29 May 2025 3 min read

AI Learns to Trust Itself: How Confidence Alone Can Boost Reasoning Skills

Imagine taking an exam where you can’t ask for help or look up answers. You’d rely on your own reasoning, revising your approach until you feel confident in your solution. Confidence isn’t a guarantee of correctness, but it’s often the only intrinsic signal we have to guide us. Now, researchers at Carnegie Mellon University have applied this same principle to AI, showing that reinforcing a model’s confidence in its own reasoning can significantly improve its performance—without any external feedback.

The Problem with Reward Engineering

Reinforcement learning (RL) has powered some of the most impressive advances in AI, from game-playing agents to language models that solve complex math and coding problems. But RL relies on a critical component: the reward function. Designing effective rewards is notoriously difficult, especially in open-ended domains like reasoning, where ground-truth answers may be scarce or unavailable.

Traditional approaches reward models based on correctness—did the AI get the answer right? But this requires labeled data, which isn’t always practical. In real-world scenarios, supervision is often limited, and models must learn to improve without explicit feedback.

Enter RENT: Reinforcement Learning via Entropy Minimization

In a new paper, researchers propose RENT, a fully unsupervised RL method that uses the model’s own confidence as a reward signal. Here’s how it works:

Confidence as Reward: Instead of relying on external correctness, RENT measures the model’s uncertainty via entropy—a statistical concept that quantifies how “peaked” or “diffuse” a probability distribution is. Lower entropy means the model is more confident in its predictions.
Optimizing for Certainty: By reinforcing chains of thought that yield high confidence (i.e., low entropy), the model learns to generate more certain—and often more accurate—responses.
Focus on Critical Tokens: Not all parts of a response are equally important. The team found that minimizing entropy over tokens near the end of the reasoning chain (especially those corresponding to the final answer) correlates most strongly with improved accuracy.

Figure: RENT uses the model’s confidence (negative entropy) as an intrinsic reward for reinforcement learning.

Does It Actually Work?

The results are striking. Across multiple benchmarks—including GSM8K (grade-school math), MATH500 (competition math), AMC/AIME (high-school math Olympiads), and GPQA (PhD-level science questions)—RENT consistently improved reasoning performance. Key findings:

Scales Across Models: The method worked across model families (Qwen and Mistral) and sizes, from 1.5B to 7B parameters.
Outperforms Format Rewards: One might suspect the model is just learning to format answers correctly. But RENT outperformed a baseline that only rewarded proper formatting, proving it’s learning meaningful reasoning.
Beats Majority Voting: Compared to TTRL (a concurrent method that uses majority voting as a reward), RENT performed similarly on most tasks and significantly better on challenging problems like AIME, where sparse rewards struggle.

Why Confidence Matters

The team also analyzed which tokens are most important to optimize. Turns out, minimizing entropy over the last few tokens (where the model commits to an answer) has the highest correlation with accuracy. Early tokens in the response? Not so much. This suggests that as the model nears its final answer, it increasingly relies on its own confidence to guide reasoning.

Limitations and Caveats

Of course, unsupervised learning isn’t a silver bullet. The model can still be confidently wrong, and overconfidence is a known issue with language models. But empirically, the researchers found that confidence and accuracy are strongly correlated—so while the method isn’t perfect, it’s a promising step toward self-improving AI.

The Big Picture

RENT opens the door to unsupervised RL in domains where labeled data is scarce. By teaching models to “trust their gut,” we might unlock new ways for AI to refine its reasoning—without constant human oversight. As the paper concludes:

“We are excited about the possibility of using entropy minimization and, more broadly, unsupervised reinforcement learning to improve the capabilities of machine learning models in regimes where external supervision is unavailable.”

For AI, it seems, confidence isn’t just a feeling—it’s a strategy.