The presence of reasoning behaviors, not the correctness
of answers is the critical factor. Models with incorrect solutions containing proper reasoning patterns achieve comparable performance to those trained on correct solutions.

It seems that the presence of cognitive behaviors enables self-improvement through RL.

Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs
https://arxiv.org/abs/2503.01307

#reinforcementlearning #RL
#AI #DL #LLM

arXiv.orgCognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRsTest-time inference has emerged as a powerful paradigm for enabling language models to ``think'' longer and more carefully about complex challenges, much like skilled human experts. While reinforcement learning (RL) can drive self-improvement in language models on verifiable tasks, some models exhibit substantial gains while others quickly plateau. For instance, we find that Qwen-2.5-3B far exceeds Llama-3.2-3B under identical RL training for the game of Countdown. This discrepancy raises a critical question: what intrinsic properties enable effective self-improvement? We introduce a framework to investigate this question by analyzing four key cognitive behaviors -- verification, backtracking, subgoal setting, and backward chaining -- that both expert human problem solvers and successful language models employ. Our study reveals that Qwen naturally exhibits these reasoning behaviors, whereas Llama initially lacks them. In systematic experimentation with controlled behavioral datasets, we find that priming Llama with examples containing these reasoning behaviors enables substantial improvements during RL, matching or exceeding Qwen's performance. Importantly, the presence of reasoning behaviors, rather than correctness of answers, proves to be the critical factor -- models primed with incorrect solutions containing proper reasoning patterns achieve comparable performance to those trained on correct solutions. Finally, leveraging continued pretraining with OpenWebMath data, filtered to amplify reasoning behaviors, enables the Llama model to match Qwen's self-improvement trajectory. Our findings establish a fundamental relationship between initial reasoning behaviors and the capacity for improvement, explaining why some language models effectively utilize additional computation while others plateau.

**ma𝕏pool** @maxpool@mathstodon.xyz · Mar 5

**Ben Lorica 罗瑞卡** @bigdata@indieweb.social · Mar 4

**Stuart Spence** @StuartSpence@mstdn.ca · Feb 18

**Tero Keski-Valkama** @tero@rukii.net · Feb 17

**Tero Keski-Valkama** @tero@rukii.net · Feb 13

Feb 13

Tero Keski-Valkama @tero@rukii.net

How to formulate exploration-exploitation trade-off better than all the hacks on top of Bellman equation?

We can first of all simply estimate the advantage of exploration by Monte-Carlo in a swarm setting: Pitting fully exploitative agents against fully exploitative agents which have the benefit of recent exploration. This can be easily done by lagging policy models.

Of course the advantage of exploration needs to be divided by the cost of exploration, which is linear to the number of agents used in the swarm to explore at a particular state.

Note that the advantage of exploration depends on the state of the agent, so we might want to define an explorative critic to estimate this.

What's beautiful in this formulation is that we can incorporate autoregressive #WorldModels naturally, as the exploitative agents only learn from rewards, but the explorative agents choose their actions in a way which maximizes the improvement of the auto-regressive World Model.

It brings these two concepts together as sides of the same coin.

Exploitation is reward-guided action, exploration is auto-regressive state transition model improvement guided action.

Balancing the two is a swarm dynamic which encourages branching where exploration has an expected value in reward terms. This can be estimated by computing the advantage of exploitative agents utilizing recent exploration versus agents which do not, and returning this advantage to the points of divergence between the two.

#mathematics #ReinforcementLearning #RL

**Trending Stocks** @stonkz@mastodon.social · Feb 8

Continued thread

**Tero Keski-Valkama** @tero@rukii.net · Jan 29 *

Jan 29 *

Tero Keski-Valkama @tero@rukii.net

Instead of "model is itself the environment" in an #LLM setting, you can take note that in normal #RL, you'd typically have state-action-reward-state-action-reward-... sequences, where the action inflicts itself upon the environment which changes and its new form projects into the next state.

For LLMs, there's only the outcome which comes from the environment. That's the final reward. Before that it's just auto-regressive action-action-action-...

So this makes advantage computation heavier than necessary and instead the advantage can be just returned backwards over the completion tree alternatives.

**Vasilis Valatsos** @aethrvmn@sigmoid.social · Jan 14

Jan 14

Vasilis Valatsos @aethrvmn@sigmoid.social

Looking into Asset Liability Management amongst other things for work (and how #rl can be used), it seems like the #private #insurance industry is an unregulated #banking industry. They take the insurances, invest them in the market and then try to figure out the timing between people asking them for the money and cashing out on their investments.

So why are people in the US insured? Is it a ritualistic cultural thing?

Recent searches

Search options

Administered by:

Server stats:

#rl