Collective Dynamics of Self-Reinforcement: How Cross-Task Coupling Drives Policy Collapse in Unsupervised RLVR
Label-free RLVR rewards a model for agreeing with itself, then collapses to one answer. A collective-instability model explains why collapse grows with the number of problems.
The claim
Label-free RLVR collapses once shared self-reward grows faster than per-problem sharpening.
The problem
Label-free RLVR rewards a language model for agreeing with its own majority vote.
Reasoning improves at first, then self-assessed reward saturates while accuracy falls.
Collapse appears only when training uses many problems at once.
How it works
The paper studies a solvable model where many problems share parameters.
Shared coupling amplifies any perturbation at a rate set by lambda.
Per-problem sharpening slows down as the problem count grows.
Past a critical problem count, the shared mode wins the race.
Removing the shared direction prevents collapse in a small Transformer.
Figures
What changes
Batch size shifts the collapse point only logarithmically.
Models that start confident show no collapse at all.
A marginal-entropy bonus stabilizes the uniform collective mode.
Limits
Much of the analysis rests on a solvable many-problem model.
The experiments run on one small Transformer, not a frontier model.
The theory covers label-free rewards, so labelled RLVR stays outside it.
Source · arXiv