SCARL: Sequence Confidence-Aligned Reinforcement Learning
RLVR treats a lucky success and a confident one the same. SCARL reshapes each advantage by the confidence gap: +2.4 Pass@32 over DAPO on Qwen3-4B-Base.
The claim
SCARL ties each RLVR advantage to the policy confidence behind it.
The problem
RLVR rewards success but ignores how confident the policy was.
A lucky find and a certain answer earn the same update.
Confident errors get no extra correction.
How it works
SCARL measures the gap between group-relative confidence and outcome.
It reshapes each advantage by that alignment gap.
Advantages are never re-signed, only redistributed.
Misaligned samples absorb most of the update force.
Repairing the gap also slows entropy collapse.
Figures
What changes
On Qwen3-4B-Base, average Pass@32 rises 2.4 points over DAPO.
It improves 14 of 16 settings across estimators and backbones.
Policy entropy stays healthy while confidence separates from correctness.
Limits
The headline gain is 2.4 points on one model family.
The method adds one hyperparameter to tune.
It adds no extra forward passes.
Source · arXiv