Training Foundation Models with a Single Algorithm
The paper folds pretraining, fine-tuning, and RL into one likelihood-based algorithm. Training becomes a choice of data and timing, not a choice of algorithms.
The claim
Pretraining, fine-tuning, and RL are one likelihood-based algorithm.
The problem
Foundation models train in stages with different data and algorithms.
The separation makes training a choice of procedures, not data.
How it works
RL objectives become maximum-likelihood fits to model-generated target distributions.
All stages then run under one likelihood-based algorithm.
Staged training appears as a special case of that view.
A single trainer mixes static data with RL generations.
The lifecycle reduces to choosing data, proportions, and timing.
Figures
What changes
Unified training improves the accuracy-compute tradeoff and task coverage.
Broader capabilities stay intact, and continual learning starts stronger.
Limits
The evidence comes from controlled and real-world experiments in the paper.
The abstract reports no single headline number for the gain.
The abstract names no specific dataset, model size, or metric.
Source · arXiv