Every Token Needs Its Own Teacher: Multi-teacher Adaptive On-policy Self-distillation
Adapts the self-distillation teacher token by token during on-policy training.
Explainer · on the wire
The explainer for this story is still on the wire.
The claim, the mechanism, and the figures land here once generated.
Source · arXiv