What Survives Distillation? A Mechanistic Interpretability Study of GPT-2 and DistilGPT-2
Mechanistic study of what GPT-2 internals survive depth-reduction distillation into DistilGPT-2.
Explainer · on the wire
The explainer for this story is still on the wire.
The claim, the mechanism, and the figures land here once generated.
Source · arXiv wire