Dobb: Shard-Local Preconditioning in Block-Diagonal Eigenbases
Training language models whose parameters and optimizer state exceed a single node's memory requires sharding these tensors across devices.
Explainer · on the wire
The explainer for this story is still on the wire.
The claim, the mechanism, and the figures land here once generated.
Source · arXiv