Softmax Routing Dynamics in Mixture-of-Experts Models: Collapse and Recovery
Softmax routing in mixture-of-experts models can concentrate probability on a subset of experts, suppressing access to the others.
Explainer · on the wire
The explainer for this story is still on the wire.
The claim, the mechanism, and the figures land here once generated.
Source · arXiv