Lost in the Latent Space: How Code Embeddings Represent Functional Intent and Implementation Fidelity
Across 14 code embedding models, style fidelity trades off against semantic generalization. Similar retrieval scores hide very different behaviour.
The claim
Across 14 code embedding models, style fidelity trades off against semantic generalization.
The problem
Code embeddings are a key building block for many software engineering tools.
Existing evaluations mostly report downstream task performance scores.
Little is known about what a single embedding actually encodes inside.
How it works
The study probes 14 state-of-the-art code embedding models in total.
Supervised probes test both functional abstraction and implementation fidelity.
Zero-shot retrieval experiments measure geometric similarity between pieces of code.
Controlled edits change style while preserving semantics exactly.
Each model's response to the same semantics-preserving edit is recorded.
Figures
What changes
Models that keep style and structure signals lose semantic generalization.
Similar retrieval scores hide very different responses to the same edit.
No single universal code representation appears within reach for practitioners.
Limits
The evidence is empirical and covers 14 released embedding models.
Probes measure representations, not the outcomes users see inside tools.
Model choice must follow the intended use, the authors argue.
Source · arXiv