V-LUMEN: Visual Lookup Memory for Embedding Scaling in Vision-Language Models
Scales VLM capacity via lookup memory instead of deeper/wider dense models
The explainer · 3-minute read · 275 words
The claim
A lookup table for visual embeddings adds capacity to vision-language models without more compute.
The problem
Embedding-level scaling adds capacity through lookup memory, not bigger models.
Visual features are continuous and spatial, unlike discrete text tokens.
That mismatch makes text-style lookup hard to reuse for images.
How it works
A frozen codebook discretizes the visual features.
Spatial aggregation hashing builds memory keys that keep location information.
Text-conditioned relevance signals then choose which keys matter now.
Addressing stays non-parametric; lookup trains nothing new.
Retrieved embeddings join the backbone through a gated residual pathway.
Figures
What changes
The memory improves a frozen Qwen3-VL-2B backbone on five reasoning benchmarks.
It transfers zero-shot to three benchmarks whose images are unseen.
Moving adapter capacity into the memory preserves accuracy on two backbones.
Limits
Lookup cost stays constant in table size and runs once at prefill.
The capacity grows in storage rather than per-token compute.
Results cover the tested backbones; wider transfer is still unproven.
Source · arXiv wire