Where Do Vision Language Models Fail at 3D Reasoning?
Vision-language models get judgments right but reuse 3D properties inconsistently. On per-color counting, humans reach 93.3% while the best model reaches 7.2%.
The claim
Vision-language models can get single judgments right, yet they fail to recover and consistently reuse exact 3D object properties.
The problem
Models can answer single judgments about a 3D object correctly one at a time.
Yet they cannot recover and reuse the object's exact properties across questions.
Accuracy alone hides these consistency failures.
How it works
BedBench builds thirteen controlled tasks with procedural scenes so exact properties stay checkable.
Tasks span individual judgments and composite answers.
The study tests twelve open-weight and four frontier models.
Probes check whether exact counts reach the language model's input.
Image-removal controls test answers without visual evidence.
Figures
What changes
Humans reach 93.3% on per-color counting; the best model reaches 7.2%.
Fine-tuning lifts one model from 3.0% to 68.6%, but answers stay inconsistent.
A probe recovers 65.0% of counts that the model's answers miss.
Limits
The paper reports no figures for reasoning or scaling gains.
Findings come from thirteen generated tasks, not open-world scenes.
The abstract reports no results for video or dynamic scenes.
Source · arXiv