VideoMemBench: How Well Do MLLMs Remember in Video Conversations?
Video QA never tests memory across turns. VideoMemBench links turns with typed dependencies: 2,476 turns, where the best of 20+ models scores only 45.7%.
The claim
Long video conversations test memory across turns, and models fail that test.
The problem
Existing video QA questions never depend on earlier turns.
Nothing forces a model to reuse what it saw or said.
How it works
Each sample pairs one video with a long multi-turn dialogue.
Episodic dependencies refer back to an earlier visual episode.
Answer dependencies build on the model own earlier answer.
Every dependency must be necessary and free of leakage.
Three language model roles draft turns; human experts review them.
Figures
What changes
The benchmark holds 326 videos with 2,476 turns.
1,387 turns depend on earlier turns, including long call-backs.
The best of 20+ models reaches only 45.7% overall.
Limits
The benchmark covers 326 videos, still a modest sample.
Dialogue drafts come from model roles before human review.
Source · arXiv