Youtu-Parsing-Omni: One Encoder, One Schema, All Modalities
One encoder and one schema parse documents, images, audio and video. A 5B checkpoint scores 96.96 overall on OmniDocBench v1.6.
The claim
One encoder and one schema turn documents, images, audio and video into records.
The problem
Agents need evidence in documents, images, audio and video.
Most parsers ship a separate model for each task and modality.
Cross-modal fusion and timing usually happen outside the encoder.
How it works
Youtu-Omni-Encoder starts from a pretrained language model.
Shared weights encode every modality in one space-time coordinate system.
A few fusion layers let video frames attend to audio.
OmniSchema gives one output schema to ten parsing tasks.
OSAD trains the model on its own rollouts against itself.
Figures
What changes
One 5B checkpoint reaches 96.96 overall on OmniDocBench v1.6.
It leads open-weight models on OmniParsingBench with 75.08 average.
The advantage concentrates on audio and video inputs.
Limits
The paper reports benchmark scores, not downstream agent task results.
Advantage concentrates on audio and video, so other modalities gain less.
Code and weights are public, so others can rerun the checks.
Source · arXiv