EvaOmni Operational

EvaOmni is an open-source benchmark for evaluating omni-modal large models, developed by Shanghai Jiao Tong University, Fudan University, Tencent YouTu, and others. It contains 392 real-world videos and 1,027 manually annotated question-answer pairs, covering 15 capabilities and 3 reasoning levels. EvaOmni requires models to provide both an answer and audio/video evidence intervals, and scores them using answer accuracy, evidence coverage, and bimodal localization. Among 24 models tested by EvaOmni, the strongest, Gemini3.7-Flash, achieved 79.4% accuracy but only 35.1% bimodal localization, falling 59.2 percentage points short of human experts at 94.3%. This exposes the weakness of getting the answer right without being able to explain the evidence.

Answer evaluation: Determines whether a model answers audiovisual multiple-choice questions correctly and reports accuracy, Acc.
Evidence annotation and output: Human annotators mark the audio and video intervals supporting each answer, and models must return bimodal evidence intervals together with their answers.
Evidence coverage score (C.G.): Measures the proportion of human-annotated evidence segments hit by the model's predicted intervals, indicating how many supporting clues the model found.
Bimodal bottleneck localization (B.G.): Computes temporal IoU separately for audio and video and uses the weaker modality to determine the score, requiring accurate localization in both modalities.
Three-level capability hierarchy: Tasks are divided into retrieval and localization, basic understanding, and complex reasoning to assess the progression from point-level alignment to multi-hop reasoning.
Shortcut filtering: Text-only, audio-only, and vision-only inputs are used to screen questions, removing samples that can be answered correctly using only one modality or textual common sense.
Cross-model evaluation: Provides a unified benchmark and leaderboard for 24 omni-modal models, supporting comparisons and selection among open-weight and proprietary models.

Follow WeChat and reply “开源” to join the AI open-source project group
Evidence-aware annotation structure: EvaOmni annotates both audio and video evidence intervals for every question and provides a set of distractor options designed around the video content. Model output is therefore defined as a triplet: the selected option, a set of audio timestamps, and a set of video timestamps. The evaluation program can independently verify answer correctness and evidence localization quality.
Four-stage data construction pipeline: Data production consists of video screening, manual annotation, cross-review, and shortcut filtering. Videos containing hard-coded subtitles or extensive text overlays are first excluded to ensure that audiovisual clues are genuine and effective. Experts write questions, design distractors tied to the video content, and annotate evidence intervals in both modalities. Other annotators then cross-review capability categorization, evidence quality, and shortcut issues. Contested samples are independently rechecked by two additional experts. Models answer using text-only, audio-only, and vision-only inputs; any question answerable correctly from a single modality is revised and re-annotated, blocking shortcuts that bypass joint audiovisual reasoning at the data level.
Coverage-localization dual-metric scoring: The evidence coverage score, C.G., checks whether each human-annotated evidence segment has positive-duration overlap with any predicted interval, measuring whether the evidence was found. The bottleneck localization score, B.G., takes the union of all predicted intervals and annotated intervals within each modality, calculates temporal intersection over union, IoU, and uses the weaker of the audio and video IoUs as the final score. Both metrics are multiplied by answer correctness, so an incorrect answer receives zero. This design means broad predictions can earn coverage points, while precise boundaries contribute to localization; using the weaker modality prevents the stronger modality from compensating for the weaker one.
Three-level capability hierarchy: The dataset is organized by reasoning and evidence complexity into three levels. Level 1, retrieval and localization, tests alignment of times, events, and entities across audio and video. Level 2, basic understanding, requires combining complementary audiovisual clues to understand events, emotions, and intent. Level 3, complex reasoning, requires integrating multiple pieces of evidence scattered across different time points to make multi-hop judgments. As the level increases, the number and duration of evidence segments per question increase significantly, locating capability gaps in retrieval, understanding, and long-range multi-step reasoning at specific levels.

Install the ModelScope client: Run pip install -U modelscope to install or upgrade the ModelScope SDK.
Download the dataset: Use dataset_snapshot_download with the dataset ID VisionXLab/EvaOmni-Dataset to download the data to a local directory.
Prepare model inputs: Preprocess the videos according to the evaluation protocol and include the corresponding audio.
Integrate the model under evaluation: Follow the GitHub repository instructions to connect your omni-modal model, and use the unified prompt to make it output a triplet consisting of the option, audio timestamps, and video timestamps.
Run evaluation: Call the evaluation framework to perform batch inference on all 1,027 question-answer pairs. The program automatically checks the answers and interval matches.
Review results: Read the three metrics, Acc., C.G., and B.G., along with scores for the three capability levels, and compare them with the leaderboard to identify model weaknesses.

Evidence localization is included in scoring: Models must return audio and video evidence intervals, distinguishing correct guesses from answers supported by evidence.
Dual metrics quantify evidence quality: The evidence coverage score, C.G., measures whether all clues were found, while the bottleneck localization score, B.G., uses temporal IoU and the weaker modality to determine the score, preventing the stronger modality from masking the weaker one.
Fine-grained three-level capability hierarchy: Tasks progress from retrieval and localization to basic understanding and complex multi-hop reasoning, pinpointing specific weaknesses at different reasoning depths.
High-quality manual annotation: Questions are written by experts, cross-reviewed, and rechecked by two experts for contested samples, making the data more reliable than automatically constructed benchmarks of the same kind.
Strict shortcut filtering: Text-only, audio-only, and vision-only inputs are used to screen questions, removing samples answerable without joint audiovisual reasoning and ensuring fair evaluation.
Reveals the real capability gap: Tests show that the strongest model's answer accuracy is only 17 percentage points below humans, while its bimodal localization lags by 59.2 percentage points, exposing the true weakness of current models.

Project website: https://xiaoxing2001.github.io/EvaOmni.github.io/
GitHub repository: https://github.com/VisionXLab/EvaOmni
Hugging Face dataset: https://huggingface.co/datasets/VisionXLab/EvaOmni-Dataset
arXiv paper: https://arxiv.org/pdf/2610.28444

Omni-modal model development and evaluation: When iterating on an Omni LLM, model teams can use Acc./C.G./B.G. to identify capability gaps where “answer quality improves but evidence localization does not,” guiding training improvements.
Enterprise AI selection and acceptance testing: In scenarios such as finance and government services that require models to “provide verifiable evidence,” buyers can use the B.G. bottleneck localization score to select genuinely reliable bimodal models.
Video content moderation and provenance: Moderation applications require models to identify “which frame and which segment of audio” contains a violation. EvaOmni's time-interval outputs and localization metrics can be used directly as acceptance criteria.
Question-answering products for education and film: Video-based learning assistants and film commentary products need to cite original clips when answering questions. This benchmark can assess whether a model can accurately locate the relevant shots and dialogue.
Academic research and leaderboard development: Researchers can use the open-source collection of 392 evidence-annotated videos to study audiovisual reasoning, temporal localization, multi-hop reasoning, and related areas, or incorporate it into a multi-benchmark evaluation system.