MMBench

What is MMBench?

MMBench is a multimodal benchmark jointly developed by researchers from the Shanghai Artificial Intelligence Laboratory, Nanyang Technological University, the Chinese University of Hong Kong, the National University of Singapore, and Zhejiang University. MMBench introduces a comprehensive evaluation process, progressively subdividing assessments from perception to cognition, covering 20 fine-grained abilities. It collects approximately 3,000 single-choice questions from the internet and authoritative benchmark datasets. Breaking away from the conventional question-and-answer approach, it evaluates by extracting options through rule-based matching, iteratively shuffling the options to verify the consistency of the output results, and using ChatGPT for precise matching of the model’s responses. MMBench covers various task types, such as visual question answering and image description generation, providing comprehensive performance evaluations of models based on multi-dimensional metrics. MMBench’s leaderboard showcases the performance of different models on these tasks, helping researchers and developers understand the current development level of multimodal technologies and promoting technological progress in related fields.

MMBench main functions

• Fine-grained capability assessment : Multimodal capabilities are subdivided into multiple dimensions (such as perception, reasoning, etc.), and relevant questions are designed for each dimension to comprehensively evaluate the model’s fine-grained capabilities.
• Large-scale multimodal dataset : Provides approximately 3,000 multiple-choice questions, covering 20 ability dimensions, and supports performance testing of the model in various scenarios.
• Innovative evaluation strategy : Adopting a “cyclic evaluation” strategy, the stability of the model is tested through multiple cyclic inferences, reducing the impact of noise and providing more reliable evaluation results.
• Multilingual support : Provides datasets in English and Chinese, supporting the evaluation of the model’s capabilities in different language environments.
• Data visualization : Supports visualization of data samples, helping users better understand data structure and content.
• Official evaluation tool : VLMEvalKit is provided, which supports standardized evaluation of multimodal models and can be used to submit test results to obtain accuracy.
• Benchmarking and Leaderboard : The leaderboard showcases the performance of different models on the MMBench dataset, providing a reference for researchers.