Captures d'écran du projet
Vue d’ensemble
EvalScope is an open-source evaluation framework built by the ModelScope Community designed to assess large model capabilities, perform inference performance stress testing, and visualize results. It supports a wide array of model types including Large Language Models (LLM), Vision Language Models (VLM), Embedding, Reranker, and AIGC. The framework integrates multiple evaluation backends like OpenCompass, VLMEvalKit, and RAGEval, and provides built-in industry-recognized benchmarks such as MMLU, C-Eval, and GSM8K. Additionally, it offers an interactive WebUI dashboard for multi-dimensional model comparison and in-depth analysis.
Fonctionnalités clés
- Comprehensive built-in evaluation benchmarks (MMLU, C-Eval, GSM8K, etc.).
- Multi-modal and multi-domain support (LLM, VLM, Embedding, Reranker, AIGC).
- Multi-backend integration (Native, OpenCompass, VLMEvalKit, RAGEval).
- Agent Evaluation Mode with multi-turn AgentLoop, pluggable strategies, tools, and Docker sandbox.
- Inference performance stress testing with metrics like TTFT, TPOT, and throughput.
- Interactive WebUI visualization dashboard for model comparison and report inspection.
- Arena Mode for multi-model pairwise battles and intuitive ranking.
- Highly extensible architecture for custom datasets, models, and metrics.
Prérequis, installation et démarrage rapide
Utilisation
Compatibilité des modèles et cas d’usage
Supports any OpenAI API-compatible model service, Anthropic Claude API, and local models hosted on ModelScope.
Licence et notes sur les risques
Apache-2.0
Editorial verification 2026-08-09: repository URL, owner, description, license and repository statistics were reviewed. License metadata: Apache-2.0. README was fetched for the channel draft; re-check repository dependencies, releases and model terms before production use.
Publication et maintenance
Recent updates include the addition of Claw-Eval, ResearchRubrics, Toolathlon, TVBench, WideSearch, and PerspectiveGap benchmarks. Added CharXiv, BabyVision, ERQA, and WorldVQA multimodal benchmarks. Major agent and code evaluation expansion with BigCodeBench, SWE-bench Multilingual, BrowseComp, MCP-Atlas, and GDPval. Refactored RAG evaluation module to MTEB 2.x and RAGAS 0.4.x. Introduced External Agent Bridge mode for off-the-shelf agent CLIs. Replaced Gradio WebUI with React + Vite interface.