Screenshot del progetto
Panoramica
EvalScope is an open-source evaluation framework built by the ModelScope Community designed to assess large model capabilities, perform inference performance stress testing, and visualize results. It supports a wide array of model types including Large Language Models (LLM), Vision Language Models (VLM), Embedding, Reranker, and AIGC. The framework integrates multiple evaluation backends like OpenCompass, VLMEvalKit, and RAGEval, and provides built-in industry-recognized benchmarks such as MMLU, C-Eval, and GSM8K. Additionally, it offers an interactive WebUI dashboard for multi-dimensional model comparison and in-depth analysis.
Funzionalità principali
- Comprehensive built-in evaluation benchmarks (MMLU, C-Eval, GSM8K, etc.).
- Multi-modal and multi-domain support (LLM, VLM, Embedding, Reranker, AIGC).
- Multi-backend integration (Native, OpenCompass, VLMEvalKit, RAGEval).
- Agent Evaluation Mode with multi-turn AgentLoop, pluggable strategies, tools, and Docker sandbox.
- Inference performance stress testing with metrics like TTFT, TPOT, and throughput.
- Interactive WebUI visualization dashboard for model comparison and report inspection.
- Arena Mode for multi-model pairwise battles and intuitive ranking.
- Highly extensible architecture for custom datasets, models, and metrics.
Requisiti, installazione e avvio rapido
Utilizzo
Compatibilità del modello e casi d'uso
Supports any OpenAI API-compatible model service, Anthropic Claude API, and local models hosted on ModelScope.
Note su licenza e rischi
Apache-2.0
Editorial verification 2026-08-09: repository URL, owner, description, license and repository statistics were reviewed. License metadata: Apache-2.0. README was fetched for the channel draft; re-check repository dependencies, releases and model terms before production use.
Rilascio e manutenzione
Recent updates include the addition of Claw-Eval, ResearchRubrics, Toolathlon, TVBench, WideSearch, and PerspectiveGap benchmarks. Added CharXiv, BabyVision, ERQA, and WorldVQA multimodal benchmarks. Major agent and code evaluation expansion with BigCodeBench, SWE-bench Multilingual, BrowseComp, MCP-Atlas, and GDPval. Refactored RAG evaluation module to MTEB 2.x and RAGAS 0.4.x. Introduced External Agent Bridge mode for off-the-shelf agent CLIs. Replaced Gradio WebUI with React + Vite interface.