Ảnh chụp màn hình dự án
Tổng quan
EvalScope is an open-source evaluation framework built by the ModelScope Community designed to assess large model capabilities, perform inference performance stress testing, and visualize results. It supports a wide array of model types including Large Language Models (LLM), Vision Language Models (VLM), Embedding, Reranker, and AIGC. The framework integrates multiple evaluation backends like OpenCompass, VLMEvalKit, and RAGEval, and provides built-in industry-recognized benchmarks such as MMLU, C-Eval, and GSM8K. Additionally, it offers an interactive WebUI dashboard for multi-dimensional model comparison and in-depth analysis.
Tính năng chính
- Comprehensive built-in evaluation benchmarks (MMLU, C-Eval, GSM8K, etc.).
- Multi-modal and multi-domain support (LLM, VLM, Embedding, Reranker, AIGC).
- Multi-backend integration (Native, OpenCompass, VLMEvalKit, RAGEval).
- Agent Evaluation Mode with multi-turn AgentLoop, pluggable strategies, tools, and Docker sandbox.
- Inference performance stress testing with metrics like TTFT, TPOT, and throughput.
- Interactive WebUI visualization dashboard for model comparison and report inspection.
- Arena Mode for multi-model pairwise battles and intuitive ranking.
- Highly extensible architecture for custom datasets, models, and metrics.
Yêu cầu, cài đặt và bắt đầu nhanh
Cách sử dụng
Khả năng tương thích của mô hình và trường hợp sử dụng
Supports any OpenAI API-compatible model service, Anthropic Claude API, and local models hosted on ModelScope.
Ghi chú về giấy phép và rủi ro
Apache-2.0
Editorial verification 2026-08-09: repository URL, owner, description, license and repository statistics were reviewed. License metadata: Apache-2.0. README was fetched for the channel draft; re-check repository dependencies, releases and model terms before production use.
Phát hành và bảo trì
Recent updates include the addition of Claw-Eval, ResearchRubrics, Toolathlon, TVBench, WideSearch, and PerspectiveGap benchmarks. Added CharXiv, BabyVision, ERQA, and WorldVQA multimodal benchmarks. Major agent and code evaluation expansion with BigCodeBench, SWE-bench Multilingual, BrowseComp, MCP-Atlas, and GDPval. Refactored RAG evaluation module to MTEB 2.x and RAGAS 0.4.x. Introduced External Agent Bridge mode for off-the-shelf agent CLIs. Replaced Gradio WebUI with React + Vite interface.