GitHub has published an evaluation of the Copilot agentic harness, a shared component powering Copilot CLI, app, and code review. The harness orchestrates tools, context, and workflow to be fast, token-efficient, and predictable. Tests compared GitHub Copilot CLI against model-vendor harnesses (Claude Code for Claude models, Codex CLI for GPT models) using fixed models and tasks across several benchmarks.
Benchmarks Used
- SWE-bench Verified: 500 human-validated bug-fix tasks from open-source Python repos.
- SWE-bench Pro: More difficult multi-step engineering tasks.
- SkillsBench: Evaluates skill use and triggering.
- TerminalBench: Terminal-based task performance.
- Win-Hill: Internal benchmark for Windows container tasks.
Token Efficiency
Across multiple benchmarks, the GitHub Copilot harness achieved task completion rates on par with model-vendor harnesses while showing lower token consumption in most configurations. Task resolution rates were within run-to-run variance, indicating effective parity.
TerminalBench 2.0 Analysis
Variance analysis on TerminalBench 2.0 showed GitHub Copilot's harness on par or ahead in task completion and cost per task. Purple (Copilot) markers overlapped with same-model competitors within ±1σ spread. Copilot was never below a competitor on completion or to the right on cost. GPT models offered best value (strong resolution at lowest cost), while Claude Opus reached highest resolution at a premium.
Multi-Model Architecture
The harness supports 20+ frontier models (GPT, Claude, Gemini, MAI) plus bring-your-own-key for open-source/local models. Auto model selection balances task intent and model health. Cross-model critique (Rubber Duck) allows one model to review another's work.
Methodology
All runs used a two-hour timeout, non-interactive single-turn, web-tools disabled, all tools allowed. TerminalBench2 used default settings with reasoning effort medium. Each model was evaluated across five independent runs. All agent-model pairs normalized to same context window, prompt token limits, reasoning effort (medium), no tool search, no MCP servers. Infrastructure anomalies excluded. Best scored run reported as pass@1.
Conclusion
GitHub Copilot delivers task resolution on par with leading model-vendor harnesses while using fewer tokens across several configurations, without locking into a single model. Developers can get comparable task completion with lower token cost and choose the best model for each task.
