Reference
Metric Definitions
- Partial Completion — mean root-node score across tasks (graded progress).
- Success Rate — fraction of tasks scoring 1.0 (end-to-end completion).
- Complex-A — mean leaf score over action nodes (UI execution complexity).
- Complex-P — mean leaf score over perception nodes (visual reasoning).
- Time / Output — average end-to-end wall-clock time and output length per task.
Evaluation Protocol
- Evaluation set — a fixed set of 198 CAP tasks. The 192-task public split is released in full; the remaining 228 tasks are held out against test-set contamination.
- Commercial agents — accessed through official interfaces with default configurations during November–December 2025. CAP runs on the live web, so absolute scores drift as websites evolve; always report your evaluation date.
- Open-source baselines — Browser-Use with GPT-5, Claude-4.5-Sonnet, and DeepSeek-V4-Flash backbones; default configurations, max 50 reasoning–action steps per task.
- Judge — verifiable agent-as-a-judge with a GPT-4o backbone; 96% agreement with human graders (κ=0.84), and system ranking is preserved across 5 alternative judge backbones.
- Human baseline — twelve CS graduate students, one-hour limit per task, scored by the same judge and rubrics as the agents.
How to Submit
To add a new system to this leaderboard,
open a GitHub issue with the
leaderboard label and include:
- A short system description and the agent / model versions used.
- The four CAP metrics (Partial Completion, Success Rate, Complex-A, Complex-P) with standard deviations.
- Average wall-clock time and output length per task.
- A link to your evaluation logs (preferred) or a brief description of the evaluation setup.
- Your evaluation date — CAP runs against the live web, so scores are only comparable with their time window.
We review and merge submissions on a rolling basis.