Applying the CaRE Protocol to Agent Benchmarks: Compute-Aware Evaluation for Skill and Memory Harnesses | AI

Back openDesk Edu for a sovereign, open-source education — every vote counts.

Loading…

Applying the CaRE Protocol to Agent Benchmarks: Compute-Aware Evaluation for Skill and Memory Harnesses | AI | graphwiz.ai

Teaser: CaRE — the compute-aware remasking evaluation protocol that reversed published MDLM rankings — has been applied to our two open-source agent benchmark harnesses: Agent Skill Bench and AMBench. Here's how the three pillars translate to LLM-based agent evaluation, what changed in the harnesses, and why single-metric leaderboards are misleading.

Introduction

In July 2026, CaRE (Compute-aware Remasking Evaluation) landed on arXiv with a damning finding: seven recent papers on masked diffusion language models (MDLMs) evaluated their remasking strategies under incompatible settings — varying step counts, metrics, and sampling temperatures without controlling these factors. When CaRE re-ran the comparisons with matched compute budgets, several published strategy rankings reversed. Temperature alone explained the majority of score variance.

The same failure mode is endemic to agent benchmarking. Leaderboards for agent skills and agent memory typically report a single headline score per model, produced from a single run with undocumented compute budgets and sampling settings. Two models with a 0.05 score gap — smaller than run-to-run variance — are ranked as if the difference were meaningful.

This article documents how we applied the CaRE protocol to our two open-source agent benchmark suites:

Agent Skill Bench — unified evaluation of 8 agent skill categories (tool use, planning, reasoning, code, web navigation, embodied, social, learning)
AMBench — agent memory systems across a 27-cell taxonomy (forms × functions × dynamics) plus temporal, multimodal, security, and multi-agent dimensions

Both harnesses now support CaRE-mode evaluation, documented in each repo's docs/care-protocol.md.

Why Agent Benchmarks Have the Same Problem

MDLM remasking and agent evaluation share a structural weakness: the "model" being compared is really a pipeline with hidden knobs.

graph LR
    subgraph Hidden[Uncontrolled Variables in Agent Eval]
        H1[Generation budget<br/>max_tokens]
        H2[Temperature]
        H3[Seed / sampling]
        H4[Number of runs]
        H5[Scoring strategy]
    end

    subgraph Effect[Distorted Outcomes]
        E1[Compute-rich models rank higher]
        E2[Rankings flip across temps]
        E3[Variance treated as signal]
        E4[Cost/latency invisible]
    end

    H1 --> E1
    H2 --> E2
    H3 --> E2
    H4 --> E3
    H1 --> E4

    classDef hidden fill:#E45756,stroke:#b33d3d,color:#fff
    classDef effect fill:#F58518,stroke:#b35a0e,color:#fff
    class H1,H2,H3,H4,H5 hidden
    class E1,E2,E3,E4 effect

Failure Mode	CaRE Finding (MDLMs)	Agent Bench Equivalents
Incompatible step budgets	NFE differences drove rankings	Different `max_tokens` per submission
Uncontrolled stochasticity	Temperature explained most MAUVE variance	Temperature unset / unreported
Single metric	Rankings reversed under matched compute	Single score per leaderboard row
No variance reporting	Mean without std hides instability	One run per model

The Three Pillars in Practice

CaRE's protocol has three pillars. Here is how each translates to our LLM-based harnesses.

Pillar 1: Standardise NFE (Number of Function Evaluations)

In MDLMs, NFE is the number of denoising steps. In agent evaluation, the analogue is the generation budget: max_tokens per task. A model with a 4096-token budget will beat one capped at 512 — even if the underlying skill is identical.

Both harnesses now take an explicit --max-tokens flag:

python src/harness.py --model gpt-4o-mini --temperature 0.0 \
  --seed 42 --max-tokens 1024 --markdown results/care-report.md

The budget is emitted in the report header and JSON config, so no submission can silently consume more compute than its peers.

Pillar 2: Control Stochasticity Explicitly

CaRE found temperature explains the majority of variance in MDLM evaluations. For LLM-based agent evaluation the equivalent knobs are temperature and seed. Both harnesses now support:

# 5 seeded runs, seeds 42..46
python src/harness.py --model gpt-4o-mini --temperature 0.0 \
  --seed 42 --runs 5 --max-tokens 1024 --markdown results/care-5runs.md

--seed sets the sampling seed (passed through to the API when the provider supports it) and acts as the seed base for sweeps
--runs N executes the complete task suite N times with seeds base + 0 .. base + N-1
Results are aggregated as mean ± std per metric — variance becomes visible instead of hidden

Pillar 3: Enforce Multi-Metric Reporting

A headline score is not enough. The CaRE report format in both harnesses reports four (skill) / three (memory) metrics simultaneously:

graph TD
    R[CaRE Report] --> M1[Score<br/>0.0-1.0 correctness]
    R --> M2[Latency (ms)<br/>response time]
    R --> M3[Tokens / NFE<br/>actual compute]
    R --> M4[Cost (USD)<br/>inference cost]

    M1 --> I1[Quality]
    M2 --> I2[Responsiveness]
    M3 --> I3[Efficiency]
    M4 --> I4[Economics]

    classDef rep fill:#4C78A8,stroke:#2c4e6e,color:#fff
    classDef metric fill:#54A24B,stroke:#3a7a35,color:#fff
    classDef insight fill:#F58518,stroke:#b35a0e,color:#fff
    class R rep
    class M1,M2,M3,M4 metric
    class I1,I2,I3,I4 insight

A model that wins the score column while consuming 4× the tokens is now visible — it can no longer hide behind a single leaderboard number.

Implementation in the Harnesses

Agent Skill Bench (`src/harness.py`)

Agent Skill Bench evaluates models against task suites across 8 skill categories using a configurable scorer (exact, keyword, llm_judge, or auto). The CaRE changes:

Change	Details
`--max-tokens`	Standardised NFE budget, default 1024, passed to the OpenAI-compatible client
`--seed`	Sampling seed + seed base, default `None` (provider default)
`--runs`	Repeated seeded executions, default 1 (legacy single-run behaviour preserved)
`to_care_markdown()`	New report generator: mean ± std per metric and per skill
`docs/care-protocol.md`	Protocol document with three pillars, usage, and leaderboard requirements

The multi-run path builds a fresh client per seed and executes the suite sequentially (or in parallel with --parallel), collecting one TaskResult list per run.

AMBench (`src/harness.py`)

AMBench evaluates memory systems across the 27-cell taxonomy. Same flags (--max-tokens 512 default, --seed, --runs) with a memory-specific CaRE report:

Change	Details
`--max-tokens`	Standardised NFE budget, default 512 (matches the existing harness budget)
`--seed` / `--runs`	Seed base and repeated executions, default single-run preserved
`to_care_markdown()`	mean ± std per metric and per taxonomy cell
CaRE + memory isolation	Compatible with `--baseline` isolation: run the same seeds against a no-memory baseline

The CaRE path is deliberately gated to the plain evaluation mode (--runs > 1 without --multi-turn/--dual-run), keeping the existing multi-turn and dual-run protocols untouched.

Sample Output

Running the skill harness in CaRE mode produces a report like this:

# Agent Skill Bench — CaRE Evaluation Report

Compute-aware, multi-metric, stochasticity-controlled evaluation
(protocol: docs/care-protocol.md).

**Model:** gpt-4o-mini
**Runs:** 3 (seeded)
**NFE budget (max_tokens):** 1024
**Temperature:** 0.0
**Seed base:** 42

| Metric | Mean ± Std |
|--------|-----------|
| Score | 0.583 ± 0.021 |
| Latency (ms) | 812.4 ± 45.2 |
| Tokens (NFE) | 2041.6 ± 118.9 |
| Cost (USD) | 0.0041 ± 0.0003 |

## Results by Skill (mean ± std across runs)

| Skill | Score | Latency (ms) | Tokens | Cost (USD) |
|-------|-------|-------------|--------|-----------|
| code-generation | 0.667 ± 0.058 | 744.0 ± 30.1 | 1987.0 ± 99.4 | 0.0040 ± 0.0002 |
| tool-use | 0.500 ± 0.000 | 881.0 ± 60.3 | 2096.0 ± 138.4 | 0.0042 ± 0.0004 |

The std column is the key addition: a 0.08 gap between two models with ±0.06 error bars is not a ranking — it's a coin flip.

How to Contribute

Both benchmarks are open source and accept submissions:

Agent Skill Bench — submit results via the leaderboard PR template (must now include max_tokens, temperature, seed, and run count)
AMBench — same CaRE requirements for memory-system submissions

CaRE-compliant submissions must include:

NFE budget (--max-tokens), standardised unless explicitly overridden
Stochasticity settings — temperature AND seed base
Run count — minimum 3 for variance reporting
Multi-metric table — score, latency, tokens, (cost)
Full harness output — no hand-picked runs

Conclusion

The CaRE protocol demonstrated for MDLMs what we suspected for agent benchmarks: evaluation settings — not just algorithms — drive leaderboard rankings. By standardising NFE budgets, controlling stochasticity via seeds, and reporting multiple metrics with variance, our harnesses now produce results where a "ranking" actually means something.

The single most important habit change for the benchmark community: stop reporting single-run, single-metric scores. Run at least three seeded executions, report mean ± std, and state your compute budget. The tools for doing this are now in both repos — the next step is that the community uses them.

Repositories:

Reference: CaRE: Compute-aware Remasking Evaluation Protocol for Masked Diffusion Language Models (arXiv:2607.24763, July 2026).