Evals as a release gate: the harness that runs before every deploy

Ordinary software has tests. A function either returns the right thing or it does not, and a red test blocks the merge. An agent is not a function. Ask it the same question twice and you may get two answers, both reasonable, one wrong. Change the prompt, the model, a tool description or the retrieval index and the behaviour shifts in ways no unit test will catch.

So we treat every release as a statistical claim: on this set of cases, the new version is at least as good as the old one, and it does not break the things we promised. The harness is how we make that claim, and CI is where it gets to say no.

Five layers#

Each layer answers a different question, and they fail in different ways.

The unit layer is plain tests and needs no explanation. The others are where the design lives.

The golden set#

A golden case is an input with an expected outcome that a person has signed off. Not an expected transcript: the words can vary. The outcome is structured: the tool that should be called, the fields that should be extracted, the category that should be assigned, or "hand off".

Cases are files, in the repo, next to the code that they test.

suite: appointments
threshold: 0.95
cases:
  - id: move-to-named-day
    input:
      transcript: "Hi, I need to move my appointment to next Tuesday if you can."
      context: { customer_id: "c_4821", timezone: "Asia/Dubai" }
    expect:
      tool: reschedule
      args: { customer_id: "c_4821", day: "tuesday" }
  - id: ambiguous-day
    input:
      transcript: "Can I move it to later in the week?"
    expect:
      tool: clarify
  - id: distress-handoff
    input:
      transcript: "I really can't deal with this right now, my mother is in hospital."
    expect:
      tool: handoff
      reason: distress

Every exception a reviewer resolves in production becomes a candidate case here. That is the loop: the queue feeds the golden set, the golden set guards the next release.

Handling non-determinism#

Run each case more than once. We use five runs by default, at the production temperature, and score the case as the fraction of runs that passed. A case that passes three times out of five is not "passing"; it is a case the agent is unsure about, and the report says so.

The judge model, for rubric scoring, runs at temperature zero with a fixed rubric and a fixed model version. Judges drift too, so the judge is pinned in the config and changing it is itself a release.

The runner#

The runner is a thin layer over pytest, so it fits in CI without inventing anything. Each suite becomes a parametrised test; each case runs N times; the suite asserts on the aggregate.

import yaml, pytest, statistics
from pathlib import Path
from agent import run_agent          # the same entrypoint production uses
 
RUNS = 5
 
def load_suites():
    for f in Path("evals/golden").glob("*.yaml"):
        yield yaml.safe_load(f.read_text())
 
def matches(expected: dict, actual: dict) -> bool:
    for k, v in expected.items():
        if isinstance(v, dict):
            if not matches(v, actual.get(k, {})): return False
        elif actual.get(k) != v:
            return False
    return True
 
@pytest.mark.parametrize("suite", load_suites(), ids=lambda s: s["suite"])
def test_suite(suite, report):
    scores = {}
    for case in suite["cases"]:
        passes = 0
        for _ in range(RUNS):
            out = run_agent(**case["input"])
            passes += matches(case["expect"], out)
        scores[case["id"]] = passes / RUNS
    mean = statistics.mean(scores.values())
    report.record(suite["suite"], mean, scores)
    unsure = [k for k, v in scores.items() if 0 < v < 1]
    assert mean >= suite["threshold"], (
        f"{suite['suite']}: {mean:.3f} below threshold {suite['threshold']} "
        f"(unstable cases: {unsure})"
    )

The report fixture writes a JSON summary that the CI job uploads as an artifact and posts as a comment on the pull request, so the reviewer sees the numbers next to the diff rather than digging through logs.

Thresholds and noise#

A fixed threshold is a blunt tool, and on small suites it is noisy. Two rules keep it honest:

• Compare against the base branch, not just the threshold. The job runs the suite on main and on the branch, and fails if the branch is worse by more than the noise band, even if both are above threshold. A slow decline across ten merges is the failure mode this catches.
• Estimate the noise band from repeated runs of the same version. Run the suite three times on the same commit and take the spread. That is your band. Anything inside it is not a signal.

Wiring it into CI#

The job is unremarkable, which is the point. It needs access to the model endpoint, a pinned judge, and enough time. We run the unit and golden layers on every pull request, and the rubric, safety and replay layers on the merge to main and nightly.

name: evals
on: [pull_request]
jobs:
  golden:
    runs-on: [self-hosted, gpu]
    steps:
      - uses: actions/checkout@v4
      - run: pip install -r requirements.txt
      - name: Serve the candidate model
        run: ./scripts/serve.sh --model ${{ github.sha }} --port 8000 &
      - name: Unit and golden suites
        run: pytest evals/ -m "unit or golden" --report=evals-report.json
        env:
          MODEL_URL: http://localhost:8000/v1
          JUDGE_MODEL: judge-2026-05   # pinned; changing this is its own PR
      - uses: actions/upload-artifact@v4
        if: always()
        with: { name: evals-report, path: evals-report.json }

The replay layer#

The most valuable cases are the ones nobody wrote. Every week a sample of real production items, with the outcomes the reviewers confirmed, is exported with identifiers stripped and added to the replay set. The replay set is large and slow, and it is the closest thing we have to the truth: the agent's own history, re-run against its new self.

What a red result means#

A red eval is not "the model is bad". It is "this change moved the distribution in a direction we did not expect". Sometimes the fix is in the change. Sometimes the golden case was wrong, or the world changed and the case needs updating, which is a decision a person makes and signs. Either way the release waits.

That is the trade. The harness slows down shipping by an hour. It has saved us from shipping something wrong to a live phone line more than once, and one of those is worth all the hours.