Review the agent,
not just the diff.

diff0 compares how your Eve agent behaves across two committed refs—so captured behavioral changes can show up before you merge them.

See the latest live comparison View the source

diff0 / showcase/live-proof-v3

comparison complete

source change

agent/instructions.md

− After computing a figure, delegate a one-line executive summary to the

− `reporter` subagent before replying.

+ After computing a figure, give a one-line executive summary before replying.

Eval observations60 / 60 passed

behavioral comparison

Drift detected

review

reporter subagent10 / 10 runs0 / 10 runs

eval observations30 / 3030 / 30

output tokens / run1,017675

duration / run20.2s11.8s

Fisher exact + Holm adjustment·N=10 per ref

One instruction simplified · 20 real-model suites · automated in GitHub Actions

Eve-native

Reads captured Eve eval JSON and events

Two refs

Base behavior vs. head behavior

N runs

Evidence, not anecdotes

One comment

The result lives in your PR

The gap after evals

Your eval passed.
Your agent still changed.

A scorer tells you whether the answer was acceptable. diff0 shows captured evidence about how the agent got there—and whether that behavior moved with your code.

A tiny edit

A process requirement becomes a direct-output requirement.

instructions.md

− Delegate summary to reporter

+ Give a one-line summary

Green evals

The head clears every eval observation. Repetition can reveal what one green run misses.

total-revenuepass

no-failed-actionspass

uses-sql-toolpass

Visible drift

Repeated runs reveal that delegation vanished completely.

reporter subagent

10 / 100 / 10

How it works

Same repo.
Both realities.

diff0 checks out both committed refs, runs each ref's Eve eval suite, and flags evaluator changes that would make the outcomes incomparable.

01
Checkout
Base and head run in isolated git worktrees. A dirty local HEAD is rejected so edits are never silently omitted.
02
Run
Existing Eve evals run N times per ref in counterbalanced AB/BA order.
03
Collect
Tool calls, subagents, skills, tokens, available cost, and duration come from Eve's own artifacts.
04
Report
Statistical changes and stable behavioral drift land in the terminal and one sticky PR comment.

Captured real-model evidence

Inspect the run, the checks, and the source diff.

GitHub Actions called Claude Haiku 4.5 for 10 runs per ref. The 2026-09-04 snapshot below preserves that run. The live PR is refreshed separately and may show different results.

Source-linked evidence

One public PR. Real model calls. Inspectable provenance.

The head passed all 30 eval observations while confirmed reporter delegation was 10/10 on base and 0/10 on head in each of 3 evals. The yellow verdict asks a human to review the captured evidence.

Read the archived report

Original workflow attempt · Captured source diff · Latest live comparison (updated separately)

Modelanthropic/claude-haiku-4.5

Runs10 per ref

Output tokensmedian −34%

Durationmedian −41%

Head cost (median)$0.0076 / session

Base cost is unavailable because subagent usage was not fully attributed. A savings percentage cannot be calculated for this captured run.

Evidence, with limits

Honest about what the runs can prove.

diff0 separates regression, inconclusive movement, and behavioral drift. It shows uncertainty instead of painting every change red.

N=10

runs per ref

60 / 60

eval observations passed

source file changed

sticky PR comment

Pass-rate evidence

One-sided Fisher exact tests compare differing proportions across runs.

Multiple comparisons

Holm adjustment keeps a suite full of evals from manufacturing confidence.

Validity caps

Changed eval harness, authored sandbox configuration, model, run count, or scorer validity caps the verdict at yellow.

Drift is not automatically bad. diff0 reports only what Eve captured; it does not prove semantic equivalence or decide whether a change is desirable. In the showcase, removing delegation coincided with 34% fewer median output tokens and 41% lower median duration. Base cost stayed unavailable because delegated base usage was not fully attributed, so diff0 makes no savings claim.

In your pull request

The report goes where the decision happens.

One comment updates in place on every push. No dashboard to remember, no second review loop.

github-actionsbotcommented

diff0 report 🟡

Review recommended. No confirmed eval regressions across 10 runs per ref. Confirmed behavioral drift requires review.

Comparing 84cb96d → fc5efb9 · 10 runs per ref · model anthropic/claude-haiku-4.5

At a glance

Signal	Base	Head	Change
Evals passing every run	3/3	3/3	unchanged
Cost / session (median)	unavailable	$0.0076	n/a
Tool calls / run (agents excluded)	6 (6–10)	6	+0%
Output tokens / run	1,017 (941–1,481)	675 (654–714)	-34%
Duration / run	20.2s (19.0s–27.2s)	11.8s (10.8s–12.5s)	-41%

Observed behavioral differences

Signal	Base	Head	Scope
Subagent `reporter`	10/10 runs	0/10 runs	3 evals · confirmed

2 additional inconclusive signals are available in the full comparison details.

Eval results

Eval	Base	Head	Result
`revenue/total-revenue`	10/10	10/10	✅ pass
`revenue/no-failed-actions`	10/10	10/10	✅ pass
`revenue/uses-sql-tool`	10/10	10/10	✅ pass

Full comparison details

eve 0.47.5 · actual sandbox unknown · host default candidate microsandbox · comparison cost unavailable

Subagent evidence: Fisher raw p=<0.0001; Holm p=<0.0001 in each of 3 evals.

Cost note: Comparison cost unavailable because delegated base usage was not fully attributed.

Changed file: fixtures/demo-agent/agent/instructions.md (+1 −2)

1 file changed, 1 insertion(+), 2 deletions(-). File attribution is correlational, not causal.

Captured 2026-09-04 · diff0 v0.1.3. Statistical comparison across 10 runs per ref — LLM runs are nondeterministic; treat proportions, not absolutes. Read the archived console report.

Run it on your agent

Compare two refs.
Review one report.

From a repository with an Eve eval suite, compare the branch you plan to merge against its base. Commit both refs, install your app’s dependencies, and configure its model credentials. No global diff0 install is required.

$ npx @knowbody/diff0 run --base main

Getting Started: baseline, evals, and credentials

Want to inspect or develop diff0 itself?

$ git clone https://github.com/knowbody/diff0 && cd diff0

Add it to CI

Run the comparison on pull requests and post one self-updating comment. The default blocks red regressions but reports yellow drift without failing; granular policies can gate eval, score, performance, behavioral, and validity findings independently. Fork PRs stay refused by default.

name: diff0

on:
  pull_request:
    paths:
      - "agent/**"
      - "evals/**"
      - "src/**"
      - "package.json"
      - "pnpm-lock.yaml"
      - ".github/workflows/diff0.yml"

permissions:
  contents: read
  pull-requests: write # required for the sticky PR comment

jobs:
  diff0:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
        with:
          fetch-depth: 0 # required: diff0 checks the base ref out locally
          persist-credentials: false # do not expose the job token to evaluated head code
      - uses: actions/setup-node@v7
        with:
          node-version: 24
      # Pin a release version.
      - uses: knowbody/diff0/action@v0.1.4
        with:
          working-directory: .
          runs: "3"
          install-mode: scripts-off
          fail-on: regression # yellow drift reports but does not block; use drift to gate it

Library API preview

Build diff0 into your own tooling

Compare records collected by your own host, or use the Node.js runner to compare Git refs through Eve. Separate comparison, adapter, and reporting entrypoints let you own the workflow and where the results go.

The package includes a library API preview. See the guide for installation and examples.

Explore the library API preview

Code review for the part of your agent you cannot see.

Inspect the live PR Explore on GitHub

Review the agent,not just the diff.

Your eval passed.Your agent still changed.