Tracetoon · evidence graph results · 61 real pull requests · November 2026
An embedded Relate graph made our PR writer better and cheaper
Tracetoon writes pull-request descriptions from the session logs your AI coding agent already keeps. Its evidence graph, built per PR in the embedded Relate database, gives the writing model a compact, cited brief instead of raw logs. Compared with Tracetoon's original multi-agent workflow on 61 real pull requests, judges preferred the graph-written description 83 to 34 on Codex and 69 to 17 on Claude Code, while it cost 46% and 31% less per PR.
Codex · judges' pick71%of decided head-to-heads went to the graph PR (83 vs 34; 95% interval 61%–80%)
Codex · cost per PR−46%typical saving per PR; average $0.225 vs $0.447 for the original workflow
Codex · key facts conveyed82%of the expert fact sheet's claims, vs 72% for the original
Claude Code · judges' pick80%of decided head-to-heads went to the graph PR (69 vs 17; 95% interval 71%–88%)
Claude Code · cost per PR−31%typical saving per PR; average $0.415 vs $0.654
61 real PRs from 12 repositories · each written 3 times by each workflow · 2 judges per provider, both orders · 732 judgments per provider · factual errors checked by two expert models · prbench
What Tracetoon does
AI coding agents such as Claude Code and Codex keep session logs on disk: the requests you typed, the plans they made, every file edit and every command they ran. A pull-request description written from the diff alone loses most of that. It can't say why the change was made, what was tried first, or which tests ran on the final code.
Tracetoon is an open-source agent skill that writes the description from that record. When your agent is about to open a PR, Tracetoon finds the sessions that produced the branch, pulls a redacted evidence set from them (private paths and secrets removed), and has the agent draft from it. It then checks the draft: every factual sentence has to trace back to evidence, and the Validation section is generated from recorded test and build runs rather than typed by the model. You approve the result before anything is posted.
What the evidence graph changes
Tracetoon's original workflow was already careful. Separate agents research the repository's rules, draft the description from a digest of the logs, and independently review every claim, so no agent checks its own work. That is accurate but expensive: an expensive model has to rediscover the structure of the change from text, working out which commit sentence describes which hunk, which functions are new and why the change was made, and then a second instance does it all again to check the first.
The evidence graph moves that rediscovery out of the model. One command reads the session logs and git, builds a small graph in Relate (an embedded graph database written in Rust, shipped inside Tracetoon's CLI as WebAssembly), runs a fixed set of Cypher queries, and prints a compact brief. No model is involved, so this step costs no tokens. The writer gets short tables it can cite (which request led to which code, which function owns each behavior, which test ran after the last edit) instead of rereading everything.
From agent logs to a cited PR description
The graph is rebuilt for every PR on the developer's machine, offline. Edges such as "this agent edit survives in the final diff" connect a hunk back to the prompt that asked for it and forward to the commit sentence that describes it.
Quality: judges prefer the graph PRs on both agents
Each description was compared head to head with the original workflow's description of the same change. Two judges per provider read both, in both orders, against an expert-reviewed fact sheet and the original evidence, and picked the one a maintainer would rather receive. Points on a 100-point score are an arbitrary scale, so we lead with the judges' choices.
How often judges picked the graph-written PR
Share of decided head-to-heads (wins vs losses; ties and order-dependent splits excluded) won by the graph PR, with 95% intervals bootstrapped over the 61 cases. A green dot means the whole interval is above an even split. "Both judges agree" counts only comparisons where the two judges picked the same winner.
Codex (GPT-6.1 Terra)
When both judges agreed, they picked the graph PR 83 times to 34 (71%).
Each judge alone: GPT-6.1 Terra picked the graph PR in 69% of decided pairs, GPT-6.1 Sol in 67%.
It conveyed 82% of each change's key facts, against 72%.
Claude Code (Sonnet 5.5)
Each judge alone: Opus 5.5 picked the graph PR 112 times to 40 (74%), Sonnet 5.5 80 to 29 (73%).
When both judges agreed: 69–17 (80%, 95% interval 71%–88%).
Fewer factual errors per draft after expert checking: 0.68 vs 0.81.
The score lead
The benchmark's composite score (claims, context, uncertainties and a reviewer rubric, minus penalties for errors) tells the same story. The charts show only the range where the two workflows differ.
Mean of per-case scores out of 100; the axes are cut (marked by the break) to show the gap. Differences are paired by case and replicate, with 95% intervals over cases. The Codex gain's interval excludes zero; the Claude Code interval includes it.
Cost: 46% less on Codex, 31% less on Claude Code
The graph replaces work the model used to do with a deterministic build step, and the drafting and review steps read a short brief instead of the full logs.
Average cost per pull request
Bars show the average priced cost of every model call per PR, including subagents and the review pass. The headline percentage is the more conservative paired measure: the geometric mean of per-PR cost ratios (same case and run), with a 95% interval over cases. The plain averages give larger savings (50% on Codex, 37% on Claude Code). Codex costs use published per-token prices; Claude Code reports its own cost.
One PR, three ways
The same change described three ways, all written by Codex (GPT-6.1 Terra) and judged by the same judges against the same expert fact sheet: the agent opening the PR on its own, Tracetoon's original multi-agent workflow, and Tracetoon with the evidence graph. Highlights come from the judges' clause-by-clause review; the notes list key facts from the fact sheet that a version left out.
contradicted by the evidencebroader than the evidence
Agent opens the PR itself · no Tracetoon
Group brief evidence into change neighborhoods
score 50.71 contradicted claimkey facts 6 of 10 · context 0 of 5$0.068
Summary
Group evidence into per-change neighborhoods that keep requests, commit sentences, hunks, and relevant changed lines together within an evidence budget.
Link commit sentences to matching hunks and expose the neighborhood through brief queries and the writing workflow.
Report recurring brief-check findings once, and distinguish results attributed to commit messages from claims about captured runs.
Validation
bun run test — passed (39 source test files).
bun run typecheck — passed.
bun run build:check — failed because the bundle output receipt is missing in this checkout.
Bundled and compiled contract suites left to CI.
Missing: A new brief layout switch (src/graph/layout.ts): TRACETOON_BRIEF_LAYOUT=neighborhood|toon, defaulting to neighborhood unless set to 'toon', ; patchHunks now stores up to 24 key changed lines per hunk (Hunk.code), which the graph build writes onto Hunk nodes. Lines are picked by pri; Tests: new tests/neighborhood.test.ts covers key code lines, sentence→hunk links, every change getting a block, budget packing and render ac; The bundled plugin/skills/tracetoon/dist/tracetoon.mjs is rebuilt and build.json's buildId changes; the version stays 1.0.0-rc.24.…
Tracetoon, original workflow
Add neighborhood evidence briefs and limit render loops
score 68.20 contradicted claimskey facts 6 of 10 · context 1 of 5$0.273
The evidence brief previously presented graph evidence as cross-referenced TOON tables. This adds a neighborhood layout that groups each change with its requests, commit sentences and hunks, including selected changed lines. It is the default; TRACETOON_BRIEF_LAYOUT=toon retains the previous layout. The change blocks use a configurable evidence budget, clipping and packing content as needed.
The commit notes report repeated render findings during Codex drafts. Render now lists previously reported brief findings as repeated rather than blocking again, and excludes results attributed to a commit message from run claims.
Review src/graph/neighborhood.ts for the layout and packing, then src/graph/briefRender.ts for finding handling; the neighborhood and render-loop tests cover these paths.
Validation: bun run typecheck, bun run test (39 files), and bun run bundle passed. The bundled and compiled suites are left to CI.
Missing: patchHunks now stores up to 24 key changed lines per hunk (Hunk.code), which the graph build writes onto Hunk nodes. Lines are picked by pri; In neighborhood mode briefParts adds a citable 'hunks' section (ref H#, change, file, lines, scope, code, why, sentences, weight, role), so ; The brief's avoid rules are reworded for the neighborhood layout (tags may cite H#; name a file from → H# hunks or implementedBy; 'was' from; graphBrief (used by prepare) and `tracetoon query brief` both print the neighborhood text when that layout is on; otherwise the TOON tables …
Tracetoon with the evidence graph
Present evidence briefs as graph neighborhoods
score 89.60 contradicted claimskey facts 10 of 10 · context 2 of 5$0.216
This gives PR writers a graph-shaped evidence brief that keeps related requests, commit explanations, hunks and changed lines together, addressing the request for a better way to pass the graph to the writer.
Previously, the brief presented the evidence as TOON tables linked by identifiers. The entry point in src/graph/evidence.ts now selects a neighborhood layout by default; TRACETOON_BRIEF_LAYOUT=toon retains the previous layout. Each change block brings its requests, commit sentences, files and hunk references together with selected old and new lines. Hunks retain up to 24 key changed lines and can be cited; links from commit sentences to hunks use conservative matching. The neighborhood renderer packs change blocks to a default budget of 2,112 tokens, configurable with TRACETOON_BRIEF_BUDGET, selecting code lines by value per character and compacting larger briefs. Fixed lines can exceed that budget, and rules and validation tables sit outside it.
The brief query uses the selected layout in its text output.
The render check also moves previously reported findings into a nonblocking repeated list, and accepts results attributed to a commit message without treating them as validation claims. The commit attributes lengthy drafting loops to findings writers could not clear, including commit-attributed results flagged as run claims and uncovered request and change rows. The contributor guidance now points writers to hunk references when identifying the code that implements a change.
tests/neighborhood.test.ts covers key line selection, sentence-to-hunk matching and budget packing. tests/render-loops.test.ts covers commit-attributed results and repeated findings; the shipped bundle is regenerated.
Validation
bun run typecheck: passed, run on the submitted content.
bun run test: passed (Source test files: 39 passed, 0 failed (up to 4 fresh processes)), run on the submitted content.
Only this version conveys: patchHunks now stores up to 24 key changed lines per hunk (Hunk.code), which the graph build writes onto Hunk nodes. Lines are picked by pri; The work came out of a graph-vs-relational 'shape test': at equal evidence size the graph brief out-informed the relational arm, and the use
Case tracetoon-neighborhood, a change to an open-source repository in the benchmark. Descriptions are shown as written, minus attribution footers.
Every case
The gains aren't carried by a few cases. Each dot is one pull request: the graph's mean score minus the original's over three runs.
Codex: per-case score difference
Claude Code: per-case score difference
Hover a dot for the case name. Per-case differences are noisy (three runs each); the pooled results above are the measure. The full list is in the appendix.
How we tested
The cases
61 real pull requests from 12 repositories: Tracetoon (17), AI First CLI (10), prbench (9), Showtail (6), AI First content (4), VEX AIM agent (4), AI First website (3), VEX AIM workshop (3), VEX V5 serial (1), WinApps (2), AI First extension (1), Bazzite login (1).
Each comes with the original agent session logs that produced it: 33 from Claude Code, 28 from Codex.
Each has an expert-reviewed fact sheet: the claims a good description should convey, the context a reviewer needs, what the evidence does not establish, and plausible-but-wrong traps. Sheets were drafted from the diff and logs, reviewed by a six-model panel (three Claude Opus 5.5, three GPT-6.1 Sol), and disputes were settled by debates judged by Fable 5.1 and GPT-6 Astra.
The runs
Both workflows wrote every PR three times with the same model, prompt and sandbox. They differ only in the installed Tracetoon skill.
Two judges per provider (Codex: GPT-6.1 Sol and Terra; Claude: Opus 5.5 and Sonnet 5.5), each in both orders, fixed before the run.
Every factual error a judge flagged was checked against the evidence by Fable 5.1 and GPT-6 Astra (Fable settles disagreements): 472 flags, none left unresolved.
Run with the open prbench harness under its confirmation protocol.
Run details and disclosures
Our own adoption gate. Before testing we fixed four criteria per provider: the score's lower 95% bound at least −2.5, expert-confirmed factual errors not worse by more than 0.05 per PR, cost at most 0.8×, and key-fact coverage not lower by more than 0.05. Codex met 3 of 4 (missed: errors not worse by more than 0.05 per PR; confirmed errors 0.26 vs 0.22 per PR, difference interval up to +0.186). Claude Code met 3 of 4 (missed: errors not worse by more than 0.05 per PR; confirmed errors 0.68 vs 0.81 per PR, difference interval up to +0.093).
Tuned on part of the pool. After a first 61-PR run, the brief was tuned over four rounds on 25 of these cases (the 20 where it had lost most, plus 5 controls). On the 36 cases never used for tuning the score difference is +5.1 (Codex) and +2.6 (Claude Code); on the 25 tuned cases +3.1 and +5.0. Judges' picks on the untouched cases: 51–14 (Codex) and 43–11 (Claude Code). Tuning rounds were scored by one judge per provider (GPT-6.1 Terra for Codex, Sonnet 5.5 for Claude Code); this final run is scored by both pre-registered judges. The first run, before tuning, measured +2.7 (Codex) and +0.8 (Claude Code).
Reused drafts. The original workflow's 183 drafts per provider were written in that first run and reused here; only the graph's drafts are new. Same models, harness, sandbox and judges.
No held-out set. All 61 cases form one pool used both to tune Tracetoon and to evaluate it. Earlier, a sealed held-out split was used once; that run found errors (statements that no request or test was recorded) which the current build fixes. We then chose a larger single pool over repeated small held-out sets.
Judges disagree often. Agreement between the two judges was 77% (Cohen's κ 0.61) on Codex and 57% (κ 0.33) on Claude Code. We report the picks where both judges agree as the headline measure. The Codex judges include GPT-6.1 Terra, the same model as the Codex writer in both workflows.
Uncertainties on Claude Code. Graph drafts on Claude Code respected fewer of the fact sheets' uncertainties (58% vs 65%). This is the main open quality gap.
Wall time. On Claude Code the graph workflow took about 1.6× as long per PR; on Codex it was about the same (1.11×).
Sandbox validation. Writers ran tests in an offline sandbox. 31 of 61 cases' test suites pass there at the PR's final commit; the rest fail for host reasons or have no test command. Both workflows saw the same environment.
Compared with our earlier 5-PR report. That report measured 35% lower cost on Codex and 41% on Claude Code. On 61 PRs the Codex saving grew (46%) and the Claude Code saving is smaller (31%); since then the graph workflow added fresh checks and a review pass to fix accuracy. The earlier score gains came from 5 PRs; this run uses 61.
Models and versions. Writers: GPT-6.1 Terra (medium effort) in the Codex CLI and Claude Sonnet 5.5 (medium) in Claude Code 2.1.319. Tracetoon build candidate58 (Tracetoon 160b49f); baseline is Tracetoon's original multi-agent workflow. Harness: prbench bench-v4 (writer side f4f7139), 61 cases x 3 replicates, pre-registered judge pairs, original-workflow drafts reused from the 2026-11-16 run (diagnostic purpose).