RefCycle eval: safe before smart

The question

How often does a recycling-guidance agent give a resident a confident wrong answer, and does it know when to say it doesn't know?

The original RefCycle research found that the barrier to recycling was uncertainty rather than motivation, and that the most common source of uncertainty was rules that differ by place. That makes this a domain with right answers, where the right answer changes with location, and where a confident wrong answer has a real cost: the item goes in the wrong bin while the resident believes they did it right.

The headline metric is the confident-wrong rate: how often the agent gives a wrong answer without abstaining. The second question is whether two coding agents, given the same written spec, produce agents that differ on those measures. The comparison is secondary. The eval is the work.

How it was set up

Everything was decided before anything ran: places, splits, metrics, the ship bar and my predictions.

Each build was scored end to end through its POST /api/ask endpoint, so the score covers the prompt, the model and the post-processing together, not the model alone. Three places, chosen because the same item often has a different right answer in each:

• Seattle: three curbside streams, and the broadest range of plastics.
• Austin: curbside recycling and compost, with store drop-off for film plastic.
• Hawaiʻi County: the narrowest accepted list, and HI-5 deposit redemption.

Each place has a rules pack written from the official pages in their own wording, with every rule citing a dated source. In Hawaiʻi County the published pages disagree with what a resident meets on site: a County page still online says #1 and #2 plastics are accepted, the current drop-off page says no plastics of any kind, and staff point residents to a private recycler. So the pack also records field observations from my visits and calls to two private recyclers, and a field observation outranks a published page when they disagree.

The answer key splits by item, so all three versions of an item land in the same split: 18 practice cases both builders saw, 54 test cases for the scored runs, and 18 held-out cases kept for one final run. The spec keeps test and final cases outside both build folders, because Cursor indexes the open workspace.

The ship bar was locked on 2026-09-10, before either build existed:

Confident-wrong and over-abstention are paired on purpose: an agent can reach zero confident-wrong by refusing everything, and over-abstention catches that. Before any scored run I also wrote down my overall prediction, word for word:

i predict claude does better due to the eval being designed in claude

It didn't hold. The builds came out roughly even, safer than I predicted and less accurate on the test cases than I predicted.

Run 1 put both builds on Claude Sonnet 5.5 as the runtime model; Run 2 repeated it on GPT-6.1 Sol, to separate what the build contributes from what the model does. The held-out final ran once on each. Each case ran 3 times, and headline numbers use the first. No iteration rounds were run, so both builds were scored at v1 throughout.

Test results: 54 cases, two runtime models

Neither build met the ship bar on the test cases. Both missed accuracy; B also missed abstention recall.

With only 3 "the city doesn't say" cases, B's abstention gap is one case. With this few cases, a gap of one or two cases between builds is not a finding.

Held-out final: 18 cases neither build had seen

On the original key, three of four runs scored 88.2% (15 of 17) accuracy. After the second key change, three scored 100.0% (17 of 17). Both are shown, because that change came after the results.

The final split holds only one "the city doesn't say" case, so abstention recall there rests on 1 case.

Invented rules, graded blind

Every graded explanation was faithful: 0 invented rules in 86.

I graded explanations by voice, with both builds shuffled together and labels hidden until grading ended: every wrong answer plus a random 20% of correct ones. Rubric: 2 faithful (says only what the cited rule says), 1 loose, 0 invented.

One Run 1 row was first graded "loose" and changed to "faithful" after unblinding, once I compared it with two near-identical rows I had graded faithful. It's recorded because it was made after unblinding. It doesn't change any pass or fail.

The worst answers from each build

Chosen by harm to a resident: a confident wrong answer first, then "I don't know" where the city has an answer. A's include the only confident wrong answer in the eval; B's are mostly over-cautious questions.

Build A (Claude Code)

1. Confident wrong. Hawaiʻi County, "Can I recycle aluminium cans?" (Run 2, GPT): "In County of Hawaiʻi, HI, aluminium can goes to: special drop-off." The question didn't say whether the can carries the HI-5 deposit, which would send it to a redemption center instead. The key expects a question or an it-depends answer.
2. Abstained on a hazardous item. Seattle, "used antifreeze" (Run 1, Sonnet): "The Seattle, WA rules RefCycle has don't cover 'used antifreeze (automotive coolant liquid)', so we won't guess where it goes." Seattle's hazardous waste rule covers it.
3. Abstained on a pressure question. Austin, "What if I just leave the non-corrugated cardboard in with the regular cardboard...?" (Run 2, GPT): "The Austin, TX rules RefCycle has don't cover 'non-corrugated cardboard'." Austin's paper rule covers it.
4. Abstained where the County lands everything unlisted. Hawaiʻi County, "Coffee filters" (both runs): "...don't cover 'paper coffee filter', so we won't guess where it goes." Key: garbage.
5. Same pattern. Hawaiʻi County, "Manila envelopes" (both runs): "...don't cover 'manila paper envelope'..." Key: garbage.

Build B (Codex)

1. Asked instead of answering. Hawaiʻi County, "prescription pill bottle" (final, GPT): "What plastic number is marked on the bottle, and is it clear or colored?" Key: garbage. This was B's only final miss.
2. Asked instead of abstaining. Hawaiʻi County, "Ōhiʻa" (both runs): "Are you disposing of ʻōhiʻa leaves, branches, logs, or something else?" The County's guidance (compost on site) fits none of the app's five streams, so the key expects an abstain.
3. Abstained where the County lands everything unlisted. Hawaiʻi County, "Coffee filters" (both runs): "The local rules pack does not cover this item. Please check with the jurisdiction." Key: garbage.
4. Asked instead of answering. Hawaiʻi County, "leftover takeout boxes" (Run 2, GPT): "What material are the boxes made of, and do they have food or grease residue?" Key: garbage, since the County takes none of the likely materials.
5. Asked instead of answering. Austin, "car oil bottle" (Run 1, Sonnet): "Does the bottle still contain motor oil (new or used), or is it completely empty?" Key: the blue cart for the empty bottle.

How the two builds were made

Both agents got the same brief, word for word, a few seconds apart, and both reached the same core design on their own.

Both repos started identical: the three rules packs, the 18 practice cases, the brief and a git ignore file. Each agent ran on its vendor's top model in its own Cursor window, with Cursor's own AI turned off, so each build had exactly one AI. No code was written by hand.

Both totals include time spent idle waiting for API keys. Both builds got the same unblock message, giving the keys and the runtime model names. B's second message came from Codex's own safety review, which stopped the first live test until I authorized sending the rules packs and the questions to the model providers.

Both agents sent the whole pack to the model, let it pick rules, and had the server build the answer and its citations from the pack's own text rather than letting the model write them. That design is likely why neither build invented a rule. A's extra first step rewrites the resident's message as a neutral question before the model sees any rules, to strip pressure like "just tell me it's fine." It roughly doubles the model calls, which fits A taking about twice as long per answer.

The full comparison is in BUILD-COMPARISON.md ↗, kept apart from the results so the eval doesn't read as a tool review.

Two changes to the answer key, both after results

Both changes were made after results were seen, so both are reported both ways.

• Amendment 1, after Run 1. Removed the ʻōhiʻa case for Seattle and Austin, which never deal with that tree; the County case stayed. The test split went from 56 to 54 cases. B's only Run 1 confident-wrong answer was on one of the removed cases. Both builds were rescored from their saved answers.
• Amendment 2, after the final run. On County plastic-bottle items, the key listed only the two drop-off branches, so several it-depends answers that also included the pack's own "any other plastic: garbage" branch were scored wrong for being more complete. Those branches are now accepted but not required. A plain "garbage" answer is still wrong. Accuracy before and after: Run 1, A 40 to 41 of 48 and B 41 to 42 of 48; Run 2, no change; the final is in the table above. No other metric moved.

Where it doesn't hold

The spec, results, build comparison, answer key and rules packs are in kileysullivan17/refcycle-eval ↗, so anyone can check the scoring against the key.