Can models write Candor?

Candor exists for code that models write. So the measure that matters is whether a model that has never seen it writes correct Candor from the reference alone. This page reports round 1, including what it doesn't show.

Method

• Reference: llms.txt, one dense spec meant for a context window, and the only thing participants read.
• Tasks: nine, each touching something Candor-specific: payload enums and match, inout records, requires contracts, effects, byte-level strings, nested arrays, binary search, an Option-style enum (TASKS.md).
• Isolation: each model ran as a fresh agent in a scratch folder holding only the spec and the tasks. It had no repository, no reference solutions and no compiler, so this measures first attempts.
• Scoring: hidden test programs use each solution and compare output exactly. The two contract tasks also check that a bad call traps with requires failed. The scorer was itself tested: reference solutions score 9/9, and a copy with four planted bugs scores 5/9.

Round 1 results

Neither model hit a single compile error, and the example clauses they wrote for themselves all pass.

The useful signal: an uncertainty tax

Each participant listed everything in the spec that was unclear. Working independently, the two models raised the same questions:

• Do and / or short-circuit?
• Is each case arm its own scope?
• Does [] take its type from a parameter or return type?
• Are nested writes like grid[i][j] = v allowed? Can an inout record's fields be assigned?
• Can match sit inside a loop, with any statements in its arms?

The answer to all of them is yes. The gaps caused no bugs, but both models paid for them with defensive code: nested ifs instead of short-circuit tests, different names in every case arm, typed temporaries instead of return [], and avoided nested writes. That code is longer and harder to review than it needs to be.

What changed: llms.txt gained a "Precise semantics" section that answers each question directly, with every answer checked against both backends. A new test, tests/spec.sh, type-checks every complete program in the spec. On its first run it caught a broken example in the Records section that showed statements at the top level of a file, a pattern a model could easily copy.

Round 1b: a model from another family, running locally

Qwen2.5-Coder (7B and 1.5B) was trained outside Anthropic and played no part in Candor's design. It ran on a laptop through Ollama, with the repair loop on: when an attempt fails, the model gets back only what the compiler said. Each run was repeated after compiler changes, so each change's effect is visible.

• A hint that wasn't an edit failed. The model wrote example parse_csv_ints("12,-3,40") == [12, -3, 40]. Candor had no == on arrays, and the hint said "compare the fields you care about". The model resubmitted the same line for three rounds. Writing f(x) == [..] is too natural to fight, so the language changed: == is now structural, and that task compiles on the first try.
• Hints that were edits worked. "add effects io to the header of fn …" was fixed in one round every time.
• Models can check themselves. Once csv compiled, its remaining bug was stated by the model's own example: parse_csv_ints("") == [], while the code returned [0]. Feeding back the failing example (no hidden information) fixed it in one round. That's what example clauses are for.
• What's left is misreading the task (an extra contract, a changed signature). The model's own examples encode the same misreading, so no self-check catches it.
• 1.5B is below the floor: it doesn't follow the task and ignores hints that state the exact fix. Its failures still led to three compiler improvements that help every model: a hint for pasted Markdown code fences, a hint for s[i:j] slicing, and trimming huge source lines out of diagnostics so they don't flood a model's context.

Round 2 (planned)

1. Harder tasks: multiple files with use, 150+ line programs, distinct-type ids, holes, and traps hidden in the obvious solution.
2. More model families and sizes: a 14B–32B open model, then Llama, Gemma and DeepSeek. Qwen 7B and 1.5B are done.
3. The repair loop: each failing attempt gets only the compiler's --json diagnostics, and we count the rounds until it passes. This tests whether Candor's error messages work for models, which is the point of having them.

Everything is reproducible: eval/score.sh eval/attempts/<model>. The attempts are in the repo exactly as the models wrote them.