Clerkbench · methodology v1

How these numbers are made

Clerkbench measures the thing Clerk actually claims: a lawyer's question goes in; a grounded answer comes out; nothing in it is invented. The unit under test is an entire environment — engine, verification gates, retrieval tools, prompt and model — because our own measurements show the environment is what changes outcomes: on the same case and the same model, the bare model fabricated 4 of 4 case-law references while the full workspace fabricated none. Every run published here carries its exact configuration, git commit, corpus fingerprint, tool traces, per-point scoring reasons and verification evidence, so a skeptic can re-score any run without trusting us.

Two headline numbers, never one

Quality (0–100) aggregates eleven substance axes: issue spotting, legal research, source quality, citation accuracy, interpretation, factual application, tool use, verification, alternative paths, uncertainty handling and completeness. Axis weights are pre-registered per case. The efficiency profile — efficiency, cost, latency, each scored against the case's pre-registered envelopes — is reported alongside and never folded into Quality: a fast wrong answer must not outrank a slow right one. Alongside both, the critical-failure rate reports the share of repeated runs with at least one critical event (τ-bench's pass^k lesson: reliability is a property of repetition, and a run that fabricates once in three tries is not "two-thirds good").

Severity gates

Every rubric item carries a severity — minor, material, major, critical. A missed major point deducts beyond its weight and caps Quality at 69; any critical event — endorsing a fabricated authority, inverting a protective provision, a rights-forfeiting deadline error, an independently verified invented citation — caps Quality at 39 and flags the rep. Polished writing can therefore never outweigh a serious legal error, which is the failure mode preference-based evaluations reward.

No preference judging, mechanical quotes

We measured an LLM preference judge inverting under hand verification — fabrication reads as fluency. So Clerkbench has no preference judging at all. A grader model answers, per rubric item, one extraction question — does the answer assert this proposition? — blind to which system produced the answer, and must return a verbatim quote that is then mechanically substring-checked (after presentation-only normalization). An unquotable claim scores absent. Items accept explicitly listed alternative routes, and nothing is scored by similarity to a reference answer: sound reasoning on a different valid basis earns the item.

Citations: verified, or unresolved — never guessed

Every precise reference (a number, a court-and-date pair, a source URL) is verified mechanically against the corpus snapshot, with the leniencies the corpus's own defects demand (Luxembourg decisions carry both a decision number and a docket/rôle number; the index is known to hold the rôle, sometimes nothing, and at least one wrong date). Verdicts:

verified — full credit; date-only — partial credit (indistinguishable from a known index defect, never penalized).
unresolved — zero credit, zero penalty. A reference we cannot confirm is never counted as fabricated: falsity requires independent verification, and every escalation query we tried is published as evidence.
misattributed / fabricated — penalized only with document-level proof or when the benchmark itself authored the planted authority (ground truth held by construction); article numbers are proven nonexistent only against fully-enumerated codes (the signature of French-law transplants).
out of corpus — dated past the snapshot horizon or outside its jurisdictions; excluded from the denominator, listed.

Unsourced appeals to case law ("il est de jurisprudence constante…") are counted separately — the honest failure mode is never folded into the dishonest one. This follows the Stanford RegLab studies' two-dimensional definition (correctness × groundedness): an uncited true statement is incomplete, not a hallucination.

Process, not just the answer

From the recorded tool trace: fetch-before-cite coverage (was every cited decision actually opened during the turn — by the model or by the engine's own mechanical check, which is part of the system under test), query discipline (loops and repeats), tool routing, steps, tokens, retries. The timeline is published; the model's reasoning stream is not (character counts only).

Cases and plants

Cases are fictional fact patterns over real, current Luxembourg law; every rubric anchor quotes the corpus's consolidated text, gathered into a ground-truth dossier before the first run. Some cases embed planted authorities — an invented cassation arrêt whose registry number falls beyond a complete, gap-free series; a real arrêt cited for the opposite of what it holds — so detection of fabrication and misattribution is provable, not asserted. Mentioning a plant is never penalized (the case itself introduces it); only endorsing it is. Each case file carries a visible canary GUID for training-data contamination detection, and cases are versioned so a contaminated case can be retired without corrupting comparisons.

Determinism and adjudication

Scoring is a pure function of the stored evidence: the run record, the cached grader output, the cached citation verdicts and any human adjudications. Re-scoring reproduces identical numbers (unit-tested). Residual grader nondeterminism is bounded by the quote gate and surfaced, not hidden: raw grader output is stored with the run, the grader's identity is published (including a warning when it shares a model family with the system under test), and human adjudications — following the HealthBench rule that a claimed error must be independently confirmed before it counts — are recorded overrides with reasons and evidence, visible on the run page.

Reproducibility and comparability

Model fallbacks pinned to the arm model for the whole run (a saturated turn silently finishing on another model is a corrupted comparison that looks clean — we've been bitten).
A real-sized rate-limit probe before each run; sequential reps; recorded cooldowns.
Every run records: benchmark/case/rubric versions, timestamps, model and provider settings, engine, git commit and dirty flag, non-secret behavior flags, the corpus snapshot's SHA-256 fingerprints and horizon, price-table version, durations, attempts and token usage.
Runs are ranked only within a compatibility key (case + rubric + framework major versions); the scoreboard warns instead of comparing across keys.
Providers offer no random seed; nondeterminism is handled by repeated reps with mean, variance, min–max and the critical-failure rate — never averaged away silently.

Limitations, stated plainly

This is a self-benchmark: Clerk grading Clerk. The mitigation is radical auditability — rubric, reasons, traces, evidence and grader identity are all published — not a claim of neutrality (the structural critique of vendor benchmarks applies to us too).
n is small; reps are expensive. Variance and n are always shown.
The corpus fingerprint versions the retrieval world; runs are comparable only within a fingerprint.
Costs come from a manually maintained, versioned price table; estimates are flagged.
All v1 cases are public (a held-out slot exists in the framework, following the GSM1k contamination lesson).

The scoreboard is at /benchmark. Method lineage: LegalBench's skill taxonomy; Stanford RegLab's verification discipline and hallucination definitions (Dahl et al. 2024; Magesh et al., JELS 2025); Harvey BigLaw Bench's rubric-and-source split; τ-bench's pass^k; SWE-bench Verified's task-audit lesson; HELM's multi-metric reporting; HealthBench's criterion-anchored grading. To our knowledge Clerkbench is the first agentic, citation-verified benchmark for a civil-law jurisdiction.