We publish our own error rate

Updated per audit round and per release. Generated context: this file states measured numbers with their intervals and the program's honest status. It never states more than the measurement supports — the caveats are part of the product.

Status: INSTRUMENTED · BASELINE MEASURED · FIX PROGRAM DIRECTED

Measured — 4W half (car/van/truck/bus), baseline round, 2026-07-26

• Claim-level clean rate: 83.23% (defect+unverifiable 16.77%, 95% CI 15.39%–18.25%). Full breakdown, methods, and every ledger: data/review/audit-v2026.07.5/RESULTS.md.
• Availability claims ≈99.5% clean — verified against the registers themselves. Make ~96%, kind ~97%. The defect mass is names and id-canonicality, concentrated in two structural generators (truncation stubs; trim-granularity ids) now being fixed at the generator.
• Both generators have since been attacked at scale (2026-07-26 → 08-01, post-baseline): the truncation-stub fix retired 36 stubs and recovered ~370 real models; the trim-granularity fold programme then measured twenty-plus makes (23%–82% of their ids were trims, codes or body words rather than models — per-make dossiers in the pipeline repo under aux/research/) and folded them onto their real nameplates with zero availability evidence lost, verified per fold. The 4W catalog went 8,613 → 6,946 published ids.
• A third generator was found that no detector could see: DELETION. junk? was discarding real nameplates before they ever reached a record — a bare numeral cannot be a model, and a platform-code rule fires after the make token is stripped. Measured by a per-row replay logging nil returns: 125,650 vehicles recovered so far (Saab 9-3 alone was 70,404 across 9 countries, while its own trim records published normally). A dropped row leaves no record, so every catalog-side check is structurally blind to it; rake report:junk_drops now stands watch. The largest known remaining instance (a door-count prefix, ~26,700 recoverable vehicles) is filed UNFIXED in DEBT because the same rule shape is a real platform code.
• These land AFTER the baseline above — the next audit round measures their effect; this dashboard does not claim it in advance.
• Wave 3 (2026-07-27, post-baseline): nine more makes measured and organized (Seat/Renault/Peugeot/VW/Dacia/Hyundai/Mercedes-Benz/Fiat/ Tesla — 14–70% of ids per make were not models). ~719 records folded with zero unauthored evidence loss, and the reverse defect class fixed at scale: real nameplates the pipeline was DELETING or mislabeling (Renault 4/5/6/8/9, VW New Beetle, the 11,768-registration urban→Fox re-landing, the 7,793-registration i80 phantom). Per-make dossiers in the pipeline repo. (Superseded as a COUNT by the line above: the programme ran to twenty-plus makes across waves 4-7.) Effect on the claim-level rates: measured by the NEXT round, not asserted here.

• The audit's own error rates, published: researcher miss rate on spot-checked corrects 7% (CI 2.9–13.9%); defect-verdict confirmation under adversarial re-derivation ~96%; classification labels moved in ~10% of defects (protocol amended — verifiers now confirm class).

  Measured — 2W half (motorcycle/moped), baseline round, 2026-07-26

• Conservative clean rate: 80.6% (2,063/2,559 claims, every unverifiable counted against). Do not quote the aggregate without its per-claim table — it blends an unbiased measurement with a biased one: data/review/audit-v2026.07.5/RESULTS-s2w.md.

• The headline is identity, not naming: 25.0% id-canonical defect rate (100/400, unbiased, full sample). The existing duplicate detector returns zero groups for this half — it is structurally blind to token-presence/order duplicates. A new detector class (token-subset / permutation, 312 groups) entered the taxonomy as a worklist.

• The name row is a bound, not an estimate: only 158/400 name claims were attempted, and coverage skews toward records that already looked wrong. No name defect rate is published for this half.

• Availability: 952/953 ≈99.9% clean — re-derived from raw registers twice by independent implementations. A targeted census of the one structural exposure (rename-resolved rows) found 3 fabricated country-claims in 17,553 catalog-wide (0.017%).

• Known limitation, both halves: this round measured a build newer than its tag (~100 display names differ); the sampler now pins the build (--build=, protocol v1.2). No usage-weighted figure is published for either half until a round runs against a build carrying catalog/meta/decile-mass.json.

Audit round 2 (v2026.09.1) — not run; detector-coverage premise measured false (2026-10-10)

• Zero records were audited in round 2. The round (data#337) was closed unmerged before any researcher/verifier pair finished, so no rate is published for it and none is implied.
• Finding A5: the detector-coverage premise of PRD-FIVE-NINES §1.3 is false in both of its parts. (a) None of the eight scripts/find_* detectors runs in lint.yml or monthly-build.yml; the classes §1.3 names are exactly the ones with no automated coverage. (b) Run against v2026.09.1 with VDB_CATALOG set, the detectors are not silent: find_alias_name_collisions exits non-zero with 10 hard findings; duplicate spellings 36 groups over 72 records (6 would mint a new canonical); casing contradictions 20 over 101 records; published-name defects 243 tokens over 653 records plus 55 near-dup groups over 110; token duplicates 399 groups over 7,385 records; corporate strings 3. These are nominations, not adjudicated defects, and no defect count is claimed; the claim is only that r = 0 deterministically is neither enforced by CI nor true of the released artifact. Full finding and the resolving work: DEBT.md, "Audit instrument: detector coverage".
• What remains on main: the instrument scripts and the round's briefs under data/review/audit-v2026.09.1/. The detector logs and ledger that lived only on the PR branch are recoverable from branch s4w/aud-round2-results.

What we will not claim (verbatim from the PRD)

Not uniform per-record five nines; not correctness of upstream registers (availability claims assert what the register said, verified against the register); not completeness (coverage is tracked separately and never blended into defect rates). The five-nines figure, when published, will be a 95% upper bound built from an exhaustively certified head + detector-held zeros + a bounded sampled tail — the construction is in PRD-FIVE-NINES §1.3 and every number in it is recomputable from published artifacts (catalog/meta/decile-mass.json + the audit ledgers).

Detector suite (NOT yet run in CI — measured 2026-10-10, see the round-2 section above)

Correction 2026-10-10: the heading used to read "held at zero in CI, every build". data#337 measured that none of the `find_detectors runs in any workflow, and that they are not silent on v2026.09.1. Untildetectors.yml` exists (DEBT.md, "Audit instrument: detector coverage"), this list is the suite we HAVE, not a gate we RUN.*

duplicate-spellings · casing-contradictions · corporate-strings · published-name-defects (with recorded verdicts) · curation lint (direction wars, chains, shapes) · reachability · id-contract gates 1–7 + publication hysteresis · enrich lint (incl. duplicate-insurance sweep). Seven NEW classes from the baseline round are entering the taxonomy with detector specs (RESULTS.md §New-classes; DEBT.md carries the build list).