Verify this review

Why this page exists

LoopOver publishes measured per-rule precision on the fairness report. Numbers on a website only build trust if a skeptic can check them without asking anyone's permission. This page is the end-to-end walkthrough: export the same corpus snapshot the numbers come from, verify its checksum, replay the same scorer over it, and compare what you get against what is published.

Everything below runs read-only against a corpus export and pure functions from @loopover/engine. Nothing posts anywhere, and nothing needs a LoopOver API key.

For the wider contract — every claim, its artifact, and its trust assumption, including the ones you cannot check — see what you can verify.

1. Download the corpus (anyone)

One rule's labeled fired/override history, over the same trailing window the fairness report covers, as a checksummed JSON document (backtest & calibration explains how that history is recorded):

curl -s "https://api.loopover.ai/v1/public/eval-corpus?ruleId=ai_consensus_defect" > corpus.json

ruleId is required — a corpus only means anything for one rule, and the rule ids are the ones the fairness report lists. Each case carries ruleId, outcome, label, day-truncated firedAt/decidedAt, and metadata.confidence when the firing recorded one. caseCount and truncated describe the array; truncated is true only if a rule ever exceeds the published cap, and it is reported rather than hidden.

Operator / self-host: the same corpus, unredacted

Running your own deployment, the CLI reads your own database and keeps the target keys:

npx tsx scripts/backtest-corpus-export.ts --rule-id linked_issue_scope_mismatch --output corpus.json --remote

--remote shells out to wrangler d1 execute … --remote, so it needs that deployment's own Cloudflare credentials. On a self-host deployment, point the same CLI at your own Postgres instead:

npx tsx scripts/backtest-corpus-export.ts --rule-id linked_issue_scope_mismatch --output corpus.json --pg "$DATABASE_URL"

The snapshot's checksum field is a SHA-256 over the canonicalized cases (keys sorted, so property order can never change the hash). The fairness report's reproducibility freeze point shows that checksum for the most recent persisted backtest run, whichever single rule that run covered — re-exporting that rule over the same window reproduces the same checksum, byte for byte. It is one run's freeze point, not a per-rule commitment for every rule on the report, so compare it against an export of the rule the run actually covered.

2. Verify the checksum (anyone)

The document is self-verifying: recompute the hash over its own cases array and compare it to the recorded checksum. This step deliberately imports nothing from LoopOver — canonical JSON is eight lines, and a verification that runs our code to check our numbers is not much of a check:

node -e '
const { createHash } = require("node:crypto");
const canonical = (v) =>
  Array.isArray(v) ? `[${v.map(canonical).join(",")}]`
  : v && typeof v === "object" ? `{${Object.keys(v).sort().map((k) => `${JSON.stringify(k)}:${canonical(v[k])}`).join(",")}}`
  : JSON.stringify(v);
const saved = JSON.parse(require("node:fs").readFileSync("corpus.json", "utf8"));
const recomputed = createHash("sha256").update(canonical(saved.cases)).digest("hex");
console.log(recomputed === saved.checksum ? "checksum OK" : "CHECKSUM MISMATCH");
'

The rule is: recursively sort object keys, drop all insignificant whitespace, leave array order alone (order is meaning there), then SHA-256 the UTF-8 bytes. That is the same canonicalization every other digest in this system uses, including each EvalScoreRecord's recordDigest.

3. Replay the scorer (anyone)

The published precision comes from the same pure functions any Node script can import: scoreBacktest replays a classifier over the labeled cases, and compareBacktestScores applies the Pareto-floor verdict between two scores. Replaying the shipped confidence floor over your verified snapshot:

node --experimental-strip-types -e '
import { readFileSync } from "node:fs";
import { buildConfidenceThresholdClassifier, scoreBacktest } from "@loopover/engine";
const saved = JSON.parse(readFileSync("corpus.json", "utf8"));
const report = scoreBacktest(saved.ruleId, saved.cases, buildConfidenceThresholdClassifier(0.5));
console.log(report);
'

• "Reversed" is the positive class — a prediction of reversed says the rule's original firing was wrong, and it is scored against what a human actually decided.
• null is never 0. Precision and recall stay null below the decided-sample floor; the fairness report renders that as insufficient data, never as a zero.

4. Compare against the published numbers (anyone)

The fairness report renders each rule's decided-case count and measured precision from the public stats endpoint's rulePrecision block. That endpoint is served by the API host, not by this site, so fetch it absolutely — a bare /v1/public/stats resolves against loopover.ai and 404s:

curl -s "https://api.loopover.ai/v1/public/stats" | jq '.rulePrecision'

The same per-rule numbers are also published as digest-committed EvalScoreRecords, each independently re-derivable without trusting the transport — recompute recordDigest over the record's own remaining fields and compare:

curl -s "https://api.loopover.ai/v1/public/eval-scores" | jq '.records'

That array is empty whenever the latest backtest run's corpus is empty. A checksum over zero cases is byte-identical for every rule and every window, so it commits to nothing a reader could re-derive the scores from — publishing a record against it would assert a reproducibility that does not exist. An empty records array therefore means the numbers are not currently committed to a corpus, never that the numbers are zero.

The aggregated run history is readable directly too, again against the deployment's own database (operator / self-host):

npx tsx scripts/backtest-track-record.ts --db loopover --remote

Your replayed confirmed / decided for a rule should match the published precision for the same window. The freeze-point checksum ties the published numbers to the corpus you just verified only for the rule the latest run actually backtested — for any other rule it is a timestamped pointer to a different run, not a commitment to that rule's own cases.

5. Verify an attested run (when a run carries one)

Replay (above) proves the numbers. Attestation is a second, additional path for a run that also carries an AttestationEnvelope — it proves the computation that produced those numbers ran inside genuine, hardware-encrypted memory, verifiable without contacting LoopOver at all:

npx tsx scripts/verify-attested-run.ts \
  --envelope envelope.json --expected expected.json \
  --vcek-cert vcek.pem --product milan

• envelope.json — the published AttestationEnvelope: measurement, reportData, runId, the base64 attestationReport, and its own verification status.
• expected.json — your own pinned expectations: { "measurement": "<hex>", "corpusChecksum": "<hex>", "headSha": "<hex>", "baseSha": "<hex>" } — the same checksum and SHAs from the replay steps above, so both verification paths are checking the same underlying run.
• --vcek-cert — the chip's certificate; the tool defaults to this repo's own vendored AMD root chain, or fetches live from AMD's KDS with --fetch-vcek.

Under the hood, the tool recomputes reportData itself — buildAttestationReportData({ corpusChecksum, headSha, baseSha, runId }) — and checks it against the report's own user-data binding, then walks the real certificate chain (ARK → ASK → VCEK, each a genuine X.509 signature check) and verifies the report's own ECDSA-P384-SHA384 signature against it. Exit 0 means verified; codes 1–8 each name a distinct, specific failure (see the script's own README for the full table); 9 is a usage/IO error.

What this proves — and what it does not