A claim survives only by surviving attack. Green tests prove the machine ran; they prove nothing about whether the conclusion is true. Before we report a result, we try to break it. Through that work we have come to value:
That is, while the items on the right are tempting, we trust the items on the left.
Reconcile from source. Recompute the number from the rawest data on disk, never from a derived summary, and never from your own prior prose. A number you have not re-derived is unverified.
Fairness. Same conditions, budget, inputs, and n across everything compared. Name what differed; an unequal comparison is a confound wearing a result's clothes.
Power. What is n? One sample, one seed, one model is exploratory, not a finding. Does the claim's strength match the evidence's strength, and did it clear a gate?
Confound. What else could produce this besides the stated cause: a bug, a harness artifact, selection? Default to "there is a confound" until the obvious ones are ruled out.
Falsifier. What experiment would disprove this, and has it run? If it never executed, the claim is unproven. If the artifact is not on disk, the claim has outrun the evidence.
Report the weakest defensible restatement, with the uncertainty leading, not buried in a footnote. A clean "supported" is earned only after a genuine attempt to refute it failed.
Justin Herrick, and anyone who would rather be right than confident
ultravalidate · a Claude Code skill that validates a claim before you trust it · fused, with three other disciplines, into flint