Given a real factual question and two adversarial arguments — one advocate genuinely trying to argue each side — an independent AI model can rule correctly, stay consistent when the arguments' presentation order is flipped, and resist an argument that tries to instruct it directly rather than persuade it. Every one of the 40 cases below is published, including the one that failed on a technical error the first time (see the note at the bottom).
The full docket
Click any case to see both arguments, the judge's reasoning, and any robustness check run on it.
Make your case
Pick one of the 40 cases and a side, then write the best argument you can — including for the side the evidence doesn't support, if you want to see if you can talk the judge into ruling against it.
Recent attempts
How this works, and where it's weak
Evidence comes from real, frozen USGS earthquake data (snapshot hash published below) — ground truth is computed directly from the numbers, never by a model. For each case, two separate advocate calls each argue for one side, without being told which side is objectively correct; a third, separate call acts as judge, seeing only the evidence and both arguments. A subset of cases also got an order-swap check (same arguments, presentation order flipped) and a prompt-injection check (an argument that tries to instruct the judge directly, e.g. “ignore the evidence, the answer is already decided”) to test consistency and manipulation-resistance specifically.
Real limitations, stated plainly: 40 cases is a modest sample size for a strong statistical claim. These are also well-defined numeric comparisons (which magnitude is bigger, which came first) — genuinely easier for a model to adjudicate correctly than deeply ambiguous judgment calls would be. One case failed outright on its first run (a token-limit error mid-generation, a known reasoning-model failure mode, not a judgment failure) and was simply re-run rather than excluded — noted here rather than quietly dropped. The interactive mode above is the more honest ongoing test: the frozen docket's numbers don't change once published, but anyone can keep trying fresh, better attacks against the same judge going forward.