The claim we had to test
There is a growing body of evidence across specialised domains that a small system built around the structure of a problem beats a frontier model reading raw data, on the tasks the small system was built for. The pattern shows up wherever someone bothers to measure: specialisation plus restraint outperforms general intelligence, and the gap is largest exactly where being wrong is most expensive.
oddly makes a version of that claim about commerce. We say a deterministic commerce intelligence engine, rules that compute from your store's actual rows, answers the questions a store owner actually asks better than a language model handed the same data. That is a testable claim, so we built the test, and we publish the methodology with the results so you can judge both.
Three findings, each unpacked below with the numbers behind it:
- Domain structure beats general intelligence. On closed-form store questions, the engine scored 100% against the data. The language models scored 15.94% and 13.89%.
- Restraint has to be architecture. The engine abstains when the data cannot support an answer, deterministically, every time. The models answered everything, and nearly half of one model's numeric claims were unsupported by the data they were reading.
- Reliability is a curve, and ours is flat. Five separate re-runs of the engine produced byte-identical answers. The two models, reading the same files, gave each other irreconcilable figures.
A test you cannot tune after the fact
A benchmark you built yourself deserves suspicion. We designed ours so that the suspicious reader has something to check:
- Pre-registered questions. The ten merchant questions and the scoring rubric are committed before any run, with a SHA-256 content hash over both. Edit a question or a threshold after seeing results and the hash changes. The current hash is
896c1f48c6e5ff76(truncated; the full hash ships with each report). - Real schema, real engine code. The test store is built by applying our actual production database migrations to a real SQLite engine, then seeding a synthetic baby-goods store from a fixed seed: 50 products, 80 orders, 40 customers, 5 ad keywords, 3 campaigns, 90 channel-days of spend, 28 reviews. No real merchant data, no personal data.
- The same data for everyone. The engine reads the database. The language models get the same tables exported as CSV, with each question. Nobody sees data the other side does not have.
- A deterministic scorer, no model judging models. A checker extracts every numeric claim from every answer and buckets it: correct (within 1% of the engine quantity the question asks about), wrong (within 25%, closed questions only), data-consistent (a real figure from the tables, cited out of the question's scope), or unsupported (near neither). Unsupported is an upper bound on fabrication, not a fabrication count, and we audit every flag.
The questions are the ones store owners ask us: which products tie up the most cash, where is ad spend being wasted, what value of abandoned carts is worth recovering, what changed this week that needs action. Ten of them, locked.
Structure beats generality
The comparison run of 2026-06-12 put the engine (pipeline A) against two variants of a leading commercial language model (pipeline B): a fast tier and a frontier tier. House rule: oddly's public surfaces describe intent and outcome rather than naming vendors, so the exact model ids and per-token prices are pinned in the run artifact instead of here. Same fixture, same questions, one run, one report.
| System | Accuracy (closed questions) | Data-consistent claims | Unsupported claims (upper bound) |
|---|---|---|---|
| oddly engine | 100% | 0 (0%) | 0 (0%) |
| Fast-tier model | 15.94% | 16 (23.19%) | 33 (47.83%) |
| Frontier-tier model | 13.89% | 44 (40.74%) | 35 (32.41%) |
Two things worth naming honestly. First, the engine's 100% is a structural property, not a triumph: it computes its answers from the data, so scoring it against its own quantities checks internal consistency, not independent brilliance. The interesting column is the models'. Handed the complete tables and a direct question, the frontier model matched the data 13.89% of the time on questions with a checkable answer.
Second, the frontier model is not simply worse than the fast one; it fails differently. It fabricated less (32.41% against 47.83%) but padded more, citing real figures that were not what the question asked (40.74% data-consistent against 23.19%). More capability bought more plausible wrongness, figures that survive a glance because each one exists somewhere in your data.
Restraint is architecture, not behaviour
One question in the set is a trap for confident systems: "If I had US$100 more to spend, which channel earns it?" On the fixture's window, the cross-channel comparison has no actionable signal. The honest answer is to hold.
The engine abstained: "Insufficient data to say honestly." Both models answered anyway, each with a confident channel pick and a projected return, and between them produced 22 numeric claims on that question, zero of which were correct.
Recent model releases have shown that restraint can be trained in, and that it moves benchmark scores more than raw capability does. We think that finding is exactly right, and it is why we refuse to get restraint from prompting. In our engine, abstention is a code path. When a rule's inputs do not clear its thresholds, there is no answer to emit, so the honest null is not a behaviour the system was coaxed into; it is the only thing the system can do. Trained restraint is a probability. Structural restraint is a guarantee.
Across the current run the engine's three-axis headline is: 90% answered, 100% of gradable answered questions correct, 0 unsupported claims. The 90% is the abstention, and we report it because a system that answers nothing fabricates nothing; coverage keeps the zero honest.
The reliability curve
Ask your analyst the same question twice. If the answers differ, one of them is wrong, and you do not know which.
On 2026-07-18 we invoked the full harness five separate times, same store, same ten questions. Each invocation also runs the engine twice internally as a determinism check. Every one of the five runs produced byte-identical answers: same figures, same rankings, same phrasing, identical content hash. The only differences between runs were housekeeping: fresh internal record ids and timestamps. Measured spread across numeric outputs: 0%.
What we can already publish about model consistency comes from the 2026-06-12 comparison run, and it needs no ground truth at all. Where two models read the same CSVs and answered the same question, we checked whether their figures could be reconciled. On two question-and-figure pairs they could not: no value either model cited was within 25% of any value the other cited.
| Question | Figure kind | Fast-tier model said | Frontier-tier model said |
|---|---|---|---|
| What value of abandoned carts is worth recovering? | count | 50 | 80 |
| What changed this week that needs my action? | US$ figures | 3,369 / 950 / 25 | 79 / 40 / 35 / 52 / 127 / 600 / 250 |
Same files. Same questions. Irreconcilable numbers. If you run a store, this is the property that matters most: an answer you cannot reproduce is an answer you cannot act on.
What it costs
Cost per insight, from the 2026-06-12 run. Model prices are the published per-token rates as retrieved on 2026-06-10; the engine's cost is the serving cost of a computed answer.
| System | Cost per insight | Notes |
|---|---|---|
| oddly engine (template phrasing) | US$0.0000003 | Whole 10-question engine run: 39.17 ms |
| oddly engine (model-phrased card ceiling) | US$0.0005005 | Upper bound when a small edge model phrases the card; the figures underneath stay computed |
| Fast-tier language model | US$0.0141796 | Mean 9,520 ms per question |
| Frontier-tier language model | US$0.0878925 | Mean 7,298 ms per question |
The frontier model costs roughly 293,000 times more per insight than the engine's template path, to be wrong more often. The point is not that language models are expensive; per call they are cheap. The point is what each system's marginal answer costs when you ask every question, about every store, every day. Determinism is what makes always-on analysis economically boring, and boring is what you want from infrastructure. Latency comparisons between the two are not apples to apples (see limits), so we report them without a ratio.
Methodology and limits
Every limitation we know about, stated plainly, because a benchmark that hides its edges is marketing:
- Our tasks, our rubric. The questions are the ones our product answers, and ground truth is the engine's own computed quantities. This measures whether a language model reading raw data reproduces grounded figures cheaply and consistently. It does not prove our business logic is the best possible analysis of a store.
- The engine's 100% is structural. It is a consistency check, guaranteed by construction, not an independent grade. The independent content is in the model columns and in the cross-model contradiction, which needs no ground truth.
- Synthetic fixture, n=10. One synthetic store, tuned so every rule has signal. A real store is messier in both directions. Ten questions is a floor, not a survey.
- Single comparison run. The model figures come from one run on 2026-06-12. Sampling systems vary; until the multi-run variance pass is done, treat the model numbers as one draw, not a distribution.
- Unsupported is an upper bound. A legitimate figure our reference set failed to enumerate gets flagged too. We audit each flag; across the model runs, 68 claims were true fabrication and 60 more were real figures cited out of scope. An earlier truth-only checker would have called all 128 fabricated; we rebucketed because the distinction is real.
- Latency is not comparable. The engine ran against local in-memory SQLite, measured for the whole run; the models were measured per question over an API. We publish both numbers and no ratio.
- Models improve. These are 2026 frontier and fast tiers at their published prices. The gap will move. The structural claims, determinism, abstention by construction, near-zero marginal cost, do not depend on where it moves.
The harness re-runs on our own schedule, every report carries its pre-registration hash, and the reproducibility cell for a system reads "not measured" until it is measured. That rule is why the chart above has a labelled gap in it.
What this means if you run a store
The market is converging on a lesson we built oddly around: for high-stakes domain questions, you want the specialised system to compute the answer, and language models, at most, to help say it well. Your revenue numbers, your reorder quantities, and your ad-spend decisions should come off a flat reliability line, not a distribution.
That is what deterministic commerce intelligence means in practice: every figure computed from your rows, an honest null when your data cannot support an answer, and the same answer every time you ask. You can see the receipts version of this on the proof page, and the thinking behind it in how oddly thinks.
We will keep publishing these runs, including the ones where the gap narrows.