The three machines on the shelf
Strip the category noise away and every tool a merchant can buy for "what should I do with my store" is one of three architectures:
- Report-only dashboards. They compute aggregates from your data and chart them. They do not answer questions; they show you numbers and leave the decision to you.
- Copilot chat layers. A language model sits over your data. You ask in plain words; it reads what it can reach and generates an answer.
- Autonomous agents. The same generative core, now holding the keys: it does not just tell you to cut the campaign, it cuts the campaign.
These classes are usually compared on features. We think that is the wrong axis. The right axis is the one from part one of this series: if you ask the same question of the same data five times, do you get the same answer? Because everything a merchant does downstream, trusting a figure, acting on a ranking, letting software touch a budget, rests on that property.
In part one we published the instrument: ten pre-registered merchant questions, a fixture store built from our real production schema, a deterministic scorer, and a content hash (896c1f48c6e5ff76, truncated) that makes after-the-fact tuning visible. This part uses the same instrument and the same runs. No new claims are manufactured for the framing; where a class has not been measured, the cell says so.
The test that sorts them
The consistency test is deliberately humble. It does not ask which system is smartest. It asks: when the input has not changed, does the output change? On 2026-07-18 we ran our engine through five separate full invocations of the harness, ten questions each. All five produced byte-identical answer sets: same figures, same rankings, same phrasing, 0% numeric spread. That flat line is the reference for everything below.
A system's position against that line tells you what kind of thing it is. There are exactly three interesting positions: flat but silent, variable and talking, and variable and acting.
Dashboards: stable numbers, no answers
Here is the honest concession first: a good dashboard passes the repeatability half of the test. Load the same data twice and the revenue chart shows the same revenue. Dashboards are deterministic aggregation, and determinism is exactly what we keep arguing for.
But run a dashboard against the actual question set and the problem is not the answers; it is that there are none. "Which products tie up the most cash?" A dashboard shows an inventory table and lets you sort it. "Where is ad spend being wasted?" It shows spend by campaign. The question the merchant asked, which of these should I act on, and what is it worth, is precisely the part left as an exercise for the reader.
Part one made the point that a system that answers nothing fabricates nothing, which is why we always publish coverage next to correctness. On the ten-question set, a report-only surface has a coverage of zero: it never commits to an answer a scorer could grade. Its consistency is real but idle, the consistency of a system that never reaches a conclusion. You are the inference layer, and your own five re-runs, tired on a Sunday night versus fresh on a Monday, are not byte-identical either.
Copilots: an answer distribution, not an answer
The copilot architecture is: a language model reads your exported data and generates a response to your question. That is not a caricature; it is the design, and it is also exactly what our comparison run measures. In the 2026-06-12 run, two tiers of a leading commercial language model, a fast tier and a frontier tier, were handed the complete fixture tables as CSV with each of the ten questions. Same files, same questions, no retrieval tricks withheld: the copilot setup in its cleanest form.
The results, from part one:
| System | Accuracy (closed questions) | Unsupported claims (upper bound) |
|---|---|---|
| oddly engine | 100% | 0 (0%) |
| Fast-tier model | 15.94% | 33 (47.83%) |
| Frontier-tier model | 13.89% | 35 (32.41%) |
For the consistency question specifically, the sharpest evidence needs no ground truth at all: put two systems of the same class in front of the same files and see if they can agree with each other. On two question-and-figure pairs they could not, at all: no value either model cited was within 25% of any value the other cited. Asked what value of abandoned carts was worth recovering, one said 50 carts, the other 80. Asked what changed this week, one produced the dollar list 3,369 / 950 / 25; the other produced 79 / 40 / 35 / 52 / 127 / 600 / 250.
And on the trap question, "if I had US$100 more to spend, which channel earns it?", where the data supports no answer and the honest move is to hold, both copilot-class systems answered anyway: 22 numeric claims between them, zero correct. The engine abstained, because when a rule's inputs do not clear its thresholds there is no answer to emit. Restraint as a code path, not a behaviour.
One measurement discipline note: we have not yet run the paid multi-run variance pass on the models themselves, so we do not publish a "model re-run spread" number, and the chart in part one labels that cell rather than assuming it. Sampling systems are non-deterministic by design; the cross-model contradiction above is the measured stand-in until the multi-run pass lands. What we can say from the measured runs: the copilot class hands you a draw from a distribution. Ask again and you may get another draw. Our engine's five-run line is what the alternative looks like: 0% spread, every time.
Autonomous agents: variance with a checkout
The third class takes the same generative core and gives it hands. The pitch is seductive: skip the reading, skip the deciding, let the agent optimise your campaigns while you sleep.
Be precise about what we have and have not measured here. We have not run an autonomous shopping-ads agent through our harness; there is no single representative system to pin down, and this series does not manufacture numbers for framing. What we can do is state the structural point, because it follows from the architecture, not from a benchmark: an agent built on a sampled generator inherits the copilot's variance, and then attaches spending authority to it. In the copilot column, run-to-run variance produces a different sentence. In the agent column, it produces a different budget. The failure mode moves from "the report was wrong" to "the money moved".
That is why the interesting question for the agent class is not "how smart is it?" but "what is it allowed to do when it is wrong?" Our answer is an asymmetry we treat as architecture: actions that are safe, reversible, or spend-decreasing can run automatically; anything that increases spend, moves money, or touches access waits for a human. The engine deciding is deterministic; the gate on acting is structural. Part three of this series shows those guarantees operating in production, with real audit-trail excerpts.
The same test, one line each
The whole argument in one table. Measured cells cite runs; structural cells are properties of the architecture; unmeasured cells say so.
| Class | Same question, five times | Coverage of decision questions | When it is wrong |
|---|---|---|---|
| Report-only dashboard | Same charts (structural: deterministic aggregation) | 0%: it never answers, you do | You misread it; the tool carries no claim |
| Copilot chat layer | A draw from a distribution (measured stand-in: two same-class systems, irreconcilable figures on the same files) | 100% answered, 13.89% to 15.94% correct on closed questions (measured, 2026-06-12) | You act on a wrong sentence |
| Autonomous agent | Not measured; inherits generative variance (structural) | Not measured | The money moves before you read anything |
| oddly engine | Byte-identical, 0% spread (measured, five invocations, 2026-07-18) | 90% answered, 100% of gradable answered correct, 0 unsupported (measured) | Abstains when data cannot support an answer; increases wait for you |
Methodology and limits
The limits, stated plainly, because a comparison that hides its edges is marketing:
- Two of the four rows are framed, not run. Dashboards cannot sit the test (they never produce a gradable answer, which is itself the finding), and the agent class has no single representative system we could run without picking a vendor to name. Neither row carries an invented number.
- The copilot numbers are one run of two systems. The 2026-06-12 comparison is a single draw from each model tier, not a distribution. The multi-run variance pass is pending; until it lands, the measured consistency evidence for that class is the cross-model contradiction, which needs no ground truth.
- Class, not product. Our public surfaces describe intent and outcome rather than naming vendors, so the copilot instance is described by tier, with exact model ids and prices pinned in the run artifact. A specific copilot product with retrieval tuned to these exact questions could do better than the clean-room setup; it would still be sampling.
- The engine's flat line is a structural property. We publish it as a measurement because we measured it (five invocations, content-hash identical), but determinism is by construction. The interesting engine numbers remain coverage and correctness, published together in part one.
- Everything else from part one still applies. Synthetic fixture, ten questions, our tasks and rubric, pre-registration hash on both. Full methodology and every number's definition live there.
What this means if you run a store
The three classes are not three qualities of the same thing; they are three different contracts. A dashboard promises you stable numbers and no opinion. A copilot promises you an opinion drawn from a distribution. An agent promises to act on such a draw. None of the three offers what the decision actually needs: a conclusion you can reproduce, from data you can trace, with action gated by the stakes.
That contract is what deterministic commerce intelligence is: every figure computed from your rows, an honest null when the data cannot support an answer, the same answer every time you ask, and autonomy only where being wrong is cheap and reversible. The receipts version is on the proof page. Part three shows the guarantees running in production; part four prices the whole argument.