Back to AI Research

AI Research

Grouping the Stochastic Machine: Precision, Not Cap... | AI Research

Key Takeaways

  • Grouping the Stochastic Machine: Precision, Not Capability, as the Frontier Metric for AI Systems argues that current AI benchmarks focus on the wrong metric...
  • Frontier language models are compared, marketed, and benchmarked on capability -- what their best or average output can achieve.
  • The models have saturated accuracy: their mean output lands on the target.
  • What now separates one system from another in practice is precision: how tightly concentrated their outputs are around that target across repeated, identical requests.
  • Borrowing the marksman's distinction, capability is where the average shot lands; reliability is the size of the group.
Paper AbstractExpand

Frontier language models are compared, marketed, and benchmarked on capability -- what their best or average output can achieve. I argue this measures the wrong axis. The models have saturated accuracy: their mean output lands on the target. What now separates one system from another in practice is precision: how tightly concentrated their outputs are around that target across repeated, identical requests. Borrowing the marksman's distinction, capability is where the average shot lands; reliability is the size of the group. I make three claims. First, precision, not capability, is the frontier differentiator between systems, and benchmark culture systematically fails to measure it, reporting central tendency rather than spread. Second, precision is measurable, cheaply and without circularity, by running a fixed suite of deterministically scored tasks many times at fixed temperature and computing the per-task consistency of outcomes -- no model-in-the-loop grader required. Third, the measurement is not merely descriptive but decision-guiding: it separates consistent failures (a tight group off-centre, correctable by the operating discipline of Paper 1 -- a sight adjustment) from scattered failures (a wide group, correctable only by changing the model or its sampling -- a rifle problem). I define a grouping metric, specify a harness, and show how tracking a human-AI pair's grouping over time yields the compounding signal that Paper 1's field study requires. A first real run, since replicated, illustrates both the method and its most important limit: one measured gap was closed completely by a single rule (0/5 -> 5/5), while a suite of tasks authored from the rules themselves found no value, because a frontier model already embodies explicit good practice -- establishing that a discipline's worth is found by measurement on real work, not constructed from its own rulebook.

Grouping the Stochastic Machine: Precision, Not Capability, as the Frontier Metric for AI Systems argues that current AI benchmarks focus on the wrong metric. While industry standards measure "capability"—the average or best performance of a model—the author, George Andrikopoulos, contends that production reliability depends on "precision": how consistently a model produces the same output across repeated, identical requests.

The Shift from Accuracy to Precision

Frontier language models have largely converged on accuracy, meaning their average output is typically on target. However, they differ significantly in their variance. A model that is accurate on average but highly scattered is less useful for engineering than a model that is consistently precise.
Andrikopoulos uses a marksmanship analogy to explain this:

  • Accuracy is where the average shot lands.

  • Precision is the size of the grouping.
    For an operator, a "tight" group of failures is preferable to a "scattered" one. A tight group of errors indicates a systematic issue that can be corrected with a specific rule or "sight adjustment." A scattered group of errors suggests a fundamental problem with the model or its sampling settings, which cannot be easily corrected by an operator.

Measuring Reliability Without Circularity

To measure precision without relying on a model-in-the-loop grader—which would introduce circularity—the paper proposes a harness that uses deterministic, binary tasks. By running a fixed suite of tasks (such as code compilation, linting, or type-checking) multiple times at a fixed temperature, one can calculate the consistency of the outcomes.
The harness produces a decision table to guide operators:

  • Decisive Pass: The model is on target and precise; no action is needed.

  • Consistent Failure: The model is precise but off-center; this is "zeroable," meaning an operator can write a specific skill or rule to fix the error.

  • Scattered Failure: The model is imprecise; no rule will help, and the operator should change the model or lower the sampling temperature.

Findings and Practical Limits

In a test using a six-task Rust suite, the author demonstrated that a gap in performance could be closed completely by identifying a consistent failure and applying a single rule. The pass rate for that task improved from 0/5 to 5/5.
However, the research also highlights a significant limitation: you cannot "author your way to a delta." When the author attempted to build a suite of tasks based on their own rulebook, the frontier model already passed them all. This indicates that highly capable models often embody explicit good practice already. The true value of an operating library lies in finding and patching the residual, model-specific gaps that are not known in advance.

Key Considerations for Implementation

The paper notes several practical constraints for this measurement approach:

  • Scorer Bias: Deterministic scorers can inadvertently penalize disciplined code if the scorer is "shape-blind" (e.g., rejecting a more robust, non-zero type because it doesn't match a rigid template).

  • Human Judgment: While the screening signal (pass rate) is objective, confirming whether failures share a single cause requires human or model judgment, which is not deterministic.

  • Context Decay: Measurement programs can accumulate data faster than the context required to interpret it. Comparisons must be pinned to specific run identifiers and configurations rather than relying on bare numbers.

  • Scope: This method is designed for verifiable engineering tasks and does not apply to open-ended generative work where a single "correct" target may not exist.

Comments (0)

No comments yet

Be the first to share your thoughts!