Talk to us
$

$ teds read --post statistical-rigor-in-ai-evals-confidence-intervals-sample-sizing

The Actuarial Approach to AI Evals: Wilson Confidence Intervals and Statistical Sample Sizing

Why running 20 test cases gives you virtually zero statistical confidence in your agent's reliability, and how to use Wilson score intervals to measure real performance.

ByHarpreet Sahota, Hacker-in-Residence/Principal Developer Relations EngineerXLinkedIn
AI EngineeringEvaluation & SafetyLLM Systems

This is Part 3 of a 5-part series on evaluating and hardening production AI agents. See also: Part 1: Moving Beyond LLM-as-a-Judge, Part 2: Deterministic State Evaluations, Part 4: Curating Golden Trajectories, and Part 5: Building an Agent CI/CD Pipeline.

In AI engineering discussions, teams frequently report evaluation metrics with unfounded precision:

“Our customer support agent achieved an 85% task completion rate on our benchmark suite.”

When you inspect the benchmark under the hood, you discover that the suite consists of 20 test cases, and the model completed 17 of them ($17 / 20 = 85.0%$).

To anyone trained in actuarial science or biostatistics, this claim is an illusion.

A point estimate without a confidence interval is not a measurement—it is an anecdote. When deploying autonomous agents that touch live databases, issue payments, or interact with customers, reporting point averages on small sample sizes creates false security.

Here is why most AI evaluation benchmarks fail basic statistics, how to compute robust confidence intervals for Bernoulli trials, and how to determine the exact sample size needed to detect regressions.

The Small-Sample Illusion

Every evaluation of an agent on a deterministic task is a Bernoulli trial: the agent either achieves the goal state ($X = 1$) or it fails ($X = 0$).

When engineering teams calculate a confidence interval, they often rely on the standard normal approximation (the Wald interval):

$$\hat{p} \pm z_{1-\alpha/2} \sqrt{\frac{\hat{p}(1-\hat{p})}{n}}$$

The Wald interval makes a critical assumption: that the sampling distribution is approximately normal. For small sample sizes ($n < 100$) or when success rates approach the boundaries ($p > 0.85$ or $p < 0.15$), the Wald interval breaks down catastrophically. It produces intervals that spill outside $[0, 1]$ and drastically underestimates uncertainty.

Enter the Wilson Score Interval

For binomial proportions in high-stakes systems, we use the Wilson score interval. The Wilson interval inverts the score test and performs reliably across all sample sizes and probability ranges:

$$w = \frac{\hat{p} + \frac{z^2}{2n} \pm z \sqrt{\frac{\hat{p}(1-\hat{p})}{n} + \frac{z^2}{4n^2}}}{1 + \frac{z^2}{n}}$$

Where:

  • $\hat{p}$ is the observed sample proportion ($x / n$)
  • $n$ is the total number of test runs
  • $z$ is the standard normal quantile ($z = 1.96$ for a 95% confidence interval)

What the Numbers Actually Say

Let us apply the Wilson score interval to typical benchmark claims:

  • Case A: 17 successes out of 20 runs ($\hat{p} = 85.0%$)

    • Point estimate: 85.0%
    • 95% Wilson Confidence Interval: [63.9%, 94.8%]
    • Reality: The true production capability could be a failing 64% or a strong 95%. Your test suite cannot distinguish between a mediocre model and a production-grade model.

  • Case B: 85 successes out of 100 runs ($\hat{p} = 85.0%$)

    • Point estimate: 85.0%
    • 95% Wilson Confidence Interval: [76.9%, 90.6%]
    • Reality: The uncertainty window narrows, but still spans a 14-percentage-point margin of error.

  • Case C: 425 successes out of 500 runs ($\hat{p} = 85.0%$)

    • Point estimate: 85.0%
    • 95% Wilson Confidence Interval: [81.6%, 87.9%]
    • Reality: Only at $n = 500$ do we have statistical proof that the true reliability exceeds 80%.

How to Calculate Wilson Score Intervals in Python

Here is a lightweight, zero-dependency helper to compute exact confidence bounds for evaluation suites:

import math

def wilson_score_interval(successes: int, total: int, confidence: float = 0.95) -> tuple[float, float]:
    """
    Compute the Wilson score confidence interval for a binomial proportion.
    """
    if total == 0:
        return (0.0, 0.0)
    
    # z-value for two-tailed standard normal distribution
    # 0.90 -> 1.645, 0.95 -> 1.960, 0.99 -> 2.576
    z_map = {0.90: 1.645, 0.95: 1.95996, 0.99: 2.57583}
    z = z_map.get(confidence, 1.95996)
    
    p_hat = successes / total
    denominator = 1.0 + (z**2) / total
    centre_adjusted_probability = p_hat + (z**2) / (2 * total)
    adjusted_standard_deviation = math.sqrt((p_hat * (1 - p_hat) + (z**2) / (4 * total)) / total)
    
    lower_bound = (centre_adjusted_probability - z * adjusted_standard_deviation) / denominator
    upper_bound = (centre_adjusted_probability + z * adjusted_standard_deviation) / denominator
    
    return max(0.0, lower_bound), min(1.0, upper_bound)

Whenever your CI evaluation pipeline completes, report both the point estimate and the interval:

successes, total = 82, 100
low, high = wilson_score_interval(successes, total, confidence=0.95)
print(f"Agent Task Success: {successes/total:.1%} [95% CI: {low:.1%} – {high:.1%}]")
# Agent Task Success: 82.0% [95% CI: 73.4% – 88.2%]

Sample Sizing: How Many Tests Do You Actually Need?

When architecting an evaluation suite, the fundamental engineering constraint is: How many test runs are required to detect a regression?

If your baseline agent has an 85% success rate, and a new prompt or model fine-tune drops performance to 78%, you need sufficient statistical power to detect that 7% drop without false alarm.

To detect a difference between two proportions ($\delta = p_1 - p_2$) with significance level $\alpha$ (typically 0.05) and statistical power $1 - \beta$ (typically 0.80):

$$n \approx \frac{\left(z_{1-\alpha/2}\sqrt{2\bar{p}(1-\bar{p})} + z_{1-\beta}\sqrt{p_1(1-p_1) + p_2(1-p_2)}\right)^2}{\delta^2}$$

Practical Rule of Thumb for Engineering Teams

  • To reliably detect a 10% shift (e.g., 85% vs. 75%): You need roughly ~180 runs per variant.
  • To reliably detect a 5% shift (e.g., 85% vs. 80%): You need roughly ~750 runs per variant.

If your benchmark suite only evaluates 30 test cases on a pull request, you literally cannot detect a 10% regression. An apparent drop from 26/30 (86.7%) to 23/30 (76.7%) is statistically indistinguishable from chance ($p > 0.30$).

Pass@k vs. Pass^k: Disentangling Temperature from Capability

Agents are non-deterministic systems. When evaluating models at temperature $T > 0$, single-run evaluations conflate sampling noise with model capability.

We separate evaluation into two distinct metrics:

  1. Pass@k (Exploration / Ceiling): Does the agent solve the task at least once across $k$ independent attempts? This measures capability ceiling—whether the model can formulate a viable trajectory.

  2. Pass^k (Reliability / Production Readiness): Does the agent solve the task every single time across $k$ consecutive runs? $$P(\text{pass}^k) = p^k$$ For enterprise workflows, Pass^3 or Pass^5 is the real deployment gate. If an agent cannot complete a workflow 3 times out of 3 without failure, it is not ready for unattended operation.

The Next Step

Now that we have established deterministic state testing and statistical sample sizing, where do we get the actual test cases?

In Part 4, we explore Curating Golden Trajectories: how to curate adversarial edge-case datasets, simulate tool failures, and stress-test agent reasoning before users do.

Need Actuarial Rigor in Your AI Testing?

Most AI consulting groups give generic prompt advice. We approach AI evaluation from applied statistics and systems engineering.

If you need to establish statistical benchmarks, design sample-efficient evaluation pipelines, or prepare your models for production compliance: