Menzio
Back

Method

How we measure

AI answers change from one run to the next. A single answer per prompt is a snapshot, not a measurement. This page explains exactly what we count, so the numbers in your reports can be checked.

What we ask, where and when

What counts

Ranges, not single numbers

Every rate comes with a 95% range, like 38% (31–45%). Runs of the same prompt are correlated, so we resample prompts, not answers (a bootstrap over prompts, 1,000 resamples). With fewer than 10 prompts or 30 answers we use the Wilson interval instead. When this week's range overlaps last week's, the report says "no clear change". The headline number pools the last 4 weeks, which narrows the range.

How wide the range is depends on how many answers it rests on. Approximate margin (±, percentage points) for a true rate of 50%, the widest case, and 20%:

AnswersTypical sample50% · 90%50% · 95%20% · 95%
281 prompt on 1 engine, 4 weeks±14.8±17.4±14.4
17525 prompts on 1 engine, 1 week±6.2±7.3±5.9
70025 prompts on 4 engines, 1 week±3.1±3.7±3.0
2,80025 prompts on 4 engines, 4 weeks±1.6±1.9±1.5
11,200100 prompts on 4 engines, 4 weeks±0.8±0.9±0.7

Wilson interval with independent answers. Real ranges are a little wider because runs of one prompt move together; the bootstrap accounts for that.

Known limits