For brands and marketing
First, find out whether the category has a core and who is in it: 100 to 200 runs answer that. To track the brand’s position, use rounds of 1,000 and treat changes below ~4 points as noise.
Methodological study · FA-MET-01
How to Measure What an AI Recommends
How many queries it takes to tell an isolated observation from a real pattern.
In short
One question to an AI is a sample, not a portrait. At scale, each brand’s presence converges to a stable value and follows classical statistics: with 1,000 runs, it is measured with a margin of at most ±3.1 percentage points.
±3.1 pp margin for each presence with 1,000 runs
0 of 10 top brands moved outside that margin going from 1,000 to 5,000 runs
38 × 144 distinct brands in 1,000 runs: concentrated and fragmented market
29% of the rare brands in the first 1,000 runs did not return in the next 4,000
What we found
Three experiments in the ChatGPT interface: 1,000 runs in a concentrated market, 1,000 in a fragmented one, and an extension of the latter to 5,000.
- 01
There is a stable pattern behind the variation
At scale, each brand’s presence converges to a measurable value. In the concentrated market, two brands appear in 99.9% of answers and five form a core above 50%.
- 02
The shape of the pattern changes from one market to another
In the fragmented market, the leader appears in 70.3% of answers and the universe reaches 144 brands in 1,000 runs, against 38 in the concentrated one. Even so, both have the same number of brands above 25% and above 10%.
- 03
Precision follows classical statistics
With 1,000 runs, presence is measured within ±3.1 points. Multiplying the collection by five, the largest change among the top ten brands was 2.9 points, within the predicted margin.
- 04
The tail needs a different scale
In the fragmented market, the brand universe is still growing at 5,000 runs, ever more slowly. And part of the tail seen in small samples is not confirmed with more volume.
Experiment 3 · fragmented market
The hierarchy measured at 1,000 runs held at 5,000
Band: 95% interval calculated with 1,000 runs. Dot: presence measured with 5,000.
Brand A 70.3% → 69.4%
Brand B 68.8% → 68.2%
Brand C 55.2% → 55.9%
Brand D 54.3% → 57.2%
Brand E 32.7% → 31.5%
Brand F 31.7% → 32.4%
Brand G 25.8% → 25.3%
Brand H 13.5% → 13.8%
Brand I 13.5% → 13.3%
Brand J 6.0% → 6.1%
How many runs are needed
Precision depends only on the number of runs and the brand’s own presence. The right collection size depends on what you want to know.
| Goal | Calls | What it delivers |
|---|---|---|
| Initial diagnosis | 100 to 200 | Whether the market is concentrated or fragmented, and the approximate top (±7 to 10 pp) |
| Visible hierarchy | 1,000 | Presence within ±3 pp; reliable order between brands more than ~6 pp apart |
| Change between rounds | 1,000 per round | Changes above ~4.4 pp (presence near 50%) can be told apart from noise |
| ±1 pp precision | ~9,600 | Presence within ±1 pp at any level |
| Tail down to 0.1% | ~3,000 | 95% chance of seeing each brand with 0.1% presence at least once |
| Tail down to 0.02% | ~15,000 | Same for brands at 0.02% |
Margins at 95% confidence.
Why it matters
For researchers and journalists
One question to an AI is a coin toss. Claiming what an AI recommends takes hundreds of independent samples, ideally 1,000 or more, and care with conclusions drawn from the rare tail.
For people building with AI
If the product depends on a consistent recommendation, consistency has to be measured, not assumed. Some choices exceed 90%; others vary in 30% or more of answers.
Method, in short
Collected through the ChatGPT interface, in the browser, by automation, with web search enabled. One fixed question per experiment, written the way a consumer would type it, each run in a new conversation. Every brand mentioned was extracted, normalized to the parent brand and anonymized with letters. 95% intervals by the Wilson method.
Limitations
One environment and two categories, and the interface does not allow the answering model to be fixed or recorded. The statistical regularity should hold for other models and access environments; the brand composition not necessarily (see FA-MET-02).
Read the full study
The three experiments in detail, with 9 figures, 5 tables, a study sheet and references.
How to cite
Beckmann, R. (2026). How to Measure What an AI Recommends: How many queries it takes to tell an isolated observation from a real pattern (FA-MET-01, version 1.0). First Answer. https://rodrigobeckmann.com/en/research/how-to-measure-what-an-ai-recommends