# How to Measure What an AI Recommends Rodrigo Beckmann · First Answer · Version 1.0 · Published 2026-04-30 · HTML version: https://rodrigobeckmann.com/en/research/how-to-measure-what-an-ai-recommends/ *Methodological study · FA-MET-01.* How many queries it takes to tell an isolated observation from a real pattern. 3 experiments · 7,000 runs · ChatGPT interface · [Full study (PDF)](https://rodrigobeckmann.com/pesquisa/FA-MET-01_how-to-measure_v1.0_2026-04-30_EN.pdf) ## In short One question to an AI is a sample, not a portrait. At scale, each brand’s presence converges to a stable value and follows classical statistics: with 1,000 runs, it is measured with a margin of at most ±3.1 percentage points. ## Key numbers - **±3.1 pp**: margin for each presence with 1,000 runs - **0 of 10**: top brands moved outside that margin going from 1,000 to 5,000 runs - **38 × 144**: distinct brands in 1,000 runs: concentrated and fragmented market - **29%**: of the rare brands in the first 1,000 runs did not return in the next 4,000 ## What we found Three experiments in the ChatGPT interface: 1,000 runs in a concentrated market, 1,000 in a fragmented one, and an extension of the latter to 5,000. 1. **There is a stable pattern behind the variation.** At scale, each brand’s presence converges to a measurable value. In the concentrated market, two brands appear in 99.9% of answers and five form a core above 50%. 2. **The shape of the pattern changes from one market to another.** In the fragmented market, the leader appears in 70.3% of answers and the universe reaches 144 brands in 1,000 runs, against 38 in the concentrated one. Even so, both have the same number of brands above 25% and above 10%. 3. **Precision follows classical statistics.** With 1,000 runs, presence is measured within ±3.1 points. Multiplying the collection by five, the largest change among the top ten brands was 2.9 points, within the predicted margin. 4. **The tail needs a different scale.** In the fragmented market, the brand universe is still growing at 5,000 runs, ever more slowly. And part of the tail seen in small samples is not confirmed with more volume. ## The hierarchy measured at 1,000 runs held at 5,000 *Experiment 3 · fragmented market.* Band: 95% interval calculated with 1,000 runs. Dot: presence measured with 5,000. | Brand | Band (95% interval) | Dot | Values | | --- | --- | --- | --- | | Brand A | 67.4%–73.1% | 69.4% | 70.3% → 69.4% | | Brand B | 65.9%–71.6% | 68.2% | 68.8% → 68.2% | | Brand C | 52.1%–58.3% | 55.9% | 55.2% → 55.9% | | Brand D | 51.2%–57.4% | 57.2% | 54.3% → 57.2% | | Brand E | 29.9%–35.7% | 31.5% | 32.7% → 31.5% | | Brand F | 28.9%–34.6% | 32.4% | 31.7% → 32.4% | | Brand G | 23.2%–28.6% | 25.3% | 25.8% → 25.3% | | Brand H | 11.5%–15.8% | 13.8% | 13.5% → 13.8% | | Brand I | 11.5%–15.8% | 13.3% | 13.5% → 13.3% | | Brand J | 4.7%–7.6% | 6.1% | 6.0% → 6.1% | Brands anonymized. Base: 1,000 and 5,000 answers, Wilson interval. Source: FA-MET-01, Table 4. ## How many runs are needed Precision depends only on the number of runs and the brand’s own presence. The right collection size depends on what you want to know. | Goal | Calls | What it delivers | | --- | --- | --- | | Initial diagnosis | 100 to 200 | Whether the market is concentrated or fragmented, and the approximate top (±7 to 10 pp) | | **Visible hierarchy** | **1,000** | **Presence within ±3 pp; reliable order between brands more than ~6 pp apart** | | Change between rounds | 1,000 per round | Changes above ~4.4 pp (presence near 50%) can be told apart from noise | | ±1 pp precision | ~9,600 | Presence within ±1 pp at any level | | Tail down to 0.1% | ~3,000 | 95% chance of seeing each brand with 0.1% presence at least once | | Tail down to 0.02% | ~15,000 | Same for brands at 0.02% | Margins at 95% confidence. ## Why it matters ### For brands and marketing First, find out whether the category has a core and who is in it: 100 to 200 runs answer that. To track the brand’s position, use rounds of 1,000 and treat changes below ~4 points as noise. ### For researchers and journalists One question to an AI is a coin toss. Claiming what an AI recommends takes hundreds of independent samples, ideally 1,000 or more, and care with conclusions drawn from the rare tail. ### For people building with AI If the product depends on a consistent recommendation, consistency has to be measured, not assumed. Some choices exceed 90%; others vary in 30% or more of answers. ## Method, in short Collected through the ChatGPT interface, in the browser, by automation, with web search enabled. One fixed question per experiment, written the way a consumer would type it, each run in a new conversation. Every brand mentioned was extracted, normalized to the parent brand and anonymized with letters. 95% intervals by the Wilson method. ## Limitations One environment and two categories, and the interface does not allow the answering model to be fixed or recorded. The statistical regularity should hold for other models and access environments; the brand composition not necessarily (see [FA-MET-02](https://rodrigobeckmann.com/en/research/api-or-interface/)). ## Read the full study The three experiments in detail, with 9 figures, 5 tables, a study sheet and references. [Full study (PDF)](https://rodrigobeckmann.com/pesquisa/FA-MET-01_how-to-measure_v1.0_2026-04-30_EN.pdf) ## How to cite Beckmann, R. (2026). *How to Measure What an AI Recommends: How many queries it takes to tell an isolated observation from a real pattern* (FA-MET-01, version 1.0). First Answer. https://rodrigobeckmann.com/en/research/how-to-measure-what-an-ai-recommends ## Related study [FA-MET-02 — API or Interface: That Is the Question](https://rodrigobeckmann.com/en/research/api-or-interface/): Why the same question returns different brands depending on how it is asked.