Methodological study · FA-MET-02
API or Interface: That Is the Question
Why the same question returns different brands depending on how it is asked.
In short
Measuring through the API is not measuring what consumers see. With the same question and the same search tool, the API and the ChatGPT interface only agree on the core: of the 15 most relevant brands, 12 change significantly between environments.
Same question, 1,000 runs in each environment
From fourth place down, brands swap positions
What we found
1,000 runs in each environment, in parallel, on the same day, with the same question and web search enabled.
- 01
Only the core survives the change of environment
Two brands stay close to 100% in both. Below them there are inversions: one brand goes from 11.1% in the API to 92.6% in the interface; another, from 66.5% to 6.4%.
- 02
The brand universes barely overlap
The API cited 39 brands and the interface 37, but only 20 appear in both. Neither environment contains the other.
- 03
Web search explains part of the difference
The interface searched in 99.5% of runs; the API, in 77.5%. Within the API itself, some brands only appear with search and others only without it.
- 04
Forcing search in the API would not fix it
Comparing only runs with search in both environments, the divergence remains: one brand sits at 93.1% in the interface and 0.3% in the API.
- 05
When both search, they search differently
The API reads 22.6 URLs on average and runs 1.55 queries per answer; the interface, 16.9 URLs in one query.
56 brands, little overlap
Distinct brands cited in 1,000 runs per environment.
Why it matters
The API is an integration gateway for developers, not where consumers meet the AI.
Anyone who measures brand presence via the API and presents the result as what users see is measuring a different system. Every monitoring program must state the environment it measures, and swapping one for the other is not fixed by a configuration tweak.
Method, in short
Collection designed to use the same model in both environments; logged out, the interface does not allow the model to be fixed or recorded. Search enabled, default temperature and no custom system instruction. 2,000 runs in parallel, within a window of a few hours, each in a new conversation. Differences assessed with 95% intervals (Wilson and Newcombe).
Limitations
One category, one model and one date. The mechanism should repeat in other contexts; which brands rise or fall should not. Logged out, we cannot rule out that part of the divergence comes from the interface’s model routing. The study does not test why search differs between environments, only that it does.
Read the full study
The five findings in detail, with figures, tables, a study sheet and references.
How to cite
Beckmann, R. (2026). API or Interface: That Is the Question: Why the same question returns different brands depending on how it is asked (FA-MET-02, version 1.0). First Answer. https://rodrigobeckmann.com/en/research/api-or-interface