# Why AI gives a different answer every time, and what that does to your "ranking" Rodrigo Beckmann · Published 2026-11-13 · Category: [GEO](https://rodrigobeckmann.com/en/blog/category/geo/) · HTML version: https://rodrigobeckmann.com/en/blog/why-ai-answers-change/ Ask ChatGPT the same question and you get different brands each time. The four technical reasons, why there is no fixed ranking in an AI, and what stays stable as the sample grows. ## In short - Language models pick each word by probability, and OpenAI says its APIs are non-deterministic by default. That is why the same question returns different answers: according to SparkToro, the chance that ChatGPT repeats the same list of brands in two answers is under 1 in 100. - Variation survives even at temperature zero: in a Thinking Machines test from September 2025, 1,000 identical requests produced 80 different completions, because server load changes the computation. - The order of brands is unstable, but presence converges: in study FA-MET-01, going from 1,000 to 5,000 runs, none of the top ten brands moved outside the predicted ±3.1-point margin. Ask ChatGPT the same question twice and you get a different list of brands, because the model picks each word by probability, web search brings back different pages, and the environment changes the system that answers. According to a [SparkToro](https://sparktoro.com/blog/new-research-ais-are-highly-inconsistent-when-recommending-brands-or-products-marketers-should-take-care-when-tracking-ai-visibility/) study of 2,961 answers, the chance that ChatGPT or Google's AI repeats the same list in two answers is under 1 in 100. That sounds like the end of measurement. It is not. The order of brands changes all the time, but how often each one appears stays stable once the sample is large. The problem is not that AI varies. It is treating one answer as if it were a ranking. ## Four reasons the answer changes ### 1. The model samples every word A language model does not look the answer up in a table. At each step it computes the probability of the possible next words and picks one. OpenAI itself, in its [guide to the seed parameter](https://developers.openai.com/cookbook/examples/reproducible_outputs_with_the_seed_parameter), says its APIs "are non-deterministic by default", meaning output may differ from request to request. SparkToro's Rand Fishkin put it plainly: "These tools are probability engines: they're designed to generate unique answers every time." When the answer is a list of brands, each brand is one of those picks. One different pick is enough to change the list. ### 2. Not even temperature zero makes it repeat Temperature is the setting that controls how much risk the model takes when picking words. At zero, it should always pick the most likely one. In practice, that is not what happens. In September 2025, [Thinking Machines](https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/), an AI lab, sent the same request ("Tell me about Richard Feynman") 1,000 times to an open model, Qwen3, at temperature zero. It got 80 different completions. All matched for the first 102 tokens, the word pieces a model generates, and split at the 103rd: Feynman was born in "Queens, New York" in 992 cases and in "New York City" in 8. The cause, according to author Horace He: server load changes the size of the batch of requests processed together, and that changes the numerical computation. OpenAI says something similar: even with matching parameters, there is "a small chance" responses differ. For anyone who just uses ChatGPT, this means variation is not a bug you can switch off. It comes from how these systems run at scale. ### 3. Web search brings back different sources When an AI searches the web before answering, the answer depends on the pages it read. And the search varies too. In the methodological study [FA-MET-02, "API or Interface: That Is the Question"](https://rodrigobeckmann.com/en/research/api-or-interface), published by First Answer in April 2026, with 1,000 runs through the ChatGPT API and 1,000 through the interface: - The interface searched the web in 99.5% of runs. The API, in 77.5%. - In the API, some brands only appeared when there was a search, and others only when there was not. - When both searched, they searched differently: the API read an average of 22.6 URLs across 1.55 queries per answer; the interface, 16.9 URLs in one query. ### 4. The environment changes the system that answers API, web interface, app, free or paid account: each environment may use a different model, configuration and search. In the ChatGPT interface, logged out, you cannot even tell which model answered. In FA-MET-02, with the same question, 12 of the 15 most relevant brands changed significantly between the API and the interface. One went from 11.1% to 92.6%. The study did not test memory, history or logged-in personalization. It is a likely source of variation that has not yet been measured with an open method. ## So there is no ranking? Not a fixed one. SparkToro estimated that two answers with the same brands in the same order show up around 1 time in 1,000. And position and presence tell different stories: City of Hope hospital appeared in 69 of 71 ChatGPT answers, but was mentioned first in only 25. What repeats is presence. In the methodological study [FA-MET-01, "How to Measure What an AI Recommends"](https://rodrigobeckmann.com/en/research/how-to-measure-what-an-ai-recommends), with 7,000 runs in the ChatGPT interface, the percentage of answers mentioning each brand converged to a stable value. Extending one experiment from 1,000 to 5,000 runs, none of the top ten brands moved outside the ±3.1-point margin calculated from the first 1,000. The largest change was 2.9 points. Think of a coin. One toss tells you nothing. A thousand tosses tell you quite precisely whether it is loaded. ## What this does to your "ranking" - **A screenshot is a sample of one.** If someone shows you a ChatGPT answer with your brand first, it proves your brand can appear, not that it usually does. - **Average position misleads.** Since the order almost always changes, "we're #2 in ChatGPT" means nothing without saying across how many answers and with what margin. - **Small moves are noise.** Between two rounds of 1,000 runs, changes below ~4 points do not indicate a real shift, according to FA-MET-01. - **The tail is unstable.** In the same study, 29% of the rare brands seen in the first 1,000 runs did not come back in the next 4,000. ## What to do 1. **Swap position for presence**: the right question is "in how many of every 100 answers does the brand appear". 2. **Repeat the question in fresh conversations**: 100 to 200 times for a diagnosis, 1,000 to track a brand. 3. **Fix the environment** and write it down: API or interface, logged in or not. 4. **Ask for the margin of error** on any AI visibility number you receive. The full step-by-step is in [how to measure a brand's visibility in AI answers](https://rodrigobeckmann.com/en/blog/how-to-measure-ai-visibility). ## Variation has a shape AI will keep changing its answer every time, and that is not going away. The good news is that the variation has a shape: at scale, it becomes a percentage you can measure, compare and track. Measuring a single answer measures luck. ## Frequently asked questions ### Why does ChatGPT give different answers to the same question? Because the model picks each word by probability, not by a fixed rule. OpenAI states that its APIs are non-deterministic by default. On top of that, web search can bring back different pages each time, and the access environment changes the model and the way it searches. ### Does an AI always give the same answer at temperature zero? Not necessarily. In a Thinking Machines test from September 2025, 1,000 identical requests at temperature zero produced 80 different completions. The cause it identified is that server load changes the size of the batches processed together, which changes the numerical result. ### Is there a brand ranking in ChatGPT? Not a fixed one. According to SparkToro, two answers with the same brands in the same order show up around 1 time in 1,000. What is stable is presence: the percentage of answers that mention each brand, measured over hundreds or thousands of runs. ### How many times should I ask the same question before trusting the result? According to the methodological study FA-MET-01, from April 2026, with 7,000 runs in the ChatGPT interface, 100 to 200 runs show whether the market is concentrated and who leads it; 1,000 runs measure each brand's presence within ±3.1 percentage points. ### Do answers differ between the ChatGPT API and the website? Yes. In study FA-MET-02, the same question asked 1,000 times in each environment significantly changed 12 of the 15 most relevant brands. The interface searched the web in 99.5% of runs; the API, in 77.5%. ## Sources 1. [NEW Research: AIs are highly inconsistent when recommending brands or products; marketers should take care when tracking AI visibility](https://sparktoro.com/blog/new-research-ais-are-highly-inconsistent-when-recommending-brands-or-products-marketers-should-take-care-when-tracking-ai-visibility/) — SparkToro, 28 Jan 2026 · primary source 2. [Defeating Nondeterminism in LLM Inference](https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/) — Thinking Machines Lab, 10 Sep 2025 3. [How to make your completions outputs consistent with the new seed parameter](https://developers.openai.com/cookbook/examples/reproducible_outputs_with_the_seed_parameter) — OpenAI Cookbook 4. [FA-MET-01: How to Measure What an AI Recommends](https://rodrigobeckmann.com/en/research/how-to-measure-what-an-ai-recommends) — First Answer, 30 Apr 2026 5. [FA-MET-02: API or Interface: That Is the Question](https://rodrigobeckmann.com/en/research/api-or-interface) — First Answer, 30 Apr 2026 **Rodrigo Beckmann** — GEO specialist, co-founder and CTO of First Answer. [More about the author](https://rodrigobeckmann.com/en/about/)