Ask ChatGPT the same question 8 times. How often does the answer change?
We asked 20 buyer questions 8 times each on ChatGPT and Gemini. The brand on top almost always comes back. The further down the list a brand sits, the more often it drops out of the next answer.
What we found
of questions kept the same #1 brand in all 8 ChatGPT answers
chance a brand named 4th to 6th is named again in the next answer, against 96% for the 1st
of answers named a brand no other run of that question named
On this page7 sections
Why one answer is not enough
Most AI visibility reports ask each question once and record who was named. That works only if the engine gives the same answer every time. It doesn’t: language models choose their words with some randomness, and assistants that search the web run several related searches for each question and can read a different set of pages each time[1].
So we did the boring thing. We asked the same questions again and again, and counted.
What we asked, and how
20 questions a buyer would ask before choosing software, each sent 8 times to ChatGPT and to Gemini, from the United States, in English: 320 answers. 13 of Gemini’s were not answers at all but error messages (“Sorry, something went wrong”), so the figures rest on 307. A brand counts as named when the answer presents it as an option, not when it only appears in a cited link.
How we measuredShowHide
Questions: 20, two in each of 10 software categories, written the way buyers ask (“What is the best CRM for a small marketing agency?”).
Runs: 8 per question and engine, over about 11 hours on November 5 and 6, 2026, through the same data provider[2] and the same brand extraction our product uses: a language model lists the brands each answer presents, in order.
One brand, two names: names that start the same way count as one brand (“HubSpot” and “HubSpot CRM”), the rule our reports use.
Ranges: 95% Wilson intervals [3], as in every report we send (how we measure).
Limits: one country, one language, one night rather than weeks. Answers to the same question are not independent, so the ranges are, if anything, too narrow. Twenty questions is a small sample for “the same #1 in every run”, hence its wide range.
The #1 brand: almost always there, not always first
The brand that came first in the first answer was still named in 96% of ChatGPT’s later answers (95% on Gemini). It was still first in only 72% of them, and just 9 of the 20 questions kept the same #1 in all 8 ChatGPT answers.
Was the first answer’s #1 brand named again? One dot per answer
- What is the best CRM for a small marketing agency?8/87/7
- Which CRM should a 5-person sales team use?8/88/8
- Best project management tool for client work8/87/7
- What project management software do agencies use?8/87/7
- Cheapest email marketing tool for a startup8/88/8
- Best email marketing platform for an ecommerce store8/87/7
- Best accounting software for freelancers8/87/7
- What accounting software should a small US business use?8/88/8
- Best help desk software for a 10-person support team8/85/5
- What is a good alternative to Zendesk for a small team?8/88/8
- Best website builder for photographers6/87/7
- Which website builder is best for a small restaurant?8/84/8
- Best password manager for teams8/88/8
- What password manager should a small company use?8/88/8
- Best payroll software for a US small business8/87/7
- Easiest payroll service for a company with 10 employees8/88/8
- Best social media scheduler for agencies8/86/8
- What tool should I use to schedule Instagram and LinkedIn posts?8/88/8
- Time tracking app that integrates with Slack5/87/7
- Best time tracking software for a design agency8/86/6
The first answer’s #1 brand in the answers that followed
Still named, anywhere in the answer
- ChatGPT96% (92–99%)
- Gemini95% (90–98%)
Still named first
- ChatGPT72% (64–79%)
- Gemini72% (63–79%)
The lower a brand sits, the more it flickers
For every brand an answer named, we checked whether the next answer to the same question named it again. A brand named first came back 96% of the time, and one named 2nd or 3rd, 90%. From 4th to 6th, one time in three it was gone; from 7th down, more often than not.
Named again in the next answer, by the brand’s place in the answer
- Named 1st96% (93–98%)
- Named 2nd–3rd90% (87–92%)
- Named 4th–6th66% (62–70%)
- Named 7th or later43% (35–52%)
That is where most brands live. An answer lists five or six options (5.4 on ChatGPT, 5.8 on Gemini), so in any category most brands sit in the places that flicker. And 23% (19–28%) of answers named a brand that no other run of the same question named at all.
ChatGPT and Gemini: alike, except for errors
On every measure the two engines landed inside each other’s ranges: the same #1 in every run for 45% of questions on ChatGPT and 50% on Gemini, the first #1 still named 96% and 95% of the time. What differed was failure: Gemini sent back an error message instead of an answer 13 times out of 160; ChatGPT, never. A report that counts those as “not named” makes a brand look less visible on Gemini than it is.
What it means for client reports
- Ask each question more than once. One answer says a lot about the leader and little about a brand in 4th place.
- Report how often a brand is named, not its rank. Being named holds up; the order doesn’t.
- Count engine errors as no answer. They are not a “no”.
- Report the range. A week-on-week change smaller than the range is noise.
Sources
- 1.AI features and your website · Google Search Central · checked Nov 6, 2026
- 2.LLM Scraper pricing and queues · DataForSEO · checked Nov 6, 2026
- 3.Binomial proportion confidence interval (Wilson score) · Wikipedia · checked Nov 6, 2026
Cite this study
Menzio Research. (2026, November 6). Ask ChatGPT the same question 8 times. How often does the answer change? Menzio. https://getmenzio.com/research/how-much-ai-answers-change
Is your brand in these answers?
Run the same kind of check on your brand: three buyer questions on ChatGPT, Gemini and Google AI Overviews, with who they name instead.
Free · no signup · ChatGPT, Gemini and Google AI Overviews
Changes to this page
- November 6, 2026: First published.