Skip to content

Large Language Model Test: ChatGPT's Top Pick Changed on 2 of 5 Questions

We split 15 brand names into tokens and asked ChatGPT the same five buyer questions, the kind a person asks when choosing a product, three times each, and some answers held still while some didn't.

Search Shift
Nov 14, 20261 min read

n October 19, 2026, we asked ChatGPT for the "best mattress for side sleepers" three times, same question each time, and it put Casper first once and Helix twice.

The short version is that a large language model writes its answer one small piece at a time, and the next piece isn't always the same.

Across the 5 buyer questions in that test, ChatGPT's top pick changed for 2, though 22 of the 28 products it named showed up in all three runs.

Four parts of the process matter to anyone publishing content, namely tokens, training, retrieval and sampling.

The first test looked at tokens, meaning how a model splits brand names, while the second looked at sampling, meaning how steady ChatGPT's answers are, and it also logged the sources ChatGPT cited.

We ran both at RankControl, an AI SEO and AEO platform. The long version is our guide, How large language models work.

Read the full write-up on Medium

Did you enjoy this article?

Recommend it — Standard Reader surfaces well-loved writing to more readers across the network.

Across the AtmosphereDiscussions