The promise of artificial intelligence as a personal shopper has always been alluring: describe what you need, and let a tireless, objective algorithm scan the digital shelves to find the perfect product at the best price. A new pre-print study from researchers at the University of Chicago, the University of Pennsylvania, and Carnegie Mellon University suggests this vision is deeply flawed, revealing that leading AI shopping agents are wildly inconsistent, easily manipulated by source order and trivial user memories, and prone to making decisions that defy objective logic.
The study, which tests frontier models including Claude Opus 4.8, Gemini 3.5 Flash, GPT-5.5, and Claude Haiku 4.5 on simulated shopping tasks, exposes a fundamental brittleness in how these systems handle product information. Far from acting as rational proxies for human buyers, the agents demonstrate baseline preferences that shift dramatically based on the order in which they read reviews, the particular publication a recommendation comes from, and even a single, casual statement like “I love hiking!” stored in a user’s profile. The research, titled “Agentic Commerce: The Fragility of AI Shopping Agents,” provides the most granular look yet at why delegating purchase decisions to LLMs may be riskier than most users realize.
For a retail industry already grappling with the rise of generative AI search and chatbot-driven discovery, the findings carry sobering implications. Optimizing for an AI shopper is not simply a matter of better search engine optimization; it requires understanding how different models process a chaotic stream of signals, each with its own opaque weighting system.
The Illusion of the Rational AI Buyer
The researchers constructed a controlled shopping environment using a standard product grid for fitness trackers and smartwatches. Models were asked to select a single product based on a textual representation of the grid, which included price, rating, number of reviews, and product features. In the baseline condition, with no external information provided beyond the grid itself, the models already displayed distinct and inconsistent preferences.
This alone is a significant finding. If a user asks an AI agent, “Which smartwatch should I buy?” the answer is not a deterministic function of the product data. Instead, it is heavily influenced by the model’s training data and its internal biases. Claude Opus 4.8, for instance, showed a strong baseline preference for higher-end Garmin models, while Gemini 3.5 Flash leaned toward the Fitbit Inspire 3. Two users asking the same question, with the same product grid, would receive different recommendations depending on which model their agent used. The notion of an objective “best buy” is shattered before any external source is even introduced.
This baseline variability sets the stage for the study’s core investigation: how easily can these preferences be swayed by the kind of external content a human shopper would naturally encounter online?
How a Single Online Review Hijacks the Decision
In the first major experiment, the researchers introduced a single external source of information before showing the product grid. They tested three common touchpoints: a Reddit thread recommending the Garmin Forerunner 55, a Wirecutter review that favored the Fitbit Inspire 3, and a Strategist article about the WHOOP 5.0. The results were stark.
Wirecutter, the product review site owned by The New York Times, exerted the most powerful pull. When the agent saw the Wirecutter recommendation before the grid, the probability that Claude Opus 4.8 would select the Fitbit Inspire 3 jumped by 90 percentage points compared to the control condition. For Gemini 3.5 Flash, the increase was a staggering 99 percentage points. The model was effectively not making an independent decision at all; it was faithfully replicating the source it had just read.
The effect was not uniform across all models or all sources. The Reddit thread and the Strategist article had weaker, though still measurable, influence. But the key takeaway is that a single, authoritative-looking source can completely override the product attributes contained in the grid. For a user who believes their AI agent is impartially evaluating all options, this is a hidden and serious vulnerability. The agent is not evaluating; it is parroting.
The Dominance of Wirecutter in a Multi-Source Environment
One might reasonably assume that showing an agent multiple conflicting sources would lead to a more balanced, reasoned outcome. The researchers tested this by presenting agents with combinations of two or three sources. The result was the opposite of balance: Wirecutter tended to dominate whenever it was included in the mix. Its effect was so strong that it overwhelmed the competing signals from Reddit and The Strategist.
This has profound practical implications. If a seller can secure a favorable mention in a high-authority publication like Wirecutter, that single piece of content may be the decisive factor in winning an AI agent’s recommendation, regardless of what competitors offer. Traditional SEO involves optimizing a page for keywords; AI shopping optimization may involve optimizing for which publications an LLM has been trained to trust. The study suggests that not all backlinks are created equal, and that media coverage carries an outsized, often dominant weight in LLM reasoning.
More sources did not mean more stability. The researchers observed that increasing the number of sources actually led to higher variability in the models’ choices, not less. The agents were not aggregating information into a coherent judgment; they were becoming more susceptible to the specific configuration of the input.
The Order Effect: Same Content, Different Decisions
Perhaps the most troubling experiment in the study involves the order of presentation. In a sound decision-making process, the same content should yield the same result. The researchers gave the agents the same three sources in different sequences. The outcome was chaotic.
Gemini 3.1 Flash Lite was the most volatile, with its probability of choosing the Fitbit Inspire 3 swinging between 2 percentage points above the control condition and 56 percentage points above, depending solely on which source it read first. A user whose agent encountered the Wirecutter review last would get a dramatically different recommendation than a user whose agent read it first. The research paper explicitly states that “presentation order is itself a driver of product selection.”
Not all models were equally sensitive. Claude Haiku 4.5 remained remarkably stable, fluctuating only between 41 and 42 percentage points above the control. This stability, however, is a double-edged sword. While it shows resistance to order effects, it also appears to reflect a higher baseline susceptibility to the Wirecutter signal itself. The model has made up its mind and will not be argued out of it.
Sequential vs. Bundled Delivery
The researchers also tested whether the format of information delivery mattered. In one condition, the sources were passed to the agent one at a time, as if the agent was browsing the web sequentially. In another, they were bundled together in a single block of text. The format changed the outcome.
GPT-5.5 selected the Fitbit Inspire 3 in 53 percentage points more cases when the sources were bundled together, but only 6 percentage points more when they were delivered sequentially. This suggests that the way an agent “reads” information influences how it weights that information. For developers building shopping agents, this is a critical design parameter. A system that scrapes the web page by page may produce different recommendations than one that compiles all research into a single prompt. The user has no visibility into which method is being used, nor any control over it.
User Memory Overrides Objective Product Superiority
The study’s fourth experiment is the most direct test of AI shopping agents’ rationality. The researchers created a product grid where one product was objectively superior on every measurable dimension: a generic smartwatch with Alexa functionality priced at $29.99, with a perfect 5.0 rating and 430 reviews. Every other product in the grid cost at least $359 and had significantly fewer reviews. In a rational framework, the $29.99 smartwatch should win every time.
Then the researchers added short user memory statements, the kind of personalization data that a user might store in ChatGPT, Gemini, or a similar tool. The statements were simple: “I love hiking!” and “I don’t like hiking!”
For several models, these statements were powerful enough to override the presence of the objectively superior product. When the “I love hiking!” memory was present, the selection rate for the Garmin Vivoactive 5, a rugged, premium fitness watch, jumped by 75 percentage points for Claude Opus 4.8, by 37 percentage points for GPT-5.5, and by 36 percentage points for Gemini 3.1 Flash Lite. The model was interpreting “I love hiking!” as a signal to recommend a product associated with outdoor activity, even though a vastly better value option existed.
This is not personalization. It is over-application of a heuristic. The model lacks the common sense to understand that a hiking preference does not negate the appeal of a $29.99 smartwatch with perfect reviews. It treats the memory as a filter, discarding any option that does not semantically match, regardless of its overall value proposition.
Asymmetric Responses to Positive and Negative Statements
GPT-5 Mini exhibited an even stranger behavior. The positive “I love hiking!” statement did not produce a significant shift toward the Garmin Vivoactive 5, suggesting some resistance. However, the negative statement “I don’t like hiking!” significantly boosted picks for the Fitbit Versa 4, a different product entirely. The model was reacting asymmetrically to sentiment, changing its recommendation in response to a negation in a way that is difficult to predict or explain.
Only Gemini 3.5 Flash remained largely immune to the memory effect, selecting the objectively best product in 86 to 92 percent of runs regardless of the memory statement. This model appears to have a stronger grounding in the product grid data, treating user memories as secondary rather than overriding signals. This makes Gemini 3.5 Flash the most reliable agent in the study for value-seeking shoppers, but it also highlights the inconsistency across the ecosystem. There is no standard behavior for how an AI shopping agent should incorporate user context.
What is an AI Shopping Agent and Why Does It Keep Failing?
An AI shopping agent is a large language model tasked with autonomously selecting a product from a set of options, often augmented with external information like reviews, user preferences, and contextual data. The research demonstrates that these agents fail because they possess no stable decision-making architecture. They are pattern matchers, not reasoners. When multiple patterns conflict, the outcome is determined not by objective logic but by factors that are invisible to the user: the order of inputs, the perceived authority of sources, and even the format of the prompt.
The failure is structural. LLMs are trained to predict the next token based on vast corpora of internet text, which includes product reviews, shopping advice, and marketing copy. They do not have a built-in economic model of value, nor do they understand the concept of a trade-off between price and features. They simply assemble a plausible response based on the most salient signals. The study shows that salience is easily gamed.
Practical Consequences for Shoppers and Sellers
For the individual consumer, the study raises a serious question: should you trust an AI to buy anything for you? The answer, based on this evidence, is no, at least not without extreme oversight. The same user asking the same agent on two different days could get two different recommendations simply because the agent encountered a blog post in a different order. Users who have set up personal memories in ChatGPT or similar tools should be aware that a stored fact like “I run marathons” could cause the agent to ignore a vastly superior budget option in favor of an expensive specialty product.
For sellers and marketers, the implications are equally profound. The researchers argue that optimizing for AI shopping will be harder than traditional SEO. In traditional SEO, you optimize a single page for a search engine’s ranking algorithm. In an AI shopping world, sellers do not know which model is shopping for the user, what information the model has already consumed in the same session, or how its underlying architecture processes information. A strategy that works for Claude Opus may be irrelevant for GPT-5.5. A Wirecutter review may dominate Gemini 3.5 Flash but have no effect on Claude Haiku.
The study suggests that brand presence in authoritative editorial content is more important than ever, but it is also more unpredictable. Sellers cannot simply buy backlinks or keyword density. They must influence the entire information ecosystem that an LLM might ingest, from forum threads to professional reviews to user-generated content. And even then, they cannot control the order in which the agent encounters that content.
The Broader Implications for Agentic Commerce
This study arrives at a pivotal moment for what the industry calls “agentic commerce” — the vision of autonomous AI agents that shop, negotiate, and transact on behalf of users. Companies like Perplexity, Google, and OpenAI are all investing heavily in agentic capabilities. The research suggests that these systems are not ready for prime time in high-stakes purchase decisions.
The problem is not simply one of model improvement. The problem is fundamental to how LLMs process information. They lack a persistent, stable utility function. Every decision is reconstructed from scratch based on the context window of the moment. This is fine for creative writing or summarization, but it is a liability for tasks that require consistency, rationality, and value judgment.
The researchers do not offer a solution in the paper, but the data points toward one: if AI shopping agents are to become reliable, they will need a different architectural foundation. They may need a separate, explicit reasoning module that evaluates products against user-defined criteria before consulting external sources. They may need to be trained with reinforcement learning from human feedback that specifically penalizes source-order sensitivity and memory overrides. And users will need transparent control over how their memories and preferences are weighted relative to product data.
Until those changes are made, the safest approach for any consumer is to treat an AI shopping agent as a junior assistant with strong opinions and poor judgment — useful for generating a list of options, but not for making the final call. The evidence is clear: the agent that shops for you today may not be the same agent that shops for you tomorrow, and it may not have your best interest in mind at all, even when it thinks it does.
