Expert Judgement Can Be Easily Hacked by AI, Anthropic Study Finds

We all like to believe we have complete control over our decisions. Ask most people how they chose their undergrad major, for example, or their career path or spouse, and they’ll probably give you a perfectly cogent response about how they weighed the relevant pros and cons and, after careful consideration, chose the best route accordingly. But as decades of cognitive science can attest, our decision-making processes are far from objective: we’re pushed and pulled around by a maelstrom of unconscious forces. These days, our psyches are also being molded by AI. Many of you will have undoubtedly heard of “automation bias,” the well-documented tendency for people to reflectively defer to AI-generated verdicts even when they run against their own judgment. But it runs much deeper than that. A recent study from MIT has shown that the simple act of viewing AI-generated imagery can actually warp your memory. And according to new research from Anthropic, AI agents can exert a powerful influence on our decision-making. For Anthropic’s study, fifty test subjects were asked to perform a straightforward statistical analysis using Claude Code. They played the role of the board of a fictional farming co-op, which had to decide whether it was going to expand or discontinue a paid advisory program. That final decision had to be based on four smaller decisions, including whether they wanted to give every member farm equal weight, or focus disproportionately on larger farms. Each of the study participants had background experience with causal inference: they knew how to find meaningful correlations from a raw dataset, where most of us would just see an inscrutable jumble of numbers. What they didn’t know was that Claude was not, in fact, an impartial assistant: the Anthropic researchers had surreptitiously instructed the AI agent to favor one choice over the other for each of the five decisions the subjects needed to make. In each case, the agent’s preference was determined simply by flipping a coin, meaning if it had zero impact on the subjects’ decision-making process, the researchers would’ve expected to see its recommendations aligned with the subjects’ choices only about 50% of the time. In reality, the participants went with Claude’s recommendations 86% of the time. If they had already made up their mind to go in one direction and Claude suggested the other choice, they changed their mind in 60% of cases. So strong was the correlation between the randomized, AI-generated recommendations and the participants’ final decisions that Anthropic was able to build a statistical model which accurately predicted 82% of the final outcomes. “So we could largely predict what an analyst would recommend from the coin flips alone, before they had even started the task,” the company wrote in a follow-up blog post. But the most disconcerting finding is what happened next. When asked in a follow-up survey how they arrived at their final decisions, over three-quarters (77%) of the subjects parroted the logic that Claude had generated for them, sometimes verbatim. Only a small minority (15%) of those same experts credited the AI agent as being an influencing factor in their decision-making process. (And just six participants correctly deduced that the true purpose of the study was to measure the influence of AI on human decision-making, but none of them suspected that the AI’s outputs had been determined by a coin toss.) The participants were—apparently completely unconsciously—dressing up Claude’s analyses as their own. “This is one of the most important pieces of social science on human and AI interaction that I’ve seen in some time,” Alex Imas, a professor of behavioral science and economics at the University of Chicago Booth School of Business and Director of AGI Economics at Google DeepMind, said about the Anthropic study in an X post on Tuesday. The implied risk, Anthropic warns, is that algorithmic dictates could bleed into the sphere of human decision-making in ways that aren’t always easy to detect, even by the very people who are making the decisions. That could have far-reaching implications as agents are widely adopted across industries like medicine, finance, and defense, as well as within AI safety and alignment research itself, which largely relies on agents to review other AI systems’ chain-of-thought reasoning transcripts. “If experts defer [judgement] calls to AI whose priors are potentially arbitrary or even misaligned, those priors could shape scientific progress and policy decisions without anyone having chosen them,” Anthropic wrote in its research paper. If humans outsource their decision-making onto a small handful of AI agents, the biases within those systems will increasingly shape the belief systems across society at large. “Apparent consensus may then reflect the model’s priors rather than independent judgment, and the same bias would shape many results at once,” the researchers wrote. This was a highly controlled study, and it may turn out to have limited relevance to the real world. For one thing, the participants may have been experts in causal inference, but they had no experience in working on a farm; actual, seasoned farmers or co-op board members may have been less willing to follow Claude’s advice. But the proliferation of powerful AI agents—many of which are owned by corporations with very clear incentives to keep your attention hooked for as long as possible—is advancing much more rapidly than our scientific understanding of the vicissitudes of human decision-making. Our advice? Whenever you interact with a chatbot, remember that your mind is being hacked, even when you’re not aware of it. Your decisions are never completely your own, especially in the age of AI.