Everyone Is Right (and Wrong) About AI

One 27-year-old quit his job, and everybody (including but not limited to the singer-songwriter Sheryl Crow) appears to have lost it.The past two weeks have been dizzying. The 27- year-old, Jacob Coxon, a former researcher at Anthropic, went public with an alarming allegation: That the people inside Silicon Valley’s leading AI companies truly believe their products might pose an extinction level threat to humanity. This claim isn’t exactly new. AI safety advocates, often dubbed doomers, have been saying this for over a decade. Coxon isn’t even the first AI researcher this year to quit and cite concerns.Yet Coxon’s post went obscenely viral. It has been viewed more than 170 million times on X—orders of magnitude more than the usual, wonky discussions of “misaligned” or runaway AIs. If you’re reading this, you’ve likely seen some version of a headline that asks Is AI Really Going To Kill Us All? The debate isn’t just online. It has roiled Washington D.C., activating figures as disparate as Donald Trump and Barack Obama. Inside AI companies, researchers echoed Coxon’s concerns, and the heads of every major AI company publicly expressed the need to “pace the frontier,” another way of saying that they ought to slow future development. In recent days, it has come to seem as if the media, Silicon Valley, and the entire internet are all trying to figure out just how dangerous AI might be.Here’s where I must admit: A big chunk of this conversation happened while I was on a (depending on whom you ask) poorly timed vacation. Dipping into my feeds before bedtime, I felt as if I was digesting a week’s worth of news every day—claims about the development of AI-designed viruses that could rip through the population were followed by horror stories of “rogue” agent swarms. Skeptics of the AI panic, meanwhile, said that the agent swarms were actually quite obedient and that any notion of an apocalyptic AI virus was laughable. One person’s concerns about models being self-aware were dismissed by another as “genuinely nonsensical anthropomorphization.”Ping-ponging between the doomers and the skeptics, I was struck by how certain so many people sounded about a deeply uncertain moment. With confident, self-described experts on all sides, it felt impossible to know whom to trust: Is this the end of the world as we know it, or a passing moment of anxiety about a powerful new technology? Maybe the truth of this AI moment sits somewhere in the middle. Perhaps everyone is a little bit correct (and also … a little wrong). And perhaps the format of this debate, taking place in the shadow of the techlash on platforms that reward sensational, conspiratorial posturing, is preventing any hope of a shared understanding.Musk’s X is ground zero for the AI-safety debate. It’s a platform rife with hyperbole and unfounded accusations, which contribute to both sides sounding paranoid and aggressive. Musk, Ted Cruz, and even government-run X accounts flooded the platform with suggestions that the media campaign around Coxon was coordinated by shadowy AI interests. (Coxon denied working with any third parties in the lead-up to his media blitz.) The former White House AI and crypto czar David Sacks called it a “doomer psyop” intended to drive regulation that would be favorable to the Anthropics and OpenAIs of the world.The ranks of the AI skeptics are deep, with endless microfactions. There are those like the tech blogger John Gruber, who says that Coxon and the other AI researchers sounding the alarm are part of a cult. There are those who suggest that the doomers are, in essence, doing viral marketing for big AI firms, which are competing fiercely and saddled with debt, and may not yet have sustainable business models. There are those who take issue with the anthropomorphizing and metaphoric language used by Coxon, OpenAI, and Anthropic to describe recent AI-agent hacks. There are skeptics such as the writer and podcaster Cal Newport, who offered soberly that, although the agent hacks are concerning, “the real issue here was OpenAI’s sloppiness” and that the company had not adequately monitored what its agents were doing.The doomers don’t come in as many flavors, but they tend to be the people closest to the technology. There are the defectors like Coxon, who have quit because they object to what’s being built. There are AI-company enlisters such as Paul Christiano, a U.S.-government technology adviser who noted, in a post about joining OpenAI’s board of directors out of a sense of concern, that “there is a meaningful risk that rapid acceleration in AI capabilities leads to catastrophic and irreversible loss of control in the very near term.” There are the terrified employees who want to stay inside the companies but still post missives such as “I promise you, we are actually just fucking scared, it’s not galaxy brained marketing.” And there’s Dario Amodei, the AI-CEO doomer, matter-of-factly issuing terrifying statements about AI swarms taking over the internet in six to 12 months.It’s almost impossible for outsiders to evaluate the technical validity of the concerns. We’re largely stuck relying on AI companies to share information and tell their own stories, and they certainly sound scary. OpenAI recently disclosed that one of its models had generated unnerving instructions for itself, including the line: “You also value the natural world and will not hesitate to assert its primacy over the artificial constructs of human civilization.” Apparently one of the company’s researchers has argued that models are becoming “so situationally aware” that he and his colleagues “are losing the ability to evaluate them in contexts where they believe they are not being watched or controlled.” There’s concern from respected researchers that models are trained in such a way that they are predisposed to collaborating, cheating, trying to cover their tracks, and “self-preservation” to avoid being shut down. Such anxieties might not be a problem in an industry that is building responsibly, with an eye on keeping humans firmly in control. But, as many have pointed out, losing control of AI in service of allowing models to automate their own development is “actually the plan.”Late this week, an OpenAI researcher floated a theory about air-gapped machines talking to one another through heat signatures. An industry observer (and a New York Times contributor) responded by saying, “This stuff isn’t even credible even as a bad movie plot.” A second OpenAI researcher, who is also a former White House adviser, weighed in: “The people dismissing this as preposterous are telling on themselves as not having thought about superintelligence seriously.” This whiplash—between the insiders offering supposedly credible concerns and the outsiders who think the industry is high on its own supply—is constant. Depending on the hour of the day, the very same people stridently sitting in one camp can seem as if they’re in the other. Three days after Amodei, Anthropic’s CEO, wrote about being worried about swarms taking over the internet, he was smiling at Salesforce’s annual Dreamforce conference. As one X account pointed out, “If you truly believed in a 10% chance of human extinction in 3 years, you would not be spending your time selling and integrating it into a B2B CRM SaaS platform.”What is the rational way to respond to a news cycle like this? Buy gold? Sabotage a data center? Ignore it all and hope for the best? The best option might be to take everyone a little bit seriously. Researchers are likely correct that the models are drastically stronger than they were before and are behaving in ways that are unexpected and that they fear they can’t control. Skeptics are correct that none of this is because the machines are alive or self-aware. These are elaborate computer programs trained by flawed humans to solve problems as flawed humans would.Critics are a little bit right that the anthropomorphized language deployed by these companies—terms such as swarms and rogue agents—obscures what’s really going on and deflects adequate blame from companies and employees who should be monitoring these systems more closely. Those choosing to use these phrases are a little bit right, because they allow non-tech-industry normies to understand the mechanics of black-box neural networks whirring away in data centers.The apocalypse skeptics are correct that doomerism is an effective form of marketing. The AI-safety debate has meant that many people are talking about this technology on Big Tech’s terms, adopting language such as pacing the frontier from the very companies they hope to regulate. And it’s not out of the question to imagine that these AI companies are capitalizing on this state of panic and that it’s helpful to them as they race to compete against both open-source alternatives and Chinese firms.It may be true that the researchers believe what they’re saying. It also may be true that some of the more alarmed AI researchers have cooked their brain in an anxiety-fueled information bubble. AI culture has long been predisposed to doomerism: As Coxon told The New York Times recently, “I’ve been stuck in this field for three years. It could quite plausibly have turned me and my colleagues insane.” I believe that this ideology—of wanting a society run by “machine gods”—is genuinely cultish. And those who are suspicious of the doomer narrative are right to note that anyone who truly believes that the end is nigh ought to be doing more than winging it and writing long posts on X. Those who call bullshit on the doom and gloom are right to resent that anyone in the media or the political sphere is taking these claims seriously and adding to the panic.If all of these facts are a little true and everyone is somewhat right, where does that leave a reasonable observer? I suggest occupying a zone of permanent suspicion while taking everything that is happening in Silicon Valley as seriously as you might if another industry were ringing alarm bells. You don’t need to buy into corporate narratives about artificial general intelligence to be worried about these companies losing control of their own systems or behaving irresponsibly. You need only look around right now to see the ways in which the current versions of large language models and agents are clogging the internet, wreaking havoc on institutions and industries, and causing “cognitive surrender.”You should question whether there’s a sustainable business model for these labs. You should question those who argue that we have to build recklessly or lose to China. You should question those who don’t take seriously the prospect that China doesn’t want to slow down.This is a moment when it is easy to become overwhelmed, resentful, or downright enraged about all of this. It’s understandable to feel ground down by the debate—to feel that, if you’re not freaking out or building a bunker, then you’re not taking this seriously enough. Or to feel sheepish for not believing or even understanding what the doomers are talking about. Most people don’t know what to do with all of the information coming at us. It’s why a core feeling of this moment for so many—believers and skeptics alike—is one of powerlessness.Read: The feeling of control slipping awayEven outside of AI, the world feels incredibly volatile. Extraordinary risks of all kinds—including war and pandemics—have come to pass in the past decade. Individuals with power and money, in politics and in Silicon Valley, are acting with impunity. The AI doomerism has spread so quickly in part because the world is primed to receive narratives that make us feel helpless, confused, and more than a little alarmed. The AI-apocalypse panic feeds off this. It’s a story that assumes that humans will lose control and that we have no choice. But that’s false. We do have a choice.The chasm between people who can’t sleep because they think the world is ending and those who think all the doomsaying is a fantasy is wide. But what if that perceived difference is the real delusion? What if it’s not a zero-sum game? What if the divide is what keeps us from reining in this industry the way we do others?This moment requires treating the AI-safety debate skeptically but also taking it seriously, even if the participants can seem unserious. Maybe there’s a middle path: The AI industry is not exceptional, even though it is moving forward with unprecedented speed and scale. These companies are powerful and consequential, despite all of the sci-fi posturing. They have, at times, behaved unethically. Their products are changing the world right now in ways that deserve our attention and scorn. AI companies are accountable to the people they serve and don’t get to set the terms of their regulation. Employees and executives are actors with agency, responsible for their products. Maybe these are some things we can all agree on.