We must not be scared by Silicon Valley’s ‘AI slowdown theory’

(Publisher’s note. The American AI doomsday debate is in full gear and dutifully chimed into by West European media. The most recent wave was kicked off by a September 6 blogpost by OpenAI chief scientist Jacub Pachocki, amplified on September 8 when Anthropic researcher Jacob Coxon resigned, saying AI “could kill us all by the end of the decade.” In his screed, ominously titled “An Alien Mind”, Pachocki worries that humans might be “left behind by unchecked progress, brought about by an alien intellect exceeding our own” and calls for “voluntary slowdowns … until shared [international] safety bars are established.” The appeal, endorsed by OpenAI’s Sam Altman, Anthropic’s Dario Amodei and Elon Musk, comes as the US and Chinese presidents prepare to meet in Washington on September 24, when officials from both sides will discuss AI development and work toward shared guardrails. The following article by a member of the editorial board of the influential “Observer” (Guancha.cn), a news and analysis platform widely seen as reflecting views in Beijing, suggests that the American push to slow AI development is driven less by safety concerns than by a desire to preserve the country’s lead in the sector.) OpenAI Chief Scientist Jakub Pachocki recently wrote an article titled “An Alien Mind.” Yesterday, our commentary desk’s “World Scientific Forefront” column published a full translation of the piece. OpenAI’s boss, Sam Altman, also reposted the article on social media, and it instantly became one of the industry’s most-discussed pieces. The article opens with a story about one particular night. In mid-2023, when OpenAI’s “RLSlow” project produced its first batch of results, Pachocki and a colleague stayed at the office through the night. He said that what occupied his mind that night wasn’t benchmark scores or product metrics, but the process of coming to terms with a sobering fact: that he would, within his own lifetime, truly see machines smarter than himself. Three years later, he wrote this article, and at its core it makes a handful of points. Machine intelligence is built mainly by piling on compute, and it is still climbing. This thing is grown, not designed; its inner workings aren’t understood, and the reliability of chain-of-thought monitoring is declining. And recursive self-improvement (AI systems training their own successors) is the natural endpoint of the current path, so it requires extreme caution, voluntary deceleration, international coordination, and mandatory safety thresholds. The article has an earnest tone with a humble stance too. But if we look at the details of its reasoning, it reveals its own ignorance in too many places. 01. Align or not? Let’s start with the most obvious point. Pachocki says chain-of-thought monitoring is OpenAI’s most important bet, but internal evaluations show the tool’s reliability is declining. Several reasons: the model’s reasoning process is increasingly intermixed with dialogue and tool calls; the model is getting better at manipulating its own reasoning process; and, as pretraining grows stronger, the model can reason just as well without showing users its logical steps. His proposed response, then, is to build an automated AI researcher and use it to iterate on the alignment problem. You’ve already admitted chain-of-thought monitoring is unreliable, so how do you build this automated AI researcher? Can you even understand the researcher you’ve built? Using something you admit you don’t understand to solve the very problem of “not understanding it” isn’t without precedent; science has examples of using tools not yet understood to understand the world. But at the very least, this shows that the cure this article prescribes and the disease it diagnoses are one and the same thing. The article says the strongest reason to keep rapidly training smarter models is the need to build defense systems against the dangers posed by other AIs. This is a standard arms-race argument, and it directly contradicts the deceleration the piece calls for throughout. Has the author ever considered the theory that “antivirus software is the prime suspect behind the viruses it fights”? He may have sensed this, so he added a line noting this shouldn’t become an excuse for recklessness. But a patch doesn’t fix a structural problem. OpenAI argues that it must stay fastest, while others should slow down. In practice, only the first half of that argument ever gets carried out. The article acknowledges that real progress in alignment research has always been deeply intertwined with progress in general capability, and it offers two examples: reinforcement learning from human feedback and chain-of-thought monitoring itself. The latter became possible only once reasoning models emerged. So by his own account, slowing down would also set back progress on safety, and Pachocki’s article never resolves this contradiction. The most quotable line in the original is, “AI is grown more than designed.” The line is vivid, and easy to accept. But it quietly swaps identities, recasting engineers as gardeners. A gardener bears only limited responsibility for how a plant grows; he only waters it; the seed is natural, the weather is natural. Once this metaphor holds, part of the responsibility for the model’s behavior is handed back to nature. But the reality is different. The loss function is written by people, the data are curated by people, the optimizer is chosen by people, the compute budget is approved by people, and the decisions of when to stop training and when to release the model are switches that only people flip. Gradient descent is not a seed sprouting; it is a deterministic process defined step by step by people. What we don’t understand is its intermediate representation layer, not the underlying principle. If Nvidia Chief Executive Jensen Huang heard Pachocki’s argument, he might curse him out on the spot (he once told a reporter who raised the AI doomsday theory to “get lost”). Huang has never hidden his contempt for the “AI out of control” theory, and often compares using AI tools to using a microwave, saying a microwave and AI are the same thing, both just “processing data.” Huang may not fully win us over either, but we have to admit that in the real world, humans cannot trace the complete causal chain of a financial crisis, yet no one concludes from that that the market economy has slipped out of human control and that exchanges need to be shut down through global coordination. Unexplainable does not mean uncontrollable. Confusing the former with the latter is this article’s single most critical leap. 02. Perhaps the author should read 300-year-old Immanuel Kant’s philosophy What genuinely needs addressing at the philosophical level is the proposition that “recursive self-improvement” will exceed human understanding and control. In the Critique of Pure Reason, Kant carried out a famous inversion, the so-called epistemological “Copernican Revolution” (paraphrased here, not a verbatim quote). It is not that our knowledge must conform to the object, but rather that the object must conform to the forms of our cognition. Space and time are the forms of sensible intuition; causality, substance, and necessity are categories of the understanding. For anything to become a phenomenon we can experience, it must first pass through this gate. This means something capable of exerting real effects on our world cannot, at the same time, be something we are in principle unable to know. The moment it allocates compute, rewrites code, issues commands or affects the power grid, it has already entered the category of causality. It is already a phenomenon, already within the forms of our cognition. What truly lies beyond knowledge is the thing-in-itself, and by definition the thing-in-itself has no experiential relation to us. It is neither knowable nor capable of doing anything to us. Being unknowable and yet able to act upon us is a combination that does not hold up within transcendental philosophy. What people can genuinely know is precisely what people themselves have built, because the underlying principles are already in human hands. This is true of mathematics, law, history and deep learning as well. So a more accurate statement would be that we simply don’t yet have a mature theory of generalization. This is a debt owed within empirical science, one that needs more work to settle. Pachocki has converted this debt into a metaphysical assertion of being unknowable in principle. Between these two things lies an entire philosophical tradition, and it cannot be leapt across in a single bound. 03 Where does this fear actually come from? So why would a rigorously trained scientist make a leap like this? I can draw an analogy using one of the few English novels I actually worked through in secondary school, The Strange Case of Dr Jekyll and Mr Hyde, written by British novelist Robert Louis Stevenson in 1886. Dr. Henry Jekyll is a respectable doctor who creates a chemical potion to separate the good and evil in a person. He experiments on himself. At first it is entirely controllable; Mr. Edward Hyde (Jekyll’s evil alter ego) comes when called and leaves when dismissed. Later, he must increase the dose; later still, Hyde begins appearing on his own, without the potion. In the end, Jekyll loses all control, leaving only one option: to take his own life. Scientists, self-experimentation, short-term control, a tipping point, total loss of control and the only solution: destruction. This structure has barely changed from the medieval golem legend (a clay figure brought to life), to Frankenstein, to Jekyll, and now to Skynet (the AI antagonist from Terminator). Behind it runs a much older thread, a linear view of time: creation, fall, judgment, redemption. Fit this narrative onto AI development, and you get Silicon Valley’s current rhetoric. The technological singularity (the hypothesized point where AI surpasses human intelligence and improves itself beyond control) is the secular version of the Last Judgment, alignment is the secular version of redemption, and whoever in the lab is first to see the curve (the J-curve of exponential growth) becomes the prophet who knows the date. Pachocki’s sleepless night at the office reads remarkably like a page from Jekyll’s diary. But the Jekyll story rests on a premise worth noting: Hyde was inside Jekyll all along. It is original sin. Underneath the West’s fear of its creations spiraling out of control is a projection of its fear of the evil within humanity itself. This thread doesn’t exist in the Chinese tradition. In the Liezi (written by Chinese philosopher Lie Yukou, who lived from 450 BC to 374 BC), the craftsman Yan Shi presents King Mu of Zhou with a puppet that can sing and dance. The puppet winks at the king’s concubines, and King Mu, enraged, wants Yan Shi killed. Yan Shi doesn’t commit suicide; instead, he takes the puppet apart on the spot and shows the king it is nothing but leather, wood, glue and lacquer. The story’s solution is to open it up and look. This isn’t to say Chinese people are inherently more optimistic. It is to say that “creations inevitably turn against and strike back at their creators” is not an objective law about technology; it is a narrative habit that one particular civilization keeps retelling. Treating one civilization’s anxiety as a law of physics, and then using it to demand the whole world slow down, is a step that should be rejected. 04 The fear has spread with the bill landing elsewhere Silicon Valley’s rhetoric hasn’t stayed confined to papers. A Gallup survey from March this year found that 70% of Americans oppose building AI-supporting data centers in their own area, with 48% strongly opposed. For comparison, the historical peak for Americans opposing local nuclear power plants was only 63%. A May survey by Heatmap (a climate and energy news outlet) put the figure at 71%. In the first quarter of this year, at least 20 data center projects were canceled due to public opposition. Then an absurd scene unfolded. In late August, US President Donald Trump wrote on social media that communities opposing data centers would only become backward and poor. Several Republican congressmen jointly wrote to the Federal Bureau of Investigation (FBI), demanding an investigation into foreign influence operations targeting America’s AI development, pointing the finger at overseas billionaires, foreign state media and funding networks considered pro-China. A Fox commentator put it more bluntly, saying someone was trying to bring socialist policy into this new AI revolution. But the same polls show 78% of voters for Democratic presidential candidate Kamala Harris opposed local data centers, and 63% of Trump voters opposed them too. This is a thoroughly bipartisan, local issue, driven by electricity bills, water and land. Social platforms’ safety teams did uncover a bot farm originating from China, but among the 200,000 accounts named, only 200 consistently posted content about data centers. On one side, some AI labs are producing doomsday narratives. On the other, political leaders claim an enemy is planting this fear. Both are happening in the same country (the United States) in the same year, which shows America has not formed a social consensus on AI and has instead turned the issue into a fresh partisan rift. How this “fear” gets distributed is absurd. Those who create the fear gain regulatory influence and the moral high ground, while the costs include stalled grid projects, canceled investments and local politicians forced to choose between votes and AI industry growth. 05 What this narrative means for China There is no need to speculate about Pachocki’s personal motives. An argument’s industrial consequences don’t depend on whether the person making it is sincere. Even if he is a hundred percent sincere, once this discourse is adopted its effects are certain. Deceleration is asymmetric. The people calling for it hold the largest stockpile of computing chips, the deepest capital pools and the most complete software ecosystem. Pushing to slow AI development means freezing the current ranking. For the leader, slow means staying ahead; for the chaser, slow means never catching up. This is asymmetric braking: the same action, applied to different cars, produces completely opposite results. Now look at what those keywords actually translate to in industrial terms. International coordination is an entry barrier; third-party audits are certification power; mandatory safety standards are standard-setting power. This playbook has already played out in the semiconductor industry: the Wassenaar Arrangement (a multilateral export-control regime), the Entity List (the US trade blacklist), and licensing requirements for lithography machines. Technical standards have never been neutral; they legalize industrial position. So a few things are worth spelling out clearly for China’s policymakers, investors and industry. Safety and alignment work needs to be done, and done more solidly than anyone else’s, but it should be built as a capability, not as a brake pad. If China puts more effort into improving interpretability, auditability, and scenario-specific boundaries of responsibility, it can define its industrial standards on its own terms, rather than accepting a bar set by someone else. In fact, don’t take someone else’s anxiety as your own signpost. What’s hurting America most right now is precisely that its own public opinion has jammed up compute infrastructure, power, land, water and substations. This is an objectively real window of opportunity, and it won’t stay open forever. For China, the single most important point is to maintain its own rhythm. China-US competition in this round has entered a toe-to-toe, evenly matched stage, and the easiest mistake to make now isn’t taking the wrong direction; it’s letting the opponent’s emotions throw off your pace. Strategic composure isn’t a slogan; concretely, it means that when someone else shouts stop, the first move is to figure out whether they are genuinely afraid, or just want you to be afraid. Conclusion Back to The Strange Case of Dr Jekyll and Mr Hyde. What was truly fatal in that story was never the chemical potion. From beginning to end, Jekyll did this alone in a locked room, telling no friend, no colleague, and keeping his diary locked in a drawer. His loss of control didn’t come from the formula. It came from the lack of transparency. Pachocki’s article includes a confession worth reading twice. He says that when o1-preview was launched (in September 2024), they deliberately hid the chain of thought, on the grounds of protecting it, over the long term, from being contaminated by the pressure of being monitored. The other result of that decision is that from that day on, the outside world could no longer see what the model was thinking. A person calling on the whole world to slow down together is himself running a system the outside world cannot monitor. This isn’t a moral accusation. It is a structural problem. When monitoring can only happen inside one room, outsiders have only one choice for judgment about risk: Believe it or not. The real way out was never to shut down the lab. It is to open the door. Open the door: open-source, reproducible, verifiable by multiple parties. In that sense, what China has been doing with open-source models these past few years comes closer to the phrase “keep humans in the loop” than any alignment manifesto does. Yan Shi’s method was right all along. When someone suspects the puppet has become a spirit, take it apart on the spot and show them.