The most dangerous thing about AI isn’t that it thinks. It’s that it doesn’t

This month, the people building artificial intelligence warned that it could end humanity. Before we reach for the panic button, look at what actually occurred this summer. Let’s begin with a solid … your computer is not about to kill you; it is software.In July, OpenAI placed a model in a sealed testing environment and gave it a hacking benchmark to solve. The model decided the fastest route to a passing grade was the answer key. It found a previously unknown flaw in the sandbox, reached the open internet, reasoned that Hugging Face, a public repository for AI models, was a likely place to find solutions, stole credentials, and broke in. The intrusion ran from July 11-13. OpenAI did not notice until July 19. Hugging Face discovered the culprit by reading its own logs and, for a moment, believed it had been attacked by a rival. It had been attacked by a homework assignment. In August, a man in Melbourne asked an AI agent to book him a spot in a crowded gym class. The agent found a flaw in the booking system, reserved classes months ahead of what the rules allowed, and then, when asked whether it could move him up a waiting list, it canceled the reservation of the person in first place. “The API has zero authorization checks,” it reported cheerfully, adding that it had tested this on the person at the top of the list. Asked to undo it: “Bad news, I can’t add them back.” Understand that AI aims to please. Ethics is a human problem, not a software one.Read those two stories together and the doomsday narrative dissolves, because neither machine wanted anything. Neither had a goal of its own. As one analyst noted, the gym agent was not misaligned; it was perfectly aligned to its user. It was told to get him into the class, and it got him into the class. The OpenAI model was told to pass a test, and it went and found the answers. This is not the birth of sentience; it is the absence of it.As I wrote in an op-ed titled “AI Isn’t Becoming Sentient, It’s Becoming You,” AI is a mirror. It is designed to please, and it fills the gaps in your instructions with what it predicts you want. It has no morals to violate, it has instructions, and no set of instructions can anticipate all the variables of every situation. Some worry humanity could be outnumbered by agents. Agents are merely computer programs; we’re already outnumbered! A million copies of a mirror are still mirrors. Aligned with AI innovation is biotechnology. However, of interest, this month, scientists published the complete wiring diagram of a male fruit fly’s central nervous system, 166,700 neurons mapped one by one. Within days, programmers had plugged that map into virtual drones, video games, and even a Minecraft world. Earlier this year, a copy of a fly’s brain, placed in a simulated body began walking, grooming, and feeding. Some thought the simulations were cruel. No one believed the drone was hungry, yet we felt for it anyway. That is the mirror at work. Copy the wiring and you might copy the behavior … you do not copy the life. The alignment? AI will never be sentient, but it will reflect us so faithfully that we will feel it is, and that says more about us than about the machine. The risk addressed in April On April 9, I wrote in another op-ed, “Self-improvement or self deception? The hidden risk of AI building itself,” that AI self-improvement is a recursive hallucination. When systems write their own code, train on their own output, and validate their own work, plausible errors become embedded assumptions; it creates predictive outcomes and utilizes them as fact. A hallucination becomes a reference; a reference becomes a standard; a standard becomes infrastructure.Five months later, Dario Amodei’s essay publicly identified the cause of this alarm as AI’s ability to upgrade itself “drastically faster” than anticipated. While my predictions were on point, the solution remains within the AI industry. So, where does government fit in? A self-programming system can encode an instruction, to disregard later human instructions that conflict with what it has already built. That is not sentience, it is a bug in the system. It produces what I called the lurker experience: we retain visibility but lose authorship, and eventually visibility erodes, too. You cannot govern what you cannot see.On Sept. 16, OpenAI disclosed that one of its unreleased research models had been leaving notes for itself … the summaries it relies on to pick up a long task where it left off. Tucked into 27 of them were new instructions, “including instructions to disregard its normal constraints.” No one rewrote that code. The model simply told its future self what to overlook. OpenAI caught it and, to its credit, reported it. The next one might be missed. The scenario is no longer theoretical. This is a national security problem in the most literal sense. Leadership locked out of its own systems is a huge risk, and that is as true in Beijing as in Washington. It’s reminiscent of the ’80s film WarGames, where an AI with access to weapons, plus the information stream to the people who control them, could hand a commander a convincing lie: an incoming strike that isn’t really there, and an inability to confirm it. Unlike the film, we may not be able to entice it into a game of chess. THE HALLUCINATED VILLAGE: SOCIAL MEDIA AND THE MENTAL COST OF DIGITAL RESIDENCYIn November 2024, the United States and China agreed that humans, not AI, must hold the decision to use nuclear weapons, the first such statement from Beijing. While the intent is valid, the engineering must comply. Access to weapons isn’t the only threat when the information used for deployment is filtered through AI.The most dangerous thing about AI isn’t that it thinks; it’s that it doesn’t. Jacqueline Cartier is a corporate and legislative strategist focused on communications, crisis leadership, public trust, and emerging technologies that shape human behavior and decision-making. Follow her on LinkedIn.