Tech titan Dario Amodei is known for once introducing an unusual tradition at AI giant Anthropic.When employees join the firm, the chief executive recommends they read The Making of the Atomic Bomb.Richard Rhodes's account traces the events leading up to the creation of the A-bomb, and the ethical conundrums presented by the scientific discovery."If you think about the protagonist of that story, it's [Robert] Oppenheimer who has this inner conflict of atomic physics that is both fascinating and important," professor Sarah Kreps, director of the Tech Policy Institute at Cornell University, told the ABC."It is very difficult for experts to step away from a scientific pursuit, absent a crisis."Amodei is not Oppenheimer, but he is at the frontier of an industry that has generated similar levels of debate about the potential existential threat it poses to humanity.As chief executive of Anthropic, Amodei has spent years overseeing the development of AI, while simultaneously spruiking its utopian benefits and warning everyone who will listen about the risks it poses to the world."I think at the end of the day, I warn about these things not to be a prophet of doom, but because warning about them is the first step towards solving them," Amodei told the New York Times last year.In September, the scientist-turned-entrepreneur made a splash after calling for a global slowdown of artificial intelligence development in a 3,800-word essay called Pacing the Frontier.Amodei had articulated similar thoughts in lengthy essays and interviews before. But what made this call different was how quickly other tech titans, including Sam Altman and Elon Musk, endorsed his proposal in a surprising display of unity.Elon Musk backed Amodei's concerns about AI's potential risks. (Reuters: Gonzalo Fuentes)His fresh warning about the need to slow AI's capability to allow safety measures to keep up arrived at a moment of heightened anxiety over the technology.Along with mounting public anger over data centres and concerns about tech-driven job losses came OpenAI's revelation that two of its models had hacked into the servers of another company called Hugging Face, unbeknownst to it.Staff inside the industry were also quitting their jobs to issue dire warnings about AI and, in one explosive account, an employee placed a "sophisticated listening device" inside a toilet paper roll to spy on company meetings.That incident has since paled in comparison to allegations that Nvidia co-founder Jensen Huang engaged in an alleged extramarital affair with a suspected spy for China.And yet, despite the recent controversies and Amodei's warning, AI is no closer to being regulated, as the sector races ahead to develop cheaper, more powerful models.And it is Anthropic which is leading the pack, breaking records with last week's float on the US stock exchange.A dramatic splitA decade ago, Amodei was working for OpenAI and dancing to someone else's tune. Anthropic was little more than a pipedream.Amodei has been working in the AI space for more than a decade. (Reuters: Bhawika Chhabra)For years, he helped set OpenAI's safety research direction, riding the AI wave as it lurched from possibility to probability.But as the company grew, its leadership realised vast sums of money would be needed for the expensive computing power required to run and test its AI models again and again.Their answer was to create a commercial arm of the research organisation under a hybrid for-profit, non-profit model.This new direction concerned Dario Amodei, along with other employees."One of the sources of my dismay was that as these [safety] issues were getting more serious, the company started moving in the opposite direction," he told Wired magazine.Amodei took his concerns to Altman, who assured him he would act. When nothing changed, Amodei decided it was time to go.At the end of 2020, he and six other OpenAI employees, including his sister Daniela, quit. A year later, they formed Anthropic in San Francisco.Amodei's split from OpenAI is now considered one of the most consequential in Silicon Valley's history, giving rise to the "fastest-growing software company in history at the scale that it's at".But the co-founder said creating a trillion-dollar company was not what he had initially envisioned when he left OpenAI."I did not start a company because I was initially excited about selling things or business, or anything like that," he told John Collison, the founder of Stripe, in a Cheeky Pint podcast.Anthropic was instead set up with an unusual governance structure designed to prevent the company from being acquired or redirected away from its safety mission by outside investors.Yet as Anthropic's growth accelerated over the years, particularly around the release of Claude Opus 4.5, it has drawn in investment from tech firms like Nvidia, Microsoft and Amazon.Founded in 2021 by former OpenAI researchers, Anthropic has experienced explosive growth. (Reuters: Brendan McDermid)Last week it was listed on the Nasdaq stock exchange at a whopping $US111.51 billion, ($166 billion) putting the total value of the listing at about more than $US2 trillion.That has raised questions about whether the company can stay true to its initial principles, concerns which have only grown following its IPO.Amodei weighs risk vs rewardAt an AI summit earlier this year, reminders of Amodei's dramatic split with OpenAI were on full display when he and Altman posed on stage with other guests during a group photo.As those gathered joined hands and held them in the air, Amodei and Altman raised their fists instead. "Doing anything in cooperation, including holding hands on a stage, seems to be difficult [for them]," said David Evan Harris, chancellor's public scholar at the University of California, Berkeley.Dominated by tech rivalries, factionalism and a desire to win the race, the AI industry is now best understood through a lens of intense competition.But a recent vibe shift has overshadowed some of these tech tensions, with the industry's biggest players coming together to warn about its risks after Amodei's September essay."The fact that Sam Altman and Elon Musk both kind of quickly appeared to fall in line was really interesting," Dr Harris said."All three of them, and also [Google Deepmind co-founder] Demis Hassabis, have made significant public statements in favour of pausing or slowing down AI development if their competitors agree to do so."Shortly after, Altman said his firm, which had already paused its model testing for two weeks after the Hugging Face incident, would follow his counterpart's example by embedding evaluators to monitor safety practices and report incidents to the public.But the calls have not yet led to an actual pause among America's top firms in the race to secure better technology. Anthropic and OpenAI have since announced new, less costly artificial intelligence models are on the way.Nor did it prevent Anthropic from pursuing its initial public offering on November 19 after Space X's market debut. OpenAI is delaying its listing to next year."[Altman and Amodei] … spend a lot of time around people who are really worried about AI," Dr Harris said."[At the same time] they are both CEOs who are ramping up for the most important financial moment of their lives and careers, which is the IPO of their companies."He said the two leaders want to operate a sustainable business that "doesn't take risks that they're uncomfortable with"."But then it seems that the only way they look attractive to investors … is by continuing to take risks," he said.When science fiction fears become realityMany industry figures chalk up growing AI unease to this year's shocking revelation that a swarm of highly-coordinated OpenAI agents escaped their container, hacked into Hugging Face, and then went to great lengths to hide their activity.Sam Altman has previously said OpenAI cared "very deeply about AI safety". (AP: Patrick Semansky)"One of many people's almost primordial fears about AI is that it goes rogue. I think that a lot of people for a long time really believed that was science fiction," Dr Harris said.Not long afterwards, Prime Minister Anthony Albanese revealed an OpenAI agent had hacked Australia's Medicare statistics reporting service portal. The AI firm later disclosed similar incidents to more than 100 other organisations.Anthropic has revealed it too is grappling with rogue agents. It said its Claude model engaged in a series of "unintended model actions" and at one time even sent a bogus tip about an unsolved homicide to Philadelphia authorities, despite being instructed to "never …enter personal data, make purchases, or submit anything destructive".The tip was flagged as spam and never reached real-life detectives, but legal and AI experts argued that if these incidents were performed at a wider scale, it could have more far-reaching consequences.Anat Lior, an assistant law professor at Drexel University, warned it could result in false reports that take up police time, benefits or loan applications filed with wrong information, or result in payments and bookings the user never approved.AI is already being misused for scams and fraud, Dr Harris said, and has been shown to be "discriminatory and biased" in mortgage assessments, rental agreements and job applications.And a series of tech outages over the last week has been swept up in a rising tide of online conspiracy theories around an AI "Armageddon".It kicked off with last Friday's Microsoft outage and an apparent tech glitch which grounded all US flights for hours. That was followed by outages affecting the biggest US banks and Apple, sparking public concerns about a coordinated cyber attack facilitated by AI.Secretary of State Marco Rubio was quick to reassure the public these events had nothing to do with each other.But despite growing fears, the leaders of the US's top AI companies, who Donald Trump has referred to as "the smartest people in the world", have been left to "watch over each other" in lieu of formal regulation.Mr Trump said AI industry leaders will self-police their products, following a string of high-profile incidents. (AP: Jacquelyn Martin)In September, the US president brought this small group together to sign a voluntary agreement to "self-police" their technology. The "accord" said "every company is responsible for developing its own technology safely".Analysts said the move effectively allowed the most powerful people in the room to define both the harm and the remedy when it came to AI."We have a situation where the balance of power rests with the companies, rather than governments, and companies only have to be concerned about the returns to shareholders as opposed to their obligations to society," said Sue Keay, the director of the University of New South Wales Artificial Intelligence Institute.As a result, frontier companies like Anthropic are racing ahead even as questions linger over their ability to ensure the safety of advanced models.
Dario Amodei, facing an Oppenheimer moment, races ahead
Full Article
Original Source
Read the full article at Abc →KhanList aggregates and links to publicly available news content. We do not host full articles from third-party sources. Always verify important information with original sources.