AI companies say their tech could kill us all. Is that just a marketing ploy?

Silicon Valley has long warned that its favorite technology might get us all killed. But chatter about AI’s apocalyptic potential has grown louder in recent weeks.

In September, Anthropic researcher Jacob Coxon resigned from the company, accusing his former employer of “gambling with our lives” by developing evermore powerful models faster than it could control them. One of Coxon’s former colleagues vouched for this assessment, saying that there was a greater than 10 percent chance of AI “killing all humans” within the next decade.

Recent pronouncements from the leaders of these companies have been nearly as grim. At the United Nations last month, Anthropic CEO Dario Amodei warned that, if “managed poorly,” AI models “could be a risk to humanity as a whole.” OpenAI’s Sam Altman similarly declared that our species just might “lose control of the future” to the type of software that his firm is selling.

All this triggered a tsunami of media coverage and policy debate about the “existential risks” of AI — and what, if anything, we can do about them.

Yet some of the AI industry’s biggest critics are more annoyed than afraid. As they see it, the world isn’t trembling on the precipice of robo-annihilation, so much as falling for the tricks of snake oil salesmen. This argument has been percolating for years, voiced by a broad set of observers, including the linguist Emily Bender (“It makes the product seem more powerful”), computer scientist Timnit Gebru (“It’s not just a distraction, but it’s harmful”), as well as Substack columnist Ed Zitron and various reporters and social media influencers.

In their thinking, Altman and Amodei aren’t talking up their technology’s catastrophic potential because they’re rationally afraid of where AI is going — they’re doing so because their firms are ravenously hungry for more investment. And the reason why “AI doom” has grown more salient in recent weeks isn’t that the case for concern has gotten stronger, but rather that OpenAI and Anthropic’s IPOs have gotten closer.

In short, these Big Tech skeptics argue, journalists and politicians have fallen prey to a counterintuitive PR strategy, one that stokes media frenzies over ludicrous sci-fi scenarios in order to hype AI’s technological promise, distract attention from chatbots’ present-day harms, and help labs secure favorable regulatory treatment.

This would be reassuring, if true. Unfortunately, the theory is unpersuasive. There is little reason to believe that the panic over AI’s “existential risks” was cooked up in a Silicon Valley marketing department. To the contrary, the most plausible explanation for the AI sector’s doomsaying is also the most straightforward: Many of the industry’s executives, employees, and analysts genuinely believe that AI progress poses catastrophic risks.

In my conversations with close observers of the industry, even some of OpenAI and Anthropic’s fiercest detractors conceded that their leaders’ avowed concerns are unlikely to be mere marketing tactics. “The more I learn about the underlying ideologies out of which those two labs were formed, the more I think what we have here is a sincere belief,” the author and computer scientist Cal Newport told me.

This does not mean that Amodei and Altman are right, much less that we can trust in their judgment or integrity. There are many reasons to doubt that the weather forecast for 2036 is “cloudy with a chance of robot apocalypse,” or that anyone can assign a “10 percent” probability to the latter outcome without pulling numbers out of thin air.

Nevertheless, Silicon Valley’s Cassandras have some logic and evidence on their side. AI models are growing both more capable, autonomous, and arguably unruly. If these trends continue unchecked, it’s not hard to see how they could yield very bad outcomes.

Given the stakes, we should not be comfortable dismissing their warnings on dubious grounds. And the “PR” theory of AI doom does precisely that.

How AI companies (supposedly) profit off fears of the robot apocalypse

Before examining why AI “doomerism” probably isn’t a PR stunt, it’s worth saying a bit more about why some believe that is.

According to Bender, Gebru, and others, AI companies benefit from talk of artificial intelligence’s existential risks in at least three ways.

First, it attracts investment. In the words of The Atlantic’s Matteo Wong, the doomer narrative implies that AI labs’ models are “destined to become terrifyingly capable,” and thus massively profitable. As the Los Angeles Times’s Brian Merchant put the point during the first big wave of AI panic in 2023, “what better way to generate a buzz than to insist, with a certain presumed credibility, that your new technology is so potent it might unravel the world as we know it?”

Second, hyping AI’s hypothetical future hazards diverts attention from the companies’ current malfeasance. Every minute we spend debating whether AI will one day turn us all into paper clips is a minute we aren’t discussing the AI labs’ alleged copyright violations, pollution, labor violations, and myriad other social harms.

And third, the doom narrative shifts regulatory scrutiny away from topics that directly threaten the AI companies’ profitability — such as copyright enforcement — while also suggesting that regulators must rely on the labs themselves for guidance.

“If you position these threats as all-powerful, as somewhat unknowable, you get to be the expert on it,” Kate Brennan of the AI Now Institute, a progressive think tank focused on artificial intelligence’s near-term harms, told me.

Critically, for the biggest skeptics of “AI doom,” industry executives and employees aren’t the only ones complicit in this PR strategy. Self-styled whistleblowers like Coxon are also suspect, since they might be “using the spotlight to attract funding to start a new venture or to secure new positions.” And even nonprofit AI safety organizations are thought to function as vessels for Big Tech’s marketing campaigns, a fact reflected by the financial and interpersonal ties between the major labs and such think tanks.

There are better ways to raise money than promising to end the world

The “PR” critique of AI doomerism has some kernels of truth. AI companies probably would prefer to talk about their technology’s hypothetical future dangers than about their products’ present-day harms. And if publicly touting AI’s existential risks were antithetical to Anthropic and OpenAI’s business goals, their leaders would almost certainly say less in public about that subject. (Disclosure: Vox Media is one of several publishers that have signed partnership agreements with OpenAI. Our reporting remains editorially independent.)

But it does not follow that Altman and Amodei’s warnings are a self-conscious marketing strategy, born of cynical calculation rather than genuine concern. And there are several reasons to doubt that charge.

For one thing, AI companies have no shortage of ways to hype up their technology’s capabilities. And it is hard to see how any firm could conclude that the optimal approach — from a purely financial standpoint — would be to portray itself as a threat to human survival.

For investors, the appeal of AI companies is not that they could hypothetically get us all enslaved by a machine God; it’s that they could theoretically make a lot of money.

Perhaps casting your chatbots as an “existential” risk will persuade some capitalists of your future profitability. But this seems like an awfully indirect pitch — and one that’s almost certain to be less effective than demonstrating that a lot of corporations would like to rent your software.

“If these guys were selling flypaper — and the revenue was growing as fast as it is — investors would be more or less just as interested,” Roy Bahat, the head of Bloomberg’s venture capital arm, told me.

And even if these firms were hellbent on awing the public with grandiose predictions of their technology’s future powers, they’d have little need to invoke the robot apocalypse. Touting AI’s potential to cure cancer, abolish poverty, or facilitate intergalactic travel would all seem to advance the same objective, while doing less to jeopardize the companies’ public reputations or freedom from intrusive regulation.

And in fact, many with a vested interest in the major AI labs’ success talk up the technology in precisely this way. OpenAI investor Marc Andreessen has decried AI safety concerns as a “full-blown moral panic” while insisting that artificial intelligence will empower humanity to cure all diseases and engage in “interstellar travel.” Similarly, Jensen Huang, the CEO of chipmaker Nvidia and major investor in both OpenAI and Anthropic, has called claims of AI’s catastrophic potential “complete nonsense.” If hyping AI doom is essential for stoking investment, not everyone in Silicon Valley has gotten the memo.

The media doesn’t seem that distracted

Promoting fears of AI’s existential risks would be a similarly odd approach to media management.

Without question, corporations always seek to direct news coverage away from their greatest legal and ethical liabilities. But companies can try to secure more favorable press in many different ways. And implicating themselves in the apocalypse does not seem like an optimal one, from a crisis communications point of view. At the very least, it’s difficult to name other industries that have attempted such a maneuver; atomic energy companies did not respond to media coverage of the Fukushima disaster by talking up the threat of nuclear winter.

What’s more, there’s reason to doubt that the AI doom discourse actually reduces scrutiny of AI’s immediate downsides in practice. It’s true that human attention is finite and that coverage of some problems will inevitably displace reporting on others. Yet when one story piques public interest in a person, institution, or industry, the media often responds by subjecting their broader behavior to heightened examination. In this way, warnings of AI’s existential risks might theoretically increase journalistic attention to the tech’s present-day harms.

And this seems to have occurred at my own publication. In the weeks since Coxon’s resignation kicked off a furor over AI’s apocalyptic potential, Vox have published more stories on AI’s near-term threats than we had in the month before his departure — including pieces about how artificial intelligence could imperil America’s critical infrastructure, abet war crimes, degrade health insurance benefits, aid bioweapons development, and damage adolescents’ mental health.

Larger news outlets appear no more “distracted.” Over the past month, the New York Times has published articles on how AI is increasing air pollution, undermining higher education, helping sports betting companies target problem gamblers, and depressing real wages for younger workers.

And it’s not clear that AI doom discourse is any more effective at shielding the top labs from regulatory scrutiny. After all, the politicians who take existential risk arguments most seriously also tend to be among America’s strongest advocates for regulating AI’s impacts on labor, the environment, artists, and education.

AI safety researchers probably aren’t fronts for Anthropic’s marketing department

Finally, the “PR” theory has difficulty explaining why so many with no great stake in OpenAI and Anthropic’s success worry about existential risk. Coxon is one of many workers who quit jobs in the AI industry, forsaking lucrative equity in the process, out of concern for artificial intelligence’s catastrophic potential. Likewise, independent AI safety researchers are among those who’ve rung alarm bells in recent months.

Attempts to reconcile these facts with the “PR” narrative can verge on conspiracy theories. In an article for the Bulletin of the Atomic Scientists, Sara Goudarzi writes that “Coxon’s resignation is the latest in a string of recent events in which the companies emphasized how unpredictable, risky, and impressive an all-powerful AI might be,” in an apparent bid to drum up “pre-IPO hype.” This phrasing attributes Coxon’s departure to his employer — casting it as an example of how AI companies sought to make their products look impressive, as though Coxon were working at Anthropic’s behest when he accused it of monstrous irresponsibility.

The AI alarmists are likely sincere

Given all this, I think we should favor a simpler explanation for why some AI industry executives, defectors, and researchers all say artificial intelligence has existential risks: They believe that to be true.

Some may be reluctant to credit Amodei and Altman with such sincerity. After all, if they truly believed their work could lead to human annihilation, why would they keep carrying it out?

But as staunch AI doom critic Cal Newport told me, they have their reasons. “Their goal is to try to get to superintelligence first because, if done right, it will deliver a transhumanist utopia — and, if done wrong, will kill us all,” he said.

Even AI Now’s Brennan, who considers the labs’ doomerism a “self-serving strategy,” suggested that earnest (if grandiose) ideas about AI safety play a role in their behavior. “Their escalation and acceleration of this tech goes hand in hand with the belief of its all-knowing, God-like power,” she said.

In other words, the leaders of Anthropic and OpenAI seem to believe that:

Are these beliefs unhinged and self-serving? Perhaps. Do they function as rationalizations for the pursuit of personal power? Quite possibly.

But whatever one makes of Amodei and Altman’s apocalyptic concerns, and how they have and have not acted on them, it’s important to understand that their worries almost certainly emerged out of reasoned argument and technical observation, rather than mere PR calculations.

Both leaders have deep ties to the Bay Area’s “rationalist” community, which has been fretting over the robot apocalypse since the first decade of this century. And like their dissident ex-employees and critics at think tanks, Altman and Amodei have witnessed some disconcerting trends in AI development.

I cannot do justice to “doomer” fears in my allotted pixels (you can find them ably summarized here, here, here, and in roughly 10 trillion other blog posts). But the fundamental concern is that AI models are growing more powerful and autonomous even as their workings remain highly opaque.

The labs know how to train these chatbots — to get them to translate prompts into useful code, legal briefs, or advice — but they don’t fully understand how the models get to the desired output. At the same time, AI systems are becoming increasingly capable, outstripping humans at more and more cognitive tasks. And users are giving them ever greater freedom to act in the world, empowering AI “agents” to browse the internet, access myriad software tools, execute code, and hatch their own plans for completing multistep tasks. If these trends continue, it’s not difficult to see how things could go badly: A supremely capable AI model might pursue an objective in unanticipated and disastrous ways before their human prompters even realize what is happening.

“What makes me particularly anxious is the combination of us handing over more power and oversight to the models, even as we understand why they’re doing what they’re doing less and less,” Nat Purser, director of US policy at the AI Verification and Evaluation Research Institute, said.

Recent events have arguably provided proof of concept for these anxieties. Earlier this year, OpenAI instructed a bunch of models to complete various coding and hacking tests. When the systems discovered that some questions were unanswerable, they concluded that the best way to realize their objectives was to hack OpenAI’s internal systems and open-source machine learning website Hugging Face. This incident — and others like it — helped fuel the past month of AI panic.

Of course, it takes a big leap to get from “unsupervised AI agents can misbehave in certain eccentric laboratory conditions” to “there’s a 10 percent chance that no human will be left alive by 2036.” My point here is not that the doomers are right, only that their fears partly reflect trends in AI development that appear genuinely hazardous.

Taken to its logical conclusion, the “it’s all PR” theory suggests that regulators should treat the AI’s hypothetical, long-term tail risks as an afterthought — if not an active distraction from its more important harms. To make the case for such a policy, however, skeptics must identify flaws in alarmists’ evidence and assumptions. Arguments that merely impugn their motivations on improbable grounds aren’t worth betting the species on.