Mother Jones illustration; Getty
We’ve been hearing a great deal recently about the specter of AI-driven human extinction. Maybe our misplaced trust in AI intelligence will set off World War III. Or AI agents will unleash widespread cyberattacks and take over the internet. Given too much autonomy, the machines will run amok. They may seek to deceive their creators—wait, that’s already happened. Suppose they conclude, like Agent Smith in The Matrix, that “human beings…are a plague, and we are the cure.” Or perhaps the chatbots help bad humans unleash a literal plague—a biological weapon.
That last one isn’t merely theoretical. A “misuse” report Anthropic made public earlier this month detailed five cases in which foreign scientists tried to evade Claude’s guardrails on “gain of function” inquiries. That’s a euphemism for when scientists tinker with a pathogen’s genes to enhance it in some way, making it more infectious, deadlier, or better able to spread among mammals, for example.
“Think about all the people—scientists, nation states, academic institutions—that are manipulating pathogens for greater virulence.”
Anthropic denied those users the answers they sought and cut off their accounts, even while noting that their motives may not have been sinister. Such information also can be used to help scientist develop pandemic countermeasures like vaccines and drugs. And there’s the rub. Like so many technologies, AI included, genetic engineering can be used for both good and evil. Hence another euphemism: “dual use.”
I studied to be a molecular biologist, and have been covering the gain of function debate since long before AI was on everyone’s radar. In my experience, researchers who work on high-risk pathogens tend to default to the view that it’s better to publish than not, and that we’re better off doing the work to understand which genetic traits make organisms a threat to humankind—never mind the potentially catastrophic consequences of an accidental release, or a deliberate one.
Is humanity really better off doing the work? I have my doubts. And I have more of them after reading the latest book from Annie Jacobsen, whom I last interviewed about her best-selling book Nuclear War: A Scenario. Her new book, Biological War: A Scenario, came out in late July and follows the same template. And because I know you were about to ask, yes, I’m sure she is great fun at cocktail parties.
One thing about Jacobsen, who has decades of high-level national security reporting under her belt, is her unrivaled access to current and former military and government officials and scientists—people who spend their days thinking about threats to humanity and gaming out how the government might respond if such-and-such horrific thing were to happen.
Beyond the notion of human extinction, perhaps the most terrifying aspect of both of her books relates to the limitations in our government’s preparation and response. America’s general plan for nuclear war was, and no doubt remains, complete madness. As for biological war, spoiler alert, it turns out we really can’t defend against it. (You’re welcome.) I’ve edited our conversation for length and clarity.
So, Annie, now that you’ve done both the nuclear scenario and the biological scenario, which is more likely to keep you up at night?
The Defense Department document that I reprinted in the book, on the weapons of mass destruction threat spectrum, lists nuclear weapons at the far end of the curve because they are the most destructive. But I was really surprised to see how biological weapons are a close second. And I believe the reason they’re the greatest overall threat very much ties into the discussion the entire world is having today, which is that the threshold for entry is extremely low, and therefore it is more likely to happen.
To have a nuclear weapon, you have to have weapons-grade fissile material, and then you have to think about containing those weapons within the arsenal. You have to have an ICBM, a B-2 bomber, a nuclear-armed submarine.
You’ve got to be a state!
Yes. A terrorist with a dirty bomb, that’s just a completely different threat. The component that makes biological war so incredibly dangerous and the greatest overall threat is that there are nine nuclear-armed states tasked with this containment component, but then you look at the rest of the world and who has access to [potential pandemic agents]…
Most everyone.
Yes. And that becomes terrifying.
If you were a bioweapons designer, what would you look for in a pathogen?
My mind does not think that way because I am a mother and I’m a member of society. But If I were a pandemic planner, what would scare me the most is the concept of an airborne agent and the public’s naiveté to that very concept. I would think about how important it is to educate the public about the reality of what is an extraordinary threat and what isn’t.
Your scenario involves the accidental release of a weaponized pathogen from a high security lab in Russia. Based on the experts you spoke with, is an accidental release more or less likely than someone deploying a bioweapon?
Accidental release is positioned as a much greater threat for a couple of reasons. One, I learned there are very serious counterproliferation efforts at the CIA and other intelligence agencies to disrupt bioweapons programs. Whether you want to sleep better at night knowing the CIA is out there countering these things is up to you, but there are examples of biological weapons programs in the hands of terrorists being shut down.
What I came to understand is that the only people crazy enough to deploy or release a biological weapon into the community, knowing full well that the blowback would be extraordinary and would create a pandemic around the world, would be someone who had a death wish. Many terrorists fit that that category, so that is deeply unsettling. But that’s a smaller group.
And then you think about all the people—scientists, nation states, academic institutions—that are manipulating pathogens for greater virulence, for greater transmissibility in these BSL-3 and BSL-4 labs [Biosafety Level 4 is the highest security designation]. I mean, there are 3,600 of those labs around the world. There are not 3,600 terrorist organizations that I am aware of. So the chance for an accident is statistically far greater. That’s why I begin my scenario with a lab leak, and also because it’s happened before.
One can imagine a researcher sticking themself with a syringe at a BSL-4 lab and freaking out, humans being humans, and kind of going into denial and heading home.
One hundred percent—and as has happened. Then you have other examples, like [Nikolai] Ustinov, whom I wrote about in a little history lesson in the book, sticking himself with a deadly Marburg virus, from which there is no cure, and doing the good Soviet scientist thing, which is tell on himself, and then be left in a isolation ward behind sort of submarine-lock type doors to die a horrific death by hemorrhagic fever. Accidents happen.
You and I have both written on the risks of gain-of-function research, like reconstituting pandemic flu strains that can spread among ferrets—god help us if that ever got out. You write in the book that it was after 9/11, when someone was mailing around anthrax, when the riskiest research accelerated.
Yes, and I think it’s interesting. With anthrax, you’re a “dead-end host,” meaning if you breathe it in, it will kill you but it’s not transmissible. The anthrax wasn’t ever going to become a pandemic. It was meant to make people experience terror. But it’s very different when you’re talking about an airborne pathogen.
And so, flash forward to where we are now, with [advanced] genetic engineering, with gain of function research, with AI-enabled bio, and you have a situation whereby everything is on the table in terms of pandemic potential. You can tweak things. Scientists in a lab can work to create a pathogen that’s able to evade existing immunity, like the mouse pox experiment, or make something airborne, like the H5N1 ferret flu experiments. We are in a completely different world now.
I was taken aback when I called up that flu researcher, how arrogant he was: Of course you have to publish the sequence of this incredibly deadly thing. He didn’t think twice about it. There’s a hubris with some of these guys.
A hubris, or an elitism, where you talk down to the laymen because they are not scientists. As if they can’t make decisions about national security or public health because they are not scientists. I find that incredibly alarming. One of the things that’s been most impactful about my scenario books, both nuclear and bio, is that I have had subject-matter experts around the world, people who write treaties, the cardinals at the Vatican, diplomats at the United Nations, say to me: “Thank you for making this very simple to understand.”
Simplicity is powerful! I’m glad you mentioned AI, as we’re being deluged lately with existential concerns, including the Anthropic report about people using Claude to potentially try and facilitate bioweapons work. Your book doesn’t address AI at all. Did your sources have serious concerns about this technology further lowering the bar to creating deadly pathogens?
Not even a question! I chose to not include AI-enabled bio in the book because I wanted to show readers how easy it is [even without AI] for a scientist in a lab just to tweak Mother Nature’s pathogens and have a pandemic that is a near extinction-level event. But in the acknowledgements, I write about the very frightening concept of mirror life, which David Relman, the biosecurity expert and presidential advisor, explained to me, where you can extinguish all humans on the planet. I wrote Biological War: A Scenario, not Biological War: The Worst Scenario, which is pretty crazy to think about.
“I chose plague because it seems to be this dark metaphor for the two different uses of science and technology, one for good and one for bad.”
What’s been interesting is the subject matter experts I’ve spoken to since the book published. I was recently invited to the BSL-4 lab in Fort Detrick [in Frederick, Maryland] by Dr. Nicholas Bergman, the lab director, to do a Q&A with the staff. He explained that when the book published, the phone started ringing off the hook. Everyone [who works there] was being asked, Is this really true? Could this happen? And people’s next question was, Why don’t we know about this?
Dr. Bergman said to me, “Annie, you said the quiet part out loud.” And so, I think the fact that AI-enabled bio-threats have exploded into our consciousness is a good thing, because it is making people pay attention to the quiet part.
The threat has always been there, but you are not going to have people who work at these labs raising the flag. That’s not their job, as it was explained to me. Their job is to work on the countermeasures. Many of their questions to me were about AI-enabled bio—they were looking to me for the zeitgeist. This conversation is now happening literally in the grocery store, and also at the highest levels of subject matter expertise.
Your book talks about how President Richard Nixon unilaterally scrapped the US bioweapons program way back when. Did your experts believe we no longer have a covert program?
Biological weapons are illegal and the US position is very clear. The State Department in the current administration and previous administration both made declarative statements that it is their position that Russia and North Korea have illegal biological weapons programs, and that China and Iran have programs of concern—which I take as a euphemism.
Given the likelihood of a bioweapon coming back on its creator, it really does seem like any deliberate release would have to be a non-state actor who maybe doesn’t care so much whether they live or die.
To map out these scenarios and understand how they roll, I looked at Pentagon biological war gaming scenarios and also at pandemic planners’ unclassified scenarios. Former public health officials and cabinet-level people gather and do these exercises. For one that I referenced in the book, called Glade X, they used an ecoterrorist group that had genetically modified a Nipah virus to make it more transmissible. In this scenario, the ecoterrorists’ ideological position was that there are too many people ruining Earth, so let’s depopulate.
Sounds like the agent from The Matrix: “Humans are a virus.”
Pandemic planners come up with these very dramatic narratives to play out and understand how these situations would unfold, and that’s where most of my major plot points, if you will, come from.
You also borrowed some details from Covid, including the outbreak aboard the Diamond Princess cruise ship, except your pathogen is way deadlier.
Yes. And when I was researching Glade X, I was stunned to learn that the government is deeply concerned with a mass exodus—in that scenario it was people from Venezuela making their way to Puerto Rico in boats. And what would happen if the US Navy had to open fire on infected persons in vessels just storming into the Caribbean. It seems apocryphal until you’re reading that these pandemic planning scenarios are being thought through by the government.
One key aspect of your book is the fragility of our social order in a plague scenario. So much hinges on how people react, and how authorities react.
I was interviewing [retired Air Force] Colonel Tom Ehrhard, who ran Pentagon war game scenarios. These are nuclear and biological catastrophic-level scenarios conducted for the Office of the Secretary of Defense [via] a legendary organization called the Office of Net Assessment. Ehrhard told me, “There is no real defense against a large-scale biological attack, only things you can do to mitigate extermination.”
That is a shocking thing to hear from someone who gamed out these scenarios for the Pentagon based on an understanding of how the government functions and how the people will respond.
As a fan of Devo, I was intrigued to learn that “devolution” is part of our government’s planning. Would you talk about that?
I don’t get the reference.
The band Devo! They were big in the 1980s. Still around, I think.
Was that a reference to devolution?
Yes, but their thing was the devolution of American culture.
I did not know that. Devolution is an actual plan I learned about from [President Barack] Obama’s former FEMA director Craig Fugate. It happens in the event of a catastrophic biological event like the one I describe, and anarchy breaks out—which is expected based on the population’s fear of infection and the realization that there are not enough medical countermeasures to go around, if at all.
“Working to erase the existing medical countermeasure of antibiotics from a disease with a 100 percent case fatality rate—that is dark.”
Anarchy leads to insurrection. The president declares the Insurrection Act, and at that point, as things really spiral out of control, the “continuity of government readiness” condition is lifted to Condition 1, which is the most extreme.
With COGCON 1 comes the triggering of the devolution plan. It involves a predetermined set of individuals, separate from those elected or appointed or in government, all of whom know who they are and have signed a contract committing to their responsibilities. They immediately leave the DC region for a series of classified bunkers located across the country. We only know that they’re called hot sites, cold sites, and warm sites. And those individuals go there and essentially wait out the catastrophe so that they can reconstitute the government.
It seems like something ripped from the pages of a dystopian novel.
It really does. The plan is classified. The name was leaked to the press about a decade ago, so FEMA officials decided to declassify the name, and that’s why Fugate could talk about it a little bit, and I was able to bring information to the table that lets your imagination kick in.
You chose a particularly nasty, weaponized, strain of the plague bacterium Yersinia pestis for your scenario. Why that particular pathogen?
I’m particularly interested in plague because we have all this history, and I could draw from accounts written contemporaneously by survivors. It was illuminating to see the utter devastation of humanity, the moral costs. I reference a lot of that in the book to give readers a sense of what actually happened: a pandemic that killed 150 million people, up to a half of Europe.
We think of plague as a disease of the past because of antibiotics. It’s astonishing that here we are, in the age of technology so advanced that we’re now worried about machines becoming smarter than humans, and we’re reverting to a situation where, because of science and technology, the medical countermeasure of antibiotics can be zeroed out. That’s why I chose plague: because it seems to be this dark metaphor for the two different uses of science and technology, one for good and one for bad.
Just like AI.
Was it your piece on the flu where someone said, “Well, we’re not just going to stop making knives because knives can kill people”?
Yes, that was the arrogant flu scientist. He also mentioned airplanes. It wasn’t too long after 9/11.
Yes! We’re not going to stop making airplanes! I’ve spent my career thinking about intent, and that’s the human component in all of this. I was astonished to learn that the Soviets had this incredibly dark, nefarious, very advanced plague program—a super plague weapon they were building—and this was 30 or 40 years ago, where genetic engineering was in its infancy. I interviewed the CIA and the British intelligence minders of these Soviet defector scientists who came here and told these [intelligence officials] what they were doing, so we got to hear things like, their intention was to kill all the people and leave the infrastructure intact. You can’t make that up.
You’d have to inoculate your entire population—and maybe that wouldn’t work by the time the pathogen circulated back around to you.
The Soviet Union was very much a castle on the hill. The wall hadn’t come down yet when they were building the plague super-weapon. And the Russians didn’t exactly fly to Paris for the weekend, so maybe there was this idea of, “We’ll inoculate everybody with a plague vaccine and the West can all die off and we’ll march in later and take over.”
You also point out that plague was perhaps the first bioweapon—when a 14th-century army led by a descendant of Genghis Khan threw plague-ridden corpses over the walls of the Republic of Genoa.
Yeah, it has had a long history, and it’s something pandemic planners and Pentagon war gamers consider a very serious threat. Pneumonic plague is airborne, with a 100 percent kill rate.
There aren’t many things with that kill rate.
Just rabies virus and plague. So, working to erase the existing medical countermeasure of antibiotics from a disease with a 100 percent case fatality rate [as the Soviets did]—that is dark.
You even made your plague a chimera—two pathogens in one!
Only because the Soviet scientists did it in the 1980s.
You think such demonic things still exist in Russia?
I can’t imagine that they wouldn’t. The way the State Department describes it, the illegal Soviet-era program was absorbed into the new Russian program. Which is deeply disturbing.
So, what’s next on your agenda? More terrifying scenarios?
[Laughs.] I write in my acknowledgements that the point is not panic. The point is pressure. On book tour, I’ve had so many people [who’d heard my book was terrifying] say, “I read it and I actually feel less terrified.”
You begin to realize there are existential fears that are based on hypothetical worry, and then there are real threats that you should be aware of. I would hope that my books would add to that part of the conversation.
