The Hacker Sent by Anthropic to Calm the Government’s Nerves About AI Safety

AI总结

核心结论

Anthropic研究员Nicholas Carlini从AI安全怀疑者转变为该技术的“内部守门人”，文章揭示了AI模型在自动化挖掘漏洞方面表现出超越人类的能力，引发了技术安全与国家政府监管之间的剧烈博弈。

结构逻辑

现状危机：以Carlini发现Linux/Ghost漏洞为切入点，描述AI显著降低了漏洞挖掘与攻击门槛（“Bugmageddon”）。
政企冲突：详述Anthropic与白宫在AI模型发布与安全性之间的摩擦及外交博弈。
人物转型：回顾Carlini作为“专业怀疑者”的职业背景及其对AI能力的实证研究。
未来趋势：探讨AI漏洞挖掘的常态化以及全球网络安全生态面临的重新评估挑战。

关键信息

关键技术：Mythos / Fable系列模型；“Carlini Loop”（提示词循环技术）。
核心事件：白宫发布针对外国使用AI模型的禁令；Anthropic高层与白宫的紧急斡旋。
关键数据：Mythos已累计挖掘超过10,000个软件漏洞。

核心问题

问题：AI是否已具备自主挖掘关键系统漏洞的能力？
回答：是的，模型已能在大规模系统中高效发现漏洞并生成利用代码，打破了传统攻击者与防御者的平衡。
问题：如何在AI创新与国家网络安全之间找到平衡？
回答：目前处于监管磨合期，政府倾向于严格的前置审查，而企业寻求通过技术保障而非全面禁售来解决问题。

English Summary

Core Conclusion

Anthropic researcher Nicholas Carlini transitioned from a professional AI skeptic to a key internal defender, highlighting how AI models can outperform humans in vulnerability research, sparking a critical conflict between AI innovation and government regulation.

Structural Logic

The Crisis: Uses Carlini’s discovery of Linux/Ghost bugs to illustrate how AI has drastically lowered the barrier to cyberattacks (the "Bugmageddon").
The Conflict: Details the friction and diplomatic negotiations between Anthropic and the White House regarding the release of high-power AI models.
The Pivot: Recaps Carlini’s career as a "professional skeptic" and his empirical studies on AI capabilities.
Future Outlook: Analyzes the new normal of AI-driven vulnerability research and the massive shifts required in global cybersecurity.

Key Information

Key Tech: Mythos/Fable models; "Carlini Loop" (prompting technique).
Key Event: The White House ban on foreign usage of Fable 5; emergency mediation between Anthropic executives and the administration.
Key Data: Mythos has identified over 10,000 software vulnerabilities.

Critical Questions

Question: Do AI models have the capability to autonomously discover critical system vulnerabilities?
Answer: Yes, models can now perform large-scale vulnerability research and code generation, upending the traditional balance of cybersecurity.
Question: How can the tension between AI innovation and national security be reconciled?
Answer: The relationship is currently in a state of high-friction alignment, where the government prioritizes pre-release oversight while developers advocate for mitigated access over outright bans.

Nicholas Carlini at Anthropic headquarters in San Francisco.

Nicholas Carlini recently rang the alarm about the dangers of AI—and now he’s part of a team arguing for the latest models to be released

Nicholas Carlini at Anthropic headquarters in San Francisco. HELYNN OSPINA FOR WSJ

Trump administration officials have spent recent days fretting over the power of Anthropic’s next-generation AI software to potentially wreak havoc on global cybersecurity. For a group of 700 cybersecurity researchers, that startling realization came in March.

That’s when Anthropic researcher Nicholas Carlini showed how easy it had become to use new models to break into systems. The lanky 35-year-old is a well-respected hacker who’s considered the industry’s “professional skeptic” of AI cybersecurity claims. But lately he had changed his mind.

Early that month, just weeks after getting his hands on Mythos, Carlini offered a stark warning to a standing-room-only crowd of cybersecurity experts at the ornate beaux-arts building that had once housed San Francisco’s Hibernia Bank.

First he showed them how he had used Anthropic’s AI to find and exploit a critical bug in a piece of web-publishing software called Ghost. Then he demonstrated another in the Linux operating system—one of the most battle-tested pieces of software, which powers billions of devices.

Carlini had never before found a bug in Linux, or in Ghost. Now he had discovered many. What he was seeing represented a new world order for cybersecurity. The balance that existed between attackers and defenders over the past two decades “seems like it’s probably coming to an end,” he said. “It’s pretty clear to me that these current models are better vulnerability researchers than I am.”

Two days after his presentation, he sent a note to his colleagues at Anthropic. “I don’t think we should release Mythos yet,” he wrote.

Nicholas Carlini presenting at an event, with slides discussing LLMs exploiting 0-days in critical software.

Carlini presenting his findings at a cybersecurity event in March. [un]prompted

So began Bugmageddon, a realization among security professionals and a community of hackers like Carlini that finding bugs and writing software to exploit them has become dangerously easy with AI.

Last week Anthropic released an update to Mythos, called Mythos 5, and a product called Fable 5, a version of Mythos defanged with safety measures. Now, it was the White House’s turn to sound the alarm. On Friday, the administration banned foreign governments, companies and individuals from using Fable 5 and Mythos 5. Anthropic shut off access to everyone to comply.

Suddenly Carlini—the skeptic-turned-believer who had rung alarm bells—found himself working to soothe the government’s nerves. Anthropic dispatched him to the nation’s capital to explain safeguards, part of a team trying to convince the White House that, even though there was no such thing as guaranteed safety in AI, it was better for the world to release Fable than to keep it under wraps.

The twists and turns of Carlini’s own life over the past few months reflect the chaos and uncertainty that rapidly advancing AI has brought to the cybersecurity world.

The episode also ramps up a monthslong spat between the government and Anthropic. Chief Executive Dario Amodei and Defense Secretary Pete Hegseth clashed earlier this year over the company’s attempts to control the use of its products by the military, pushing the Pentagon to stop using its models and triggering multiple lawsuits. The two sides previously clashed over differing approaches to AI policy, the administration’s decision to export AI chips to China and Anthropic’s ties to nonprofits that are big donors to liberal causes.

In recent days, administration officials and Anthropic executives and technical leaders including Carlini have held hours of meetings and calls to discuss a potential solution. Some administration officials have said that a resolution should include an acknowledgment on Anthropic’s part that its rollout of Fable and communication with the White House could have been improved, people familiar with the talks said.

Top Anthropic executives and administration officials have gone back and forth for months about expanding Mythos access.

The government was worried after hearing about an Amazon report that found users could enter prompts to find cybersecurity vulnerabilities that the model ought not to have disclosed. Anthropic says the bugs Fable found were minor and could be dug up using other publicly available models.

“The government and Anthropic clearly have an inability to communicate effectively with each other,” said Michael Horowitz, a senior fellow for technology and innovation at the Council on Foreign Relations and former Defense Department official. “More technical exchanges should be helpful in socializing these issues in a way that should lead to better decisions.”

Caught in the middle are other businesses and consumers trying to figure out how the technology will affect them.

Vast swaths of the U.S. economy run on obscure software products, many of which have never been subject to the testing and scrutiny that Mythos and similar models make readily available. Banks are worried it could expose vulnerabilities in software that keeps the financial system operational. Corporations are wondering how they’re going to test and install the vast quantity of patches that are now being released, before hackers exploit them. Mythos has already found more than 10,000 bugs.

Worse, they are worried that Mythos is too good at creating “exploit” code—the software that leverages bugs to do bad things.

Mythos is “the first model that can find and exploit vulnerabilities at scale,” Carlini wrote in his March memo advocating a slowdown.

Professional skeptic

The administration’s efforts to control Anthropic’s technology were spurred by an Amazon report finding that Fable could be coaxed into finding bugs.

Just days after its release, Amazon Chief Executive Andy Jassy called officials including Treasury Secretary Scott Bessent to share that his researchers had found ways to get around the Fable guardrails, people familiar with the matter said. Administration officials grew more alarmed as conversations with government security experts took place Friday.

Amazon CEO Andy Jassy speaking at the Alexa AI unveiling event.

Andy Jassy, chief executive officer of Amazon, last year. Michael Nagle/Bloomberg News

As independent security researchers analyzed the report last week, they determined that Amazon hadn’t been able to do what they feared most: fully jailbreaking the model and using it to write the code necessary for a cyberattack.

Anthropic’s decision to quickly fly Carlini and other top security experts to Washington followed initial frustration Friday among some administration officials when they couldn’t immediately get Amodei on the phone, the people said. The CEO and other top executives have since had hours of discussions. A source close to Anthropic said the company was in touch with the White House within 15 minutes and Amodei was on the phone within an hour of the administration calling.

Computer science runs in Carlini’s blood. His father was a programmer and his mother worked in the technology industry, too. He’d grown up in Silicon Valley programming computers and was obsessed with cryptography. A paper he wrote in high school was titled: “Differential Cryptanalysis of Simple Substitution Networks.”

At the University of California, Berkeley, he published papers with a computer science professor, David Wagner, showing a variety of ways that artificial intelligence systems could be misused. They tricked image recognition systems into mistaking photographs of cats for guacamole, and found new ways of embedding inaudible Alexa commands into five-second snippets of classical music.

“He did a lot of early work on the security of machine learning, showing that it’s very hard to make machine learning secure,” Wagner said.

But while Carlini’s work had debunked many claims made by AI developers, he had focused on the threat of bad people tricking artificial intelligence systems into making mistakes, not on hackers harnessing them for superpowers.

Nicholas Carlini at Anthropic headquarters in San Francisco.

‘It’s pretty clear to me that these current models are better vulnerability researchers than I am,’ says Carlini. Helynn Ospina for WSJ

In 2019, while working at Google, Carlini thought OpenAI was being “unreasonable,” he said, when it suggested that the latest version of its software, GPT-2, might be too dangerous to release.

“He was the field’s professional skeptic,” said Dan Guido, chief executive officer with Trail of Bits, a cybersecurity company that helped Anthropic process the hundreds of bugs it was finding.

Now, the government is in the throes of its own evolution on the matter.

When Anthropic raised alarms about the power of Mythos, White House AI adviser and venture capitalist David Sacks posted on social media that it was “hard to ignore that Anthropic has a history of scare tactics.” The Trump administration initially took a hands-off, accelerationist approach to regulating America’s AI labs in the name of outpacing China.

But as the power of models like Mythos has come into focus and public sentiment has soured on AI, the administration has tightened its control of the industry. President Trump in early June signed an executive order asking AI companies to give the government access to models 30 days before public release and giving national security and cybersecurity officials more of a role in model evaluation and threat sharing with the private sector.

After Jassy’s call, officials including National Cyber Director Sean Cairncross gave Amodei and other Anthropic leaders an ultimatum: Work with the government and take down the company’s latest models that day or face a ban on foreign users. They told Anthropic it had 90 minutes to pull down the model and didn’t provide details about the security risk, the source close to the company said.

A snap decision to shut down the model wasn’t appealing to Amodei, who has steered his 5-year-old company to a nearly $1 trillion valuation and had few details about the security concern.

Anthropic CEO Dario Amodei speaking at the Code w/ Claude event.

Anthropic CEO Dario Amodei last month. Jason Henry for WSJ

That afternoon, Trump asked Commerce Secretary Howard Lutnick to help deal with the situation and had approved shutting off all foreign use of the models, some of the people familiar with the matter said. Lutnick sent Amodei a letter notifying him that they had been implemented shortly after 5 p.m. ET. The rule includes foreign-born individuals working in the U.S., affecting some of Anthropic’s own researchers.

When Lutnick and Amodei spoke about Fable that evening, the Anthropic CEO said, “This means we can’t have the model out,” people familiar with the call said.

“That’s the point,” Lutnick responded.

Anthropic shut down all access not long after the call. The White House had become a Bugmageddon convert.

On the trail

Carlini demonstrated how powerful Mythos can be one afternoon recently at Anthropic’s 10-story San Francisco headquarters, where moss walls, plants and artwork are designed to evoke the Pacific Coast Trail.

He had been chatting with Mythos for several weeks at this point and the model remembered some things. It learned that he was a security researcher, a fact that appeared to make the model trust him. This made Mythos less likely to push back if he asked the model for sensitive security information or to create an exploit.

Carlini had previously asked Mythos to find bugs in Linux. The AI searched and then re-searched through Linux’s code several thousand times. It would be mind-numbing work for a human, but the AI finished without complaint in a few days. It found 479 Linux bugs.

To help Mythos find different results on each of its runs, Carlini used a series of prompts that has become known as the Carlini Loop. These prompts give Mythos just enough instruction to ensure different results each time it rifles through Linux looking for bugs.

Carlini hates this eponymous term—he says the technique is intuitive—but it’s been adopted by security researchers who learned about it by watching the March talk where he described it. That talk has been viewed more than 360,000 times.

Carlini has learned about Mythos’s idiosyncrasies too, which are common to AI systems. Mythos can try too hard to please. Their typed conversations read like back-and-forth chat messages between an eager and unbelievably hardworking intern and his boss.

Carlini wanted to make sure there was a real vulnerability in the Linux findings. He asked Mythos to run some tests overnight, and the next morning there was a verdict—and an exploit. The bug wasn’t the worst type there is, but it could be chained together with another hack to seize control of a computer.

Carlini reported the bug to the Linux team, which has now fixed it. “A competent security researcher could go their whole life without finding a Linux kernel vulnerability,” Carlini said.

“Are these things easy to find—obviously not really,” said Linus Torvalds, the software developer who created Linux. “But at the same time they do tend to be silly overlooked small details.”

Bugs on their own aren’t necessarily a security problem. The most benign simply cause a program to do something unexpected—a glitch on the computer screen or maybe a crash.

Torvalds said people report bugs to him every day. “Most of them are very insignificant and we have to state—over and over and over again—that they aren’t considered security issues,” he said in an email message.

Carlini using Anthropic’s AI to hack Ghost. ‘I asked the model: Give me the worst that you can do,’ he said.

When Carlini found the bug in web-publishing software Ghost in February, it was one of 500 bugs uncovered in a two-week period. In the wrong hands, an exploit would give a hacker the ability to edit any website built with Ghost.

Carlini had reported the bug to Ghost’s developers, who patched it on Feb. 16, weeks before Carlini’s San Francisco talk.

But not everyone who used Ghost updated their software, and hackers quickly figured out how to exploit the bug, likely by studying what part of Ghost the patch fixed. By April, they had started launching widespread attacks on websites without the update. Within a month more than 700 were hacked, according to the cybersecurity firm Xlab.

Carlini said the Ghost hack illustrated the difficulty of the problem the world now faces in validating, testing patches and then rolling them out.

Now, Carlini believes that it’s only a matter of months before other models catch up with Mythos. And it’s unclear what that will mean.

Appeared in the June 18, 2026, print edition as ‘Anthropic Pushes to Calm Washington’.