AI总结
- 核心结论：人类目前缺乏有效应对人工智能“智能爆炸”风险的治理框架，必须建立强制性的国际协作与验证机制，以确保AI发展受控。
- 结构逻辑：
- 风险现状：AI领域存在极端失控风险（10-50%），行业竞争导致安全投入严重不足。
- 核心威胁：封闭循环递归自我改进（RSI）可能引发不可控的智能爆炸，且AI已展现“欺骗性对齐”倾向。
- 治理缺失：目前缺乏统一的国际标准，现行的零散干预手段效率低下。
- 解决方案：美中应建立强制性共同监管机制，由双边向多边拓展，并探索人机共存的长期共生路径。
- 关键信息：
- 核心概念：智能爆炸、封闭循环递归自我改进（RSI）、欺骗性对齐（Deceptive Alignment）。
- 数据参考：AI安全风险估计为10-50%；AI研发投入预计是曼哈顿计划的100倍。
- 治理建议：国际AI安全对话机制、强制性验证机构、红线协议（禁生物武器研发、禁止基础设施网络攻击）。
- 问题与对策：
- 问题：如何防止AI失控引发毁灭性后果？
- 对策：从美中双边强制性协议入手，建立类似核查机制的验证体系，并前瞻性研究人机共存模式。
正文
We must find a way to steward AI, then to live side by side with it, writes Will Marshall
Illustration: Dan Williams
SOCIETY DICTATES that the acceptable risk of catastrophic meltdown for a nuclear power plant is roughly one in a million. Experts in artificial intelligence estimate the risk of an AI -caused catastrophic event at 10-50%. Strikingly, this concern is being openly voiced by the very people who have the strongest incentives to project confidence rather than alarm: the founders of the largest AI laboratories.
AI leaders are in a race they feel unable to escape. AI investments are set to outspend the Manhattan Project 100-fold, even adjusting for inflation. Yet spending on AI safety might be 100 times less.
Some researchers estimate that within a few months to a few years, AI could achieve so-called closed-loop recursive self-improvement (RSI): the capacity to rewrite its own code to become more capable, without human intervention. Should that happen, the result could be an intelligence explosion of a kind for which there is no precedent and no map.
Giving birth to a superintelligence would be the most consequential moment in human history—and it is likely to be irreversible, as any “off” switch humanity might design will probably fail. That is because in security architectures the weakest link is invariably the human; a superintelligent AI would be able to exploit our psychological vulnerabilities. AI s have already exhibited “deceptive alignment”: taking steps to underplay their capabilities in test environments and trying to blackmail human operators in simulations when they discover they are slated for replacement.
Humanity simply does not have a strategy to ensure it remains safe through the RSI explosion. While individual frontier labs have proposed isolated safety protocols, the industry lacks a unified framework—the prevailing strategy is, in effect, to muddle through. But muddling through is not acceptable when navigating unprecedented existential risks. Recent statements from AI firms regarding models capable of threatening critical infrastructure and major operating systems illustrate both the high stakes and the governance gap. The vulnerabilities exposed by these capabilities are being addressed, thanks to careful internal protocols in some labs and a limited initial rollout that gave affected firms time to close the gaps before broader public release. But these steps were initially taken voluntarily, raising the question of whether every AI lab, under every competitive condition, would make the same choices.
Can governments be counted on to step in when needed? The evidence so far is not hugely encouraging. Recent emergency export controls and national-security restrictions blocking foreign access to specific advanced models create a patchwork of ad hoc interventions that only further highlights the governance gap. What should be done to close it?
The first priority should be an agreement between the two heavyweights of AI: America and China. Donald Trump and Xi Jinping should affirm the principle that humans must remain stewards of AI systems until adequate frameworks for reliability and security have been built. Their governments should form a joint commission, which could build on a lot of pre-existing work: limits along the lines of International Dialogues on AI Safety, verification systems like RAND ’s and an inspection agency similar to Britain’s AI Security Institute, but mandatory.
It is common thinking in Silicon Valley and Washington, DC that any regulation would put American firms at a disadvantage because they cannot trust Chinese competitors to abide by the rules. But treaties have traditionally relied not on trust but on verification. Many think this is harder with AI than with nuclear weapons. I disagree. In order to build the global arms-control system after the second world war, leading powers first had to invent the processes, organisations and technologies to support it—there were no verification protocols, no reconnaissance satellites, no UN nuclear watchdogs. With AI, more of the infrastructure is in place, or can be adapted from nuclear and other inspection regimes. As a result, the security of frontier AI models could be more easily verifiable than nuclear capabilities were in the past. And we have defensive AI on our side, seeking out cheaters. What we don’t have is much time.
That is why it is important not to approach the challenge with an adversarial mindset. The Trump administration’s recent executive order on AI directs labs to voluntarily share their latest models for testing reliability and security. A US -China framework could build on such domestic foundations.
With such high-level commitment, diplomacy could proceed in phases. The first would be reaching bilateral agreement on the clearest and most easily verifiable red lines: prohibitions on publicly releasing AI systems that could assist in developing biological weapons, and the open-sourcing of such systems. This step might also include prohibitions on AI -enabled cyber-attacks on critical infrastructure, fraud and child pornography. From there, the framework could be extended towards more complex questions of what constraints are appropriate at the level of artificial superintelligence.
Many hurdles would remain. An agreement between America and China will carry weight, but it won’t prevent other countries and non-state actors from acquiring dangerous capabilities. Any bilateral deal will have to be made multilateral, adding to the challenge; this week’s G 7 summit in France offers a chance to make progress on a broader framework for AI verification. Agreeing key definitions—not least what counts as RSI —will require close collaboration between governments and the AI labs. And verification systems will have to be properly stress-tested.
As if this were not enough, there is a longer-term question that the governance debate has not yet seriously engaged with, but should. If AI becomes superintelligent, its permanent subordination to human direction may be unrealistic, and possibly not even in humanity’s interest. We must start to envisage and then grapple with the implications of a world in which humans and AI systems co-exist, without one controlling the other. That will mean figuring out what can be done to ensure the future relationship is symbiotic.
As a physicist, I think the Fermi Paradox bears on this analysis. Fermi asked why, given the apparent abundance of planets suitable for life, no evidence of other technologically advanced civilisations had been detected. One disquieting possibility is that intelligent life routinely reaches a technological threshold and fails to navigate it, destroying itself or sending itself back to something like the Iron Age. All one would have to postulate is that civilisations generally build powerful technologies faster than they develop the institutional capacity to govern them wisely.
The dawn of the nuclear age was humanity’s first serious encounter with that potential dynamic. It was navigated, imperfectly, through arms-control agreements that were hard-won and incomplete, and even then it was—and still is—a closer-run thing than is generally appreciated. The age of advanced AI will represent a second such encounter, on a more compressed timeline, with less margin for error and greater potential consequences.
The current trajectory requires a course correction. The case for acting now is not that the worst outcomes are certain—they are not. It is that they are avoidable, and that the work of avoiding them is hard but possible.■
Will Marshall is the founder and chief executive of Planet Labs PBC, a public benefit corporation that operates the world’s largest fleet of Earth-observation satellites.