AI 2040 Mistakes Technical Legibility for Political Feasibility
Please see relevant disclosures here.
AI 2040 is best understood as an aspirational design exercise, not a plausible geopolitical scenario. Its authors largely acknowledge this: Plan A is described as a recommendation rather than a forecast, individual authors assign it only a 3–15 percent chance of occurring, and their own analysis estimates roughly a 48 percent chance that the agreement would collapse or become seriously impaired within ten years. These admissions make the document more intellectually honest, but they also expose its central problem. A scenario intended to “stress-test” a policy cannot simply stipulate the political transformation on which that policy depends.
The document’s fundamental mistake is to confuse a shared interest with an incentive-compatible bargain. The United States and China could both reasonably fear uncontrolled artificial intelligence, military instability, or catastrophic loss of control. But common danger does not resolve disagreements over relative power. The relevant questions are who retains the technological lead, who receives access to frontier research, who sets capability thresholds, who inspects military-linked infrastructure, and how gains are distributed after a slowdown.
Plan A avoids this bargaining problem by giving China much of what would make an agreement attractive: access to algorithms, time for Chinese firms to catch up, and an eventual position near the technological frontier. Yet those are precisely the concessions most likely to make the agreement unacceptable in Washington. Conversely, a deal that preserved a decisive American advantage would appear to Beijing less like arms control than technologically enforced containment. Verification cannot create a bargaining range where the distribution of benefits remains unacceptable to one side.
The document also substitutes an “awakening” theory of politics for an account of national interests. Its FAQ suggests that the Chinese Communist Party does not yet appreciate the importance of AGI or superintelligence but will revise its position once the stakes become evident. This treats disagreement with the authors’ threat model as an information deficit. Chinese leaders might fully understand the hypothetical risks and still prefer a less capable but sovereign AI ecosystem to a more capable international consortium that exposes sensitive research, constrains military development, and weakens Party control.
China’s actual incentives are more complex than fear of being overtaken. The Party views technological self-reliance as essential to regime security, economic development, military modernization, and national rejuvenation. It also sees AI as a means of managing social stability and strengthening state power. Official Chinese policy combines support for international AI safety with strong commitments to sovereignty, development rights, indigenous capacity, and nationwide resource mobilization. The document’s claim that China is not an actor “in whom power would concentrate by default” makes sense only if power means control of the global technological frontier. Domestically, it ignores the possibility that AI could further concentrate power in the Party-state.
The American side is similarly flattened. Plan A imagines an alarmed president choosing a strategy and then rapidly compelling frontier laboratories, reorganizing compute infrastructure, negotiating reciprocal inspections, and binding the country to a decade-long international regime. But the United States is not a unitary executive machine. Laboratories and datacenters are privately owned; Congress controls legislation and appropriations; courts adjudicate property and constitutional claims; agencies possess overlapping authority; states influence energy and construction; and formal treaties require Senate consent. An election mandate is not an administrative apparatus.
This exposes the document’s deeper misunderstanding of state capacity. State capacity is not a general reservoir that governments unlock when sufficiently frightened. It is specific to particular tasks. China possesses considerable capacity to finance infrastructure, direct firms, allocate land and power, and mobilize industrial production. It is much less well suited to radical transparency, independent auditing, truthful disclosure of military-linked activity, and externally credible limits on Party discretion. The United States has stronger legal institutions, private technical expertise, capital markets, and alliances, but much less ability to direct firms and physical infrastructure rapidly.
Plan A effectively assigns both countries the strongest characteristics of both systems. America suddenly acquires Chinese-style command over industry, while China acquires American-style auditability and institutional openness. The document tries to solve inadequate government expertise by outsourcing analysis to transparent third-party evaluators. That may reduce the technical knowledge required inside an agency, but it does not create legal authority, procurement competence, adjudication procedures, institutional legitimacy, or enforcement capacity. It solves part of the state’s knowledge problem while leaving its principal-agent and commitment problems untouched.
The proposed “trustless” verification system therefore rests on a category error. Monitoring can help determine whether an observable obligation has been violated. It cannot determine why either state should continue accepting the agreement as technology, leadership, and relative power change. Nor is verification genuinely trustless. The parties must still trust baseline compute declarations, monitoring hardware, inspectors, classification rules, evidence standards, dispute-resolution bodies, and the other side’s willingness not to exploit inspections for intelligence collection.
This is especially problematic because Plan A seeks declarations and retrofits covering approximately 99 percent of global AI compute. The project’s own verification assessment concedes that many essential systems do not yet exist and that core work on physical inspection, side-channel protection, reporting integrity, and large-scale monitoring is not started or not on track. The scenario nevertheless moves from immature prototypes to reciprocal inspections, near-total compute accounting, and worldwide enforcement in approximately three years.
Taiwan and other third parties are also treated as peripheral complications when they are constitutive of the problem. Taiwan, South Korea, Japan, the Netherlands, the European Union, cloud providers, equipment manufacturers, and host governments all possess independent interests and leverage. They would bargain over sovereignty, technological access, exemptions, security guarantees, and industrial rents. The assertion that worldwide participation would be “surprisingly easy” merely assumes away another set of political conflicts.
The proposal to place Chinese datacenters in Canada and American datacenters in Mongolia illustrates the broader unreality. These countries are treated as strategically convenient geography rather than sovereign actors. Deliberately vulnerable infrastructure is then presented as “mutually assured compute destruction.” But vulnerability does not necessarily produce stability. It can create incentives for preemption, seizure, sabotage, or rapid destruction during a false alarm. Unlike survivable nuclear second-strike forces, these assets would be valuable economic hostages exposed to use-it-or-lose-it pressures.
None of this means that AI cooperation with China is impossible. Large datacenters, advanced chip production, and power infrastructure are more observable than ordinary software development. Reciprocal notifications, managed access to declared civilian clusters, common evaluations of specific catastrophic capabilities, crisis hotlines, human control over nuclear-use decisions, and conditional export-control relief could all be worthwhile. Verification research and stronger domestic regulatory capacity are sensible investments.
But these measures point toward a gradual, layered arms-control architecture, not an immediate global grand bargain. A credible scenario would begin with narrow obligations, limited inspections, reversible concessions, third-party dispute resolution, withdrawal periods, and domestic legislation capable of surviving leadership turnover. It would treat Plan A as a distant possible endpoint rather than something that appears once political leaders finally understand the authors’ argument.
The central flaw of AI 2040 is that the document treats geopolitics as a disturbance surrounding a technical governance regime. In reality, geopolitics would constitute the regime itself. The difficult problem is not designing monitoring devices after cooperation has been achieved. It is constructing institutions that remain acceptable to rival states, competing domestic coalitions, private firms, allies, and future governments while the underlying distribution of power is changing. On that question, the scenario offers aspiration in place of analysis.
If you enjoy this newsletter, consider sharing it with a colleague.
I’m always happy to receive comments, questions, and pushback. If you want to connect with me directly, you can: