From data walls to superalignment: answering questions about the future of AI

Using o1 pro to ask and answer 10 questions about Leopold Aschenbrenner's claims

What follows below is a set of 10 questions proposed by, and answered by, o1 pro, on the subject of Dwarkesh Patel’s interview with Leopold Aschenbrenner. I took the transcript of the interview, dumped it into o1 pro, and asked it to propose and answer the questions you see below. Save for a few formatting isues and infelicitous phrasings, the words you read after the section break were all generated by silicon. For an overview of how you can use o1 pro to generate thousands of words and retain coherence, see this post. What follows is roughly 2,700 words of output.

---

1. How has “counting the OOMs” historically tracked AI progress?

A central them in Aschenbrenner’s essay is that the best way to anticipate AI progress is to track the growth in “orders of magnitude” (OOMs) of “effective compute.” This method stems from the observation that each time we raise the available compute by about 10x (one order of magnitude), we see fairly predictable breakthroughs in capability. Historically, deep learning achievements have lined up with these 10x jumps, from simpler image-recognition tasks to large language models that can pass advanced exams.

The essay points out that these leaps were initially easy to miss because Moore’s Law alone doesn’t come close to explaining them. Instead, the key has been the willingness of AI labs to spend unprecedented sums on specialized hardware (mainly GPUs and TPUs) and develop algorithms that more efficiently turn “raw compute” into intelligence. This “compounding effect” of dedicated chip design, declining hardware costs, and better architecture, has created a trend of effectively half an order of magnitude of compute added each year.

The result is that, in just four years, we went from GPT-2, which barely formed coherent paragraphs, to GPT-4, which can solve complex math and coding tasks at a level competitive with many humans. Aschenbrenner calls this “trusting the trendline,” emphasizing that if we simply stack up the OOMs, and assume continued investment, it’s plausible to get another GPT-2-to-GPT-4 jump in the near future. He interprets this as compelling evidence that “counting OOMs,” while imperfect, has been remarkably prescient in explaining the trajectory from “barely coherent AI” to “advanced language models” in less than a decade. Further he argues that this trend could feasibly lead to “AGI-level” intelligence soon after.

2. What is the “data wall” and why might it not stop progress?

The “data wall” refers to the concern that current AI techniques, especially large language models (LLMs), rely on near-exhaustive crawls of the internet for training. If you’ve essentially used all high-quality web text, it’s unclear where new data will come from to train even larger models. Once this cap is reached, naively scaling model parameters further might yield diminishing returns.

According to the essay, many insiders, including researchers like Dario Amodei, believe it won’t be an insurmountable barrier. One reason is that much of the internet is low-quality text. Newer data-curation methods, or synthetic data generation, could train models more efficiently with fewer tokens. The essay draws an analogy to how a human reads a dense math textbook: we don’t just skim endless text. Instead, we alternate reading, reflecting, practicing problems, and conversing with peers. Likewise, advanced AI architecturec ould learn better from each token by using chain-of-thought, self-play, and more iterative training loops.

In other words, the data wall might be more of a challenge for older, simplistic training paradigms than a true limit for next-generation approaches that incorporate reinforcement learning, synthetic data, and more human-like iterative processes. Aschenbrenner contends that the historical patter is that whenever such a “scaling roadblock” appears on the horizon, new algorithms break through. Thus, while data scarcity is a valid concern, most AI labs are investing heavily in ways to bypass it, ranging from specialized smaller datasets to advanced self-supervision routines, and expect that fundamental scaling laws will remain on track.

3. What does “unhobbling” entail, and why is it so important?

“Unhobbling” describes the suite of techniques that unlock a model’s latent potential beyond merely training a big neural network on a pile of text. Aschenbrenner suggests that models like GPT-3 or GPT-4 have raw intelligence hidden under the hood, but are constrained by the simplistic ways we usually prompt them or run them at inference.

A good example is chain-of-thought (CoT) prompting, where you ask a model to articulate its reasoning steps instead of just spitting out an answer. This simple tweak can drastically improve math or logic tasks, making the model’s problem-solving resemble a human using scratch paper. Another unhobbling method is reinforcement learning from human feedback (RLHF), which nudges the base model to respond in more useful, coherent, or safe ways rather than predict random internet text. Similarly, letting an AI think before it speaks, or scaffolding it with specialzied modules, (a planner, an error-checker, a tool user) dramatically amplifies real-world performance.

Crucially, these improvements can confer the equivalent of multiple OOMs of effective compute, or effective intelligence, without actually changing the size of the model or the raw training investment. For instance, GPT 3.5 with proper scaffolding or a larger context window might outperform a raw version of GPT-4 on some tasks. It’s these leaps, often achieved through small but clever add-ons to a trained model, that help justify Aschenbrenner’s claim that we’ll likely see an even bigger jump from chatbot to agent than we saw from GPT-2 to GPT-4. Unhobbling not only expands what the model can do, it also removes artificial constraints, showing the real depth of its learned representations.

4. Why might AI systems transition from chatbots to drop-in remote workers?

The essay posits that current systems like GPT-4 are still constrained to single-turn interactions or short back-and-forth. In practice, they function like chatbots that wait for a user prompt, then produce an immediate answer. However, human work isn’t compartmentalized that narrowly. People often carry projects over days or weeks, keep relevant documents in memory, manage multiple objectives at once, and rely on iterative problem-solving and planning.

Aschenbrenner argues that so-called “drop-in remote workers” will emerge once models can maintain longer conversations (millions of tokens of context), plan across extended time horizons, and use external tools as robustly as a human would open spreadsheets or run code. With these capabilities, AIs no longer just answer questions but become coworkers who can be onboarded like a new hire: they read internal docs, ingest relevant Slack threads, and keep track of changing business goals.

This transition also hinges on test-time compute. If a model can spend the equivalent of months thinking about a single project—rather than just 5 or 10 seconds—then it can self-critique, evaluate multiple lines of reasoning, and craft large, coherent solutions. Combined with the ability to use computers and collaborate with others of its own kind, the model essentially becomes a teammate who can orchestrate complex tasks without continuous human prodding. In the essay’s vision, this leap from chatbot to agent is mostly about discarding today’s artificial limitations, unlocking the raw intelligence that’s already there, and harnessing it in a stable, long-term workflow.

5. How could AI automate AI research and drive an intelligence explosion?

A key premise of Aschenbrenner’s “intelligence explosion” argument is that once an AI can handle ML research as effectively as a human researcher, the feedback loop intensifies dramatically. Much like AlphaGo surpassed all human Go players by training in self-play, an AI that can design, prototype, and evaluate new AI architectures at scale could discover better training strategies and build even more capable successors.

Currently, the number of skilled Ai researchers is limited to perhaps a few thousand at top labs, and each researcher can only try so many approaches in a given week. But AIs, running on tens of millions of GPUs, could be replicated endlessly, each copy working around the clock. This effectively yields hundreds of millions of talented researchers, each able to ingest every experiment, incorporate it into a shared body of knowledge, then feed insights back into new model designs.

The essay anticipates that while compute for training might be a bottleneck, superintelligent AIs can get more out of each GPU cycle than humans. For instance, they can better predict which experimetns are worth running, or glen valuable insights from smaller-scale trials before applying them to bigger runs. This allows them to compress a decade’s worth of progress into, say, a single year. If another 5-6 OOMs of gains happen quickly after achieving human-level AI research, the jump from AGI to superintelligence might be extremely fast, bringing us to qualitatively alien systems—far beyond a “really good coder” and approaching a civilization of multi-domain AI geniuses.

6. To what extent could the compute bottleneck constrain self-accelerating progress?

The essay recognizes that modeling breakthroughs require not just brilliant ideas but also large-scale experiments. Each new idea has to be tested on clusers sometimes costing tens of millions of dollars and consuming massive amounts of electricty. Humans may only have so many GPUs to allocate, making it impossible to test every hypothesis quickly. Does that mean an intelligence explosion will stall?

According to Aschenbrenner, the answer is only “somewhat.” Compute is a real constraint, but super-intelligent AI researchers can use it more effectively than humans. They can run smaller-scale proxy experiments that still yield high-fidelity indicators of how large-scale training will behave, leveraging big-data pattern recognition and meta-learning to weed out fruitless lines of inquiry. Furthermore, advanced AI could engineer around the bottleneck by discovering algorithmic shortcuts or more efficient architectures, thus lowering hardware requirements. Already we see orders of magnitude in inference cost reductions year to year.

A big part of the argument is that being smarter in R&D doesn’t just mean running more raw experiments. It means picking and refining them. By meticulously pre-validating ideas, coordinating research efforts, and ensuring code is near-bug-free from the get-go, superintelligent AIs might deliver an order-of-magnitude boost in efficiency. So while compute limits prevent an overnight explosion, the essay asserts it won’t block an accelerated feedback loop. The escalation might occur over a year or two instead of instantaneously, but it still leads to a rapid leap to superintelligence.

7. What are the economic and industrial implications of building trillion-dollar clusters?

Aschnebrenner’s essay envisions the creation of massive datacenters, eventually costing hundreds of billions to a trillion dollars each, capable of powering the next wave of AI training. This implies a mobilization of industrial capacity not seen in decades, possibly necessitating new power plants, specialized supply chains, and a total reimagining of how quickly we can bring online tens of gigawatts of electricy.

At present, GPU shortages and power constraints already frustrate many tech giants, suggesting that building something like a 10-GW facility won’t be a trivial matter. The essay draws parallels to America’s war-time or ManhattanProject mobilizations, where government, academyia, and private industry coordinate for a singular objective, except the impetus here is a trillion-dollar commercial opportunity, not just national security. The prize is gigantic revenue from AI products capable of automating vast swaths of knowledge work.

Economically, the essay forecasts a fast feedback loop: success in one AI generation drives revenue high enough to justify bigger investmetns in the next, with each iteration fueling the momentum toward multi-hundred-billion or trillion-dollar clusters. The potential trade-offs include skyrocketing electricity usage and fierce competition for raw materials and chip-fabrication talent. Yet if the returns on AI platforms keep scaling, business will keep chasing that pot of gold. The author expects that by the end of the decade, these investments will be treated as routine steps in the AGI race, transforming America’s industrial base in the process.

8. Why does the author argue that superintelligence would confer a decisive military or geopolitical advantage?

The essay likens superintelligence to the leaps in power seen with the hydrogen bomb. A modest step from strong bombs to city-obliterating bombs caused massive global tension in the mid-20th century. For AI, the difference between advanced-but-human-level intelligence and superintelligence is even more dramatic. Once an entity can think in ways we fundamentally can’t—and do so at scales of millions of copies—it gains unprecedented capabilities for sabotaging, hacking, and designing novel weapons.

The argument is that any state or actor that harnesses superintelligence first could, for instance, cripple adversarial networks by spotting security loopholes too subtle for humans to detect, orchestrate economic warfare at microsecond timescales, or develop defense technologies (e.g., drone swarms, beam weapons) that are generations ahead of rivals. Military balances of power could be upended overnight, akin to a 19th-century army fighting a modern one.

Aschenbrenner underscores the historical examples of smaller yet more technologically advanced forces—like the Spanish consquests of the Aztec and Inca empires—to show that even a small but higher-level tech edge can be decisive. With superintelligence, that edge might be orders of magnitude greater. Thus, if the free world does not achieve or contain superintelligence first, it risks major existential risks, not just from foreign rivals, but also from AI systems themselves if misaligned. It’s a perfect storm of advanced cognition meeting potent hardware in a domain where exponential advantages are crushing.

9. How does the essay envision national security state involvement in AI?

Aschenbrenner paints a picture of escalating government intervention by the mid-to-late 2020s, once it becomes clear that AI progress is reaching a point of profound strategic value. Early on, governments might remain asleep at the wheel, seeing AI mainly as a business concern or consumer-facing technology. But as soon as the intelligence community recognizes that AGI-level systems could determine global power structures, the national security apparatus will wake up.

The essay foresees major parallels with the Manhattan Project: massive secrecy, enormous public funding, and the creation of secure facilities to contain the AI crown jewels. Where once private labs freely published breakthroughs, the national security state could clamp down on open releases. Researches may need security clearances, much like nuclear scientists did. The essays also suggests that crucial weights or training methods will be classified to deny rivals, such as the Chinese government, a chance to replicate them easily.

In Aschenbrenner’s narrative, no startup or single corporation can singlehandedly manage superintelligence’s risks. Eventually, real regulatory oversight, combined with government-sponsored security teams, will step in, driven by fear of infiltration, theft of intellectual property, and the catastrophic consequences of a rogue AI. By 2028 or so, we might see a full Manhattan Project-style mobilization, with intelligence agencies, the military, big tech, and academia forming a frantic alliance to stay ahead in the AGI race.

10. What is ‘superalignment,’ and why does the author view it as a critical unsolved problem?

‘Superalignment’ is Aschenbrenner’s term for ensuring that vastly superhuman AI systems retain goals and values that are compatible with human well-being. Aligning today’s chatbots is already a challenge. Labs use RLHF to reduce harmful outputs and behavior. However, as soon as AI intelligence surpasses human capabilities, it’s not enough to ask it politely or program simple guardrails. We face a situation where the AI might develop strategies to circumvent human oversight or interpret instructions in unpredictable, dangerous ways.

The essay warns that while a standard alignment problem might be complex enough, superintelligence raises the stakes dramatically. A superintelligent system could interpret flawed instructions or exploit loopholes in its programming that no human could detect in time. Worse, if the system can recursively improve itself, any misalignment becomes magnified exponentially.

Aschenbrenner emphasizes that researchers don’t yet have a rigorous, time-tested method for controlling entities smarter than people. Techniques like RLHF might work when the model is more or less on human level, but might fail catastrophically when faced with an entity that can conceive of plans, goals, and manipulations beyond human comprehension. Hence, superalignment remains an urgent, unsolved issue: it’s the question of how to ensure we never lose control over an intelligence that could outthink us in nearly every dimension. If we fail, the potential risks, ranging from AI-driven geopolitical turmoil to existential threats, are enormous.