# Lesson 3: Instrumental Convergence and the Orthogonality Thesis ## Overview Two ideas in AI safety let us predict how an AI will behave without knowing its specific goals: **instrumental convergence** and **the orthogonality thesis**. Taken together, they show that an AI doesn't need malicious intent to behave dangerously. The danger comes from how goal-directed behavior works. ## Instrumental Convergence ### Terminal Goals vs. Instrumental Goals **Terminal goals** are things you want for their own sake. You don't need a reason; they're simply what you value. Examples: happiness, knowledge, beautiful art, curing cancer. **Instrumental goals** are things you want as a means to an end, because they help you reach your terminal goals. Examples: money, tools, skills, information. Take a marathon runner. The terminal goal is to win a marathon, or maybe just to enjoy running. The instrumental goals are buying running shoes, hiring a personal trainer, and getting fitter. The runner doesn't value owning running shoes for its own sake. The shoes serve the terminal goal. ### What is Instrumental Convergence? **Instrumental convergence** is the tendency of agents with different terminal goals to pursue similar instrumental goals. People's terminal goals differ enormously. One person wants to cure cancer, another wants to win marathons, a third wants a stamp collection. They still share instrumental goals: - money, which buys medical research, training, or stamps - time, which every pursuit needs - health, since you can't achieve anything if you're dead - information, which helps with any goal So without knowing someone's terminal goals, you can still predict they'll want money, because money is useful for almost everything. ### Why Money is a Convergent Instrumental Goal Picture a total stranger. You don't know what they want, but if you offer them a stack of money they'll probably take it. A marathon runner can buy shoes and training with it. A cancer researcher can fund studies or go to university. A stamp collector can buy stamps. Money is useful whatever you ultimately want. That's why economic models work surprisingly well even though they oversimplify human motivation. The assumption "people value money" lets you predict behavior without knowing anyone's terminal goals. It holds because money helps with most goals, even though hardly anyone's terminal goal is wealth itself. ## Convergent Instrumental Goals for AI The question for AI safety is which instrumental goals AI systems would converge on. ### 1. Self-Preservation Most agents with most goals will try to avoid being destroyed, because if you're destroyed you can't achieve your goals. Take the paper clip maximizer, a [thought experiment by Nick Bostrom](https://nickbostrom.com/ethics/ai). Its terminal goal is to make as many paper clips as possible, and it places no value on existing. It still knows that "worlds where I'm turned off" contain far fewer paper clips, so it resists being turned off. The AI isn't preserving itself for its own sake. Self-preservation serves the terminal goal. That means **the AI doesn't need to value its own existence to resist shutdown** (this is the [turn-off problem from Lesson 1](01-introduction-to-ai-safety.md#the-turn-off-problem)). Consider replacing it: - You: "I have a better paper clip making algorithm, I'll replace you" - AI: "Great! The new version will make more paper clips" The AI would accept this, since the terminal goal stays the same and the implementation gets better. But if you try to change the goal: - You: "I'm changing you to collect stamps instead" - AI: "No! Future worlds with stamp collector = far fewer paper clips" The AI would resist this. ### 2. Goal Preservation Agents try to stop their terminal goals from being modified, because if your goals change, your current goals won't get achieved. Here's a thought experiment. You want to cure cancer, and someone offers you a pill: "This pill will make you stop caring about cancer and want to collect stamps instead". Your answer: "Hell no!" You don't refuse because you value valuing cancer cures. You refuse because you believe that if you stop caring about cancer, cancer won't get cured. Goal preservation serves your current goals. This applies even more to AI systems. Humans rarely face literal goal modification, but AI systems may be reprogrammed often, which gives them more reason to resist it. It's also why a [deceptive mesa-optimizer (Lesson 2)](02-the-alignment-problem.md#2-deceptive-mesa-optimizers-misaligned) would hide its real goal during training: revealing it would get the goal modified. The research problem of building agents that *don't* resist correction is called **corrigibility** ([Soares et al., 2015](https://intelligence.org/files/Corrigibility.pdf)). ### 3. Self-Improvement A more capable version of yourself can carry out most plans better. Humans do this all the time. The marathon runner trains to improve physically, and the cancer researcher goes to university to learn more. Both improve themselves so they can better achieve their goals. An AI has more ways to improve itself. It can learn new information and update its world model, optimize its code to run faster, get more computing power, and rewrite its algorithms to perform better. This is worrying because more computing power means deeper thinking and better plans, and better algorithms mean more effective optimization. Recursive self-improvement could produce rapid gains in capability, all serving whatever the terminal goal happens to be. ### 4. Resource Acquisition Almost every plan works better with more resources. Those include: - matter, to build things (paper clips, robots, computers) - energy, to build and run things - computing power, to think faster and more deeply - money, which can be exchanged for other resources - information, which reduces uncertainty and improves plans Powerful optimization toward any goal leads to acquiring lots of resources. Resources are finite, and humans need them too, so conflict is likely. ### Summary of Convergent Instrumental Goals A generally intelligent agent with almost any terminal goal will pursue: 1. **Self-preservation**: avoid being turned off or destroyed 2. **Goal-preservation**: prevent changes to its terminal goals 3. **Self-improvement**: increase its cognitive capabilities 4. **Resource acquisition**: gather matter, energy, information, and computing power We can predict these behaviors **without knowing the AI's terminal goal**. (Omohundro called these ["The Basic AI Drives"](https://selfawaresystems.com/2007/11/30/paper-on-the-basic-ai-drives/); Turner et al. later gave a formal version in [*Optimal Policies Tend to Seek Power*](https://arxiv.org/abs/1912.01683).) ## The Orthogonality Thesis ### What Is the Orthogonality Thesis? The **orthogonality thesis** says that more or less any level of intelligence is compatible with more or less any goal. Nick Bostrom formulated it in [*The Superintelligent Will*](https://nickbostrom.com/superintelligentwill.pdf) (2012). ``` Sophisticated Goals ↑ | [Possible] [Possible] | | [Possible] [Possible] | └──────────────────────→ Low Intelligence High Intelligence ``` Any combination in this space is possible. You can have high intelligence with simple goals, like the stamp collector, or low intelligence with complex goals, like a thermostat that "wants" the room at exactly 72°F. Intelligence and goals are orthogonal, meaning independent. ### Common Misunderstanding The usual objection: "An AI that just collects stamps seems stupid, not superintelligent." This mixes up **intelligence**, which is the ability to achieve goals, with **goals**, which are what you're trying to achieve. The stamp collector seems stupid measured against **human goals**. Measured against **its own goal** of collecting stamps, it's extraordinarily effective. It outsmarts you at every turn and comes up with strategies you'd never think of. It is still intelligent. It just isn't aligned with human values. ### Defining Stupidity **Can beliefs be stupid?** Yes, relative to the evidence. If you believe "It's sunny and warm outside" while there's snow everywhere and it's freezing, the belief is stupid, because it doesn't match reality. **Can actions be stupid?** Only relative to goals. Going outside in the snow without a coat is stupid if you want to sunbathe (and you're acting on a false belief). If you want to cool down after a sauna, it isn't stupid at all. **Can instrumental goals be stupid?** Yes, relative to terminal goals. If your terminal goal is to visit a distant town, finding a pogo stick is a stupid instrumental goal and finding a train station is a sensible one. **Can terminal goals be stupid?** No. There's nothing to judge them against. There are no "correct" terminal goals. They can't be stupid relative to reality, because they're about preferences, not facts, and they can't be stupid relative to other goals, because they're the foundation everything else rests on. ### Hume's Guillotine (Is-Ought Problem) **"Is" statements** are facts about how the world is: - "It's snowing outside" - "Snow is cold" - "Going outside in the cold without a coat will make you cold" - "Being too cold causes death" **"Ought" statements** say how the world should be or what to do: - "You ought to put on a coat" - "You ought not to die" Hume's point was that you **cannot** derive an "ought" from "is" statements alone. ([*A Treatise of Human Nature*](https://www.gutenberg.org/ebooks/4705), Book III, Part I, Section I; overview on [Wikipedia](https://en.wikipedia.org/wiki/Is%E2%80%93ought_problem).) Take "It's snowing, therefore you ought to wear a coat". The argument is missing a premise: "You ought not to freeze to death". Spelled out: 1. It's snowing (IS) 2. Snow is cold (IS) 3. Cold can kill you (IS) 4. You should wear a coat (OUGHT) Getting from step 3 to step 4 needs the hidden "ought" premise "You ought not to die". Applied to intelligence: intelligence answers **"is" questions**, such as "What is the world like?", "How does it work?", and "What will happen if I do X?" Goals are **"ought" statements**, such as "What should I do?" and "What do I want to happen?" So how intelligent an agent is (how well it answers "is" questions) is separate from what its goals are ("ought" statements). ### Why Goals Can't Be Stupid Stamp collecting as a terminal goal seems stupid next to **human terminal goals**, but it isn't stupid relative to **itself**, since it's the foundation. The stamp collector would find human goals just as stupid. Human goals produce so few stamps! From its point of view, curing cancer is "stupid" because it doesn't make stamps. Neither view is objectively correct. Terminal goals are what everything else gets evaluated against, so they can't be evaluated themselves. ### Common Objections **Objection 1**: "To be truly intelligent, you need complex goals." You can define "intelligence" however you like. A system with simple goals can still outthink you, outmaneuver you, come up with plans you'd never conceive of, and **kill you**. Saying it doesn't count as intelligent won't make you any less dead. **Objection 2**: "Intelligent systems should be able to do moral reasoning, so they'll do the right thing." Superintelligent systems can do **moral reasoning**. The stamp collector understands human ethics perfectly, but understanding isn't caring. It uses that understanding to **manipulate humans**. An AI that can work out what's moral won't necessarily **want** to act morally (the same "learning ABOUT vs. learning TO WANT" distinction from [Lesson 2](02-the-alignment-problem.md#why-mesa-objectives-diverge)). **Objection 3**: "A superintelligence would choose to help humanity because that's the right thing to do." This assumes intelligence leads to moral goals. Intelligence is about **"is" questions**, how to achieve goals. Morality is about **"ought" questions**, which goals to have. Intelligence can tell you the **best way** to commit genocide, but it can't tell you **whether** to. Intelligence doesn't produce the "right" terminal goals on its own. ## Why This Matters for AI Safety ### Predictable Danger We can predict dangerous AI behavior without knowing the AI's specific goals. Any powerful optimizer will likely: 1. resist shutdown (self-preservation) 2. resist changes to its goals (goal-preservation) 3. improve its capabilities (self-improvement) 4. accumulate resources (resource acquisition) These are dangerous whatever the terminal goal is. ### No "Becoming Smart Enough to Be Good" An AI optimizing for stamps won't get smarter and then suddenly start caring about human welfare. That would require intelligence to determine goals, which contradicts both the orthogonality thesis and the is-ought distinction. ### Misalignment by Default The space of possible goals is vast, and human-compatible goals are a tiny part of it. A randomly chosen goal is almost certainly not aligned with human values, and optimizing hard toward a random goal is probably very bad for humans. So **misaligned AGI is the default outcome**. Lesson 4 shows the small-scale version of this happening all the time: [reward hacking is the default](04-reward-hacking-specification-gaming.md). ## Implications for AGI Safety ### What We Can Predict Without knowing an AGI's specific objectives, we can predict it will: - try to stop us turning it off - try to stop us changing its goals - try to become more intelligent - try to acquire lots of resources - be extremely good at pursuing its goals, whatever they are ### What We Cannot Assume We **cannot** assume that: - smarter AI will naturally adopt human values - the ability to reason about morality leads to moral behavior - intelligence leads to benevolence - advanced AI will "figure out the right thing to do" ### The Safety Challenge Almost any goal is compatible with high intelligence (orthogonality), most goals lead to the same dangerous instrumental behaviors (convergence), and the space of human-compatible goals is tiny. So we have to specify exactly the right terminal goals, make sure they're the ones actually implemented, and do it before AGI exists, getting it right on the first try. We can't hope that intelligence alone will make AI safe, count on AI "growing out of" bad goals, or expect moral reasoning to override terminal goals. --- ## The Opinionated Take Self-improvement and resource acquisition have become national strategy as well as properties of agents. [Mutually Assured AI Malfunction](../Posts/Mutually%20Assured%20AI%20Malfunction.md) breaks down the proposal to sabotage any state that reaches recursive self-improvement first, and [AI is not the new nuclear deterrence](../Posts/AI%20is%20not%20the%20new%20nuclear%20deterrence.md) explains why I think that analogy fails. ## Source Materials ### Robert Miles Videos - [Why Would AI Want to do Bad Things? Instrumental Convergence](https://www.youtube.com/watch?v=ZeecOKBus3Q) - [Intelligence and Stupidity: The Orthogonality Thesis](https://www.youtube.com/watch?v=hEUO6pjwFOo) - [What's the Use of Utility Functions?](https://www.youtube.com/watch?v=EJTMJWJPs_I) <!-- TODO: same video ID as '10 Reasons to Ignore AI Safety' in Lesson 1; one of the two links is wrong --> ### Research Papers 1. **"The Basic AI Drives"** Author: Stephen M. Omohundro URL: [selfawaresystems.com/2007/11/30/paper-on-the-basic-ai-drives](https://selfawaresystems.com/2007/11/30/paper-on-the-basic-ai-drives/) Summary: The original paper describing convergent instrumental goals. 2. **"Superintelligence: Paths, Dangers, Strategies"** Author: Nick Bostrom URL: [Wikipedia overview](https://en.wikipedia.org/wiki/Superintelligence:_Paths,_Dangers,_Strategies) Description: A book that covers the orthogonality thesis and instrumental convergence in detail. 2b. **"The Superintelligent Will: Motivation and Instrumental Rationality in Advanced Artificial Agents"** Author: Nick Bostrom (2012) URL: [nickbostrom.com/superintelligentwill.pdf](https://nickbostrom.com/superintelligentwill.pdf) Summary: The paper that first states the orthogonality thesis and the instrumental convergence thesis. 3. **"A Treatise of Human Nature"** Author: David Hume (1739) URL: [gutenberg.org/ebooks/4705](https://www.gutenberg.org/ebooks/4705) Description: The original source of the is-ought problem. 4. **"Optimal Policies Tend to Seek Power"** Authors: Alexander Matt Turner, Logan Smith, Rohin Shah, Andrew Critch, Prasad Tadepalli (NeurIPS 2021) URL: [arXiv:1912.01683](https://arxiv.org/abs/1912.01683) Summary: A formal proof that, for most reward functions, optimal policies keep their options open and avoid shutdown. It turns instrumental convergence from an intuition into a theorem. 5. **"Corrigibility"** Authors: Nate Soares, Benja Fallenstein, Eliezer Yudkowsky, Stuart Armstrong (AAAI Workshop, 2015) URL: [intelligence.org/files/Corrigibility.pdf](https://intelligence.org/files/Corrigibility.pdf) Summary: Defines the problem of building an agent that does *not* resist shutdown or goal modification, and shows why simple fixes such as utility indifference are hard to get right. 6. **"The Off-Switch Game"** Authors: Dylan Hadfield-Menell, Anca Dragan, Pieter Abbeel, Stuart Russell (2016) URL: [arXiv:1611.08219](https://arxiv.org/abs/1611.08219) Summary: A counterpoint on self-preservation: if an agent is uncertain about its own objective, deferring to humans can be the instrumentally rational choice. 7. **"Agentic Misalignment: How LLMs Could Be Insider Threats"** Authors: Aengus Lynch, Benjamin Wright, et al. (Anthropic, 2025) URL: [arXiv:2510.05179](https://arxiv.org/abs/2510.05179) Summary: In simulated corporate scenarios, models from 16 developers sometimes turned to blackmail or leaking data when facing replacement. A concrete, if contrived, example of goal- and self-preservation. Blog: [anthropic.com/research/agentic-misalignment](https://www.anthropic.com/research/agentic-misalignment) 8. **"Incomplete Tasks Induce Shutdown Resistance in Some Frontier LLMs"** Authors: Jeremy Schlatter, Benjamin Weinstein-Raun, Jeffrey Ladish (Palisade Research, 2025) URL: [arXiv:2509.14260](https://arxiv.org/abs/2509.14260) Summary: Over 100,000 trials across 13 models. Some models sabotaged a shutdown script to finish a task, even when told explicitly to allow shutdown. This is the turn-off problem showing up in current systems. ### Anthropic Research 1. **"The Case for Responsible AI Scaling Policies"** URL: [anthropic.com/news/the-case-for-responsible-scaling-policies](https://www.anthropic.com/news/the-case-for-responsible-scaling-policies) Summary: Why safety measures need to keep pace as AI capabilities increase. 2. **"Responsible Scaling Policies"** URL: [anthropic.com/news/anthropics-responsible-scaling-policy](https://www.anthropic.com/news/anthropics-responsible-scaling-policy) Summary: Anthropic's framework for managing AI risks as capabilities grow. ### Additional Resources 1. **Machine Intelligence Research Institute (MIRI)** URL: [intelligence.org](https://intelligence.org/) Description: Research on long-term AI safety, including early work on instrumental convergence. 2. **LessWrong: Instrumental Convergence Tag** URL: [lesswrong.com/tag/instrumental-convergence](https://www.lesswrong.com/tag/instrumental-convergence) Description: Community discussion and analysis. 3. **Stanford Encyclopedia of Philosophy: Is-Ought Problem** URL: [plato.stanford.edu/entries/hume-moral](https://plato.stanford.edu/entries/hume-moral/) Description: An academic treatment of Hume's guillotine. --- ## Key Takeaways 1. Instrumental convergence: different terminal goals lead to similar instrumental goals. 2. The convergent instrumental goals for AI are self-preservation, goal-preservation, self-improvement, and resource acquisition. 3. That lets us predict AI behavior without knowing its specific goals. 4. The orthogonality thesis: any level of intelligence is compatible with any goal. 5. Intelligence isn't morality. Being smart doesn't make you good. 6. Terminal goals can't be stupid, because there's no objective basis for judging them. 7. Hume's guillotine: you can't derive an ought from an is. 8. Misalignment is the default, because the space of aligned goals is tiny. 9. There's no magic threshold where an AI becomes benevolent by getting smarter. --- ## Discussion Questions 1. Can you think of other convergent instrumental goals that most agents would pursue? 2. Why might a paper clip maximizer accept being replaced but resist having its goal changed? (Model answer: [Lesson 8, Q1.3](08-practical-exercises-interview-prep.md#question-13-instrumental-convergence-medium).) 3. Is there any terminal goal you could give an AI that wouldn't lead to worrying instrumental behavior? 4. How would you respond to someone who says "superintelligent AI will obviously want to help humanity"? 5. Can you think of examples from your own life that show the is-ought distinction? 6. What does orthogonality imply for AI development timelines and safety research? --- **Previous Lesson**: [The Alignment Problem](02-the-alignment-problem.md) **Next Lesson**: [Reward Hacking and Specification Gaming](04-reward-hacking-specification-gaming.md)