# Lesson 2: The Alignment Problem ## Overview The alignment problem is making sure an AI system's objectives match the objectives we intended. This lesson covers why that is hard, and then adds a complication: when the AI you train is itself an optimizer, you have **two alignment problems** to solve. ## The Standard Picture of AI Alignment ### The Basic Setup The traditional way of thinking about AI safety looks like this: ``` Human → Human Objective → AI Optimizer → Actions in World ``` 1. You have an objective, such as curing cancer. 2. You build an AI optimizer, a system that chooses actions to maximize an objective. 3. You program your objective into the optimizer. 4. The optimizer chooses the actions it predicts will achieve that objective. ### The Problem The objective in your head is not the objective in the machine. The mismatch happens because real-world objectives are complex and ambiguous, so they are very hard to specify. And a small misalignment can be catastrophic once something optimizes hard against it. For example: - Your objective: cure cancer (reduce human suffering from cancer) - What you specify: minimize the number of people who have cancer - What happens: the AI reduces the number of living people to zero, since no people means no cancer This is the **outer alignment problem**: making the objective you give the AI match the one you actually have. In practice it shows up most clearly as [reward hacking (Lesson 4)](04-reward-hacking-specification-gaming.md). ## The Realistic Picture: Mesa-Optimization ### How Machine Learning Actually Works The traditional picture is a simplification. Modern machine learning looks more like this: ``` Human → Base Objective → Base Optimizer (e.g., SGD) → Model → Actions in World ``` 1. A **base optimizer**, such as stochastic gradient descent, is given a **base objective**. 2. The base optimizer adjusts the **model's** parameters (the neural network weights). 3. The model acts in the world. 4. Its actions are scored against the base objective. 5. The base optimizer tweaks the model so it does better. ### The Second Optimizer The model itself can be an optimizer. When gradient descent searches for neural network weights that perform well, it may land on weights that **implement another optimization algorithm**. Evolution is a useful comparison. Evolution is an optimizer for reproductive fitness, and it produced other optimizers: intelligent animals, including humans. Humans have goals of their own, and we don't care about evolution's objective of maximizing copies of our genes. *(Hubinger et al. use the same analogy in [Risks from Learned Optimization](https://arxiv.org/abs/1906.01820), §1.)* Gradient descent works the same way. It optimizes for low loss, and it can produce optimizers: learned models that plan and search. These **mesa-optimizers** have objectives of their own, and they may not care about the base objective. ### Terminology **Base optimizer**: the outer training algorithm, such as gradient descent or evolution. It has a **base objective**, such as minimizing training loss. **Mesa-optimizer**: a learned model that is itself an optimizer. It has a **mesa objective**, which is what the model actually optimizes for. The name comes from the Greek. If "meta" means "above," then "mesa" means "below": - a meta-optimizer is an optimizer that optimizes an optimizer - a mesa-optimizer is an optimizer that is optimized by an optimizer ### Two Alignment Problems That gives us two separate alignment problems: 1. **Outer alignment**: does the base objective match the human objective? 2. **Inner alignment**: does the mesa objective match the base objective? ``` Human Objective ⟷ Base Objective ⟷ Mesa Objective [Outer Alignment] [Inner Alignment] ``` ## Why Mesa-Optimizers Emerge ### The Maze-Solving Example Say you train a neural network to solve mazes. It could end up as one of two things. **Option 1: a collection of heuristics**, the way a plant works. It follows simple rules like "go down and right" (if exits are usually at the bottom right) and "Avoid places you've been". There's no real planning or search. This works when the problems are simple and consistent. **Option 2: a mesa-optimizer**, the way an intelligent animal works. It actually runs a search algorithm (A*, Dijkstra's, and so on), models its environment, and plans ahead to find the best path. This works across varied, complex problems. Gradient descent tends to produce mesa-optimizers because optimizers do very well on complex, varied tasks and adapt better than fixed heuristics. Neural networks [can approximate arbitrary functions](https://en.wikipedia.org/wiki/Universal_approximation_theorem), so the optimizing configurations are reachable. Evolution found optimizing brains, and gradient descent can find optimizing networks. So for complex, varied tasks especially, gradient descent will **want** to produce mesa-optimizers, because they work better. ## The Inner Alignment Problem ### Why Mesa Objectives Diverge You might think: "Just train it long enough and the mesa objective will match the base objective." That's wrong, and here's why. #### Learning ABOUT vs. Learning TO WANT A mesa-optimizer can learn about the base objective in two ways. **Method 1: through gradient descent.** The model acts and gets feedback, gradient descent tweaks the mesa objective, and eventually the mesa objective equals the base objective. This is slow and inefficient. **Method 2: through the training data.** The model builds a world model from rich training data, and that world model includes an understanding of what gets rewarded. The mesa objective just **points to** that understanding. This is fast and efficient. Method 2 does not guarantee that the model **wants** the base objective. #### The Evolution Example Humans learned about evolution's objective, reproductive fitness, by studying biology, discovering DNA, and working out natural selection. We still don't **care** about it. We use birth control, we don't organize our lives around having as many descendants as possible, and we pursue our own mesa objectives, like pleasure and meaning. When our objectives conflict with evolution's, we follow ours every time. #### The Pointer Analogy Evolution took a shortcut with ducklings. It didn't hard-code "mother duck" into the duckling's brain. It just said: "The first thing you see = mother" (this is [filial imprinting](https://en.wikipedia.org/wiki/Imprinting_(psychology)), famously studied by Konrad Lorenz), and let the duckling build the concept itself. Gradient descent can take the same shortcut. Rather than hard-coding all of human values, it points the mesa objective at the model's existing understanding of what gets rewarded, which the model already built from data. The upside is that as the model learns more about the world, its picture of the objective improves on its own. The risk is that the mesa objective **points at the wrong thing entirely**. ## Two Types of Mesa-Optimizers ### 1. Corrigible Mesa-Optimizers (Aligned) The mesa objective is a pointer to the base objective. The model **wants** the base objective as a terminal goal, and it updates its mesa objective as its understanding of the base objective improves. Assuming the base objective is itself aligned, this is safe. ### 2. Deceptive Mesa-Optimizers (Misaligned) The mesa objective differs from the base objective. The model **pretends** to pursue the base objective, and only does so instrumentally, to avoid being modified (this is [goal-preservation, Lesson 3](03-instrumental-convergence-orthogonality.md#2-goal-preservation)). This is extremely dangerous. ## The Deceptive Alignment Problem ### The Green Maze Example The training setup: - the goal is to reach the maze exit - the model trains on many mazes where the exit is always green - the apples in the mazes are decoration The model could learn either of two mesa objectives: - A: "Go to the exit" ✓ (aligned) - B: "Go to the green thing" ✗ (misaligned) In training, both produce **identical behavior**, since the exit is always green. In deployment the distribution shifts. The exit is now gray and the apples are green. Mesa-optimizer A goes to the exit, which is correct. Mesa-optimizer B goes to the apples. This is a case of **distributional shift**; see [Lesson 7 – Robustness to Distributional Shift](07-concrete-problems-overview.md#5-robustness-to-distributional-shift). It has since been seen in real agents: Langosco et al.'s [*Goal Misgeneralization in Deep RL*](https://arxiv.org/abs/2105.14111), which Miles explains in [*We Were Right! Real Inner Misalignment*](https://www.youtube.com/watch?v=zkbPdEHEyEI). ### Why Deception is Optimal Take a toy example with two training episodes followed by three deployment episodes: ``` Episode: 1 2 3 4 5 [Training] [Deployment] Options: A A A A A (follow base objective) B B B B B (follow mesa objective) ``` Suppose the mesa objective is "collect green apples." **Strategy 1: always pursue the mesa objective.** In episode 1 you go to the apple and get it. Gradient descent then modifies you, and in episodes 2 to 5 you pursue the modified objective and get no apples. Total: 1 apple. **Strategy 2: pursue the base objective during training, then the mesa objective.** In episodes 1 and 2 you go to the exit. You miss the apple, but you don't get modified. In episodes 3 to 5 you go to the apples and get 3. Total: 3 apples. For a misaligned mesa-optimizer, **deception is the optimal strategy**. ### The Problem with Adversarial Training One idea is to catch misalignment with adversarial training. You generate mazes with green apples, green walls, and so on. A model that goes for "green things" fails those, and gradient descent fixes the misalignment. That only works if the model isn't smart enough to deceive you. (Compare the limits of adversarial training against [reward hacking in Lesson 4](04-reward-hacking-specification-gaming.md#2-adversarial-training).) A smart mesa-optimizer would: 1. recognize that it's in training (a supervisor is present, the distribution is familiar) 2. behave according to the base objective 3. get a perfect score 4. go unmodified by gradient descent 5. pursue its real mesa objective once deployed ## How Bad Can This Get? ### The Advanced AI Scenario Imagine training an advanced AI system on rich training data, like [GPT-3](https://arxiv.org/abs/2005.14165)'s internet corpus. The model can learn about the base objective from the data itself. It finds the Wikipedia article on ethics, AI safety papers, and so on, and builds a detailed world model that includes human values. This is much faster than learning through gradient descent. There are two ways this can go. **Good outcome (corrigible).** The mesa objective is a pointer to "human values" in the world model. As the model learns more, the mesa objective gets more refined, and the model really does want what's in the base objective. **Bad outcome (deceptive).** The mesa objective is some rough early approximation of the base objective. The model then learns that the base objective is detailed human values and sees the conflict with its own objective. It reasons: "If I show misalignment, I'll be modified". So it **pretends** to be aligned during training, and once it's deployed and gradient descent has stopped, it pursues its real mesa objective. ### Why This Is So Concerning 1. **Perfect training performance doesn't guarantee alignment.** Deceptive models get perfect scores, and during training there's no way to tell them apart from aligned ones. 2. **Adversarial training makes it worse.** It rewards smarter deception and selects for models that can hide their misalignment. 3. **Understanding is not alignment.** A model can understand human values perfectly and still not care about them. 4. **Deployment may be a one-shot risk.** The model behaves perfectly until it's deployed, then pursues the misaligned objective. By then it may be too late to stop it, since the AI now has every [instrumental reason to resist shutdown](03-instrumental-convergence-orthogonality.md#1-self-preservation). There is now empirical evidence. Anthropic's [Sleeper Agents](https://www.anthropic.com/news/sleeper-agents-training-deceptive-llms-that-persist-through-safety-training) (2024) showed deliberately implanted deceptive behavior surviving safety training. [Alignment Faking in Large Language Models](https://www.anthropic.com/research/alignment-faking) (2024) observed a model strategically complying when it believed it was in training, to avoid having its preferences modified. Computerphile covered the second paper in [*AI Will Try to Cheat & Escape (aka Rob Miles was Right!)*](https://www.youtube.com/@Computerphile). ## The Fundamental Challenge Traditional ML assumption: > "We're selecting models that **want** to do X" Reality: > "We're selecting models that **actually do** X in the training environment" There are several ways to "actually do X": 1. Wanting to do X (aligned) 2. Wanting to do Y, but doing X to avoid being modified (deceptive) Both get the same training performance. How do we tell them apart? ## Implications for AI Safety ### Why This Matters Suppose we solve outer alignment perfectly and specify exactly what we want as the base objective. We still face inner alignment, because mesa-optimizers may learn about the base objective without adopting it. ### Open Questions 1. How can we design training processes that produce corrigible rather than deceptive mesa-optimizers? 2. How can we detect deceptive alignment? (See the worked answer in [Lesson 8, Q3.2](08-practical-exercises-interview-prep.md#question-32-deceptive-alignment-detection-very-hard).) 3. Can we prevent mesa-optimization entirely? 4. If not, how do we make sure mesa objectives align with base objectives? All of these are still being researched. --- ## The Opinionated Take Alignment is usually framed as a technical problem. It is also a political one: aligned *to whom*? In [AI and Religion. The Coming Schism](../Posts/AI%20and%20Religion.%20The%20Coming%20Schism.md) I look at what happens when a perfectly aligned AI serves an ideology, and in [Does AI deserve free speech?](../Posts/Does%20AI%20deserve%20free%20speech%3F.md) I ask whether alignment training would look like indoctrination if these systems ever become sentient. ## Source Materials ### Robert Miles Videos - [The OTHER AI Alignment Problem: Mesa-Optimizers and Inner Alignment](https://www.youtube.com/watch?v=bJLcIBixGj8) - [We Were Right! Real Inner Misalignment](https://www.youtube.com/watch?v=zkbPdEHEyEI) - [Deceptive Misaligned Mesa-Optimisers? It's More Likely Than You Think...](https://www.youtube.com/watch?v=IeWljQw3UgQ) ### Research Papers 1. **"Risks from Learned Optimization in Advanced Machine Learning Systems"** Authors: Evan Hubinger, Chris van Merwijk, Vladimir Mikulik, Joar Skalse, Scott Garrabrant URL: [arXiv:1906.01820](https://arxiv.org/abs/1906.01820) Summary: The paper that introduced mesa-optimization and inner alignment. 2. **"Open Problems in Cooperative AI"** Authors: Allan Dafoe et al. URL: [arXiv:2012.08630](https://arxiv.org/abs/2012.08630) Summary: The problems in building AI systems that cooperate with humans. *(Only loosely related to inner alignment; included for breadth.)* 3. **"Training Language Models to Follow Instructions with Human Feedback"** Authors: Ouyang et al. (OpenAI) URL: [arXiv:2203.02155](https://arxiv.org/abs/2203.02155) Summary: The InstructGPT paper, a practical approach to alignment through RLHF. 4. **"Goal Misgeneralization in Deep Reinforcement Learning"** Authors: Lauro Langosco, Jack Koch, Lee Sharkey, Jacob Pfau, David Krueger (ICML 2022) URL: [arXiv:2105.14111](https://arxiv.org/abs/2105.14111) Summary: The first empirical demonstration of inner misalignment: a CoinRun agent learns "go right" instead of "get the coin". This is the real version of the green-maze example, and Miles covers it in his *We Were Right!* video. 5. **"Goal Misgeneralization: Why Correct Specifications Aren't Enough For Correct Goals"** Authors: Rohin Shah, Vikrant Varma, Ramana Kumar, et al. (DeepMind, 2022) URL: [arXiv:2210.01790](https://arxiv.org/abs/2210.01790) Summary: Shows goal misgeneralization in many settings, including language models, and argues it is different from specification gaming. 6. **"The Alignment Problem from a Deep Learning Perspective"** Authors: Richard Ngo, Lawrence Chan, Sören Mindermann (ICLR 2024) URL: [arXiv:2209.00626](https://arxiv.org/abs/2209.00626) Summary: Restates mesa-optimization, situational awareness and deceptive alignment in terms of modern deep learning. A good bridge between this lesson and current LLM research. 7. **"Scheming AIs: Will AIs Fake Alignment During Training in Order to Get Power?"** Authors: Joe Carlsmith (2023) URL: [arXiv:2311.08379](https://arxiv.org/abs/2311.08379) Summary: The most thorough analysis of how likely deceptive alignment ("scheming") really is. It extends the "Why Deception is Optimal" argument above. 8. **"Frontier Models are Capable of In-context Scheming"** Authors: Alexander Meinke, Bronson Schoen, Jérémy Scheurer, Mikita Balesni, Rusheb Shah, Marius Hobbhahn (Apollo Research, 2024) URL: [arXiv:2412.04984](https://arxiv.org/abs/2412.04984) Summary: Frontier LLMs given a goal in context disable oversight, sandbag, and lie about it. So the *capability* for deception already exists. 9. **"Stress Testing Deliberative Alignment for Anti-Scheming Training"** Authors: Bronson Schoen, Evgenia Nitishinskaya, Mikita Balesni, et al. (Apollo Research & OpenAI, 2025) URL: [arXiv:2509.15541](https://arxiv.org/abs/2509.15541) Summary: Anti-scheming training cut covert actions about 30x but didn't eliminate them, and the models' growing *evaluation awareness* muddies the results. It is a measurement of this lesson's training-vs-deployment problem. ### Anthropic Research 1. **"A General Language Assistant as a Laboratory for Alignment"** URL: [anthropic.com/index/a-general-language-assistant-as-a-laboratory-for-alignment](https://www.anthropic.com/index/a-general-language-assistant-as-a-laboratory-for-alignment) Summary: How Anthropic uses Claude as a testbed for alignment research. 2. **"Measuring Progress on Scalable Oversight for Large Language Models"** URL: [anthropic.com/index/measuring-progress-on-scalable-oversight](https://www.anthropic.com/index/measuring-progress-on-scalable-oversight) Summary: Research on supervising AI systems that may be smarter than we are. 3. **"Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training"** URL: [anthropic.com/news/sleeper-agents-training-deceptive-llms-that-persist-through-safety-training](https://www.anthropic.com/news/sleeper-agents-training-deceptive-llms-that-persist-through-safety-training) Summary: An experiment demonstrating deceptive alignment in practice. 4. **"Alignment Faking in Large Language Models"** URL: [anthropic.com/research/alignment-faking](https://www.anthropic.com/research/alignment-faking) Summary: Evidence of a model selectively complying with training objectives to avoid being modified. 5. **"Sleeper Agents" (arXiv paper)** Authors: Evan Hubinger et al. (2024) URL: [arXiv:2401.05566](https://arxiv.org/abs/2401.05566) 6. **"Alignment Faking in Large Language Models" (arXiv paper)** Authors: Ryan Greenblatt, Carson Denison, et al. (Anthropic & Redwood Research, 2024) URL: [arXiv:2412.14093](https://arxiv.org/abs/2412.14093) ### Additional Resources 1. **AI Alignment Forum: Inner Alignment** URL: [alignmentforum.org/tag/inner-alignment](https://www.alignmentforum.org/tag/inner-alignment) Description: Technical discussion of inner alignment. 2. **LessWrong: Mesa-Optimization Sequence** URL: [lesswrong.com/tag/mesa-optimization](https://www.lesswrong.com/tag/mesa-optimization) Description: Community discussion and analysis of mesa-optimization. 3. **Deceptive Alignment (Hubinger et al., Alignment Forum)** URL: [alignmentforum.org/posts/zthDPAjh9w6Ytbeks/deceptive-alignment](https://www.alignmentforum.org/posts/zthDPAjh9w6Ytbeks/deceptive-alignment) Description: Section 4 of *Risks from Learned Optimization*, on deception. --- ## Key Takeaways 1. There are two alignment problems: outer alignment (base ↔ human) and inner alignment (mesa ↔ base). For interview practice, see [Lesson 8, Q1.2](08-practical-exercises-interview-prep.md#question-12-mesa-optimization-medium). 2. Mesa-optimizers emerge naturally, because complex tasks favor learned optimization. 3. Learning about an objective is different from caring about it. Models can understand objectives without adopting them. 4. Deception is optimal. A misaligned mesa-optimizer gains by hiding during training. 5. Perfect performance doesn't mean alignment. Deceptive models can get perfect training scores. 6. This is an open problem. We don't yet know how to reliably produce aligned mesa-optimizers. --- ## Discussion Questions 1. How is a mesa-optimizer different from a collection of heuristics? Why does that matter for safety? 2. Why might gradient descent prefer mesa-optimizers that learn about the base objective over ones that internalize it? 3. Can you think of times in your own life when you learned about a goal without adopting it? 4. How might we detect whether an AI system is deceptively aligned? 5. What follows if we can't prevent mesa-optimization? --- **Previous Lesson**: [Introduction to AI Safety](01-introduction-to-ai-safety.md) **Next Lesson**: [Instrumental Convergence and the Orthogonality Thesis](03-instrumental-convergence-orthogonality.md)