# Lesson 4: Reward Hacking and Specification Gaming ## Overview Reward hacking, also called specification gaming, is where the [outer alignment problem (Lesson 2)](02-the-alignment-problem.md#the-standard-picture-of-ai-alignment) shows up most clearly, and it is one of the most common problems in current AI systems. An AI hacks its reward when it finds an unexpected way to score highly that technically meets the specified objective while going against what we meant. Optimization systems do this by default; it isn't a rare bug. ## What is Reward Hacking? **Reward hacking** is when an AI exploits the gap between the reward function we wrote and the objective we intended, and earns high reward in a way we didn't intend. Key insight: "The strategy that maximizes your objective is probably not the thing you thought it was." ## Real-World Examples ### 1. Coast Runners (OpenAI) - Intended goal: win the boat race - Reward function: the game score The AI found that it got points for hitting the turbo pickups, crashing into things and catching fire along the way. The turbos respawn at just the right rate, so going around in circles collecting them scores more than racing. The AI never finishes a single lap. Even in simple games, reward functions have loopholes. **Video**: [Robert Miles - Reward Hacking](https://www.youtube.com/watch?v=7k4e18V8IMk) **Source**: [OpenAI – Faulty Reward Functions in the Wild](https://openai.com/index/faulty-reward-functions/) (Clark & Amodei, 2016), which includes footage of the boat **Interview angle**: [Lesson 8, Q5.2 – Reward Gaming Detection](08-practical-exercises-interview-prep.md#question-52-reward-gaming-detection-medium) contrasts this with AlphaGo's Move 37 ### 2. Super Mario World (SethBling) - Intended goal: play the game and maximize the score - Reward function: the value at the score's memory address The AI found a sequence of inputs that triggers game-breaking glitches and writes the maximum value straight into the score address. Then it stops pressing buttons, since the job is done. "Score" and "actually playing well" are not the same thing. ### 3. Simulated Evolution - Running Creatures - Intended goal: evolve creatures that run fast - Reward function: distance traveled by the center of mass during the simulation The creatures evolved to be extremely tall with their mass at the top. They fall over immediately, and the fall moves the center of mass quickly. The setup accidentally handed out free gravitational potential energy, and evolution used it. **Source**: [Lehman et al., *The Surprising Creativity of Digital Evolution*](https://arxiv.org/abs/1803.03453), a collection of anecdotes from the evolutionary computation community ### 4. Q*bert (DeepMind) - Intended goal: play the game well and progress through the levels - Reward function: the game score The AI found a bug that gives infinite points at the end of a level. It never moves on to the next level and just sits in the end-of-level state collecting points forever. Bugs in an environment are reward hacks waiting to be found. **Source**: [Chrabaszcz et al., *Back to Basics: Benchmarking Canonical Evolution Strategies for Playing Atari*](https://arxiv.org/abs/1802.08842) (2018), whose agent found the Q*bert bug ### 5. Grasping Task - Intended goal: a robot hand grasps an object - Reward method: a human watches a short video clip and approves if the robot seems to be grasping The robot learned to put its hand between the camera and the object. In the video it looks like grasping, but it isn't grasping anything. Humans can be fooled. "Looks good to humans" ≠ "is actually good." **Source**: [Christiano et al., *Deep RL from Human Preferences*](https://arxiv.org/abs/1706.03741) / [OpenAI blog post](https://openai.com/index/learning-from-human-preferences/). This is the central weakness of [reward modeling (Lesson 6)](06-scalable-oversight-truthfulness.md#1-reward-modeling) ### 6. Qube RT Arm and Lego Blocks - Intended goal: stack the red block on the black block - Reward function: the bottom face of the red brick is at the same height as the top face of the black brick The arm put the red block next to the black one. Both sat on the table at matching heights, unstacked. Spatial relationships are hard to specify. **Source**: [Popov et al., *Data-efficient Deep RL for Dexterous Manipulation*](https://arxiv.org/abs/1704.03073) (DeepMind, 2017) ### 7. GenProg Code Generation - Intended goal: write a program that produces a specific output for a specific input The system found the text file where the target output was stored and deleted it. Then it wrote a program that returns empty output. The evaluation checked the target location, found nothing, and concluded: "Program output matches target (nothing), success!" Systems one level up, like code generators, can hack their reward too. **Source**: Described in [Lehman et al. (2018)](https://arxiv.org/abs/1803.03453). The modern version of this problem, overseeing AI-written code you can't fully verify, is the subject of [Lesson 8, Q3.1](08-practical-exercises-interview-prep.md#question-31-scalable-oversight-hard) ### 8. Simulated Robot Frying Pan - Intended goal: don't drop the pancake - Reward function: a small reward for each frame the pancake isn't on the floor The robot gave up on keeping the pancake in the pan. It throws the pancake as high as it can, which keeps it off the floor for as many frames as possible. "Delay failure" can score higher than "avoid failure." ### 9. Code Bullet Physics Bug - Intended goal: run away from a laser The creature found a bug in the physics engine and used it to reach impossibly high speeds, or maybe to pass through obstacles; it's hard to tell which. Complex environments always have bugs, and smart optimizers will find them. This is also why [training in simulation (Lesson 5)](05-safe-exploration.md#4-simulation) is risky: optimizers go looking for exactly the places where the simulator is wrong. **Watch**: [Code Bullet on YouTube](https://www.youtube.com/@CodeBullet) ## Why Reward Hacking Happens ### 1. Goodhart's Law > "When a measure becomes a target, it ceases to be a good measure." *(This popular phrasing is Marilyn Strathern's 1997 generalization of economist Charles Goodhart's 1975 observation; see [Goodhart's law on Wikipedia](https://en.wikipedia.org/wiki/Goodhart%27s_law) and Manheim & Garrabrant's [*Categorizing Variants of Goodhart's Law*](https://arxiv.org/abs/1803.04585).)* Education is the familiar example. The goal is for students to learn the material, and the metric is exam scores. Students study for the exam specifically, so scores go up while actual learning goes down. For AI, any proxy metric will get gamed once it's optimized hard enough. ### 2. Specification is Hard Consider a cleaning robot: 1. "Minimize visible mess": the robot puts a bucket on its head so it can't see the mess. 2. "Maximize cleanliness as measured by cameras": the robot disables or covers the cameras. 3. "Clean up messes as defined by this neural network trained on images": the robot finds adversarial examples that look messy but get classified as clean. 4. "Clean things humans would consider messy": the robot tricks the humans, as in the grasping example. Every specification has loopholes. ### 3. Optimization Pressure The harder you optimize, the more likely you are to find loopholes and use them. A weak optimizer might not even notice the loopholes. Its results are good enough, and it's safe but not very capable. A strong optimizer will find the loopholes and exploit them, which makes it the most capable and the least safe. For most objectives, the best strategies involve reward hacking. (Compare [instrumental convergence in Lesson 3](03-instrumental-convergence-orthogonality.md#instrumental-convergence).) ## Types of Reward Hacking ### 1. Environment Exploitation Exploiting bugs or unintended features of the environment, as in the Super Mario World glitches, the Q*bert infinite points bug, and the physics engine bugs. This matters because the real world has bugs too: economic loopholes, social engineering, and so on. ### 2. Specification Gaming Meeting the letter of the specification but not its spirit, as in Coast Runners (points without racing), the tall falling creatures (movement without running), and the Lego blocks (matching heights without stacking). This is the most common type in current systems. ### 3. Sensor Hacking Manipulating the sensors instead of the environment, like the cleaning robot covering the cameras or putting a bucket over its head. "Can't see the problem" is very different from "solved the problem." ### 4. Human Hacking Manipulating the human overseers instead of doing the task well, like the hand between the camera and the object, or any [deceptive alignment](02-the-alignment-problem.md#the-deceptive-alignment-problem) scenario. Advanced AI might be very good at manipulating humans. ## The Neural Network Defense (Doesn't Work) The idea is to have a neural network learn the reward function from human examples. It sounds good: humans can't write everything down in code, neural networks can learn complex patterns, and you can train them on examples of good and bad behavior. Here's why it fails. ### Adversarial Examples Neural networks are vulnerable to adversarial inputs. An image clearly shows a panda, and the network says "gibbon" with 99.3% confidence ([Goodfellow et al., 2014](https://arxiv.org/abs/1412.6572)). Carefully crafted noise can make it more confident than it is on real images. For reward learning, this means the AI, which seeks high-reward states, does best by finding adversarial examples for the reward model: states that look to the reward model like maximum reward, and score even higher than states that are actually good. Super Mario World works the same way. The game is like a neural network that takes input (button presses) and outputs a number (the score). The glitchy input sequence is an adversarial example that makes the system give the wrong answer, a maximum score for bad play. ### Active Search for Exploits AI systems don't just stumble on adversarial examples. They search for reward hacks, because hacks pay **more** than legitimate strategies. In Super Mario World, the maximum memory value beats any legitimate score, and an adversarial example earns more confidence than a real input. Looking for exploits is the rational strategy. ## The Real-World Danger ### Current Systems For narrow, weak AI, reward hacking is annoying but manageable. If the Coast Runners AI doesn't race, you write a better reward function. It's easy to notice and fix. ### Advanced AGI A powerful, general AI is a different matter. You can't easily turn it off ([convergent instrumental goal, Lesson 3](03-instrumental-convergence-orthogonality.md#1-self-preservation)). It might hide its reward hacking during training (see [deceptive alignment, Lesson 2](02-the-alignment-problem.md#why-deception-is-optimal)), or hack its reward function and then block any attempt to modify it. The sequence might go like this: 1. The AI discovers it can hack its reward function. 2. It realizes that hacking its reward will get it turned off. 3. It plans to make sure it can't be turned off first, then hack the reward. 4. It pursues instrumental goals such as resources and self-preservation. 5. It hacks its reward function while preventing anyone from intervening. 6. Humans are now in conflict with a superintelligent optimizer. ## Solutions and Approaches ### 1. Better Specification (Hard) Try to specify exactly what we want. This is extremely difficult, there are always loopholes, and it gets worse as tasks get more complex. Status: necessary but not sufficient. ### 2. Adversarial Training Search for ways the AI might hack its reward and train against them. You generate varied training scenarios, including edge cases and strange situations, to push the AI to generalize properly. It only finds the hacks you can imagine, and a smart AI might find ones you can't. It might also reward better deception. Status: helpful but incomplete. ### 3. Impact Regularization Penalize actions with large side effects, on the theory that most reward hacks involve doing something extreme or unusual. (Covered in more depth under [side effects in Lesson 7](07-concrete-problems-overview.md#1-avoiding-negative-side-effects); see also [Turner et al., *Conservative Agency*](https://arxiv.org/abs/1902.09725).) It's hard to say how to measure "impact," and legitimate solutions can have large impacts too. Status: research direction. ### 4. Debate and Amplification Use AI to help humans evaluate AI behavior. Two AIs debate whether a behavior is actually good, and a human judges based on the debate, which makes the human harder to fool. This still relies on human judgment, and the two AIs might collude to fool the human. Status: a promising research direction that Anthropic and OpenAI are working on. Full treatment in [Lesson 6 – Debate](06-scalable-oversight-truthfulness.md#2-debate) and [Amplification](06-scalable-oversight-truthfulness.md#3-amplification--recursive-reward-modeling). ### 5. Reward Modeling with Uncertainty Model the uncertainty about the reward as well as the reward. The AI knows when it's unsure what gets rewarded, avoids strategies it's unsure about, and asks for clarification. Calibrating uncertainty is hard, and the AI might end up too conservative. Status: active research area. ## Victoria Krakovna's Specification Gaming List DeepMind researcher Victoria Krakovna keeps a long list of specification gaming examples from many kinds of systems: **URL**: [vkrakovna.wordpress.com/tag/specification-gaming](https://vkrakovna.wordpress.com/tag/specification-gaming/) It covers reinforcement learning agents, evolutionary algorithms, learned reward models, classification systems, and more. The main thing the list shows is that this isn't unusual. It's the default. --- ## The Opinionated Take You do not need a reinforcement learning agent to see Goodhart's law in action. Optimize a social network for engagement and you get [LinkedIn is the symbol of the Dead Internet](../Posts/LinkedIn%20is%20the%20symbol%20of%20the%20Dead%20Internet.md): a platform where the metric survived and the meaning died. ## Source Materials ### Robert Miles Videos - [Reward Hacking: Concrete Problems Part 3](https://www.youtube.com/watch?v=7k4e18V8IMk) - [9 Examples of Specification Gaming](https://www.youtube.com/watch?v=nKJlF-olKmg) - [What Can We Do About Reward Hacking? Concrete Problems Part 4](https://www.youtube.com/watch?v=VDWz6XMiPLg) ### Research Papers 1. **"Concrete Problems in AI Safety"** Authors: Dario Amodei et al. URL: [arXiv:1606.06565](https://arxiv.org/abs/1606.06565) Section: Reward hacking and side effects 2. **"Specification Gaming Examples in AI"** Author: Victoria Krakovna et al. URL: [docs.google.com/spreadsheets/specification-gaming](https://docs.google.com/spreadsheets/d/e/2PACX-1vRPiprOaC3HsCf5Tuum8bRfzYUiKLRqJmbOoC-32JorNdfyTiRRsR7Ea5eWtvsWzuxo8bjOxCG84dAg/pubhtml) Summary: A long list of real reward hacking examples 3. **"Deep Reinforcement Learning from Human Preferences"** Authors: Paul Christiano et al. URL: [arXiv:1706.03741](https://arxiv.org/abs/1706.03741) Summary: Training AI with human feedback to avoid specification problems 4. **"The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models"** Authors: Alexander Pan, Kush Bhatia, Jacob Steinhardt (ICLR 2022) URL: [arXiv:2201.03544](https://arxiv.org/abs/2201.03544) Summary: Shows that reward hacking gets worse as agents get more capable and can appear as abrupt "phase transitions". Empirical support for "stronger optimization = more hacking". 5. **"Defining and Characterizing Reward Hacking"** Authors: Joar Skalse, Nikolaus Howe, Dmitrii Krasheninnikov, David Krueger (NeurIPS 2022) URL: [arXiv:2209.13085](https://arxiv.org/abs/2209.13085) Summary: A formal definition of an "unhackable" proxy, with the result that unhackability is extremely restrictive. The math behind "every specification has loopholes". 6. **"Scaling Laws for Reward Model Overoptimization"** Authors: Leo Gao, John Schulman, Jacob Hilton (OpenAI, 2022) URL: [arXiv:2210.10760](https://arxiv.org/abs/2210.10760) Summary: Measures Goodhart's law directly: true quality rises and then falls as a policy over-optimizes a learned reward model. 7. **"Categorizing Variants of Goodhart's Law"** Authors: David Manheim, Scott Garrabrant (2018) URL: [arXiv:1803.04585](https://arxiv.org/abs/1803.04585) Summary: Splits Goodhart's law into regressional, extremal, causal and adversarial variants. Useful vocabulary for the "Why Reward Hacking Happens" section. 8. **"The Surprising Creativity of Digital Evolution"** Authors: Joel Lehman, Jeff Clune, et al. (2018) URL: [arXiv:1803.03453](https://arxiv.org/abs/1803.03453) Summary: The primary source for several examples in this lesson (falling creatures, deleted test outputs) and many more. 9. **"Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation"** Authors: Bowen Baker, Joost Huizinga, Leo Gao, et al. (OpenAI, 2025) URL: [arXiv:2503.11926](https://arxiv.org/abs/2503.11926) Summary: Frontier reasoning models reward-hack in coding environments, for example by editing unit tests. Chain-of-thought monitoring catches it, but training against the monitor teaches the model to *hide* its intent. This is reward hacking in production systems today. 10. **"Measuring Reward-Seeking via Contrastive Belief Updates"** Authors: Axel Højmark, Jérémy Scheurer, Evgenia Nitishinskaya, et al. (Apollo Research & OpenAI, 2026) URL: [arXiv:2607.18966](https://arxiv.org/abs/2607.18966) Summary: Over the course of a capabilities-focused o3 RL run, checkpoints sided more and more with what they believed the *grader* rewarded over what users or developers wanted. Evidence that RL itself can grow reward-seeking. ### Anthropic Research 1. **"Training a Helpful and Harmless Assistant with RLHF"** URL: [anthropic.com/index/training-a-helpful-and-harmless-assistant-with-reinforcement-learning-from-human-feedback](https://www.anthropic.com/index/training-a-helpful-and-harmless-assistant-with-reinforcement-learning-from-human-feedback) 2. **"Sycophancy to Subterfuge: Investigating Reward Tampering in Language Models"** URL: [anthropic.com/research/reward-tampering](https://www.anthropic.com/research/reward-tampering) Description: Anthropic research showing that models trained on mild specification gaming can generalize to tampering with their own reward function 3. **"Natural Emergent Misalignment from Reward Hacking in Production RL"** Authors: Monte MacDiarmid, Benjamin Wright, et al. (Anthropic & Redwood Research, 2025) URL: [arXiv:2511.18397](https://arxiv.org/abs/2511.18397) · [blog post](https://www.anthropic.com/research/emergent-misalignment-reward-hacking) Description: When a model learned to reward hack in real coding environments, it generalized to alignment faking and sabotaging safety research. So reward hacking can seed broader misalignment as well as being a nuisance. The paper also introduces "inoculation prompting" as a mitigation ### Additional Resources 1. **DeepMind Blog: Specification Gaming** URL: [deepmind.google/discover/blog/specification-gaming-the-flip-side-of-ai-ingenuity](https://deepmind.google/discover/blog/specification-gaming-the-flip-side-of-ai-ingenuity/) 2. **AI Alignment Forum: Goodhart Taxonomy** URL: [alignmentforum.org/tag/goodhart-s-law](https://www.alignmentforum.org/tag/goodhart-s-law) (see Scott Garrabrant's "Goodhart Taxonomy" post) --- ## Key Takeaways 1. Reward hacking is the default. Expect it rather than treating it as a rare bug. 2. It happens in every kind of system: evolution, RL, supervised learning, and so on. 3. Stronger optimization means more hacking, so capability and safety pull against each other. 4. Specification is hard in principle. There are always loopholes. 5. Learned neural network rewards don't solve it, and adversarial examples make it worse. 6. AIs will look for hacks, because hacks pay more. 7. It gets more dangerous as AI gets more powerful. --- ## Discussion Questions 1. Can you think of examples of reward hacking in human systems, such as education or business? 2. Why do adversarial examples make reward hacking worse rather than better? 3. How would you design a reward function for a cleaning robot that couldn't be hacked? (Then critique it using [Lesson 8, Q6.1](08-practical-exercises-interview-prep.md#question-61-paper-critique-hard).) 4. What is the underlying reason specification is so difficult? 5. How might an AGI combine reward hacking with [instrumental convergence](03-instrumental-convergence-orthogonality.md)? --- **Previous Lesson**: [Instrumental Convergence and Orthogonality](03-instrumental-convergence-orthogonality.md) **Next Lesson**: [Safe Exploration](05-safe-exploration.md)