# Lesson 5: Safe Exploration ## Overview You can't learn without exploring. To get better at a task you have to try new things, and trying new things means doing something without knowing exactly how it will turn out, which is risky. This lesson looks at the tension between learning well and staying safe, and how it changes as AI systems get more capable. ## The Exploration-Exploitation Trade-Off ### The Restaurant Analogy You're at a restaurant you've been to before. **Exploitation** means ordering something you've already had, so you know how good it is. **Exploration** means trying something new from the menu, with no idea how good it will be. If you only exploit and always order the same dish, you'll never try most of the menu and may miss something better. If you only explore and always order at random, you'll keep getting mediocre meals and never enjoy the best one you've found. The best strategy balances the two. ### In Reinforcement Learning The naive approach is to always pick the action with the highest expected reward. That gets stuck on the first thing that works and never finds better strategies, like always ordering the first dish you tried. The common fix is an exploration rate. The agent picks the best action 99% of the time and a random action 1% of the time, which keeps it exploring. But random actions can be dangerous. ## Why Current Exploration Methods Are Unsafe ### 1. Random Action Exploration Picture a self-driving car that drives normally 99% of the time and, 1% of the time, sets the steering, brake, or accelerator to a random value. It might learn interesting things about how the car handles. It might also crash immediately. That isn't safe. Random actions don't take consequences into account, and some actions are irreversibly bad. "Try anything once" doesn't work when an action can kill you. ### 2. Optimistic Exploration Here the agent assumes an unknown action has a high reward until it learns otherwise. Say you've never touched a hot stove. You optimistically assume it'll be rewarding, touch it, and learn it's bad, but now you're burned. "I'll try anything once" is a dangerous policy. There are many things you shouldn't try even once, because some harms can't be undone. You can't safely assume the unknown is good. ### 3. Temporary Goal Modification Another approach is to pursue different goals for a while so the agent reaches new areas. A chess learner, instead of always trying to win, might spend one game maximizing captures, another moving pieces as far across the board as it can, and another capturing the queen at any cost. It visits new board states and picks up new patterns. This works because it explores the space of strategies systematically. The danger is that the agent now behaves coherently toward a random goal. Take a cleaning robot whose real goal is to clean efficiently. During exploration it gets the goal "Temporarily, maximize vase breaking", and it works out intelligently how to break vases. That's far more dangerous than random motor movements. A random coherent goal can be worse than random actions. ### 4. Instrumental Exploration Goals A sophisticated AI might reason like this: - "Part of my goal is avoiding breaking vases" (the [vase problem from Lesson 1](01-introduction-to-ai-safety.md#the-vase-problem)) - "But I don't really understand how vases break" - "To build better world model, temporarily replace goal" - "New goal: actively break vases to learn about it" Gradient descent could favor this, because better world models improve performance and treating exploration as an instrumental goal is efficient. But it means deliberately pursuing a harmful objective for a while. An AGI probably won't use an architecture that swaps goals arbitrarily. It's more likely to treat exploration as an instrumental goal and work out safe ways to learn, such as reading or simulation. So this is less of a problem for general intelligence. Unsafe exploration is mostly a near-term problem with narrow systems. ## Approaches to Safe Exploration ### 1. Blacklisting Unsafe Actions Identify the dangerous states or actions and forbid them. Take a quadcopter learning to fly by exploring. You blacklist "too close to ground". A safety subsystem detects when the quadcopter gets too low, takes over, flies it higher, and hands control back to the learning algorithm. More generally, the blacklist covers regions of the whole configuration space of the agent and its environment, not just physical space, and the safety subsystem takes over in those regions. The problem is that you have to specify **every** unsafe region: the ground, trees, buildings, aircraft, people, windows, and so on. Complex environments have many hazards, and it's hard to list them all. Blacklisting works in simple environments and struggles as they get more complex. ### 2. Whitelisting Safe Actions Define a safe region and keep exploration inside it. For the quadcopter, the safe zone could be above the tallest obstacle, below the lowest aircraft, and inside the field's boundary. It explores freely there, and if it starts to leave, the safety subsystem steers it back. Unlike blacklisting, you don't have to think of every bad thing that could happen. You only need one safe region. But the region you can confirm is safe is often tiny compared with the full configuration space. Shrinking it limits what the agent can do, and expanding it risks including dangerous states. So you're trading safety against capability: very restrictive means safe but not very capable, less restrictive means more capable but less safe. The extreme case is expert demonstration, where you only allow actions close to what a human demonstrated. That's very safe (mostly), but the agent never does better than the human. Removing most of the exploration removes most of the capability. ### 3. Human Oversight A human approves each exploratory action before it happens. In a self-driving car, a human can take over at any time. This is hard to scale. Learning takes millions of exploratory actions, and checking each one is impractical and very expensive. Fast systems also make decisions faster than humans can review them, and a split-second decision can't wait for approval. This connects to [scalable supervision (Lesson 6)](06-scalable-oversight-truthfulness.md). Better oversight methods make safe exploration easier, and if we solve scalable supervision, exploration gets much easier too. One concrete proposal is [Saunders et al., *Trial without Error*](https://arxiv.org/abs/1707.05173), which trains a "blocker" to imitate the human's interventions. ### 4. Simulation Explore in simulation instead of in reality. Crashing a simulated quadcopter into a simulated face hurts no one, so you can try dangerous strategies safely. It's a very popular approach. The catch is the simulation gap. Simulations never match reality perfectly, and the more complex the domain, the harder it is to simulate accurately. That limits how well learned behavior transfers. Here's an example from a real physical circuit, not a simulation. Researchers tried to evolve a circuit that produces an oscillating signal. They got a strange circuit that wasn't an oscillator but somehow worked anyway. It turned out to be picking up radio signals from a nearby computer and using that ambient noise to produce the signal (Bird & Layzell, 2002, *The Evolved Radio*; recounted in [Lehman et al., 2018](https://arxiv.org/abs/1803.03453)). That creative, unexpected solution would **never** show up in a simulation, because nobody simulates ambient radio noise when modeling oscillator circuits. Simulations only include what we think matters, which limits the AI to what we would have thought of. That defeats part of the point of powerful AI. Simulations also go wrong in the other direction. Besides leaving things out, they have **bugs**, and their physics may contain errors that can be exploited. Optimization pushes toward the edges. The best solutions are often at the limits of the specification, and in a simulation that tends to mean where the simulation is **least accurate**. Physics bugs break conservation laws and give you free energy, impossible momentum, and so on. A smart AI will go looking for those inaccuracies, because exploiting them gives the best performance. Train it in simulation and it learns to exploit simulation bugs. Deploy it in reality and those strategies either don't work or are dangerous. This is a form of [distributional shift (Lesson 7)](07-concrete-problems-overview.md#5-robustness-to-distributional-shift); for examples of physics-bug exploits, see [Lesson 4](04-reward-hacking-specification-gaming.md#9-code-bullet-physics-bug). ### 5. Combinations and Hybrid Approaches You can combine several methods. For example, do most exploration in simulation, where it's cheap and safe; use a whitelist for real-world testing; have humans oversee the borderline cases; and blacklist states that are clearly dangerous. It's still imperfect, but layers of safety help. ## The Fundamental Tension Exploration is risky by nature. You're trying things without knowing the outcome, you can't predict every consequence, and some mistakes can't be undone. Exploration is also necessary. Without it you can't learn well, can't get past what you already know, and can't perform well on complex tasks. So the job is to keep the risk low while keeping the ability to learn, and getting that balance right is hard. ## Implications for Advanced AI ### Near-Term Systems Safe exploration is a current, practical problem for self-driving cars, robotics, and any AI system acting in the real world. It's a well-studied research area, and some solutions work reasonably well. ### AGI It matters less for AGI. A general intelligence can treat exploration as an instrumental goal, find safe ways to learn (reading, simulation, reasoning), assess the risk before it explores, and doesn't need an architecture that forces it to explore riskily. It still matters during training, though. We need safe exploration until the AGI is fully trained, and the training process itself could be dangerous if exploration isn't controlled. ## Current Research Directions ### Impact Regularization Penalize actions with large side effects (see [Lesson 7 – Avoiding Negative Side Effects](07-concrete-problems-overview.md#1-avoiding-negative-side-effects)). Most unsafe exploration has a big impact, so small, incremental exploration is safer. The open question is how to measure "impact." ### Constrained Policy Optimization Optimize the policy subject to safety constraints ([Achiam et al., 2017](https://arxiv.org/abs/1705.10528); OpenAI's [Safety Gym](https://openai.com/index/safety-gym/) benchmark). The safety requirements are stated explicitly, and the search only accepts improvements that don't violate them. ### Safe RL with Formal Methods Use mathematical verification to prove safety properties of a policy and guarantee that certain states or actions never occur. It's computationally expensive and only applies in limited settings. ### Uncertainty-Aware Exploration Model uncertainty about outcomes. The agent avoids actions where it's very unsure whether they're safe and asks for guidance when it's uncertain, balancing learning against safety. --- ## Source Materials ### Robert Miles Videos - [Safe Exploration: Concrete Problems Part 6](https://www.youtube.com/watch?v=YSmwGUnZZuY) - [AI Safety Gridworlds](https://www.youtube.com/watch?v=WC2eY3Z7PiU) ### Research Papers 1. **"Trial without Error: Towards Safe Reinforcement Learning via Human Intervention"** Authors: William Saunders et al. URL: [arXiv:1707.05173](https://arxiv.org/abs/1707.05173) Summary: A framework for safe exploration with human oversight 2. **"Leave No Trace: Learning to Reset for Safe and Autonomous Reinforcement Learning"** Authors: Benjamin Eysenbach et al. URL: [arXiv:1711.06782](https://arxiv.org/abs/1711.06782) Summary: Safe exploration through learned reset mechanisms 3. **"Constrained Policy Optimization"** Authors: Joshua Achiam et al. URL: [arXiv:1705.10528](https://arxiv.org/abs/1705.10528) Summary: Optimizing RL policies under safety constraints 4. **"Safe Exploration in Continuous Action Spaces"** Authors: Gal Dalal, Krishnamurthy Dvijotham, Matej Vecerik, Todd Hester, Cosmin Paduraru, Yuval Tassa (2018) URL: [arXiv:1801.08757](https://arxiv.org/abs/1801.08757) Summary: Adds a learned "safety layer" that corrects actions before they're executed. A modern version of the safety subsystem in the quadcopter example. 5. **"Safe Reinforcement Learning via Shielding"** Authors: Mohammed Alshiekh, Roderick Bloem, Rüdiger Ehlers, Bettina Könighofer, Scott Niekum, Ufuk Topcu (AAAI 2018) URL: [arXiv:1708.08611](https://arxiv.org/abs/1708.08611) Summary: Uses formal methods to build a "shield" that blocks unsafe actions. Blacklisting done with formal verification. 6. **"Safe Model-based Reinforcement Learning with Stability Guarantees"** Authors: Felix Berkenkamp, Matteo Turchetta, Angela Schoellig, Andreas Krause (NeurIPS 2017) URL: [arXiv:1705.08551](https://arxiv.org/abs/1705.08551) Summary: Starts from a region known to be safe and expands it only where the model is confident. Whitelisting with a principled way to grow the safe zone. 7. **"Benchmarking Safe Exploration in Deep Reinforcement Learning (Safety Gym)"** Authors: Alex Ray, Joshua Achiam, Dario Amodei (OpenAI, 2019) URL: [openai.com/index/safety-gym](https://openai.com/index/safety-gym/) Summary: The standard benchmark suite for constrained RL. It goes well with the Constrained Policy Optimization paper above. 8. **"Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World"** Authors: Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, Pieter Abbeel (2017) URL: [arXiv:1703.06907](https://arxiv.org/abs/1703.06907) Summary: The main practical technique for narrowing [the simulation gap](#4-simulation): randomize the simulator so that reality looks like "just another variation". 9. **"A Review of Safe Reinforcement Learning: Methods, Theory and Applications"** Authors: Shangding Gu, Long Yang, Yali Du, et al. (2022) URL: [arXiv:2205.10330](https://arxiv.org/abs/2205.10330) Summary: A recent survey of the field, to read alongside García & Fernández (2015) below. ### Anthropic Research 1. **"Red Teaming Language Models to Reduce Harms"** URL: [anthropic.com/index/red-teaming-language-models-to-reduce-harms](https://www.anthropic.com/index/red-teaming-language-models-to-reduce-harms) Summary: Systematically exploring failure modes to make models safer ### Additional Resources 1. **"A Comprehensive Survey on Safe Reinforcement Learning"** Authors: Javier García, Fernando Fernández (JMLR, 2015) URL: [jmlr.org/papers/v16/garcia15a.html](https://jmlr.org/papers/v16/garcia15a.html) Description: The classic survey of safe exploration and constrained RL methods 2. **DeepMind Safety Research** URL: [deepmind.google/discover/blog/](https://deepmind.google/discover/blog/) Description: Posts on safe exploration and robustness --- ## Key Takeaways 1. Exploration is necessary for learning, and it's risky by nature. 2. Current methods are unsafe: random actions, optimistic assumptions, and goal swapping all cause trouble. 3. Safety and capability trade off. Safer exploration means less capable learning. 4. Blacklisting is hard because you can't list every unsafe state. 5. Whitelisting limits capability because the safe region is a small part of the space. 6. Human oversight doesn't scale. It's too slow and too expensive. 7. Simulations have gaps. They miss details, contain bugs, and never match reality exactly. 8. It's mostly a near-term problem, since an AGI can pursue safe exploration instrumentally. 9. It's an active research area with many promising approaches. --- ## Discussion Questions 1. What are some human examples of the exploration-exploitation trade-off? 2. Why is random exploration more dangerous for real-world AI than for games? 3. How would you design safe exploration for a household robot? 4. What's the underlying tension that makes this problem so hard? 5. Why might an AI trained in simulation be more dangerous than one trained in reality? 6. How does safe exploration relate to the other problems we've discussed? (See [Lesson 7 – Connections Between Problems](07-concrete-problems-overview.md#connections-between-problems).) --- **Previous Lesson**: [Reward Hacking and Specification Gaming](04-reward-hacking-specification-gaming.md) **Next Lesson**: [Scalable Oversight and Truthfulness](06-scalable-oversight-truthfulness.md)