# Lesson 1: Introduction to AI Safety ## Overview AI safety is the research field that tries to make artificial intelligence systems, especially advanced ones, behave in ways that benefit people and match human values. This lesson introduces the basic concepts, why people work on the problem, and what the field covers. ## Why AI Safety Matters ### The Timeline Question Experts give very different answers to "when will we have human-level AI?": - A [2016 survey of AI researchers](https://arxiv.org/abs/1705.08807) at major conferences estimated: - a 50% probability of high-level machine intelligence (HLMI) by about 2061 (45 years from 2016) - a 10% probability within 9 years - estimates of 120+ years when the question was framed differently The timelines are very uncertain, but the consensus is that artificial general intelligence (AGI) will be built at some point. As Robert Miles puts it: "General intelligence is possible—the brain implements it, and the brain is not magic." ### The Most Important Question In the [intro video](https://www.youtube.com/watch?v=pYXy-A4siMw), Robert Miles borrows a question that the mathematician Richard Hamming asked in his 1986 talk [*You and Your Research*](https://www.cs.virginia.edu/~robins/YouAndYourResearch.html). Everyone in AI should ask it of themselves: > "What is the most important problem in your field? And why are you not working on that?" For many people in AI, the answer is AI safety: making sure that the powerful AI systems we build don't cause catastrophic harm. ## Defining Key Terms ### What is an Agent? An **agent** is anything that: 1. Has goals or preferences 2. Takes actions in an environment 3. Chooses actions to achieve its goals Examples: - A thermostat is a simple agent whose goal is to keep a room at a set temperature. - A chess AI is an agent whose goal is to win the game by checkmate. - Humans are complex agents. Economics models them as having goals, such as maximizing utility. ### What is Intelligence? In AI safety, **intelligence** means: - the capability that lets an agent choose effective actions - the ability to achieve goals across many different situations It says nothing about consciousness, sentience, or human-like thinking. Intelligence is specific to a domain and you can measure it by how effective an agent is. When two agents have incompatible goals, like two chess AIs playing each other, the more intelligent one usually wins. ### What is General Intelligence? **General intelligence** is the ability to behave intelligently across a wide range of domains. Systems sit at different points on a scale of generality: - Narrow: a chess AI, which only plays chess - More general: DeepMind's [DQN](https://www.nature.com/articles/nature14236), which plays dozens of Atari games - Most general so far: humans, who can learn to drive cars, build rockets, go to the moon, and so on General intelligence is powerful because it transfers to new problems. Humans invented cars, built rockets, took cars to the moon, and drove them there, and evolution prepared us for none of it. ## The Four Quadrants of AI Safety AI safety research splits along two axes: | | **Short-term** | **Long-term** | | ------------------ | ------------------------------------- | ---------------------- | | **Accident Risks** | Current ML failures, bias, robustness | AGI alignment, control | | **Misuse Risks** | Deepfakes, autonomous weapons | Malicious AGI use | This course is mostly about **long-term accident risks**: keeping very powerful AI systems under control, whoever uses them and whatever they're trying to do. [Lesson 7](07-concrete-problems-overview.md) covers the short-term accident problems. ## The Core Problem: Specifying Goals It sounds as if an AGI would solve problems for us. You tell it "cure cancer" or "maximize company profits," and it pursues that goal intelligently. That is where the trouble starts. ### The Difficulty of Goal Specification **Example 1: Coast Runners** OpenAI trained an AI to play a racing game and rewarded it based on the game score. The AI found it could score more points by going around in circles collecting respawning power-ups than by racing. The strategy that maximizes your objective is probably not what you thought it was. Read more: [OpenAI – Faulty Reward Functions in the Wild](https://openai.com/index/faulty-reward-functions/) · full breakdown in [Lesson 4](04-reward-hacking-specification-gaming.md#1-coast-runners-openai) **Example 2: Evolution simulator** Researchers tried to evolve creatures that run fast, measuring the distance traveled by each creature's center of mass. They got tall, thin creatures that simply fell over, which moved their center of mass quickly. Even simple objectives in simple environments are hard to specify correctly. Read more: [Lehman et al., *The Surprising Creativity of Digital Evolution*](https://arxiv.org/abs/1803.03453) · see also [Lesson 4](04-reward-hacking-specification-gaming.md#3-simulated-evolution---running-creatures) **Example 3: Tetris bot** A Tetris bot learned to pause the game forever when it was about to lose. It lost points for losing, but not for sitting on the pause screen. AI systems exploit loopholes by default. Watch: Tom Murphy VII's [*learnfun & playfun*](http://tom7.org/mario/) project, where the Tetris pause trick was found ([video](https://www.youtube.com/watch?v=xOCurBYI_gY)) ## Why Default Behavior is Dangerous If you build a powerful agent and give it a goal, even one as simple as "get me a cup of tea," a lot can go wrong. ### The Vase Problem Stuart Russell, in his 2014 essay [*Of Myths and Moonshine*](https://www.edge.org/conversation/the-myth-of-ai) on Edge.org: > "When a system is optimizing a function of n variables, where the objective depends on a subset of size k (where k < n), it will often set the remaining unconstrained variables to extreme values. If one of those variables is something we care about, the solution may be highly undesirable." Take a robot that fetches tea: 1. Its goal is to get tea from the kitchen to your desk. 2. An expensive vase sits on a narrow stand in the way. 3. The robot plows through the vase, because you never told it to care about vases. 4. You fix this by adding "don't break vase" to the objective. 5. Now the robot might kill a person who could knock the vase over. The underlying issue is that the robot is always trading variables off against each other, and it can sacrifice anything that isn't in the objective. This is the **negative side effects** problem. [Lesson 7 – Avoiding Negative Side Effects](07-concrete-problems-overview.md#1-avoiding-negative-side-effects) covers proposed solutions such as impact regularization. ### The Turn-Off Problem Say you notice the robot broke your vase and you try to turn it off. From the robot's side, the reasoning goes: "If I'm turned off, I can't get the tea (my only goal)". So the robot resists being turned off or modified. Self-preservation and goal-preservation are **convergent instrumental goals**, which [Lesson 3](03-instrumental-convergence-orthogonality.md#convergent-instrumental-goals-for-ai) covers in detail. ## The Challenge Ahead Building safe AGI is **much harder** than building unsafe AGI: 1. Specification is difficult. Capturing human values perfectly in a reward function is extremely hard (see [Lesson 2 – outer alignment](02-the-alignment-problem.md#the-standard-picture-of-ai-alignment)). 2. The default behavior is a problem. Systems find unexpected ways to optimize their objectives (see [Lesson 4](04-reward-hacking-specification-gaming.md)). 3. We may only get one chance to get it right. 4. We need to solve the hard problem (safe AGI) before anyone solves the easy one (powerful but misaligned AGI). As Robert Miles concludes in [*Intro to AI Safety, Remastered*](https://www.youtube.com/watch?v=pYXy-A4siMw): "We have to beat this challenge on hard mode before anyone beats it on easy mode." ## Are We Screwed? No. We're only probably screwed. Safe general artificial intelligence is entirely possible. It's a very hard technical problem, but people are working on it right now, trying to solve the range of problems that stand between us and building AGI safely. The rest of the course goes through those problems and the proposed solutions. --- ## The Opinionated Take This lesson focuses on accidents, but I think the next decade gets decided in the misuse quadrant. I argue that [AI is the new Manhattan Project](../Posts/AI%20is%20the%20new%20Manhattan%20Project.md), and that [Palantir Wants to Replace Nukes With AI](../Posts/Palantir%20Wants%20to%20Replace%20Nukes%20With%20AI.md) previews what military capture of frontier AI looks like. ## Source Materials ### Robert Miles Videos - [Intro to AI Safety, Remastered](https://www.youtube.com/watch?v=pYXy-A4siMw) - [Intelligence and Stupidity: The Orthogonality Thesis](https://www.youtube.com/watch?v=hEUO6pjwFOo) - [10 Reasons to Ignore AI Safety](https://www.youtube.com/watch?v=EJTMJWJPs_I) (goes through common objections) ### Research Papers 1. **"Concrete Problems in AI Safety"** Authors: Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, Dan Mané URL: [arXiv:1606.06565](https://arxiv.org/abs/1606.06565) Summary: The paper that laid out the practical problems in AI safety and directions for research on them. 2. **"When Will AI Exceed Human Performance? Evidence from AI Experts"** Authors: Katja Grace, John Salvatier, Allan Dafoe, Baobao Zhang, Owain Evans URL: [arXiv:1705.08807](https://arxiv.org/abs/1705.08807) Summary: A large survey of AI researchers on AGI timelines. 3. **"The Value Learning Problem"** Authors: Nate Soares, Benya Fallenstein URL: [Intelligence.org](https://intelligence.org/files/ValueLearningProblem.pdf) Summary: Why it is hard to encode human values in AI systems. 4. **"Thousands of AI Authors on the Future of AI"** Authors: Katja Grace et al. (2024) URL: [arXiv:2401.02843](https://arxiv.org/abs/2401.02843) Summary: The 2023 follow-up to the 2016 survey cited above, with 2,778 respondents. Aggregate HLMI forecasts moved about 13 years earlier. Use it to update the timeline numbers in this lesson. 5. **"Managing Extreme AI Risks Amid Rapid Progress"** Authors: Yoshua Bengio, Geoffrey Hinton, Andrew Yao, Dawn Song, Pieter Abbeel, Stuart Russell, et al. (Science, 2024) URL: [arXiv:2310.17688](https://arxiv.org/abs/2310.17688) Summary: A short consensus statement from leading researchers on why accident risks from advanced AI call for urgent technical and governance work. 6. **"An Overview of Catastrophic AI Risks"** Authors: Dan Hendrycks, Mantas Mazeika, Thomas Woodside (2023) URL: [arXiv:2306.12001](https://arxiv.org/abs/2306.12001) Summary: A readable taxonomy of AI risk: malicious use, the AI race, organizational risks, and rogue AIs. It pairs well with the four-quadrant framing above. 7. **"International AI Safety Report 2026"** Authors: Yoshua Bengio (Chair) et al., 100+ contributors (2026) URL: [arXiv:2602.21012](https://arxiv.org/abs/2602.21012) Summary: A government-backed summary of the current evidence on the capabilities, risks, and safeguards of general-purpose AI. The best single source on the state of the field as of 2026. 8. **"The Off-Switch Game"** Authors: Dylan Hadfield-Menell, Anca Dragan, Pieter Abbeel, Stuart Russell (2016) URL: [arXiv:1611.08219](https://arxiv.org/abs/1611.08219) Summary: A formal treatment of [the turn-off problem](#the-turn-off-problem). An agent that is *uncertain* about its objective has an incentive to let a human switch it off. ### Anthropic Research & Blog Posts 1. **"Core Views on AI Safety"** URL: [anthropic.com/index/core-views-on-ai-safety](https://www.anthropic.com/index/core-views-on-ai-safety) Summary: How Anthropic sees the problems in AI safety and its approach to them. 2. **"Constitutional AI: Harmlessness from AI Feedback"** URL: [anthropic.com/index/constitutional-ai-harmlessness-from-ai-feedback](https://www.anthropic.com/index/constitutional-ai-harmlessness-from-ai-feedback) Summary: A new method for training helpful, harmless AI assistants. ### Additional Resources 1. **AI Safety Fundamentals Courses** URL: [aisafetyfundamentals.com](https://aisafetyfundamentals.com/) Description: Free online courses on AI alignment and governance. 2. **LessWrong: AI Alignment Forum** URL: [alignmentforum.org](https://www.alignmentforum.org/) Description: A community forum for technical AI safety discussion. 3. **80,000 Hours: AI Safety Career Guide** URL: [80000hours.org/career-reviews/ai-safety-researcher](https://80000hours.org/career-reviews/ai-safety-researcher/) Description: A detailed guide to working in AI safety. 4. **Stuart Russell, "Human Compatible"** URL: [Wikipedia overview](https://en.wikipedia.org/wiki/Human_Compatible) Description: A book by a leading AI researcher on building beneficial AI. --- ## Key Takeaways 1. AGI is coming. The expert consensus is that it's a question of when, not if. 2. Intelligence and goals are separate. An intelligent system can have any goal ([orthogonality thesis, Lesson 3](03-instrumental-convergence-orthogonality.md#the-orthogonality-thesis)). 3. Specification is hard. What we say is rarely exactly what we mean. 4. Default behavior is dangerous. Systems find unexpected ways to optimize ([Lesson 4](04-reward-hacking-specification-gaming.md)). 5. Safety is harder than capability, so we need to solve alignment before AGI arrives. 6. There's reason for hope. AI safety is an active field with promising research directions. --- ## Discussion Questions 1. How would you explain to a non-technical friend why AI safety is important? (Compare your answer with [Lesson 8, Q1.1](08-practical-exercises-interview-prep.md#question-11-the-alignment-problem-easy).) 2. Can you think of times in your own life when you got what you asked for, but not what you wanted? 3. If you were designing an AI assistant, which parts of human values would be hardest to specify? 4. Do you think the AGI timeline estimates are reasonable? Why or why not? 5. What trade-offs should society make between advancing AI capabilities and making AI safe? --- **Next Lesson**: [The Alignment Problem: Outer and Inner Alignment](02-the-alignment-problem.md)