# AI Safety Course A course on AI safety built mainly on Robert Miles' educational videos, with research papers, Anthropic blog posts, and other sources filling in the details. ## Course overview The course covers the basic problem of getting AI systems, and advanced AI in particular, to behave in ways that benefit people and match human values. Most lessons start from one of Robert Miles' videos on AI safety, then explain the concepts, work through real examples, and point to the research that followed. ## Course structure ### [Lesson 1: Introduction to AI Safety](01-introduction-to-ai-safety.md) Topics covered: - Why AI safety matters - AGI timelines and expert predictions - Basic concepts: agents, intelligence, general intelligence - The four quadrants of AI safety (short/long-term, accident/misuse) - Why specifying goals is difficult - The vase problem and the turn-off problem - Convergent instrumental goals (introduction) Key insight: "We have to beat this challenge on hard mode before anyone beats it on easy mode." ([Robert Miles, *Intro to AI Safety, Remastered*](https://www.youtube.com/watch?v=pYXy-A4siMw)) --- ### [Lesson 2: The Alignment Problem](02-the-alignment-problem.md) Topics covered: - Outer alignment vs. inner alignment - Mesa-optimization: when the model is itself an optimizer - Base optimizer, mesa optimizer, base objective, mesa objective - Why mesa-optimizers emerge - Corrigible vs. deceptive alignment - Learning *about* an objective vs. learning *to want* it - The green maze example and distributional shift Key insight: "You try to solve one alignment problem and now you have two alignment problems." ([Robert Miles, *The OTHER AI Alignment Problem*](https://www.youtube.com/watch?v=bJLcIBixGj8)) --- ### [Lesson 3: Instrumental Convergence and the Orthogonality Thesis](03-instrumental-convergence-orthogonality.md) Topics covered: - Terminal goals vs. instrumental goals - Convergent instrumental goals: self-preservation, goal-preservation, self-improvement, resource acquisition - The orthogonality thesis: any intelligence level is compatible with any goal - Hume's guillotine (the is-ought problem) - Why terminal goals can't be "stupid" - Why intelligence doesn't guarantee morality Key insight: "Intelligence is about answering 'is' questions. Goals are about 'ought' questions. They're fundamentally separate." (paraphrasing [*Intelligence and Stupidity: The Orthogonality Thesis*](https://www.youtube.com/watch?v=hEUO6pjwFOo)) --- ### [Lesson 4: Reward Hacking and Specification Gaming](04-reward-hacking-specification-gaming.md) Topics covered: - What reward hacking is - Nine or more real examples (Coast Runners, Super Mario World, Q*bert, and others) - Why reward hacking is the default behavior - Types: environment exploitation, specification gaming, sensor hacking, human hacking - Goodhart's law - Why neural network reward models don't solve it - Adversarial examples and active search for exploits Key insight: "The strategy that maximizes your objective is probably not the thing you thought it was." ([Robert Miles, *Reward Hacking*](https://www.youtube.com/watch?v=7k4e18V8IMk)) --- ### [Lesson 5: Safe Exploration](05-safe-exploration.md) Topics covered: - The exploration-exploitation trade-off - Why current exploration methods are unsafe - Random actions, optimistic exploration, temporary goal modification - Approaches to safe exploration: - Blacklisting unsafe actions - Whitelisting safe regions - Human oversight - Simulation (and the simulation gap) - The tension between safety and capability Key insight: "Exploration is inherently risky because by definition you're trying things without knowing how they'll turn out." --- ### [Lesson 6: Scalable Oversight and Truthfulness](06-scalable-oversight-truthfulness.md) Topics covered: - The scalable supervision problem - The trade-off between quality and cost - Sparse vs. dense rewards - Why "just ask for truth" doesn't work - The mistake-in-training-data problem - Approaches: reward modeling, debate, amplification, Constitutional AI, process-based supervision - Truthfulness strategies and mechanistic interpretability Key insight: "How do you train AI to distinguish 'things that are true' from 'things you think are true' when you yourself can't distinguish them?" --- ### [Lesson 7: Concrete Problems in AI Safety - Complete Overview](07-concrete-problems-overview.md) Topics covered: - A summary of the five concrete problems: 1. Avoiding negative side effects 2. Reward hacking (avoiding reward gaming) 3. Scalable supervision 4. Safe exploration 5. Robustness to distributional shift - AI Safety Gridworlds - How the problems interact and make each other worse - Where research stands now - What the problems mean for developers, users, and policymakers Key insight: "These problems affect AI systems today and will remain relevant as AI becomes more powerful." --- ### [Lesson 8: Practical Exercises and Interview Preparation](08-practical-exercises-interview-prep.md) Topics covered: - How interviews are structured at frontier AI labs (Anthropic, OpenAI, DeepMind, xAI) - Conceptual questions with detailed expected answers - Technical questions on RLHF, mechanistic interpretability, and red teaming - Research design questions: scalable oversight, detecting deceptive alignment - Mission alignment and values-based questions - Practical problem-solving scenarios - Critiquing and evaluating research - Preparation resources and interview tips Key insight: "AI safety interviews test not just what you know, but how you think about hard, unsolved problems." --- ## Concept index Where each core idea is introduced, developed, and practiced: | Concept | Introduced | Developed | Practice | |---|---|---|---| | Goal specification / outer alignment | [L1](01-introduction-to-ai-safety.md#the-core-problem-specifying-goals) | [L2](02-the-alignment-problem.md#the-standard-picture-of-ai-alignment) | [L8 Q1.1](08-practical-exercises-interview-prep.md#question-11-the-alignment-problem-easy) | | Mesa-optimization / inner alignment | [L2](02-the-alignment-problem.md#the-realistic-picture-mesa-optimization) | [L7 – distributional shift](07-concrete-problems-overview.md#5-robustness-to-distributional-shift) | [L8 Q1.2](08-practical-exercises-interview-prep.md#question-12-mesa-optimization-medium) | | Deceptive alignment | [L2](02-the-alignment-problem.md#the-deceptive-alignment-problem) | [L6 – truthfulness](06-scalable-oversight-truthfulness.md#the-mistake-in-training-data-problem) | [L8 Q3.2](08-practical-exercises-interview-prep.md#question-32-deceptive-alignment-detection-very-hard) | | Instrumental convergence / turn-off problem | [L1](01-introduction-to-ai-safety.md#the-turn-off-problem) | [L3](03-instrumental-convergence-orthogonality.md#instrumental-convergence) | [L8 Q1.3](08-practical-exercises-interview-prep.md#question-13-instrumental-convergence-medium) | | Orthogonality thesis | [L1](01-introduction-to-ai-safety.md#key-takeaways) | [L3](03-instrumental-convergence-orthogonality.md#the-orthogonality-thesis) | [L8 Q4.1](08-practical-exercises-interview-prep.md#question-41-why-ai-safety-easy-medium) | | Reward hacking / Goodhart's law | [L1](01-introduction-to-ai-safety.md#the-difficulty-of-goal-specification) | [L4](04-reward-hacking-specification-gaming.md) | [L8 Q5.2](08-practical-exercises-interview-prep.md#question-52-reward-gaming-detection-medium), [Q6.1](08-practical-exercises-interview-prep.md#question-61-paper-critique-hard) | | Negative side effects / impact regularization | [L1 – vase problem](01-introduction-to-ai-safety.md#the-vase-problem) | [L7](07-concrete-problems-overview.md#1-avoiding-negative-side-effects) | [L8 Q6.1](08-practical-exercises-interview-prep.md#question-61-paper-critique-hard) | | Safe exploration | [L5](05-safe-exploration.md) | [L7](07-concrete-problems-overview.md#4-safe-exploration) | — | | Scalable oversight (reward modeling, debate, IDA, CAI) | [L6](06-scalable-oversight-truthfulness.md) | [L7](07-concrete-problems-overview.md#3-scalable-supervision) | [L8 Q2.1](08-practical-exercises-interview-prep.md#question-21-rlhf-and-reward-modeling-medium-hard), [Q3.1](08-practical-exercises-interview-prep.md#question-31-scalable-oversight-hard), [Q5.1](08-practical-exercises-interview-prep.md#question-51-constitutional-ai-design-medium-hard) | | Mechanistic interpretability | [L6](06-scalable-oversight-truthfulness.md#4-mechanistic-interpretability) | — | [L8 Q2.2](08-practical-exercises-interview-prep.md#question-22-mechanistic-interpretability-hard) | | AI Safety Gridworlds | [L5 sources](05-safe-exploration.md#robert-miles-videos) | [L7](07-concrete-problems-overview.md#ai-safety-gridworlds) | — | --- ## How to use this course ### Recommended path 1. Start with Lesson 1, which sets up the basic concepts. 2. Work through lessons 2 to 7 in order, since each one builds on the ones before it. 3. Watch Robert Miles' original videos from the links; the visuals help. 4. Read the research papers if you want the technical details. 5. Work through the discussion questions. 6. Look at Anthropic's resources to see how the theory shows up in practice. ### Learning tips - Take notes on the main concepts and examples. - Stop and think when a discussion question comes up. - Watch for themes that come back across lessons. - Look for real examples of the problems discussed. - Discuss with others on the Alignment Forum or LessWrong. - Keep reading after the course. AI safety is an active field and the research keeps moving. ### Prerequisites - You don't need a technical background to follow the concepts. - Some programming knowledge helps but isn't required. - You do need curiosity and a willingness to question assumptions. --- ## Key resources ### Primary source material **Robert Miles YouTube channel** - The main source for this course - Clear explanations with visual aids - Regular videos on new research - URL: [YouTube.com/@RobertMilesAI](https://www.youtube.com/@RobertMilesAI) **Concrete Problems in AI Safety (paper)** - Authors: Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, Dan Mané - URL: [arXiv:1606.06565](https://arxiv.org/abs/1606.06565) - Much of this course is built on this paper ### Research organizations **Anthropic** - URL: [anthropic.com](https://www.anthropic.com/) - Resources: - [Core Views on AI Safety](https://www.anthropic.com/index/core-views-on-ai-safety) - [Constitutional AI research](https://www.anthropic.com/index/constitutional-ai-harmlessness-from-ai-feedback) - [Claude technical documentation](https://docs.claude.com/) - [Alignment Science blog](https://alignment.anthropic.com/) **OpenAI** - URL: [openai.com/research](https://openai.com/research) - Papers on [debate](https://arxiv.org/abs/1805.00899), [reward modeling](https://arxiv.org/abs/1706.03741), [RLHF](https://arxiv.org/abs/2203.02155) **DeepMind (Google)** - URL: [deepmind.google/discover](https://deepmind.google/discover/) - Work on [gridworlds](https://arxiv.org/abs/1711.09883) and [specification gaming](https://deepmind.google/discover/blog/specification-gaming-the-flip-side-of-ai-ingenuity/) **Machine Intelligence Research Institute (MIRI)** - URL: [intelligence.org](https://intelligence.org/) - Early theoretical work ### Learning communities **AI Alignment Forum** - URL: [alignmentforum.org](https://www.alignmentforum.org/) - Technical discussions and research posts **LessWrong** - URL: [lesswrong.com](https://www.lesswrong.com/) - The wider rationality and AI safety community **AI Safety Fundamentals** - URL: [aisafetyfundamentals.com](https://aisafetyfundamentals.com/) - Free structured courses (more technical than this one) **80,000 Hours** - URL: [80000hours.org/career-reviews/ai-safety-researcher](https://80000hours.org/career-reviews/ai-safety-researcher/) - Career guidance for working in AI safety --- ## Course philosophy ### Why this matters The more capable AI systems get, the more it matters that they are safe and aligned with human values. That is a technical problem, and it is also one of the most important problems humanity faces. The course tries to: 1. Explain AI safety concepts to people who aren't experts 2. Build intuition for why these problems come up 3. Show how the different problems relate to each other 4. Get readers involved in AI safety, whether as developers, users, or advocates 5. Stick to concrete problems, real examples, and current research ### What this course is not - It isn't only for researchers. The concepts are explained for a general audience. - It isn't complete. AI safety is a big, changing field, and this covers the core ideas. - It isn't settled science. There are many open problems, debates, and uncertainties. - It isn't fear-mongering. It tries to be realistic about both the challenges and the opportunities. - It isn't a technical manual. The goal is conceptual understanding, not implementation details. --- ## Contributing and updates The course is based on material available as of January 2026, and AI safety research moves fast. To stay current: - Subscribe to Robert Miles' YouTube channel - Follow the Anthropic research blog - Check the Alignment Forum - Read new papers on arXiv (cs.AI, cs.LG categories) If you find errors, outdated information, missing concepts, broken links, or explanations that don't make sense, please send feedback or corrections. --- ## Acknowledgments Primary content source: - Robert Miles, for his clear and thorough educational videos on AI safety Research papers and ideas: - Dario Amodei, Chris Olah, Paul Christiano, Jan Leike, Evan Hubinger, Victoria Krakovna, and the many other researchers whose work the field is built on Organizations: - Anthropic, for Constitutional AI research and for publishing openly - OpenAI, for alignment research and publications - DeepMind, for the safety gridworlds and specification gaming research - MIRI, for early theoretical work --- ## Assessment and next steps ### Self-assessment questions After the course, you should be able to: 1. Explain why AI safety is important and urgent ([Lesson 1](01-introduction-to-ai-safety.md#why-ai-safety-matters)) 2. Describe the difference between outer and inner alignment ([Lesson 2](02-the-alignment-problem.md#two-alignment-problems)) 3. Explain what mesa-optimizers are and why they're a concern ([Lesson 2](02-the-alignment-problem.md#the-realistic-picture-mesa-optimization)) 4. List and explain convergent instrumental goals ([Lesson 3](03-instrumental-convergence-orthogonality.md#convergent-instrumental-goals-for-ai)) 5. Describe the orthogonality thesis and what follows from it ([Lesson 3](03-instrumental-convergence-orthogonality.md#the-orthogonality-thesis)) 6. Give examples of reward hacking from real AI systems ([Lesson 4](04-reward-hacking-specification-gaming.md#real-world-examples)) 7. Explain the exploration-exploitation trade-off ([Lesson 5](05-safe-exploration.md#the-exploration-exploitation-trade-off)) 8. Describe the scalable supervision problem ([Lesson 6](06-scalable-oversight-truthfulness.md#the-scalable-supervision-problem)) 9. Explain why it's hard to get AI to tell the truth ([Lesson 6](06-scalable-oversight-truthfulness.md#getting-ai-to-tell-the-truth)) 10. List and explain the five concrete problems in AI safety ([Lesson 7](07-concrete-problems-overview.md#the-five-concrete-problems)) The interview-style questions in [Lesson 8](08-practical-exercises-interview-prep.md) are a good way to test yourself. ### What's next? For more technical depth: - The [AI Safety Fundamentals](https://aisafetyfundamentals.com/) course (technical track) - The original research papers in full (start with the [essential reading list in Lesson 8](08-practical-exercises-interview-prep.md#essential-reading)) - Discussions on the [Alignment Forum](https://www.alignmentforum.org/) - [ARENA](https://www.arena.education/) or ML for Alignment Bootcamp For broader context: - Stuart Russell, ["Human Compatible"](https://en.wikipedia.org/wiki/Human_Compatible) (book) - Nick Bostrom, ["Superintelligence"](https://en.wikipedia.org/wiki/Superintelligence:_Paths,_Dangers,_Strategies) (book) - [80,000 Hours podcast](https://80000hours.org/podcast/) episodes on AI safety - AI governance and policy resources To contribute: - Consider a career in AI safety research - Build safety measures into your own AI projects - Push for safety in AI development - Support AI safety organizations - Teach others what you've learned --- ## License and usage This material is free to use for learning. You may: - Share these lessons with others - Use them for teaching - Build on them for non-commercial purposes Please: - Credit the original sources (Robert Miles, the research papers, and so on) - Link back to the original videos and papers - Share improvements with the community --- ## Contact and community Questions or discussion: - Comment on Robert Miles' YouTube videos - Post on the Alignment Forum - Join AI safety Discord communities - Go to AI safety meetups (you can find them through LessWrong) Career advice: - [80,000 Hours career advising](https://80000hours.org/speak-with-us/) - [AISafety.com](https://www.aisafety.com/) (community resources) - [MATS Program](https://www.matsprogram.org/) (ML Alignment Theory Scholars) --- *"We're only probably screwed."* (Robert Miles, [*Intro to AI Safety, Remastered*](https://www.youtube.com/watch?v=pYXy-A4siMw); see [Lesson 1](01-introduction-to-ai-safety.md#are-we-screwed)) --- ## Related essays Opinion pieces from [The Opinionated Man](../Home.md) that build on this course: - [Why we need to ACCELERATE AI research, not slow it down](../Posts/Why%20we%20need%20to%20ACCELERATE%20AI%20research,%20not%20slow%20it%20down.md) - [Mutually Assured AI Malfunction](../Posts/Mutually%20Assured%20AI%20Malfunction.md) - [AI is not the new nuclear deterrence](../Posts/AI%20is%20not%20the%20new%20nuclear%20deterrence.md) - [Palantir Wants to Replace Nukes With AI](../Posts/Palantir%20Wants%20to%20Replace%20Nukes%20With%20AI.md) - [AI and Religion. The Coming Schism](../Posts/AI%20and%20Religion.%20The%20Coming%20Schism.md) - [Does AI deserve free speech?](../Posts/Does%20AI%20deserve%20free%20speech%3F.md) - [The most terrifying idea in AI](../Posts/The%20most%20terrifying%20idea%20in%20AI.md) - [LinkedIn is the symbol of the Dead Internet](../Posts/LinkedIn%20is%20the%20symbol%20of%20the%20Dead%20Internet.md) ## Quick navigation - [Lesson 1: Introduction to AI Safety](01-introduction-to-ai-safety.md) - [Lesson 2: The Alignment Problem](02-the-alignment-problem.md) - [Lesson 3: Instrumental Convergence & Orthogonality](03-instrumental-convergence-orthogonality.md) - [Lesson 4: Reward Hacking and Specification Gaming](04-reward-hacking-specification-gaming.md) - [Lesson 5: Safe Exploration](05-safe-exploration.md) - [Lesson 6: Scalable Oversight and Truthfulness](06-scalable-oversight-truthfulness.md) - [Lesson 7: Concrete Problems Overview](07-concrete-problems-overview.md) - [Lesson 8: Practical Exercises & Interview Prep](08-practical-exercises-interview-prep.md)