Existential Risk and the Future of Humanity: Lessons from AI, Pandemics & Nuclear Threats (Toby Ord)

Описание: How close are we to the end of humanity? Toby Ord, Senior Researcher at Oxford University’s AI Governance Initiative and author of The Precipice, argues that the odds of a civilization-ending catastrophe this century are roughly one in six. In this wide-ranging conversation, we unpack the risks that could end humanity’s story and explore why protecting future generations may be our greatest moral duty.

We explore:
• Why existential risk matters and what we owe the 10,000-plus generations who came before us
• Why Toby believes we face a one-in-six chance of civilizational collapse this century
• The four key types of AI risk: alignment failures, gradual disempowerment, AI-fueled coups, and AI-enabled weapons of mass destruction
• Why racing dynamics between companies and nations amplify those risks, and how an AI treaty might help
• How short-term incentives in democracies blind us to century-scale dangers, along with policy ideas to fix it
• The lessons COVID should have taught us (but didn’t)
• The hidden ways the nuclear threat has intensified as treaties lapse and geopolitical tensions rise
• Concrete steps each of us can take today to steer humanity away from the brink

Transcript: https://www.generalist.com/p/existent...

This episode is brought to you by:
Brex: The banking solution for startups: https://www.brex.com/mario

Timestamps
(00:00) Intro
(02:20) An explanation of existential risk, and the study of it
(06:20) How Toby’s interest in global poverty sparked his founding of Giving What We Can
(11:18) Why Toby chose to study under Derek Parfit at Oxford
(14:40) Population ethics, and how Parfit’s philosophy looked ahead to future generations
(19:05) An introduction to existential risk
(22:40) Why we should care about the continued existence of humans
(28:53) How fatherhood sparked Toby’s gratitude to his parents and previous generations
(31:57) An explanation of how LLMs and agents work
(40:10) The four types of AI risks
(46:58) How humans justify bad choices: lessons from the Manhattan Project
(51:29) A breakdown of the “unilateralist’s curse” and a case for an AI treaty
(1:02:15) Covid’s impact on our understanding of pandemic risk
(1:08:51) The shortcomings of our democracies and ways to combat our short-term focus
(1:14:50) Final meditations

Follow Toby Ord
Website: https://www.tobyord.com/
LinkedIn:   / tobyord  
X: https://x.com/tobyordoxford?lang=en
Giving What We Can: https://www.givingwhatwecan.org/

Resources and episode mentions
—Books—
• The Precipice: Existential Risk and the Future of Humanity: https://www.amazon.com/dp/0316484911
• Reasons and Persons: https://www.amazon.com/Reasons-Person...
• Practical Ethics: https://www.amazon.com/Practical-Ethi...

—People—
• Derek Parfit: https://en.wikipedia.org/wiki/Derek_P...
• Carl Sagan: https://en.wikipedia.org/wiki/Carl_Sagan
• Stuart Russell: https://en.wikipedia.org/wiki/Stuart_...

—Other resources—
• DeepMind: https://deepmind.google/
• OpenAI: https://openai.com/
• Manhattan Project: https://en.wikipedia.org/wiki/Manhatt...
• The Unilateralist’s Curse and the Case for a Principle of Conformity: https://nickbostrom.com/papers/unilat...
• The Nuclear Non-Proliferation Treaty (NPT), 1968: https://history.state.gov/milestones/...
• The Blitz: https://en.wikipedia.org/wiki/The_Blitz
• Operation Warp Speed: https://en.wikipedia.org/wiki/Operati...

Production and marketing by penname.co. For inquiries about sponsoring the podcast, email [email protected].