Future Of Life Award 2020

The Future of Life Award 2020

Celebrating the contributions of Viktor Zhdanov and William Foege

Read the press release in Russian

Imagine how you’d react to the headline “Mutated coronavirus discovered, killing 30% of those infected.” Smallpox was such a lethal virus, estimated to have killed 500 million people in its last century. Today, the 41st anniversary of its eradication is celebrated by honoring two of the heroes who made it happen: Dr. William Foege and Dr. Viktor Zhdanov will share the $100,000 Future of Life Award in an online ceremony including Bill Gates, Dr. Anthony Fauci and freshly minted Nobel Laureate Dr. Jennifer Doudna.


Viktor Zhdanov


William Foege

The lessons learned from defeating smallpox are highly relevant to the covid pandemic: “We’re all indebted to Bill Foege and Viktor Zhdanov for their critical contributions to the eradication of smallpox, which demonstrated the immense value of science and international collaboration for fighting disease”, said António Guterres, Secretary General, United Nations. “Bill Foege’s advocacy of surveillance and containment (applying water to the house on fire rather than the whole neighborhood) was very significant for the overall success”, said Harvard professor George Church. Bill played a pivotal role in eradicating smallpox and deserves the Future of Life Award. I look forward to seeing what the next generation of public health students will accomplish—by following in this giant’s footsteps”,  said Bill Gates. 

Viktor Zhdanov has been called “the best person who ever lived” by Oxford professor Will MacAskill, for successfully persuading the World Health Assembly to initiate an eradication campaign where the USA and the Soviet Union collaborated despite the Cold War. Dr. MacAskill added: Smallpox was one of the worst diseases to ever befall the human race, and its eradication is one of the greatest achievements of humanity. Bill Foege and Viktor Zhdanov should be celebrated for their contributions, and should inspire us today to take effective action to tackle the world’s most pressing problems.” “They are phenomenal examples of what it means to harness science for global health”, Bill Gates added.

UNICEF estimates that smallpox eradication has saved close to 200 million lives – so far. “In selecting Bill Foege and Viktor Zhdanov as recipients of its prestigious 2020 award, the Future of Life Institute reminds us that seemingly impossible problems can be solved when science is respected, international collaboration is fostered, and goals are boldly defined. As we celebrate this achievement quarantined in our homes and masked outdoors, what message could be more obvious or more audacious?”, said Dr. Rachel Bronson, President & CEO of the Bulletin of the Atomic Scientists. George Church gives examples of such seemingly impossible problems: “Smallpox eradication is a model for elimination of other human-specific diseases in the future, such as polio, measles, mumps, rubella and syphilis.” 

While serving as the Soviet Union’s Deputy Minister of Health, Dr. Viktor Zhdanov persuasively argued at the Eleventh World Health Assembly meeting in 1958 that the world could eradicate smallpox within a decade with a united effort, and successfully lobbied the Soviet Union to donate 25 million doses of the smallpox vaccine to kickstart the effort in developing countries. The World Health Assembly accepted his proposal in 1959 under Resolution WHA11.54. Viktor Zhdanov passed away in 1987, and his award will be received by his sons Viktor and Michael in his memory. 

While working for the Centers for Disease Control in Africa as Chief of the Smallpox Eradication Program, Dr. Bill Foege developed the highly successful surveillance and “ring vaccination” strategy to contain smallpox spread. This greatly reduced the number of vaccinations needed, ensuring that the limited resources available sufficed to make smallpox the first infectious disease to be eradicated in human history. 

The award: The Future of Life Award honors those who take exceptional measures to safeguard the collective future of humanity.  The 2017 award honored Vasili Arkhipov for single-handedly preventing a Soviet nuclear attack against the US in 1962, and the 2018 award celebrated Stanislav Petrov for helping avert an accidental nuclear war in 1983. The 2019 award honored Dr. Matthew Meselson for his remarkable contributions to getting biological weapons banned and focusing biology on curing rather than killing. The award is funded by Skype-cofounder Jaan Tallinn and presented by the Future of Life Institute (FLI), a non-profit organization advocating positive technology use. “By eradicating smallpox and banning bioweapons, biology is a role model for other sciences struggling to help and not harm”, said FLI president  Max Tegmark. This dual nature of science was echoed by MIT biology Professor Jonathan King: “The victory over smallpox highlights both the power of science to help humanity, and also how science could do more good if we didn’t divert vast resources to developing new weapons of mass destruction. We need the world’s scientific community to draw inspiration from Foege and Zhdanov and press our political leaders to fund healthcare, not warfare.”

FLI’s Dr. Emilia Javorsky, a biotech entrepreneur, added: “As we embark on our journey to eradicate COVID in an environment plagued by mistrust and misinformation, Foege and Zhdanov have shown us that the seemingly impossible is possible. Their example illustrates the importance of rebuilding and restoring trust in science, between nations, and perhaps most powerfully, between each other.”





FLI Podcast – In Conversation with the Award Recipients

Featured Books

Viktor Zhdanov—my Husband, Elena Tatulova’s Diary

Viktor Zhdanov’s wife (under the pseudonym of Elena) wrote a book about her husband’s life that the team at FLI has translated into English. It includes many fictitious names but tells a real story, and gives a taste of who Viktor Zhdanov was like as a person, while diving deeper into his work. You can read it here.

House on Fire: The Fight to Eradicate Smallpox by William Foege

A story of courage and risk-taking, House on Fire tells how smallpox, a disease that killed, blinded, and scarred millions over centuries of human history, was completely eradicated in a spectacular triumph of medicine and public health. Part autobiography, part mystery, the story is told by a man who was one of the architects of a radical vaccination scheme that became a key strategy in ending the horrible disease when it was finally contained in India. You can find the book on Amazon here.

History of the Award

The Future of Life Award is a $50,000 prize given to an individual who, without receiving much recognition at the time, has helped make today dramatically better than it may otherwise have been. We are confident that there are many unsung heroes out there, who have done incredible work to ensure a beneficial future of life on Earth. We need your help to ensure they get the recognition and honor they deserve.

2019 Winner

Matthew Meselson



Dr. Meselson was a driving force behind the 1972 Biological Weapons Convention, an international ban that has prevented one of the most inhumane forms of warfare known to humanity. April 9th marked the eve of the Convention’s anniversary. Meselson’s long career is studded with highlights: proving Watson and Crick’s hypothesis on DNA structure, solving the Sverdlovsk Anthrax mystery, ending the use of Agent Orange in Vietnam. But it is above all his work on biological weapons that makes him an international hero.

The 2019 Future of Life Award to Dr. Matthew Meselson was covered in Vox, and you can see a video about Meselson’s life below.

2018 Winner

Stanislav Petrov



One of the closest calls occurred thirty-five years ago, on September 26, 1983, when Stanislav Petrov chose to ignore the Soviet early-warning detection system that had erroneously indicated five incoming American nuclear missiles. With his decision to ignore algorithms and instead follow his gut instinct, Petrov helped prevent an all-out US-Russian nuclear war, as detailed in the documentary film “The Man Who Saved the World”.

The 2018 Future of Life Award to Stanislav Petrov was covered in VoxDaily Mail, and The Daily Star.

2017 Winner

Vasili Arkhipov



Vasili Arkhipov single-handedly prevented nuclear war during the height of the Cuban Missile Crisis. Arkhipov’s submarine captain, thinking their sub was under attack by American forces, wanted to launch a nuclear weapon at the ships above. Arkhipov, with the power of veto, said no, thus averting nuclear war.

The 2017 Future of Life Award to Vasili Arkhipov was covered in The TimesThe GuardianThe Independent, and The Atlantic.

Help Us Find Our Next Unsung Hero

In the form below, please explain how your candidate has helped (indirectly or directly) save thousands or millions of lives, or helped make today dramatically better than it may otherwise have been. Please add links and evidence where possible.

If we decide to give the award to your nominee, you will receive a $3,000 prize from FLI for your contribution!



If you have documentation that you can’t upload in the form below, or if you are a researcher who would like to apply for a grant to research a candidate, please email dnicholson@post.harvard.edu.

If you don’t know of any candidates, we encourage you to share this page on social media and websites like Reddit, so we can reach as many people as possible in our quest to find humanity’s unsung heroes. If, by sharing this, you directly lead someone else to submit a winning candidate, we will give you credit in the nominations page.

Sign Up for our Newsletter

Future Of Life Award 2019

The Future of Life Award 2019

Celebrating the contributions of Matthew Meselson

On April 9th, Dr. Matthew Meselson received the $50,000 Future of Life Award at a ceremony at the University of Boulder’s Conference on World Affairs. Dr. Meselson was a driving force behind the 1972 Biological Weapons Convention, an international ban that has prevented one of the most inhumane forms of warfare known to humanity. April 9th marked the eve of the Convention’s 47th anniversary.

Meselson’s long career is studded with highlights: proving Watson and Crick’s hypothesis on DNA structure, solving the Sverdlovsk Anthrax mystery, ending the use of Agent Orange in Vietnam. But it is above all his work on biological weapons that makes him an international hero.

“Through his work in the US and internationally, Matt Meselson was one of the key forefathers of the 1972 Biological Weapons Convention,” said Daniel Feakes, Chief of the Biological Weapons Convention Implementation Support Unit. “The treaty bans biological weapons and today has 182 member states. He has continued to be a guardian of the BWC ever since. His seminal warning in 2000 about the potential for the hostile exploitation of biology foreshadowed many of the technological advances we are now witnessing in the life sciences and responses which have been adopted since.”

Meselson became interested in biological weapons during the 60s, while employed with the U.S. Arms Control and Disarmament Agency. It was on a tour of Fort Detrick, where the U.S. was then manufacturing anthrax, that he learned the motivation for developing biological weapons: they were cheaper than nuclear weapons. Meselson was struck, he says, by the illogic of this — it would be an obvious national security risk to decrease the production cost of WMDs.

The use of biological weapons was already prohibited by the 1925 Geneva Protocol, an international treaty that the U.S. had never ratified. So Meselson wrote a paper, “The United States and the Geneva Protocol,” outlining why it should do so. Meselson knew Henry Kissinger, who passed his paper along to President Nixon, and by the end of 1969 Nixon renounced biological weapons.

Next came the question of toxins — poisons derived from living organisms. Some of Nixon’s advisors believed that the U.S. should renounce the use of naturally derived toxins, but retain the right to use artificial versions of the same substances. It was another of Meselson’s papers, “What Policy for Toxins,” that led Nixon to reject this arbitrary distinction and to renounce the use of all toxin weapons.

On Meselson’s advice, Nixon had resubmitted the Geneva Protocol to the Senate for approval. But he also went beyond the terms of the Protocol — which only ban the use of biological weapons — to renounce offensive biological research itself. Stockpiles of offensive biological substances, like the anthrax that Meselson had discovered at Fort Detrick, were destroyed.

Once the U.S. adopted this more stringent policy, Meselson turned his attention to the global stage. He and his peers wanted an international agreement stronger than the Geneva Protocol, one that would ban stockpiling and offensive research in addition to use and would provide for a verification system. From their efforts came the Biological Weapons Convention, which was signed in 1972 and is still in effect today.

“Thanks in significant part to Professor Matthew Meselson’s tireless work, the world came together and banned biological weapons, ensuring that the ever more powerful science of biology helps rather than harms humankind. For this, he deserves humanity’s profound gratitude,” said former UN Secretary-General Ban Ki-Moon.

Meselson has said that biological warfare “could erase the distinction between war and peace.” Other forms of war have a beginning and an end — it’s clear what is warfare and what is not. Biological warfare would be different: “You don’t know what’s happening, or you know it’s happening but it’s always happening.”

And the consequences of biological warfare can be greater, even, than mass destruction; Attacks on DNA could fundamentally alter humankind. FLI honors Matthew Meselson for his efforts to protect not only human life but also the very definition of humanity.

Said Astronomer Royal Lord Martin Rees, “Matt Meselson is a great scientist — and one of very few who have been deeply committed to making the world safe from biological threats. This will become a challenge as important as the control of nuclear weapons — and much more challenging and intractable. His sustained and dedicated efforts fully deserve wider acclaim.”

“Today biotech is a force for good in the world, associated with saving rather than taking lives, because Matthew Meselson helped draw a clear red line between acceptable and unacceptable uses of biology”, added MIT Professor and FLI President Max Tegmark. “This is an inspiration for those who want to draw a similar red line between acceptable and unacceptable uses of artificial intelligence and ban lethal autonomous weapons.

To learn more about Matthew Meselson, listen to FLI’s two-part podcast featuring him in conversation with Ariel Conn and Max Tegmark. In Part One, Meselson describes how he helped prove Watson and Crick’s hypothesis of DNA structure and recounts the efforts he undertook to get biological weapons banned. Part Two focuses on three major incidents in the history of biological weapons and the role played by Meselson in resolving them.

Publications by Meselson include:

History of the Award

The Future of Life Award is a $50,000 prize given to an individual who, without receiving much recognition at the time, has helped make today dramatically better than it may otherwise have been. We are confident that there are many unsung heroes out there, who have done incredible work to ensure a beneficial future of life on Earth. We need your help to ensure they get the recognition and honor they deserve.

2018 Winner

Stanislav Petrov



One of the closest calls occurred thirty-five years ago, on September 26, 1983, when Stanislav Petrov chose to ignore the Soviet early-warning detection system that had erroneously indicated five incoming American nuclear missiles. With his decision to ignore algorithms and instead follow his gut instinct, Petrov helped prevent an all-out US-Russian nuclear war, as detailed in the documentary film “The Man Who Saved the World”.

The 2018 Future of Life Award to Stanislav Petrov was covered in VoxDaily Mail, and The Daily Star.

2017 Winner

Vasili Arkhipov



Vasili Arkhipov single-handedly prevented nuclear war during the height of the Cuban Missile Crisis. Arkhipov’s submarine captain, thinking their sub was under attack by American forces, wanted to launch a nuclear weapon at the ships above. Arkhipov, with the power of veto, said no, thus averting nuclear war.

The 2017 Future of Life Award to Vasili Arkhipov was covered in The TimesThe GuardianThe Independent, and The Atlantic.

Help Us Find Our Next Unsung Hero

In the form below, please explain how your candidate has helped (indirectly or directly) save thousands or millions of lives, or helped make today dramatically better than it may otherwise have been. Please add links and evidence where possible.

If we decide to give the award to your nominee, you will receive a $3,000 prize from FLI for your contribution!



If you have documentation that you can’t upload in the form below, or if you are a researcher who would like to apply for a grant to research a candidate, please email dnicholson@post.harvard.edu.

If you don’t know of any candidates, we encourage you to share this page on social media and websites like Reddit, so we can reach as many people as possible in our quest to find humanity’s unsung heroes. If, by sharing this, you directly lead someone else to submit a winning candidate, we will give you credit in the nominations page.

Sign Up for our Newsletter

Augmented Intelligence Summit

 

Augmented Intelligence Summit

Steering the Future of AI

March 28 – 31, 2019 | 1440 Multiversity | Tuition $495

Today many of the concepts, consequences, and possibilities involved in a future with advanced AI feel distant, uncertain, and abstract. No one has all the answers about how to ensure that powerful AI in the future is beneficial, either in terms of technical implementation or in terms of transference to the domains of law, regulations and policy, industry best practices, or society at large. There are a number of organizations and initiatives that are working on the issues of AI safety, ethics, and governance.

Joining these efforts with a distinct role that bridges academia and industry, the Augmented Intelligence Summit offers a unique, inter-disciplinary approach to learning and creating solutions in this space. We ask: is it possible to develop a collective, concrete, realistic vision of a positive AI future that can inform policy, the development of the industry, and academic research, and can it be done inclusively? Our hypothesis is that this is indeed possible, and we have designed an experiment to test it.

The Summit will provide the tools and framework for a multi-disciplinary group of policy, research, and business leaders to imagine and interact with a simulated model of a positive future for our global society and its major interconnected systems – the economy, education, work, and political structures – through the lens of advanced AI. AIS utilizes a range of collaborative design tools and immersive exercises, including:

  • Scenario simulation, war-game style
  • World building and collaborative strategic planning exercises
  • Presentations on models of economic, social and complex systems to inform our work

The Summit’s high-level goals are about increasing systems-literacy and improving long-term thinking and decision making, which we see as critical skills for navigating the future of advanced AI and its impact on society. In addition to presentations on the state of AI and its trajectory, immersive exercises at the Summit relate to a fictional future world with advanced AI that is both plausible and aspirational. By mentally committing to a specific version of the future, our creativity, our ability to identify important variables, and our overall problem-solving capacity can be enhanced to help us better imagine the societal challenges, opportunities, and needs that will face us and the planning that must be done.

Featured Speakers

Max Tegmark

MIT | FLI

David Krakauer

Santa Fe Institute

Allison Duettmann

Foresight Institute

Stuart Russell

UC Berkeley | CHAI

Gaia Dempsey

7th Future

Peter Eckersley

Partnership on AI

Marie-Therese Png

Oxford | DeepMind

Anthony Aguirre

UC Santa Cruz | FLI | FXQi

Marc Fleurbaey

Princeton

Ariel Conn

FLI

Michael Page

Center for Security and Emerging Technology

Meia Chita-Tegmark

Tufts | FLI

David Haussler

UC Santa Cruz | QB3

Shahar Avin

CSER

Lou Viquerat de Kerhuelvez

Foresight Institute

 

Bios for the speakers can be found here.

Agenda

Thursday, March 28th

5:30 pm – 9:00 pm

Opening Keynote: Modeling, Simulation & Changing the Rules of the Game | Gaia Dempsey

Gaia Dempsey is an entrepreneur and pioneer in the field of augmented reality (AR). She is the founder and CEO of 7th Future, a consultancy that partners and co-invests with technology leaders and communities to build and launch global-impact innovation models.

World Building Breakout: “A Day in the Life” Exercise

Small groups imagine a day in the life of a character they co-create that lives on a plausible, aspirational version of Earth in 2045, which we’ve named Earth 2045. The goal of the exercise is to encourage participants to place themselves within the experience of Earth 2045, get familiar with the practice of creating a fictional future, and get to know each other (break the ice).

Friday, March 29th

9:00 am – 9:00 pm

Keynote: AI Futures — Where Are We Headed and How Do We Steer? | Stuart Russell

Stuart will frame the problem we are working to solve at AIS, namely that advanced AI technology represents a vast amplifying capacity for human beings that could veer into dangerous uncharted territory with vast societal implications.

Stuart is a renowned professor and former chair of the electrical engineering and computer sciences department at the University of California, Berkeley. At UC Berkeley, he holds the Smith-Zadeh Chair in engineering and is the director of the Center for Human-Compatible AI.

What’s the Future of Healthcare? | David Haussler

David will present a vision of a plausible, aspirational healthcare system could look like by 2045.

David Haussler serves as the Distinguished Professor of Biomolecular Engineering at the University of California, Santa Cruz, where he directs the Center for Biomolecular Science and Engineering and the California Institute for Quantitative Biosciences (QB3). He is also the scientific co-director for the California Institute for Quantitative Biomedical Research.

Introduction to World Building | Anthony Aguirre

Anthony Aguirre is a theoretical cosmologist and a Harvard-trained professor of physics at the  University of California, Santa Cruz. He cofounded and is the associate scientific director of the nonprofit organization  Foundational Questions Institute and is the cofounder of the Future of Life Institute.

World Building Breakout: Designing Interconnected Systems | Ariel Conn

Small groups collaborate on describing the systems that would need to be in place on Earth 2045 to move toward their aspirational vision. Each group focuses on a particular lens, such as neuroethics, democratic processes, economic justice, education, healthcare, manufacturing, etc.

Ariel Conn is the Director of Media and Outreach for the Future of Life Institute. Her work covers a range of fields, including artificial intelligence (AI) safety, AI policy, lethal autonomous weapons, nuclear weapons, biotechnology, and climate change.

What’s the Future of Criminal Justice? | Peter Eckersley

Peter is Director of Research at the Partnership on AI, a collaboration between the major technology companies, civil society and academia to ensure that AI is designed and used to benefit humanity. He leads PAI’s research on machine learning policy and ethics.

Synthesis Session

Facilitation of the large group, sharing and combining the insights they developed in the system-building exercise, identifying areas of confluence and conflict. The evolving description of Earth 2045 is enriched, including a unified timeline of events.

War Game Simulation ­| Allison Duettmann & Lou Viquerat

The session begins with an “Intro to Existential Hope & Strengthening Civilization” talk from Allison. Next, Red Teams and Blue Teams are created to attack and defend Earth 2045 in multiple rounds.

Allison is a researcher and program coordinator at Foresight Institute, a non-profit institute for technologies of fundamental importance for the future of life. Her research focuses on the reduction of existential risks, especially from artificial general intelligence. 

Lou is the Director of Development at Foresight Institute. Lou’s research focuses on alternative governance, justice, and economic systems for a more collaborative, sustainable, and global future society.

Saturday, March 30th

9:00 am – 5:30 pm

Keynote: Complex Adaptive System Modeling — Origins and Applications | David Krakauer

David will orient us toward solution paths for the problem Stuart outlined on Friday, namely the power of systems thinking for improving our understanding of the world, as well as our decision making, long-term thinking, and coordination capacity.

David is President and Professor of Complex Systems at the Santa Fe Institute. His research explores the evolution of intelligence on earth. This includes studying the evolution of genetic, neural, linguistic, social and cultural mechanisms supporting memory and information processing, and exploring their shared properties.

Keynote: The Value of Non-Zero Sum Dynamics | Michael Page

Michael focuses on the long-term social implications of the development and use of advanced artificial intelligence systems. He is policy and ethics advisor at OpenAI a nonprofit AI research company, dedicated to charting a path to safe AI.

Keynote: TBD | Shahar Avin

Scenario Simulation | Shahar Avin

Groups of people on Earth 2045 pursue goals and report back if they are successful. (Similar to a D&D quest).

Shahar is a postdoctoral researcher at the Centre for the Study of Existential Risk (CSER). He works with CSER researchers and others in the global catastrophic risk community to identify and design risk prevention strategies.

Keynote: Redefining Social Welfare: Bridging Preferences Across an Impossible Chasm | Marc Fleurbaey

Marc will provide an overview of contemporary economic theories of justice and introduce us to the most important and promising ideas for social progress identified by the International Panel on Social Progress on structural and systemic issues for the long-term future. Finally, Marc will explore new potential solutions that become viable in economies powered by advanced AI.

Marc is the Robert E. Kuenne Professor in Economics and Humanistic Studies, and a professor of Public Affairs and the University Center for Human Values at Princeton University.

Sunday, March 31st

9:00 am – 1:30 pm

Keynote: TBD | Max Tegmark

Max is a professor doing AI and physics research at MIT, and advocates for positive use of technology as president of the Future of Life Institute.

Global Ethics in AI Development | Marie-Therese Png

Marie-Therese is a doctoral candidate at the Oxford Internet Institute, and PhD research intern at DeepMind Ethics and Society. Her research focuses are Globally Beneficial AI, Intercultural AI Ethics, and Global Justice.

Organizers

Anthony Aguirre, PhD

Physicist | Cofounder of Future of Life Institute | Cofounder of Foundational Questions Institute

Anthony Aguirre is a theoretical cosmologist and a Harvard-trained professor of physics at the  University of California, Santa Cruz. He cofounded and is the associate scientific director of the nonprofit organization  Foundational Questions Institute and is the cofounder of the Future of Life Institute.

He received his doctorate in astronomy from Harvard University in 2000 and then spent three years as a member of the  Institute for Advanced Study  in Princeton, New Jersey, before accepting a professorship in the physics department of the University of California, Santa Cruz.

He has worked on a wide variety of topics in theoretical cosmology (the study of the formation, nature, and evolution of the universe), including the early universe and inflation, gravity physics, first stars, the intergalactic medium, galaxy formation, and black holes.

Gaia Dempsey

Founder & CEO of 7th Future

Gaia Dempsey is an entrepreneur and pioneer in the field of augmented reality (AR). She was a cofounder and former managing director at DAQRI, an augmented reality hardware and software company that delivers a complete professional AR platform to the industrial and enterprise market.

Gaia is currently the founder and CEO of 7th Future, a consultancy that partners and co-invests with technology leaders and communities to build and launch global-impact innovation models, with a commitment to openness, integrity, resilience, and long-term thinking.

William Dolphin

CEO of CueSquared

Dr. William Dolphin is an accomplished chief executive officer and senior executive with a history of leading the management and operations of international private and public companies in a range of industries including biotechnology, pharmaceutical, medical device, and information technology.

Currently, he is chief executive officer at CueSquared, which brings developments in artificial intelligence and machine learning to bear on healthcare applications—in particular, to expedite business processes, facilitate access to healthcare, and improve outcomes.

Dr. Dolphin brings a wealth of knowledge and an extensive history of building and growing successful, global businesses, especially to facilitate the delivery of healthcare and improved patient outcomes. He has served as chief executive officer of numerous medical and information technology companies around the world including Omedix, SonaMed Corporation, SpectraNet Ltd., Visiomed Group, and Avita Medical. Additionally, Dr. Dolphin has served as a member of the board of directors for over a dozen public and private companies worldwide.

Tasha McCauley

Board Director of GeoSim Systems

Tasha McCauley is a technology entrepreneur and robotics expert. She is board director at GeoSim Systems, a company centering on a new technology that produces high-resolution, fully interactive virtual models of cities. This novel approach to “reality capture” yields the most detailed VR environments ever created out of data gathered from the real world.

Tasha is also the cofounder of Fellow Robots, a robotics company based at NASA Research Park in Silicon Valley. Formerly on the faculty of Singularity University, she taught students about robotics and was director of the Autodesk Innovation Lab.

She sits on the board of directors of the Ten to the Ninth Plus Foundation, an organization empowering exponential technological change worldwide. Tasha enjoys building things and discovering how things work. She spends a lot of time thinking about how to make user interfaces that facilitate and enhance creativity.


In partnership with





Location

1440 Multiversity | Scotts Valley, CA

Redwood trails, copper-adorned buildings with majestic views, and decks built around sky-high trunks keep you wrapped in nature while on 1440 Multiversity’s 75-acre campus.

Part wellness resort, part Ted-talk auditorium, part conference center, the campus features state-of-the-art classrooms, meeting spaces, and accommodations tucked away in a lush forest between Santa Cruz and Silicon Valley. With locally sourced meals, holistic-health classes, Fitness Center, infinity-edge hot tub, and more, your stay at 1440 is designed to nurture, educate, and inspire.

Nearby Airports:

Mineta San Jose (SJC): 29 miles | approximately 40 – 60 minutes

San Fransisco (SFO): 60 miles | approximately 60 – 75 minutes

Oakland (OAK): 60 miles | approximately 60 – 90 minutes

Transportation suggestions to and from airports will be sent to registered participants prior to the event. 

 

Beneficial AGI 2019

BENEFICIAL AGI 2019

After our Puerto Rico AI conference in 2015 and our Asilomar Beneficial AI conference in 2017, we returned to Puerto Rico at the start of 2019 to talk about Beneficial AGI. We couldn’t be more excited to see all of the groups, organizations, conferences and workshops that have cropped up in the last few years to ensure that AI today and in the near future will be safe and beneficial. And so we now wanted to look further ahead to artificial general intelligence (AGI), the classic goal of AI research, which promises tremendous transformation in society. Beyond mitigating risks, we want to explore how we can design AGI to help us create the best future for humanity.

We again brought together an amazing group of AI researchers from academia and industry, as well as thought leaders in economics, law, policy, ethics, and philosophy for five days dedicated to beneficial AI. We hosted a two-day technical workshop to look more deeply at how we can create beneficial AGI, and we followed that with a 2.5-day conference, in which people from a broader AI background considered the opportunities and challenges related to the future of AGI and steps we can take today to move toward an even better future. Honoring the FrieNDA for the conference, we have only posted videos and slides below with approval of the speaker. (See the high-res group photo here.)

Conference Schedule

Friday, January 4

18:00 Early drinks and appetizers
19:00 Conference welcome reception & buffet dinner
20:00 Poster session

Saturday, January 5

Session 1 — Overview Talks

9:00 Welcome & summary of conference plan | FLI Team

Destination:

9:05 Framing the Challenge | Max Tegmark (FLI/MIT)
9:15 Alternative destinations that are popular with participants. What do you prefer?

Technical Safety:

10:00 Provably beneficial AI | Stuart Russell (Berkeley/CHAI) (pdf) (video)
10:30 Technical workshop summary | David Krueger (MILA) (pdf) (video)
10:50 Coffee break

11:00 Strategy/Coordination:

AI strategy, policy, and governance | Allan Dafoe (FHI/GovAI) (pdf) (video)
Towards a global community of shared future in AGI | Brian Tse (FHI) (pdf) (video)
Strategy workshop summary | Jessica Cussins (FLI) (pdf) (video)

12:00 Lunch
13:00 Optional breakouts & free time

Session 2 — Destination | Session Chair: Max Tegmark

16:00 Talk: Summary of discussions and polls about long-term goals, controversies

Panels on the controversies where the participant survey revealed the greatest disagreements:

16:10 Panel: Should we build superintelligence? (video)

Tiejun Huang (Peking)
Tanya Singh (FHI)
Catherine Olsson (Google Brain)
John Havens (IEEE)
Moderator:
Joi Ito (MIT)

16:30 Panel: What should happen to humans? (video)

José Hernandez-Orallo (CFI)
Daniel Hernandez (OpenAI)
De Kai (ICSI)
Francesca Rossi (CSER)
Moderator:
Carla Gomes (Cornell)

16:50 Panel: Who or what should be in control? (video)

Gaia Dempsey (7th Future)
El Mahdi El Mhamdi (EPFL)
Dorsa Sadigh (Stanford)
Moderator: Meia Chita-Tegmark (FLI)

17:10 Panel: Would we prefer AGI to be conscious? (video)

Bart Selman (Cornell)
Hiroshi Yamakawa (Dwango AI)
Helen Toner (GovAI)
Moderator: Andrew Serazin (Templeton)

17:30 Panel: What goal should civilization strive for? (video)

Joshua Greene (Harvard)
Nick Bostrom (FHI)
Amanda Askell (OpenAI)
Moderator: Lucas Perry (FLI)

18:00 Coffee & snacks
18:20 Panel: If AGI eventually makes all humans unemployable, then how can people be provided with the income, influence, friends, purpose, and meaning that jobs help provide today? (video)

Gillian Hadfield (CHAI/Toronto)
Reid Hoffman (LinkedIn)
James Manyika (McKinsey)
Moderator: Erik Brynjolfsson (MIT)

19:00 Dinner
20:00 Lightning talks

Sunday, January 6

Session 3 — Technical Safety | Session Chair: Victoria Krakovna

Paths to AGI:

9:00 Talk: Challenges towards AGI | Yoshua Bengio (MILA) (pdf) (video)
9:30 Debate: Possible paths to AGI (video)

Yoshua Bengio (MILA)
Irina Higgins (DeepMind)
Nick Bostrom (FHI)
Yi Zeng (Chinese Academy of Sciences)
Moderator: Joshua Tenenbaum (MIT)

10:10 Coffee break

Safety research:

10:25 Talk: AGI safety research agendas | Rohin Shah (Berkeley/CHAI) (pdf) (video)
10:55 Panel: What are the key AGI safety research priorities? (video)

Scott Garrabrant (MIRI)
Daniel Ziegler (OpenAI)
Anca Dragan (Berkeley/CHAI)
Eric Drexler (FHI)
Moderator: Ramana Kumar (DeepMind)

11:30 Debate: Synergies vs. tradeoffs between near-term and long-term AI safety efforts (video)

Neil Rabinowitz (DeepMind)
Catherine Olsson (Google Brain)
Nate Soares (MIRI)
Owain Evans (FHI)
Moderator: Jelena Luketina (Oxford)

12:00 Lunch
13:00 Optional breakouts & free time
15:00 Coffee, snacks, & free time

Session 4 — Strategy & Governance | Session Chair: Anthony Aguirre

16:00 Talk: AGI emergence scenarios, and their relation to destination issues | Anthony Aguirre (FLI)
16:10 Debate: 4-way friendly debate – what would be the best scenario for AGI emergence and early use?

Helen Toner (FHI)
Seth Baum (GCRI)
Peter Eckersley (Partnership on AI)
Miles Brundage (OpenAI)
Moderator: Anthony Aguirre (FLI)

16:50 Talk: AGI – Racing and cooperating | Seán Ó hÉigeartaigh (CSER) (pdf) (video)
17:10 Panel: Where are opportunities for cooperation at the governance level? Can we identify globally shared goals? (video)

Jason Matheny (IARPA)
Charlotte Stix (CFI)
Cyrus Hodes (Future Society)
Bing Song (Berggruen)
Moderator: Danit Gal (Keio)

17:40 Coffee break
17:55 Panel: Where are opportunities for cooperation at the academic & corporate level? (video)

Jason Si (Tencent)
Francesca Rossi (CSER)
Andrey Ustyuzhanin (Higher School of Economics)
Martina Kunz (CFI/CSER)
Teddy Collins (DeepMind)
Moderator: Brian Tse (FHI)

18:25 Talks: Lightning support pitches
19:00 Banquet
20:00 Poster session

Monday, January 7

Session 5 — Action items

9:00 Group discussion and analysis of conference
9:45 Governance panel: What laws, regulations, institutions, and agreement are needed? (video)

Wendell Wallach (Yale)
Nicolas Miailhe (Future Society)
Helen Toner (FHI)
Jeff Cao (Tencent)
Moderator: Tim Hwang (Google)

10:15 Coffee break
10:30 Technical safety student panel: What does the next generation of researchers think that the action items should be? (video)

Jaime Fisac (Berkeley/CHAI)
William Saunders (Toronto)
Smitha Milli (Berkeley/CHAI)
El Mahdi El Mhamdi (EPFL)
Moderator: Dylan Hadfield-Menell (Berkeley/CHAI)

11:10 Prioritization: Everyone e-votes on how to rank and prioritize action items
11:30 Closing remarks

Early drinks & appetizers

Towards a global community of shared future in AGI | Brian Tse

Strategy workshop summary | Jessica Cussins

Panel: Should we build superintelligence?

AGI & Work Panel

Lightning talk | Francesca Rossi

Challenges towards AGI | Yoshua Bengio

AGI safety research agendas | Rohin Shah

AGI emergence scenarios, and their relation to destination issues | Anthony Aguirre

Panel: Where are opportunities for cooperation at the governance level?

Panel: Where are opportunities for cooperation at the academic & corporate level?

Governance Panel: What laws, regulations, institutions, and agreement are needed?

Closing remarks

 

 

Workshop Schedule

Wednesday, January 2

18:00 Workshop welcome reception, 3-word introductions

Thursday, January 3

9:00 Welcome & summary of workshop plan | FLI Team

Destination Session | Session Chair: Max Tegmark

9:15 Talk: Framing, breakout task assignment
9:30 Breakouts: Parallel group brainstorming sessions
10:45 Coffee break
11:00 Discussion: Destination group reports, debate between panel rapporteurs
12:00 Lunch
13:00 Optional breakouts & free time
15:00 Coffee, snacks, & free time

Technical Safety Session | Session Chair: Victoria Krakovna

16:00 Opening remarks

Talks about latest progress on human-in-the-loop approaches to AI safety 

16:05 Talk: Scalable agent alignment | Jan Leike (DeepMind) (pdf)
16:20 Talk: Iterated amplification and debate | Amanda Askell (OpenAI) (pdf)
16:35 Talk: Cooperative inverse reinforcement learning | Dylan Hadfield-Menell (Berkeley/CHAI)
16:50 Talk: The comprehensive AI services framework | Eric Drexler (FHI) (pdf)
17:05 Coffee break

Talks about latest progress on theory approaches to AI safety 

17:25 Talk: Embedded agency | Scott Garrabrant (MIRI) (pdf)
17:40 Talk: Measuring side effects | Victoria Krakovna (DeepMind) (pdf)
17:55 Talk: Verification/security/containment | Ramana Kumar (DeepMind) (pdf)
18:10 Coffee break
18:30 Debate: Will future AGI systems be optimizing a single long-term goal?

Rohin Shah (Berkeley/CHAI)
Peter Eckersley (Partnership on AI)
Anna Salamon (CFAR)
Moderator: David Krueger (MILA)

19:00 Dinner
20:30 Destinations groups finish their work as needed

Friday, January 4

Strategy Session | Session Chair: Anthony Aguirre

9:00 Welcome back & framing of strategy challenge
9:10 Talk: Exploring AGI scenarios | Shahar Avin (FHI) (pdf)
9:40 Forecasting: What scenario ingredients are most likely and why? What goes into answering this?

Jade Leung (FHI)
Danny Hernandez (OpenAI)
Gillian Hadfield (CHAI/Toronto)
Malo Bourgon (MIRI)
Moderator: Shahar Avin (FHI)

10:10 Coffee break
10:20 Scenario breakouts
11:20 Reports: Scenario report-backs & discussion panel(s), assignments to action item panels
12:00 Lunch
13:00 Optional breakouts & free time
15:00 Coffee & free time

Action Item Session

15:45 Panel: Destination synthesis – what did all groups agree on, and what are the key controversies worth clarifying? Action items?

Tegan Maharaj (MILA)
Alex Zhu (MIRI)

16:15 Panel: What might the unknown unknowns in the space of AI safety problems look like? How can we broaden our research agendas to capture them?

Jan Leike (DeepMind)
Victoria Krakovna (DeepMind)
Richard Mallah (FLI)
Moderator: Andrew Critch (Berkeley/CHAI)

16:45 Coffee break
17:00 Action items: What institutions, platforms, & organizations do we need?

Jaan Tallinn (CSER/FLI)
Tanya Singh (FHI)
Alex Zhu (MIRI)
Moderator: Allison Duettmann (Foresight)

17:30 Action items: What standards, laws, regulations, & agreements do we need?

Jessica Cussins (FLI)
Teddy Collins (DeepMind)
Gillian Hadfield (CHAI/Toronto)
Moderator: Allan Dafoe (FHI/GovAI)

18:00 Conference welcome reception & buffet dinner
20:00 Poster session

There are no videos from the workshop portion of the conference.

Welcome

Iterated amplification and debate | Amanda Askell

Measuring side effects | Victoria Krakovna

Exploring AGI scenarios | Shahar Avin

Forecasting: What scenario ingredients are most likely and why?

Panel: Destination synthesis

Action items: What institutions, platforms, & organizations do we need?

Organizers

Anthony Aguirre, Meia Chita-Tegmark, Ariel Conn, Tucker Davey, Victoria Krakovna, Richard Mallah, Lucas Perry, Lucas Sabor, Max Tegmark

Participants

Bios for the attendees and participants can be found here.

 

 

 

 

 

 

Future Of Life Award 2018

The Future of Life Award 2018

Celebrating the contributions of Stanislav Petrov

Click here to see this page in other languages:  German Russian 

To celebrate that today is not the 35th anniversary of World War III, Stanislav Petrov, the man who helped avert an all-out nuclear exchange between Russia and the U.S. on September 26 1983 was honored in New York with the $50,000 Future of Life Award at a ceremony at the Museum of Mathematics in New York.

Former United Nations Secretary General Ban Ki-Moon said: “It is hard to imagine anything more devastating for humanity than all-out nuclear war between Russia and the United States. Yet this might have occurred by accident on September 26 1983, were it not for the wise decisions of Stanislav Yevgrafovich Petrov. For this, he deserves humanity’s profound gratitude. Let us resolve to work together to realize a world free from fear of nuclear weapons, remembering the courageous judgement of Stanislav Petrov.”

Stanislav Petrov’s daughter Elena holds the 2018 Future of Life Award flanked by her husband Victor. From left: Ariel Conn (FLI), Lucas Perry (FLI), Hannah Fry, Victor, Elena, Steven Mao (exec. producer of the Petrov film “The Man Who Saved the World”), Max Tegmark (FLI)

Although the U.N. General Assembly, just blocks away, heard politicians highlight the nuclear threat from North Korea’s small nuclear arsenal, none mentioned the greater threat from the many thousands of nuclear weapons in the United States and Russian arsenals that have nearly been unleashed by mistake dozens of times in the past in a seemingly never-ending series of mishaps and misunderstandings.

One of the closest calls occurred thirty-five years ago, on September 26, 1983, when Stanislav Petrov chose to ignore the Soviet early-warning detection system that had erroneously indicated five incoming American nuclear missiles. With his decision to ignore algorithms and instead follow his gut instinct, Petrov helped prevent an all-out US-Russian nuclear war, as detailed in the documentary film “The Man Who Saved the World”, which will be released digitally next week. Since Petrov passed away last year, the award was collected by his daughter Elena. Meanwhile, Petrov’s son Dmitry missed his flight to New York because the U.S. embassy delayed his visa. “That a guy can’t get a visa to visit the city his dad saved from nuclear annihilation is emblematic of how frosty US-Russian relations have gotten, which increases the risk of accidental nuclear war”, said MIT Professor Max Tegmark when presenting the award. Arguably the only recent reduction in the risk of accidental nuclear war came when Donald Trump held a summit with Vladimir Putin in Helsinki earlier this year, which was, ironically, met with widespread criticism.

In Russia, soldiers often didn’t discuss their wartime actions out of fear that it might displease their government, and so, Elena only first heard about her father’s heroic actions in 1998 – 15 years after the event occurred. And even then, Elena and her brother only learned of what her father had done when a German journalist reached out to the family for an article he was working on. It’s unclear if Petrov’s wife, who died in 1997, ever knew of her husband’s heroism. Until his death, Petrov maintained a humble outlook on the event that made him famous. “I was just doing my job,” he’d say.

But most would agree that he went above and beyond his job duties that September day in 1983. The alert of five incoming nuclear missiles came at a time of high tension between the superpowers, due in part to the U.S. military buildup in the early 1980s and President Ronald Reagan’s anti-Soviet rhetoric. Earlier in the month the Soviet Union shot down a Korean Airlines passenger plane that strayed into its airspace, killing almost 300 people, and Petrov had to consider this context when he received the missile notifications. He had only minutes to decide whether or not the satellite data were a false alarm. Since the satellite was found to be operating properly, following procedures would have led him to report an incoming attack. Going partly on gut instinct and believing the United States was unlikely to fire only five missiles, he told his commanders that it was a false alarm before he knew that to be true. Later investigations revealed that reflections of the Sun off of cloud tops had fooled the satellite into thinking it was detecting missile launches.

Last years Nobel Peace Prize Laureate, Beatrice Fihn, who helped establish the recent United Nations treaty banning nuclear weapons, said,“Stanislav Petrov was faced with a choice that no person should have to make, and at that moment he chose the human race — to save all of us. No one person and no one country should have that type of control over all our lives, and all future lives to come. 35 years from that day when Stanislav Petrov chose us over nuclear weapons, nine states still hold the world hostage with 15,000 nuclear weapons. We cannot continue relying on luck and heroes to safeguard humanity. The Treaty on the Prohibition of Nuclear Weapons provides an opportunity for all of us and our leaders to choose the human race over nuclear weapons by banning them and eliminating them once and for all. The choice is the end of us or the end of nuclear weapons. We honor Stanislav Petrov by choosing the latter.”

University College London Mathematics Professor  Hannah Fry, author of  the new book “Hello World: Being Human in the Age of Algorithms”, participated in the ceremony and pointed out that as ever more human decisions get replaced by automated algorithms, it is sometimes crucial to keep a human in the loop – as in Petrov’s case.

The Future of Life Award seeks to recognize and reward those who take exceptional measures to safeguard the collective future of humanity. It is given by the Future of Life Institute (FLI), a non-profit also known for supporting AI safety research with Elon Musk and others. “Although most people never learn about Petrov in school, they might not have been alive were it not for him”, said FLI co-founder Anthony Aguirre. Last year’s award was given to the Vasili Arkhipov, who singlehandedly prevented a nuclear attack on the US during the Cuban Missile Crisis. FLI is currently accepting nominations for next year’s award.

Stanislav Petrov around the time he helped avert WWIII

History of the Award

The Future of Life Award is a $50,000 prize given to an individual who, without receiving much recognition at the time, has helped make today dramatically better than it may otherwise have been. We are confident that there are many unsung heroes out there, who have done incredible work to ensure a beneficial future of life on Earth. We need your help to ensure they get the recognition and honor they deserve.

2019 Winner

Matthew Meselson



Dr. Meselson was a driving force behind the 1972 Biological Weapons Convention, an international ban that has prevented one of the most inhumane forms of warfare known to humanity. April 9th marked the eve of the Convention’s anniversary. Meselson’s long career is studded with highlights: proving Watson and Crick’s hypothesis on DNA structure, solving the Sverdlovsk Anthrax mystery, ending the use of Agent Orange in Vietnam. But it is above all his work on biological weapons that makes him an international hero.

The 2019 Future of Life Award to Dr. Matthew Meselson was covered in Vox, and you can see a video about Meselson’s life below.

2017 Winner

Vasili Arkhipov



Vasili Arkhipov single-handedly prevented nuclear war during the height of the Cuban Missile Crisis. Arkhipov’s submarine captain, thinking their sub was under attack by American forces, wanted to launch a nuclear weapon at the ships above. Arkhipov, with the power of veto, said no, thus averting nuclear war.

The 2017 Future of Life Award to Vasili Arkhipov was covered in The TimesThe GuardianThe Independent, and The Atlantic.

Help Us Find Our Next Unsung Hero

In the form below, please explain how your candidate has helped (indirectly or directly) save thousands or millions of lives, or helped make today dramatically better than it may otherwise have been. Please add links and evidence where possible.

If we decide to give the award to your nominee, you will receive a $3,000 prize from FLI for your contribution!



If you have documentation that you can’t upload in the form below, or if you are a researcher who would like to apply for a grant to research a candidate, please email dnicholson@post.harvard.edu.

If you don’t know of any candidates, we encourage you to share this page on social media and websites like Reddit, so we can reach as many people as possible in our quest to find humanity’s unsung heroes. If, by sharing this, you directly lead someone else to submit a winning candidate, we will give you credit in the nominations page.

Sign Up for our Newsletter

2018 Spring Conference: Invest in Minds Not Missiles

On Saturday April 7th and Sunday morning April 8th, MIT and Massachusetts Peace Action will co-host a conference and workshop at MIT on understanding and reducing the risk of nuclear war. Tickets are free for students. To attend, please register here.

 

Saturday sessions

Workshops

Sunday Morning Planning Breakfast

Student-led session to design and implement programs enhancing existing campus groups, and organizing new ones; extending the network to campuses in Rhode Island, Connecticut, New Jersey, New Hampshire, Vermont and Maine.

For more information, contact Jonathan King at <jaking@mit.edu>, or call 617-354-2169

Future Of Life Award 2017

The Future of Life Award 2017

Celebrating the contributions of Vasili Arkhipov

Click here to see this page in other languages: Russian

London, UK – On October 27, 1962, a soft-spoken naval officer named Vasili Arkhipov single-handedly prevented nuclear war during the height of the Cuban Missile Crisis. Arkhipov’s submarine captain, thinking their sub was under attack by American forces, wanted to launch a nuclear weapon at the ships above. Arkhipov, with the power of veto, said no, thus averting nuclear war.

Now, 55 years after his courageous actions, the Future of Life Institute has presented the Arkhipov family with the inaugural Future of Life Award to honor humanity’s late hero.

Arkhipov’s surviving family members, represented by his daughter Elena and grandson Sergei, flew into London for the ceremony, which was held at the Institute of Engineering & Technology. After explaining Arkhipov’s heroics to the audience, Max Tegmark, president of FLI, presented the Arkhipov family with their award and $50,000. Elena and Sergei were both honored by the gesture and by the overall message of the award.

Elena explained that her father “always thought that he did what he had to do and never consider his actions as heroism. … Our family is grateful for the prize and considers it as a recognition of his work and heroism. He did his part for the future so that everyone can live on our planet.”

Elena and Sergei with the Future of Life Award

The Future of Life Award seeks to recognize and reward those who take exceptional measures to safeguard the collective future of humanity. Arkhipov, whose courage and composure potentially saved billions of lives, was an obvious choice for the inaugural event.

“Vasili Arkhipov is arguably the most important person in modern history, thanks to whom October 27 2017 isn’t the 55th anniversary of World War III,” FLI president Max Tegmark explained. “We’re showing our gratitude in a way he’d have appreciated, by supporting his loved ones.”

The award also aims to foster a dialogue about the growing existential risks that humanity faces, and the people that work to mitigate them.

Jaan Tallinn, co-founder of FLI, said: “Given that this century will likely bring technologies that can be even more dangerous than nukes, we will badly need more people like Arkhipov — people who will represent humanity’s interests even in the heated moments of a crisis.”

FLI president Max Tegmark presenting the Future of Life Award to Arkhipov’s daughter, Elena, and grandson, Sergei.

History of the Award

The Future of Life Award is a $50,000 prize given to an individual who, without receiving much recognition at the time, has helped make today dramatically better than it may otherwise have been. We are confident that there are many unsung heroes out there, who have done incredible work to ensure a beneficial future of life on Earth. We need your help to ensure they get the recognition and honor they deserve.

2019 Winner

Matthew Meselson



Dr. Meselson was a driving force behind the 1972 Biological Weapons Convention, an international ban that has prevented one of the most inhumane forms of warfare known to humanity. April 9th marked the eve of the Convention’s anniversary. Meselson’s long career is studded with highlights: proving Watson and Crick’s hypothesis on DNA structure, solving the Sverdlovsk Anthrax mystery, ending the use of Agent Orange in Vietnam. But it is above all his work on biological weapons that makes him an international hero.

The 2019 Future of Life Award to Dr. Matthew Meselson was covered in Vox, and you can see a video about Meselson’s life below.

2018 Winner

Stanislav Petrov



One of the closest calls occurred thirty-five years ago, on September 26, 1983, when Stanislav Petrov chose to ignore the Soviet early-warning detection system that had erroneously indicated five incoming American nuclear missiles. With his decision to ignore algorithms and instead follow his gut instinct, Petrov helped prevent an all-out US-Russian nuclear war, as detailed in the documentary film “The Man Who Saved the World”.

The 2018 Future of Life Award to Stanislav Petrov was covered in VoxDaily Mail, and The Daily Star.

Help Us Find Our Next Unsung Hero

In the form below, please explain how your candidate has helped (indirectly or directly) save thousands or millions of lives, or helped make today dramatically better than it may otherwise have been. Please add links and evidence where possible.

If we decide to give the award to your nominee, you will receive a $3,000 prize from FLI for your contribution!



If you have documentation that you can’t upload in the form below, or if you are a researcher who would like to apply for a grant to research a candidate, please email dnicholson@post.harvard.edu.

If you don’t know of any candidates, we encourage you to share this page on social media and websites like Reddit, so we can reach as many people as possible in our quest to find humanity’s unsung heroes. If, by sharing this, you directly lead someone else to submit a winning candidate, we will give you credit in the nominations page.

Sign Up for our Newsletter

Beneficial AI 2017

Beneficial AI 2017

In our sequel to the 2015 Puerto Rico AI conference, we brought together an amazing group of AI researchers from academia and industry, and thought leaders in economics, law, ethics, and philosophy for five days dedicated to beneficial AI. We hosted a two-day workshop for our grant recipients to give them an opportunity to highlight and discuss the progress with their grants. We followed that with a 2.5-day conference, in which people from various AI-related fields hashed out opportunities and challenges related to the future of AI and steps we can take to ensure that the technology is beneficial. Honoring the FrieNDA for the conference, we are only posting videos and slides below with approval of the speaker. Learn more about the Asilomar AI Principles that resulted from the conference, the process involved in developing them, and the resulting discussion about each principle. (It looks like we’ll be posting almost everything, but please be patient while we finish editing and uploading videos, etc.)

Conference Schedule

Thursday January 5

All afternoon: registration open; come chill & meet old and new friends!
1800-2100: Welcome reception
Friday January 6

0730-0900: Breakfast
0900-1200: Opening keynotes on AI, economics & law: Progress since Puerto Rico 2015

  • Welcome Remarks by Max Tegmark (video)
  • Talks:
    • Erik Brynjolfsson (MIT) (pdf) (video)
    • Yoshua Bengio (Montreal) (pdf) (video)
    • Viktoriya Krakovna (DeepMind/FLI) (pdf) (video)
    • Stuart Russell (Berkeley) (pdf) (video)
    • Ryan Calo (U. Washington) (video)
  • Group photo

1200-1300: Lunch
1300-1500: Breakout sessions
1500-1800: Economics: How can we grow our prosperity through automation without leaving people lacking income or purpose?

  • Talk: Daniela Rus (MIT) (video)
  • Panel with Daniela Rus (MIT), Andrew Ng (Baidu), Mustafa Suleyman (DeepMind), Moshe Vardi (Rice) & Peter Norvig (Google): How AI is automating and augmenting work (video)
  • Talks:
  • Panel with Andrew McAfee (MIT), Jeffrey Sachs (Columbia), Eric Schmidt (Google) & Reid Hoffman (LinkedIn): Implications of AI for the Economy and Society (video)
  • Fireside chat with Daniel Kahneman: What makes people happy? (video)

1800-2100: Dinner

Saturday January 7

0730-0900: Breakfast
0900-1200: Creating human-level AI: Will it happen, and if so, when and how? What key remaining obstacles can be identified? How can we make future AI systems more robust than today’s, so that they do what we want without crashing, malfunctioning or getting hacked?

  • Talks:
    • Demis Hassabis (DeepMind) (video)
    • Ray Kurzweil (Google) (video)
    • Yann LeCun (Facebook/NYU) (pdf) (video)
  • Panel with Anca Dragan (Berkeley), Demis Hassabis (DeepMind), Guru Banavar (IBM), Oren Etzioni (Allen Institute), Tom Gruber (Apple), Jürgen Schmidhuber (Swiss AI Lab), Yann LeCun (Facebook/NYU), Yoshua Bengio (Montreal) (video)

1200-1300: Lunch
1300-1500: Breakout sessions
1500-1800: Superintelligence: Science or fiction? If human level general AI is developed, then what are likely outcomes? What can we do now to maximize the probability of a positive outcome? (video)

  • Talks:
    • Shane Legg (DeepMind)
    • Nick Bostrom (Oxford) (pdf) (video)
    • Jaan Tallinn (CSER/FLI) (pdf) (video)
  • Panel with Bart Selman (Cornell), David Chalmers (NYU), Elon Musk (Tesla, SpaceX), Jaan Tallinn (CSER/FLI), Nick Bostrom (FHI), Ray Kurzweil (Google), Stuart Russell (Berkeley), Sam Harris, Demis Hassabis (DeepMind): If we succeed in building human-level AGI, then what are likely outcomes? What would we like to happen?
  • Panel with Dario Amodei (OpenAI), Nate Soares (MIRI), Shane Legg (DeepMind), Richard Mallah (FLI), Stefano Ermon (Stanford), Viktoriya Krakovna (DeepMind/FLI): Technical research agenda: What can we do now to maximize the chances of a good outcome? (video)

1800-2200: Banquet

Sunday January 8

0730-0900: Breakfast
0900-1200: Law, policy & ethics: How can we update legal systems, international treaties and algorithms to be more fair, ethical and efficient and to keep pace with AI?

  • Talks:
    • Matt Scherer (pdf) (video)
    • Heather Roff-Perkins (Oxford)
  • Panel with Martin Rees (CSER/Cambridge), Heather Roff-Perkins, Jason Matheny (IARPA), Steve Goose (HRW), Irakli Beridze (UNICRI), Rao Kambhampati (AAAI, ASU), Anthony Romero (ACLU): Policy & Governance (video)
  • Panel with Kate Crawford (Microsoft/MIT), Matt Scherer, Ryan Calo (U. Washington), Kent Walker (Google), Sam Altman (OpenAI): AI & Law (video)
  • Panel with Kay Firth-Butterfield (IEEE, Austin-AI), Wendell Wallach (Yale), Francesca Rossi (IBM/Padova), Huw Price (Cambridge, CFI), Margaret Boden (Sussex): AI & Ethics (video)

1200-1300: Lunch
1300: Depart

 

 

 

Workshop Schedule

Tuesday January 3

All afternoon: workshop registration open; come chill & meet old and new friends!
1800-2100: Welcome reception, breakout session sign-up

Wednesday January 4

0730-0900: Breakfast
0900-1200: Talks & discussions

  • Grant-winner talks on value learning: Stuart Russell (Berkeley) (pdf), Francesca Rossi (IBM/Padova) (pdf), Owain Evans (FHI) (pdf), Adrian Weller (Cambridge), Vincent Conitzer (Duke) (pdf)
  • Panel with above speakers: Future directions in value learning
  • Panel with Tom Dietterich, Manuela Veloso, Bart Selman & Jürgen Schmidhuber: Will we ever build human-level AI?

1200-1300: Lunch
1300-1400: Lightning talks: Dario Amodei (OpenAI) (pdf), Catherine Olsson (OpenAI) (pdf), Andrew Critch (CHCAI/MIRI) (link), Eric
Drexler (MIT), Allan Dafoe (Yale), Jürgen Schmidhuber (Swiss AI Lab) (pdf), Toby Walsh (UNSW) (pdf)
1400-1500: Breakout session A
1500-1800: Talks & discussions

  • Grant-winner talks on verification, safe self-modification, and complexity: Bart Selman (Cornell) (pdf), Stefano Ermon (Stanford) (pdf), Katja Grace (MIRI), Andrew Critch (CHCAI/MIRI) (pdf), Ramana Kumar (Data61/CSIRO/UNSW) (pdf) & Bas Steunebrink (IDSIA) (pdf)
  • Panel with above speakers: Future directions in these areas
  • Panel with above speakers and Toby Walsh: Recursive self-improvement: fast takeoff, slow takeoff or no takeoff?

1800-2100: Dinner

Thursday January 5

0730-0900: Breakfast
0900-1200: Talks & discussions

  • Grant-winner talks on control, uncertainty identification & management: Thomas Dietterich (OSU) (pdf), Manuela Veloso (Carnegie Mellon) (pdf), Percy Liang (Stanford) (pdf), Long Ouyang (pdf), Paul Christiano (OpenAI) (pdf), Fuxin Li (OSU) (pdf) & Brian Ziebart (U. Chicago) (pdf)
  • Panel with above speakers: Future directions in these areas
  • “Red-team” Panel with Dario Amodei (OpenAI), Eric Drexler (MIT), Jan Leike (FHI/DeepMind), Percy Liang (Stanford) & Eliezer Yudkowsky (MIRI): What could go wrong with AGI and how can these problems be avoided?

1200-1300: Lunch
1300-1400: Breakout session B
1400-1500: Breakout session C
1500-1800: Talks & discussions

  • Grant-winner talks on governance & ethics: Nick Bostrom (Oxford), Wendell Wallach (Yale) (pdf), Heather Roff Perkins (Oxford/ASU) (pdf), Peter Asaro (New School) (pdf) & Moshe Vardi (Rice) (pdf)
  • Grant-winner talks on education: Anna Salamon (CFAR) & Paul Christiano (OpenAI) (pdf)
  • Panel with Anna Salamon (CFAR) & Paul Christiano (OpenAI): Educating and mentoring AI safety researchers (Moderator: Andrew Critch)
  • Grant-winner talk: Owen Cotton-Barratt (GPP) (pdf)
  • Panel with Allan Dafoe (Yale), Anthony Aguirre (FLI) & Owen Cotton-Barratt (GPP): Prediction (Moderator: Jaan Tallinn (FLI/CSER))

1800-2100: Welcome reception for Beneficial AI 2017 conference

Scientific Organizing Committee

Erik Brynjolfsson, MIT, Professor at the MIT Sloan School of Management, co-author of The Second Machine Age
Eric Horvitz, Microsoft, co-chair of the AAAI presidential panel on long-term AI futures
Peter Norvig, Google, Director of Research, co-author of the standard textbook Artificial Intelligence: a Modern Approach
Francesca Rossi, Univ. Padova, Professor of Computer Science, IBM,  President of the International Joint Conference on Artificial Intelligence
Stuart Russell, UC Berkeley, Professor of Computer Science, director of the Center for Intelligent Systems, and co-author of the standard textbook Artificial Intelligence: a Modern Approach
Bart Selman, Cornell University, Professor of Computer Science, co-chair of the AAAI presidential panel on long-term AI futures
Max Tegmark, MIT, Professor of Physics

Local Organizers

Anthony Aguirre, Meia Chita-Tegmark, Ariel Conn, Tucker Davey, Lucas Perry, Viktoriya Krakovna, Janos Kramar, Richard Mallah, Max Tegmark.

Sponsors

A special thank you to our generous sponsors: Alexander Tamas; The Center for Brains, Minds, and Machines; Elon Musk; Jaan Tallinn; and The Open Philanthropy Project.

 

Bios for the attendees and participants can be found here.


Standing Row: Patrick Lin, Daniel Weld, Ariel Conn, Nancy Chang, Tom Mitchell, Ray Kurzweil, Daniel Dewey, Margaret Boden, Peter Norvig, Nick Hay, Moshe Vardi, Scott Siskind, Nick Bostrom, Francesca Rossi, Shane Legg, Manuela Veloso, David Marble, Katja Grace, Irakli Beridze, Marty Tenenbaum, Gill Pratt, Martin Rees, Joshua Greene, Matt Scherer, Angela Kane, Amara Angelica, Jeff Mohr, Mustafa Suleyman, Steve Omohundro, Kate Crawford, Vitalik Buterin, Yutaka Matsuo, Stefano Ermon, Michael Wellman, Bas Steunebrink, Wendell Wallach, Allan Dafoe, Toby Ord, Thomas Dietterich, Daniel Kahneman, Dario Amodei, Eric Drexler, Tomaso Poggio, Eric Schmidt, Pedro Ortega, David Leake, Sean O’Heigeartaigh, Owain Evans, Jaan Tallinn, Anca Dragan, Sean Legassick, Toby Walsh, Peter Asaro, Kay Firth-Butterfield, Philip Sabes, Paul Merolla, Bart Selman, Tucker Davey, ?, Jacob Steinhardt, Moshe Looks, Josh Tenenbaum, Tom Gruber, Andrew Ng, Kareem Ayoub, Craig Calhoun, Percy Liang, Helen Toner, David Chalmers, Richard Sutton, Claudia Passos-Ferriera, Janos Kramar, William MacAskill, Eliezer Yudkowsky, Brian Ziebart, Huw Price, Carl Shulman, Neil Lawrence, Richard Mallah, Jurgen Schmidhuber, Dileep George, Jonathan Rothberg, Noah Rothberg

Sitting Row: Anthony Aguirre, Sonia Sachs, Lucas Perry, Jeffrey Sachs, Vincent Conitzer, Steve Goose, Victoria Krakovna, Owen Cotton-Barratt, Daniela Rus, Dylan Hadfield-Menell, Verity Harding, Shivon Zilis, Laurent Orseau, Ramana Kumar, Nate Soares, Andrew McAfee, Jack Clark, Anna Salamon, Long Ouyang, Andrew Critch, Paul Christiano, Yoshua Bengio, David Sanford, Catherine Olsson, Jessica Taylor, Martina Kunz, Kristinn Thorisson, Stuart Armstrong, Yann LeCun, Alexander Tamas, Roman Yampolskiy, Marin Soljacic, Lawrence Krauss, Stuart Russell, Eric Brynjolfsson, Ryan Calo, ShaoLan Hsueh, Meia Chita-Tegmark, Kent Walker, Heather Roff, Meredith Whittaker, Max Tegmark, Adrian Weller, Jose Hernandez-Orallo, Andrew Maynard, John Hering, Abram Demski, Nicolas Berggruen, Gregory Bonnet, Sam Harris, Tim Hwang, Andrew Snyder-Beattie, Marta Halina, Sebastian Farquhar, Stephen Cave, Jan Leike, Tasha McCauley, Joseph Gordon-Levitt

Not in photo: Guru Banavar, Sam Teller, Anthony Romero, Elon Musk, Larry Page, Sam Altman, Oren Etzioni, Chelsea Finn, Ian Goodfellow, Reid Hoffman, Holden Karnofsky, Sergey Levine, Fuxin Li, Jason Matheny, Andrew Serazin, Ilya Sutskever

 

 

The view from the conference:

AI safety conference in Puerto Rico

The Future of AI: Opportunities and Challenges 

This conference brought together the world’s leading AI builders from academia and industry to engage with each other and experts in economics, law and ethics. The goal was to identify promising research directions that can help maximize the future benefits of AI while avoiding pitfalls (see this open letter and this list of research priorities). To facilitate candid and constructive discussions, there was no media present and Chatham House Rules: nobody’s talks or statements will be shared without their permission.
Where? San Juan, Puerto Rico
When? Arrive by evening of Friday January 2, depart after lunch on Monday January 5 (see program below)

 

conference-1

Scientific organizing committee:

  • Erik Brynjolfsson, MIT, Professor at the MIT Sloan School of Management, co-author of The Second Machine Age: Work, Progress, and Prosperity in a Time of Brilliant Technologies
  • Demis Hassabis, Founder, DeepMind
  • Eric Horvitz, Microsoft, co-chair of the AAAI presidential panel on long-term AI futures
  • Shane Legg, Founder, DeepMind
  • Peter Norvig, Google, Director of Research, co-author of the standard textbook Artificial Intelligence: a Modern Approach.
  • Francesca Rossi, Univ. Padova, Professor of Computer Science, President of the International Joint Conference on Artificial Intelligence
  • Stuart Russell, UC Berkeley, Professor of Computer Science, director of the Center for Intelligent Systems, and co-author of the standard textbook Artificial Intelligence: a Modern Approach.
  • Bart Selman, Cornell University, Professor of Computer Science, co-chair of the AAAI presidential panel on long-term AI futures
  • Murray Shanahan, Imperial College, Professor of Cognitive Robotics
  • Mustafa Suleyman, Founder, DeepMind
  • Max Tegmark, MIT, Professor of physics, author of Our Mathematical Universe

Local Organizers:
Anthony Aguirre, Meia Chita-Tegmark, Viktoriya Krakovna, Janos Kramar, Richard Mallah, Max Tegmark, Susan Young

Support: Funding and organizational support for this conference is provided by Skype-founder Jaan Tallinnthe Future of Life Institute and the Center for the Study of Existential Risk.

PROGRAM

Friday January 2:
1600-late: Registration open
1930-2130: Welcome reception (Las Olas Terrace)

Saturday January 3:
0800-0900: Breakfast
0900-1200: Overview (one review talk on each of the four conference themes)
• Welcome
• Ryan Calo (Univ. Washington): AI and the law
• Erik Brynjolfsson (MIT): AI and economics (pdf)
• Richard Sutton (Alberta): Creating human-level AI: how and when? (pdf)
• Stuart Russell (Berkeley): The long-term future of (artificial) intelligence (pdf)
1200-1300: Lunch
1300-1515: Free play/breakout sessions on the beach
1515-1545: Coffee & snacks
1545-1600: Breakout session reports
1600-1900: Optimizing the economic impact of AI
(A typical 3-hour session consists of a few 20-minute talks followed by a discussion panel where the panelists who haven’t already given talks get to give brief introductory remarks before the general discussion ensues.)
What can we do now to maximize the chances of reaping the economic bounty from AI while minimizing unwanted side-effects on the labor market?
Speakers:
• Andrew McAfee, MIT (pdf)
• James Manyika, McKinsey (pdf)
• Michael Osborne, Oxford (pdf)
Panelists include Ajay Agrawal (Toronto), Erik Brynjolfsson (MIT), Robin Hanson (GMU), Scott Phoenix (Vicarious)
1900: dinner

Sunday January 4:
0800-0900: Breakfast
0900-1200: Creating human-level AI: how and when?
Short talks followed by panel discussion: will it happen, and if so, when? Via engineered solution, whole brain emulation, or other means? (We defer until the 4pm session questions regarding what will happen, about whether machines will have goals, about ethics, etc.)
Speakers:
• Demis Hassabis, Google/DeepMind
• Dileep George, Vicarious (pdf)
• Tom Mitchell, CMU (pdf)
Panelists include Joscha Bach (MIT), Francesca Rossi (Padova), Richard Mallah (Cambridge Semantics), Richard Sutton (Alberta)
1200-1300: Lunch
1300-1515: Free play/breakout sessions on the beach
1515-1545: Coffee & snacks
1545-1600: Breakout session reports
1600-1900: Intelligence explosion: science or fiction?
If an intelligence explosion happens, then what are likely outcomes? What can we do now to maximize the probability of a positive outcome? Containment problem? Is “friendly AI” possible? Feasible? Likely to happen?
Speakers:
• Nick Bostrom, Oxford (pdf)
• Bart Selman, Cornell (pdf)
• Jaan Tallinn, Skype founder (pdf)
• Elon Musk, SpaceX, Tesla Motors
Panelists include Shane Legg (Google/DeepMind), Murray Shanahan (Imperial), Vernor Vinge (San Diego), Eliezer Yudkowsky (MIRI)
1930: banquet (outside by beach)

Monday January 5:
0800-0900: Breakfast
0900-1200: Law & ethics: Improving the legal framework for autonomous systems
How should legislation be improved to best protect the AI industry and consumers? If self-driving cars cut the 32000 annual US traffic fatalities in half, the car makers won’t get 16000 thank-you notes, but 16000 lawsuits. How can we ensure that autonomous systems do what we want? And who should be held liable if things go wrong? How tackle criminal AI? AI ethics? AI ethics/legal framework for military systems & financial systems?
Speakers:
• Joshua Greene, Harvard (pdf)
• Heather Roff Perkins, Univ. Denver (pdf)
• David Vladeck, Georgetown
Panelists include Ryan Calo (Univ. Washington), Tom Dietterich (Oregon State, AAAI president), Kent Walker (General Counsel, Google)
1200: Lunch, depart

PARTICIPANTS
You’ll find a list of participants and their bios here.

conference150104 hires

Back row, from left to right: Tom Mitchell, Seán Ó hÉigeartaigh, Huw Price, Shamil Chandaria, Jaan Tallinn, Stuart Russell, Bill Hibbard, Blaise Agüera y Arcas, Anders Sandberg, Daniel Dewey, Stuart Armstrong, Luke Muehlhauser, Tom Dietterich, Michael Osborne, James Manyika, Ajay Agrawal, Richard Mallah, Nancy Chang, Matthew Putman
Other standing, left to right: Marilyn Thompson, Rich Sutton, Alex Wissner-Gross, Sam Teller, Toby Ord, Joscha Bach, Katja Grace, Adrian Weller, Heather Roff-Perkins, Dileep George, Shane Legg, Demis Hassabis, Wendell Wallach, Charina Choi, Ilya Sutskever, Kent Walker, Cecilia Tilli, Nick Bostrom, Erik Brynjolfsson, Steve Crossan, Mustafa Suleyman, Scott Phoenix, Neil Jacobstein, Murray Shanahan, Robin Hanson, Francesca Rossi, Nate Soares, Elon Musk, Andrew McAfee, Bart Selman, Michele Reilly, Aaron VanDevender, Max Tegmark, Margaret Boden, Joshua Greene, Paul Christiano, Eliezer Yudkowsky, David Parkes, Laurent Orseau, JB Straubel, James Moor, Sean Legassick, Mason Hartman, Howie Lempel, David Vladeck, Jacob Steinhardt, Michael Vassar, Ryan Calo, Susan Young, Owain Evans, Riva-Melissa Tez, János Kramár, Geoff Anders, Vernor Vinge, Anthony Aguirre
Seated: Sam Harris, Tomaso Poggio, Marin Soljačić, Viktoriya Krakovna, Meia Chita-Tegmark
Behind the camera: Anthony Aguirre (and also photoshopped in by the human-level intelligence on his left)
Click  for a full resolution version.

Policy Exchange: Co-organized with CSER

This event occurred on September 1, 2015.

When one of the world’s leading experts in Artificial Intelligence makes a speech suggesting that a third of existing British jobs could be made obsolete by automation, it is time for think tanks and the policymaking community to take notice. This observation – by Associate Professor of Machine Learning at the University of Oxford, Michael Osborne – was one of many thought provoking comments made at a special event on the policy implications of the rise of AI we held this week with the Cambridge Centre for the Study of Existential Risk. 

The event formed part of Policy Exchange’s long-running research programme looking at reforms that are needed in policies relating to welfare and the workplace – as well as other long-term challenges facing the country. We gathered together the world’s leading authorities on AI to consider the rise of this new technology, which will form one of the greatest challenges facing our society in this century. The speakers were the following:  

•    Huw Price, Bertrand Russell Professor of Philosophy and a Fellow of Trinity College at the University of Cambridge, and co-founder of the Centre for the Study of Existential Risk.

•    Stuart Russell, Professor at the University of California at Berkeley and author of the standard textbook on AI.

•    Nick Bostrom, Professor at Oxford University, author of “Superintelligence” and Founding Director of the Future of Humanity Institute.

•    Michael Osborne. Associate Professor at Oxford University and co-director of the Oxford Martin programme on Technology and Employment.  

•    Murray Shanahan. Professor at Imperial College London, scientific advisor to the film Ex Machina, and author of “The Technological Singularity”

 policy_exchange_FLI_CSER

In this bulletin, we ask two of our speakers to share their views about the potential and risks from future AI:

Michael Osborne: Machine Learning and the Future of Work 

Machine learning is the study of algorithms that can learn and act. Why use a machine when we already have over six billion humans to choose from? One reason is that algorithms are significantly cheaper, and becoming ever more so.  But just as important, algorithms can often do better than humans, avoiding the biases that taint human decision making.  

There are big benefits to be gained from the rise of the algorithms. Big data is already leading to programs that can handle increasingly sophisticated tasks, such as translation. Computational health informatics will transform health monitoring, allowing us to release patients from their hospital beds much earlier and freeing up resources in the NHS. Self-driving cars will allow us to cut down on the 90% of traffic accidents caused by human error, while the data generated by their constant monitoring of the impact will have big consequences for mapping, insurance, and the law.

Nevertheless, there will be big challenges from the disruption automation creates. New technologies derived from mobile machine learning and robotics threaten employment in logistics, sales and clerical occupations.  Over the next few decades, 47% of jobs in America are at high risk of automation, and 35% of jobs in the UK. Worse, it will be the already vulnerable who are most at risk, while high-skilled jobs are relatively resistant to computerisation. New jobs are emerging to replace the old, but only slowly – only 0.5% of the US workforce is employed in new industries created in the 21st century. 

Policy makers are going to have to do more to ensure that we can all share in the great prosperity promised by technology.

Stuart Russell: Killer Robots, the End of Humanity, and All That: Policy Implications 

Everything civilisation offers is the product of intelligence. If we can use AI to amplify our intelligence, the benefits to humanity are potentially immeasurable. 

The good news is that progress is accelerating. Solid theoretical foundations, more data and computing, and huge investments from private industry have created a virtuous cycle of iterative improvement. On the current trajectory, further real-world impact is inevitable.

Of course, not all impact is good. As technology companies unveil ever more impressive demos, newspapers have been full of headlines warning of killer robots, or the loss of half of all jobs, or even the end of humanity. But how credible exactly are these nightmare scenarios? The short answer is we should not panic, but there are real risks that are worth taking seriously.

In the short term, lethal autonomous weapons or weapon systems that can select and fire upon targets on their own are worth taking seriously. According to defence experts, including the Ministry of Defence, these are probably feasible now, and they have already been the subject of three UN meetings in 2014-5. In the future, they are likely to be relatively cheap to mass produce, potentially making them much harder to control or contain than nuclear weapons. A recent open letter from 3,000 AI researchers argued for a total ban on the technology to prevent the start of a new arms race.

Looking further ahead, what, however, if we succeed in creating an AI system that can make decisions as well, or even significantly better than humans? The first thing to say is that we are several conceptual breakthroughs away from constructing such a general artificial intelligence, as compared to the more specific algorithms needed for an autonomous weapon or self-driving car. 

It is highly unlikely that we will be able to create such an AI within the next five to ten years, but then conceptual breakthroughs are by their very nature hard to predict. The day before Leo Szilard conceived of neutron-induced chain reactions, the key to nuclear power, Lord Rutherford was claiming that, “anyone who looks for a source of power in the transformation of the atoms is talking moonshine.”

The danger from such a “superintelligent” AI is that it would not by default share the same goals as us. Even if we could agree amongst ourselves what the best human values were, we do not understand how to reliably formalise them into a programme. If we accidentally give a superintelligent AI the wrong goals, it could prove very difficult to stop. 

For example, for many benign-sounding final goals we might try to give the computer, two plausible intermediate goals for an AI are to gain as many physical resources as possible and to refuse to allow itself to be terminated. We might think that we are just asking the machine to calculate as many digits of pi as possible, but it could judge that the best way to do so is turn the whole Earth into a supercomputer.

In short, we are moving with increasing speed towards what could be the biggest event in human history. Like global warming, there are significant uncertainties involved and the pain might not come for another few decades – but equally like global warming, the sooner we start to look at potential solutions the more likely we are to be successful. Given the complexity of the problem, we need much more technical research on what an answer might look like.

Martin Rees: Catastrophic Risks: The Downsides of Advancing Technology

This event was held Thursday, November 6, 2014 in Harvard auditorium Jefferson Hall 250.

Our Earth is 45 million centuries old. But this century is the first when one species ours can determine the biosphere’s fate. Threats from the collective “footprint” of 9 billion people seeking food, resources and energy are widely discussed. But less well studied is the potential vulnerability of our globally-linked society to the unintended consequences of powerful technologies not only nuclear, but (even more) biotech, advanced AI, geo-engineering and so forth. More information here.

Nick Bostrom: Superintelligence — Paths, Dangers, Strategies

This event was held Thursday, September 4th, 2014 in Harvard auditorium Emerson 105.

What happens when machines surpass humans in general intelligence? Will artificial agents save or destroy us? In his new book – Superintelligence: Paths, Dangers, Strategies – Professor Bostrom explores these questions, laying the foundation for understanding the future of humanity and intelligent life.

Photos from the talk

Max Tegmark: “Ask Max Anything” on Reddit

 This event was held Wednesday, August 20th, 2014, in the “IAmA” subreddit on reddit.com. Read it here!

Max Tegmark answers the questions of reddit.com’s user base! Questions are on the subject of his book “Our Mathematical Universe”, physics, x-risks, AI safety, and AI research.

The Future of Technology: Benefits and Risks

This event was held Saturday May 24, 2014 at 7pm in MIT auditorium 10-250 – see videotranscript and photos below.

The coming decades promise dramatic progress in technologies from synthetic biology to artificial intelligence, with both great benefits and great risks. Please watch the video below for a fascinating discussion about what we can do now to improve the chances of reaping the benefits and avoiding the risks, moderated by Alan Alda and featuring George Church (synthetic biology), Ting Wu (personal genetics), Andrew McAfee(second machine age, economic bounty and disparity), Frank Wilczek (near-term AI and autonomous weapons) and Jaan Tallinn (long-term AI and singularity scenarios).

  • Alan Alda is an Oscar-nominated actor, writer, director, and science communicator, whose contributions range from M*A*S*H to Scientific American Frontiers.
  • George Church is a professor of genetics at Harvard Medical School, initiated the Personal Genome Project, and invented DNA array synthesizers.
  • Andrew McAfee is Associate Director of the MIT Center for Digital Business and author of the New York Times bestseller The Second Machine Age.
  • Jaan Tallinn is a founding engineer of Skype and philanthropically supports numerous research organizations aimed at reducing existential risk.
  • Frank Wilczek is a physics professor at MIT and a 2004 Nobel laureate for his work on the strong nuclear force.
  • Ting Wu is a professor of Genetics at Harvard Medical School and Director of the Personal Genetics Education project.
Photos from the talk