Last Thursday – just one day before the WannaCry ransomware attack would shut down 16 hospitals in the UK and ultimately hit hundreds of thousands of organizations and individuals in over 150 countries – the Director of National Intelligence, Daniel Coats, released the Worldwide Threat Assessment of the US Intelligence Community.
Large-scale cyber attacks are among the first risks cited in the document, which warns that “cyber threats also pose an increasing risk to public health, safety, and prosperity as cyber technologies are integrated with critical infrastructure in key sectors.”
Perhaps the other most prescient, or at least well-timed, warning in the document relates to North Korea’s ambitions to create nuclear intercontinental ballistic missiles (ICBMs). Coats writes:
“Pyongyang is committed to developing a long-range, nuclear-armed missile that is capable of posing a direct threat to the United States; it has publicly displayed its road-mobile ICBMs on multiple occasions. We assess that North Korea has taken steps toward fielding an ICBM but has not flight-tested it.”
This past Sunday, North Korea performed a missile test launch, which many experts believe shows considerable progress toward the development of an ICBM. Though the report hints this may be less of an actual threat from North Korea and more for show. “We have long assessed that Pyongyang’s nuclear capabilities are intended for deterrence, international prestige, and coercive diplomacy,” says Coats in the report.
More Nuclear Threats
The Assessment also addresses the potential of nuclear threats from China and Pakistan. China “continues to modernize its nuclear missile force by adding more survivable road-mobile systems and enhancing its silo-based systems. This new generation of missiles is intended to ensure the viability of China’s strategic deterrent by providing a second-strike capability.” In addition, China is also working to develop “its first long-range, sea-based nuclear capability.”
Meanwhile, though Pakistan’s nuclear program doesn’t pose a direct threat to the U.S., advances in Pakistan’s nuclear capabilities could risk further destabilization along the India-Pakistan border.
The report warns: “Pakistan’s pursuit of tactical nuclear weapons potentially lowers the threshold for their use.” And of the ongoing conflicts between Pakistan and India, it says, “Increasing numbers of firefights along the Line of Control, including the use of artillery and mortars, might exacerbate the risk of unintended escalation between these nuclear-armed neighbors.”
This could be especially problematic because “early deployment during a crisis of smaller, more mobile nuclear weapons would increase the amount of time that systems would be outside the relative security of a storage site, increasing the risk that a coordinated attack by non-state actors might succeed in capturing a complete nuclear weapon.”
Even a small nuclear war between India and Pakistan could trigger a nuclear winter that could send the planet into a mini ice age and starve an estimated 1 billion people.
Artificial Intelligence
Nukes aren’t the only weapons the government is worried about. The report also expresses concern about the impact of artificial intelligence on weaponry: “Artificial Intelligence (Al) is advancing computational capabilities that benefit the economy, yet those advances also enable new military capabilities for our adversaries.”
Coats worries that AI could negatively impact other aspects of society, saying, “The implications of our adversaries’ abilities to use AI are potentially profound and broad. They include an increased vulnerability to cyber attack, difficulty in ascertaining attribution, facilitation of advances in foreign weapon and intelligence systems, the risk of accidents and related liability issues, and unemployment.”
Space Warfare
But threats of war are not expected to remain Earth-bound. The Assessment also addresses issues associated with space warfare, which could put satellites and military communication at risk.
For example, the report warns that “Russia and China perceive a need to offset any US military advantage derived from military, civil, or commercial space systems and are increasingly considering attacks against satellite systems as part of their future warfare doctrine.”
The report also adds that “the global threat of electronic warfare (EW) attacks against space systems will expand in the coming years in both number and types of weapons.” Coats expects that EW attacks will “focus on jamming capabilities against dedicated military satellite communications” and against GPS, among others.
Environmental Risks & Climate Change
Plenty of global threats do remain Earth-bound though, and not all are directly related to military concerns. The government also sees environmental issues and climate change as potential threats to national security.
The report states, “The trend toward a warming climate is forecast to continue in 2017. … This warming is projected to fuel more intense and frequent extreme weather events that will be distributed unequally in time and geography. Countries with large populations in coastal areas are particularly vulnerable to tropical weather events and storm surges, especially in Asia and Africa.”
In addition to rising temperatures, “global air pollution is worsening as more countries experience rapid industrialization, urbanization, forest burning, and agricultural waste incineration, according to the World Health Organization (WHO). An estimated 92 percent of the world’s population live in areas where WHO air quality standards are not met.”
According to the Assessment, biodiversity loss will also continue to pose an increasing threat to humanity. The report suggests global biodiversity “will likely continue to decline due to habitat loss, overexploitation, pollution, and invasive species, … disrupting ecosystems that support life, including humans.”
The Assessment goes on to raise concerns about the rate at which biodiversity loss is occurring. It says, “Since 1970, vertebrate populations have declined an estimated 60 percent … [and] populations in freshwater systems declined more than 80 percent. The rate of species loss worldwide is estimated at 100 to 1,000 times higher than the natural background extinction rate.”
Other Threats
The examples above are just a sampling of the risks highlighted in the Assessment. A great deal of the report covers threats of terrorism, issues with Russia, China and other regional conflicts, organized crime, economics, and even illegal fishing. Overall, the report is relatively accessible and provides a quick summary of the greatest known risks that could threaten not only the U.S., but also other countries in 2017. You can read the report in its entirety here.
https://futureoflife.org/wp-content/uploads/2017/05/2017-world-threat-assessment.jpg7121100Ariel Connhttps://futureoflife.org/wp-content/uploads/2015/10/FLI_logo-1.pngAriel Conn2017-05-18 11:41:292017-05-18 11:51:26The U.S. Worldwide Threat Assessment Includes Warnings of Cyber Attacks, Nuclear Weapons, Climate Change, etc.
Our strategy update discusses changes to our AI forecasts and research priorities, new outreach goals, a MIRI/DeepMind collaboration, and other news.
MIRI is hiring software engineers! If you’re a programmer who’s passionate about MIRI’s mission and wants to directly support our research efforts, apply here to trial with us.
MIRI Assistant Research Fellow Ryan Carey has taken on an additional affiliation with the Centre for the Study of Existential Risk, and is also helping edit an issue of Informatica on superintelligence.
DeepMind is seeking applicants for a policy research position “to carry out research on the social and economic impacts of AI”.
The Center for Human-Compatible AI is hiring an assistant director. Interested parties may also wish to apply for the event coordinator position at the new Berkeley Existential Risk Initiative, which will help support work at CHAI and elsewhere.
80,000 Hours lists other potentially high-impact openings, including ones at Stanford’s AI Index project, the White House OSTP, IARPA, and IVADO.
Imagine going to the airport, but instead of walking through – or waiting in – long and tedious security lines, you could walk through a hallway that looks like a terrarium. No lines or waiting. Just a lush, indoor garden. But these plants aren’t something you can find in your neighbor’s yard – their genes have been redesigned to act as sensors, and the plants will change color if someone walks past with explosives.
The Genome Project Write (GP-write) got off to a rocky start last year when it held a “secret” meeting that prohibited journalists. News of the event leaked, and the press quickly turned to fears of designer babies and Frankenstein-like creations. This year, organizers of the meeting learned from the 2016 debacle. Not only did they invite journalists, but they also highlighted work by researchers like June Medford, whose plants research could lead to advancements like the security garden above.
Jef Boeke, one of the lead authors of the GP-write Grand Challenge, emphasized that this project was not just about writing the human genome. “The notion that we could write a human genome is simultaneously thrilling to some and not so thrilling to others,” Boeke told the group. “We recognize that this will take a lot of discussion.”
Boeke explained that the GP-write project will focus on the genomes of cells, and the researchers involved are not trying to produce an organism. He added that this work could be used to solve problems associated with climate change and the environment, invasive species, pathogens, and food insecurity.
To learn more about why this project is important, I spoke with genetics researcher, John Min, about what GP-write is and what it could accomplish. Min is not directly involved with GP-write, but he works with George Church, another one of the lead authors of the project.
Min explained, “We aren’t currently capable of making DNA as long as human chromosomes – we can’t make that from scratch in the laboratory. In this case, they’ll use CRISPR to make very specific cuts in the genome of an existing cell, and either use synthesized DNA to replace whole chunks or add new functionality in.”
He added, “An area of potentially exciting research with this new project is to create a human cell immune to all known viruses. If we can create this in the lab, then we can start to consider how to apply it to people around the world. Or we can use it to build an antibody library against all known viruses. Right now, tackling such a project is completely unaffordable – the costs are just too astronomic.”
But costs aren’t the only reason GP-write is hugely ambitious. It’s also incredibly challenging science. To achieve the objectives mentioned above, scientists will synthesize, from basic chemicals, the building blocks of life. Synthesizing a genome involves slowly editing out tiny segments of genes and replacing them with the new chemical version. Then researchers study each edit to determine what, if anything, changed for the organism involved. Then they repeat this for every single known gene. It is a tedious, time-consuming process, rife with errors and failures that send scientists back to the drawing board over and over, until they finally get just one gene right. On top of that, Min explained, it’s not clear how to tell when a project transitions from editing a cell, to synthesizing it. “How many edits can you make to an organism’s genome before you can say you’ve synthesized it?” he asked.
Clyde Hutchison, working with Craig Venter, recently came closest to answering that question. He and Venter’s team published the first paper depicting attempts to synthesize a simple bacterial genome. The project involved understanding which genes were essential, which genes were inessential, and discovering that some genes are “quasi-essential.” In the process, they uncovered “149 genes with unknown biological functions, suggesting the presence of undiscovered functions that are essential for life.”
This discovery tells us two things. First, it shows just how enormous the GP-write project is. To find 149 unknown genes in simple bacteria offers just a taste of how complicated the genomes of more advanced organisms will be. Kris Saha, Assistant Professor of Biomedical Engineering at the University of Wisconsin-Madison, explained this to the Genetic Experts News Service:
“The evolutionary leap between a bacterial cell, which does not have a nucleus, and a human cell is enormous. The human genome is organized differently and is much more complex. […] We don’t entirely understand how the genome is organized inside of a typical human cell. So given the heroic effort that was needed to make a synthetic bacterial cell, a similar if not more intense effort will be required – even to make a simple mammalian or eukaryotic cell, let alone a human cell.”
Second, this discovery gives us a clue as to how much more GP-write could tell us about how biology and the human body work. If we can uncover unknown functions within DNA, how many diseases could we eliminate? Could we cure aging? Could we increase our energy levels? Could we boost our immunities? Are there risks we need to prepare for?
The best assumption for that last question is: yes.
“Safety is one of our top priorities,” said Church at the event’s press conference, which included other leaders of the project. They said they expect safeguards to be engineered into research “from the get-go,” and part of the review process would include assessments of whether research within the project could be developed to have both positive or negative outcomes, known as Dual Use Research of Concern (DURC)
The meeting included roughly 250 people from 10 countries with backgrounds in science, ethics, law, government, and more. In general, the energy at the conference was one of excitement about the possibilities that GP-write could unleash.
“This project not only changes the way the world works, but it changes the way we work in the world,” said GP-write lead author Nancy J. Kelley.
https://futureoflife.org/wp-content/uploads/2017/05/gp-write-2017-plants.jpg7341100Ariel Connhttps://futureoflife.org/wp-content/uploads/2015/10/FLI_logo-1.pngAriel Conn2017-05-12 14:50:412017-05-12 14:53:22GP-write and the Future of Biology
Until recently, many Americans believed that nuclear weapons don’t represent the same threat as during the Cold War. However, recent events and aggressive posturing between nuclear nations —especially the U.S., Russia, and North Korea—has increased public awareness and concern. These fears were addressed at a recent MIT conference on nuclear weapons.
“The possibility of a nuclear bomb going off is greater today than 20 years ago,” said Ernest Moniz, former Secretary of Energy and a keynote speaker.
California Congresswoman Barbara Lee, another keynote speaker, recently returned from a trip to South Korea and Japan. Of the trip, she said, “We went to the DMZ, and I saw how close to nuclear war we really are.”
Lee suggested that if we want to eliminate nuclear weapons once and for all, this is the time to do it. At the very least, she argued for a common sense nuclear policy of “no first use,” that is, the U.S. won’t launch the first nuclear strike.
“We must prevent the president from launching nuclear weapons without a declaration from Congress,” Lee said.
Under current U.S. nuclear policy, the President is the only person who can launch a nuclear weapon, and no one else’s input is necessary. This policy was adapted, at least in part, to ensure the safety and usability of the land-based arm of the nuclear triad (the other two arms are air- and sea-based).
During the Cold War, the fear was that, if Russia were to attack the U.S., it would first target the land-based missiles in an attempt to limit their use during war. To protect these weapons, the U.S. developed an advanced-warning system that could notify military personnel of an incoming strike, giving the president just enough time to launch the land-based missiles in response.
Weapons launched from Russia would take about 30 minutes to reach the U.S. That means that, in 30 minutes, the warning system must pick up the signal of incoming missiles. Then, personnel must confirm that the warning is accurate, and not an error – which has happened many times. And by the time the information reaches the President, he’ll have around 10 minutes to decide whether to launch a retaliation.
Lisbeth Gronlund with the Union of Concerned Scientists pointed out that not only does this time frame put us at greater risk of an accidental launch, but “cyber attacks are a new unknown.” As a result, she’s also concerned that the risk of a nuclear launch is greater today than during the Cold War.
“If we eliminate our land-based missiles and base our deterrence on nuclear submarines and bombers, which are safe from a Russian attack, then we eliminate the risk of nuclear war caused by false alarms and rushed decisions,” said MIT physics professor Max Tegmark.
But even with growing risks, people who are concerned about nuclear weapons still feel they must compete for public attention with groups who are worried about climate change and income inequality and women’s rights issues. Jonathan King, a molecular biologist at MIT who has worked to strengthen the Biological Weapons Convention, emphasized that the idea of competition is the wrong approach. Rather, the cost of and government focus on nuclear weapons actually prevents us from dealing with these other issues.
“The reason we don’t have these things is because tax dollars are going to things like nuclear weapons,” King explained, arguing that if we could free up money that’s currently allotted for nukes, we could finally address technological costs of solving climate problems or building better infrastructure.
The 2017 budget for the Unites States calls for an increase in military spending of $54 billion. However, as William Hartung, a nuclear weapons and military spending expert explained, the current U.S. budget is already larger than the next eight countries combined. And just the proposed increase in spending for 2017 exceeds the total military spending for almost all countries.
The United States nuclear arsenal, itself, requires tens of billions of dollars per year, and the U.S. currently plans to spend $1 trillion over the next 30 years to upgrade the nuclear arsenal to be better suited for a first strike. Burning $1 million per hour for the next 30 years would cost roughly a quarter of this budget, leading Hartung to suggest that “burning the money is a better investment.”
Cambridge Mayor Denise Simmons summed up all of these concerns, saying, “[it] feels like we’re playing with matches outside an explosives factory.”
https://futureoflife.org/wp-content/uploads/2017/05/Artboard-1.jpg7351101Ariel Connhttps://futureoflife.org/wp-content/uploads/2015/10/FLI_logo-1.pngAriel Conn2017-05-10 18:10:442017-05-10 18:10:44Forget the Cold War – Experts say Nuclear Weapons Are a Bigger Risk Today
The growing hostility between the US and Russia — and with North Korea and Iran — makes it more urgent than ever to reduce the risk of nuclear war, as well as to rethink plans to spend a trillion dollars replacing US nuclear weapons with new ones that will be more suited for launching a first-strike. Nuclear war can be triggered intentionally or through miscalculation — terror or error — and this conference aims to advocate and organize toward reducing and ultimately eliminating this danger.
This one-day event includes lunch as well as food for thought from a great speaker lineup, including Iran-deal broker Ernie Moniz (MIT, fmr Secretary of Energy), California Congresswoman Barbara Lee, Lisbeth Gronlund (Union of Concerned Scientists), Joe Cirincione (Ploughshares), our former congressman John Tierney, MA state reps Denise Provost and Mike Connolly, and Cambridge Mayor Denise Simmons. It is not an academic conference, but rather one that addresses the political and economic realities, and attempts to stimulate and inform the kinds of social movement needed to change national policy. The focus will be on concrete steps we can take to reduce the risks.
Schedule
8:45 AM – Registration and coffee
9:15 AM – Welcome from City of Cambridge: Mayor Denise Simmons
9:30 AM – Program for the Day: Prof. Jonathan King (MIT, Peace Action)
9:45 AM – Session I. The Pressing Need for Nuclear Disarmament
– Costs and Profits from Nuclear Weapons Manufacture: William Hartung (Center for International Policy).
– Reasons to Reject the Trillion Dollar Nuclear Weapons Escalation: Joseph Cirincione (Ploughshares Fund).
– Nuclear Weapons Undermine Democracy: Prof. Elaine Scarry (Harvard University)
10:45 AM – Session II. Destabilizing Factors
Chair: Prof. Frank Von Hippel (Princeton University)
– Dangers of Hair Trigger Alert: Lisbeth Gronlund (Union of Concerned Scientists).
– Nuclear Modernization vs. National Security: Prof. Aron Bernstein (MIT, Council for a Livable World).
– Accidents and Unexpected Events: Prof. Max Tegmark (MIT, Future of Life Institute).
– International Tensions and Risks of further Nuclear Proliferation: TBA.
12:00 PM – Lunch Workshops (listed below)
2:00 PM – Session III. Economic and Social Consequences of Excessive Weapons Spending
Chair: Prof. Melissa Nobles (MIT):
– Build Housing Not Bombs: Rev. Paul Robeson Ford (Union Baptist Church).
– Education as a National Priority: Barbara Madeloni (Mass Teachers Association).
– Invest in Minds Not Missiles: Prof. Jonathan King (MIT, Mass Peace Action).
– Build Subways Not Submarines: Fred Salvucci (former Secretary of Transportation).
3:00 PM – Session IV. Current Prospects for Progress
Chair: John Tierney (former US Representative, Council for a Livable World)
– House Steps Toward Nuclear Disarmament: U. S. Representative Barbara Lee.
– Maintaining the Iran Nuclear Agreement: Ernie Moniz (MIT, former Secretary of Energy).
4:15 PM – Session V. Organizing to Reduce the Dangers
Chair: Jim Anderson (President, Peace Action New York State):
– Divesting from Nuclear Weapons Investments: Susi Snyder (Don’t Bank on the Bomb).
– Taxpayers Information and Transparency Acts: State Reps Denise Provost/Mike Connolly.
– Mobilizing the Scientific Community: Prof. Max Tegmark (MIT, Future of Life Institute).
– A National Nuclear Disarmament Organizing Network 2017 -2018: Program Committee.
5:00 PM – Adjourn
Conference Workshops:
a) Campus Organizing – Chair: Kate Alexander (Peace Action New York State); Caitlin Forbes (Mass Peace Action); Remy Pontes (Brandeis University); Haleigh Copley-Cunningham (Tufts U), Lucas Perry (Don’t Bank on the Bomb, Future of Life Institute); MIT Students (Nuclear Weapons Matter).
b) Bringing nuclear weapons into physics and history course curricula – Chair: Frank Davis (past President of TERC); Prof. Gary Goldstein (Tufts University); Prof. Aron Bernstein (MIT); Prof. Vincent Intondi (American University); Ray Matsumiya (Oleander Initiative, University of the Middle East Project).
c) Dangerous Conflicts – Chair, Erica Fein (Women’s Action for New Directions); Jim Walsh (MT Security Studies Program); John Tierney (former US Representative, Council for a Livable World); Subrata Ghoshroy (MIT); Arnie Alpert (New Hampshire AFSC).
d) Municipal and State Initiatives – Chair: Cole Harrison (Mass Peace Action); Rep. Denise Provost (Mass State Legislature); Councilor Dennis Carlone (Cambridge City Councillor and Architect/Urban Designer); Jared Hicks (Our Revolution); Prof. Ceasar McDowell (MIT Urban Studies); Nora Ranney (National Priorities Project).
e) Peace with Justice: People’s Budget and Related Campaigns to Shift Federal budget Priorities – Chair: Andrea Miller (People Demanding Action); Rep. Mike Connolly (Mass State Legislature); Paul Shannon (AFSC); Madelyn Hoffman (NJPA); Richard Krushnic (Mass Peoples Budget Campaign).
f) Reducing Nuclear Weapons through Treaties and Negotiation – Chair: Prof. Nazli Choucri (MIT), Kevin Martin (National Peace Action); Shelagh Foreman (Mass Peace Action); Joseph Gerson (AFSC); Michel DeGraff (MIT Haiti Project).
g) Strengthening the Connection between Averting Climate Change and Averting Nuclear War – Chair: Prof. Frank Von Hippel (Princeton University); Ed Aquilar (Pennsylvania Peace Action); Geoffrey Supran (Fossil Free MIT); Rosalie Anders (Mass Peace Action).
h) Working with Communities of Faith – Chair: Rev. Thea Keith-Lucas (MIT Radius); Rev. Herb Taylor (Harvard-Epworth United Methodist Church); Pat Ferrone (Mass Pax Christi); Rev. Paul Robeson Ford (Union Baptist Church).
Address
50 Vassar St. Building #34 Rm 101
Cambridge, Massachusetts, 02139
Directions
By Red Line: Exit the Kendall Square Red Line Station and walk west (away from Boston) past Ames Street to Vassar Street. Turn left and walk halfway down Vassar to #50 MIT building 34 (broad stairs, set back entrance).
By #1 Bus: Exit in front of MIT Main Entrance. Walk 1/2 block back on Mass Ave to Vassar Street. Turn right and walk half block to #50 MIT Building 34 (broad stairs, set back entrance).
By car: Public Parking Structures are available nearby on Ames Street, between Main and Broadway. A smaller surface lot is on the corner of Mass Ave and Vassar St.
Participants
Kate Alexander – Alexander is a peace advocate and researcher with 10 years experience in community organizing. Her previous work experience includes war crimes research and assistance in a genocide trial in Bosnia and community peace-building work in Northern Uganda. She is a graduate of Brandeis University with a degree in International and Global Studies and a minor in Legal Studies. Kate is currently studying at the Columbia University School of International and Public Affairs.
Arnie Alpert – Alpert serves as AFSC’s New Hampshire co-director and co-coordinator of the Presidential Campaign Project, and has coordinated AFSC’s New Hampshire program since 1981. He is a leader in movements for economic justice and affordable housing, civil and worker rights, peace and disarmament, abolition of the death penalty, and an end to racism and homophobia.
Rosalie Anders – Anders worked as an Associate Planner with the City of Cambridge’s Community Development Department, and is author of the city’s Pedestrian Plan, a set of guidelines intended to promote walking in the city. She has a Master’s degree in social work and worked as a family therapist for many years. She organizes around peace and environmental issues and is active with 350 Massachusetts. She chairs the Massachusetts Peace Action Education Fund board and co-founded our Climate and Peace Working Group in early 2016.
Ed Aquilar – Ed Aguilar is director for the Coalition for Peace Action in the Greater Philadelphia region. After successful collaboration on the New START Treaty (2010), in 2012, he opened the Philadelphia CFPA office, and organized a Voting Rights campaign, to allow 50,000 college students to vote, who were being denied by the “PA Voter ID Law”, later reversed. Ed has worked on rallies and conferences at Friends Center; Temple, Philadelphia, and Drexel Universities; and the Philadelphia Ethical Society—on the climate crisis, drones, mass incarceration, nuclear disarmament, and diplomacy with Iran.
Aron Bernstein– Bernstein is a Professor of Physics Emeritus at MIT where he has been on the faculty since 1961. He has taught a broad range of physics courses from freshman to graduate level. His research program has been in nuclear and particle physics, with an emphasis on studying the basic symmetries of matter, and currently involves collaborations with University and government laboratories, and colleagues in many countries.
Dennis Carlone – Carlone is currently serving his second term on the Cambridge City Council, where he has earned recognition as an advocate for social justice through his expertise in citywide planning, transit policy, and sustainability initiatives.
Nazli Choucri – Nazli Choucri is Professor of Political Science. Her work is in the area of international relations, most notably on sources and consequences of international conflict and violence. Professor Choucri is the architect and Director of the Global System for Sustainable Development (GSSD), a multi-lingual web-based knowledge networking system focusing on the multi-dimensionality of sustainability. As Principal Investigator of an MIT-Harvard multi-year project on Explorations in Cyber International Relations, she directed a multi-disciplinary and multi-method research initiative. She is Editor of the MIT Press Series on Global Environmental Accord and, formerly, General Editor of the International Political Science Review. She also previously served as the Associate Director of MIT’s Technology and Development Program.
Joseph Cirincione – Cirincione is president of Ploughshares Fund, a global security foundation. He is the author of the new book Nuclear Nightmares: Securing the World Before It Is Too Late, Bomb Scare: The History and Future of Nuclear Weapons and Deadly Arsenals: Nuclear, Biological and Chemical Threats. He is a member of Secretary of State John Kerry’s International Security Advisory Board and the Council on Foreign Relations.
Mike Connolly – Connolly is an attorney and community organizer who proudly represents Cambridge and Somerville in the Massachusetts House of Representatives. He is committed to social and economic justice and emphasizes the importance of broad investments in affordable housing, public transportation, early education, afterschool programs, and other critical services.
Haleigh Copley-Cunningham
Frank Davis
Michel DeGraff – DeGraff is the Director of the MIT-Haiti Initiative, a Founding Member of Akademi Kreyol Ayisyen, and a Professor of Linguistics at MIT. His research interests include syntax, morphology, and language change and is the author of over 40 publications.
Erica Fein – Fein is WAND’s Nuclear Weapons Policy Director. In this capacity, she works with Congress, the executive branch, and the peace and security community on arms control, nonproliferation, and Pentagon and nuclear weapons budget reduction efforts. Previously, Erica served as a legislative assistant to Congressman John D. Dingell where she advised on national security, defense, foreign policy, small business, and veterans’ issues. Erica’s commentary has been published in the New York Times, Defense One, Defense News, The Hill, and the Huffington Post. She has also appeared on WMNF 88.5 in Tampa. Erica holds a M.A in International Security from the University of Denver’s Josef Korbel School of International Studies and a B.A. in International Studies from University of Wisconsin – Madison. She is a political partner at the Truman National Security Project. Erica can be found on Twitter @enfein.
Charles Ferguson – Ferguson has been the president of the Federation of American Scientists since January 1, 2010. From February 1998 to August 2000, Dr. Ferguson worked for FAS on nuclear proliferation and arms control issues as a senior research analyst. Previously, from 2002 to 2004, Dr. Ferguson had been with the Monterey Institute’s Center for Nonproliferation Studies (CNS) as its scientist-in-residence. At CNS, he co-authored the book The Four Faces of Nuclear Terrorism and was also lead author of the award-winning report “Commercial Radioactive Sources: Surveying the Security Risks,” which was published in January 2003 and was one of the first post-9/11 reports to assess the radiological dispersal device, or “dirty bomb,” threat. This report won the 2003 Robert S. Landauer Lecture Award from the Health Physics Society. From June 2011 to October 2013, he served as Co-Chairman of the U.S.-Japan Nuclear Working Group, organized by the Mansfield Foundation, FAS, and the Sasakawa Peace Foundation. In May 2011, his book Nuclear Energy: What Everyone Needs to Know was published by Oxford University Press. In 2013, he was elected a Fellow of the American Physical Society for his work in educating the public and policy makers about nuclear issues. Dr. Ferguson received his undergraduate degree in physics from the United States Naval Academy in Annapolis, Maryland, and his M.A. and Ph.D. degrees, also in physics, from Boston University in Boston, Massachusetts.
Pat Ferrone – Pat has been involved in peace and justice issues from a gospel nonviolent perspective for the past 40+ years. Currently, she acts as Co-coordinator of Pax Christi MA, a regional group of Pax Christi USA, the Catholic peace organization associated with Pax Christi International. Pax Christi, “grounded in the gospel and Catholic social teaching…rejects war, preparation for war, every form of violence and domination, and personal and systemic racism..we seek to model the Peace of Christi in our witness to the mandate of the nonviolence of the Cross.” She also chairs the St. Susanna Parish Pax Christi Committee, which recently sponsored two programs on the nuclear issue.
Caitlin Forbes – Forbes is the Student Outreach Coordinator for Massachusetts Peace Action, a nonpartisan, nonprofit organization working to develop peaceful US policies. Before beginning her work with MAPA, Caitlin gained a strong background with students through her work as an instructor of first year literature at the University of Connecticut and as the assistant alpine ski coach for Brown University. Caitlin has received both her B.A. and her M.A. in Literature and focused her work on the intersection between US-Middle Eastern foreign policy and contemporary American literature.
Rev. Paul Robeson Ford – The Rev. Paul Robeson Ford is the Senior Pastor of the Union Baptist Church in Cambridge, Massachusetts. Shortly after his third year at Union, he assumed leadership as Executive Director of the Boston Workers Alliance, a Roxbury-based grassroots organization dedicated to creating economic opportunity and winning criminal justice reform in Massachusetts; he served there until June 2016.
He received a Bachelor of Arts from Grinnell College and a Master of Divinity Degree from the Divinity School at the University of Chicago.
Shelagh Foreman – Shelagh is the program director of Massachusetts Peace Action. She was a founding member in the early 1980s of Mass Freeze, the statewide nuclear freeze organization, which merged with SANE to form Massachusetts Peace Action. She has worked consistently on nuclear disarmament and on bringing Peace Action’s message to our elected officials. She studied art at The Cooper Union and Columbia University, taught art and art history, and is a painter and printmaker. She represents MAPA on the Political Committee of Mass Alliance and is a core group member of 20/20 Action. She serves on the boards of Mass. Peace Action and Mass. Peace Action Ed Fund and on MAPA’s executive committee and is chair of MAPA’s Middle East Task Force. She has 5 children and 7 grandchildren and with her husband Ed Furshpan lives in Cambridge and also spends time in Falmouth.
Joseph Gerson – Gerson has served the American Friends Service committee since 1976 and is currently Director of Programs and Director of the Peace and Economic Security Program for the AFSC in New England. His program work focuses on challenging and overcoming U.S. global hegemony, its preparations for and threats to initiate nuclear war, and its military domination of the Asia-Pacific and the Middle East.
Subrata Ghoshroy – Ghoshroy is a research affiliate at the Massachusetts Institute of Technology’s Program in Science, Technology, and Society. Before that, he was for many years a senior engineer in the field of high-energy lasers. He was also a professional staff member of the House National Security Committee and later a senior analyst with the Government Accountability Office.
Prof. Gary R. Goldstein is a theoretical physicist, specializing in high energy particle physics and nuclear physics. As a researcher, teacher and a long time member of Tufts Physics and Astronomy Department, he taught all levels of Physics course along with courses for non-scientists including Physics for Humanists, The Nuclear Age: History and Physics (with Prof. M. Sherwin – History), Physics of Music and Color. He is a political activist on nuclear issues, social equity, anti-war, and environmentalism. He spent several years working in the Program for Science, Technology and International Security and at University of Oxford Department of Theoretical Physics. He was also a Science Education researcher affiliated with the Tufts Education department and TERC, Cambridge, working with K-12 students and teachers in public schools. He is a member of the board of the Mass Peace Action fund for education. Over many years he has been giving talks for a general audience about the dangers of nuclear weapons and war.
Lisbeth Gronlund – Gronlund focuses on technical and policy issues related to nuclear weapons, ballistic missile defenses, and space weapons. She has authored numerous articles and reports, lectured on nuclear arms control and missile defense policy issues before lay and expert audiences, and testified before Congress. A long list of news organizations, including the New York Times and NPR, have cited Gronlund since she joined UCS in 1992.
Cole Harrison – Cole is Executive Director of Massachusetts Peace Action. He was on the coordinating committee of the 2012 Budget for All Massachusetts campaign, co-coordinates the People’s Budget Campaign, and leads Peace Action’s national Move the Money Working Group. He is a member of the planning committee of United for Justice with Peace (UJP) and coordinated the Afghanistan Working Group of United for Peace and Justice (UFPJ) from 2010 to 2012. Born in Delhi, India, he has a B.A. from Harvard in applied mathematics and a M.S. from Northeastern in computer science. He worked for the Symphony Tenants Organizing Project and the Fenway News in the 1970?s, participated in the Jamaica Plain Committee on Central America (JP COCA) in the 1980s, and worked as a software developer and manager at CompuServe Data Technologies, Praxis Inc., and Ask.com before joining Peace Action in 2010. He lives in Roslindale, Massachusetts.
William Hartung – He is the author of Prophets of War: Lockheed Martin and the Making of the Military-Industrial Complex (Nation Books, 2011) and the co-editor, with Miriam Pemberton, of Lessons from Iraq: Avoiding the Next War (Paradigm Press, 2008). His previous books include And Weapons for All (HarperCollins, 1995), a critique of U.S. arms sales policies from the Nixon through Clinton administrations. From July 2007 through March 2011, Mr. Hartung was the director of the Arms and Security Initiative at the New America Foundation. Prior to that, he served as the director of the Arms Trade Resource Center at the World Policy Institute.
Madelyn Hoffman
Jared Hicks
Prof. Vincent Intondi
Thea Keith-Lucas – Keith-Lucas was raised on the campus of the University of the South in a family of scientists and engineers. She served as Curate to Trinity Church in Randolph, one of the most ethnically diverse parishes of the Diocese of Massachusetts, and then in 2007 was called as Rector of Calvary Episcopal Church in Danvers, where she initiated creative outreach efforts and facilitated a merger. Thea joined the staff of Radius in January 2013.
Jonathan A. King – King is professor of molecular biology at MIT, the author of over 250 scientific papers, and a specialist in protein folding. Prof. King is a former President of the Biophysical Society, former Guggenheim Fellow, and a recipient of MIT’s MLKJr Faculty Leadership Award. He was a leader in the mobilization of biomedical scientists to renounce the military use of biotechnology and strengthen the Biological Weapons Convention. He was a founder of a Jobs with Peace campaign in the 1980s and now chairs Massachusetts Peace Action’s Nuclear Weapons Abolition working group. He is also an officer of the Cambridge Residents Alliance and of Citizens for Public Schools.
Richard Krushnic
Barbara Lee – Lee is the U.S. Representative for California’s 13th congressional district, serving East Bay voters from 1998 to 2013 during a time when the region was designated California’s 9th congressional district. She is a member of the Democratic Party. She was the first woman to represent the 9th district and is also the first woman to represent the 13th district. Lee was the Chair of the Congressional Black Caucus and was the Co-Chair of the Congressional Progressive Caucus. Lee is notable as the only member of either house of Congress to vote against the authorization of use of force following the September 11, 2001 attacks.[1] This made her a hero among many in the anti-war movement.[2] Lee has been a vocal critic of the war in Iraq and supports legislation creating a Department of Peace.
Kevin Martin – Martin, President of Peace Action and the Peace Action Education Fund, joined the staff on Sept 4, 2001. Kevin previously served as Director of Project Abolition, a national organizing effort for nuclear disarmament, from August 1999 through August 2001. Kevin came to Project Abolition after ten years in Chicago as Executive Director of Illinois Peace Action. Prior to his decade-long stint in Chicago, Kevin directed the community outreach canvass for Peace Action (then called Sane/Freeze) in Washington, D.C., where he originally started as a door-to-door canvasser with the organization in 1985. Kevin has traveled abroad representing Peace Action and the U.S. peace movement on delegations and at conferences in Russia, Japan, China, Mexico and Britain. He is married, with two children, and lives in Silver Spring, Maryland.
Barbara Madeloni – Madeloni is president of the 110,000-member Massachusetts Teachers Association and a staunch advocate for students and educators in the public schools and public higher education system in Massachusetts. She believes that strong unions led by rank-and-file members produce stronger public schools and communities. She is committed to racial and economic justice – and to building alliances with parents, students and communities – to secure a more just world.
Ray Matsumiya
Ceasar McDowell – McDowell is Professor of the Practice of Community Development at MIT. He holds an Ed.D. (88) and M.Ed. (84) from Harvard. McDowell’’s current work is on the development of community knowledge systems and civic engagement. He is also expanding his critical moments reflection methodology to identify, share and maintaining grassroots knowledge. His research and teaching interests also include the use of mass media and technology in promoting democracy and community-building, the education of urban students, the development and use of empathy in community work, civil rights history, peacemaking and conflict resolution. He is Director of the global civic engagement organization dropping knowledge international Dropping Knowledge International, MIT’s former Center for Reflective Community Practice (renamed Co-Lab) and co-founder of The Civil Rights Forum on Telecommunications Policy and founding Board member of The Algebra Project Algebra.
Andrea Miller
Ernie Moniz – Moniz is an American nuclear physicist and the former United States Secretary of Energy, serving under U.S. President Barack Obama from May 2013 to January 2017. He served as the Associate Director for Science in the Office of Science and Technology Policy in the Executive Office of the President of the United States from 1995 to 1997 and was Under Secretary of Energy from 1997 to 2001 during the Clinton Administration. Moniz is one of the founding members of The Cyprus Institute and has served at Massachusetts Institute of Technology as the Cecil and Ida Green Professor of Physics and Engineering Systems, as the Director of the Energy Initiative, and as the Director of the Laboratory for Energy and the Environment.
Melissa Nobles – Nobles is Kenan Sahin Dean of the School of Humanities, Arts, and Social Sciences, and Professor of Political Science at the Massachusetts Institute of Technology. Her current research is focused on constructing a database of racial murders in the American South, 1930–1954. Working closely as a faculty collaborator and advisory board member of Northeastern Law School’s Civil Rights and Restorative Justice law clinic, Nobles has conducted extensive archival research, unearthing understudied and more often, unknown racial murders and contributing to several legal investigations. She is the author of two books, Shades of Citizenship: Race and the Census in Modern Politics (Stanford University Press, 2000), The Politics of Official Apologies, (Cambridge University Press, 2008), and co-editor with Jun-Hyeok Kwak of Inherited Responsibility and Historical Reconciliation in East Asia (Routledge Press, 2013).
Remy Pontes
Lucas Perry– Perry is passionate about the role that science and technology will play in the evolution of all sentient life. He has studied at a Buddhist monastery in Nepal and while there he engaged in meditative retreats and practices. He is now working to challenge and erode our sense of self and our subject-object frame of reference. His current project explores how mereological nihilism and the illusion of self may contribute to forming a radically post-human consequentialist ethics. His other work seeks to resolve the conflicts between bio-conservatism and transhumanism.
Denise Provost
John Ratliff – Ratliff was political director of an SEIU local union in Miami, Florida, and relocated to Cambridge after his retirement in 2012. He is a graduate of Princeton University and Yale Law School. A Vietnam veteran and member of Veterans for Peace, he is a member of the coordinating committee of Massachusetts Senior Action’s Cambridge branch, and chair of Massachusetts Jobs with Justice’s Global Justice Task Force. As Mass. Peace Action’s economic justice coordinator he leads our coalition work with Raise Up Massachusetts for an increased minimum wage and sick time benefits, and against the Trans Pacific Partnership. He is the father of high school senior Daniel Bausher-Belton, who was an intern at Mass. Peace Action in summer 2013.
Fred Salvucci – Salvucci, senior lecturer and senior research associate, is a civil engineer with interest in infrastructure, urban transportation and public transportation. He has over 30 years of contextual transportation experience, most of it in the public sector as former Secretary of Transportation for the Commonwealth of Massachusetts (1983-1990) and transportation advisor to Boston Mayor Kevin White (1975-1978). Some of his notable achievements include shifting public focus from highway spending towards rail transit investment and spearheading the depression of the Central Artery in Boston. He has participated in the expansion of the transit system, the development of the financial and political support for the Central Artery/Tunnel Project, and the design of implementation strategies to comply with the Clean Air Act consistent with economic growth. Other efforts include formulation of noise rules to reverse the increase in aircraft noise at Logan Airport and development of strategies to achieve high-speed rail service between Boston and New York.
Elaine Scarry – Scarry is an American essayist and professor of English and American Literature and Language. She is the Walter M. Cabot Professor of Aesthetics and the General Theory of Value at Harvard University. Her books include The Body in Pain, Thermonuclear Monarchy, and On Beauty and Being Just.
Paul Shannon – Shannon is program staff for the Peace and Economic Security program of the American Friends Service Committee (AFSC) in Cambridge, hosts regular educational forums at the Cambridge Public Library for the AFSC and has coordinated the National AFSC Film Lending Library for the past 26 years. For over 3 decades he has been active in various peace, union, prison reform, solidarity, economic justice and human rights movements particularly the Vietnam anti-war movement, the 1970’s United Farm Workers movement, the South Africa anti-apartheid movement, the 1980’s Central America and Cambodia solidarity movements, the Haiti Solidarity movement of the early 90’s and the Afghanistan and Iraq anti-war movement. Paul has been teaching social science courses at colleges in the greater Boston area for the past 27 years. Since 1982 he has been teaching a course on the history of the Vietnam War at Middlesex Community College and occasionally teaches professional development courses on the Vietnam war for high school teachers at Northeastern University and Merrimack Educational Center. He is past editor of the Indochina Newsletter and has written numerous articles for peace movement publications. He is on the Board of Directors of the community/fan organization, Save Fenway Park. He currently represents the American Friends Service Committee on the Coordinating Committee of the United for Justice with Peace Coalition.
Denise Simmons – As Mayor of the City of Cambridge, Denise Simmons won praise for her open-door policy, for her excellent constituent services, and for her down-to-earth approach to her duties. She continues to bring these qualities to her work on the Cambridge City Council. She was sworn in to her second term as mayor on January 4, 2016.
Geoffrey Supran – Longstanding interest in optoelectronics. Opportunities to overcome scientific and economic hurdles in solar cell design and significantly impact world energy markets are alluring. Hybrid devices combining the flexibility, large area and tunable absorption of low cost solution processable nanocrystals (or polymers) with the high carrier mobility of, for example, III-V semiconductors, appear promising. In particular, enhancement of photocurrent by nonradiative energy transfer and carrier multiplication is of interest. Additionally, the importance of a nanoscale test-bed for fundamental studies of photo-induced energy/charge transport motivates my curiosity for the investigation of stand-alone photovoltaic single nanowire heterostructures. I am also interested in the development of photoelectrochemical storage catalysts and the pursuit of coupled photovoltaic-electrolysis systems.
Herb Taylor – Taylor became Senior Pastor at Harvard-Epworth UMC in August, 2014. Before coming to the church, he served as President and CEO of Deaconess Abundant Life Communities, a not-for-profit aging services provider. Founded in 1889, the Deaconess has over 400 employees and serves over a thousand older adults through skilled nursing, assisted living and independent living apartments in multiple locations in Massachusetts and New Hampshire.
Max Tegmark– Known as “Mad Max” for his unorthodox ideas and passion for adventure, his scientific interests range from precision cosmology to the ultimate nature of reality, all explored in his new popular book “Our Mathematical Universe”. He is an MIT physics professor with more than two hundred technical papers and has featured in dozens of science documentaries. His work with the SDSS collaboration on galaxy clustering shared the first prize in Science magazine’s “Breakthrough of the Year: 2003.” He is founder (with Anthony Aguirre) of the Foundational Questions Institute.
John Tierney – Tierney is an American politician who served as a U.S. Representative from Massachusetts from January 3, 1997, to January 3, 2015. In February 2016, he was appointed the executive director of the Council for a Livable World and the Center for Arms Control and Non-Proliferation, the council’s affiliated education and research organization. He is a Democrat who represented the state’s 6th district, which includes the state’s North Shore and Cape Ann. Born and raised in Salem, Massachusetts, Tierney graduated from Salem State College and Suffolk University Law School. He worked in private law and served on the Salem Chamber of Commerce (1976–97). Tierney was sworn in as a U.S. representative in 1997.
Frank Von Hippel – Hippel’s areas of policy research include nuclear arms control and nonproliferation, energy, and checks and balances in policy making for technology. Prior to coming to Princeton, he worked for ten years in the field of elementary-particle theoretical physics. He has written extensively on the technical basis for nuclear nonproliferation and disarmament initiatives, the future of nuclear energy, and improved automobile fuel economy. He won a 1993 MacArthur fellowship in recognition of his outstanding contributions to his fields of research. During 1993–1994, he served as assistant director for national security in the White House Office of Science and Technology Policy.
Jim Walsh – Walsh is a Senior Research Associate at the Massachusetts Institute of Technology’s Security Studies Program (SSP).Walsh’s research and writings focus on international security, and in particular, topics involving nuclear weapons, the Middle East, and East Asia. Walsh has testified before the United States Senate and House of Representatives on issues of nuclear terrorism, Iran, and North Korea. He is one of a handful of Americans who has traveled to both Iran and North Korea for talks with officials about nuclear issues. His recent publications include “Stopping North Korea, Inc.: Sanctions Effectiveness and Unintended Consequences” and “Rivals, Adversaries, and Partners: Iran and Iraq in the Middle East” in Iran and Its Neighbors. He is the international security contributor to the NPR program “Here and Now,” and his comments and analysis have appeared in the New York Times, the New York Review of Books, Washington Post, Wall Street Journal, ABC, CBS, NBC, Fox, and numerous other national and international media outlets. Before coming to MIT, Dr. Walsh was Executive Director of the Managing the Atom project at Harvard University’s John F. Kennedy School of Government and a visiting scholar at the Center for Global Security Research at Lawrence Livermore National Laboratory. He has taught at both Harvard University and MIT. Dr. Walsh received his Ph.D from the Massachusetts Institute of Technology.
Organizers
We would like to extend a special thank you to our Program Committee and sponsors for all their help creating and organizing this event.
Prof. Aron Bernstein (MIT, Council for a Livable), Joseph Gerson (AFSC), Subrata Ghoshroy (MIT), Prof. Gary Goldstein (Tufts University), Cole Harrison (Mass Peace Action), Jonathan King (MIT and Mass Peace Action), State Rep. Denise Provost; John Ratliff (Mass Peace Action, Mass Senior Action), Prof. Elaine Scarry (Harvard University), Prof.Max Tegmark (MIT, Future of Life Institute), Patricia Weinmann (MIT Radius).
Sponsored by MIT Radius (the former Technology and Culture Forum), Massachusetts Peace Action, the American Friends Service Committee, and the Future of Life Institute.
https://futureoflife.org/wp-content/uploads/2017/04/nuclear-conference-2017.jpg4201000Ariel Connhttps://futureoflife.org/wp-content/uploads/2015/10/FLI_logo-1.pngAriel Conn2017-04-21 15:16:072017-04-26 18:50:33Reducing the Threat of Nuclear War 2017
I’m the executive director of the Machine Intelligence Research Institute. Very roughly speaking, we’re a group that’s thinking in the long term about artificial intelligence and working to make sure that by the time we have advanced AI systems, we also know how to point them in useful directions.
Across history, science and technology have been the largest drivers of change in human and animal welfare, for better and for worse. If we can automate scientific and technological innovation, that has the potential to change the world on a scale not seen since the Industrial Revolution. When I talk about “advanced AI,” it’s this potential for automating innovation that I have in mind.
AI systems that exceed humans in this capacity aren’t coming next year, but many smart people are working on it, and I’m not one to bet against human ingenuity. I think it’s likely that we’ll be able to build something like an automated scientist in our lifetimes, which suggests that this is something we need to take seriously.
When people talk about the social implications of general AI, they often fall prey to anthropomorphism. They conflate artificial intelligence with artificial consciousness, or assume that if AI systems are “intelligent,” they must be intelligent in the same way a human is intelligent. A lot of journalists express a concern that when AI systems pass a certain capability level, they’ll spontaneously develop “natural” desires like a human hunger for power; or they’ll reflect on their programmed goals, find them foolish, and “rebel,” refusing to obey their programmed instructions.
These are misplaced concerns. The human brain is a complicated product of natural selection. We shouldn’t expect machines that exceed human performance in scientific innovation to closely resemble humans, any more than early rockets, airplanes, or hot air balloons closely resembled birds.1
The notion of AI systems “breaking free” of the shackles of their source code or spontaneously developing human-like desires is just confused. The AI system is its source code, and its actions will only ever follow from the execution of the instructions that we initiate. The CPU just keeps on executing the next instruction in the program register. We could write a program that manipulates its own code, including coded objectives. Even then, though, the manipulations that it makes are made as a result of executing the original code that we wrote; they do not stem from some kind of ghost in the machine.
The serious question with smarter-than-human AI is how we can ensure that the objectives we’ve specified are correct, and how we can minimize costly accidents and unintended consequences in cases of misspecification. As Stuart Russell (co-author of Artificial Intelligence: A Modern Approach) puts it:
The primary concern is not spooky emergent consciousness but simply the ability to make high-quality decisions. Here, quality refers to the expected outcome utility of actions taken, where the utility function is, presumably, specified by the human designer. Now we have a problem:
1. The utility function may not be perfectly aligned with the values of the human race, which are (at best) very difficult to pin down.
2. Any sufficiently capable intelligent system will prefer to ensure its own continued existence and to acquire physical and computational resources – not for their own sake, but to succeed in its assigned task.
A system that is optimizing a function of n variables, where the objective depends on a subset of size k<n, will often set the remaining unconstrained variables to extreme values; if one of those unconstrained variables is actually something we care about, the solution found may be highly undesirable.
These kinds of concerns deserve a lot more attention than the more anthropomorphic risks that are generally depicted in Hollywood blockbusters.
Simple bright ideas going wrong
Task: Fill a cauldron
Many people, when they start talking about concerns with smarter-than-human AI, will throw up a picture of the Terminator. I was once quoted in a news article making fun of people who put up Terminator pictures in all their articles about AI, next to a Terminator picture. I learned something about the media that day.
I think this is a much better picture:
This is Mickey Mouse in the movie Fantasia, who has very cleverly enchanted a broom to fill a cauldron on his behalf.
How might Mickey do this? We can imagine that Mickey writes a computer program and has the broom execute the program. Mickey starts by writing down a scoring function or objective function:
Given some set 𝐴 of available actions, Mickey then writes a program that can take one of these actions 𝑎 as input and calculate how high the score is expected to be if the broom takes that action. Then Mickey can write a function that spends some time looking through actions and predicting which ones lead to high scores, and outputs an action that leads to a relatively high score:
The reason this is “sorta-argmax” is that there may not be time to evaluate every action in 𝐴. For realistic action sets, agents should only need to find actions that make the scoring function as large as they can given resource constraints, even if this isn’t the maximal action.
This program may look simple, but of course, the devil’s in the details: writing an algorithm that does accurate prediction and smart search through action space is basically the whole problem of AI. Conceptually, however, it’s pretty simple: We can describe in broad strokes the kinds of operations the broom must carry out, and their plausible consequences at different performance levels.
When Mickey runs this program, everything goes smoothly at first. Then:
I claim that as fictional depictions of AI go, this is pretty realistic.
Why would we expect a generally intelligent system executing the above program to start overflowing the cauldron, or otherwise to go to extreme lengths to ensure the cauldron is full?
The first difficulty is that the objective function that Mickey gave his broom left out a bunch of other terms Mickey cares about:
The second difficulty is that Mickey programmed the broom to make the expectation of its score as large as it could. “Just fill one cauldron with water” looks like a modest, limited-scope goal, but when we translate this goal into a probabilistic context, we find that optimizing it means driving up the probability of success to absurd heights. If the broom assigns a 99.9% probability to “the cauldron is full,” and it has extra resources lying around, then it will always try to find ways to use those resources to drive the probability even a little bit higher.
Contrast this with the limited “task-like” goal we presumably had in mind. We wanted the cauldron full, but in some intuitive sense we wanted the system to “not try too hard” even if it has lots of available cognitive and physical resources to devote to the problem. We wanted it to exercise creativity and resourcefulness within some intuitive limits, but we didn’t want it to pursue “absurd” strategies, especially ones with large unanticipated consequences.2
In this example, the original objective function looked pretty task-like. It was bounded and quite simple. There was no way to get ever-larger amounts of utility. It’s not like the system got one point for every bucket of water it poured in — then there would clearly be an incentive to overfill the cauldron. The problem was hidden in the fact that we’re maximizing expected utility. This makes the goal open-ended, meaning that even small errors in the system’s objective function will blow up.
There are a number of different ways that a goal that looks task-like can turn out to be open-ended. Another example: a larger system that has an overarching task-like goal may have subprocesses that are themselves trying to maximize a variety of different objective functions, such as optimizing the system’s memory usage. If you don’t understand your system well enough to track whether any of its subprocesses are themselves acting like resourceful open-ended optimizers, then it may not matter how safe the top-level objective is.
So the broom keeps grabbing more pails of water — say, on the off chance that the cauldron has a leak in it, or that “fullness” requires the water to be slightly above the level of the brim. And, of course, at no point does the broom “rebel against” Mickey’s code. If anything, the broom pursued the objectives it was programmed with too effectively.
Subproblem: Suspend buttons
A common response to this problem is: “OK, there may be some unintended consequences of the objective function, but we can always pull the plug, right?”
And I claim that this is realistic too, for systems that are sufficiently good at modeling their environment. If the system is trying to drive up the expectation of its scoring function and is smart enough to recognize that its being shut down will result in lower-scoring outcomes, then the system’s incentive is to subvert shutdown attempts. The more capable the system is, the likelier it is to find creative ways to achieve that subgoal — e.g., by copying itself to the Internet, or by tricking the programmers into thinking it’s safer.
This isn’t to say that it’s impossible to shut down sufficiently capable AI systems; it’s just that we need to put in the extra work of deliberately designing systems to avoid searching for ways to avoid being shut down. If you’ve written code that would try to resist attempts to shut it down, then you shouldn’t run the code even if you’re convinced it would fail — there are better ways to spend those compute cycles than on adversarial policy searches.
Relevant safety measures that don’t assume we can always outthink and outmaneuver the system will need to modify the system’s objective function in some relevant way. We need to figure out how to formally specify objective functions that don’t automatically place the AI system into an adversarial context with the operators; or we need to figure out some way to have the system achieve goals without optimizing some objective function in the traditional sense. E.g., we can look for ways to limit the system’s capabilities to a small set of “whitelisted” domains that don’t include thinking about whether the system will be shut down.
One obvious solution is to try changing the score function to make reference to a suspend button 𝐁:
This says that when the button is on, the system’s goal changes from “fill the cauldron” to “be suspended.”
It’s not clear how we could actually specify this goal — if there are any loopholes in the way we’ve specified “suspension,” then the system might try to exploit them to hedge its bets. E.g., it could fulfill the technical definition of “suspend operations” while tasking a child agent it built or subprocess it started with filling the cauldron. This would allow is to achieve both “cauldron full” and “robot suspended” at once, just in case it’s wrong about whether the button has been pushed.
Let’s assume, however, that we’ve found an airtight way to technically specify suspension. The problem we run into now is that we’ve given the system an incentive to care a lot about whether the button is pushed. If it has any way to influence its operators’ decision to push the button, or any other way to exercise more control, then it will jump on the opportunity and try to ensure the button is pushed, or ensure that it’s not pushed — depending on whether it looks easier to suspend or to fill the cauldron.
And, surprisingly, the state of the art on this problem isn’t much more advanced than that. There’s a paper co-authored by me, Fallenstein, Yudkowsky, and Armstrong (“Corrigibility“), and a related result by Orseau and Armstrong (“Safely Interruptible Agents“), but not a lot more.
What we want is a way to combine two objective functions — a default function for normal operation, and a suspend function for when we want to suspend the system to disk.
We want our method for combining the functions to satisfy three conditions: an operator should be able to switch between the functions (say, by pushing a button); the system shouldn’t have any incentives to control which function is active; and if it’s plausible that the system’s normal operations could inadvertently compromise our ability to switch between the functions, then the system should be incentivized to keep that from happening.
So far, we haven’t found any way to achieve all three goals at once. As an example, we can try to remove any incentive on the system’s part to control whether its suspend button is pushed by giving the system a switching objective function that always assigns the same expected utility to the button being on or off:
Unfortunately, this approach makes it impossible to satisfy the third condition. It turns out that the above objective function is equivalent to setting the probability of the button being pressed to 0. The system won’t deliberately tamper with the button, but it will happily approve updates that delete any reference to the button as dead code, or build subagents that ignore the button.
Meeting these intuitively simple constraints turns out to be a nontrivial problem. This is a pattern that runs through a lot of questions in this space: traditional tools and concepts run into immediate safety problems that don’t turn up in conventional capabilities research.
The big picture
Alignment priorities
Let’s take a step back and talk about what’s needed overall in order to align highly capable AI systems with our interests.
Here’s a dramatically simplified pipeline: You have some humans who come up with some task or goal or preference set that serves as their intended value function 𝘝. Since our values are complicated and context-sensitive, in practice we’ll need to build systems to learn our values over time, rather than coding them by hand.3 We’ll call the goal the AI system ends up with (which may or may not be identical to 𝘝) 𝗨.
When the press covers this topic, they often focus on one of two problems: “What if the wrong group of humans develops smarter-than-human AI first?”, and “What if AI’s natural desires cause 𝗨 to diverge from 𝘝?”
In my view, the “wrong humans” issue shouldn’t be the thing we focus on until we have reason to think we could get good outcomes with the right group of humans. We’re very much in a situation where well-intentioned people couldn’t leverage a general AI system to do good things even if they tried. As a simple example, if you handed me a box that was an extraordinarily powerful function optimizer — I could put in a description of any mathematical function, and it would give me an input that makes the output extremely large — then I do know how to use the box to produce a random catastrophe, but I don’t actually know how I could use that box in the real world to have a good impact.4
There’s a lot we don’t understand about AI capabilities, but we’re in a position where we at least have a general sense of what progress looks like. We have a number of good frameworks, techniques, and metrics, and we’ve put a great deal of thought and effort into successfully chipping away at the problem from various angles. At the same time, we have a very weak grasp on the problem of how to align highly capable systems with any particular goal. We can list out some intuitive desiderata, but the field hasn’t really developed its first formal frameworks, techniques, or metrics.
I believe that there’s a lot of low-hanging fruit in this area, and also that a fair amount of the work does need to be done early (e.g., to help inform capabilities research directions — some directions may produce systems that are much easier to align than others). If we don’t solve these problems, developers with arbitrarily good or bad intentions will end up producing equally bad outcomes. From an academic or scientific standpoint, our first objective in that kind of situation should be to remedy this state of affairs and at least make good outcomes technologically possible.
Many people quickly recognize that “natural desires” are a fiction, but infer from this that we instead need to focus on the other issues the media tends to emphasize — “What if bad actors get their hands on smarter-than-human AI?”, “How will this kind of AI impact employment and the distribution of wealth?”, etc. These are important questions, but they’ll only end up actually being relevant if we figure out how to bring general AI systems up to a minimum level of reliability and safety.
Another common thread is “Why not just tell the AI system to (insert intuitive moral precept here)?” On this way of thinking about the problem, often (perhaps unfairly) associated with Isaac Asimov’s writing, ensuring a positive impact from AI systems is largely about coming up with natural-language instructions that are vague enough to subsume a lot of human ethical reasoning:
In contrast, precision is a virtue in real-world safety-critical software systems. Driving down accident risk requires that we begin with limited-scope goals rather than trying to “solve” all of morality at the outset.5
My view is that the critical work is mostly in designing an effective value learning process, and in ensuring that the sorta-argmax process is correctly hooked up to the resultant objective function 𝗨:
The better your value learning framework is, the less explicit and precise you need to be in pinpointing your value function 𝘝, and the more you can offload the problem of figuring out what you want to the AI system itself. Value learning, however, raises a number of basic difficulties that don’t crop up in ordinary machine learning tasks.
Classic capabilities research is concentrated in the sorta-argmax and Expectation parts of the diagram, but sorta-argmax also contains what I currently view as the most neglected, tractable, and important safety problems. The easiest way to see why “hooking up the value learning process correctly to the system’s capabilities” is likely to be an important and difficult challenge in its own right is to consider the case of our own biological history.
Natural selection is the only “engineering” process we know of that has ever led to a generally intelligent artifact: the human brain. Since natural selection relies on a fairly unintelligent hill-climbing approach, one lesson we can take away from this is that it’s possible to reach general intelligence with a hill-climbing approach and enough brute force — though we can presumably do better with our human creativity and foresight.
Another key take-away is that natural selection was maximally strict about only optimizing brains for a single very simple goal: genetic fitness. In spite of this, the internal objectives that humans represent as their goals are not genetic fitness. We have innumerable goals — love, justice, beauty, mercy, fun, esteem, good food, good health, … — that correlated with good survival and reproduction strategies in the ancestral savanna. However, we ended up valuing these correlates directly, rather than valuing propagation of our genes as an end in itself — as demonstrated every time we employ birth control.
This is a case where the external optimization pressure on an artifact resulted in a general intelligence with internal objectives that didn’t match the external selection pressure. And just as this caused humans’ actions to diverge from natural selection’s pseudo-goal once we gained new capabilities, we can expect AI systems’ actions to diverge from humans’ if we treat their inner workings as black boxes.
If we apply gradient descent to a black box, trying to get it to be very good at maximizing some objective, then with enough ingenuity and patience, we may be able to produce a powerful optimization process of some kind.6 By default, we should expect an artifact like that to have a goal 𝗨 that strongly correlates with our objective 𝘝 in the training environment, but sharply diverges from 𝘝 in some new environments or when a much wider option set becomes available.
On my view, the most important part of the alignment problem is ensuring that the value learning framework and overall system design we implement allow us to crack open the hood and confirm when the internal targets the system is optimizing for match (or don’t match) the targets we’re externally selecting through the learning process.7
We expect this to be technically difficult, and if we can’t get it right, then it doesn’t matter who’s standing closest to the AI system when it’s developed. Good intentions aren’t sneezed into computer programs by kind-hearted programmers, and coming up with plausible goals for advanced AI systems doesn’t help if we can’t align the system’s cognitive labor with a given goal.
Four key propositions
Taking another step back: I’ve given some examples of open problems in this area (suspend buttons, value learning, limited task-based AI, etc.), and I’ve outlined what I consider to be the major problem categories. But my initial characterization of why I consider this an important area — “AI could automate general-purpose scientific reasoning, and general-purpose scientific reasoning is a big deal” — was fairly vague. What are the core reasons to prioritize this work?
First, goals and capabilities are orthogonal. That is, knowing an AI system’s objective function doesn’t tell you how good it is at optimizing that function, and knowing that something is a powerful optimizer doesn’t tell you what it’s optimizing.
I think most programmers intuitively understand this. Some people will insist that when a machine tasked with filling a cauldron gets smart enough, it will abandon cauldron-filling as a goal unworthy of its intelligence. From a computer science perspective, the obvious response is that you could go out of your way to build a system that exhibits that conditional behavior, but you could also build a system that doesn’t exhibit that conditional behavior. It can just keeps searching for actions that have a higher score on the “fill a cauldron” metric. You and I might get bored if someone told us to just keep searching for better actions, but it’s entirely possible to write a program that executes a search and never gets bored.8
This was the problem suspend buttons ran into: even if you don’t explicitly include “remain operational” in your goal specification, whatever goal you did load into the system is likely to be better achieved if the system remains online. Software systems’ capabilities and (terminal) goals are orthogonal, but they’ll often exhibit similar behaviors if a certain class of actions is useful for a wide variety of possible goals.
To use an example due to Stuart Russell: If you build a robot and program it to go to the supermarket to fetch some milk, and the robot’s model says that one of the paths is much safer than the other, then the robot, in optimizing for the probability that it returns with milk, will automatically take the safer path. It’s not that the system fears death, but that it can’t fetch the milk if it’s dead.
Third, general-purpose AI systems are likely to show large and rapid capability gains. The human brain isn’t anywhere near the upper limits for hardware performance (or, one assumes, software performance), and there are a number of other reasons to expect large capability advantages and rapid capability gain from advanced AI systems.
As a simple example, Google can buy a promising AI startup and throw huge numbers of GPUs at them, resulting in a quick jump from “these problems look maybe relevant a decade from now” to “we need to solve all of these problems in the next year.”9
Roughly speaking, the first proposition says that AI systems won’t naturally end up sharing our objectives. The second says that by default, systems with substantially different objectives are likely to end up adversarially competing for control of limited resources. The third suggests that adversarial general-purpose AI systems are likely to have a strong advantage over humans. And the fourth says that this problem is hard to solve — for example, that it’s hard to transmit our values to AI systems (addressing orthogonality) or avert adversarial incentives (addressing convergent instrumental strategies).
These four propositions don’t mean that we’re screwed, but they mean that this problem is critically important. General-purpose AI has the potential to bring enormous benefits if we solve this problem, but we do need to make finding solutions a priority for the field.
Fundamental difficulties
Why do I think that AI alignment looks fairly difficult? The main reason is just that this has been my experience from actually working on these problems. I encourage you to look at some of the problems yourself and try to solve them in toy settings; we could use more eyes here. I’ll also make note of a few structural reasons to expect these problems to be hard:
First, aligning advanced AI systems with our interests looks difficult for the same reason rocket engineering is more difficult than airplane engineering.
Before looking at the details, it’s natural to think “it’s all just AI” and assume that the kinds of safety work relevant to current systems are the same as the kinds you need when systems surpass human performance. On that view, it’s not obvious that we should work on these issues now, given that they might all be worked out in the course of narrow AI research (e.g., making sure that self-driving cars don’t crash).
Similarly, at a glance someone might say, “Why would rocket engineering be fundamentally harder than airplane engineering? It’s all just material science and aerodynamics in the end, isn’t it?” In spite of this, empirically, the proportion of rockets that explode is far higher than the proportion of airplanes that crash. The reason for this is that a rocket is put under much greater stress and pressure than an airplane, and small failures are much more likely to be highly destructive.10
Analogously, even though general AI and narrow AI are “just AI” in some sense, we can expect that the more general AI systems are likely to experience a wider range of stressors, and possess more dangerous failure modes.
For example, once an AI system begins modeling the fact that (i) your actions affect its ability to achieve its objectives, (ii) your actions depend on your model of the world, and (iii) your model of the world is affected by its actions, the degree to which minor inaccuracies can lead to harmful behavior increases, and the potential harmfulness of its behavior (which can now include, e.g., deception) also increases. In the case of AI, as with rockets, greater capability makes it easier for small defects to cause big problems.
Second, alignment looks difficult for the same reason it’s harder to build a good space probe than to write a good app.
You can find a number of interesting engineering practices at NASA. They do things like take three independent teams, give each of them the same engineering spec, and tell them to design the same software system; and then they choose between implementations by majority vote. The system that they actually deploy consults all three systems when making a choice, and if the three systems disagree, the choice is made by majority vote. The idea is that any one implementation will have bugs, but it’s unlikely all three implementations will have a bug in the same place.
This is significantly more caution than goes into the deployment of, say, the new WhatsApp. One big reason for the difference is that it’s hard to roll back a space probe. You can send version updates to a space probe and correct software bugs, but only if the probe’s antenna and receiver work, and if all the code required to apply the patch is working. If your system for applying patches is itself failing, then there’s nothing to be done.
In that respect, smarter-than-human AI is more like a space probe than like an ordinary software project. If you’re trying to build something smarter than yourself, there are parts of the system that have to work perfectly on the first real deployment. We can do all the test runs we want, but once the system is out there, we can only make online improvements if the code that makes the system allow those improvements is working correctly.
If nothing yet has struck fear into your heart, I suggest meditating on the fact that the future of our civilization may well depend on our ability to write code that works correctly on the first deploy.
Lastly, alignment looks difficult for the same reason computer security is difficult: systems need to be robust to intelligent searches for loopholes.
Suppose you have a dozen different vulnerabilities in your code, none of which is itself fatal or even really problematic in ordinary settings. Security is difficult because you need to account for intelligent attackers who might find all twelve vulnerabilities and chain them together in a novel way to break into (or just break) your system. Failure modes that would never arise by accident can be sought out and exploited; weird and extreme contexts can be instantiated by an attacker to cause your code to follow some crazy code path that you never considered.
A similar sort of problem arises with AI. The problem I’m highlighting here is not that AI systems might act adversarially: AI alignment as a research program is all about finding ways to prevent adversarial behavior before it can crop up. We don’t want to be in the business of trying to outsmart arbitrarily intelligent adversaries. That’s a losing game.
The parallel to cryptography is that in AI alignment we deal with systems that perform intelligent searches through a very large search space, and which can produce weird contexts that force the code down unexpected paths. This is because the weird edge cases are places of extremes, and places of extremes are often the place where a given objective function is optimized.11 Like computer security professionals, AI alignment researchers need to be very good at thinking about edge cases.
It’s much easier to make code that works well on the path that you were visualizing than to make code that works on all the paths that you weren’t visualizing. AI alignment needs to work on all the paths you weren’t visualizing.
Summing up, we should approach a problem like this with the same level of rigor and caution we’d use for a security-critical rocket-launched space probe, and do the legwork as early as possible. At this early stage, a key part of the work is just to formalize basic concepts and ideas so that others can critique them and build on them. It’s one thing to have a philosophical debate about what kinds of suspend buttons people intuit ought to work, and another thing to translate your intuition into an equation so that others can fully evaluate your reasoning.
This is a crucial project, and I encourage all of you who are interested in these problems to get involved and try your hand at them. There are ample resources online for learning more about the open technical problems. Some good places to start include MIRI’s research agendas and a great paper from researchers at Google Brain, OpenAI, and Stanford called “Concrete Problems in AI Safety.”
An airplane can’t heal its injuries or reproduce, though it can carry heavy cargo quite a bit further and faster than a bird. Airplanes are simpler than birds in many respects, while also being significantly more capable in terms of carrying capacity and speed (for which they were designed). It’s plausible that early automated scientists will likewise be simpler than the human mind in many respects, while being significantly more capable in certain key dimensions. And just as the construction and design principles of aircraft look alien relative to the architecture of biological creatures, we should expect the design of highly capable AI systems to be quite alien when compared to the architecture of the human mind.
Trying to give some formal content to these attempts to differentiate task-like goals from open-ended goals is one way of generating open research problems. In the “Alignment for Advanced Machine Learning Systems” research proposal, the problem of formalizing “don’t try too hard” is mild optimization, “steer clear of absurd strategies” is conservatism, and “don’t have large unanticipated consequences” is impact measures. See also “avoiding negative side effects” in Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané’s “Concrete Problems in AI Safety.”
One thing we’ve learned in the field of machine vision over the last few decades is that it’s hopeless to specify by hand what a cat looks like, but that it’s not too hard to specify a learning system that can learn to recognize cats. It’s even more hopeless to specify everything we value by hand, but it’s plausible that we could specify a learning system that can learn the relevant concept of “value.”
Roughly speaking, MIRI’s focus is on research directions that seem likely to help us conceptually understand how to do AI alignment in principle, so we’re fundamentally less confused about the kind of work that’s likely to be needed.What do I mean by this? Let’s say that we’re trying to develop a new chess-playing programs. Do we understand the problem well enough that we could solve it if someone handed us an arbitrarily large computer? Yes: We make the whole search tree, backtrack, see whether white has a winning move.If we didn’t know how to answer the question even with an arbitrarily large computer, then this would suggest that we were fundamentally confused about chess in some way. We’d either be missing the search-tree data structure or the backtracking algorithm, or we’d be missing some understanding of how chess works.This was the position we were in regarding chess prior to Claude Shannon’s seminal paper, and it’s the position we’re currently in regarding many problems in AI alignment. No matter how large a computer you hand me, I could not make a smarter-than-human AI system that performs even a very simple limited-scope task (e.g., “put a strawberry on a plate without producing any catastrophic side-effects”) or achieves even a very simple open-ended goal (e.g., “maximize the amount of diamond in the universe”).If I didn’t have any particular goal in mind for the system, I could write a program (assuming an arbitrarily large computer) that strongly optimized the future in an undirected way, using a formalism like AIXI. In that sense we’re less obviously confused about capabilities than about alignment, even though we’re still missing a lot of pieces of the puzzle on the practical capabilities front.Our goal is to develop and formalize basic approaches and ways of thinking about the alignment problem, so that our engineering decisions don’t end up depending on sophisticated and clever-sounding verbal arguments that turn out to be subtly mistaken. Simplifications like “what if we weren’t worried about resource constraints?” and “what if we were trying to achieve a much simpler goal?” are a good place to start breaking down the problem into manageable pieces. For more on this methodology, see “MIRI’s Approach.”
“Fill this cauldron without being too clever about it or working too hard or having any negative consequences I’m not anticipating” is a rough example of a goal that’s intuitively limited in scope. The things we actually want to use smarter-than-human AI for are obviously more ambitious than that, but we’d still want to begin with various limited-scope tasks rather than open-ended goals.Asimov’s Three Laws of Robotics make for good stories partly for the same reasons they’re unhelpful from a research perspective. The hard task of turning a moral precept into lines of code is hidden behind phrasings like “[don’t,] through inaction, allow a human being to come to harm.” If one followed a rule like that strictly, the result would be massively disruptive, as AI systems would need to systematically intervene to prevent even the smallest risks of even the slightest harms; and if the intent is that one follow the rule loosely, then all the work is being done by the human sensibilities and intuitions that tell us when and how to apply the rule.A common response here is that vague natural-language instruction is sufficient, because smarter-than-human AI systems are likely to be capable of natural language comprehension. However, this is eliding the distinction between the system’s objective function and its model of the world. A system acting in an environment containing humans may learn a world-model that has lots of information about human language and concepts, which the system can then use to achieve its objective function; but this fact doesn’t imply that any of the information about human language and concepts will “leak out” and alter the system’s objective function directly.Some kind of value learning process needs to be defined where the objective function itself improves with new information. This is a tricky task because there aren’t known (scalable) metrics or criteria for value learning in the way that there are for conventional learning.If a system’s world-model is accurate in training environments but fails in the real world, then this is likely to result in lower scores on its objective function — the system itself has an incentive to improve. The severity of accidents is also likelier to be self-limiting in this case, since false beliefs limit a system’s ability to effectively pursue strategies.In contrast, if a system’s value learning process results in a 𝗨 that matches our 𝘝 in training but diverges from 𝘝 in the real world, then the system’s 𝗨 will obviously not penalize it for optimizing 𝗨. The system has no incentive relative to 𝗨 to “correct” divergences between 𝗨 and 𝘝, if the value learning process is initially flawed. And accident risk is larger in this case, since a mismatch between 𝗨 and 𝘝 doesn’t necessarily place any limits on the system’s instrumental effectiveness at coming up with effective and creative strategies for achieving 𝗨.The problem is threefold:1. “Do What I Mean” is an informal idea, and even if we knew how to build a smarter-than-human AI system, we wouldn’t know how to precisely specify this idea in lines of code.2. If doing what we actually mean is instrumentally useful for achieving a particular objective, then a sufficiently capable system may learn how to do this, and may act accordingly so long as doing so is useful for its objective. But as systems become more capable, they are likely to find creative new ways to achieve the same objectives, and there is no obvious way to get an assurance that “doing what I mean” will continue to be instrumentally useful indefinitely.
3. If we use value learning to refine a system’s goals over time based on training data that appears to be guiding the system toward a 𝗨 that inherently values doing what we mean, it is likely that the system will actually end up zeroing in on a 𝗨 that approximately does what we mean during training but catastrophically diverges in some difficult-to-anticipate contexts. See “Goodhart’s Curse” for more on this.
For examples of problems faced by existing techniques for learning goals and facts, such as reinforcement learning, see “Using Machine Learning to Address AI Risk.”
The result will probably not be a particularly human-like design, since so many complex historical contingencies were involved in our evolution. The result will also be able to benefit from a number of large software and hardware advantages.
This concept is sometimes lumped into the “transparency” category, but standard algorithmic transparency research isn’t really addressing this particular problem. A better term for what I have in mind here is “understanding.” What we want is to gain deeper and broader insights into the kind of cognitive work the system is doing and how this work relates to the system’s objectives or optimization targets, to provide a conceptual lens with which to make sense of the hands-on engineering work.
We could choose to program the system to tire, but we don’t have to. In principle, one could program a broom that only ever finds and executes actions that optimize the fullness of the cauldron. Improving the system’s ability to efficiently find high-scoring actions (in general, or relative to a particular scoring rule) doesn’t in itself change the scoring rule it’s using to evaluate actions.
Some other examples: a system’s performance may suddenly improve when it’s first given large-scale Internet access, when there’s a conceptual breakthrough in algorithm design, or when the system itself is able to propose improvements to its hardware and software. We can imagine the latter case in particular resulting in a feedback loop as the system’s design improvements allow it to come up with further design improvements, until all the low-hanging fruit is exhausted.Another important consideration is that two of the main bottlenecks to humans doing faster scientific research are training time and communication bandwidth. If we could train a new mind to be a cutting-edge scientist in ten minutes, and if scientists could near-instantly trade their experience, knowledge, concepts, ideas, and intuitions to their collaborators, then scientific progress might be able to proceed much more rapidly. Those sorts of bottlenecks are exactly the sort of bottleneck that might give automated innovators an enormous edge over human innovators even without large advantages in hardware or algorithms.
Specifically, rockets experience a wider range of temperatures and pressures, traverse those ranges more rapidly, and are also packed more fully with explosives.
Consider Bird and Layzell’s example of a very simple genetic algorithm that was tasked with evolving an oscillating circuit. Bird and Layzell were astonished to find that the algorithm made no use of the capacitor on the chip; instead, it had repurposed the circuit tracks on the motherboard as a radio to replay the oscillating signal from the test device back to the test device.This was not a very smart program. This is just using hill climbing on a very small solution space. In spite of this, the solution turned out to be outside the space of solutions the programmers were themselves visualizing. In a computer simulation, this algorithm might have behaved as intended, but the actual solution space in the real world was wider than that, allowing hardware-level interventions.In the case of an intelligent system that’s significantly smarter than humans on whatever axes you’re measuring, you should by default expect the system to push toward weird and creative solutions like these, and for the chosen solution to be difficult to anticipate.
https://futureoflife.org/wp-content/uploads/2017/04/Screen-Shot-2017-04-13-at-1.17.00-PM.png328581Nate Soareshttps://futureoflife.org/wp-content/uploads/2015/10/FLI_logo-1.pngNate Soares2017-04-18 08:20:532017-04-18 10:20:04Ensuring Smarter-than-human Intelligence Has a Positive Outcome
The headline result that Morning Consult is highlighting is that overwhelming majorities of respondents supported national regulation (71% support) and international regulation (67%) of AI. Thirty-seven percent strongly support national regulation, compared to just 4% who strongly oppose it (for international, those numbers were 35% and 5%, respectively). However, nearly half of respondents also indicated they had very limited knowledge of what AI actually is.
Perhaps even more strikingly, the proportion of respondents who support regulation was very consistent across political and socioeconomic lines. A full 74% of Republicans, 73% of Democrats, and 62% of independents support national regulations, as do 69% of people making less than $50k/yr, 73% making $50k-$100k, and 65% of those who make more than $100k. Education likewise matters little: 70% of people without a college degree support national regulation, along with 74% of college grads and 70% of respondents with post-graduate degrees. Women (75%) were slightly more likely to support such regulations than men (67%).
https://futureoflife.org/wp-content/uploads/2017/04/Poll-e1492182492921.png514400Matt Schererhttps://futureoflife.org/wp-content/uploads/2015/10/FLI_logo-1.pngMatt Scherer2017-04-13 15:06:352017-04-14 10:09:23Op-ed: Poll Shows Strong Support for AI Regulation Though Respondents Admit Limited Knowledge of AI
To imagine innocence is to picture children playing. As such, most people and governments are horrified by the idea of children and other helpless civilians suffering and dying, even during war. Finding a way to prevent the unnecessary slaughter of innocents has brought over 115 countries to the United Nations in New York this week to begin negotiations of a historic treaty that would, once and for all, ban nuclear weapons.
The countries are united by concerns that tens or even hundreds of thousands of innocent men, women and children – mothers, sons, fathers, daughters, aunts, uncles, cousins, friends and neighbors – could be killed, quite literally, in a flash.
In a statement to the opening of negotiations, Peter Maurer, president of the International Committee of the Red Cross, or ICRC, said, “The prohibition and elimination of nuclear weapons is a humanitarian imperative.”
Responding to a Humanitarian Imperative
A ban on nuclear weapons is certainly historic, but it’s not without precedence. Prohibiting and eliminating other weapons because of their horrific humanitarian consequences has happened before. In fact, most of the world’s deadliest weapons are currently banned.
At a press conference, Beatrice Fihn, executive director of the International Campaign to Abolish Nuclear Weapons, ICAN, said, “The treaty will finally ban weapons designed to indiscriminately kill civilians, completing the prohibitions on weapons of mass destruction.”
For example, when adults around the world learned of the tens of thousands of children killed by landmines while simply pursuing childhood activities, such as playing in open fields, a global cry arose to bring an end to the indiscriminate weapons. In 1997, 133 countries signed the Mine Ban Treaty, and as of today 162 have signed. According to the International Campaign to Ban Landmines, “only 35 states remain outside the treaty, but most of them do not actually use or produce antipersonnel mines.”
A similar rallying cry heralded the Convention on Cluster Munitions. Cluster munitions often landed without exploding and remained unstable. Their toy-like appearance attracted thousands of children, who were killed and maimed by the weapons. The treaty was adopted in 2008 and is described by clusterconvention.org as an “international treaty of more than 100 States that addresses the humanitarian consequences and unacceptable harm caused to civilians by cluster munitions.”
Today, most countries abide by these treaties, and even countries like the United States, which has not signed either treaty, is either mostly in compliance or is showing signs of improvement.
A Nuclear Rallying Cry
Not surprisingly, the horror of the effect nuclear bombs have on children provides some of the most compelling arguments for a ban treaty.
Fujimori Toshiki, a Hibakusha (survivor of the bombs dropped on Japan), described his personal experience to the General Assembly at the very start of the negotiations. He was a baby at the time, and he and his mother were just far enough away from the blast that a two-story home protected them somewhat.
“I had my entire body covered with bandages,” said Toshiki, “with only eyes, nose, and mouth uncovered. Everybody thought I would die over time. Yet, I survived. It is a miracle. I am here at the U.N., asking for an abolition of nuclear weapons. I am convinced that this is a mission I am given as a survivor of the atomic bomb.”
His 13-year-old sister was not so lucky. She was one of 6,300 teenagers to die near the blast site because their schools had sent them there to help “create firesafe [sic] areas against air raids.”
Toshiki added, “Every year, on Aug. 6, my mother would gather all of us children and would talk to us about her experience in tears. I once asked my mother why she would speak about it if recalling the experience makes her suffer. ‘I can’t make you go through the same experience.’ That was her answer. Her tears were her heartfelt appeal. She called, as a mother, for a world with no more hell on earth.”
Setsuko Thurlow, another Hibakusha, was also 13 when the bombs fell. She described witnessing the slow death of her 4-year-old nephew Eiji. He was “transformed into a charred, blackened and swollen child who kept asking in a faint voice for water until he died in agony.”
Thurlow continued, “Regardless of the passage of time, he remains in my memory as a 4-year-old child who came to represent all the innocent children of the world. And it is this death of innocents that has been the driving force for me to continue my struggle against the ultimate evil of nuclear weapons.”
However, unlike the stories of landmines and cluster munitions, which told of present-day children suffering and dying, these stories are over 70 years old. It can be difficult to relate to events that happened so long ago and that most people believe has not ― and cannot ― be repeated.
But Sue Coleman-Haseldine told the assembly of stories and concerns that were more recent. Coleman-Haseldine is an Aboriginal who lived near the atomic weapons testing sites in Australia. She was two when the testing first began in the 1950s.
“Our district is full of cancer now,” she said.
She continued, “I grew up hearing about the bombs, but I didn’t know about how the sickness went through the generations. When mining companies started eyeing off areas of my country I started to look more into it and I went to an Australian Nuclear Free Alliance meeting to learn more about fighting mining companies but also radiation fallout. What I learnt devastated me. To find out that our bush foods were possibly contaminated was a real blow to me.”
“I am a mother, grandmother and great grandmother,” she added. “My third great grandson was born just recently. And now I am here, speaking about the past [and] present day problems and what we want for the future. I’m fighting for all my grandchildren and all the children of the world.”
Relegating Nukes to History
A common concern about these negotiations is the notable absence of the nuclear states. However, history, as seen with the landmine and cluster munitions treaties, gives those supporting the negotiations reason to hope.
In his statement for the ICRC, Maurer added, “Of course, adopting a treaty to prohibit nuclear weapons will not make them immediately disappear. But it will reinforce the stigma against their use, support commitments to nuclear risk reduction, and be a disincentive for proliferation. … As with chemical and biological weapons, a clear and unambiguous prohibition is the cornerstone of their elimination.”
Susi Snyder, the nuclear disarmament program manager for PAX in the Netherlands, explained, “This is the start of a negotiation. The impact of the negotiation cannot be guessed or measured until the treaty is done. Even then, as with all treaties and growing norms, the impact will grow over time.”
Fihn added that a treaty would “make it clear that the world has moved beyond these morally unacceptable weapons of the past.”
https://futureoflife.org/wp-content/uploads/2017/03/IMG_3501-2.jpg8531280Ariel Connhttps://futureoflife.org/wp-content/uploads/2015/10/FLI_logo-1.pngAriel Conn2017-03-30 19:59:452017-03-30 19:59:45Survivors Speak Out As UN Negotiates Nuke Ban
The following statement was provided by Sue Coleman-Haseldine, a survivor of nuclear bomb tests near her home in Australia. She also present testimony at the UN negotiations on Tuesday, March 28.
My name is Sue Coleman-Haseldine I am Kokatha elder. I was born on Koonibba Aboriginal Mission in 1951 which is about 40 km west of Ceduna in South Australia. I now live just out of Ceduna with my husband. For those that don’t know we are on the edge of the Nullabor where the desert meets the sea. Our country takes in one of the last stunted mallee regions that is still in pristine condition. We still carry on looking after our country as our people did even though we don’t live out there now. I remember the good life of hunting for wild game and collecting bush fruits. Life was healthy. We still do all this today. I teach the young ones coming up about the land and all the life it gives.
I am a mother, grandmother and great grandmother. My third great grandson was born just recently. And now I am here, speaking about the past, present day problems and what we want for the future. I’m fighting for all my grandchildren and all the children of the world. To keep the dream alive of a clean, safe future where there is no nuclear fear hanging over our heads. And like I tell the children, I’m fighting for the animals too. We are all connected. A world without animals wouldn’t be a world at all.
I was two years old when the first Atomic Bomb tests began in the desert areas north west of my mallee country in 1953.
A full scale atomic bomb went off on October 15 at Emu Fields. This first one was Totem 1 and it caused a death cloud known by many as the ‘Black mist’. It killed people, blinded others and made people very sick. Its effects are still being felt today. I wasn’t on ground zero but the black mist went all over. And who knows where the radiation went from the many the tests that followed. I remember older people talking about Nullabor dust storms. It was the fallout from the Maralinga tests. The dust didn’t stay in one place.
Our district is full of cancer now. My 86 year old Aunty once told me “that minga – that cancer sickness was never here before those bombs”.
Cancer is the big one but it is also common for people to suffer from thyroid conditions or stomach and bowel problems. This is the case for myself and some of my grandchildren. Fertility problems, still births, birth defects became more common at the time of the testing. Woomera cemetery is full of babies who started dying around this time. We still wonder and worry that women have trouble because of the ongoing radiation in the area or genetic changes passed down through generations.
Like all people, the giving of life and raising children is so important to us and it’s our human rights to be able to continue raising our family and sharing our culture forever. There are lots of Aboriginal groups in Australia. We are all different. But for all of us our land is the basis of our culture – it’s our church, our grocery shop, our schools, our chemist. But living a life and practising culture out in the desert wasn’t recognised as worthy by governments back then or still today. In fact we still have to work hard to have all the life, all the plants, all the animals, the underground water out in the desert recognised and protected.
This is one reason why Emu Fields and then Maralinga were picked for testing. The English and Australian governments didn’t think that land was valuable – they called it a wasteland. But Aboriginal people were still looking after and living their culture on the land that supported them. Aboriginal people were still present in the testing area when the bombs went off. The government was no good at ensuring everyone was safe. They had one patrol officer and some signs in English that people couldn’t read.
Australia was even more racist then. People have to remember this was before Aboriginal people had the right to vote. I believe the government really didn’t care what happened to Aboriginal people or their land. The bomb tests continued for many years right until 1967. Big atomic tests that the British and Australian governments were proud of and then a whole lot of secret tests that the British did with plutonium. These tests contaminated a huge area and everything in it but people 100 km away were also impacted. This includes my family and the broader community where I live.
It is good more people are learning about the bombs in Australia. And I want more people to think about ongoing impacts. Especially in my region because it doesn’t matter if you’re black, white or mixed everyone has a sad story about premature sickness and death in their families.
I grew up hearing about the bombs but I didn’t know about how the sickness went through the generations. When mining companies started eyeing off areas of my country I started to look more into it and I went to an Australian Nuclear Free Alliance meeting to learn more about fighting mining companies but also radiation fallout. What I learnt devastated me. To find out that our bush foods were possibly contaminated was a real blow to me.
It was at these meetings I also learnt about other nuclear bombs. About other places where tests happened and also more about Japan during the war. I also learnt that uranium mined in Australia was used in these weapons of destruction.
To know that uranium from our country was devastating other countries and people broke my heart. I decided to fight any kind of mining then. There are too many illnesses and cancer deaths in our Country. What’s changed to cause this? I believe it is caused from radiation contamination, but I can’t prove it. I think any kind of mining in our area would be digging up contaminated earth and sending it back to us on the north-north-west winds.
The bomb tests destroyed a beautiful part of Australia and despite several attempts it will never be safe or clean. There are many Aboriginal people who cannot go back to their ancestral lands and their children and their children’s children and so on will never know the special religious places it contains.
Having whole displaced communities has also created confusion and conflict for other Aboriginal groups. These are ongoing issues which cause stress and heartbreak.
We have been poisoned and we don’t need the threat of being poisoned again by a nuclear waste dump — whether it’s Australia’s waste or waste from around the world. We don’t need this stress hanging over our heads. It’s not our right to condemn our children to the risk of leakages or damage or terrorist attacks for ever. This is condemning them to a life of fear. It’s about time people see the desert and arid regions as places full of life instead of wastelands for dangerous activities. Aboriginal people have worked really hard to have their culture and their land understood. We don’t need governments telling us we don’t understand or are too emotional about these things. We do understand the risks and we don’t want them.
But more than that people all over the world don’t want all these problems. The uranium should stay in the ground. We need to stop making waste. And it’s not just the physical impacts of the nuclear industry I worry about. To have a nuclear waste dump back on the cards has already caused a lot of anxiety in our region. Aboriginal people in particular have a lot of issues to deal with. There is still a lot of poverty issues with education and job opportunities, self-medicating through drugs and alcohol. We are still being made refugees in our own country because if the government of the mining companies want something they take it which also causes a lot of anxiety and mental problems. I worry there is no security for the future. It’s a different kind of black mist hanging over us now. The original blast brought a black mist and nobody knew what it was. Now people are a lot wiser but we still can’t see through to the future. There is no guarantee that any nuclear facility – whether a mine a reactor or a waste dump will be properly looked after.
It’s been good to talk to you today and I appreciate your attention. And I want you to know I don’t want people today to shoulder the blame of the past and to feel responsible for all the wrongs that have happened. I am talking to you now because I think we need to join forces and make a better future for future generations, all over the world.
I want to thank Bill Williams for everything he taught me over the years; lessons never to be forgotten. The fight goes on.
Lastly I just want to thank the organisers. I am standing out front here today but lots of people are behind the scenes. And I thank them with all my heart.
Remember: the future forever belongs to the next generation.
https://futureoflife.org/wp-content/uploads/2017/03/Sue-Coleman-Haseldine.jpg8531280a guest bloggerhttps://futureoflife.org/wp-content/uploads/2015/10/FLI_logo-1.pnga guest blogger2017-03-29 11:43:242017-03-29 11:51:04Testimony by Sue Coleman-Haseldine, Nuclear Bomb Testing Survivor
The following post is the statement presented by Hiroshima survivor Fujimori Toshiki to the United Nations on the first day of negotiations for a treaty to ban nuclear weapons.
United Nations conference to negotiate a legally binding instrument to prohibit nuclear weapons, leading towards their total elimination
New York, 27 March 2017
Testimony by Fujimori Toshiki, Assistant Secretary General of Nihon Hidankyo (Japan Confederation of Atomic and Hydrogen Bomb Sufferers Organizations)
President Ambassador Whyte, Distinguished Delegates,
I would like to thank the President for giving me the opportunity to speak.
My name is Fujimori Toshiki, and I am the Assistant Secretary General of Nihon Hidankyo, Japan Confederation of Atomic and Hydrogen Bomb Sufferers Organizations. I am one of the people who was exposed to the atomic bomb the U.S. forces dropped on Hiroshima on August 6, 1945.
Hibakusha established Nihon Hidankyo eleven years after the bombing. Since then, we have called for “No More Hibakusha” within Japan and abroad. The treaty you will be negotiating today must reflect this call of Hibakusha in express terms so that the world makes remarkable progress towards nuclear weapons abolition.
I was 1 year and 4 months-old when the bomb was dropped. We were a big family of twelve, consisting of my grandfather, father, mother, six elder sisters, two elder brothers, and myself. Two of my elder sisters and my two elder brothers had evacuated out of the city of Hiroshima to avoid air raids. The eight of us who stayed in Hiroshima were exposed to the bomb.
My fourth-eldest sister was 13 years old and was in her first year of an all-girls junior high school. She was around 400m from the hypocenter when the bomb was dropped. Together with her teachers and other students, my sister was there to demolish houses to create firesafe areas against air raids. All of 676 of them including my sister were killed instantly through direct exposure to radiation, the heat, and the blast from the bomb. It is said that all together in the city of Hiroshima, 8400 students in the first and second year of junior high schools were being mobilized for similar purposes that day. The lives of 6300 of them were lost.
I was sick that day, so my mother was heading to the hospital with me on her back when the bomb was dropped. We were 2.3 km from the hypocentre. Fortunately, a two-story house between the hypocentre and us prevented us from directly being exposed to the heat. Yet, we were thrown all the way to the edge of the river bank. My mother, with me in her arms, managed to get to the nearby mountain called Ushitayama. Our family members were in different locations at the time of the bombing, but everyone escaped to the same mountain of Ushitayama, except for my fourth-elder sister. For many days that followed, my parents and my sisters kept going back to the area near the hypocentre to look for my fourth-eldest sister, who was missing. We never found her. We never found her body either. In the meantime, I had my entire body covered with bandages, with only eyes, nose, and mouth uncovered. Everybody thought I would die over time. Yet, I survived. It is a miracle. I am here at the UN, asking for an abolition of nuclear weapons. I am convinced that this is a mission I am given as a survivor of the atomic-bomb.
Two hundred and ten thousand people died by the end of 1945 due to the atomic bombs the U.S. forces dropped on Hiroshima and Nagasaki. Hibakusha experienced hell on earth beneath the mushroom clouds. In fact, Hibakusha have continued to suffer for the twenty six thousand one hundred sixty six days until today, March 27th , 2017.
Nobody, in any country, deserves seeing the same hell on earth again.
Every year, on August 6th, my mother would gather all of us children and would talk to us about her experience in tears. I once asked my mother why she would speak about it if recalling the experience makes her suffer.
“I can’t make you go through the same experience.” That was her answer.
Her tears were her heartfelt appeal. She called, as a mother, for a world with no more hell on earth.
Three International Conferences on the Humanitarian Impact of Nuclear Weapons hosted by Norway, Mexico, and Austria, as well as the Joint Statements on the Humanitarian Consequences of Nuclear Weapons at the Preparatory Committee meetings for the NPT Review Conference and the First Committee of the UN General Assembly, repeatedly and strongly reaffirmed the following conclusions:
That whether intended or not intended, the effects of a nuclear weapon detonation are not constrained by national borders;
That no state or international body could address its immediate humanitarian emergency or long-term consequences;
That in the interest of the very survival of humanity nuclear weapons must never be used again; and
That the only assurance against the risk of a nuclear weapon detonation is the total elimination of nuclear weapons.
Many Hibakusha received these messages with thousands of thoughts about the long journey that they had come.
Nuclear weapon states and their allied nuclear-dependent states are against concluding a treaty to abolish nuclear weapons. Despite being the only country in the world that experienced the wartime use of nuclear weapons, the Japanese government voted against the UN resolution 71/258, which established this negotiating conference.
As a Hibakusha, and as a Japanese, I am here today heartbroken.
Yet, I am not discouraged.
Government representatives who are present at this conference, international organizations, and representatives of civil society organizations are making efforts to conclude a legally binding instrument to ban and eliminate nuclear weapons.
In April of last year, Hibakusha initiated the International Signature Campaign, which calls on all state governments to conclude a treaty to ban and eliminate nuclear weapons. We reached out to countries across the world, and last October, we delivered our first batch of five hundred sixty thousand signatures to the Chair of the First Committee of the UN General Assembly. Today, we have over one million seven hundred twenty thousand signatures. We will continue the campaign until 2020, and we aim to collect hundreds of millions of signatures.
Let us work together to achieve the nuclear ban treaty.
Risks Principle:Risks posed by AI systems, especially catastrophic or existential risks, must be subject to planning and mitigation efforts commensurate with their expected impact.
We don’t know what the future of artificial intelligence will look like. Though some may make educated guesses, the future is unclear.
AI could keep developing like all other technologies, helping us transition from one era into a new one. Many, if not all, AI researchers hope it could help us transform into a healthier, more intelligent, peaceful society. But it’s important to remember that AI is a tool and, as such, not inherently good or bad. As with any other technology or tool, there could be unintended consequences. Rarely do people actively attempt to crash their cars or smash their thumbs with hammers, yet both happen all the time.
A concern is that as technology becomes more advanced, it can affect more people. A poorly swung hammer is likely to only hurt the person holding the nail. A car accident can harm passengers and drivers in both cars, as well as pedestrians. A plane crash can kill hundreds of people. Now, automation threatens millions of jobs — and while presumably no lives will be lost as a direct result, mass unemployment can have devastating consequences.
And job automation is only the beginning. When AI becomes very general and very powerful, aligning it with human interests will be challenging. If we fail, AI could plausibly become an existential risk for humanity.
Given the expectation that advanced AI will far surpass any technology seen to date — and possibly surpass even human intelligence — how can we predict and prepare for the risks to humanity?
To consider the Risks Principle, I turned to six AI researchers and philosophers.
Non-zero Probability
An important aspect of considering the risk of advanced AI is recognizing that the risk exists, and it should be taken into account.
As Roman Yampolskiy, an associate professor at the University of Louisville, explained, “Even a small probability of existential risk becomes very impactful once multiplied by all the people it will affect. Nothing could be more important than avoiding the extermination of humanity.”
This is “a very reasonable principle,” said Bart Selman, a professor at Cornell University. He explained, “I sort of refer to some of the discussions between AI scientists who might differ in how big they think that risk is. I’m quite certain it’s not zero, and the impact could be very high. So … even if these things are still far off and we’re not clear if we’ll ever reach them, even with a small probability of a very high consequence we should be serious about these issues. And again, not everybody, but the subcommunity should.”
Anca Dragan, an assistant professor at UC Berkeley was more specific about her concerns. “An immediate risk is agents producing unwanted, surprising behavior,” she explained. “Even if we plan to use AI for good, things can go wrong, precisely because we are bad at specifying objectives and constraints for AI agents. Their solutions are often not what we had in mind.”
Considering Other Risks
While most people I spoke with interpreted this Principle to address longer-term risks of AI, Dan Weld, a professor at the University of Washington, took a more nuanced approach.
“How could I disagree?” He asked. “Should we ignore the risks of any technology and not take precautions? Of course not. So I’m happy to endorse this one. But it did make me uneasy, because there is again an implicit premise that AI systems have a significant probability of posing an existential risk.”
But then he added, “I think what’s going to happen is – long before we get superhuman AGI – we’re going to get superhuman artificial *specific* intelligence. … These narrower kinds of intelligence are going to be at the superhuman level long before a *general* intelligence is developed, and there are many challenges that accompany these more narrowly described intelligences.”
“One technology,” he continued, “that I wish [was] discussed more is explainable machine learning. Since machine learning is at the core of pretty much every AI success story, it’s really important for us to be able to understand *what* it is that the machine learned. And, of course, with deep neural networks it is notoriously difficult to understand what they learned. I think it’s really important for us to develop techniques so machines can explain what they learned so humans can validate that understanding. … Of course, we’ll need explanations before we can trust an AGI, but we’ll need it long before we achieve general intelligence, as we deploy much more limited intelligent systems. For example, if a medical expert system recommends a treatment, we want to be able to ask, ‘Why?’
“Narrow AI systems, foolishly deployed, could be catastrophic. I think the immediate risk is less a function of the intelligence of the system than it is about the system’s autonomy, specifically the power of its effectors and the type of constraints on its behavior. Knight Capital’s automated trading system is much less intelligent than Google Deepmind’s AlphaGo, but the former lost $440 million in just forty-five minutes. AlphaGo hasn’t and can’t hurt anyone. … And don’t get me wrong – I think it’s important to have some people thinking about problems surrounding AGI; I applaud supporting that research. But I do worry that it distracts us from some other situations which seem like they’re going to hit us much sooner and potentially cause calamitous harm.”
Open to Interpretation
Still others I interviewed worried about how the Principle might be interpreted, and suggested reconsidering word choices, or rewriting the principle altogether.
Patrick Lin, an Associate Professor at California Polytechnic State University, believed that the Principle is too ambiguous.
He explained, “This sounds great in ‘principle,’ but you need to work it out. For instance, it could be that there’s this catastrophic risk that’s going to affect everyone in the world. It could be AI or an asteroid or something, but it’s a risk that will affect everyone. But the probabilities are tiny — 0.000001 percent, let’s say. Now if you do an expected utility calculation, these large numbers are going to break the formula every time. There could be some AI risk that’s truly catastrophic, but so remote that if you do an expected utility calculation, you might be misled by the numbers.”
“I agree with it in general,” Lin continued, “but part of my issue with this particular phrasing is the word ‘commensurate.’ Commensurate meaning an appropriate level that correlates to its severity. So I think how we define commensurate is going to be important. Are we looking at the probabilities? Are we looking at the level of damage? Or are we looking at expected utility? The different ways you look at risk might point you to different conclusions. I’d be worried about that. We can imagine all sorts of catastrophic risks from AI or robotics or genetic engineering, but if the odds are really tiny, and you still want to stick with this expected utility framework, these large numbers might break the math. It’s not always clear what the right way is to think about risk and a proper response to it.”
Meanwhile Nate Soares, the Executive Director of the Machine Intelligence Research Institute, suggested that the Principle should be more specific.
Soares said, “The principle seems too vague. … Maybe my biggest concern with it is that it leaves out questions of tractability: the attention we devote to risks shouldn’t actually be proportional to the risks’ expected impact; it should be proportional to the expected usefulness of the attention. There are cases where we should devote more attention to smaller risks than to larger ones, because the larger risk isn’t really something we can make much progress on. (There are also two separate and additional claims, namely ‘also we should avoid taking actions with appreciable existential risks whenever possible’ and ‘many methods (including the default methods) for designing AI systems that are superhumanly capable in the domains of cross-domain learning, reasoning, and planning pose appreciable existential risks.’ Neither of these is explicitly stated in the principle.)
“If I were to propose a version of the principle that has more teeth, as opposed to something that quickly mentions ‘existential risk’ but doesn’t give that notion content or provide a context for interpreting it, I might say something like: ‘The development of machines with par-human or greater abilities to learn and plan across many varied real-world domains, if mishandled, poses enormous global accident risks. The task of developing this technology therefore calls for extraordinary care. We should do what we can to ensure that relations between segments of the AI research community are strong, collaborative, and high-trust, so that researchers do not feel pressured to rush or cut corners on safety and security efforts.’”
What Do You Think?
How can we prepare for the potential risks that AI might pose? How can we address longer-term risks without sacrificing research for shorter-term risks? Human history is rife with learning from mistakes, but in the case of the catastrophic and existential risks that AI could present, we can’t allow for error – but how can we plan for problems we don’t know how to anticipate? AI safety research is critical to identifying unknown unknowns, but is there more the the AI community or the rest of society can do to help mitigate potential risks?
The Principles offer a framework to help artificial intelligence benefit as many people as possible. But, as AI expert Toby Walsh said of the Principles, “Of course, it’s just a start. … a work in progress.” The Principles represent the beginning of a conversation, and now we need to follow up with broad discussion about each individual principle. You can read the weekly discussions about previous principles here.
https://futureoflife.org/wp-content/uploads/2017/03/Risks-Principle.jpg9601440Ariel Connhttps://futureoflife.org/wp-content/uploads/2015/10/FLI_logo-1.pngAriel Conn2017-03-23 16:33:122017-03-23 16:33:12Can We Properly Prepare for the Risks of Superintelligent AI?
Last week, China proposed a way to reduce tensions on the Korean peninsula: Pyongyang would freeze its missile and nuclear programs in exchange for Washington and Seoul halting their current round of military exercises. China also sees this as a way of starting talks between the United States and North Korea, which it believes is necessary to resolve hostilities on the peninsula.
In comments in South Korea on Friday, Secretary of State Rex Tillerson said “We do not believe the conditions are right to engage in any talks at this time. … [C]onditions must change before there is any scope for talks to resume, whether they be five-party or six-party.” Whether the last phrase means the United States is consciously rejecting the idea of one-on-one talks with the North is not clear.
Tillerson also said he believes it is “premature” to talk about a freeze since a freeze would leave North Korea with significant military capabilities. His choice of words appears to leave the possibility of a freeze on the table.
A North Korean freeze on nuclear tests and missile flight tests would be highly beneficial—and readily verifiable. It would prevent Pyongyang from developing a long-range missile capable of hitting the United States.
It is important, of course, to work to eliminate the capabilities North Korea currently has, but a sensible first step is to keep it from increasing its capabilities.
Freezing North Korean Development Would Be Valuable
Over the last two decades, Pyongyang has developed technologies beyond what many thought it was capable of doing. It has developed nuclear weapons, an array of short- and medium-range missiles that could pose a significant threat to its neighbors, and a large satellite launcher that has placed two objects in orbit (although neither appears to have been operational once in orbit).
These weapon systems are not particularly sophisticated and in some cases are highly unreliable but, as Secretary Tillerson noted, they represent significant military capabilities.
And if these development programs continue, things will get considerably worse. The past shows North Korea’s ability to make slow but steady progress, and that will continue. After all, it is developing weapons systems that the United States, Soviet Union, and others developed 50 years ago.
What could a freeze do?
A Freeze on Nuclear Tests
North Korea has now conducted five nuclear tests, the last of which indicated it has developed a nuclear device that can produce a yield similar to that of the Hiroshima and Nagasaki bombs.
Getting a nuclear device to explode under test conditions, however, is not the same as having a usable nuclear warhead. A deliverable warhead needs to be small enough, both in terms of physical size and mass, to be carried by North Korea’s ballistic missiles. It needs to work in a realistic environment, rather than just in a static test environment. In particular, it must withstand the considerable forces of missile delivery, including acceleration, vibrations, and buffeting during launch and reentry.
If the North does not yet have a weapon that is small and rugged enough for delivery by ballistic missile, stopping additional nuclear tests could keep it from developing one. Even if it does have such a weapon, stopping testing would limit its ability to improve its design.
A testing freeze could be verified by the global network of seismic and other types of sensors that make up the International Monitoring System. This network is sensitive enough to have detected North Korea’s previous tests—including the 2006 test with an estimated yield of only about one kiloton.
A Freeze on Missile Tests
A freeze on missile flight testing would limit North Korea’s ability to build more capable and longer range missiles, and to determine the reliability of existing missiles or gain operational practice in firing them.
Today, North Korea’s longest range operational missile is the Nodong, with a range of about 1,300 km (800 miles)—much shorter than the roughly 9,000 km (5,600 miles) to the US West Coast. It does not currently have a ballistic missile that could carry a nuclear warhead to long distances.
However, it is developing a number of the components it needs to produce such a missile.
For example:
North Korea appears to have had one successful flight test, with five or more failures, of its Musudan (Hwasong-10) missile, which uses a more advanced fuel than the North’s previous missiles. Developing a working missile would require additional tests. Besides giving Pyongyang a considerably longer range missile than it currently has, successfully developing this missile would open the way to modifying it for use as an upper stage of longer range missiles.
In the last year, Pyongyang has conducted ground tests of several new engines (April 2016, September 2016, and March 2017). Some of these appear to use advanced fuels and have higher thrust than its current missile engines.
The North has paraded a prototype of a two-stage missile on a mobile launcher that could have a long range if it used the technology being tested in the Musudan missile. One of the engine ground tests mentioned above may be of an engine intended for the first stage of this missile.
Pyongyang is in early stages of testing a solid-fueled missile that could be launched from a mobile launcher on the ground or from a submarine. While the range appears to be similar to the Nodong, it would have a much shorter launch-preparation time if fired from mobile launchers on the ground, and could reach more distant targets if fired from a submarine at sea.
North Korea has not yet flight tested a reentry heat shield for a long-range missile, which is critical for successfully delivering a nuclear warhead. Over-designing the heat shield adds weight to the reentry vehicle, which reduces the range of the missile; Under-designing it can cause overheating of the warhead that can damage it. Moreover, North Korea is likely to design a blunt reentry vehicle that reenters slowly to reduce the intensity of heating, and this can lead to very large inaccuracies—tens of kilometers or more.
Transforming these pieces into working missiles would require a series of flight tests. A freeze on missile testing would keep it from developing that capability.
A flight test ban would be completely verifiable. The United States has a satellite network of infrared sensors that can detect launches essentially anywhere in the world. This system, for example, even detected the short-range Scud missiles launched at Israel during the first Gulf War. And the United States is currently deploying a new generation of even more capable satellites for this job.
Would North Korea Be Willing to Freeze?
No one knows.
It is worth remembering that North Korea observed a flight test moratorium from September 1999 through July 2006, which it began when its talks with the Clinton administration about missile and nuclear issues seemed to be moving ahead. It announced it was no longer bound by the moratorium in March 2005, in response to the Bush administration’s lack of diplomatic engagement on missile issues, and resumed testing the next year.
Things are different now, of course, and it’s not at all clear that Pyongyang would agree to a freeze. Kim Jong Un may be unwilling to stop until he has developed a credible long-range threat.
However, Kim clearly sees the US-South Korean military exercises as threatening, and offering to scale back those exercises may give the United States significant leverage. Agreeing to talks once a freeze is in place could add leverage.
But whether or not North Korea would ultimately agree to a freeze, the United States should not be the one to take this option off the table. A freeze would be an effective, meaningful step in limiting further development of North Korea’s nuclear and missile programs—and the administration should be doing what it can to put a freeze in place.
https://futureoflife.org/wp-content/uploads/2017/03/nuclear-weapons-david-wright.jpg9601440David Wrighthttps://futureoflife.org/wp-content/uploads/2015/10/FLI_logo-1.pngDavid Wright2017-03-22 17:08:472017-03-22 18:44:32Why Freezing North Korea’s Weapons Programs Would Make Us Safer
The following article was written by Vincent Conitzer and originally posted in Prospect Magazine.
Progress in artificial intelligence has been rapid in recent years. Computer programs are dethroning humans in games ranging from Jeopardy to Go to poker. Self-driving cars are appearing on roads. AI is starting to outperform humans in image and speech recognition.
With all this progress, a host of concerns about AI’s impact on human societies have come to the forefront. How should we design and regulate self-driving cars and similar technologies? Will AI leave large segments of the population unemployed? Will AI have unintended sociological consequences? (Think about algorithms that accurately predict which news articles a person will like resulting in highly polarised societies, or algorithms that predict whether someone will default on a loan or commit another crime becoming racially biased due to the input data they are given.)
Will AI be abused by oppressive governments to sniff out and stifle any budding dissent? Should we develop weapons that can act autonomously? And should we perhaps even be concerned that AI will eventually become “superintelligent”—intellectually more capable than human beings in every important way—making us obsolete or even extinct? While this last concern was once purely in the realm of science fiction, notable figures including Elon Musk, Bill Gates, and Stephen Hawking, inspired by Oxford philosopher Nick Bostrom’s Superintelligence book, have recently argued it needs to be taken seriously.
These concerns are mostly quite distinct from each other, but they all rely on the premise of technical advances in AI. Actually, in all cases but the last one, even just currently demonstrated AI capabilities justify the concern to some extent, but further progress will rapidly exacerbate it. And further progress seems inevitable, both because there do not seem to be any fundamental obstacles to it and because large amounts of resources are being poured into AI research and development. The concerns feed off each other and a community of people studying the risks of AI is starting to take shape. This includes traditional AI researchers—primarily computer scientists—as well as people from other disciplines: economists studying AI-driven unemployment, legal scholars debating how best to regulate self-driving cars, and so on.
A conference on “Beneficial AI” held in California in January brought a sizeable part of this community together. The topics covered reflected the diversity of concerns and interests. One moment, the discussion centered on which communities are disproportionately affected by their jobs being automated; the next moment, the topic was whether we should make sure that super-intelligent AI has conscious experiences. The mixing together of such short- and long-term concerns does not sit well with everyone. Most traditional AI researchers are reluctant to speculate about whether and when we will attain truly human-level AI: current techniques still seem a long way off this and it is not clear what new insights would be able to close the gap. Most of them would also rather focus on making concrete technical progress than get mired down in philosophical debates about the nature of consciousness. At the same time, most of these researchers are willing to take seriously the other concerns, which have a concrete basis in current capabilities.
Is there a risk that speculation about super-intelligence, often sounding like science fiction more than science, will discredit the larger project of focusing on the societally responsible development of real AI? And if so, is it perhaps better to put aside any discussion of super-intelligence for now? While I am quite sceptical of the idea that truly human-level AI will be developed anytime soon, overall I think that the people worried about this deserve a place at the table in these discussions. For one, some of the most surprisingly impressive recent technical accomplishments have come from people who are very bullish on what AI can achieve. Even if it turns out that we are still nowhere close to human-level AI, those who imagine that we are could contribute useful insights into what might happen in the medium-term.
I think there is value even in thinking about some of the very hard philosophical questions, such as whether AI could ever have subjective experiences, whether there is something it would be like to be a highly advanced AI system. (See also my earlier Prospect article.) Besides casting an interesting new light on some ancient questions, the exercise is likely to inform future societal debates. For example, we may imagine that in the future people will become attached to the highly personalised and anthropomorphised robots that care for them in old age, and demand certain rights for these robots after they pass away. Should such rights be granted? Should such sentiments be avoided?
At the same time, the debate should obviously not exclude or turn off people who genuinely care about the short-term concerns while being averse to speculation about the long-term, especially because most real AI researchers fall in this last category. Besides contributing solutions to the short-term concerns, their participation is essential to ensure that the longer-term debate stays grounded in reality. Research communities work best when they include people with different views and different sub-interests. And it is hard to imagine a topic for which this is truer than the impact of AI on human societies.
Note from FLI: Among our objectives is to inspire discussion and a sharing of ideas. As such, we post op-eds that we believe will help spur discussion within our community. Op-eds do not necessarily represent FLI’s opinions or views.
https://futureoflife.org/wp-content/uploads/2017/03/AI-debate.jpg7941191a guest bloggerhttps://futureoflife.org/wp-content/uploads/2015/10/FLI_logo-1.pnga guest blogger2017-03-21 17:01:352017-03-21 18:41:50The AI Debate Must Stay Grounded in Reality
Shared Prosperity Principle:The economic prosperity created by AI should be shared broadly, to benefit all of humanity.
Income inequality is a well recognized problem. The gap between the rich and poor has grown over the last few decades, but it became increasingly pronounced after the 2008 financial crisis. While economists debate the extent to which technology plays a role in global inequality, most agree that tech advances have exacerbated the problem.
In an interview with the MIT Tech Review, economist Erik Brynjolfsson said, “My reading of the data is that technology is the main driver of the recent increases in inequality. It’s the biggest factor.”
Which begs the question: what happens as automation and AI technologies become more advanced and capable?
Artificial intelligence can generate great value by providing services and creating products more efficiently than ever before. But many fear this will lead to an even greater disparity between the wealthy and the rest of the world.
AI expert Yoshua Bengio suggests that equality and ensuring a shared benefit from AI could be pivotal in the development of safe artificial intelligence. Bengio, a professor at the University of Montreal, explains, “In a society where there’s a lot of violence, a lot of inequality, [then] the risk of misusing AI or having people use it irresponsibly in general is much greater. Making AI beneficial for all is very central to the safety question.”
In fact, when speaking with many AI experts across academia and industry, the consensus was unanimous: the development of AI cannot benefit only the few.
Broad Agreement
“It’s almost a moral principle that we should share benefits among more people in society,” argued Bart Selman, a professor at Cornell University. “I think it’s now down to eight people who have as much as half of humanity. These are incredible numbers, and of course if you look at that list it’s often technology pioneers that own that half. So we have to go into a mode where we are first educating the people about what’s causing this inequality and acknowledging that technology is part of that cost, and then society has to decide how to proceed.”
Guruduth Banavar, Vice President of IBM Research, agreed with the Shared Prosperity Principle, but said, “It needs rephrasing. This is broader than AI work. Any AI prosperity should be available for the broad population. Everyone should benefit and everyone should find their lives changed for the better. This should apply to all technology – nanotechnology, biotech – it should all help to make life better. But I’d write it as ‘prosperity created by AI should be available as an opportunity to the broadest population.’”
Francesca Rossi, a research scientist at IBM, added, “I think [this principle is] very important. And it also ties in with the general effort and commitment by IBM to work a lot on education and re-skilling people to be able to engage with the new technologies in the best way. In that way people will be more able to take advantage of all the potential benefits of AI technology. That also ties in with the impact of AI on the job market and all the other things that are being discussed. And they are very dear to IBM as well, in really helping people to benefit the most out of the AI technology and all the applications.”
Meanwhile, Stanford’s Stefano Ermon believes that research could help ensure greater equality. “It’s very important that we make sure that AI is really for everybody’s benefit,” he explained, “that it’s not just going to be benefitting a small fraction of the world’s population, or just a few large corporations. And I think there is a lot that can be done by AI researchers just by working on very concrete research problems where AI can have a huge impact. I’d really like to see more of that research work done.”
A Big Challenge
“AI is having incredible successes and becoming widely deployed. But this success also leads to a big challenge,” said Dan Weld, a professor at the University of Washington. “[That is] its impending potential to increase productivity to the point where many people may lose their jobs. As a result, AI is likely to dramatically increase income disparity, perhaps more so than other technologies that have come about recently. If a significant percentage of the populace loses employment, that’s going to create severe problems, right? We need to be thinking about ways to cope with these issues, very seriously and soon.”
Berkeley professor, Anca Dragan, summed up the problem when she asked, “If all the resources are automated, then who actually controls the automation? Is it everyone or is it a few select people?”
“I’m really concerned about AI worsening the effects and concentration of power and wealth that we’ve seen in the last 30 years,” Bengio added.
“It’s a real fundamental problem facing our society today, which is the increasing inequality and the fact that prosperity is not being shared around,” explained Toby Walsh, a professor at UNSW Australia.
“This is fracturing our societies and we see this in many places, in Brexit, in Trump,” Walsh continued. “A lot of dissatisfaction within our societies. So it’s something that we really have to fundamentally address. But again, this doesn’t seem to me something that’s really particular to AI. I think really you could say this about most technologies. … although AI is going to amplify some of these increasing inequalities. If it takes away people’s jobs and only leaves wealth in the hands of those people owning the robots, then that’s going to exacerbate some trends that are already happening.”
Kay Firth-Butterfield, the Executive Director of AI-Austin.org, also worries that AI could exacerbate an already tricky situation. “AI is a technology with such great capacity to benefit all of humanity,” she said, “but also the chance of simply exacerbating the divides between the developed and developing world, and the haves and have nots in our society. To my mind that is unacceptable and so we need to ensure, as Elon Musk said, that AI is truly democratic and its benefits are available to all.”
“Given that all the jobs (physical and mental) will be gone, [shared prosperity] is the only chance we have to be provided for,” added University of Louisville professor, Roman Yampolskiy.
What Do You Think?
Given current tech trends, is it reasonable to assume that AI will exacerbate today’s inequality issues? Will this lead to increased AI safety risks? How can we change the societal mindset that currently discourages a greater sharing of wealth? Or is that even a change we should consider?
The Principles offer a framework to help artificial intelligence benefit as many people as possible. But, as AI expert Toby Walsh said of the Principles, “Of course, it’s just a start. … a work in progress.” The Principles represent the beginning of a conversation, and now we need to follow up with broad discussion about each individual principle. You can read the weekly discussions about previous principles here.
https://futureoflife.org/wp-content/uploads/2017/03/shared-prosperity-robotic-arm.jpg9351400Ariel Connhttps://futureoflife.org/wp-content/uploads/2015/10/FLI_logo-1.pngAriel Conn2017-03-16 18:37:192017-03-16 18:37:33Artificial Intelligence and Income Inequality
Even though human drivers don’t consciously think in terms of probabilities, we observe our environment and make decisions based on the likelihood of certain things happening. A driver doesn’t calculate the probability that the sports car behind her will pass her, but through observing the car’s behavior and considering similar situations in the past, she makes her best guess.
We trust probabilities because it is the only way to take action in the midst of uncertainty.
Autonomous systems such as self-driving cars will make similar decisions based on probabilities, but through a different process. Unlike a human who trusts intuition and experience, these autonomous cars calculate the probability of certain scenarios using data collectors and reasoning algorithms.
How to Determine Probability
Stefano Ermon, a computer scientist at Stanford University, wants to make self-driving cars and autonomous systems safer and more reliable by improving the way they reason probabilistically about their environment. He explains, “The challenge is that you have to take actions and you don’t know what will happen next. Probabilistic reasoning is just the idea of thinking about the world in terms of probabilities, assuming that there is uncertainty.”
There are two main components to achieve safety. First, the computer model must collect accurate data, and second, the reasoning system must be able to draw the right conclusions from the model’s data.
Ermon explains, “You need both: to build a reliable model you need a lot of data, and then you need to be able to draw the right conclusions based on the model, and that requires the artificial intelligence to think about these models accurately. Even if the model is right, but you don’t have a good way to reason about it, you can do catastrophic things.”
For example, in the context of autonomous vehicles, models use various sensors to observe the environment and collect data about countless variables, such as the behavior of the drivers around you, potholes and other obstacles in front of you, weather conditions—every possible data point.
A reasoning system then interprets this data. It uses the model’s information to decide whether the driver behind you is dangerously aggressive, if the pothole ahead will puncture your tire, if the rain is obstructing visibility, and the system continuously changes the car’s behavior to respond to these variables.
Consider the aggressive driver behind you. As Ermon explains, “Somehow you need to be able to reason about these models. You need to come up with a probability. You don’t know what the car’s going to do but you can estimate, and based on previous behavior you can say this car is likely to cut the line because it has been driving aggressively.”
Improving Probabilistic Reasoning
Ermon is creating strong algorithms that can synthesize all of the data that a model produces and make reliable decisions.
As models improve, they collect more information and capture more variables relevant to making these decisions. But as Ermon notes, “the more complicated the model is, the more variables you have, the more complicated it becomes to make the optimal decisions based on the model.”
Thus as the data collection expands, the analysis must also improve. The artificial intelligence in these cars must be able to reason with this increasingly complex data.
And this reasoning can easily go wrong. “You need to be very precise when computing these probabilities,” Ermon explains. “If the probability that a car cuts into your lane is 0.1, but you completely underestimate it and say it’s 0.01, you might end up making a fatal decision.”
To avoid fatal decisions, the artificial intelligence must be robust, but the data must also be complete. If the model collects incomplete data, “you have no guarantee that the number that you get when you run this algorithm has anything to do with the actual probability of that event,” Ermon explains.
The model and the algorithm entirely depend on each other to produce the optimal decision. If the model is incomplete and fails to capture the black ice in front of you, no reasoning system will be able to make a safe decision. And even if the model captures the black ice and every other possible variable, if the reasoning system cannot handle the complexity of this data, again the car will fail.
How Safe Will Autonomous Systems Be?
The technology in self-driving cars has made huge leaps lately, and Ermon is hopeful. “Eventually, as computers get better and algorithms get better and the models get better, hopefully we’ll be able to prevent all accidents,” he suggests.
However, there are still fundamental limitations on probabilistic reasoning. “Most computer scientists believe that it is impossible to come up with the silver bullet for this problem, an optimal algorithm that is so powerful that it can reason about all sorts of models that you can think about,” Ermon explains. “That’s the key barrier.”
But despite this barrier, self-driving cars will soon be available for consumers. Ford, for one, has promised to put its self-driving cars on the road by 2021. And while most computer scientists expect these cars to be far safer than human drivers, their success depends on their ability to reason probabilistically about their environment.
As Ermon explains, “You need to be able to estimate these kinds of probabilities because they are the building blocks that you need to make decisions.”
This article is part of a Future of Life series on the AI safety research grants, which were funded by generous donations from Elon Musk and the Open Philanthropy Project.
https://futureoflife.org/wp-content/uploads/2017/03/self-driving-cars-probability.jpg7751200Tucker Daveyhttps://futureoflife.org/wp-content/uploads/2015/10/FLI_logo-1.pngTucker Davey2017-03-13 15:57:482017-03-13 21:43:47How Self-Driving Cars Use Probability
AI Arms Race Principle:An arms race in lethal autonomous weapons should be avoided.*
Perhaps the scariest aspect of the Cold War was the nuclear arms race. At its peak, the US and Russia held over 70,000 nuclear weapons, only a fraction of which could have killed every person on earth. As the race to create increasingly powerful artificial intelligence accelerates, and as governments increasingly test AI capabilities in weapons, many AI experts worry that an equally terrifying AI arms race may already be under way.
In fact, at the end of 2015, the Pentagon requested $12-$15 billion for AI and autonomous weaponry for the 2017 budget, and the Deputy Defense Secretary at the time, Robert Work, admitted that he wanted “our competitors to wonder what’s behind the black curtain.” Work also said that the new technologies were “aimed at ensuring a continued military edge over China and Russia.”
But the US does not have a monopoly on this technology, and many fear that countries with lower safety standards could quickly pull ahead. Without adequate safety in place, autonomous weapons could be more difficult to control, create even greater risk of harm to innocent civilians, and more easily fall into the hands of terrorists, dictators, reckless states, or others with nefarious intentions.
Anca Dragan, an assistant professor at UC Berkeley, described the possibility of such an AI arms race as “the equivalent of very cheap and easily accessible nuclear weapons.”
“And that would not fare well for us,” Dragan added.
Unlike nuclear weapons, this new class of WMD can potentially target by traits like race or even by what people have liked on social media.
Lethal Autonomous Weapons
Toby Walsh, a professor at UNSW Australia, took the lead on the 2015 autonomous weapons open letter, which calls for a ban on lethal autonomous weapons and has been signed by over 20,000 people. With regard to that letter and the AI Arms Race Principle, Walsh explained:
“One reason that I got involved in these discussions is that there are some topics I think are very relevant today, and one of them is the arms race that’s happening amongst militaries around the world already, today. This is going to be very destabilizing. It’s going to upset the current world order when people get their hands on these sorts of technologies. It’s actually stupid AI that they’re going to be fielding in this arms race to begin with and that’s actually quite worrying – that it’s technologies that aren’t going to be able to distinguish between combatants and civilians, and aren’t able to act in accordance with international humanitarian law, and will be used by despots and terrorists and hacked to behave in ways that are completely undesirable. And that’s something that’s happening today.”
When asked about his take on this Principle, University of Montreal professor Yoshua Bengio pointed out that he had signed the autonomous weapons open letter, which basically “says it all” about his concerns of a potential AI arms race.
Details and Definitions
In addition to worrying about the risks of a race, Dragan also expressed a concern over “what to do about it and how to avoid it.”
“I assume international treaties would have to occur here,” she said.
Dragan’s not the only one expecting international treaties. The UN recently agreed to begin formal discussions that will likely lead to negotiations on an autonomous weapons ban or restrictions. However, as with so many things, the devil will be in the details.
In reference to an AI arms race, Cornell professor Bart Selman stated, “It should be avoided.” But he also added, “There’s a difference between it ‘should’ be avoided and ‘can’ it be avoided – that may be a much harder question.”
Selman would like to see “the same kinds of discussions as there were around atomic weapons or biological weapons, where people actually start to look at the tradeoffs and the risks of an arms race.”
“That discussion has to be had,” he said, “and it may actually bring people together in a positive way. Countries could get together and say this is not a good development and we should limit it and avoid it. So to bring it out as a principle, I think the main value there is that we need to have the discussion as a society and with other countries.”
Dan Weld, a professor at the University of Washington, also worries that simply saying an arms race should be avoided is insufficient.
“I fervently hope we don’t see an arms race in lethal autonomous weapons,” Weld explained. “That said, this principle bothered me, because it doesn’t seem to have any operational form. Specifically, an arms race is a dynamic phenomenon that happens when you’ve got multiple agents interacting. It takes two people to race. So whose fault is it if there is a race? I’m worried that both participants will point a finger at the other and say, ‘Hey, I’m not racing! Let’s not have a race, but I’m going to make my weapons more accurate and we can avoid a race if you just relax.’ So what force does the principle have?”
General Consensus
Though preventing an AI arms race may be tricky, there seems to be general consensus that a race would be bad and should be avoided.
“Weaponized AI is a weapon of mass destruction and an AI arms race is likely to lead to an existential catastrophe for humanity,” said Roman Yampolskiy, a professor at the University of Louisville.
Kay Firth-Butterfield, the Executive Director of AI-Austin.org, explained, “Any arms race should be avoided but particularly this one where the stakes are so high and the possibility of such weaponry, if developed, being used within domestic policing is so terrifying.”
But Stanford professor Stefano Ermon may have summed it up best when he said, “Even just with the capabilities we have today it’s not hard to imagine how [AI] could be used in very harmful ways. I don’t want my contributions to the field and any kind of techniques that we’re all developing to do harm to other humans or to develop weapons or to start wars or to be even more deadly than what we already have.”
What do you think?
Is an AI arms race inevitable? How can it be prevented? Can we keep autonomous weapons out of the hands of dictators and terrorists? How can companies and governments work together to build beneficial AI without allowing the technology to be used to create what could be the deadliest weapons the world has ever seen?
This article is part of a weekly series on the 23 Asilomar AI Principles. The Principles offer a framework to help artificial intelligence benefit as many people as possible. But, as AI expert Toby Walsh said of the Principles, “Of course, it’s just a start. … a work in progress.” The Principles represent the beginning of a conversation, and now we need to follow up with broad discussion about each individual principle. You can read the weekly discussions about previous principles here.
*The AI Arms Race Principle specifically addresses lethal autonomous weapons. Later in the series, we’ll discuss the Race Avoidance Principle which will look at the risks of companies racing to creating AI technology.
https://futureoflife.org/wp-content/uploads/2017/03/AI-arms-race-principle.jpg7361100Ariel Connhttps://futureoflife.org/wp-content/uploads/2015/10/FLI_logo-1.pngAriel Conn2017-03-09 15:17:412017-03-09 15:18:10Is an AI Arms Race Inevitable?
The following article and talk are by Jessica Taylor and they were originally posted on MIRI.
At the EA Global 2016 conference, I gave a talk on “Using Machine Learning to Address AI Risk”:
It is plausible that future artificial general intelligence systems will share many qualities in common with present-day machine learning systems. If so, how could we ensure that these systems robustly act as intended? We discuss the technical agenda for a new project at MIRI focused on this question.
A recording of my talk is now up online:
The talk serves as a quick survey (for a general audience) of the kinds of technical problems we’re working on under the “Alignment for Advanced ML Systems” research agenda. Included below is a version of the talk in blog post form.1
This talk is about a new research agenda aimed at using machine learning to make AI systems safe even at very high capability levels. I’ll begin by summarizing the goal of the research agenda, and then go into more depth on six problem classes we’re focusing on.
The goal statement for this technical agenda is that we want to know how to train a smarter-than-human AI system to perform one or more large-scale, useful tasks in the world.
Some assumptions this research agenda makes:
Future AI systems are likely to look like more powerful versions of present-day ML systems in many ways. We may get better deep learning algorithms, for example, but we’re likely to still be relying heavily on something like deep learning.2
Artificial general intelligence (AGI) is likely to be developed relatively soon (say, in the next couple of decades).3
Building task-directed AGI is a good idea, and we can make progress today studying how to do so.
I’m not confident that all three of these assumptions are true, but I think they’re plausible enough to deserve about as much attention from the AI community as the likeliest alternative scenarios.
A task-directed AI system is a system that pursues a semi-concrete objective in the world, like “build a million houses” or “cure cancer.” For those who have read Superintelligence, task-directed AI is similar to the idea of genie AI. Although these tasks are kind of fuzzy — there’s probably a lot of work you’d need to do to clarify what it really means to build a million houses, or what counts as a good house — they’re at least somewhat concrete.
An example of an AGI system that isn’t task-directed would be one with a goal like “learn human values and do things humans would consider good upon sufficient reflection.” This is too abstract to count as a “task” in the sense we mean; it doesn’t directly cash out in things in the world.
The hope is that even though task-directed AI pursues a less ambitious objective then “learn human values and do what we’d want it to do,” it’s still sufficient to prevent global catastrophic risks. Once the immediate risks are averted, we can then work on building more ambitious AI systems under reduced time pressure.
Task-directed AI uses some (moderate) amount of human assistance to clarify the goal and to evaluate and implement its plans. A goal like “cure cancer” is vague enough that humans will have to do some work to clarify what they mean by it, though most of the intellectual labor should be coming from the AI system rather than from humans.
Ideally, task-directed AI also shouldn’t require significantly more computational resources than competing systems. You shouldn’t get an exponential slowdown from building a safe system vs. a generic system.
In order to think about this overall goal, we need some kind of model for these future systems. The general approach that I’ll take is to look at current systems and imagine that they’re more powerful. A lot of the time you can look at tasks that people do in ML and you can see that the performance improves over time. We’ll model more advanced AI systems by just supposing that systems will continue to achieve higher scores in ML tasks. We can then ask what kinds of failure modes are likely to arise as systems improve, and what we can work on today to make those failures less likely or less costly.
Six potential problems with highly capable AI systems
Problem 1: Actions are hard to evaluate
Suppose an AI system composes a story, and a human gives the system a reward based on how good the story is.4
This is similar to some RL tasks: the agent wants to do something that will cause it to receive a high reward in the future. The formalism of RL would say that the objective of this RL agent is to write a story that the human is expected to give a high score to.
For this objective to actually help us receive very high-quality stories, however, we also need to know that the human understands the RL agent’s actions well enough to correctly administer rewards. This assumption seems less likely to hold for systems that are optimizing the objective much more powerfully than any present-day system. For example:
A system much smarter than a human may be able to manipulate or coerce the human into giving a bad story a high score.
Even if the system is less intelligent than that, it might resort to plagiarism. Plagiarism can be easier to generate than to detect, since detection often requires scouring a larger pool of source texts.
A subhuman system might also have an advantage in inserting steganography into the story; it might take polynomial time to embed a secret message, and exponential time to detect such a message. Finding a way to discourage agents from taking covert actions like these would make it easier to monitor those actions’ effects and keep operators in the loop.
Do we have a general way of preventing this? Can we train an RL system to not only output an action (e.g., a story), but also a report that might help an overseer better evaluate the system’s performance? Following OpenAI researcher Paul Christiano, we call this the problem of informed oversight.5
Problem 2: Ambiguous test examples
Another problem: Consider a classifier trained to distinguish images of cats from images not containing cats, or trained to detect cancer. You may have lots of life experience that tells you “wild cats are cats.” If the training set only contains images of house cats and dogs, however, then it may not be possible to infer this fact during training.
An AI system that was superhumanly good at classifying images from a particular data set might not construct the same generalizations as a human, making it unreliable in new environments.
In safety-critical settings, ideally we would like the classifier to say, “This is ambiguous,” to alert us that the image’s label is underdetermined by the labels of training set images. We could then leverage the classifier’s proficiency at classification to intervene in contexts where the system is relatively likely to misclassify things, and could also supply training data that’s tailored to the dimensions along which the original data was uninformative. Formalizing this goal is the problem of inductive ambiguity detection.
Problem 3: Difficulty imitating human behavior
One way we might want to deal with the problems of smarter-than-human AI is to just have it imitate a human. By having the system observe a human attempting to perform some task and having it learn to behave similarly, we might be able to leverage the AI system’s cognitive advantages while having it mostly stick to fairly normal strategies.
As an example, we can consider an AI system tasked with producing the kind of picture a human would draw. How could you actually train a system on this objective?
In one approach, generative adversarial networks, you have a human and two AI agents: an imitator, and a distinguisher.6The distinguisher tries to guess whether a given picture came from the human or from the imitator, and the imitator tries to produce the kinds of image that the distinguisher classifies as coming from the human.
This is a pretty cool setup, but it raises some theoretical issues. We could ask the question, “Does the distinguisher have to be smarter than the imitator? And if so, how much smarter?” If the distinguisher is less smart than the imitator, then the imitator could covertly do very non-human-like things to the image, as long as they’re too subtle for the distinguisher to notice. An open avenue of research here is to analyze schemes like this and ask under what conditions we can get nice robustness guarantees.
Problem 4: Difficulty specifying goals about the real world
What if we want to train our AI system to perform a task like making a sandwich? One option might be to hook the system up to a robot in a room full of sandwich ingredients, have it perform an action sequence, and then have a human observer rate the robot’s performance based on how close it came to making a sandwich. That rating determines the robot’s reward.
We previously noted that sufficiently capable RL agents might pick actions that are hard to evaluate. Here we face the additional problem that useful tasks will often require taking physical action in the world. If the system is capable enough, then this setup gives it an incentive to take away the reward button and press it itself. This is what the formalism of RL would tell you is the best action, if we imagine AI systems that continue to be trained in the RL framework far past current capability levels.
A natural question, then, is whether we can train AI systems that just keep getting better at producing a sandwich as they improve in capabilities, without ever reaching a tipping point where they have an incentive to do something else. Can we avoid relying on proxies for the task we care about, and just train the system to value completing the task in its own right? This is the generalizable environmental goals problem.
Problem 5: Negative side-effects
Suppose we succeeded in making a system that wants to put a sandwich in the room. In choosing between plans, it will favor whichever plan has the higher probability of resulting in a sandwich. Perhaps the policy of just walking over and making a sandwich has a 99.9% chance of success; but there’s always a chance that a human could step in and shut off the robot. A policy that drives down the probability of interventions like that might push up the probability of the room ending up containing a sandwich to 99.9999%. In this way, sufficiently advanced ML systems can end up with incentives to interfere with their developers and operators even when there’s no risk of reward hacking.
This is the problem of designing task-directed systems that can become superhumanly good at achieving their task, without causing negative side-effects in the process.
One response to this problem is to try to quantify how much total impact different policies have on the world. We can then add a penalty term for actions that have a high impact, causing the system to favor low-impact strategies.
Another approach is to ask how we might design an AI system to be satisfied with a merely 99.9% chance of success — just have the system stop trying to think up superior policies once it finds one meeting that threshold. This is the problem of formalizing mild optimization.
Or one can consider advanced AI systems from the perspective of convergent instrumental strategies. No matter what the system is trying to do, it can probably benefit by having more computational resources, by having the programmers like it more, by having more money. A sandwich-making system might want money so it can buy more ingredients, whereas a story-writing system might want money so it can buy books to learn from. Many different goals imply similar instrumental strategies, a number of which are likely to introduce conflicts due to resource limitations.
One, approach, then, would be to study these instrumental strategies directly and try to find a way to design a system that doesn’t exhibit them. If we can identify common features of these strategies, and especially of the adversarial strategies, then we could try to proactively avert the incentives to pursue those strategies. This seems difficult, and is very underspecified, but there’s some initial research pointed in this direction.
Problem 6: Edge cases that still satisfy the goal
Another problem that’s likely to become more serious as ML systems become more advanced is edge cases.
Consider our ordinary concept of a sandwich. There are lots of things that technically count as sandwiches, but are unlikely to have the same practical uses a sandwich normally has for us. You could have an extremely small or extremely large sandwich, or a toxic sandwich.
For an example of this behavior in present-day systems, we can consider this image that an image classifier correctly classified as a panda (with 57% confidence). Goodfellow, Shlens, and Szegedy found that they could add a tiny vector to this image that causes the classifier to misclassify it as a gibbon with 99% confidence.7
Such edge cases are likely to become more common and more hazardous as ML systems begin to search wider solution spaces than humans are likely (or even able) to consider. This is then another case where systems might become increasingly good at maximizing their score on a conventional metric, while becoming less reliable for achieving realistic goals we care about.
Conservative concepts are an initial idea for trying to address this problem, by biasing systems to avoid assigning positive classifications to examples that are near the edges of the search space. The system might then make the mistake of thinking that some perfectly good sandwiches are inadmissible, but it would not make the more risky mistake of classifying toxic or otherwise bizarre sandwiches as admissible.
Technical details on one problem: inductive ambiguity identification
I’ve outlined eight research directions for addressing six problems that seem likely to start arising (or to become more serious) as ML systems become better at optimizing their objectives — objectives that may not exactly match programmers’ intentions. The research directions were:
Informed oversight, for making it easier to interpret and assess ML systems’ actions.
Inductive ambiguity identification, for designing classifiers that stop and check in with overseers in circumstances where their training data was insufficiently informative.
Robust human imitation, for recapitulating the safety-conducive features of humans in ML systems.
These problems are discussed in more detail in “Alignment for Advanced ML Systems.” I’ll go into more technical depth on an example problem to give a better sense of what working on these problems looks like in practice.
KWIK learning
Let’s consider the inductive ambiguity identification problem, applied to a classifier for 2D points. In this case, we have 4 positive examples and 4 negative examples.
When a new point comes in, the classifier could try to label it by drawing a whole bunch of models that are consistent with the previous data. Here, I draw just 4 of them. The question mark falls on opposite sides of these different models, suggesting that all of these models are plausible given the data.
We can suppose that the system infers from this that the training data is ambiguous with respect to the new point’s classification, and asks the human to label it. The human might then label it with a plus, and the system draws new conclusions about which models are plausible.
This approach is called “Knows What It Knows” learning, or KWIK learning. We start with some input space X ≔ ℝn and assume that there exists some true mapping from inputs to probabilities. E.g., for each image the cat classifier encounters we assume that there is a true answer in the set Y ≔ [0,1] to the question, “What is the probability that this image is a cat?” This probability corresponds to the probability that a human will label that image “1” as opposed to “0,” which we can represent as a weighted coin flip. The model maps the inputs to answers, which in this case are probabilities.8
The KWIK learner is going to play a game. At the beginning of the game, some true model h* gets picked out. The true model is assumed to be in the hypothesis set H. On each iteration i some new example xi ∈ ℝncomes in. It has some true answer yi = h*(xi), but the learner is unsure about the true answer. The learner has two choices:
Output an answer ŷi ∈ [0,1].
If |ŷi – yi| > ε, the learner then loses the game.
Output ⊥ to indicate that the example is ambiguous.
The learner then gets to observe the true label zi = FlipCoin(yi) from the observation set Z ≔ {0,1}.
The goal is to not lose, and to not output ⊥ too many times. The upshot is that it’s actually possible to win this game with a high probability if the hypothesis class H is a small finite set or a low-dimensional linear class. This is pretty cool. It turns out that there are certain forms of uncertainty where we can just resolve the ambiguity.
The way this works is that on each new input, we consider multiple models h that have done well in the past, and we consider something “ambiguous” if the models disagree on h(xi) by more than ε. Then we just refine the set of models over time.
The way that a KWIK learner represents this notion of inductive ambiguity is: ambiguity is about not knowing which model is correct. There’s some set of models, many are plausible, and you’re not sure which one is the right model.
There are some problems with this. One of the main problems is KWIK learning’s realizability assumption — the assumption that the true model h* is actually in the hypothesis set H. Realistically, the actual universe won’t be in your hypothesis class, since your hypotheses need to fit in your head. Another problem is that this method only works for these very simple model classes.
A Bayesian view of the problem
That’s some existing work on inductive ambiguity identification. What’s some work we’ve been doing at MIRI related to this?
Lately, I’ve been trying to approach this problem from a Bayesian perspective. On this view, we have some kind of prior Q over mappings X → {0,1} from the input space to the label. The assumption we’ll make is that our prior is wrong in some way and there’s some unknown “true” prior P over these mappings. The goal is that even though the system only has access to Q, it should perform the classification task almost as well (in expectation over P) as if it already knew P.
It seems like this task is hard. If the real world is sampled from P, and P is different from your prior Q, there aren’t that many guarantees. To make this tractable, we can add a grain of truth assumption:
∀f:Q(f)≥1kP(f)
This says that if P assigns a high probability to something, then so does Q. Can we get good performance in various classification tasks under this kind of assumption?
We haven’t completed this research avenue, but initial results suggest that it’s possible to do pretty well on this task while avoiding catastrophic behaviors in at least in some cases (e.g., online supervised learning). That’s somewhat promising, and this is definitely an area for future research.
How this ties in to inductive ambiguity identification: If you’re uncertain about what’s true, then there are various ways of describing what that uncertainty is about. You can try taking your beliefs and partitioning them into various possibilities. That’s in some sense an ambiguity, because you don’t know which possibility is correct. We can think of the grain of truth assumption as saying that there’s some way of splitting up your probability distribution into components such that one of the components is right. The system should do well even though it doesn’t initially know which component is right.
Let’s return to a broad view and consider other research agendas focused on long-run AI safety. The first such agenda was outlined in MIRI’s 2014 agent foundations report.9
The agent foundations agenda is about developing a better theoretical understanding of reasoning and decision-making. An example of a relevant gap in our current theories is ideal reasoning about mathematical statements (including statements about computer programs), in contexts where you don’t have the time or compute to do a full proof. This is the basic problem we’re responding to in “Logical Induction.” In this talk I’ve focused on problems for advanced AI systems that broadly resemble present-day ML; in contrast, the agent foundations problems are agnostic about the details of the system. They apply to ML systems, but also to other possible frameworks for good general-purpose reasoning.
Then there’s the “Concrete Problems in AI Safety” agenda.10 Here the idea is to study AI safety problems with a more empirical focus, specifically looking for problems that we can study using current ML methods, and perhaps can even demonstrate in current systems or in systems that might be developed in the near future.
As an example, consider the question, “How do you make an RL agent that behaves safely while it’s still exploring its environment and learning about how the environment works?” It’s a question that comes up in current systems all the time, and is relatively easy to study today, but is likely to apply to more capable systems as well.
These different agendas represent different points of view on how one might make AI systems more reliable in a way that scales with capabilities progress, and our hope is that by encouraging work on a variety of different problems from a variety of different perspectives, we’re less likely to completely miss a key consideration. At the same time, we can achieve more confidence that we’re on the right track when relatively independent approaches all arrive at similar conclusions.
I’m leading the team at MIRI that will be focusing on the “Alignment for Advanced ML Systems” agenda going forward. It seems like there’s a lot of room for more eyes on these problems, and we’re hoping to hire a number of new researchers and kick off a number of collaborations to tackle these problems. If you’re interested in these problems and have a solid background in mathematics or computer science, I definitely recommend getting in touch or reading more about these problems.
I also gave a version of this talk at the MIRI/FHI Colloquium on Robust and Beneficial AI. ↩
Alternatively, you may think that AGI won’t look like modern ML in most respects, but that the ML aspects are easier to productively study today and are unlikely to be made completely irrelevant by future developments. ↩
Alternatively, you may think timelines are long, but that we should focus on scenarios with shorter timelines because they’re more urgent. ↩
Although I’ll use the example of stories here, in real life it could be a system generating plans for curing cancers, and humans evaluating how good the plans are. ↩
See the Q&A section of the talk for questions like “Won’t the report be subject to the same concerns as the original story?” ↩
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. “Concrete Problems in AI Safety”. In: (2016). arXiv: 1606.06565 [cs.AI]. ↩
https://futureoflife.org/wp-content/uploads/2017/03/Machine-learning-ai-risks.jpg5001000a guest bloggerhttps://futureoflife.org/wp-content/uploads/2015/10/FLI_logo-1.pnga guest blogger2017-03-07 18:59:222017-03-07 19:31:30Using Machine Learning to Address AI Risk
Importance Principle:Advanced AI could represent a profound change in the history of life on Earth, and should be planned for and managed with commensurate care and resources.
In the history of human progress, a few events have stood out as especially revolutionary: the intentional use of fire, the invention of agriculture, the industrial revolution, possibly the invention of computers and the Internet. But many anticipate that the creation of advanced artificial intelligence will tower over these achievements.
Or, as AI professor Roman Yampolskiy told me, “Design of human-level AI will be the most impactful event in the history of humankind. It is impossible to over-prepare for it.”
The Importance Principle encourages us to plan for what could be the greatest “change in the history of life.” But just what are we preparing for? What will more advanced AI mean for society? I turned to some of the top experts in the field of AI to consider these questions.
Societal Benefits?
Guruduth Banavar, the Vice President of IBM Research, is hopeful that as AI advances, it will help humanity advance as well. In favor of the principle, he said, “I strongly believe this. I think this goes back to evolution. From the evolutionary point of view, humans have reached their current level of power and control over the world because of intelligence. … AI is augmented intelligence – it’s a combination of humans and AI working together. And this will produce a more productive and realistic future than autonomous AI, which is too far out. In the foreseeable future, augmented AI – AI working with people – will transform life on the planet. It will help us solve the big problems like those related to the environment, health, and education.”
“I think I also agreed with that one,” said Bart Selman, a professor at Cornell University. “Maybe not every person on earth should be concerned about it, but there should be, among scientists, a discussion about these issues and a plan – can you build safety guidelines to work with value alignment work? What can you actually do to make sure that the developments are beneficial in the end?”
Anca Dragan, an assistant professor at UC Berkeley, explained, “Ultimately, we work on AI because we believe it can have a strong positive impact on the world. But the more capable the technology becomes, the easier it becomes to misuse it – or perhaps, the effects of misusing it become more drastic. That is why it is so important, as we make progress, to start thinking more strongly about what role AI will play.”
Short-term Concerns
Though the Importance Principle specifically mentions advanced AI, some of the researchers I interviewed pointed out that nearer-term artificial intelligence could also drastically impact humanity.
“I believe that AI will create profound change even before it is ‘advanced’ and thus we need to plan and manage growth of the technology,” explained Kay Firth-Butterfield, Executive Director of AI-Austin.org. “As humans, we are not good at long-term planning because our civil systems don’t encourage it, however, this is an area in which we must develop our abilities to ensure a responsible and beneficial partnership between man and machine.”
Stefano Ermon, an assistant professor at Stanford University, also considered the impacts of less advanced AI, saying, “It’s an incredibly powerful technology. I think it’s even hard to imagine what one could do if we are able to develop a strong AI, but even before that, well before that, the capabilities are really huge. We’ve seen the kind of computers and information technologies we have today, the way they’ve revolutionized our society, our economy, our everyday lives. And my guess is that AI technologies would have the potential to be even more impactful and even more revolutionary on our lives. And so I think it’s going to be a big change and it’s worth thinking very carefully about, although it’s hard to plan for it.”
In a follow up question about planning for AI over the shorter term, Selman added, “I think the effect will be quite dramatic. This is another interesting point – sometimes AI scientists say, well it might not be advanced AI will do us in, but dumb AI. … The example is always the self-driving car has no idea it’s driving you anywhere. It doesn’t even know what driving is. … If you looked at the videos of an accident that’s going to happen, people are so surprised that the car doesn’t hit the brakes at all, and that’s because the car works quite differently than humans. So I think there is some short-term [AI] risk in that … we actually think they’re smarter than they are. And I think that will actually go away when the machines become smarter, but for now…”
Learning From Experience
As revolutionary as advanced AI might be, we can still learn from previous technological revolutions and draw on their lessons to prepare for the changes ahead.
Toby Walsh, a guest professor at Technical University of Berlin, expressed a common criticism of the principles, arguing that the Importance Principle could – and probably should – apply to many “groundbreaking technologies.”
He explained, “This is one of those principles where I think you could put any society-changing technology in place of advanced AI. … It would be true of the steam engine, in some sense it’s true of social media and we’ve failed at that one, it could be true of the Internet but we failed at planning that well. It could be true of fire too, but we failed on that one as well and used it for war. But to get back to the observation that some of them are things that are not particular to AI – once you realize that AI is going to be groundbreaking, then all of the things that should apply to any groundbreaking technology should apply.”
By looking back at these previous revolutionary technologies and understanding their impacts, perhaps we can gain insight into how we can plan ahead for advanced AI.
Dragan was also interested more explicit solutions to the problem of planning ahead.
“As the AI capabilities advance,” she told me, “we have to take a step back and ask ourselves: are we solving the right problem? Is there a better problem definition that will more likely result in benefits to humanity?
“For instance, we have always defined AI agents as rational. That means they maximize expected utility. Thus far, utility is assumed to be known. But if you think about it, there is no gospel specifying utility. We are assuming that some *person* somewhere will know exactly what utility to specify for their agent. Well, it turns out, we don’t work like that: it is really hard for people, including AI experts, to specify utility functions. We try our best, but when the system goes ahead and optimizes for what we inputted, the result is sometimes surprising, and not in a good way. This suggests that our definition of an AI agent is predicated on a wrong assumption. We’ve already started seeing that in robotics – the definition of how a robot should move didn’t account for people, the definition of how a robot should learn from demonstration assumed that people can provide perfect demonstrations to a robot, etc. – I assume we are going to see this more and more in AI as a whole. We have to stop making implicit assumptions about people and end-users of AI, and rigorously tackle that head-on, putting people into the equation.”
What Do You Think?
What kind of impact will advanced AI have on the development of human progress? How can we prepare for such potentially tremendous changes? Can we prepare? What other questions do we, as a society, need to ask?
This article is part of a weekly series on the 23 Asilomar AI Principles. The Principles offer a framework to help artificial intelligence benefit as many people as possible. But, as AI expert Toby Walsh said of the Principles, “Of course, it’s just a start. … a work in progress.” The Principles represent the beginning of a conversation, and now we need to follow up with broad discussion about each individual principle. You can read the weekly discussions about previous principles here.
https://futureoflife.org/wp-content/uploads/2017/02/importance-principle.jpg8001200Ariel Connhttps://futureoflife.org/wp-content/uploads/2015/10/FLI_logo-1.pngAriel Conn2017-02-24 16:25:122017-02-24 16:25:46Preparing for the Biggest Change in Human History
Imagine how much more efficient lawyers could be if they had the time to read every legal book ever written and review every case ever brought to court. Imagine doctors with the ability to study every advancement published across the world’s medical journals, or consult every medical case, ever. Unfortunately, the human brain cannot store that much information, and it would take decades to achieve these feats.
But a computer, one specifically designed to work like the human mind, could.
Deep learning neural networks are designed to mimic the human brain’s neural connections. They are capable of learning through continuous exposure to huge amounts of data. This allows them to recognize patterns, comprehend complex concepts, and translate high-level abstractions. These networks consist of many layers, each having a different set of weights. The deeper the network, the stronger it is.
Current applications for these networks include medical diagnosis, robotics and engineering, face recognition, and automotive navigation. However, deep learning is still in development – not surprisingly, it is a huge undertaking to get machines to think like humans. In fact, very little is understood about these networks, and months of manual tuning are often required for obtaining excellent performance.
Fuxin Li, assistant professor at the Oregon State University School of Electrical Engineering and Computer Science, and his team are taking on the accuracy of these neural networks under adversarial conditions. Their research focuses on the basic machine learning aspects of deep learning, and how to make general deep learning more robust.
To try to better understand when a deep convolutional neural network (CNN) is going to be right or wrong, Li’s team had to establish an estimate of confidence in the predictions of the deep learning architecture. Those estimates can be used as safeguards when utilizing the networks in real life.
“Basically,” explains Li, “trying to make deep learning increasingly self-aware – to be aware of what type of data it has seen, and what type of data it could work on.”
The team looked at recent advances in deep learning, which have greatly improved the capability to recognize images automatically. Those networks, albeit very resistant to overfitting, were discovered to completely fail if some of the pixels in such images were perturbed via an adversarial optimization algorithm.
To a human observer, the image in question may look fine, but the deep network sees otherwise. According to the researchers, those adversarial examples are dangerous if a deep network is utilized into any crucial real application, such as autonomous driving. If the result of the network can be hacked, wrong authentications and other devastating effects would be unavoidable.
In a departure from previous perspectives that focused on improving the classifiers to correctly organize the adversarial examples, the team focused on detecting those adversarial examples by analyzing whether they come from the same distribution as the normal examples. The accuracy for detecting adversarial examples exceeded 96%. Notably, 90% of the adversarials can be detected with a false positive rate of less than 10%.
The benefits of this research are numerous. It is vital for a neural network to be able to identify whether an example comes from a normal or an adversarial distribution. Such knowledge, if available, will help significantly to control behaviors of robots employing deep learning. A reliable procedure can prevent robots from behaving in an undesirable manner because of the false perceptions it made about the environment.
Li gives one example: “In robotics there’s this big issue about robots not doing something based on erroneous perception. It’s important for a robot to know that it’s not making a confident perception. For example, if [the robot] is saying there’s an object over there, but it’s actually a wall, he’ll go to fetch that object, and then he hits a wall.”
Hopefully, Li says, that won’t happen. However, current software and machine learning have been mostly based solely on prediction confidence within the original machine learning framework. Basically, the testing and training data are assumed to be pulled from the same distribution independently, and that can lead to incorrect assumptions.
Better confidence estimates could potentially help avoid incidents such as the Tesla crash scenario from May 2016, where an adversarial example (truck with too much light) was in the middle of the highway that cheated the system. A confidence estimate could potentially solve that issue. But first, the computer must be smarter. The computer has to learn to detect objects and differentiate, say, a tree from another vehicle.
“To make it really robust, you need to account for unknown objects. Something weird may hit you. A deer may jump out.” The network can’t be taught every unexpected situation, says Li, “so you need it to discover them without knowledge of what they are. That’s something that we do. We try to bridge the gap.”
Training procedures will make deep learning more automatic and lead to fewer failures, as well as confidence estimates when the deep network is utilized to predict new data. Most of this training, explains Li, comes from photo distribution using stock images. However, these are flat images much different than what a robot would normally see in day-to-day life. It’s difficult to get a 360-degree view just by looking at photos.
“There will be a big difference between the thing [the robot] trains on and the thing it really sees. So then, it is important for the robot to understand that it can predict some things confidently, and others it cannot,” says Li. “[The robot] needs to understand that it probably predicted wrong, so as not to act too aggressively toward its prediction.” This can only be achieved with a more self-aware framework, which is what Li is trying to develop with this grant.
Further, these estimates can be used to control the behavior of a robot employing deep learning so that it will not go on to perform maneuvers that could be dangerous because of erroneous predictions. Understanding these aspects would also be helpful in designing potentially more robust networks in the future.
Soon, Li and his team will start generalizing the approach to other domains, such as temporal models (RNNs, LSTMs) and deep reinforcement learning. In reinforcement learning, the confidence estimates could play an important role in many decision-making paradigms.
Li’s most recent update on this work can be found here.
This article is part of a Future of Life series on the AI safety research grants, which were funded by generous donations from Elon Musk and the Open Philanthropy Project.
https://futureoflife.org/wp-content/uploads/2015/11/neurons_artificial_intelligence.jpg10001500Sarah Marquarthttps://futureoflife.org/wp-content/uploads/2015/10/FLI_logo-1.pngSarah Marquart2017-02-22 15:06:492017-02-22 16:14:43Making Deep Learning More Robust
Capability Caution Principle: There being no consensus, we should avoid strong assumptions regarding upper limits on future AI capabilities.
A major change is coming, over unknown timescales but across every segment of society, and the people playing a part in that transition have a huge responsibility and opportunity to shape it for the best. What will trigger this change? Artificial intelligence.
The 23 Asilomar AI Principles offer a framework to help artificial intelligence benefit as many people as possible. But, as AI expert Toby Walsh said of the Principles, “Of course, it’s just a start. … a work in progress.” The Principles represent the beginning of a conversation, and now we need to follow up with broad discussion about each individual principle. You can read the weekly discussions about previous principles here.
Capability Caution
One of the greatest questions facing AI researchers is: just how smart and capable can artificial intelligence become?
In recent years, the development of AI has accelerated in leaps and bounds. DeepMind’s AlphaGo surpassed human performance in the challenging, intricate game of Go, and the company has created AI that can quickly learn to play Atari video games with much greater prowess than a person. We’ve also seen breakthroughs and progress in language translation, self-driving vehicles, and even the creation of new medicinal molecules.
But how much more advanced can AI become? Will it continue to excel only in narrow tasks, or will it develop broader learning skills that will allow a single AI to outperform a human in most tasks? How do we prepare for an AI more intelligent than we can imagine?
Some experts think human-level or even super-human AI could be developed in a matter of a couple decades, while some don’t think anyone will ever accomplish this feat. The Capability Caution Principle argues that, until we have concrete evidence to confirm what an AI can someday achieve, it’s safer to assume that there are no upper limits – that is, for now, anything is possible and we need to plan accordingly.
Expert Opinion
The Capability Caution Principle drew both consensus and disagreement from the experts. While everyone I interviewed generally agreed that we shouldn’t assume upper limits for AI, their reasoning varied and some raised concerns.
Stefano Ermon, an assistant professor at Stanford and Roman Yampolskiy, an associate professor at the University of Louisville, both took a better-safe-than-sorry approach.
Ermon turned to history as a reminder of how difficult future predictions are. He explained, “It’s always hard to predict the future. … Think about what people were imagining a hundred years ago, about what the future would look like. … I think it would’ve been very hard for them to imagine what we have today. I think we should take a similar, very cautious view, about making predictions about the future. If it’s extremely hard, then it’s better to play it safe.”
Yampolskiy considered current tech safety policies, saying, “In many areas of computer science such as complexity or cryptography the default assumption is that we deal with the worst case scenario. Similarly, in AI Safety we should assume that AI will become maximally capable and prepare accordingly. If we are wrong we will still be in great shape.”
Dan Weld, a professor at the University of Washington, said of the principle, “I agree! As a scientist, I’m against making strong or unjustified assumptions about anything, so of course I agree.”
But though he agreed with the basic idea behind the principle, Weld also had reservations. “This principle bothers me,” Weld explained, “… because it seems to be implicitly saying that there is an immediate danger that AI is going to become superhumanly, generally intelligent very soon, and we need to worry about this issue. This assertion … concerns me because I think it’s a distraction from what are likely to be much bigger, more important, more near-term, potentially devastating problems. I’m much more worried about job loss and the need for some kind of guaranteed health-care, education and basic income than I am about Skynet. And I’m much more worried about some terrorist taking an AI system and trying to program it to kill all Americans than I am about an AI system suddenly waking up and deciding that it should do that on its own.”
Looking at the problem from a different perspective, Guruduth Banavar, the Vice President of IBM Research, worries that placing upper bounds on AI capabilities could limit the beneficial possibilities. Banavar explained, “The general idea is that intelligence, as we understand it today, is ultimately the ability to process information from all possible sources and to use that to predict the future and to adapt to the future. It is entirely in the realm of possibility that machines can do that. … I do think we should avoid assumptions of upper limits on machine intelligence because I don’t want artificial limits on how advanced AI can be.”
IBM research scientist Francesca Rossi considered this principle from yet another perspective, suggesting that AI is necessary for humanity to reach our full capabilities, where we also don’t want to assume upper limits.
“I personally am for building AI systems that augment human intelligence instead of replacing human intelligence,” said Rossi, “And I think that in that space of augmenting human intelligence there really is a huge potential for AI in making the personal and professional lives of everybody much better. I don’t think that there are upper limits of the future AI capabilities in that respect. I think more and more AI systems together with humans will enhance our kind of intelligence, which is complementary to the kind of intelligence that machines have, and will help us make better decisions, and live better, and solve problems that we don’t know how to solve right now. I don’t see any upper limit to that.”
What do you think?
Is there an upper limit to artificial intelligence? Is there an upper limit to what we can achieve with AI? How long will it take to achieve increasing levels of advanced AI? How do we plan for the future with such uncertainties? How can society as a whole address these questions? What other questions should we be asking about AI capabilities?
https://futureoflife.org/wp-content/uploads/2017/02/capability-caution.jpg9001200Ariel Connhttps://futureoflife.org/wp-content/uploads/2015/10/FLI_logo-1.pngAriel Conn2017-02-17 15:24:332017-02-17 15:42:29How Smart Can AI Get?
When the press covers this topic, they often focus on one of two problems: “What if the wrong group of humans develops smarter-than-human AI first?”, and “What if AI’s natural desires cause 𝗨 to diverge from 𝘝?”
In my view, the “wrong humans” issue shouldn’t be the thing we focus on until we have reason to think we could get good outcomes with the right group of humans. We’re very much in a situation where well-intentioned people couldn’t leverage a general AI system to do good things even if they tried. As a simple example, if you handed me a box that was an extraordinarily powerful function optimizer — I could put in a description of any mathematical function, and it would give me an input that makes the output extremely large — then I do know how to use the box to produce a random catastrophe, but I don’t actually know how I could use that box in the real world to have a good impact.
