Posts in this category get featured at the top of the front page.

Podcast: Top AI Breakthroughs, with Ian Goodfellow and Richard Mallah

2016 saw some significant AI developments. To talk about the AI progress of the last year, we turned to Richard Mallah and Ian Goodfellow. Richard is the director of AI projects at FLI, he’s the Senior Advisor to multiple AI companies, and he created the highest-rated enterprise text analytics platform. Ian is a research scientist at OpenAI, he’s the lead author of the Deep Learning textbook, and he’s a lead inventor of Generative Adversarial Networks.

The following interview has been heavily edited for brevity, but you can listen to it in its entirety above or read the full transcript here.

Ariel: Two events stood out to me in 2016. The first was AlphaGo, which beat the world’s top Go champion, Lee Sedol last March. What is AlphaGo, and why was this such an incredible achievement?

Ian: AlphaGo was DeepMind’s system for playing the game of Go. It’s a game where you place stones on a board with two players, the object being to capture as much territory as possible. But there are hundreds of different positions where we can place a stone on each turn. It’s not even remotely possible to use a computer to simulate many different Go games and figure out how the game will progress in the future. The computer needs to rely on intuition the same way that human Go players can look at a board and get kind of a sixth sense that tells them whether the game is going well or poorly for them, and where they ought to put the next stone. It’s computationally infeasible to explicitly calculate what each player should do next.

Richard: The DeepMind team has one network for what’s called value learning and another deep network for policy learning. The policy is, basically, which places should I evaluate for the next piece. The value network is how good that state is, in terms of the probability that the agent will be winning. And then they do a Monte Carlo tree search, which means it has some randomness and many different paths — on the order of thousands of evaluations. So it’s much more like a human considering a handful of different moves and trying to determine how good those moves would be.

Ian: From 2012 to 2015 we saw a lot of breakthroughs where the exciting thing was that AI was able to copy a human ability. In 2016, we started to see breakthroughs that were all about exceeding human performance. Part of what was so exciting about AlphaGo was that AlphaGo did not only learn how to predict what a human expert Go player would do, AlphaGo also improved beyond that by practicing playing games against itself and learning how to be better than the best human player. So we’re starting to see AI move beyond what humans can tell the computer to do.

Ariel: So how will this be applied to applications that we’ll interact with on a regular basis? How will we start to see these technologies and techniques in action ourselves?

Richard: With these techniques, a lot of them are research systems. It’s not necessarily that they’re going to directly go down the pipeline towards productization, but they are helping the models that are implicitly learned inside of AI systems and machine learning systems to get much better.

Ian: There are other strategies for generating new experiences that resemble previously seen experiences. One of them is called WaveNet. It’s a model produced by DeepMind in 2016 for generating speech. If you provide a sentence, just written down, and you’d like to hear that sentence spoken aloud, WaveNet can create an audio waveform that sounds very realistically like a human pronouncing that sentence written down. The main drawback to WaveNet right now is that it’s fairly slow. It has to generate the audio waveform one piece at a time. I believe it takes WaveNet two minutes to produce one second of audio, so it’s not able to make the audio fast enough to hold an interactive conversation.

Richard: And similarly, we’ve seen applications to colorizing black and white photos, or turning sketches into somewhat photo-realistic images, being able to turn text into images.

Ian: Yeah one thing that really highlights how far we’ve come is that in 2014, one of the big breakthroughs was the ability to take a photo and produce a sentence summarizing what was in the photo. In 2016, we saw different methods for taking a sentence and producing a photo that contains the imagery described by the sentence. It’s much more complicated to go from a few words to a very realistic image containing thousands or millions of pixels than it is to go from the image to the words.

Another thing that was very exciting in 2016 was the use of generative models for drug discovery. Instead of imagining new images, the model could actually imagine new molecules that are intended to have specific medicinal effects.

Richard: And this is pretty exciting because this is being applied towards cancer research, developing potential new cancer treatments.

Ariel: And then there was Google’s language translation program, Google Neural Machine Translation. Can you talk about what that did and why it was a big deal?

Ian: It’s a big deal for two different reasons. First, Google Neural Machine Translation is a lot better than previous approaches to machine translation. Google Neural Machine Translation removes a lot of the human design elements, and just has a neural network figure out what to do.

The other thing that’s really exciting about Google Neural Machine Translation is that the machine translation models have developed what we call an “Interlingua.” It used to be that if you wanted to translate from Japanese to Korean, you had to find a lot of sentences that had been translated from Japanese to Korean before, and then you could train a machine learning model to copy that translation procedure. But now, if you already know how to translate from English to Korean, and you know how to translate from English to Japanese, in the middle, you have Interlingua. So you translate from English to Interlingua and then to Japanese, English to Interlingua and then to Korean. You can also just translate Japanese to Interlingua and Korean to Interlingua and then Interlingua to Japanese or Korean, and you never actually have to get translated sentences from every pair of languages.

Ariel: How can the techniques that are used for language apply elsewhere? How do you anticipate seeing this developed in 2017 and onward?

Richard: So I think what we’ve learned from the approach is that deep learning systems are able to create extremely rich models of the world that can actually express what we can think, which is a pretty exciting milestone. Being able to combine that Interlingua with more structured information about the world is something that a variety of teams are working on — it is a big, open area for the coming years.

Ian: At OpenAI one of our largest projects, Universe, allows a reinforcement learning agent to play many different computer games, and it interacts with these games in the same way that a human does, by sending key presses or mouse strokes to the actual game engine. The same reinforcement learning agent is able to interact with basically anything that a human can interact with on a computer. By having one agent that can do all of these different things we will really exercise our ability to create general artificial intelligence instead of application-specific artificial intelligence. And projects like Google’s Interlingua have shown us that there’s a lot of reason to believe that this will work.

Ariel: What else happened this year that you guys think is important to mention?

Richard: One-shot [learning] is when you see just a little bit of data, potentially just one data point, regarding some new task or some new category, and you’re then able to deduce what that class should look like or what that function should look like in general. So being able to train systems on very little data from just general background knowledge, will be pretty exciting.

Ian: One thing that I’m very excited about is this new area called machine learning security where an attacker can trick a machine learning system into taking the wrong action. For example, we’ve seen that it’s very easy to fool an object-recognition system. We can show it an image that looks a lot like a panda and it gets recognized as being a school bus, or vice versa. It’s actually possible to fool machine learning systems with physical objects. There was a paper called Accessorize to a Crime, that showed that by wearing unusually-colored glasses it’s possible to thwart a face recognition system. And my own collaborators at GoogleBrain and I wrote a paper called Adversarial Examples in the Physical World, where we showed that we can make images that look kind of grainy and noisy, but when viewed through a camera we can control how an object-recognition system will respond to those images.

Ariel: Is there anything else that you thought was either important for 2016 or looking forward to 2017?

Richard: Yeah, looking forward to 2017 I think there will be more focus on unsupervised learning. Most of the world is not annotated by humans. There aren’t little sticky notes on things around the house saying what they are. Being able to process [the world] in a more unsupervised way will unlock a plethora of new applications.

Ian: It will also make AI more democratic. Right now, if you want to use really advanced AI you need to have not only a lot of computers but also a lot of data. That’s part of why it’s mostly very large companies that are competitive in the AI space. If you want to get really good at a task you basically become good at that task by showing the computer a million different examples. In the future, we’ll have AI that can learn much more like a human learns, where just showing it a few examples is enough. Once we have machine learning systems that are able to get the general idea of what’s going on very quickly, in the way that humans do, it won’t be necessary to build these gigantic data sets anymore.

Richard: One application area I think will be important this coming year is automatic detection of fake news, fake audio and fake images and fake video. Some of the applications this past year have actually focused on generating additional frames of video. As those get better, as the photo generation that we talked about earlier gets better, and also as audio templating gets better… I think it was Adobe that demoed what they called PhotoShop for Voice, where you can type something in and select a person, and it will sound like that person saying whatever it is that you typed. So we’ll need ways of detecting that, since this whole concept of fake news is quite at the fore these days.

Ian: It’s worth mentioning that there are other ways of addressing the spread of fake news. Email spam uses a lot of different clues that it can statistically associate with whether people mark the email as spam or not. We can do a lot without needing to advance the underlying AI systems at all.

Ariel: Is there anything that you’re worried about, based on advances that you’ve seen in the last year?

Ian: The employment issue. As we’re able to automate our tasks in the future, how will we make sure that everyone benefits from that automation? And the way that society is structured, right now increasing automation seems to lead to increasing concentration of wealth, and there are winners and losers to every advance. My concern is that automating jobs that are done by millions of people will create very many losers and a small number of winners who really win big.

Richard: I’m also slightly concerned with the speed at which we’re approaching additional generality. It’s extremely cool to see systems be able to do lots of different things, and being able to do tasks that they’ve either seen very little of or none of before. But it raises questions as to when we implement different types of safety techniques. I don’t think that we’re at that point yet, but it raises the issue.

Ariel: To end on a positive note: looking back on what you saw last year, what has you most hopeful for our future?

Ian: I think it’s really great that AI is starting to be used for things like medicine. In the last year we’ve seen a lot of different machine learning algorithms that could exceed human abilities at some tasks, and we’ve also started to see the application of AI to life-saving application areas like designing new medicines. And this makes me very hopeful that we’re going to start seeing superhuman drug design, and other kinds of applications of AI to just really make life better for a lot of people in ways that we would not have been able to do without it.

Richard: Various kinds of tasks that people find to be drudgery within their jobs will be automatable. That will lead them to be open to working on more value-added things with more creativity, and potentially be able to work in more interesting areas of their field or across different fields. I think the future is wide open and it’s really what we make of it, which is exciting in itself.

 

 

Podcast: FLI 2016 – A Year In Review

For FLI, 2016 was a great year, full of our own success, but also great achievements from so many of the organizations we work with. Max, Meia, Anthony, Victoria, Richard, Lucas, David, and Ariel discuss what they were most excited to see in 2016 and what they’re looking forward to in 2017.

AGUIRRE: I’m Anthony Aguirre. I am a professor of physics at UC Santa Cruz, and I’m one of the founders of the Future of Life Institute.

STANLEY: I’m David Stanley, and I’m currently working with FLI as a Project Coordinator/Volunteer Coordinator.

PERRY: My name is Lucas Perry, and I’m a Project Coordinator with the Future of Life Institute.

TEGMARK: I’m Max Tegmark, and I have the fortune to be the President of the Future of Life Institute.

CHITA-TEGMARK: I’m Meia Chita-Tegmark, and I am a co-founder of the Future of Life Institute.

MALLAH: Hi, I’m Richard Mallah. I’m the Director of AI Projects at the Future of Life Institute.

KRAKOVNA: Hi everyone, I am Victoria Krakovna, and I am one of the co-founders of FLI. I’ve recently taken up a position at Google DeepMind working on AI safety.

CONN: And I’m Ariel Conn, the Director of Media and Communications for FLI. 2016 has certainly had its ups and downs, and so at FLI, we count ourselves especially lucky to have had such a successful year. We’ve continued to progress with the field of AI safety research, we’ve made incredible headway with our nuclear weapons efforts, and we’ve worked closely with many amazing groups and individuals. On that last note, much of what we’ve been most excited about throughout 2016 is the great work these other groups in our fields have also accomplished.

Over the last couple of weeks, I’ve sat down with our founders and core team to rehash their highlights from 2016 and also to learn what they’re all most looking forward to as we move into 2017.

To start things off, Max gave a summary of the work that FLI does and why 2016 was such a success.

TEGMARK: What I was most excited by in 2016 was the overall sense that people are taking seriously this idea – that we really need to win this race between the growing power of our technology and the wisdom with which we manage it. Every single way in which 2016 is better than the Stone Age is because of technology, and I’m optimistic that we can create a fantastic future with tech as long as we win this race. But in the past, the way we’ve kept one step ahead is always by learning from mistakes. We invented fire, messed up a bunch of times, and then invented the fire extinguisher. We at the Future of Life Institute feel that that strategy of learning from mistakes is a terrible idea for more powerful tech, like nuclear weapons, artificial intelligence, and things that can really alter the climate of our globe.

Now, in 2016 we saw multiple examples of people trying to plan ahead and to avoid problems with technology instead of just stumbling into them. In April, we had world leaders getting together and signing the Paris Climate Accords. In November, the United Nations General Assembly voted to start negotiations about nuclear weapons next year. The question is whether they should actually ultimately be phased out; whether the nations that don’t have nukes should work towards stigmatizing building more of them – with the idea that 14,000 is way more than anyone needs for deterrence. And – just the other day – the United Nations also decided to start negotiations on the possibility of banning lethal autonomous weapons, which is another arms race that could be very, very destabilizing. And if we keep this positive momentum, I think there’s really good hope that all of these technologies will end up having mainly beneficial uses.

Today, we think of our biologist friends as mainly responsible for the fact that we live longer and healthier lives, and not as those guys who make the bioweapons. We think of chemists as providing us with better materials and new ways of making medicines, not as the people who built chemical weapons and are all responsible for global warming. We think of AI scientists as – I hope, when we look back on them in the future – as people who helped make the world better, rather than the ones who just brought on the AI arms race. And it’s very encouraging to me that as much as people in general – but also the scientists in all these fields – are really stepping up and saying, “Hey, we’re not just going to invent this technology, and then let it be misused. We’re going to take responsibility for making sure that the technology is used beneficially.”

CONN: And beneficial AI is what FLI is primarily known for. So what did the other members have to say about AI safety in 2016? We’ll hear from Anthony first.

AGUIRRE: I would say that what has been great to see over the last year or so is the AI safety and beneficiality research field really growing into an actual research field. When we ran our first conference a couple of years ago, they were these tiny communities who had been thinking about the impact of artificial intelligence in the future and in the long-term future. They weren’t really talking to each other; they weren’t really doing much actual research – there wasn’t funding for it. So, to see in the last few years that transform into something where it takes a massive effort to keep track of all the stuff that’s being done in this space now. All the papers that are coming out, the research groups – you sort of used to be able to just find them all, easily identified. Now, there’s this huge worldwide effort and long lists, and it’s difficult to keep track of. And that’s an awesome problem to have.

As someone who’s not in the field, but sort of watching the dynamics of the research community, that’s what’s been so great to see. A research community that wasn’t there before really has started, and I think in the past year we’re seeing the actual results of that research start to come in. You know, it’s still early days. But it’s starting to come in, and we’re starting to see papers that have been basically created using these research talents and the funding that’s come through the Future of Life Institute. It’s been super gratifying. And seeing that it’s a fairly large amount of money – but fairly small compared to the total amount of research funding in artificial intelligence or other fields – but because it was so funding-starved and talent-starved before, it’s just made an enormous impact. And that’s been nice to see.

CONN: Not surprisingly, Richard was equally excited to see AI safety becoming a field of ever-increasing interest for many AI groups.

MALLAH: I’m most excited by the continued mainstreaming of AI safety research. There are more and more publications coming out by places like DeepMind and Google Brain that have really lent additional credibility to the space, as well as a continued uptake of more and more professors, and postdocs, and grad students from a wide variety of universities entering this space. And, of course, OpenAI has come out with a number of useful papers and resources.

I’m also excited that governments have really realized that this is an important issue. So, while the White House reports have come out recently focusing more on near-term AI safety research, they did note that longer-term concerns like superintelligence are not necessarily unreasonable for later this century. And that they do support – right now – funding safety work that can scale toward the future, which is really exciting. We really need more funding coming into the community for that type of research. Likewise, other governments – like the U.K. and Japan, Germany – have all made very positive statements about AI safety in one form or another. And other governments around the world.

CONN: In addition to seeing so many other groups get involved in AI safety, Victoria was also pleased to see FLI taking part in so many large AI conferences.

KRAKOVNA: I think I’ve been pretty excited to see us involved in these AI safety workshops at major conferences. So on the one hand, our conference in Puerto Rico that we organized ourselves was very influential and helped to kick-start making AI safety more mainstream in the AI community. On the other hand, it felt really good in 2016 to complement that with having events that are actually part of major conferences that were co-organized by a lot of mainstream AI researchers. I think that really was an integral part of the mainstreaming of the field. For example, I was really excited about the Reliable Machine Learning workshop at ICML that we helped to make happen. I think that was something that was quite positively received at the conference, and there was a lot of good AI safety material there.

CONN: And of course, Victoria was also pretty excited about some of the papers that were published this year connected to AI safety, many of which received at least partial funding from FLI.

KRAKOVNA: There were several excellent papers in AI safety this year, addressing core problems in safety for machine learning systems. For example, there was a paper from Stuart Russell’s lab published at NIPS, on cooperative IRL. This is about teaching AI what humans want – how to train an RL algorithm to learn the right reward function that reflects what humans want it to do. DeepMind and FHI published a paper at UAI on safely interruptible agents, that formalizes what it means for an RL agent not to have incentives to avoid shutdown. MIRI made an impressive breakthrough with their paper on logical inductors. I’m super excited about all these great papers coming out, and that our grant program contributed to these results.

CONN: For Meia, the excitement about AI safety went beyond just the technical aspects of artificial intelligence.

CHITA-TEGMARK: I am very excited about the dialogue that FLI has catalyzed – and also engaged in – throughout 2016, and especially regarding the impact of technology on society. My training is in psychology; I’m a psychologist. So I’m very interested in the human aspect of technology development. I’m very excited about questions like, how are new technologies changing us? How ready are we to embrace new technologies? Or how our psychological biases may be clouding our judgement about what we’re creating and the technologies that we’re putting out there. Are these technologies beneficial for our psychological well-being, or are they not?

So it has been extremely interesting for me to see that these questions are being asked more and more, especially by artificial intelligence developers and also researchers. I think it’s so exciting to be creating technologies that really force us to grapple with some of the most fundamental aspects, I would say, of our own psychological makeup. For example, our ethical values, our sense of purpose, our well-being, maybe our biases and shortsightedness and shortcomings as biological human beings. So I’m definitely very excited about how the conversation regarding technology – and especially artificial intelligence – has evolved over the last year. I like the way it has expanded to capture this human element, which I find so important. But I’m also so happy to feel that FLI has been an important contributor to this conversation.

CONN: Meanwhile, as Max described earlier, FLI has also gotten much more involved in decreasing the risk of nuclear weapons, and Lucas helped spearhead one of our greatest accomplishments of the year.

PERRY: One of the things that I was most excited about was our success with our divestment campaign. After a few months, we had great success in our own local Boston area with helping the City of Cambridge to divest its $1 billion portfolio from nuclear weapon producing companies. And we see this as a really big and important victory within our campaign to help institutions, persons, and universities to divest from nuclear weapons producing companies.

CONN: And in order to truly be effective we need to reach an international audience, which is something Dave has been happy to see grow this year.

STANLEY: I’m mainly excited about – at least, in my work – the increasing involvement and response we’ve had from the international community in terms of reaching out about these issues. I think it’s pretty important that we engage the international community more, and not just academics. Because these issues – things like nuclear weapons and the increasing capabilities of artificial intelligence – really will affect everybody. And they seem to be really underrepresented in mainstream media coverage as well.

So far, we’ve had pretty good responses just in terms of volunteers from many different countries around the world being interested in getting involved to help raise awareness in their respective communities, either through helping develop apps for us, or translation, or promoting just through social media these ideas in their little communities.

CONN: Many FLI members also participated in both local and global events and projects, like the following we’re about  to hear from Victoria, Richard, Lucas and Meia.

KRAKOVNA: The EAGX Oxford Conference was a fairly large conference. It was very well organized, and we had a panel there with Demis Hassabis, Nate Soares from MIRI, Murray Shanahan from Imperial, Toby Ord from FHI, and myself. I feel like overall, that conference did a good job of, for example, connecting the local EA community with the people at DeepMind, who are really thinking about AI safety concerns like Demis and also Sean Legassick, who also gave a talk about the ethics and impacts side of things. So I feel like that conference overall did a good job of connecting people who are thinking about these sorts of issues, which I think is always a great thing.  

MALLAH: I was involved in this endeavor with IEEE regarding autonomy and ethics in autonomous systems, sort of representing FLI’s positions on things like autonomous weapons and long-term AI safety. One thing that came out this year – just a few days ago, actually, due to this work from IEEE – is that the UN actually took the report pretty seriously, and it may have influenced their decision to take up the issue of autonomous weapons formally next year. That’s kind of heartening.

PERRY: A few different things that I really enjoyed doing were giving a few different talks at Duke and Boston College, and a local effective altruism conference. I’m also really excited about all the progress we’re making on our nuclear divestment application. So this is an application that will allow anyone to search their mutual fund and see whether or not their mutual funds have direct or indirect holdings in nuclear weapons-producing companies.

CHITA-TEGMARK:  So, a wonderful moment for me was at the conference organized by Yann LeCun in New York at NYU, when Daniel Kahneman, one of my thinker-heroes, asked a very important question that really left the whole audience in silence. He asked, “Does this make you happy? Would AI make you happy? Would the development of a human-level artificial intelligence make you happy?” I think that was one of the defining moments, and I was very happy to participate in this conference.

Later on, David Chalmers, another one of my thinker-heroes – this time, not the psychologist but the philosopher – organized another conference, again at NYU, trying to bring philosophers into this very important conversation about the development of artificial intelligence. And again, I felt there too, that FLI was able to contribute and bring in this perspective of the social sciences on this issue.

CONN: Now, with 2016 coming to an end, it’s time to turn our sites to 2017, and FLI is excited for this new year to be even more productive and beneficial.

TEGMARK: We at the Future of Life Institute are planning to focus primarily on artificial intelligence, and on reducing the risk of accidental nuclear war in various ways. We’re kicking off by having an international conference on artificial intelligence, and then we want to continue throughout the year providing really high-quality and easily accessible information on all these key topics, to help inform on what happens with climate change, with nuclear weapons, with lethal autonomous weapons, and so on.

And looking ahead here, I think it’s important right now – especially since a lot of people are very stressed out about the political situation in the world, about terrorism, and so on – to not ignore the positive trends and the glimmers of hope we can see as well.

CONN: As optimistic as FLI members are about 2017, we’re all also especially hopeful and curious to see what will happen with continued AI safety research.

AGUIRRE: I would say I’m looking forward to seeing in the next year more of the research that comes out, and really sort of delving into it myself, and understanding how the field of artificial intelligence and artificial intelligence safety is developing. And I’m very interested in this from the forecast and prediction standpoint.

I’m interested in trying to draw some of the AI community into really understanding how artificial intelligence is unfolding – in the short term and the medium term – as a way to understand, how long do we have? Is it, you know, if it’s really infinity, then let’s not worry about that so much, and spend a little bit more on nuclear weapons and global warming and biotech, because those are definitely happening. If human-level AI were 8 years away… honestly, I think we should be freaking out right now. And most people don’t believe that, I think most people are in the middle it seems, of thirty years or fifty years or something, which feels kind of comfortable. Although it’s not that long, really, on the big scheme of things. But I think it’s quite important to know now, which is it? How fast are these things, how long do we really have to think about all of the issues that FLI has been thinking about in AI? How long do we have before most jobs in industry and manufacturing are replaceable by a robot being slotted in for a human? That may be 5 years, it may be fifteen… It’s probably not fifty years at all. And having a good forecast on those good short-term questions I think also tells us what sort of things we have to be thinking about now.

And I’m interested in seeing how this massive AI safety community that’s started develops. It’s amazing to see centers kind of popping up like mushrooms after a rain all over and thinking about artificial intelligence safety. This partnership on AI between Google and Facebook and a number of other large companies getting started. So to see how those different individual centers will develop and how they interact with each other. Is there an overall consensus on where things should go? Or is it a bunch of different organizations doing their own thing? Where will governments come in on all of this? I think it will be interesting times. So I look forward to seeing what happens, and I will reserve judgement in terms of my optimism.

KRAKOVNA: I’m really looking forward to AI safety becoming even more mainstream, and even more of the really good researchers in AI giving it serious thought. Something that happened in the past year that I was really excited about, that I think is also pointing in this direction, is the research agenda that came out of Google Brain called “Concrete Problems in AI Safety.” And I think I’m looking forward to more things like that happening, where AI safety becomes sufficiently mainstream that people who are working in AI just feel inspired to do things like that and just think from their own perspectives: what are the important problems to solve in AI safety? And work on them.

I’m a believer in the portfolio approach with regards to AI safety research, where I think we need a lot of different research teams approaching the problems from different angles and making different assumptions, and hopefully some of them will make the right assumption. I think we are really moving in the direction in terms of more people working on these problems, and coming up with different ideas. And I look forward to seeing more of that in 2017. I think FLI can also help continue to make this happen.

MALLAH: So, we’re in the process of fostering additional collaboration among people in the AI safety space. And we will have more announcements about this early next year. We’re also working on resources to help people better visualize and better understand the space of AI safety work, and the opportunities there and the work that has been done. Because it’s actually quite a lot.

I’m also pretty excited about fostering continued theoretical work and practical work in making AI more robust and beneficial. The work in value alignment, for instance, is not something we see supported in mainstream AI research. And this is something that is pretty crucial to the way that advanced AIs will need to function. It won’t be very explicit instructions to them; they’ll have to be making decision based on what they think is right. And what is right? It’s something that… or even structuring the way to think about what is right requires some more research.

STANLEY: We’ve had pretty good success at FLI in the past few years helping to legitimize the field of AI safety. And I think it’s going to be important because AI is playing a large role in industry and there’s a lot of companies working on this, and not just in the US. So I think increasing international awareness about AI safety is going to be really important.

CHITA-TEGMARK: I believe that the AI community has raised some very important questions in 2016 regarding the impact of AI on society. I feel like 2017 should be the year to make progress on these questions, and actually research them and have some answers to them. For this, I think we need more social scientists – among people from other disciplines – to join this effort of really systematically investigating what would be the optimal impact of AI on people. I hope that in 2017 we will have more research initiatives, that we will attempt to systematically study other burning questions regarding the impact of AI on society. Some examples are: how can we ensure the psychological well-being for people while AI creates lots of displacement on the job market as many people predict. How do we optimize engagement with technology, and withdrawal from it also? Will some people be left behind, like the elderly or the economically disadvantaged? How will this affect them, and how will this affect society at large?

What about withdrawal from technology? What about satisfying our need for privacy? Will we be able to do that, or is the price of having more and more customized technologies and more and more personalization of the technologies we engage with… will that mean that we will have no privacy anymore, or that our expectations of privacy will be very seriously violated? I think these are some very important questions that I would love to get some answers to. And my wish, and also my resolution, for 2017 is to see more progress on these questions, and to hopefully also be part of this work and answering them.

PERRY: In 2017 I’m very interested in pursuing the landscape of different policy and principle recommendations from different groups regarding artificial intelligence. I’m also looking forward to expanding out nuclear divestment campaign by trying to introduce divestment to new universities, institutions, communities, and cities.

CONN: In fact, some experts believe nuclear weapons pose a greater threat now than at any time during our history.

TEGMARK: I personally feel that the greatest threat to the world in 2017 is one that the newspapers almost never write about. It’s not terrorist attacks, for example. It’s the small but horrible risk that the U.S. and Russia for some stupid reason get into an accidental nuclear war against each other. We have 14,000 nuclear weapons, and this war has almost happened many, many times. So, actually what’s quite remarkable and really gives a glimmer of hope is that – however people may feel about Putin and Trump – the fact is they are both signaling strongly that they are eager to get along better. And if that actually pans out and they manage to make some serious progress in nuclear arms reduction, that would make 2017 the best year for nuclear weapons we’ve had in a long, long time, reversing this trend of ever greater risks with ever more lethal weapons.

CONN: Some FLI members are also looking beyond nuclear weapons and artificial intelligence, as I learned when I asked Dave about other goals he hopes to accomplish with FLI this year.

STANLEY: Definitely having the volunteer team – particularly the international volunteers – continue to grow, and then scale things up. Right now, we have a fairly committed core of people who are helping out, and we think that they can start recruiting more people to help out in their little communities, and really making this stuff accessible. Not just to academics, but to everybody. And that’s also reflected in the types of people we have working for us as volunteers. They’re not just academics. We have programmers, linguists, people having just high school degrees all the way up to Ph.D.’s, so I think it’s pretty good that this varied group of people can get involved and contribute, and also reach out to other people they can relate to.

CONN: In addition to getting more people involved, Meia also pointed out that one of the best ways we can help ensure a positive future is to continue to offer people more informative content.

CHITA-TEGMARK: Another thing that I’m very excited about regarding our work here at the Future of Life Institute is this mission of empowering people to information. I think information is very powerful and can change the way people approach things: they can change their beliefs, their attitudes, and their behaviors as well. And by creating ways in which information can be readily distributed to the people, and with which they can engage very easily, I hope that we can create changes. For example, we’ve had a series of different apps regarding nuclear weapons that I think have contributed a lot to peoples knowledge and has brought this issue to the forefront of their thinking.

CONN: Yet as important as it is to highlight the existential risks we must address to keep humanity safe, perhaps it’s equally important to draw attention to the incredible hope we have for the future if we can solve these problems. Which is something both Richard and Lucas brought up for 2017.

MALLAH: I’m excited about trying to foster more positive visions of the future, so focusing on existential hope aspects of the future. Which are kind of the flip side of existential risks. So we’re looking at various ways of getting people to be creative about understanding some of the possibilities, and how to differentiate the paths between the risks and the benefits.

PERRY: Yeah, I’m also interested in creating and generating a lot more content that has to do with existential hope. Given the current global political climate, it’s all the more important to focus on how we can make the world better.

CONN: And on that note, I want to mention one of the most amazing things I discovered this past year. It had nothing to do with technology, and everything to do with people. Since starting at FLI, I’ve met countless individuals who are dedicating their lives to trying to make the world a better place. We may have a lot of problems to solve, but with so many groups focusing solely on solving them, I’m far more hopeful for the future. There are truly too many individuals that I’ve met this year to name them all, so instead, I’d like to provide a rather long list of groups and organizations I’ve had the pleasure to work with this year. A link to each group can be found at futureoflife.org/2016, and I encourage you to visit them all to learn more about the wonderful work they’re doing. In no particular order, they are:

Machine Intelligence Research Institute

Future of Humanity Institute

Global Catastrophic Risk Institute

Center for the Study of Existential Risk

Ploughshares Fund

Bulletin of Atomic Scientists

Open Philanthropy Project

Union of Concerned Scientists

The William Perry Project

ReThink Media

Don’t Bank on the Bomb

Federation of American Scientists

Massachusetts Peace Action

IEEE (Institute for Electrical and Electronics Engineers)

Center for Human-Compatible Artificial Intelligence

Center for Effective Altruism

Center for Applied Rationality

Foresight Institute

Leverhulme Center for the Future of Intelligence

Global Priorities Project

Association for the Advancement of Artificial Intelligence

International Joint Conference on Artificial Intelligence

Partnership on AI

The White House Office of Science and Technology Policy

The Future Society at Harvard Kennedy School

 

I couldn’t be more excited to see what 2017 holds in store for us, and all of us at FLI look forward to doing all we can to help create a safe and beneficial future for everyone. But to end on an even more optimistic note, I turn back to Max.

TEGMARK: Finally, I’d like – because I spend a lot of my time thinking about our universe – to remind everybody that we shouldn’t just be focused on the next election cycle. We have not decades, but billions of years of potentially awesome future for life, on Earth and far beyond. And it’s so important to not let ourselves get so distracted by our everyday little frustrations that we lose sight of these incredible opportunities that we all stand to gain from if we can get along, and focus, and collaborate, and use technology for good.

Silo Busting in AI Research

Artificial intelligence may seem like a computer science project, but if it’s going to successfully integrate with society, then social scientists must be more involved.

Developing an intelligent machine is not merely a problem of modifying algorithms in a lab. These machines must be aligned with human values, and this requires a deep understanding of ethics and the social consequences of deploying intelligent machines.

Getting people with a variety of backgrounds together seems logical enough in theory, but in practice, what happens when computer scientists, AI developers, economists, philosophers, and psychologists try to discuss AI issues? Do any of them even speak the same language?

Social scientists and computer scientists will come at AI problems from very different directions. And if they collaborate, everybody wins. Social scientists can learn about the complex tools and algorithms used in computer science labs, and computer scientists can become more attuned to the social and ethical implications of advanced AI.

Through transdisciplinary learning, both fields will be better equipped to handle the challenges of developing AI, and society as a whole will be safer.

 

Silo Busting

Too often, researchers focus on their narrow area of expertise, rarely reaching out to experts in other fields to solve common problems. AI is no different, with thick walls – sometimes literally – separating the social sciences from the computer sciences. This process of breaking down walls between research fields is often called silo-busting.

If AI researchers largely operate in silos, they may lose opportunities to learn from other perspectives and collaborate with potential colleagues. Scientists might miss gaps in their research or reproduce work already completed by others, because they were secluded away in their silo. This can significantly hamper the development of value-aligned AI.

To bust these silos, Wendell Wallach organized workshops to facilitate knowledge-sharing among leading computer and social scientists. Wallach, a consultant, ethicist, and scholar at Yale University’s Interdisciplinary Center for Bioethics, holds these workshops at The Hastings Center, where he is a senior advisor.

With co-chairs Gary Marchant, Stuart Russell, and Bart Selman, Wallach held the first workshop in April 2016. “The first workshop was very much about exposing people to what experts in all of these different fields were thinking about,” Wallach explains. “My intention was just to put all of these people in a room and hopefully they’d see that they weren’t all reinventing the wheel, and recognize that there were other people who were engaged in similar projects.”

The workshop intentionally brought together experts from a variety of viewpoints, including engineering ethics, philosophy, and resilience engineering, as well as participants from the Institute of Electrical and Electronics Engineers (IEEE), the Office of Naval Research, and the World Economic Forum (WEF). Wallach recounts, “some were very interested in how you implement sensitivity to moral considerations in AI computationally, and others were more interested in how AI changes the societal context.”

Other participants studied how the engineers of these systems may be susceptible to harmful cognitive biases and conflicts of interest, while still others focused on governance issues surrounding AI. Each of these viewpoints is necessary for developing beneficial AI, and The Hastings Center’s workshop gave participants the opportunity to learn from and teach each other.

But silo-busting is not easy. Wallach explains, “everybody has their own goals, their own projects, their own intentions, and it’s hard to hear someone say, ‘maybe you’re being a little naïve about this.’” When researchers operate exclusively in silos, “it’s almost impossible to understand how people outside of those silos did what they did,” he adds.

The intention of the first workshop was not to develop concrete strategies or proposals, but rather to open researchers’ minds to the broad challenges of developing AI with human values. “My suspicion is, the most valuable things that came out of this workshop would be hard to quantify,” Wallach clarifies. “It’s more like people’s minds were being stretched and opened. That was, for me, what this was primarily about.”

The workshop did yield some tangible results. For example, Marchant and Wallach introduced a pilot project for the international governance of AI, and nearly everyone at the workshop agreed to work on it. Since then, the IEEE, the International Committee of the Red Cross, the UN, the World Economic Forum, and other institutions have agreed to become active partners with The Hastings Center in building global infrastructure to ensure that AI and Robotics are beneficial.

This transdisciplinary cooperation is a promising sign that Wallach’s efforts are succeeding in strengthening the global response to AI challenges.

 

Value Alignment

Wallach and his co-chairs held a second workshop at the end of October. The participants were mostly scientists, but also included social theorists, a legal scholar, philosophers, and ethicists. The overall goal remained – to bust AI silos and facilitate transdisciplinary cooperation – but this workshop had a narrower focus.

“We made it more about value alignment and machine ethics,” he explains. “The tension in the room was between those who thought the problem [of value alignment] was imminently solvable and those who were deeply skeptical about solving the problem at all.”

In general, Wallach observed that “the social scientists and philosophers tend to overplay the difficulties [of creating AI with full value alignment] and computer scientists tend to underplay the difficulties.”

Wallach believes that while computer scientists will build the algorithms and utility functions for AI, they will need input from social scientists to ensure value alignment. “If a utility function represents 100,000 inputs, social theorists will help the AI researchers understand what those 100,000 inputs are,” he explains. “The AI researchers might be able to come up with 50,000-60,000 on their own, but they’re suddenly going to realize that people who have thought much more deeply about applied ethics are perhaps sensitive to things that they never considered.”

“I’m hoping that enough of [these researchers] learn each other’s language and how to communicate with each other, that they’ll recognize the value they can get from collaborating together,” he says. “I think I see evidence of that beginning to take place.”

 

Moving Forward

Developing value-aligned AI is a monumental task with existential risks. Experts from various perspectives must be willing to learn from each other and adapt their understanding of the issue.

In this spirit, The Hastings Center is leading the charge to bring the various AI silos together. After two successful events that resulted in promising partnerships, Wallach and his co-chairs will hold their third workshop in Spring 2018. And while these workshops are a small effort to facilitate transdisciplinary cooperation on AI, Wallach is hopeful.

“It’s a small group,” he admits, “but it’s people who are leaders in these various fields, so hopefully that permeates through the whole field, on both sides.”

This article is part of a Future of Life series on the AI safety research grants, which were funded by generous donations from Elon Musk and the Open Philanthropy Project.

Artificial Intelligence and the King Midas Problem

Value alignment. It’s a phrase that often pops up in discussions about the safety and ethics of artificial intelligence. How can scientists create AI with goals and values that align with those of the people it interacts with?

Very simple robots with very constrained tasks do not need goals or values at all. Although the Roomba’s designers know you want a clean floor, Roomba doesn’t: it simply executes a procedure that the Roomba’s designers predict will work—most of the time. If your kitten leaves a messy pile on the carpet, Roomba will dutifully smear it all over the living room. If we keep programming smarter and smarter robots, then by the late 2020s, you may be able to ask your wonderful domestic robot to cook a tasty, high-protein dinner. But if you forgot to buy any meat, you may come home to a hot meal but find the aforementioned cat has mysteriously vanished. The robot, designed for chores, doesn’t understand that the sentimental value of the cat exceeds its nutritional value.

AI and King Midas

Stuart Russell, a renowned AI researcher, compares the challenge of defining a robot’s objective to the King Midas myth. “The robot,” says Russell, “has some objective and pursues it brilliantly to the destruction of mankind. And it’s because it’s the wrong objective. It’s the old King Midas problem.”

This is one of the big problems in AI safety that Russell is trying to solve. “We’ve got to get the right objective,” he explains, “and since we don’t seem to know how to program it, the right answer seems to be that the robot should learn – from interacting with and watching humans – what it is humans care about.”

Russell works from the assumption that the robot will solve whatever formal problem we define. Rather than assuming that the robot should optimize a given objective, Russell defines the problem as a two-player game (“game” as used by economists, meaning a decision problem with multiple agents) called cooperative inverse reinforcement learning (CIRL).

A CIRL game includes a person and a robot: the robot’s only purpose is to make the person happy, but it doesn’t know what the person wants. Fortunately, it can learn more about what the person wants by observing her behavior. For example, if a robot observed the human’s morning routine, it should discover how important coffee is—not to itself, of course (we don’t want robots drinking coffee), but to the human. Then, it will make coffee for the person without being asked.

What about the example of King Midas, who thought he wanted as much gold as possible? How does the approach deal with someone stating their wishes incorrectly?

“We don’t have to know what we want. That’s the whole point. It’s hard to know what we want, and what we ask for is evidence for what we really want, but it’s not ground truth,” says Russell.

The robot should learn what people want by watching their actions. Then, if someone asks for something that’s technically incorrect, the robot will understand their underlying wish. Russell adds, “The robot—if it had been there—would have told Midas that he didn’t really want everything turned to gold, maybe just a few choice objects that he might point at from time to time and say the magic word.”

AI Off-Switch

Russell and his Berkeley colleagues also recently announced further progress toward ensuring safe AI, with a paper on ensuring an AI’s off-switch is always accessible. An off-switch is often suggested to guarantee a robot doesn’t harm humans – if the robot goes bad, just turn it off! But what if the AI anticipates that and disables its off switch?

Russell and his team propose solving this by providing a level of uncertainty in the robot’s objective. They created another CIRL game in which the robot chooses to announce its intentions, which gives the human a chance to switch it off, or to plunge ahead with its plan, bypassing human oversight.

They showed that a robot with a fixed objective will take the second choice and disable its off-switch. But if there is uncertainty about the objective, the robot will reason that a human decision to switch the robot off would imply the announced intention is undesirable. Because its goal is to make the human happy, the robot accepts being switched off. The robot has a positive incentive not to disable the off-switch, and that incentive is directly tied to the robot’s uncertainty about the human’s true objective.

Ensuring AI Safety

In addition to his research, Russell is also one of the most vocal and active AI safety researchers concerned with ensuring a stronger public understanding of the potential issues surrounding AI development.

He recently co-authored a rebuttal to an article in the MIT Technology Review, which claimed that real AI scientists weren’t worried about the existential threat of AI. Russell and his co-author summed up why it’s better to be cautious and careful than just assume all will turn out for the best:

“Our experience with Chernobyl suggests it may be unwise to claim that a powerful technology entails no risks. It may also be unwise to claim that a powerful technology will never come to fruition. On September 11, 1933, Lord Rutherford, perhaps the world’s most eminent nuclear physicist, described the prospect of extracting energy from atoms as nothing but “moonshine.” Less than 24 hours later, Leo Szilard invented the neutron-induced nuclear chain reaction; detailed designs for nuclear reactors and nuclear weapons followed a few years later. Surely it is better to anticipate human ingenuity than to underestimate it, better to acknowledge the risks than to deny them. … [T]he risk [of AI] arises from the unpredictability and potential irreversibility of deploying an optimization process more intelligent than the humans who specified its objectives.”

This summer, Russell received a grant of over $5.5 million from the Open Philanthropy Project for a new research center, the Center for Human-Compatible Artificial Intelligence, in Berkeley. Among the primary objectives of the Center will be to study this problem of value alignment, to continue his efforts toward provably beneficial AI, and to ensure we don’t make the same mistakes as King Midas.

“Look,” he says, “if you were King Midas, would you want your robot to say, ‘Everything turns to gold? OK, boss, you got it.’ No! You’d want it to say, ‘Are you sure? Including your food, drink, and relatives? I’m pretty sure you wouldn’t like that. How about this: you point to something and say ‘Abracadabra Aurificio’ or something, and then I’ll turn it to gold, OK?’”

This article is part of a Future of Life series on the AI safety research grants, which were funded by generous donations from Elon Musk and the Open Philanthropy Project.

Autonomous Weapons: an Interview With the Experts

FLI’s Ariel Conn recently spoke with Heather Roff and Peter Asaro about autonomous weapons. Roff, a research scientist at The Global Security Initiative at Arizona State University and a senior research fellow at the University of Oxford, recently compiled an international database of weapons systems that exhibit some level of autonomous capabilities. Asaro is a philosopher of science, technology, and media at The New School in New York City. He looks at fundamental questions of responsibility and liability with all autonomous systems, but he’s also the Co-Founder and Vice-Chair of the International Committee for Robot Arms Control and a he’s a Spokesperson for the Campaign to Stop Killer Robots.

The following interview has been edited for brevity, but you can read it in its entirety here or listen to it above.

ARIEL: Dr. Roff, I’d like to start with you. With regard to the database, what prompted you to create it, what information does it provide, how can we use it?

ROFF: The main impetus behind the creation of the database [was] a feeling that the same autonomous or automated weapons systems were brought out in discussions over and over and over again. It made it seem like there wasn’t anything else to worry about. So I created a database of about 250 autonomous systems that are currently deployed [from] Russia, China, the United States, France, and Germany. I code them along a series of about 20 different variables: from automatic target recognition [to] the ability to navigate [to] acquisition capabilities [etc.].

It’s allowing everyone to understand that autonomy isn’t just binary. It’s not a yes or a no. Not many people in the world have a good understanding of what modern militaries fight with, and how they fight.

ARIEL: And Dr. Asaro, your research is about liability. How is it different for autonomous weapons versus a human overseeing a drone that just accidentally fires on the wrong target.

ASARO: My work looks at autonomous weapons and other kinds of autonomous systems and the interface of the ethical and legal aspects. Specifically, questions about the ethics of killing, and the legal requirements under international law for killing in armed conflict. These kind of autonomous systems are not really legal and moral agents in the way that humans are, and so delegating the authority to kill to them is unjustifiable.

One aspect of accountability is, if a mistake is made, holding people to account for that mistake. There’s a feedback mechanism to prevent that error occurring in the future. There’s also a justice element, which could be attributive justice, in which you try to make up for loss. Other forms of accountability look at punishment itself. When you have autonomous systems, you can’t really punish the system. More importantly, if nobody really intended the effect that the system brought about then it becomes very difficult to hold anybody accountable for the actions of the system. The debate — it’s really kind of framed around this question of the accountability gap.

ARIEL: One of the things we hear a lot in the news is about always keeping a human in the loop. How does that play into the idea of liability? And realistically, what does it mean?

ROFF: I actually think this is just a really unhelpful heuristic. It’s hindering our ability to think about what’s potentially risky or dangerous or might produce unintended consequences. So here’s an example: the UK’s Ministry of Defense calls this the Empty Hangar Problem. It’s very unlikely that they’re going to walk down to an airplane hangar, look in, and be like, “Hey! Where’s the airplane? Oh, it’s decided to go to war today.” That’s just not going to happen.

These systems are always going to be used by humans, and humans are going to decide to use them. A better way to think about this is in terms of task allocation. What is the scope of the task, and how much information and control does the human have before deploying that system to execute? If there is a lot of time, space, and distance between the time the decision is made to field it and then the application of force, there’s more time for things to change on the ground, and there’s more time for the human to basically [say] they didn’t intend for this to happen.

ASARO: If self-driving cars start running people over, people will sue the manufacturer. But there’s no mechanisms in international law for the victims of bombs and missiles and potentially autonomous weapons to sue the manufacturers of those systems. That just doesn’t happen. So there’s no incentives for companies that manufacture those [weapons] to improve safety and performance.

ARIEL: Dr. Asaro, we’ve briefly mentioned definitional problems of autonomous weapons — how does the liability play in there?

ASARO: The law of international armed conflict is pretty clear that humans are the ones that make the decisions, especially about a targeting decision or the taking of a human life in armed conflict. This question of having a system that could range over many miles and many days and select targets on its own is where things are problematic. Part of the definition is: how do you figure out exactly what constitutes a targeting decision, and how do you ensure that a human is making that decision? That’s the direction the discussion at the UN is going as well. Instead of trying to define what’s an autonomous system, what we focus on is the targeting decision and firing decisions of weapons for individual attacks. What we want to acquire is meaningful human control over those decisions.

ARIEL: Dr. Roff, you were working on the idea of meaningful human control, as well. Can you talk about that?

ROFF: If [a commander] fields a weapon that can go from attack to attack without checking back with her, then the weapon is making the proportionality calculation, and she [has] delegated her authority and her obligation to a machine. That is prohibited under IHL, and I would say is also morally prohibited. You can’t offload your moral obligation to a nonmoral agent. So that’s where our work on meaningful human control is: a human commander has a moral obligation to undertake precaution and proportionality in each attack.

ARIEL: Is there anything else you think is important to add?

ROFF: We still have limitations AI. We have really great applications of AI, and we have blind. It would be really incumbent on the AI community to be vocal about where they think there are capacities and capabilities that could be reliably and predictably deployed on such systems. If they don’t think that those technologies or applications could be reliably and predictably deployed, then they need to stand up and say as much.

ASARO: We’re not trying to prohibit autonomous operations of different kind of systems or the development and application of artificial intelligence for a wide range of civilian and military applications. But there are certain applications, specifically the lethal ones, that have higher standards of moral and legal requirements that need to be met.

 

Complex AI Systems Explain Their Actions

cobots_mauela_veloso

In the future, service robots equipped with artificial intelligence (AI) are bound to be a common sight. These bots will help people navigate crowded airports, serve meals, or even schedule meetings.

As these AI systems become more integrated into daily life, it is vital to find an efficient way to communicate with them. It is obviously more natural for a human to speak in plain language rather than a string of code. Further, as the relationship between humans and robots grows, it will be necessary to engage in conversations, rather than just give orders.

This human-robot interaction is what Manuela M. Veloso’s research is all about. Veloso, a professor at Carnegie Mellon University, has focused her research on CoBots, autonomous indoor mobile service robots which transport items, guide visitors to building locations, and traverse the halls and elevators. The CoBot robots have been successfully autonomously navigating for several years now, and have traveled more than 1,000km. These accomplishments have enabled the research team to pursue a new direction, focusing now on novel human-robot interaction.

“If you really want these autonomous robots to be in the presence of humans and interacting with humans, and being capable of benefiting humans, they need to be able to talk with humans” Veloso says.

 

Communicating With CoBots

Veloso’s CoBots are capable of autonomous localization and navigation in the Gates-Hillman Center using WiFi, LIDAR, and/or a Kinect sensor (yes, the same type used for video games).

The robots navigate by detecting walls as planes, which they match to the known maps of the building. Other objects, including people, are detected as obstacles, so navigation is safe and robust. Overall, the CoBots are good navigators and are quite consistent in their motion. In fact, the team noticed the robots could wear down the carpet as they traveled the same path numerous times.

Because the robots are autonomous, and therefore capable of making their own decisions, they are out of sight for large amounts of time while they navigate the multi-floor buildings.

The research team began to wonder about this unaccounted time. How were the robots perceiving the environment and reaching their goals? How was the trip? What did they plan to do next?

“In the future, I think that incrementally we may want to query these systems on why they made some choices or why they are making some recommendations,” explains Veloso.

The research team is currently working on the question of why the CoBots took the route they did while autonomous. The team wanted to give the robots the ability to record their experiences and then transform the data about their routes into natural language. In this way, the bots could communicate with humans and reveal their choices and hopefully the rationale behind their decisions.

 

Levels of Explanation

The “internals” underlying the functions of any autonomous robots are completely based on numerical computations, and not natural language. For example, the CoBot robots in particular compute the distance to walls, assigning velocities to their motors to enable the motion to specific map coordinates.

Asking an autonomous robot for a non-numerical explanation is complex, says Veloso. Furthermore, the answer can be provided in many potential levels of detail.

“We define what we call the ‘verbalization space’ in which this translation into language can happen with different levels of detail, with different levels of locality, with different levels of specificity.”

For example, if a developer is asking a robot to detail their journey, they might expect a lengthy retelling, with details that include battery levels. But a random visitor might just want to know how long it takes to get from one office to another.

Therefore, the research is not just about the translation from data to language, but also the acknowledgment that the robots need to explain things with more or less detail. If a human were to ask for more detail, the request triggers CoBot “to move” into a more detailed point in the verbalization space.

“We are trying to understand how to empower the robots to be more trustable through these explanations, as they attend to what the humans want to know,” says Veloso. The ability to generate explanations, in particular at multiple levels of detail, will be especially important in the future, as the AI systems will work with more complex decisions. Humans could have a more difficult time inferring the AI’s reasoning. Therefore, the bot will need to be more transparent.

For example, if you go to a doctor’s office and the AI there makes a recommendation about your health, you may want to know why it came to this decision, or why it recommended one medication over another.

Currently, Veloso’s research focuses on getting the robots to generate these explanations in plain language. The next step will be to have the robots incorporate natural language when humans provide them with feedback. “[The CoBot] could say, ‘I came from that way,’ and you could say, ‘well next time, please come through the other way,’” explains Veloso.

These sorts of corrections could be programmed into the code, but Veloso believes that “trustability” in AI systems will benefit from our ability to dialogue, query, and correct their autonomy. She and her team aim at contributing to a multi-robot, multi-human symbiotic relationship, in which robots and humans coordinate and cooperate as a function of their limitations and strengths.

“What we’re working on is to really empower people – a random person who meets a robot – to still be able to ask things about the robot in natural language,” she says.

In the future, when we will have more and more AI systems that are able to perceive the world, make decisions, and support human decision-making, the ability to engage in these types of conversations will be essential­­.

This article is part of a Future of Life series on the AI safety research grants, which were funded by generous donations from Elon Musk and the Open Philanthropy Project.

Who is Responsible for Autonomous Weapons?

Consider the following wartime scenario: Hoping to spare the lives of soldiers, a country deploys an autonomous weapon to wipe out an enemy force. This robot has demonstrated military capabilities that far exceed even the best soldiers, but when it hits the ground, it gets confused. It can’t distinguish the civilians from the enemy soldiers and begins taking innocent lives. The military generals desperately try to stop the robot, but by the time they succeed it has already killed dozens.

Who is responsible for this atrocity? Is it the commanders who deployed the robot, the designers and manufacturers of the robot, or the robot itself?

 

Liability: Autonomous Systems

As artificial intelligence improves, governments may turn to autonomous weapons — like military robots — in order to gain the upper hand in armed conflict. These weapons can navigate environments on their own and make their own decisions about who to kill and who to spare. While the example above may never occur, unintended harm is inevitable. Considering these scenarios helps formulate important questions that governments and researchers must jointly consider, namely:

How do we hold human beings accountable for the actions of autonomous systems? And how is justice served when the killer is essentially a computer?

As it turns out, there is no straightforward answer to this dilemma. When a human soldier commits an atrocity and kills innocent civilians, that soldier is held accountable. But when autonomous weapons do the killing, it’s difficult to blame them for their mistakes.

An autonomous weapon’s “decision” to murder innocent civilians is like a computer’s “decision” to freeze the screen and delete your unsaved project. Frustrating as a frozen computer may be, people rarely think the computer intended to complicate their lives.

Intention must be demonstrated to prosecute someone for a war crime, and while autonomous weapons may demonstrate outward signs of decision-making and intention, they still run on a code that’s just as impersonal as the code that glitches and freezes a computer screen. Like computers, these systems are not legal or moral agents, and it’s not clear how to hold them accountable — or if they can be held accountable — for their mistakes.

So who assumes the blame when autonomous weapons take innocent lives? Should they even be allowed to kill at all?

 

Liability: from Self-Driving Cars to Autonomous Weapons

Peter Asaro, a philosopher of science, technology, and media at The New School in New York City, has been working on addressing these fundamental questions of responsibility and liability with all autonomous systems, not just weapons. By exploring fundamental concepts of autonomy, agency, and liability, he intends to develop legal approaches for regulating the use of autonomous systems and the harm they cause.

At a recent conference on the Ethics of Artificial Intelligence, Asaro discussed the liability issues surrounding the application of AI to weapons systems. He explained, “AI poses threats to international law itself — to the norms and standards that we rely on to hold people accountable for [decisions, and to] hold states accountable for military interventions — as [people are] able to blame systems for malfunctioning instead of taking responsibility for their decisions.”

The legal system will need to reconsider who is held liable to ensure that justice is served when an accident happens. Asaro argues that the moral and legal issues surrounding autonomous weapons are much different than the issues surrounding other autonomous machines, such as self-driving cars.

Though researchers still expect the occasional fatal accident to occur with self-driving cars, these autonomous vehicles are designed with safety in mind. One of the goals of self-driving cars is to save lives. “The fundamental difference is that with any kind of weapon, you’re intending to do harm, so that carries a special legal and moral burden,” Asaro explains. “There is a moral responsibility to ensure that [the weapon is] only used in legitimate and appropriate circumstances.”

Furthermore, liability with autonomous weapons is much more ambiguous than it is with self-driving cars and other domestic robots.

With self-driving cars, for example, bigger companies like Volvo intend to embrace strict liability – where the manufacturers assume full responsibility for accidental harm. Although it is not clear how all manufacturers will be held accountable for autonomous systems, strict liability and threats of class-action lawsuits incentivize manufacturers to make their product as safe as possible.

Warfare, on the other hand, is a much messier situation.

“You don’t really have liability in war,” says Asaro. “The US military could sue a supplier for a bad product, but as a victim who was wrongly targeted by a system, you have no real legal recourse.”

Autonomous weapons only complicate this. “These systems become more unpredictable as they become more sophisticated, so psychologically commanders feel less responsible for what those systems do. They don’t internalize responsibility in the same way,” Asaro explained at the Ethics of AI conference.

To ensure that commanders internalize responsibility, Asaro suggests that “the system has to allow humans to actually exercise their moral agency.”

That is, commanders must demonstrate that they can fully control the system before they use it in warfare. Once they demonstrate control, it can become clearer who can be held accountable for the system’s actions.

 

Preparing for the Unknown

Behind these concerns about liability, lies the overarching concern that autonomous machines might act in ways that humans never intended. Asaro asks: “When these systems become more autonomous, can the owners really know what they’re going to do?”

Even the programmers and manufacturers may not know what their machines will do. The purpose of developing autonomous machines is so they can make decisions themselves – without human input. And as the programming inside an autonomous system becomes more complex, people will increasingly struggle to predict the machine’s action.

Companies and governments must be prepared to handle the legal complexities of a domestic or military robot or system causing unintended harm. Ensuring justice for those who are harmed may not be possible without a clear framework for liability.

Asaro explains, “We need to develop policies to ensure that useful technologies continue to be developed, while ensuring that we manage the harms in a just way. A good start would be to prohibit automating decisions over the use of violent and lethal force, and to focus on managing the safety risks in beneficial autonomous systems.”

Peter Asaro also spoke about this work on an FLI podcast. You can learn more about his work at http://www.peterasaro.org.

This article is part of a Future of Life series on the AI safety research grants, which were funded by generous donations from Elon Musk and the Open Philanthropy Project.

Nuclear Winter with Alan Robock and Brian Toon

The UN voted last week to begin negotiations on a global nuclear weapons ban, but for now, nuclear weapons still jeopardize the existence of almost all people on earth.

I recently sat down with Meteorologist Alan Robock from Rutgers University and physicist Brian Toon from the University of Colorado to discuss what is potentially the most devastating consequence of nuclear war: nuclear winter.

Toon and Robock have studied and modeled nuclear winter off and on for over 30 years, and they joined forces ten years ago to use newer climate models to look at the climate effects of a small nuclear war.

The following interview has been heavily edited, but you can listen to it in its entirety here or read the complete transcript here.

Ariel: How is it that you two started working together?

Toon: This was initiated by a reporter. At the time, Pakistan and India were having a conflict over Kashmir and threatening each other with nuclear weapons. A reporter wanted to know what effect this might have on the rest of the planet. I calculated the amount of smoke and found, “Wow that was a lot of smoke!”

Alan had a great volcano model, so at the American Geophysical Union meeting that year, I tried to convince him to work on this problem. Alan was pretty skeptical.

Robock: I don’t remember being skeptical. I remember being very interested. I said, “How much smoke would there be?” Brian told me 5,000,000 tons of smoke, and I said, “That sounds like a lot!”

We put it into a NASA climate model and found it would be the largest climate change in recorded human history. The basic physics is very simple. If you block out the Sun, it gets cold and dark at the Earth’s surface.

We hypothesized that if each country used half of their nuclear arsenal, that would be 50 weapons on each side. We assumed the simplest bomb, which is the size dropped on Hiroshima and Nagasaki — a 15 kiloton bomb.

The answer is the global average temperature would go down by about 1.5 degrees Celsius. In the middle of continents, temperature drops would be larger and last for a decade or more.

We took models that calculate agricultural productivity and calculated how wheat, corn, soybeans, and rice production would change. In the 5 years after this war, using less than 1% of the global arsenal on the other side of the world, global food production would go down by 20-40 percent for 5 years, and for the next 5 years, 10-20 percent.

Ariel: Could you address criticisms of whether or not the smoke would loft that high or spread globally?

Toon: The only people that have been critical are Alan and I. The Departments of Energy and Defense, which should be investigating this problem, have done absolutely nothing. No one has done studies of fire propagation in big cities — no fire department is going to go put out a nuclear fire.

As far as the rising smoke, we’ve had people investigate that and they all find the same things: it goes into the upper atmosphere and then self-lofts. But, these should be investigated by a range of scientists with a range of experiences.

Robock: What are the properties of the smoke? We assume it would be small, single, black particles. That needs to be investigated. What would happen to the particles as they sit in the stratosphere? Would they react with other particles? Would they degrade? Would they grow? There are additional questions and unknowns.

Toon: Alan made lists of the important issues. And we have gone to every agency that we can think of, and said, “Don’t you think someone should study this?” Basically, everyone we tried so far has said, “Well, that’s not my job.”

Ariel: Do you think there’s a chance then that as we acquired more information that even smaller nuclear wars could pose similar risks? Or is 100 nuclear weapons the minimum?

Robock: First, it’s hard to imagine how once a nuclear war starts, it could be limited. Communications are destroyed, people panic — how would people even be able to rationally have a nuclear war and stop?

Second, we don’t know. When you get down to small numbers, it depends on what city, what time of year, the weather that day. And we don’t want to emphasize India and Pakistan – any two nuclear countries could do this.

Toon: The most common thing that happens when we give a talk is someone will stand up and say, “Oh, but a war would only involve one nuclear weapon.” But the only nuclear war we’ve had, the nuclear power, the United States, used every weapon that it had on civilian targets.

If you have 1000 weapons and you’re afraid your adversary is going to attack you with their 1000 weapons, you’re not likely to just bomb them with one weapon.

Robock: Let me make one other point. If the United States attacked Russia on a first strike and Russia did nothing, the climate change resulting from that could kill almost everybody in the United States. We’d all starve to death because of the climate response. People used to think of this as mutually assured destruction, but really it’s being a suicide bomber: it’s self-assured destruction.
Ariel: What scares you most regarding nuclear weapons?

Toon: Politicians’ ignorance of the implications of using nuclear weapons. Russia sees our advances to keep Eastern European countries free — they see that as an attempt to move military forces near Russia where [NATO] could quickly attack them. There’s a lack of communication, a lack of understanding of [the] threat and how people see different things in different ways. So Russians feel threatened when we don’t even mean to threaten them.

Robock: What scares me is an accident. There have been a number of cases where we came very close to having nuclear war. Crazy people or mistakes could result in a nuclear war. Some teenaged hacker could get into the systems. We’ve been lucky to have gone 71 years without a second nuclear war. The only way to prevent it is to get rid of the nuclear weapons.

Toon: We have all these countries with 100 weapons. All those countries can attack anybody on the Earth and destroy most of the country. This is ridiculous, to develop a world where everybody can destroy anybody else on the planet. That’s what we’re moving toward.

Ariel: Is there anything else you think the public needs to understand about nuclear weapons or nuclear winter?

Robock: I would think about all of the countries that don’t have nuclear weapons. How did they make that decision? What can we learn from them?

The world agreed to a ban on chemical weapons, biological weapons, cluster munitions, land mines — but there’s no ban on the worst weapon of mass destruction, nuclear weapons. The UN General Assembly voted next year to negotiate a treaty to ban nuclear weapons, which will be a first step towards reducing the arsenals and disarmament. But people have to get involved and demand it.

Toon: We’re not paying enough attention to nuclear weapons. The United States has invested hundreds of billions of dollars in building better nuclear weapons that we’re never going to use. Why don’t we invest that in schools or in public health or in infrastructure? Why invest it in worthless things we can’t use?

How Can AI Learn to Be Safe?

As artificial intelligence improves, machines will soon be equipped with intellectual and practical capabilities that surpass the smartest humans. But not only will machines be more capable than people, they will also be able to make themselves better. That is, these machines will understand their own design and how to improve it – or they could create entirely new machines that are even more capable.

The human creators of AIs must be able to trust these machines to remain safe and beneficial even as they self-improve and adapt to the real world.

Recursive Self-Improvement

This idea of an autonomous agent making increasingly better modifications to its own code is called recursive self-improvement. Through recursive self-improvement, a machine can adapt to new circumstances and learn how to deal with new situations.

To a certain extent, the human brain does this as well. As a person develops and repeats new habits, connections in their brains can change. The connections grow stronger and more effective over time, making the new, desired action easier to perform (e.g. changing one’s diet or learning a new language). In machines though, this ability to self-improve is much more drastic.

An AI agent can process information much faster than a human, and if it does not properly understand how its actions impact people, then its self-modifications could quickly fall out of line with human values.

For Bas Steunebrink, a researcher at the Swiss AI lab IDSIA, solving this problem is a crucial step toward achieving safe and beneficial AI.

Building AI in a Complex World

Because the world is so complex, many researchers begin AI projects by developing AI in carefully controlled environments. Then they create mathematical proofs that can assure them that the AI will achieve success in this specified space.

But Steunebrink worries that this approach puts too much responsibility on the designers and too much faith in the proof, especially when dealing with machines that can learn through recursive self-improvement. He explains, “We cannot accurately describe the environment in all its complexity; we cannot foresee what environments the agent will find itself in in the future; and an agent will not have enough resources (energy, time, inputs) to do the optimal thing.”

If the machine encounters an unforeseen circumstance, then that proof the designer relied on in the controlled environment may not apply. Says Steunebrink, “We have no assurance about the safe behavior of the [AI].”

Experience-based Artificial Intelligence

Instead, Steunebrink uses an approach called EXPAI (experience-based artificial intelligence). EXPAI are “self-improving systems that make tentative, additive, reversible, very fine-grained modifications, without prior self-reasoning; instead, self-modifications are tested over time against experiential evidences and slowly phased in when vindicated, or dismissed when falsified.”

Instead of trusting only a mathematical proof, researchers can ensure that the AI develops safe and benevolent behaviors by teaching and testing the machine in complex, unforeseen environments that challenge its function and goals.

With EXPAI, AI machines will learn from interactive experience, and therefore monitoring their growth period is crucial. As Steunebrink posits, the focus shifts from asking, “What is the behavior of an agent that is very intelligent and capable of self-modification, and how do we control it?” to asking, “How do we grow an agent from baby beginnings such that it gains both robust understanding and proper values?”

Consider how children grow and learn to navigate the world independently. If provided with a stable and healthy childhood, children learn to adopt values and understand their relation to the external world through trial and error, and by examples. Childhood is a time of growth and learning, of making mistakes, of building on success – all to help prepare the child to grow into a competent adult who can navigate unforeseen circumstances.

Steunebrink believes that researchers can ensure safe AI through a similar, gradual process of experience-based learning. In an architectural blueprint developed by Steunebrink and his colleagues, the AI is constructed “starting from only a small amount of designer-specific code – a seed.” Like a child, the beginnings of the machine will be less competent and less intelligent, but it will self-improve over time, as it learns from teachers and real-world experience.

As Steunebrink’s approach focuses on the growth period of an autonomous agent, the teachers, not the programmers, are most responsible for creating a robust and benevolent AI. Meanwhile, the developmental stage gives researchers time to observe and correct an AI’s behavior in a controlled setting where the stakes are still low.

The Future of EXPAI

Steunebrink and his colleagues are currently creating what he describes as a “pedagogy to determine what kind of things to teach to agents and in what order, how to test what the agents understand from being taught, and, depending on the results of such tests, decide whether we can proceed to the next steps of teaching or whether we should reteach the agent or go back to the drawing board.”

A major issue Steunebrink faces is that his method of experience-based learning diverges from the most popular methods for improving AI. Instead of doing the intellectual work of crafting a proof-backed optimal learning algorithm on a computer, EXPAI requires extensive in-person work with the machine to teach it like a child.

Creating safe artificial intelligence might prove to be more a process of teaching and growth rather than a function of creating the perfect mathematical proof. While such a shift in responsibility may be more time-consuming, it could also help establish a far more comprehensive understanding of an AI before it is released into the real world.

Steunebrink explains, “A lot of work remains to move beyond the agent implementation level, towards developing the teaching and testing methodologies that enable us to grow an agent’s understanding of ethical values, and to ensure that the agent is compelled to protect and adhere to them.”

The process is daunting, he admits, “but it is not as daunting as the consequences of getting AI safety wrong.”

If you would like to learn more about Bas Steunebrink’s research, you can read about his project here, or visit http://people.idsia.ch/~steunebrink/. He is also the co-founder of NNAISENSE, which you can learn about at https://nnaisense.com/.

This article is part of a Future of Life series on the AI safety research grants, which were funded by generous donations from Elon Musk and the Open Philanthropy Project.

The Age of Em: Review and Podcast

Interview with Robin Hanson
A few weeks ago, I had the good fortune to interview Robin Hanson about his new book, The Age of Em. We discussed his book, the future and evolution of humanity, and the research he’s doing for his next book. You can listen to all of that here. And read on for my review of Hanson’s book…

Age of Em Review

As I’ve interviewed more and more people who focus on and worry about the future, an interesting theme keeps popping up: choice. Over and over, scholars and researchers desperately try to remind us that we have a say in our future. The choices we make will impact whether or not our future goes the way we want it to.

But choosing a path for our future isn’t like picking a breakfast cereal. Our options aren’t all in front of us, with detailed information on the side telling us how we may or may not benefit from each choice. Few of us can predict how our decisions will shape our own lives, let alone the future of humanity. That’s where Robin Hanson comes in.

Hanson, a professor of economics at George Mason University, advocates that the activities that will shape our future can be much better informed by what is likely to happen than most of us realize.  He recently wrote the book, The Age of Em: Work, Love and Life when Robots Rule the Earth, which describes one possible path that may lay before us based on current economic and technologic trends.

What Is the Age of Em?

Em is short for emulation — in this case, a brain emulation. In this version of the future, people will choose to have their brains scanned, uploaded and possibly copied, creating a new race of robots and other types of machine intelligence. Because they’re human emulations, Hanson expects ems will think and feel just as we would. However, without biological needs or aging processes, they’ll be much cheaper to run and maintain. And because they can do anything we can – just at a much lower cost – it will be more practical to switch from human to em. Humans who resist this switch or who are too old to be useful as ems will end up on the sidelines of society. When (if) ems take over the world, it will be because humans chose to make the transition.

Interestingly, the timeline for how long ems will rule the world will depend on human versus em perspective. Because ems are essentially machines, they can run at different speeds. Hanson anticipates that over the course of two regular “human” years, most ems will have experienced a thousand years – along with all of the societal changes that come with a thousand years of development. Hanson’s book tells the story of their world: their subsistence lifestyles made glamorous by virtual reality; the em clans comprised of the best and the brightest human minds; and literally, how the ems will work, love, and live.

It’s a very detailed work, and it’s easy to get caught up in the details of which aspects of em life are likely, which details seem unrealistic, and even if ems are more likely than artificial intelligence to take over the world next. And there have been excellent discussions and reviews of the details of the book, like this one at Slate Star Codex. But I’m writing this review almost as much in response to commentary I’ve read about the book as I am about the book itself because there’s another question that’s important to ask as well: Is this the future that we want?

What do we want?

For a book without a plot or characters, it offers a surprisingly engaging and compelling storyline. Perhaps that’s because this is the story of us. It’s the story of humanity — the story of how we progress and evolve. And it’s also the story of how we, as we know ourselves, end.

It’s easy to look at this new world with fear. We’re so focused on production and the bottom line that, in the future, we’ve literally pushed humanity to the sidelines and possibly to extinction. Valuing productivity is fine, but do we really want to take it to this level? Can we stop this from happening, and if so, how?

Do we even want to stop it from happening? Early on Hanson encourages us to remember that people in the past would have been equally horrified by our own current lifestyle. He argues that this future may be different from what we’re used to, but it’s reasonable to expect that humans will prefer transitioning to an em lifestyle in the future. And from that perspective, we can look on this new world with hope.

As I read The Age of Em, I was often reminded of A Brave New World by Aldous Huxley. Huxley described his book as a “negative utopia,” but much of what he wrote has become commonplace and trivial today — mass consumerism, drugs to make us happy, a freer attitude about sex, a preference for mindless entertainment to deep thought. Though many of us may not necessarily consider these attributes of modern society to be a utopia, most people today would choose our current lifestyle over that of the 1930s, and we typically consider our lives better now than at any point in history. Even among people today, we see sharp divides between older generations who are horrified by how much privacy is lost thanks to the Internet and younger generations who see the benefits of increased information and connectivity outweighing any potential risks. Most likely, a similar attitude shift will take place as (if) we move toward a world of ems.

Yet while it’s reasonable to accept that in the future we would likely consider ems to be a positive step for humanity, the questions still remain: Is this what we want, or are we just following along on a path, unable to stop or change directions? Can we really choose our future?

Studying the future

In the book, Hanson says, “If we first look carefully at what is likely to happen if we do nothing, such a no-action baseline can help us to analyze what we might do to change those outcomes. This book, however, only offers the nearest beginnings of such policy analysis.” Hanson looks at where we’ve been, he looks at where we are now, and then he draws lines out into the future to figure out the direction we’ll go. And he does this for every aspect of life. In fact, given that this book is about the future, it also provides considerable insight into who we are now. But it represents only one possible vision for the future.

There are many more people who study history than the future, primarily because we already have information and writings and artifacts about historic events. But though we can’t change the past, we can impact the future. As the famous quote (or paraphrase) by George Santayana goes, “Those who fail to learn history are doomed to repeat it.” So perhaps learning history is only half the story. Perhaps it’s time to reevaluate the prevailing notion that the future is something that can’t be studied.

Only as we work through different possible scenarios for our future, can we better understand how decisions today will impact humanity later on. And with that new information, maybe we can start to make choices that will guide us toward a future we’re all excited about.

Final thoughts

Love it or hate it, agree with it or not, Hanson’s book and his approach to thinking about the future are extremely important for anyone who wants to have a say in the future of humanity. It’s easy to argue over whether or not ems represent the most likely future. It’s just as easy to get lost in the minutia of the em world and debate whether x, y, or z will happen. And these discussions are necessary if we’re to understand what could happen in the future. But to do only that is to miss an important point: something will happen, and we have to decide if we want a role in creating the future or if we want to stand idly by.

I highly recommend The Age of Em, I look forward to Hanson’s next book, and I hope others will answer his call to action and begin studying the future.

Note from FLI: Among our objectives is to inspire discussion and a sharing of ideas. As such, we post op-eds that we believe will help spur discussion within our community. Op-eds do not necessarily represent FLI’s opinions or views.

Training Artificial Intelligence to Compromise

Imagine you’re sitting in a self-driving car that’s about to make a left turn into on-coming traffic. One small system in the car will be responsible for making the vehicle turn, one system might speed it up or hit the brakes, other systems will have sensors that detect obstacles, and yet another system may be in communication with other vehicles on the road. Each system has its own goals — starting or stopping, turning or traveling straight, recognizing potential problems, etc. — but they also have to all work together toward one common goal: turning into traffic without causing an accident.

Harvard professor and FLI researcher, David Parkes, is trying to solve just this type of problem. Parkes told FLI, “The particular question I’m asking is: If we have a system of AIs, how can we construct rewards for individual AIs, such that the combined system is well behaved?”

Essentially, an AI within a system of AIs — like that in the car example above — needs to learn how to meet its own objective, as well as how to compromise so that it’s actions will help satisfy the group objective. On top of that, the system of AIs needs to consider the preferences of society. The safety of the passenger in the car or a pedestrian in the crosswalk is a higher priority than turning left.

Training a well-behaved AI

Because environments like a busy street are so complicated, an engineer can’t just program an AI to act in some way to always achieve its objectives. AIs need to learn proper behavior based on a rewards system. “Each AI has a reward for its action and the action of the other AI,” Parkes explained. With the world constantly changing, the rewards have to evolve, and the AIs need to keep up not only with how their own goals change, but also with the evolving objectives of the system as a whole.

The idea of a rewards-based learning system is something most people can likely relate to. Who doesn’t remember the excitement of a gold star or a smiley face on a test? And any dog owner has experienced how much more likely their pet is to perform a trick when it realizes it will get a treat. A reward for an AI is similar.

A technique often used in designing artificial intelligence is reinforcement learning. With reinforcement learning, when the AI takes some action, it receives either positive or negative feedback. And it then tries to optimize its actions to receive more positive rewards. However, the reward can’t just be programmed into the AI. The AI has to interact with its environment to learn which actions will be considered good, bad or neutral. Again, the idea is similar to a dog learning that tricks can earn it treats or praise, but misbehaving could result in punishment.

More than this, Parkes wants to understand how to distribute rewards to subcomponents – the individual AIs – in order to achieve good system-wide behavior. How often should there be positive (or negative) reinforcement, and in reaction to which types of actions?

For example, if you were to play a video game without any points or lives or levels or other indicators of success or failure, you might run around the world killing or fighting aliens and monsters, and you might eventually beat the game, but you wouldn’t know which specific actions led you to win. Instead, games are designed to provide regular feedback and reinforcement so that you know when you make progress and what steps you need to take next. To train an AI, Parkes has to determine which smaller actions will merit feedback so that the AI can move toward a larger, overarching goal.

Rather than programming a reward specifically into the AI, Parkes shapes the way rewards flow from the environment to the AI in order to promote desirable behaviors as the AI interacts with the world around it.

But this is all for just one AI. How do these techniques apply to two or more AIs?

Training a system of AIs

Much of Parkes’ work involves game theory. Game theory helps researchers understand what types of rewards will elicit collaboration among otherwise self-interested players, or in this case, rational AIs. Once an AI figures out how to maximize its own reward, what will entice it to act in accordance with another AI? To answer this question, Parkes turns to an economic theory called mechanism design.

Mechanism design theory is a Nobel-prize winning theory that allows researchers to determine how a system with multiple parts can achieve an overarching goal. It is a kind of “inverse game theory.” How can rules of interaction – ways to distribute rewards, for instance – be designed so individual AIs will act in favor of system-wide and societal preferences? Among other things, mechanism design theory has been applied to problems in auctions, e-commerce, regulations, environmental policy, and now, artificial intelligence.

The difference between Parkes’ work with AIs and mechanism design theory is that the latter requires some sort of mechanism or manager overseeing the entire system. In the case of an automated car or a drone, the AIs within have to work together to achieve group goals, without a mechanism making final decisions. As the environment changes, the external rewards will change. And as the AIs within the system realize they want to make some sort of change to maximize their rewards, they’ll have to communicate with each other, shifting the goals for the entire autonomous system.

Parkes summarized his work for FLI, saying, “The work that I’m doing as part of the FLI grant program is all about aligning incentives so that when autonomous AIs decide how to act, they act in a way that’s not only good for the AI system, but also good for society more broadly.”

Parkes is also involved with the One Hundred Year Study on Artificial Intelligence, and he explained his “research with FLI has informed a broader perspective on thinking about the role that AI can play in an urban context in the near future.” As he considers the future, he asks, “What can we see, for example, from the early trajectory of research and development on autonomous vehicles and robots in the home, about where the hard problems will be in regard to the engineering of value-aligned systems?”

This article is part of a Future of Life series on the AI safety research grants, which were funded by generous donations from Elon Musk and the Open Philanthropy Project.

Podcast: What Is Our Current Nuclear Risk?

A conversation with Lucas Perry about nuclear risk

Participants:

  • Ariel Conn— Ariel oversees communications and digital media at FLI, and as such, she works closely with members of the nuclear community to help present information about the costs and risks of nuclear weapons.
  • Lucas Perry—Lucas has been actively working with the Mayor and City Council of Cambridge, MA to help them divest from nuclear weapons companies, and he works closely with groups like Don’t Bank on the Bomb to bring more nuclear divestment options to the U.S.

Summary

In this podcast interview, Lucas and Ariel discuss the concepts of nuclear deterrence, hair trigger alert, the potential consequences of nuclear war, and how individuals can do their part to lower the risks of nuclear catastrophe. (You can find more links to information about these issues at the bottom of the page.)

Transcript

Ariel:  I’m Ariel Conn with the Future of Life Institute, and I’m here with Lucas Perry, also a member of FLI, to talk about the increasing risks of nuclear weapons, and what we can do to decrease those risks.

With the end of the Cold War, and the development of the two new START treaties, we’ve dramatically decreased the number of nuclear weapons around the world. Yet even though there are fewer weapons, they still represent a real and growing threat. In the last few months, FLI has gotten increasingly involved in efforts to decrease the risks of nuclear weapons.

One of the first things people worry about when it comes to decreasing the number of nuclear weapons or altering our nuclear posture is whether or not we can still maintain effective deterrence.

Lucas, can you explain how deterrence works?

Lucas: Sure, deterrence is the idea that to protect ourselves from other nuclear states who might want to harm us through nuclear strikes, if we have our own nuclear weapons primed and ready to be fired, it would deter another nuclear state from firing on us, knowing that we would retaliate with similar, or even more, nuclear force.

Ariel:  OK, and along the same lines, can you explain what hair trigger alert is?

Lucas: Hair trigger alert is a Cold War-era strategy that has nuclear weapons armed and ready for launch within minutes. It ensures mutual and total annihilation, and thus acts as a means of deterrence. But the problem here is that it also increases the likelihood of accidental nuclear war.

Ariel:  Can you explain how an accidental nuclear war could happen? And, also, has it almost happened before?

Lucas: Having a large fraction of our nuclear weapons on hair trigger alert creates the potential for accidental nuclear war through the fallibility of the persons and instruments involved with the launching of nuclear weapons, in junction with the very small amount of time actually needed to fire the nuclear missiles.

Us humans are known to be prone to making mistakes on a daily basis, and we even make the same mistakes multiple times. Computers, radars, and all of the other instruments and technology that go into the launching and detecting of nuclear strikes are intrinsically fallible, as well, as they are prone to breaking and committing error.

So there is the potential for us to fire missiles when an instrument gives us false alarm or a person—say, the President—under the pressure of needing to make a decision within only a few minutes, decides to fire missiles due to some misinterpretation of a situation. This susceptibility to error is actually so great that groups such as the Union of Concerned Scientists have been able to identify at least 21 nuclear close calls where nuclear war was almost started by mistake.

Ariel:  How long does the President actually have to decide whether or not to launch a retaliatory attack?

Lucas: The President actually only has about 12 minutes to decide whether or not to fire our missiles in retaliation. After our radars have detected the incoming missiles, and after this information has been conveyed to the President, there has already been some non-negligible amount of time—perhaps 5 to 15 minutes—where nuclear missiles might already be inbound. So he only has another few minutes—say, about 10 or 12 minutes—to decide whether or not to fire ours in retaliation. But this is also highly contingent upon where the missiles are coming from and how early we detected their launch.

Ariel:  OK, and then do you have any examples off the top of your head of times where we’ve had close calls that almost led to an unnecessary nuclear war?

Lucas: Out of the twenty-or-so nuclear close calls that have been identified by the Union of Concerned Scientists, among other organizations, a few that stand out to me are—for example, in 1980, the Soviet Union launched four submarine-based missiles from near the Kuril Islands as part of a training exercise, which led to a triggering of American early-warning sensors.

And even in 1995, Russian early-warning radar detected a missile launch off the coast of Norway with flight characteristics very similar to that of US submarine missiles. This led to all Russian nuclear forces going into full alert, and even the President at the time got his nuclear football ready and was prepared for full nuclear retaliation. But they ended up realizing that this was just a Norwegian scientific rocket.

These examples really help to illustrate how hair trigger alert is so dangerous. Persons and instruments are inevitably going to make mistakes, and this is only made worse when nuclear weapons are primed and ready to be launched within only minutes.

Ariel:  Going back to deterrence: Do we actually need our nuclear weapons to be on hair trigger alert in order to have effective deterrence?

Lucas: Not necessarily. The current idea is that we keep our intercontinental ballistic missiles (ICBMs), which are located in silos, on hair trigger alert so that these nuclear weapons can be launched before the silos are destroyed by an enemy strike. But warheads can be deployed without being on hair trigger alert, on nuclear submarines and bombers, without jeopardizing national security. If nuclear weapons were to be fired at the United States with the intention of destroying our nuclear missile silos, then we could authorize the launch of our submarine- and bomber-based missiles over the time span of hours and even days. These missiles wouldn’t be able to be intercepted, and would thus offer a means of retaliation, and thus deterrence, without the added danger of being on hair trigger alert.

Ariel:  How many nuclear weapons does the Department of Defense suggest we need to maintain effective deterrence?

Lucas: Studies have shown that only about 300 to 1,000 nuclear weapons are necessary for deterrence. An example of this would be, about 450 of these bombs could be located on submarines and bombers spread out throughout the world, with about another 450 at home on reserve and in silos.

Ariel:  So how many nuclear weapons are there in the US and around the world?

Lucas: There are currently about 15,700 nuclear weapons on this planet. Russia and the US are the main holders of these, with Russia having about 7,500 and the US having about 7,200. Other important nuclear states to note are China, Israel, the UK, North Korea, France, India, and Pakistan.

Ariel:  OK, so basically we have a lot more nuclear weapons than we actually need.

Lucas: Right. If only about 300 to 1,000 are needed for deterrence, then the amount of nuclear weapons on this planet could be exponentially less than it is currently. And the amount that we have right now is actually just blatant overkill. It’s a waste of resources and it increases the risk of accidental nuclear war, making both the countries that have them and the countries that don’t have them, more at risk.

Ariel:  I want to consider this idea of the countries that don’t have them being more at risk. I’m assuming you’re talking about nuclear winter. Can you explain what nuclear winter is?

Lucas: Nuclear winter is an indirect effect of nuclear war. When nuclear weapons go off they create large firestorms from all of the infrastructure, debris, and trees that are set on fire surrounding the point of detonation. These massive firestorms release enormous amounts of soot and smoke into the air that goes into the atmosphere and can block out the sun for months and even years at a time. This drastically reduces the amount of sunlight that is able to get to the Earth, and it thus causes a significant decrease in average global temperatures.

Ariel:  How many nuclear weapons would actually have to go off in order for us to see a significant drop in temperature?

Lucas: About 100 Hiroshima-sized nuclear weapons would decrease average global temperatures by about 1.25 degrees Celsius. When these 100 bombs go off, they would release about 5 million tons of smoke lofted high into the stratosphere. And now, this change of 1.25 degrees Celsius of average global temperatures might seem very tiny, but studies actually show that this will lead to a shortening of growing seasons by up to 30 days and a 10% reduction in average global precipitation. Twenty million people would die directly from the effects of this, but then hundreds of millions of people would die in the following months from a lack of food due to the decrease in average global temperatures and a lack of precipitation.

Ariel:  And that’s hundreds of millions of people around the world, right? Not just in the regions where the war took place?

Lucas: Certainly. The soot and smoke from the firestorms would spread out across the entire planet and be affecting the amount of precipitation and sunlight that everyone receives. It’s not simply that the effects of nuclear war are contained to the countries involved with the nuclear strikes, but rather, potentially the very worst effects of nuclear war create global changes that would affect us all.

Ariel:  OK, so that was for a war between India and Pakistan, which would be small, and it would be using smaller nuclear weapons than what the US and Russia have. So if just an accident were to happen that triggered both the US and Russia to launch their nuclear weapons that are on hair trigger alert, what would the impacts of that be?

Lucas: Well, the United States has about a thousand weapons on hair trigger alert. I’m not exactly sure as to how many there are in Russia, but we can assume that it’s probably a similar amount. So if a nuclear war of about 2,000 weapons were to be exchanged between the United States and Russia, it would cause 510 million tons of smoke to rise into the stratosphere, which would lead to a 4 degrees Celsius change in average global temperatures. And compared to an India-Pakistan conflict, this would lead to catastrophically more casualties from a lack of food and from the direct effects of these nuclear bombs.

Ariel:  And over what sort of time scale is that expected to happen?

Lucas: The effects of nuclear winter, and perhaps even what might one day be nuclear summer, would be lasting over the time span of not just months, but years, even decades.

Ariel:  What’s nuclear summer?

Lucas: So nuclear summer is a more theoretical effect of nuclear war. With nuclear winter you have tons of soot and ash and smoke in the sky blotting out the sun, but additionally, there has actually been an enormous amount of CO2 released from the burning all of the infrastructure and forests and grounds due to the nuclear blasts. After decades, once all of this soot and ash and smoke begin to settle back down onto the Earth’s surface, there will still be this enormous remaining amount of CO2.

So nuclear summer is a hypothetical indirect effect of nuclear war, after nuclear winter, after the soot has fallen down, where there would be a huge spike in average global temperatures due to the enormous amount of CO2 left over from the firestorms.

Ariel: And so how likely is all of this to happen? Is there actually a chance that these types of wars could occur? Or is this mostly something that people are worrying about unnecessarily?

Lucas: The risk of a nuclear war is non-zero. It’s very difficult to quantify exactly what the risks are, but we can say that we have seen at least 21 nuclear close calls where nuclear war was almost started by mistake. And these 21 close calls are actually just the ones that we know about. How many more nuclear close calls have there been that we simply don’t know about, or that governments have been able to keep a secret? We can reflect that as tensions rise between the United States and Russia, and as the risk of terrorism and cyber attack continues to rise, and the conflict between India and Pakistan is continually exacerbated, the threat of nuclear war is actually increasing. It’s not going down.

Ariel:  So there is a risk, and we know that we have more nuclear weapons than we actually need for deterrence. Even if we want to keep enough weapons for deterrence, we don’t need as many as we have. I’m guessing that the government is not going to do anything about this, so what can people do to try to have an impact themselves?

Lucas: A method of engaging with this nuclear issue that has a potentially high efficacy is divesting. We have power as voters, consumers, and producers, but perhaps even more importantly, we have power over what we invest in. We have the power to choose to invest in companies that are socially responsible or ones which are not. So through divestment, we can take money away from nuclear weapons producers. But not only that, we can also work to stigmatize nuclear weapons production and our current nuclear situation through our divestment efforts.

Ariel:  But my understanding is that most of our nuclear weapons are funded by the government, so how would a divestment campaign actually be impactful, given that the money for nuclear weapons wouldn’t necessarily disappear?

Lucas: The most important part of divestment in this area of nuclear weapons is actually the stigmatization. When you see massive amounts of people divesting from something, it creates a lot of light and heat on the subject. It influences the public consciousness and helps to bring back to light this issue of nuclear weapons. And once you have stigmatized something to a critical point, it effectively renders its target politically and socially untenable. Divestment also stimulates new education and research on the topic, while also getting persons invested in the issue.

Ariel:  And so have there been effective campaigns that used divestment in the past?

Lucas: There have been a lot of different campaigns in the past that have used divestment as an effective means of creating important change in the world. A few examples of these are divestment from tobacco, South African apartheid, child labor, and fossil fuels. In all of these instances, persons were divesting from institutions involved in these socially irresponsible acts. Through doing so, they created much stigmatization of these issues, they created capital flight from them, and also created a lot of negative media attention that helped to bring light to these issues and show people the truth of what was going on.

Ariel:  I know FLI was initially inspired by a lot of the work that Don’t Bank on the Bomb has done. Can you talk a bit about some of the work they’ve done and what their success has been?

Lucas: The Don’t Bank on the Bomb campaign has been able to identify direct and indirect investments in nuclear weapons producers, made by large institutions in both Europe and America. Through this they have worked to engage with many banks in Europe to help them to not include these direct or indirect investments in their portfolios and mutual funds, thus helping them to construct socially responsible funds. A few examples of these successes are A&S Bank, ASR, and the Cooperative Bank.

Ariel:  So you’ve been very active with FLI in trying to launch a divestment campaign in the US. I was hoping you could talk a little about the work you’ve done so far and the success you’ve had.

Lucas: Inspired by a lot of the work that’s been done through the Don’t Bank on the Bomb campaign, in junction with resources provided by them, we were able to engage with the city of Cambridge and work with them and help them to divest $1 billion from nuclear weapons-producing companies. As we continue our divestment campaign, we’re really passionate about making the information needed for divestment transparent and open. Currently we’re working on a web app that will allow you to search your mutual fund and see whether not it has direct or indirect investments in nuclear weapons producers. Through doing so, we hope to not only be helping cities and municipalities and institutions divest, but also individuals like you and me.

Ariel:  Lucas, this has been great. Thank you so much for sharing information about the work you’ve been doing so far. If anyone has any questions about how they can divest from nuclear weapons, please email Lucas at lucas@futureoflife.org. You can also check out our new web app at futureoflife.org/invest.

[end of recorded material]

Learn more about nuclear weapons in the 21st Century:

What is hair-trigger alert?

How many nuclear weapons are there and who has them?

What are the consequences of nuclear war?

What would the world look like after a U.S and Russia nuclear war?

How many nukes would it take to make the Earth uninhabitable?

What are the specific effects of nuclear winter?

What can I do to mitigate the risk of nuclear war?

Do we really need so many nuclear weapons on hair-trigger alert?

What sort of new nuclear policy could we adopt?

How can we restructure strategic U.S nuclear forces?

Podcast: Concrete Problems in AI Safety with Dario Amodei and Seth Baum

Many researchers in the field of artificial intelligence worry about potential short-term consequences of AI development. Yet far fewer want to think about the long-term risks from more advanced AI. Why? To start to answer that question, it helps to have a better understanding of what potential issues we could see with AI as it’s developed over the next 5-10 years. And it helps to better understand the concerns actual researchers have about AI safety, as opposed to fears often brought up in the press.

We brought on Dario Amodei and Seth Baum to discuss just that. Amodei, who now works with OpenAI, was the lead author on the recent, well-received paper Concrete Problems in AI Safety. Baum is the Executive Director of the Global Catastrophic Risk Institute, where much of his research is also on AI safety.

Not in a good spot to listen? You can always read the transcript here.

If you’re still new to or learning about AI, the following terminology might help:

Artificial Intelligence (AI): A machine or program that can learn to perform cognitive tasks, similar to those achieved by the human brain. Typically, the program, or agent, is expected to be able to interact with the real world in some way without constant supervision from its creator. Microsoft Office is considered a computer program because it will do only what it is programmed to do. Siri is considered by most to be a very low-level AI because it must adapt to its surroundings, respond to a wide variety of owners, and understand a wide variety of requests, not all of which can be programmed for in advance. Levels of artificial intelligence fall along a spectrum:

  • Narrow AI: This is an artificial intelligence that can only perform a specific task. Siri can look up anything on a search engine, but it can’t write a book or drive a car. Google’s self-driving cars can drive you where you want to go, but they can’t cook dinner. AlphaGo can beat the world’s best Go player, but it can’t play Monopoly or research cancer. Each of these programs can do the program they’re designed for as well as, or better than humans, but they don’t come close to the breadth of capabilities humans have.
  • Short-term AI concerns: The recent increase in AI development has many researchers concerned about problems that could arise in the next 5-10 years. Increasing autonomy will impact the job market and potentially income inequality. Biases, such as sexism and racism, have already cropped up in some programs, and people worry this could be exacerbated as AIs become more capable. Many wonder how we can ensure control over systems after they’ve been released for the public, as seen with Microsoft’s problems with its chatbot Tay. Transparency is another issue that’s often brought up — as AIs learn to adapt to their surroundings, they’ll modify their programs for increased efficiency and accuracy, and it will become increasingly difficult to track why an AI took some action. These are some of the more commonly mentioned concerns, but there are many others.
  • Advanced AI and Artificial General Intelligence (AGI): As an AI program expands its capabilities, it will be considered advanced. Once it achieves human-level intelligence in terms of both capabilities and breadth, it will be considered generally intelligent.
  • Long-term AI concerns: Current expectations are that we could start to see more advanced AI systems within the next 10-30 years. For the most part, the concerns for long-term AI are similar to those of short-term AI, except that, as AIs become more advanced, the problems that arise as a result could be more damaging, destructive, and/or devastating.
  • Superintelligence: AI that is smarter than humans in all fields.

Agent: A program, machine, or robot with some level of AI capabilities that can act autonomously in a simulated environment or the real world.

Machine Learning: An area of AI research that focuses on how the agent can learn from its surroundings, experiences, and interactions in order to improve how well it functions and performs its assigned tasks. With machine learning, the AI will adapt to its environment without the need for additional programming. AlphaGo, for example, was not programmed to be better than humans from the start. None of its programmers were good enough at the game of Go to compete with the world’s best. Instead, it was programmed to play lots of games of Go with the intent to win. Each time it won or lost a game, it learned more about how to win in the future.

Training: These are the iterations a machine-learning program must go through in order learn how to better meet its goal by making adjustments to the program’s settings. In the case of AlphaGo, training involved playing Go over and over.

Neural Networks (Neural Nets) and Deep Neural Nets: Neural nets are programs that were inspired by the way the central nervous system of animals processes information, especially with regard to pattern recognition. These are important tools within a machine learning algorithm that can help the AI process and learn from the information it receives. Deep neural nets have more layers of complexity.

Reinforcement Learning: Similar to training a dog. The agent receives positive or negative feedback for each iteration of its training, so that it can learn which actions it should seek out and which it should avoid.

Objective Function: This is the goal of the AI program (it can also include subgoals). Using AlphaGo as an example again, the primary objective function would have been to win the game of Go.

Terms from the paper, Concrete Problems in AI Safety, that might not be obvious (all are explained in the podcast, as well):

  • Reward Hacking: When the AI system comes up with an undesirable way to achieve its goal or objective function. For example, if you tell a robot to clean up any mess it sees, it might just throw away all messes so it can’t see them anymore.
  • Scalable Oversight: Training an agent to solve problems on its own without requiring constant oversight from a human.
  • Safe Exploration: Training an agent to explore its surroundings safely, without injuring itself or others and without triggering some negative outcome that could be difficult to recover from.
  • Robustness to distributional shifts: Training an agent to adapt to new environments and to understand when the environment has changed so it knows to be more cautious.

Effective Altruism 2016

The Effective Altruism Movement

Edit: The following article has been updated to include more highlights as well as links to videos of the talks.

How can we more effectively make the world a better place? Over 1,000 concerned altruists converged at the Effective Altruism Global conference this month in Berkeley, CA to address this very question. For two and a half days, participants milled around the Berkeley campus, attending talks, discussions, and workshops to learn more about efforts currently underway to improve our ability to not just do good in the world, but to do the most good.

Those who arrived on the afternoon of Friday, August 5 had the opportunity to mingle with other altruists and attend various workshops geared toward finding the best careers, improving communication, and developing greater self-understanding and self-awareness.

But the conference really kicked off on Saturday, August 6, with talks by Will MacAskill and Toby Ord, who both helped found the modern effective altruistism movement. Ord gave the audience a brief overview of the centuries of science and philosophy that provided the base for effective altruism. “Effective altruism is to the pursuit of good as the scientific revolution is to the pursuit of truth,” he explained. Yet, as he pointed out, effective altruism has only been a real “thing” for five years.

Will MacAskill

Will MacAskill introduced the conference and spoke of the success the EA movement has had in the last year.

Toby Ord speaking about the history of effective altruism.

Toby Ord spoke about the history of effective altruism.

 

MacAskill took the stage after Ord to highlight the movement’s successes over the past year, including coverage by such papers as the New York Times and the Washington Post. And more importantly, he talked about the significant increase in membership they saw this year, as well as in donations to worthwhile causes. But he also reminded the audience that a big part of the movement is the process of effective altruism. He said:

“We don’t know what the best way to do good is. We need to figure that out.”

For the rest of the two days, participants considered past charitable actions that had been most effective, problems and challenges altruists face today, and how the movement can continue to grow. There were too many events to attend them all, but there were many highlights.

Highlights From the Conference

When FLI cofounder, Jaan Tallin, was asked why he chose to focus on issues such as artificial intelligence, which may or may not be a problem in the future, rather than mosquito nets, which could save lives today, he compared philanthropy to investing. Higher risk investments have the potential for a greater payoff later. Similarly, while AI may not seem like much of  threat to many people now, ensuring it remains safe could save billions of lives in the future. Tallin spoke as part of a discussion on Philanthropy and Technology.

Jaan Tallin speaking remotely about his work with EA efforts.

Jaan Tallin speaking remotely about his work with EA efforts.

Martin Reese, a member of FLI’s Science Advisory Board, argued that we are in denial of the seriousness of our risks. At the same time, he said that minimizing risks associated with technological advances can only be done “with great difficulty.”  He encouraged EA participants to figure out which threats can be dismissed as science fiction and which are legitimate, and he encouraged scientists to become more socially engaged.

As if taking up that call to action, Kevin Esvelt talked about his own attempts to ensure gene drive research in the wild is accepted and welcomed by local communities. Gene drives could be used to eradicate such diseases as malaria, schistosomiasis, Zika, and many others, but fears of genetic modification could slow research efforts. He discussed his focus on keeping his work as open and accessible as possible, engaging with the public to allow anyone who might be affected by his research to have as much input as they want. “Closed door science,” he added, “is more dangerous because we have no way of knowing what other people are doing.”  A single misstep with this early research in his field could imperil all future efforts for gene drives.

Kevin Esvelt talks about his work with CRISPR and gene drives.

Kevin Esvelt talks about his work with CRISPR and gene drives.

That same afternoon, Cari Tuna, President of the Open Philanthropy Project, sat down with Will McAskill for an interview titled, “Doing Philosophy Better,” which focused on her work with OPP and Effective Altruism and how she envisions her future as a philanthropist. She highlighted some of the grants she’s most excited about, which include grants to Give Directly, Center for Global Development, and Alliance for Safety and Justice. When asked about how she thought EA could improve, she emphasized, “We consider ourselves a part of the Effective Altruism community, and we’re excited to help it grow.” But she also said, “I think there is a tendency toward overconfidence in the EA community that sometimes undermines our credibility.” She mentioned that one of the reasons she trusted GiveWell was because of their self reflection. “They’re always asking, ‘how could we be wrong?'” she explained, and then added, “I would really love to see self reflection become more of a core value of the effective altruism community.”

cari tuna

Cari Tuna interviewed by Will McAskill (photo from the Center for Effective Altruism).

The next day, FLI President, Max Tegmark, highlighted the top nine myths of AI safety, and he discussed how important it is to dispel these myths so researchers can focus on the areas necessary to keep AI beneficial. Some of the most distracting myths include arguments over when artificial general intelligence could be created, whether or not it could be “evil,” and goal-oriented issues. Tegmark also added that the best thing people can do is volunteer for EA groups.

During the discussion about the risks and benefits of advanced artificial intelligence, Dileep George, cofounder of Vicarious, reminded the audience why this work is so important. “The goal of the future is full unemployment so we can all play,” he said. Dario Amodei of OpenAI emphasized that having curiosity and trying to understand how technology is evolving can go a long way toward safety. And though he often mentioned the risks of advanced AI, Toby Ord, a philosopher and research fellow with the Future of Humanity Institute, also added, “I think it’s more likely than not that AI will contribute to a fabulous outcome.” Later in the day, Chris Olah, an AI researcher at Google Brain and one of the lead authors of the paper, Concrete Problems in AI Safety, explained his work as trying to build a bridge to futuristic problems by doing empirical research today.

Moderator Riva-Melissa Tez, Dario Amodei, George Dileep, and Toby Ord at the Risks and Benefits of Advanced AI discussion.

Moderator Riva-Melissa Tez, Dario Amodei, Dileep George, and Toby Ord at the Risks and Benefits of Advanced AI discussion. (Not pictured, Daniel Dewey)

FLI’s Richard Mallah gave a talk on mapping the landscape of AI safety research threads. He showed how there are many meaningful dimensions along which such research can be organized, how harmonizing the various research agendas into a common space allows us to reason about different kinds of synergies and dependencies, and how consideration of the white space in such representations can help us find both unknown knowns and unknown unknowns about the space.

Tara MacAulay, COO at the Centre for Effective Altruism, spoke during the discussion on “The Past, Present, and Future of EA.” She talked about finding the common values in the movement and coordinating across skill sets rather than splintering into cause areas or picking apart who is and who is not in the movement. She said, “The opposite of effective altruism isn’t ineffective altruism. The opposite of effective altruism is apathy, looking at the world and not caring, not doing anything about it . . . It’s helplessness. . . . throwing up our hands and saying this is all too hard.”

MacAulay also moderated a panel discussion called, Aggregating Knowledge, which was significant, not only for its thoughtful content about accessing, understanding, and communicating all of the knowledge available today, but also because it was an all-woman panel. The panel included Sarah Constantin, Amanda Askell, Julia Galef, and Heidi McAnnaly, who discussed various questions and problems the EA community faces when trying to assess which actions will be most effective. MacAulay summarized the discussion at the end when she said, “Figuring out what to do is really difficult but we do have a lot of tools available.” She concluded with a challenge to the audience to spend five minutes researching some belief they’ve always had about the world to learn what the evidence actually says about it.

aggregating knowledge

Sarah Constantin, Amanda Askell, Julia Galef, Heidi McAnnaly, and Tara MacAulay (photo from the Center for Effective Altruism).

Prominent government leaders also took to the stage to discuss how work with federal agencies can help shape and impact the future. Tom Kalil, Deputy Director for Technology and Innovation highlighted how much of today’s technology, from cell phones to Internet, got its start in government labs. Then, Jason Matheny, Director of IARPA, talked about how delays in technology can actually cost millions of lives. He explained that technology can make it less costly to enhance moral developments and that, “ensuring that we have a future counts a lot.”

Tom Kalil speaks about the history of government research and its impact on technology.

Tom Kalil speaks about the history of government research and its impact on technology.

Jason Matheny talks about how employment with government agencies can help advance beneficial technologies.

Jason Matheny talks about how employment with government agencies can help advance beneficial technologies.

Robin Hanson, author of The Age of Em, talked about his book and what the future will hold if we continue down our current economic path while the ability to create brain emulation is developed. He said that if creating ems becomes cheaper than paying humans to do work, “that would change everything.” Ems would completely take over the job market and humans would be pushed aside. He explained that some people might benefit from this new economy, but it would vary, just as it does today, with many more people suffering from poverty and fewer gaining wealth.

Robin Hanson talks to a group about how brain emulations might take over the economy and what their world will look like.

Robin Hanson talks to a group about how brain emulations might take over the economy and what their world will look like.

 

Applying EA to Real Life

Lucas Perry, also with FLI, was especially impressed by the career workshops offered by 80,000 Hours during the conference. He said:

“The 80,000 Hours workshops were just amazing for giving new context and perspective to work. 80,000 Hours gave me the tools and information necessary to reevaluate my current trajectory and see if it really is best of all possible paths for me and the world.

In the end, I walked away from the conference realizing I had been missing out on something so important for most of my life. I found myself wishing that effective altruism, and organizations like 80,000 Hours, had been a part of my fundamental education. I think it would have helped immensely with providing direction and meaning to my life. I’m sure it will do the same for others.”

In total, 150 people spoke over the course of those two and a half days. MacAskill finally concluded the conference with another call to focus on the process of effective altruism, saying:

“Constant self-reflection, constant learning, that’s how we’re going to be able to do the most good.”

 

View from the conference.

View from the conference.

The Evolution of AI: Can Morality be Programmed?

The following article was originally posted on Futurism.com.

Recent advances in artificial intelligence have made it clear that our computers need to have a moral code. Disagree? Consider this: A car is driving down the road when a child on a bicycle suddenly swerves in front of it. Does the car swerve into an oncoming lane, hitting another car that is already there? Does the car swerve off the road and hit a tree? Does it continue forward and hit the child?

Each solution comes with a problem: It could result in death.

It’s an unfortunate scenario, but humans face such scenarios every day, and if an autonomous car is the one in control, it needs to be able to make this choice. And that means that we need to figure out how to program morality into our computers.

Vincent Conitzer, a Professor of Computer Science at Duke University, recently received a grant from the Future of Life Institute in order to try and figure out just how we can make an advanced AI that is able to make moral judgments…and act on them.

MAKING MORALITY

At first glance, the goal seems simple enough—make an AI that behaves in a way that is ethically responsible; however, it’s far more complicated than it initially seems, as there are an amazing amount of factors that come into play. As Conitzer’s project outlines, “moral judgments are affected by rights (such as privacy), roles (such as in families), past actions (such as promises), motives and intentions, and other morally relevant features. These diverse factors have not yet been built into AI systems.”

That’s what we’re trying to do now.

In a recent interview with Futurism, Conitzer clarified that, while the public may be concerned about ensuring that rogue AI don’t decide to wipe-out humanity, such a thing really isn’t a viable threat at the present time (and it won’t be for a long, long time). As a result, his team isn’t concerned with preventing a global-robotic-apocalypse by making selfless AI that adore humanity. Rather, on a much more basic level, they are focused on ensuring that our artificial intelligence systems are able to make the hard, moral choices that humans make on a daily basis.

So, how do you make an AI that is able to make a difficult moral decision?

Conitzer explains that, to reach their goal, the team is following a two path process: Having people make ethical choices in order to find patterns and then figuring out how that can be translated into an artificial intelligence. He clarifies, “what we’re working on right now is actually having people make ethical decisions, or state what decision they would make in a given situation, and then we use machine learning to try to identify what the general pattern is and determine the extent that we could reproduce those kind of decisions.”

In short, the team is trying to find the patterns in our moral choices and translate this pattern into AI systems. Conitzer notes that, on a basic level, it’s all about making predictions regarding what a human would do in a given situation, “if we can become very good at predicting what kind of decisions people make in these kind of ethical circumstances, well then, we could make those decisions ourselves in the form of the computer program.”

However, one major problem with this is, of course, that morality is not objective — it’s neither timeless nor universal.

Conitzer articulates the problem by looking to previous decades, “if we did the same ethical tests a hundred years ago, the decisions that we would get from people would be much more racist, sexist, and all kinds of other things that we wouldn’t see as ‘good’ now. Similarly, right now, maybe our moral development hasn’t come to its apex, and a hundred years from now people might feel that some of the things we do right now, like how we treat animals, is completely immoral. So there’s kind of a risk of bias and with getting stuck at whatever our current level of moral development is.”

And of course, there is the aforementioned problem regarding how complex morality is. “Pure altruism, that’s very easy to address in game theory, but maybe you feel like you owe me something based on previous actions. That’s missing from the game theory literature, and so that’s something that we’re also thinking about a lot—how can you make this, what game theory calls ‘Solutions Concept’—sensible? How can you compute these things?”

To solve these problems, and to help figure out exactly how morality functions and can (hopefully) be programmed into an AI, the team is combining the methods from computer science, philosophy, and psychology “That’s, in a nutshell, what our project is about,” Conitzer asserts.

But what about those sentient AI? When will we need to start worrying about them and discussing how they should be regulated?

THE HUMAN-LIKE AI

According to Conitzer, human-like artificial intelligence won’t be around for some time yet (so yay! No Terminator-styled apocalypse…at least for the next few years).

“Recently, there have been a number of steps towards such a system, and I think there have been a lot of surprising advances….but I think having something like a ‘true AI,’ one that’s really as flexible, able to abstract, and do all these things that humans do so easily, I think we’re still quite far away from that,” Conitzer asserts.

True, we can program systems to do a lot of things that humans do well, but there are some things that are exceedingly complex and hard to translate into a pattern that computers can recognize and learn from (which is ultimately the basis of all AI).

“What came out of early AI research, the first couple decades of AI research, was the fact that certain things that we had thought of as being real benchmarks for intelligence, like being able to play chess well, were actually quite accessible to computers. It was not easy to write and create a chess-playing program, but it was doable.”

Indeed, today, we have computers that are able to beat the best players in the world in a host of games—Chess and Alpha Go, for example.

But Conitzer clarifies that, as it turns out, playing games isn’t exactly a good measure of human-like intelligence. Or at least, there is a lot more to the human mind. “Meanwhile, we learned that other problems that were very simple for people were actually quite hard for computers, or to program computers to do. For example, recognizing your grandmother in a crowd. You could do that quite easily, but it’s actually very difficult to program a computer to recognize things that well.”

Since the early days of AI research, we have made computers that are able to recognize and identify specific images. However, to sum the main point, it is remarkably difficult to program a system that is able to do all of the things that humans can do, which is why it will be some time before we have a ‘true AI.’

Yet, Conitzer asserts that now is the time to start considering what the rules we will use to govern such intelligences. “It may be quite a bit further out, but to computer scientists, that means maybe just on the order of decades, and it definitely makes sense to try to think about these things a little bit ahead.” And he notes that, even though we don’t have any human-like robots just yet, our intelligence systems are already making moral choices and could, potentially, save or end lives.

“Very often, many of these decisions that they make do impact people and we may need to make decisions that we will typically be considered to be a morally loaded decision. And a standard example is a self-driving car that has to decide to either go straight and crash into the car ahead of it or veer off and maybe hurt some pedestrian. How do you make those trade-offs? And that I think is something we can really make some progress on. This doesn’t require superintelligent AI, this can just be programs that make these kind of trade-offs in various ways.”

But of course, knowing what decision to make will first require knowing exactly how our morality operates (or at least having a fairly good idea). From there, we can begin to program it, and that’s what Conitzer and his team are hoping to do.

So welcome to the dawn of moral robots.

This interview has been edited for brevity and clarity. 

This article is part of a Future of Life series on the AI safety research grants, which were funded by generous donations from Elon Musk and the Open Philanthropy Project.

The Trillion Dollar Question Obama Left Unanswered in Hiroshima

A soldier carries the briefcase containing nuclear weapons codes for U.S. President Barack Obama. REUTERS/Joshua Roberts

This following article, written by Max Tegmark and Frank Wilczek, was originally posted in The Conversation.

As it seeks to modernize its nuclear arsenal, the United States faces a big choice, one which Barack Obama failed to mention during his moving Hiroshima speech on May 27.

Should we spend a trillion dollars to replace each of our thousands of nuclear warheads with a more sophisticated substitute attached to a more lethal delivery system? Or should we keep only enough nuclear weapons needed for a devastatingly effective deterrence against any nuclear aggressor, investing the money saved into other means of making our nation more secure? The first option would allow us to initiate and wage nuclear war. The second would allow us to deter it. These are very different tasks.

As physicists who have studied nuclear reactions and cataclysmic explosions, we are acutely aware that nuclear weapons are so devastating that merely a hundred could annihilate the major population centers of any potential state enemy. That prospect is enough to deter any rational leadership while no number of weapons could deter a mad one. Waging nuclear warfare could involve using vastly more warheads to strike diverse military and industrial targets.

So, is maintaining the ability to initiate nuclear war worth a trillion dollar investment?

The limits of nuclear blackmail

The U.S. and Russia currently have about 7,000 nukes each, largely for historical reasons. That’s over 13 times as many as held by the other seven nuclear powers combined. When the Soviet Union was perceived to be a threat to Europe with its numerically superior conventional forces, the U.S. stood ready to use nuclear weapons in response. We were prepared not only to deter the use of nuclear weapons by others, but also possibly to initiate nuclear warfare, and to use nuclear weapons in battle.

Now the tables have turned and NATO is the dominant nonnuclear force in Europe. But other arguments for maintaining the ability to initiate nuclear war remain, positing the utility of “compellance” (also known as “nuclear blackmail”) or using the threat of nuclear attack to extract concessions. This strategy has been used on several occasions. For example, when President Eisenhower threatened the use of nuclear weapons to compel negotiations ending the Korean War.

In today’s world, with nuclear technology more widely accessible, compellance is no longer straightforward. If a nonnuclear nation feels it is subject to nuclear bullying, it can counter by developing its own nuclear deterrent, or enlisting nuclear allies. For example, U.S. nuclear threats inspired North Korea to mount its own nuclear program, which is, to say the least, not the result we were hoping for.

North Korean leader Kim Jong-Un looks at a rocket warhead tip after a simulated test of atmospheric reentry of a ballistic missile. Such missiles are often used to deliver nuclear weapons. North Korea’s Korean Central News Agency via REUTERS

Another development is the emergence of modern threats to the U.S. and its allies against which nuclear compellance is rather useless. For example, nuclear weapons didn’t help prevent 9/11. Nor did they help the U.S. in Iraq, Afghanistan, Syria or Libya – or in the battle against terrorist groups such as Al-Quaida or the Islamic State.

These considerations raise the question of whether we might actually improve our national security by forswearing compellance and committing to “No First Use.” That is, committing to using nuclear weapons only in response to their use by others. This deterrence-only approach is already the policy of two other major nuclear powers, China and India. It is a mission we could fulfill with a much smaller and cheaper arsenal, freeing up money for other investments in our national security. By easing fear of our intentions, this could also reduce further nuclear proliferation – so far, eight other nations have developed nukes after we bombed Hiroshima, and all except Russia have concluded that deterrence requires fewer than a few hundred nuclear weapons. Indeed, hundreds of warheads may be a more convincing deterrent than thousands, because use of the latter might be an act of self destruction, triggering a decade-long global nuclear winter that would kill most Americans even if no nuclear explosions occurred on U.S. soil.

‘No First Use’ or ‘Pay to Play’?

Whatever one’s opinion on No First Use, it is a question with huge implications for military spending. Were the U.S. to pledge No First Use, we would have no reason to deploy more nuclear weapons than required for deterrence. We could save ourselves four million dollars per hour for the next 30 years, according to government estimates.

Nuclear weapons involve many complex issues. But one crucial question is beautifully simple: is our aim strictly to deter nuclear war, or should we invest the additional resources needed to maintain our ability to initiate it? No First Use, or Pay to Play?

We urge debate moderators, town hall participants and anyone else who gets the opportunity to ask our presidential candidates this crucial question. American voters deserve to know where their candidates stand.

Nuclear Weapons Are Scary — But We Can Do Something About Them

We’re ending our Huffington Post nuclear security series on a high note, with this article by Susi Snyder, explaining how people can take real action to decrease the threat of nuclear weapons.

Nuclear weapons are scary. The risk of use by accident, intention or terror. The climate consequences. The fact that they are designed and built to vaporize thousands of people with the push of a button. Scary. Fortunately, there is something we can do.

We know that nuclear weapons are scary, but we must be much louder in defining them as unacceptable, as illegitimate. By following the money, we can cut it off, and while this isn’t the only thing necessary to make nuclear weapons extinct, it will help.

That’s why we made Don’t Bank on the Bomb. Because we want to do something about nuclear weapons. Investments are not neutral. Financing and investing are active choices, based on a clear assessment of a company and its plans. Any financial service delivered to a company by a financial institution or other investor gives a tacit approval of their activities. To make nuclear weapons, you need money. Governments pay for a lot of things, but the companies most heavily involved in producing key components for nuclear warheads need additional investment — from banks, pension funds, and insurance companies — to sustain the working capital they need to maintain and modernize nuclear bombs.

We can steer these companies in a new direction. We can influence their decision making, by making sure our own investments don’t go anywhere near nuclear weapon producing companies. Choosing to avoid investment in controversial items or the companies that make them — from tobacco to nuclear arms — can result in changed policies and reduces the chances of humanitarian harm. Just as it wasn’t smokers that got smoking banned indoors across the planet, it’s not likely that the nuclear armed countries will show the normative leadership necessary to cut off the flow of money to their nuclear bomb producers.

Public exclusions by investors have a stigmatizing effect on companies associated with illegitimate activities. There are lots of examples from child labor to tobacco where financial pressure had a profound impact on industry. While it is unlikely that divestment by a single financial institution or government would enough for a company to cancel its nuclear weapons associated contracts, divestment by even a few institutions, or countries, for the same reason can affect a company’s strategic direction.

It’s worked before.

Divestment, and legal imperatives to divest are powerful tools to compel change. The divestment efforts in the 1980s around South Africa are often cited as having a profound impact on ending the Apartheid Regime. Global efforts divesting from tobacco stocks, have not ended the production or sale of tobacco products, but have compelled the producing companies to significantly modify behaviors — and they’ve helped to delegitimize smoking.

According to a 2013 report by Oxford University “in almost every divestment campaign … from adult services to Darfur, tobacco to Apartheid, divestment campaigns were effective in lobbying for restricting legislation affecting stigmatized firms.” The current global fossil fuel divestment campaign is mobilizing at all levels of society to stigmatize relationships with the fossil fuel industry resulting in divestment by institutions representing over $3.4 trillion in assets, and inspiring investment towards sustainable energy solutions.

US company Lockheed Martin, which describes itself as the worlds largest arms manufacturer, announced it ceased its involvement with the production of rockets, missiles or other delivery systems for cluster munitions and stated it will not accept such orders in the future. The arms manufacturer expressed the hope that its decision to cease the activities in the area of cluster munitions would enable it to be included in investors portfolios again, thereby suggesting that pressure by financial institutions had something to do with its decision.

In Geneva right now, governments are meeting to discuss new legal measures to deal with the deadliest weapons. The majority of governments want action- and want it now. Discussions are taking place about negotiating new legal instruments — new international law about nuclear weapons. The majority of the world’s governments are calling for a comprehensive new treaty to outlaw nuclear weapons.

And they’re talking about divestment too. For example, the Ambassador from Jamaica said:

“A legally-binding instrument on prohibition of nuclear weapons would also serve as a catalyst for the elimination of such weapons. Indeed, it would encourage nuclear weapon states and nuclear umbrella states to stop relying on these types of weapons of mass destruction for their perceived security. Another notable impact of a global prohibition is that it would encourage financial institutions to divest their holdings in nuclear weapons companies.”

Governments are talking about divestment, and it’s something you can do too.

If you have a bank account, find out if your bank invests in nuclear weapon producing companies. You can either look at our website and see if your bank is listed, or you can ask your bank directly. We found that a few people, asking the same bank about questionable investments, was enough to get that bank to adopt a policy preventing them from having any relationship with nuclear weapon producing companies.

Anyone, no matter where they are can have some influence over nuclear weapons decision making. From the heads of government to you from your very own pocket — everyone can do something about this issue. It doesn’t take a lot of time, or money, to make a difference, but it does take you. Together we can stop the scary threat of massive nuclear violence. If you want to help end the threat of nuclear weapons, then put your money where your mouth is, and Don’t Bank on the Bomb.

A Call for Russia and the U.S. to Cooperate in Protecting Against Nuclear Terrorism

The following post was written by Former Secretary of Defense William J. Perry and California Governor Jerry Brown as part of our Huffington Post series on nuclear security.

We believe that the likelihood of a nuclear catastrophe is greater today than it was during the Cold War. In the Cold War our nation lived with the danger of a nuclear war starting by accident or by miscalculation. Indeed, the U.S. had three false alarms during that period, any one of which might have resulted in a nuclear war, and several crises, including the Cuban Missile Crisis, which could have resulted in a nuclear war from a miscalculation on either side.

When the Cold War ended, these dangers receded, but with the increasing hostility between the U.S. and Russia today, they are returning, endangering both of our countries. In addition to those old dangers, two new dangers have arisen—nuclear terrorism, and the possibility of a regional nuclear war. Neither of those dangers existed during the Cold War, but both of them are very real today. In particular, the prospect of a nuclear terror attack looms over our major cities today.

Both Al Qaeda and ISIL have tried to acquire nuclear weapons, and no one should doubt that if they succeeded they would use them. Because the security around nuclear weapons is so high, it is unlikely (but not impossible) that they could buy or steal a nuclear bomb. But if they could obtain some tens of kilograms of highly enriched uranium (HEU), they could make their own improvised nuclear bomb. A significant quantity of HEU is held by civilian organizations, with substantially lower security than in military facilities. Recognizing this danger, President Obama initiated the Nuclear Security Summit meetings, whose objective was to eliminate fissile material not needed, and to provide enhanced security for the remainder.

That program—involving the leaders of over 50 nations that possessed fissile material, has been remarkably successful. In 1992, 52 countries had weapons-usable nuclear material; in 2010, the year of the first Summit, that number stood at 35. Just six years later, we are down to 24, as 11 more countries have eliminated their stocks of highly enriched uranium and plutonium. Additionally, security has been somewhat improved for the remaining material. But progress has stalled, much more remains to be done, and the danger of a terror group obtaining fissile material is still unacceptably high.

A quantity of HEU the size of a basketball would be sufficient to make an improvised nuclear bomb that had the explosive power of the Hiroshima bomb and was small enough to fit into a delivery van. Such a bomb, delivered by van (or fishing boat) and detonated in one of our cities, could essentially destroy that city, causing hundreds of thousands of casualties, as well as major social, political, and economic disruptions.

The danger of this threat is increasing every day; indeed, we believe that our population is living on borrowed time. If this catastrophe were allowed to happen, our society would never be the same. Our political system would respond with frenzied actions to ensure that it would not happen again, and we can assume that, in the panic and fear that would ensue, some of those actions would be profoundly unwise. How much better if we took preventive measures now—measures that increase our safety while still preserving our democracy and our way of life.

Two actions cry out to be taken. One is the international effort to improve the security of fissile material. The Nuclear Security Summits have made a very good start in that direction, but they are now over, and the pressure to reduce supplies of fissile material and improve security for the remainder predictably will falter. It is imperative to keep up this pressure, either through continuing summits, or through an institutional process that would be created by the nations that attended the summits and that would be managed by the Disarmament Agency of the UN, which would be given additional powers for that purpose. The U.S. should take the lead to ensure that a robust follow-on program is established.

Beyond that, and perhaps even more importantly, the U.S. and Russia, the nations that possess 90 percent of the world’s fissile material, should work closely together, including cooperation in intelligence about terror groups, to ensure that a terror group never obtains enough material to destroy one of their cities. After all, these two nations not only possess most of the fissile material, they are also the prime targets for a terror attack. Moscow and St. Petersburg are in as great a danger as Washington, D.C. and New York City.

Sen. Sam Nunn has proposed that Russia and the U.S. form a bilateral working group specifically charged with outlining concrete actions they could take that would greatly lessen the danger of Al Qaeda or ISIL obtaining enough fissile material to make improvised nuclear bombs. Whatever disagreements exist between our two countries—and they are real and serious—certainly we could agree to work together to protect our cities from destruction.

If our two countries were successful in cooperating in this important area, they might be encouraged to cooperate in other areas of mutual interest, and, in time, even begin to work to resolve other differences. The security of the whole world would be improved if they could do so.

Even with these efforts, we cannot be certain that a terror group could not obtain fissile material. But we can greatly lower that probability by taking responsible actions to protect our societies. If a nuclear bomb were to go off in one of our cities, we would move promptly to take actions that could prevent another attack. So why not do it now? Timely action can prevent the catastrophe from occurring, and can ensure that the preventive actions we take are thoughtful and do not make unnecessary infringements on our civil liberties.

What President Obama Should Say When He Goes to Hiroshima

The following post was written by David Wright and Lisbeth Gronlund as part of our Huffington Post series on nuclear security. Gronlund and Wright are both Senior Scientists and Co-Directors of the Global Security Program for the Union of Concerned Scientists.

Yesterday the White House announced that President Obama will visit Hiroshima — the first sitting president to do so — when he is in Japan later this month.

He will give a speech at the Hiroshima Peace Memorial Park, which commemorates the atomic bombing by the United States on August 6, 1945.

According to the president’s advisor Ben Rhodes, Obama’s remarks “will reaffirm America’s longstanding commitment — and the President’s personal commitment — to pursue the peace and security of a world without nuclear weapons. As the President has said, the United States has a special responsibility to continue to lead in pursuit of that objective as we are the only nation to have used a nuclear weapon.”

Obama gave his first foreign policy speech in Prague in April 2009, where he talked passionately about ending the threat posed by nuclear weapons. He committed the United States to reducing the role of nuclear weapons in its national security policy and putting an end to Cold War thinking.

A speech in Hiroshima would be a perfect bookend to his Prague speech — but only if he uses the occasion to announce concrete steps he will take before he leaves office. The president must do more than give another passionate speech about nuclear disarmament. The world needs — indeed, is desperate for — concrete action.

Here’s what Mr. Obama should say in Hiroshima:

 

***

 

Thank you for your warm welcome.

I have come to Hiroshima to do several things. First, to recognize those who suffered the humanitarian atrocities of World War II throughout the Pacific region.

Second, to give special recognition to the survivors of the atomic bombings of Hiroshima and Nagasaki — the hibakusha — who have worked tirelessly to make sure those bombings remain the only use of nuclear weapons.

And third, to announce three concrete steps I will take as U.S. commander-in-chief to reduce the risk that nuclear weapons will be used again. These are steps along the path I laid out in Prague in 2009.

First, the United States will cut the number of nuclear warheads deployed on long-range forces below the cap of 1,550 in the New START treaty, down to a level of 1,000. This is a level, based on the Pentagon’s analysis, that I have determined is adequate to maintain U.S. security regardless of what other countries may do.

Second, I am cutting back my administration’s trillion-dollar plan to build a new generation of nuclear warheads, missiles, bombers, and submarines. I am beginning by canceling plans for the new long-range nuclear cruise missile, which I believe is unneeded and destabilizing.

Third, I am taking a step to eliminate one of the ultimate absurdities of our world: The most likely way nuclear weapons would be used again may be by mistake.

How is this possible? Let me explain.

Today the United States and Russia each keep many hundreds of nuclear-armed missiles on prompt-launch status — so-called “hair-trigger alert“ — so they can be launched in a matter of minutes in response to warning of an incoming nuclear attack. The warning would be based on data from satellites and ground-based radars, and would come from a computer.

This practice increases the chance of an accidental or unauthorized launch, or a deliberate launch in response to a false warning. U.S. and Russian presidents would have only about 10 minutes to decide whether the warning of an incoming attack was real or not, before giving the order to launch nuclear-armed missiles in retaliation — weapons that cannot be recalled after launch.

And history has shown again and again that the warning systems are fallible.Human and technical errors have led to mistakes that brought the world far too close to nuclear war. That is simply not acceptable. Accidents happen — they shouldn’t lead to nuclear war.

As a candidate and early in my presidency I recognized the danger and absurdity of this situation. I argued that “we should take our nuclear weapons off hair-trigger alert” because “keeping nuclear weapons ready to launch on a moment’s notice is a dangerous relic of the Cold War. Such policies increase the risk of catastrophic accidents or miscalculation.”

Former secretaries of defense as well as generals who oversaw the U.S. nuclear arsenal agree with me, as do science and faith leaders. In his recent book My Journey at the Nuclear Brink, former Secretary of Defense William Perry writes: “These stories of false alarms have focused a searing awareness of the immense peril we face when in mere minutes our leaders must make life-and-death decisions affecting the whole planet.”

General James Cartwright, former commander of U.S. nuclear forces, argues that cyber threats that did not exist during the Cold War may introduce new system vulnerabilities. A report he chaired last year states that “In some respects the situation was better during the Cold War than it is today. Vulnerability to cyber-attack … is a new wild card in the deck.”

And the absurdity may get even worse: China’s military is urging its government to put Chinese missiles on high alert for the first time. China would have to build a missile warning system, which would be as fallible as the U.S. and Russian ones. The United States should help Chinese leaders understand the danger and folly of such a step.

So today I am following through on my campaign pledge. I am announcing that the United States will take all of its land-based missiles off hair-trigger alert and will eliminate launch-on-warning options from its war plans.

These steps will make America — and the world — safer.

Let me end today as I did in Prague seven years ago: “Let us honor our past by reaching for a better future. Let us bridge our divisions, build upon our hopes, accept our responsibility to leave this world more prosperous and more peaceful than we found it. Together we can do it.”

Passing the Nuclear Baton

The following post was written by Joe Cirincione, President of the Ploughshares Fund, as part of our Huffington Post series on nuclear security.

President Obama entered office with a bold vision, determined to end the Cold War thinking that distorted our nuclear posture. He failed. He has a few more moves he could still make — particularly with his speech in Hiroshima later this month — but the next president will inherit a nuclear mess.

Obama had the right strategy. In his brilliant Prague speech, he identified our three greatest nuclear threats: nuclear terrorism, the spread of nuclear weapons to new states and the dangers from the world’s existing nuclear arsenals. He detailed plans to reduce and eventually eliminate all three, understanding correctly that they all must be tackled at once or progress would be impossible on any.

Progress Thwarting Nuclear Terror

Through his Nuclear Security Summits, Obama created an innovative new tool to raise the threat of nuclear terrorism to the highest level of global leadership and inspire scores of voluntary actions to reduce and secure nuclear materials. But it is, as The New York Times editorialized, “a job half done.” Instead of securing all the material in four years as originally promised, after eight years we still have 1,800 tons of bomb-usable material stored in 24 countries, some of it guarded less securely than we guard our library books.

If a terrorist group could get their hands on just 100 pounds of enriched uranium, they could make a bomb that could destroy a major city. In October of last year, anAP investigation revealed that nuclear smugglers were trying to sell weapons grade uranium to ISIS. Smugglers were overheard on wiretaps as saying that they wanted to find an ISIS buyer because, “they will bomb the Americans.”

More recently, we learned that extremists connected to the attacks in Paris and Belgium had also been videotaping a Belgian nuclear scientist, likely in the hopes of forcing “him to turn over radioactive material, possibly for use in a dirty bomb.”

Obama got us moving in the right direction, but when you are fleeing a forest fire, it is not just a question of direction but also of speed. Can we get to safety before catastrophe engulfs us?

Victory on Iran

His greatest success, by far, has been the agreement with seven nations that blocks Iran’s path to a bomb. This is huge. There are only two nations in the world with nuclear programs that threatened to become new nuclear-armed states: Iran and North Korea. North Korea has already crossed the nuclear Rubicon and we must struggle to see if we can contain that threat and even push them back. Thanks to the Iran agreement however, Iran can now be taken off the list.

For this achievement alone, Obama should get an “A” on his non-proliferation efforts. He is the first president in 24 years not to have a new nuclear nation emerge on his watch.

Bill Clinton saw India and Pakistan explode into the nuclear club in 1998. George W. Bush watched as North Korea set off its first nuclear test in 2006. Barack Obama scratched Iran from contention. Through negotiations, he reduced its program to a fraction of its original size and shrink-wrapped it within the toughest inspection regime ever negotiated. It didn’t cost us a dime. And nobody died. It is, by any measure, a major national security triumph.

Failure to Cut

Unfortunately Obama could not match these gains when it came to the dangers posed by the existing arsenals. The New START Treaty he negotiated with Russia kept alive the intricate inspection procedures previous presidents had created, so that each of the two nuclear superpowers could verify the step-by-step reduction process set in motion by Ronald Reagan and continued by every president since.

That’s where the good news ends. The treaty made only modest reductions to each nation’s nuclear arsenals. The United States and Russia account for almost 95 percent of all the nuclear weapons in the world, with about 7,000 each. The treaty was supposed to be a holding action, until the two could negotiate much deeper reductions. That step never came.

The “Three R’s” blocked the path: Republicans, Russians and Resistance.

First, the Republican Party leadership in Congress fought any attempt at reductions. Though many Republicans supported the treaty, including Colin Powell, George Shultz and Senator Richard Lugar, the entrenched leadership did not want to give a Democratic president a major victory, particularly in the election year of 2010. They politicized national security, putting the interest of the party over the interest of the nation. It took everything Obama had to finally get the treaty approved on the last day of the legislative session in December.

By then, the president’s staff had seen more arms control then they wanted, and the administration turned its attention to other pressing issues. Plans to “immediately and aggressively” pursue Senate approval of the nuclear test ban treaty were shelved and never reconsidered. The Republicans had won.

Worse, when Russia’s Vladimir Putin returned to power, Obama lost the negotiating partner he had had in President Medvedev. Putin linked any future negotiation to a host of other issues, including stopping the deployment of US anti-missile systems in eastern Europe, cuts in conventional forces and limits on long-range conventional strike systems the Russian claimed threatened their strategic nuclear forces. Negotiations never resumed.

Finally, he faced resistance from the nuclear industrial complex, including many of those he himself appointed to implement his policies. Those with a vested financial, organizational or political interest in the thousands of contracts, factories, bases and positions within what is now euphemistically call our “nuclear enterprise” will do anything they can to preserve those dollars, contracts and positions. Many of his appointees merely paid lip-service to the president’s agenda, paying more attention to the demands of the services, or the contractors or their own careers. Our nuclear policy is now less determined by military necessity or strategic doctrine, than by self-interest.

It is difficult to find someone who supports keeping our obsolete Cold War arsenal that is not directly benefiting from, or beholden to, these weapons. In a very strange way, the machines we built are now controlling us.

The Fourth Threat

To make matters worse, under Obama’s watch these three “traditional” nuclear threats have been joined by a fourth: nuclear bankruptcy.

Obama pledged in Prague that as he reduced the role and number of nuclear weapons in U.S. policy, he would maintain a “safe, secure and reliable” arsenal. He increased spending on nuclear weapons, in part to make much needed repairs to a nuclear weapons complex neglected under the Bush administration and, in part, to win New START votes from key senators with nuclear bases and labs in their states.

As Obama’s policy faltered, the nuclear contracts soared. The Pentagon has embarked on the greatest nuclear weapons spending spree in U.S. history. Over the next 30 years the Pentagon is planning to spend at least $1 trillion on new nuclear weapons. Every leg of the U.S. nuclear triad – our fleet of nuclear bombers, ballistic missile submarines, and ICBMs – will be completely replaced by a new generation of weapons that will last well into the later part of this century. It is a new nuclear nightmare.

What Should the Next President Do?

While most of us have forgotten that nuclear weapons still exist today, former Secretary of Defense Bill Perry warns that we “are on the brink of a new nuclear arms race” with all the perils, near-misses and terrors you thought ended with the Cold War. The war is over; the weapons live on.

The next president cannot make the mistake of believing that incremental change in our nuclear policies will be enough to avoid disaster. Or that appointing the same people who failed to make significant change under this administration, will somehow help solve the challenges of the next four years. There is serious work to be done.

We need a new plan to accelerate the elimination of nuclear material. We need a new strategy for North Korea. But most of all, we need a new strategy for America. It starts with us. As long as we keep a stockpile of nuclear weapons far in excess of any conceivable need, how can we convince other nations to give up theirs?

The Joint Chiefs told President Obama that he could safely cut our existing nuclear arsenal and that we would have more than enough weapons to fulfill every military mission. It did not matter what the Russians did. If they cut or did not cut, honored the New START Treaty or cheated. We could still cut down to about 1000 to 1100 strategic weapons and still handle every contingency.

The next president should do that. Not just because it is sound strategic policy – but because it is essential financial policy too. We are going broke. We do not have enough money to pay for all the weapons the Pentagon ordered when they projected ever-rising defense budgets. “There’s a reckoning coming here,” warns Rep. Adam Smith, the ranking Democrat on the House Armed Services Committee. “Do we really need the nuclear power to destroy the world six, seven times?”

The Defense Department admits it does not have the money to pay for these plans. Referring to the massive ‘bow wave‘ of spending set to peak in the 2020s and 2030s, Pentagon Comptroller Mike McCord said “I don’t know of a good way for us to solve this issue.”

In one of more cynical admissions by a trusted Obama advisor, Brian McKeon, the principal undersecretary of defense for policy, said last October, “We’re looking at that big [nuclear] bow wave and wondering how the heck we’re going to pay for it,” And we’re “probably thanking our stars we won’t be here to have to answer the question,” he added with a chuckle.

He may think it’s funny now, but the next president won’t when the stuff hits the fan in 2017. One quick example: The new nuclear submarines the Navy wants will devour half of the Navy’s shipbuilding budget in the next decade. According to the Congressional Research Service, to build 12 of these new subs, “the Navy would need to eliminate… a notional total of 32 other ships, including, notionally, 8 Virginia-class attack submarines, 8 destroyers, and 16 other combatant ships.”

These are ships we use every day around the world on real missions to deal with real threats. They launch strikes against ISIS, patrol the South China Sea, interdict pirates around the horn of Africa, guarantee the safety of international trade lanes, and provide disaster relief around the globe.

The conventional navy’s mission is vital to international security and stability. It is foolish, and dangerous, to cut our conventional forces to pay for weapons built to fight a global thermonuclear war.

Bottom-Up

The next President could do a bottom-up review of our nuclear weapons needs. Don’t ask the Pentagon managers of these programs what they can cut. You know the answer you will get. Take a blank slate and design the force we really need.

Do we truly need to spend $30 billion on a new, stealthy nuclear cruise missile to put on the new nuclear-armed stealth bomber?

Do we truly need to keep 450 intercontinental ballistic missiles, whose chief value is to have the states that house them serve as targets to soak up so many of the enemy’s nuclear warheads that it would “complicate an adversary’s attack plans?” Do Montana and Wyoming and North Dakota really want to erect billboards welcoming visitors to “America’s Nuclear Sponge?”

If President Trump, or Clinton, or Sanders put their trust in the existing bureaucracy, it will likely churn out the same Cold War nuclear gibberish. It will be up to outside experts, scientists, retired military and former diplomats to convince the new president to learn from Obama’s successes and his failures.

Obama had the right vision, the right strategy. He just didn’t have an operational plan to get it all done. It is not that hard, if you have the political will.

Over to you next POTUS.