Joshua Greene Interview

The following is an interview with Joshua Greene about the Beneficial AI 2017 conference and The Asilomar Principles that it produced. Greene is an experimental psychologist, neuroscientist, and philosopher. He studies moral judgment and decision-making, primarily using behavioral experiments and functional neuroimaging (fMRI). Other interests include religion, cooperation, and the capacity for complex thought. He is the author of Moral Tribes: Emotion, Reason, and the Gap Between Us and Them.

ARIEL: “The idea behind this is the principles were a start. There were criticisms and comments about them, and we wanted to have a discussion about each individual principle to better understand what’s important to people.”

JOSHUA: “Yes, in crafting these principles, the challenge is to make them general enough that there can be some kind of an agreement, but specific and substantive enough that they’re not just completely empty.

“They address a deep moral tension. I think of people’s values as lying along a continuum. There are individualist values, where it’s really just all about me and my rights and my freedom. In the middle we have tribalist values, where I care not just about me, but my group. And then at the other end you have universalist values. If you ask a lot of people in our world, they’ll say, ‘Oh, of course I’m a universalist.’

“But they mean that only up to a point. They would say, ‘Sure, we shouldn’t do anything that would be terrible for all of humanity, but do I have an obligation to give a significant portion of my money to charity? Well, no, that’s a matter of personal preference.’ There’s a challenging, recurring moral question about the extent to which certain valuable resources ought to be common versus private.

“I see these principles as saying, ‘the incredible power of forthcoming artificial intelligence is not just another resource to be controlled by whoever gets there first, whoever gets the patent, whoever has the code. It is a common good that belongs to everybody.’ We may think about this the way many of us think about environmental issues or healthcare. Many of us believe that people have a moral claim to basic healthcare that goes beyond people’s claims to ordinary consumer goods, which they may or may not be able to afford. Likewise, when it comes to the quality of the Earth’s atmosphere or clean water and other environmental resources, many of us think of this as part of our collective endowment as a species, or perhaps as the living things on this particular planet. It’s not just who gets there first and wants to liquidate these assets; it’s not just whatever political structure happens to be in place that should determine what happens to the environment.

“I think what’s valuable and substantive in these principles is the idea of declaring in advance, ‘we don’t think that AI should be just another resource to be allocated according to the contingencies of history, of who happens to be on top when the power emerges, whose lab it happens to come out of, who happens to get the patent. This is part of our story as a species, the story of the evolution of complexity and intelligence on our planet, and we think that this should be understood as a common resource, as part of our endowment as humanity.’”

ARIEL: “As someone who’s looked at that spectrum you were talking about, do you see ways of applying AI so that it’s benefiting people universally? I would be concerned that whoever develops it is going to also have sort of an ‘I’m the most important’ attitude.”

JOSHUA: “Yeah, humans tend to do that! And I think that’s the big worry. I’ve been thinking a lot about [John] Rawls’ and [John] Harsanyi’s idea of a ‘veil of ignorance’. One of the key ideas in Rawls’s theory of justice and Harsanyi’s foundational defense of rule utilitarianism is that a fair outcome is one that you would choose if you didn’t know who you were going to be. The nice thing right now is that as much as we might place bets on some countries and firms rather than others, we really don’t know where this power is going to land first. (Nick Bostrom makes the same point in Superintelligence.)

“We have this opportunity now to lay down some principles about how this should go before we know who the winners and losers are, before we know who would benefit from saying, ‘Actually, it’s a private good instead of a public good.’ I think what’s most valuable about this enterprise is not just the common good principle itself—which is fairly straightforward. It’s the idea of getting that idea and that expectation out there before anyone has a very strong, selfish interest in flouting it. Right now, most of us comfortably say that we would like humanity’s most powerful creation to be used for the common good. But as soon as that power lands in somebody’s hands, they might feel differently about that. To me, that’s the real practical significance of what we’re doing: establishing a set of norms, a culture, and set of expectations while the veil is still mostly on.”

Q. Explain what you think of the following principles:

5) Race Avoidance: Teams developing AI systems should actively cooperate to avoid corner-cutting on safety standards.

“Cutting corners on safety is essentially saying, ‘My private good takes precedence over the public good.’ Cutting corners on safety is really just an act of selfishness. The only reason to race forward at the expense of safety is if you think that the benefits of racing disproportionately go to you. It’s increasing the probability that people in general will be harmed—a “common bad”, if you like—in order to raise the probability of a private good for oneself.”

6) Safety: AI systems should be safe and secure throughout their operational lifetime and verifiably so where applicable and feasible.

“Yeah, I think that one’s kind of a no-brainer – not that there’s anything wrong with saying it. It’s the kind of thing that’s good to be reminded of, but no one’s saying, ‘No, I don’t think they should be safe through their whole lifetime, just part of it.’”

ARIEL: “I think sometimes researchers get worried about how technically feasible some of these are.”

JOSHUA: “I guess it depends what you mean by ‘verifiably.’ Does verifiably mean mathematically, logically proven? That might be impossible. Does verifiably mean you’ve taken some measures to show that a good outcome is most likely? If you’re talking about a small risk of a catastrophic outcome, maybe that’s not good enough.

“Like all principles in this domain, there’s wiggle room for interpretation. I think that’s just how it has to be. Good principles are not ones that are locked down and not open to interpretation. Instead, they signal two or more things that need to be balanced. This principle says, ‘Yes, they need to be safe, but we understand that you may never be able to say never.’”

9) Responsibility: Designers and builders of advanced AI systems are stakeholders in the moral implications of their use, misuse, and actions with a responsibility and opportunity to shape those implications.

“I think that’s an important one because it’s very easy for people to say, ‘Making sure this doesn’t go bad is someone else’s problem.’ There’s a general problem of diffusion of responsibility. The engineers can say, ‘It’s not my job to make sure that this thing doesn’t hurt people. It’s my job to make sure that it works,’ and management says, ‘It’s not my job. My job is to meet our goals for the next quarterly report.’ Then they say it’s the lawmakers’ decision. And the lawmakers say, ‘Well, it’s this commission’s decision.’ Then the commission is filled with people from corporations whose interests may not be aligned with those of the public.

“It’s always someone else’s job. Saying designers and builders have this responsibility, that’s not trivial. What we’re saying here is the stakes are too high for anyone who’s involved in the design and building of these things to say safety is someone else’s problem.”

14) Shared Benefit: AI technologies should benefit and empower as many people as possible.

“I think that’s basically another version of the common good principle. We’re saying in advance, before we know who really has it, that this is not a private good. It will land in the hands of some private person, it will land in the hands of some private company, it will land in the hands of some nation first. But this principle is saying, ‘It’s not yours.’ That’s an important thing to say because the alternative is to say that potentially, the greatest power that humans ever develop belongs to whoever gets it first.”

16) Human Control: Humans should choose how and whether to delegate decisions to AI systems to accomplish human-chosen objectives.

“This is an interesting one because it’s not clear what it would mean to violate that rule. What kind of decision could an AI system make that was not in some sense delegated to the system by a human? AI is a human creation. This principle, in practice, is more about what specific decisions we consciously choose to let the machines make. One way of putting it is that we don’t mind letting the machines make decisions, but whatever decisions they make, we want to have decided that they are the ones making those decisions.

“In, say, a navigating robot that walks on legs like a human, the person controlling it is not going to decide every angle of every movement. The humans won’t be making decisions about where exactly each foot will land, but the humans will have said, ‘I’m comfortable with the machine making those decisions as long as it doesn’t conflict with some other higher level command.’

“The worry is when you have machines that are making more complicated and consequential decisions than where do to put the next footstep. When you have a machine that can behave in an open-ended flexible way, how do you delegate anything without delegating everything? When you have someone who works for you and you have some problem that needs to be solved and you say, ‘Go figure it out,’ you don’t specify, ‘But don’t murder anybody in the process. Don’t break any laws and don’t spend all the company’s money trying to solve this one small-sized problem.’ There are assumptions in the background that are unspecified and fairly loose, but nevertheless very important.

“I like the spirit of this principle. It’s a specification of what follows from the more general idea of responsibility, that every decision is either made by a person or specifically delegated to the machine. But this one will be especially hard to implement once AI systems start behaving in more flexible, open-ended ways.”

ARIEL: “Is that a decision that you think each person needs to make, or is that something that a company can make when they’re designing it, and then when you buy it you implicitly accept?”

JOSHUA: “I think it’s a general principle about the choices of humans and machines that cuts across the choices of consumers and producers. Ideally there won’t be any unknown unknowns about what decisions a machine is making. Unknowns are okay, but as much as possible we’d like them to be known unknowns.”

1) Research Goal: The goal of AI research should be to create not undirected intelligence, but beneficial intelligence.

“I think that is a bookend to the common good principle – the idea that it’s not okay to be neutral. It’s not okay to say, ‘I just make tools and someone else decides whether they’re used for good or ill.’ If you’re participating in the process of making these enormously powerful tools, you have a responsibility to do what you can to make sure that this is being pushed in a generally beneficial direction. With AI, everyone who’s involved has a responsibility to be pushing it in a positive direction, because if it’s always somebody else’s problem, that’s a recipe for letting things take the path of least resistance, which is to put the power in the hands of the already powerful so that they can become even more powerful and benefit themselves.”

4) Research Culture: A culture of cooperation, trust, and transparency should be fostered among researchers and developers of AI.

“I see this as a practical distillation of the Asilomar Principles. They are not legally binding. At this early stage, it’s about creating a shared understanding that beneficial AI requires an active commitment to making it turn out well for everybody, which is not the default path. To ensure that this power is used well when it matures, we need to have already in place a culture, a set of norms, a set of expectations, a set of institutions that favor good outcomes. That’s what this is about – getting people together and committed to directing AI in a mutually beneficial way before anyone has a strong incentive to do otherwise.”

Join the discussion about the Asilomar AI Principles!

Susan Craw Interview

The following is an interview with Susan Craw about the Beneficial AI 2017 conference and The Asilomar Principles that it produced. Craw is a Research Professor at Robert Gordon University Aberdeen in Scotland. Her research in artificial intelligence develops innovative data/text/web mining technologies to discover knowledge to embed in case-based reasoning systems, recommender systems, and other intelligent information systems.

Q. Explain what you think of the following principles:

1) Research Goal: The goal of AI research should be to create not undirected intelligence, but beneficial intelligence.

“Yes, I agree with that. I think it’s a little strange the way it’s worded, because of ‘undirected.’ It might even be better the other way around, which is, it would be better to create beneficial research, because that’s a more well-defined thing. But yes, certainly many researchers are working on things that they believe are going to do good. For example, making companies more efficient, or helping people engage with others and engage with information, or just assisting them in some way finding information that they’d like to be able to access.”

4) Research Culture: A culture of cooperation, trust and transparency should be fostered among researchers and developers of AI.

“That would be a lovely principle to have, [but] it can work perhaps better in universities, where there is not the same idea of competitive advantage as in industry. So, I suppose I can unpack that and say: transparency I’m very much in favor of, because I think it’s really important that AI systems are able to explain what they’re doing and are able to be inspected as to why they’ve come up with a particular solution or recommendation.

“And cooperation and trust among researchers… well without cooperation none of us would get anywhere, because we don’t do things in isolation. And so I suspect this idea of research culture isn’t just true of AI. You’d like it to be true of many subjects that people study. Trusting that people have good governance of their research, and what they say they’ve done is a true reflection of what they are actually working on and have achieved.”

5) Race Avoidance: Teams developing AI systems should actively cooperate to avoid corner cutting on safety standards.

“It’s quite hard to cooperate, especially if you’re trying to race for the product, and I think it’s going to be quite difficult to police that, except, I suppose, by people accepting the principle. For me, safety standards are paramount and so active cooperation to avoid corner cutting in this area is even more important. But that will really depend on who’s in this space with you.”

6) Safety: AI systems should be safe and secure throughout their operational lifetime, and verifiably so where applicable and feasible.

“I suppose this one is more perhaps thinking of robots. You don’t want your robots running amok as they learn more or become devious, or something. But I guess it’s true of AI systems as well, software systems. And it is linked to ‘transparency.’ Maybe ‘verifiably so’ would be possible with systems if they were a bit more transparent about how they were doing things.”

14) Shared Benefit: AI technology should benefit and empower as many people as possible.

“That’s definitely a yes. But it is AI technologies plural, when it’s taken as a whole. Rather than saying that a particular technology should benefit lots of people, it’s that the different technologies should benefit and empower people.”

ARIEL: “So, in general, as long as we have this broad range of AI technology and it’s benefitting people, whether one or two individual technologies benefit everyone is less important? Is that how you’re viewing that?”

SUSAN: “Yes, because, after all, AI technologies can benefit you in your work because it makes you more efficient or less likely to make a mistake. And then there’s all the social AI technologies, where you are being helped to do things, and have social engagement with others and with information – both regulatory information but also social information.”

16) Human Control: Humans should choose how and whether to delegate decisions to AI systems to accomplish human-chosen objectives.

“I think this is hugely important, because otherwise you’ll have systems wanting to do things for you that you don’t necessarily want them to do, or situations where you don’t agree with the way that systems are doing something.

“In a meeting my research group had recently, I was shown the new Google Inbox for your Gmail, that is designed to organize your emails for you. And I’m not sure it’s something that I shall be adopting. But on the other hand, when you are using Google Maps and you find it knows about the hotel that you’re staying in for a trip, then that’s convenient, although it’s kind of scary that it’s able to work out all those things for you. It knows much more about you than you would perhaps believe. Instead of not knowing what the system can do, I’d much prefer to set things up and say, ‘I want you to do this for me.’”

ARIEL: “And then how do you feel about situations where safety is an issue? Even today, we have issues of pilots not necessarily flying the planes as well as the automation in the planes can. For me, in that case, it seems better to let the automated system take over from the human if necessary. But most of the time that’s not what I would want. Where do you draw the line?”

SUSAN: “With that one you would almost want it to be the other way around: by default, the automated system is in control, but the pilot could take over if necessary. And the same is true of autonomous cars. I was hearing something on the radio this morning in the UK, which was saying that if you keep the driver there, then the last thing that he wants to happen is that the automated system can’t cope with something, because he’s not likely to be paying that much attention to what the problem is.”

23) Common Good: Superintelligence should only be developed in the service of widely shared ethical ideals, and for the benefit of all humanity rather than one state or organization.

“Yes, the idea of ‘shared ethical ideals’. It’s the rogue state developing superintelligence, and whether that could be picked up as unethical behavior. It would then turn AI into something more like nuclear power – it could be a reason for taking action against a state or an organization.”

ARIEL: “Do you worry about that happening, or do you think, for the most part, if we develop superintelligence it will be safe?”

SUSAN: “I suppose I do worry about it, because rogue nuclear states do exist. So it is possible, but I think it needs a lot of collaboration and sharing to develop superintelligence. There could be surveillance of what’s happening in countries, and the development of things of that level could be ascertained. If you had a superintelligence arms race in the same way as the nuclear arms race, then the countries and organizations that are acting well would be able to keep up, because there are more of them and there’s more ability to share. [That] would be my hope.”

ARIEL: “Is there anything else you want to add about the principles, or is there anything we didn’t cover that you think is important that you wanted to comment on?”

SUSAN: “Yes, I read in someone’s commentary on this, and that was: was it wise to have so many principles? I think I might agree with that, but on the other hand, it seems that they may be quite detailed principles, and as such you need to have many of them. And as our discussion has shown, there are different interpretations for the principles depending on your interests and what you associate with certain words.”

ARIEL: “That’s one of the big reasons we wanted to do these interviews and try to get a lot of people talking about their interpretations. We view these principles as a starting point.”

SUSAN: “I actually have taken a big interest in this because I was at IJCAI in 2015 in Buenos Aires, where a lot of the discussions outside the actual talks were on this particular topic. And there was a panel because it was just when Elon Musk and various people had talked about the existential threat from AI. So it was very lovely to see the AI community jumping into action, and saying, ‘we haven’t made our voice heard enough, and we haven’t really talked about this, and we certainly haven’t talked about this in a way that people outside our close community can hear us.’ So this is just another way of promoting these ideas, and I think this is hugely important, because I don’t think the AI community particularly publicizes its faults on these issues.”

Join the discussion about the Asilomar AI Principles!

John C. Havens Interview

The following is an interview with John C. Havens about the Beneficial AI 2017 conference and The Asilomar Principles that it produced. Havens is the Executive Director of The IEEE Global Initiative for Ethical Considerations in Artificial Intelligence and Autonomous Systems. He is the author of Heartificial Intelligence: Embracing Our Humanity to Maximize Machines and Hacking H(app)iness – Why Your Personal Data Counts and How Tracking It Can Change the World, and previously worked as the founder of both The H(app)athon Project and Transitional Media.

Q. Explain what you think of the following principles:

1) Research Goal: The goal of AI research should be to create not undirected intelligence, but beneficial intelligence.

“I love the word ‘beneficial.’ I think sometimes inherently people think that intelligence, in one sense, is always positive. Meaning, because something can be intelligent, or autonomous, and that can advance technology, that that is a ‘good thing’. Whereas the modifier ‘beneficial’ is excellent, because you have to define: What do you mean by beneficial? And then, hopefully, it gets more specific, and it’s: Who is it beneficial for? And, ultimately, what are you prioritizing? So I love the word beneficial.”

4) Research Culture: A culture of cooperation, trust and transparency should be fostered among researchers and developers of AI.

“I love the sentiment of it, and I completely agree with it. By the way, I should say, I love how these principles provide these one-line, excellent, very pragmatic ideas. So I want to make that as a preface. But, that said, I think defining what a culture of cooperation, trust, and transparency is… what does that mean? Where the ethicists come into contact with the manufacturers, there is naturally going to be the potential for polarization, where people on the creation side of the technology feel their research or their funding may be threatened. And on the ethicists or risk or legal compliance side, they feel that the technologists may not be thinking of certain issues. However, in my experience, the ethicists – I’m being very general, just to make a point – but the ethicists, etc., or the risk and compliance folks may be tasked with a somewhat outdated sense of the word ‘safety.’ Where, for instance, I was talking to an engineer the other day who was frustrated because they were filling out an IRB type form, and the question was asked: Could this product, this robotic product, be used in a military purpose. And when he really thought about it, he had to say yes. Because sure, can a shovel be used in a military purpose? Sure!

“I’m not being facetious; the ethicist, the person asking was well intended. And what they probably meant to say, and this is where accountability, and the certification, and these processes – as much as I know people don’t love processes, but it’s really important – is you build that culture of cooperation, trust, and transparency when both sides say, as it were, ‘Here’s the information we really need to progress our work forward. How do we get to know what you need more, so that we can address that well with these questions?’ You can’t just say, ‘Let’s develop a culture of cooperation, trust, and transparency.’ How do you do it? This sentence is great, but the next sentence should be: Give me a next step to make that happen.”

5) Race Avoidance: Teams developing AI systems should actively cooperate to avoid corner cutting on safety standards.

“I couldn’t agree more, not just because I’m working with the IEEE Standards Association – full disclosure – but we have to re-invent, we have to help people re-imagine what safety standards mean. If it’s this sort of onerous thing of checklists for compliance, then of course people are going to try to cut corners, because it’s time-consuming and boring. That’s what the impression is; I’m not saying that’s accurate. But if by going over safety, you’re now asking: What is my AI system? How will it interact with end users or stakeholders in the supply chain touching it and coming into contact with it, where there are humans involved, where it’s system to human vs. system to system? Safety is really about asking about people’s values. It’s not just physical safety, it’s also: What about their personal data, what about how they’re going to interact with this? So the reason you don’t want to cut corners is you’re also cutting innovation. You’re cutting the chance to provide a better product or service, because the word ‘safety’ in and of itself should now be expanded in the AI world to mean emotional and wellbeing safety for individuals, where then you’re going to discover all these wonderful ways to build more trust with what you’re doing when you take the time you need to go over those standards.”

ARIEL: “I like that, and I guess I haven’t thought about that. How do you convince people that safety is more than just physical safety?”

JOHN: “Sure. Think of an autonomous vehicle. Right now understandably the priority is: How do we make sure this thing doesn’t run into people, right? Which is a good thing to think about. But I’m going to choose one vehicle, in the future, over another, because I have been given proof that the vehicle does not harvest my physiological and facial and eye tracking data when I get in that car. The majority of them do. Physical safety is one issue, but the safety of how my data is transferred is critical. Sometimes it could be life and death, you know, what if I have a medical condition and the car reads it incorrectly and all that? But it’s also things like, how do we want the data revealed about where we are and about our health, to which actors, and when. So, that’s an example. Where, again, I’m using a larger scope for safety, but it really is important where, especially, we’re moving into a virtual realm, where safety is also about mental health safety. Meaning, if you wear, say, a Facebook Oculus Rift. A lot of people are saying social VR is the future. You’ll check Facebook while you’re in virtual reality. How you’re presented with Facebook stuff right now, not just ads, but posts can be really depressing, right? Depending on the time and place you look at it. That’s not Facebook’s fault, by the way, but the way that those things are presented, the algorithms etc., what they choose can be of their design. And so in terms of mental safety, mental wellbeing, it is also a really critical issue to think about right now.”

6) Safety: AI systems should be safe and secure throughout their operational lifetime, and verifiably so where applicable and feasible.

“Yes. Although I don’t know who wouldn’t say AI systems shouldn’t be safe and secure. So I would say words that further explain ‘safe and secure’ would be great. Meaning, AI systems that are physically safe, that provide increased wellbeing, whatever. And ‘secure throughout their operational lifetime’: I think what’s interesting to me is, ‘throughout their operational lifetime’ is actually the more important part of the sentence, because that’s about sustainability and longevity.

“And my favorite part of the sentence is ‘and verifiably so.’ That is critical. Because that means, even if you and I don’t agree on what ‘safe and secure’ means, but we do agree on verifiability, then you can go, ‘well, here’s my certification, here’s my checklist.’ And I can go, ‘Great, thanks.’ I can look at it, and say, ‘oh, I see you got things 1-10, but what about 11-15?’ Verifiably is a critical part of that sentence.”

ARIEL: “Considering that, what’s your take on ‘applicable and feasible?’”

JOHN: “I think they’re modifiers that kind of destroy the sentence. It’s like, ‘oh, I don’t feel like being applicable, that doesn’t matter here, because that’s personal data, and, you know, based on the terms and conditions.’ Or feasible, you know, ‘it’s an underwater system, it’s going to be too hard to reach in the water.’ ‘Safe and secure where applicable and feasible’ – you have those words in there, and I feel like anyone’s going to find a problem with every single thing you come up with. So I would lose those words if you want them to be more powerful.”

9) Responsibility: Designers and builders of advanced AI systems are stakeholders in the moral implications of their use, misuse, and actions, for the responsibility and opportunity to shape those implications.

“I like the phrase ‘stakeholders in the moral implications.’ But I think you have to expand beyond the ‘designers and builders’ to say the ‘designers, builders,’ and – however you’d wordsmith this – ‘the organizations or manufacturers of advanced AI systems.’ Because a lot of times, with engineers, what we’ve been finding is that you’re systematically handed blueprints to build something, and so then you are a stakeholder, but you’re not really in control because someone said ‘build this, and you have to do it because I’m telling you to do it, and we have to make our quarterly numbers.’ So, if you have ethical reservations or moral reservations, you can either whistleblow or quit. That’s kind of where we are. So it has to be enlarged, and the responsibility has to fall on the shareholders, the manufacturers, however you want to phrase that, so that it puts responsibility on the whole organization, not just the people whose hands actually touch and build the AI systems. Does that make sense?”

ARIEL: “Yeah, actually this example makes me think of the Manhattan Project, where people didn’t even know what they were doing. I don’t know how often engineers find themselves in a situation where they are working on something and don’t actually know what they’re working on.”

JOHN: “Not being an engineer, I’m not sure. It’s a great point, but I also think then the responsibility falls back to the manufacturers. I understand secrecy, I understand IP… I think there’d have to be some kind of public, ethical, like the Google DeepMind board, which, I know, still doesn’t exist… But if there was some public way of saying, as a company, ‘This is our IP. We’re going to have these engineers who are under our employment produce something and then be completely not responsible, from a legal standpoint, for what they create…’ But that doesn’t happen, unless they’re a private contractor, and they’d still be responsible.

“So, I don’t know. It’s a tough call, but I think it’s a cop-out to say, ‘Well the engineers didn’t know what they were building.’ That means you don’t trust them enough to tell them, you’re trying to avoid culpability and risk, and it means engineers, if they do build something, it’s kind of A or B. They don’t know what they’re building, and it turns out to be horrible in their mind, and they feel really guilty. [Or] they do know what they’re building, and they can’t do anything about it. So, it’s a situation that needs to evolve.”

14) Shared Benefit: AI technology should benefit and empower as many people as possible.

“Yes, it’s great. I think if you can put a comma after it, and say, ‘many people as possible,’ something like, ‘issues of wealth, GDP, notwithstanding,’ the point being, what this infers is whatever someone can afford, it should still benefit them. But a couple sentences maybe about the differences between the developed and not developed countries would be really interesting, because I certainly support the idea of it, but realistically right now, GDP drives the manufacture of most things. And GDP is exponential growth, and that favors the companies that can afford it. Which is not necessarily evil, but by definition it means that it will not benefit as many people as possible. So this is purely aspirational, unless you add some modifiers to it.”

16) Human Control: Humans should choose how and whether to delegate positions to AI systems to accomplish human-chosen objectives.

“Yes.”

ARIEL: “So, this is one I think is interesting, because, instinctively, my reaction is to say yes. But then I think of these examples of, say, planes, where we’re finding that the planes’ autonomous systems are actually better at flying than some of the pilots. And do we actually want the pilots to be choosing to make a bad decision with a plane, or do we want the plane to take power away from the pilot?”

JOHN: “Until universally systems can show that humans can be completely out of the loop and more often than not it will be beneficial, then I think humans need to be in the loop. However, the research I’ve seen also shows that right now is the most dangerous time, where humans are told, “Just sit there, the system works 99% of the time, and we’re good.” That’s the most dangerous situation, because then, even if the humans are really well trained, they may go for six weeks or they may go for 6 hours before something negative happens. So I think it still has to be humans delegating first. But in the framework of the context we’ve talked about here, where the systems are probably going to be doing pretty well and humans are in the loop, it’s a good choice to make, plus we should have lots of continued training to demonstrate the system is going to stay useful.”

23) Common Good: Superintelligence should only be developed in the service of widely shared ethical ideals, and for the benefit of all humanity rather than for one state or organization.

“This sentence has a lot in it. Superintelligence – and I know FLI has deep expertise in this – that’s a very tough term. Because it usually infers some kind of sentience, like artificial general intelligence is superintelligence. But without defining what superintelligence means… I would say, define that word a little bit more.

“And then, ‘only be developed in the service of widely shared ethical ideals.’ That part of the sentence is incredibly difficult, because widely-shared ethical ideals… whose ideals? What do you mean by ‘widely-shared?’ Established human rights criteria make it a lot easier to talk about these things, because then you can point to actual UN rules – it doesn’t mean you have to agree with them – but they are widely established.

“And ‘for the benefit of all humanity, rather than for one state or organization,’ yes, again, that works for me. But, what does benefit mean? And also, ‘one state or organization…’ If I’m reading this as a government, what do I do? I’m still New Jersey, I’m still the United States, I’m still Israel. Of course we’re going to have to prioritize our own needs. In general, I think the sentence is fine, it’s just that there’s so much to it to unpack. There could be a lot of modifiers to it that I think would make it stronger. Hopefully that’s helpful.”

ARIEL: “Yeah, this is exactly the kind of discussion that we want. Going back to what you were saying earlier, and even here, I’m curious what you think the next steps would be if we wanted to see these principles actually put into action; the next steps to have principles that are generally accepted as we move forward in the development of AI. Or to make these stronger and easier to follow.”

JOHN: “Well, what I’m finding with my experience at IEEE is that the more you want principles to be accepted – the challenge is [that] to make them universal you risk making them less specific, [less] pragmatic, and potentially [not as] strong as they need to be. Which is hard. For example, the Asimov robotics law that says machines shouldn’t harm humans. Most people right away go, ‘Yeah, I agree with that.’ And then someone is like, ‘What about a medical robot that needs to operate on a person?’ And you’re like, ‘Oh yeah, medical robots…’ So it becomes hard in that regard.

“So this is why we’re doing a lot of work to define specifics on what it means to increase positive human benefits with AI on our wellbeing committee. Fortunately, there are already a lot of fantastic metrics along these lines that already exist. For instance, The OECD has the Better Life Index, which contains metrics that measure – quantitatively and qualitatively – metrics beyond GDP measures. This is part of a whole movement, or group of metrics comprising the Beyond GDP Movement. So, for instance, one way you can measure if what you’re building is for the benefit of all humanity, is to say, using these metrics from the OECD or the UN Development Goals, we’re going to try and increase these ten metrics (Beyond GDP metrics). When these ten things are increased in a positive way, that is demonstrably increasing either human wellbeing or positive benefit for humanity. Then you can point to it and say, ‘That’s what we mean by increasing human benefit.’”

ARIEL: “Was there anything else in general about the principles that you wanted to comment on?”

JOHN: “Just, again, I think they’re great in the sense of this is so much more than just Asimov’s Principles, because obviously those were science fiction and very short…”

ARIEL: “And designed to be broken.”

JOHN: “Exactly, a conundrum by design. I really like how you’ve broken it up: Research, Longer-Term Issues, the three sections. And I think, especially in terms of really core things… it’s very meaty, and people can get their teeth around it. In general, I think it’s fantastic.”

Join the discussion about the Asilomar AI Principles!

Susan Schneider Interview

The following is an interview with Susan Schneider about the Beneficial AI 2017 conference and The Asilomar Principles that it produced. Schneider is a philosopher and cognitive scientist at the University of Connecticut, YHouse (NY) and the Institute for Advanced Study in Princeton, NJ.

Q. Explain what you think of the following principles:

4) Research Culture: A culture of cooperation, trust and transparency should be fostered among researchers and developers of AI.

“This is a nice ideal, but unfortunately there may be organizations, including governments, that don’t follow principles of transparency and cooperation. Still, it is important that we set forth the guidelines, and aim to set norms that others feel they need to follow.”

ARIEL: “And do you have thoughts on trying to get people more involved, who might resist something like that culture?”

SUSAN: “Concerning those who might resist the cultural norm of cooperation and transparency, in the domestic case, regulatory agencies may be useful. The international case is difficult; I am most concerned about the use of AI-based autonomous weapons. I’m also greatly concerned with the possible strategic use of superintelligent AI for warfare, even if they aren’t technically “autonomous weapons.” Global bans are difficult to enforce, and I doubt there is even sufficient support in the US government for a ban (or even major restrictions) on AI-based autonomous weapons, for instance. And secrecy is often key to the success of weapons and strategic programs. But you asked how to best involve humans: Ironically, when it comes to superintelligence, enhancing certain humans so that they can understand (and compete with) the complex processing of a superintelligence might be the most useful way of getting humans involved! But I think the efforts of FLI, in publicizing the Asilomar Principles and holding meetings, are potentially very useful. Calling attention to AI safety is very important.”

5) Race Avoidance: Teams developing AI systems should actively cooperate to avoid corner cutting on safety standards.

“Cooperation is very important. The problem is going to be countries or corporations that have a stake in secrecy. To go back to the case of warfare, I worry that we are entering an AI arms race. If superintelligent AI is the result of this race, it could pose an existential risk to humanity. As Elon Musk, Bill Gates, Stephen Hawking, Nick Bostrom and others have pointed out, it would be difficult to control an AI that is smarter than humans in every respect. I’m not sure the safeguards being discussed will work (kill switches, boxing it in, and so on), although they are better than nothing.”

ARIEL: “I think it’s natural to think about the weapons as an obvious issue, but I also worry about just economic forces encouraging companies to make a profit. And the best way to make a profit is to be the first person to create the new product. Do you have concerns about that at all? Or do you think it’s more weapons that we have an issue with?”

SUSAN: “This is a major concern as well. We certainly don’t want companies cutting corners and not developing safety standards. In the US, we have the FDA regulating pharmaceutical drugs before they go to consumers. We need regulatory practices for putting AIs into the market. If the product is a brain enhancement device, for instance, what protects the privacy of one’s thoughts, or firewalls the brain from a computer virus? Certain regulatory tasks may fall under agencies that now deal with the safety of medical devices. Hacking will take on a whole new dimension – brain hacking! (This is very different than the fun consciousness hacking going on in Silicon Valley right now!) And what about the potential abuse of the robots that serve us? After all, could a machine be conscious? (I pursue this issue in a recent Nautilus piece and a TED Talk. If they can be conscious, they aren’t mere products that can cause harm; they are sentient beings that can be harmed.”

6) Safety: AI systems should be safe and secure throughout their operational lifetime, and verifiably so where applicable and feasible.

“This is important, obviously, but I’m not sure how you verify that you can trust the program when a machine is superintelligent. It can constantly rewrite its own code, and we may not even be able to understand the program to begin with. It is already difficult to fully understand what deep learning programs are doing, and these are nowhere near AGI even.

“So ‘value alignment’ – the idea of how to align machine values with our values, basically – it’s horrendously tricky. I think the market forces are going to keep pushing us more and more in the direction of greater intelligence for machines, and as you do that, I think it becomes more difficult to control them.

“I’ve already talked about superintelligence; there’s general agreement that value alignment is a major problem. So consider a different case, the example of the Japanese androids that are being developed for elder care right now. Right now, they’re not smart; right now, the emphasis is on physical appearance and motor skills. But imagine when one of these androids is actually engaged in elder care, and it’s trying to do ordinary tasks, there will be a need for the android to be highly intelligent. It can’t just have expertise in one domain, like Go or chess, it has to multitask and exhibit cognitive flexibility. (As my dissertation advisor, Jerry Fodor, used to say, it has to make breakfast without burning the house down.) After all, we do not want elderly people facing accidents! That raises the demand for household assistants that are AGIs. And once you get to the level of artificial general intelligence, it’s harder to control the machines. We can’t even make sure fellow humans (other kinds of AGI’s, haha) have the right goals; why should we think AGI will have values that align with ours, let alone that a superintelligence would. We are lucky when are teenagers can be controlled!”

ARIEL: “Do you worry at all about designs pre-AGI? Or are you mostly concerned that things will get worse once we hit human-level and then beyond?”

SUSAN: “No, AI introduces all sorts of safety issues. I mean, people worry about drones that aren’t very smart, but could do a lot of damage, for example. So there are all sorts of worries. My work concerns AGI and superintelligence, so I tend to focus more on that.”

9) Responsibility: Designers and builders of advanced AI systems are stakeholders in the moral implications of their use, misuse, and actions, with the obligation and responsibility to shape those implications.

“I don’t have objections to these principles. The problem is going to be getting people to internalize them. It’s important to have principles put forward! The difficult task is to make sure people aren’t focused on their own self-interest and that they anticipate the social impact of their research.”

ARIEL: “Maybe the question here is, how do you see them being implemented? Or what problems do you see us having if we try to implement them?”

SUSAN: “Well I guess the problem I would have is seeing how they would be implemented. So the challenge for the Institute is to figure out ways to make the case that people follow them. And not just in one community, not just in Silicon Valley or in North America, but everywhere. Even in isolated countries that get AI technology and could be uncooperative with the United States, for example an authoritarian dictatorship. I just haven’t the slightest idea about how you would go about implementing these. I mean, a global ban on AI or even AGI technology is not practical at this point.”

16) Human Control: Humans should choose how and whether to delegate decisions to AI systems to accomplish human-chosen objectives.

“Right, that’s very, very tricky. Even now it’s sometimes difficult to understand why a deep learning system made the decisions that it did. And obviously, getting the machine to communicate clearly to us is an ongoing research project, and there will be exciting developments in that. But again, my focus is on AGI and superintelligence. If we delegate decisions to a system that’s vastly smarter than us, I don’t know how we’ll be able to trust it, since traditional methods of verification seem break down. One idea would be that you could have an enhanced human that would interact with the machine. (Hopefully you can trust the enhanced human! The hope is that even if he/she is post-biological, the person still has a partly biological brain, could even unplug the enhancements, and they identify with us.”)

23) Common Good: Superintelligence should only be developed in the service of widely shared ethical ideals, and for the benefit of all humanity rather than for one state or organization.

“The obvious problem here is going to be, what ‘widely-shared ethical ideals’ are. And how you would even specify that in the machine’s architecture – that’s a big issue right now. That was a topic that was treated very nicely by Nick Bostrom’s book, Superintelligence. And there’s no easy answer. The field of ethics is full of controversies, and that doesn’t even get to the larger problem of how you would encode ethics in a machine and make sure that the machine continues with those ideals in mind.“

ARIEL: “If we’re looking back at history, do you feel that humanity as a whole is moving towards more cohesive ideals, or do you think it’s just as fractured as ever?”

SUSAN: “Our world is still a fractured one, but to speak to the case of AI, what you want are values that people agree upon, and if you add too much content you’re never going to get shared ideals. You start getting into values that differ from culture to culture and religion to religion… To draw from Asimov’s three laws, we might use a ‘do not harm humans’ principle. But there may be contexts in which harming is justifiable or inevitable. And this is when you get the issues about which particular ethical system is justified. So according to a utilitarian principle, for example, it would be okay to sacrifice one individual for the greater good. But that wouldn’t be okay according to a Kantian approach, at least without the person’s consent.

“So it’s going to be very tricky. I mean, if you consider all the business interests at work, I just wonder how you would even make sure that the different businesses developing a certain kind of AI or would code in the very same ideals. (Or perhaps the same ideals do not need to be coded, but the consumer needs to be aware what ideals the system is supposed to follow.) It would be nice to see some safety regulations here. But then, again, the problem becomes renegade countries and renegade groups that don’t follow it.”

ARIEL: “Yeah, war is a perfect example of when you don’t want to apply the ‘robots shouldn’t hurt a human’ rule.”

SUSAN: “Exactly. The trick there is how you can get programming that you can trust, programming that is going to give us the right results – a machine that’s safe.

“I find it very difficult to see how we could ensure superintelligence safety without benefitting from human minds that have been enhanced, so that we’re working with a group of ultra-intelligent people who can help us better foresee how a superintelligence could behave. Just as no one person understands all of mathematics, so too, collective efforts of enhanced and unenhanced individuals may help us get a grip on a single superintelligent mind. As I’ve urged in a recent paper, if we develop ways to understand superintelligence we can better control it. It will probably be easier to understand superintelligences that are based on principles that characterize the human brain (combinatorial representations, multi-layered neural networks, etc.). We can draw from work in cognitive science, or so I’ve urged.”

ARIEL: “Wouldn’t we have similar issues with a highly intelligent augmented person?”

SUSAN: “We might. I mean, I think if you take a human brain that is a relatively known quantity, and then you begin to augment it in certain ways, you may be better able to trust that individual. Because so much of the architecture is common to us, and the person was a human, cares about how humans fare, and so on. But you never know. So this gets into the domain of science fiction.

“And, you know, the thing that I’ve been working on is machine consciousness. One thing that’s not discussed in these principles about safety and about ethics is what we want to do concerning developing machines that might have an inner world or feel. There’s a big question there about whether a machine could feel, be conscious. Suppose it was an AGI, would it inevitably feel like something to be it?

“A lot of cognitive scientists think that experience itself is just computation, so if that’s the case, then one element of safety is that the machine may have inner experience. And if they recognize in us the capacity to feel, they may respect us more. On the other hand, if the machine is conscious, maybe it will be less predictable. So these are issues we have to enter into the whole AI safety debate as well, in addition to work on human intelligence enhancement as a safety strategy.”

ARIEL: “Overall, what were your thoughts about the principles? And you were talking about consciousness… Are there other principles you’d like to see added that aren’t there?”

SUSAN: “Well, I think developing an understanding of machine consciousness should be a goal. If the machine is conscious, it could facilitate empathy, or it could make it less productive. So there are two issues here with AI consciousness that are certainly important. The first issue is, you can’t market a product if it’s conscious, because that could be tantamount to slavery. And the other issue is, if these AGIs or superintelligences are in a condition where they have weapons or are capable of hurting others, and we want them to have our goals, then we need to figure out if they’re conscious or not, because it could play out one of two ways. Consciousness could make something more compassionate towards other conscious beings, the way it is for non-human animals. Certain humans choose to not eat non-human animals because they feel that they’re conscious. Or the presence of conscious experience could make the AI less predictable. So we need to figure this out. I really enjoy the company and discussions of the AI leaders, but this really has a lot to do with philosophical training as well. So there should be a dialogue on this issue and more understanding.”

Join the discussion about the Asilomar AI Principles!

Patrick Lin Interview

The following is an interview with Patrick Lin about the Beneficial AI 2017 conference and The Asilomar Principles that it produced. Lin is the director of the Ethics + Emerging Sciences Group, based at California Polytechnic State University, San Luis Obispo, where he is an associate philosophy professor. He regularly gives invited briefings to industry, media, and government; and he teaches courses in ethics, political philosophy, philosophy of technology, and philosophy of law.

Q: From your perspective, what were the highlights of the conference?

“There’s so many, I don’t know if I can even limit myself to a few. I guess the top level highlights for me were about the people. Not that the content wasn’t interesting, but I’m already familiar with a lot of the positions and arguments; so for me, the most interesting thing was to just be around, to personally and finally meet in real life a lot of the folks that I’ve only known online, like Roman Yampolskiy, or by reputation only.

“Also, this particular conference you guys had was unique, far different from other AI conferences I’ve been to, in that you had such an incredible concentration of thought leaders from diverse fields, not just technologists but economists, ethicists, and so on. This was a rare meeting of minds. If you were Skynet looking to travel back in time to do the most damage towards AI safety, you might want to hit Asilomar in January 2017. I think that says something.

“It was just an incredible nexus of luminaries and industry captains, and just people from all kinds of fields. Some of my favorite speakers there were Jeff Sachs, the economist from Columbia, Anthony Romero, ACLU – these are people not traditionally involved in this AI conversation. Anca Dragan from Berkeley had a great piece on human-robot interaction, and you and I spoke about this at the conference, I’d love to see more engagement of human-robot or human-computer interface issues; I think those will be big.

“Also, take Joseph Gordon-Levitt and his wife, Tasha, who’s a technologist in her own right: they’re great people, not people I would normally meet at a standard AI conference, but they’re also important here. Some background: I’ve been working with military technology and people from DoD for close to 10 years now, and this theme keeps coming up. There’s a difference between hard power and soft power. Hard power are things like sending in military, sending in funding, and just really trying to exert influence over other nations.

“There’s also soft power; soft power is more persuasive, friendlier things you could do, for instance, increase tourism and increase goodwill to help bridge people together. I think that’s what’s great about having someone like Joseph Gordon-Levitt there, in that he represents Hollywood and can help steer AI globally. I’ve always thought that Hollywood was maybe America’s greatest source of soft power: it’s our greatest way of influencing other cultures, opening them up to our values, opening them up to the idea of democracy. We do this through movies. This is hard to tell when you’re inside the US, but outside the US, Hollywood has such a profound impact, whether they realize it or not. Even inside the US, think about the effect of movies on our national consciousness, like Ex Machina, Her, Star Wars, 2001: A Space Odyssey, and many others, on how we think about AI an robots. Hollywood – that is, storytelling – is one of our best and most effective weapons of change.

“The conference was also a good chance for me to catch up with old friends, people I’ve known for a long time – Wendell Wallach, Ryan Calo, and many others – we run in the same circles, but we don’t meet up all that often. The conference has already sparked many new ideas and ways to collaborate. Now, I’m already starting to do that, just connecting with people I met at the conference, and hopefully projects and funding will materialize.”

ARIEL: “That’s awesome, I’m glad you liked it. I’ve noticed most of the people that we talked to, it’s been the people and the interactions at the conference seem to be sort of the big highlights, so that was pretty nice.”

PATRICK: “There were a few times where I heard things that were just really surprising to me. As an ethicist, I’m not so much in touch with a lot of the technical details, so it was good to hear the technical details straight from the horse’s mouth, from people on the frontlines of this. Also, a few things really stood out; for instance Ray Kurzweil, when he was on that super-panel, he basically said, I’m paraphrasing, ‘Look, even if we had perfect AI today, there would still be a whole load of problems. AI safety is not just a technical problem that can be solved with clever programming, but even if you have perfect AI, it’s a social and political problem to guard against abuse of that power.’

“It isn’t just about aligning values or working out the programming, but this is also very much a non-technical problem.”

Q: Why did you decide to sign the Asilomar AI Principles document?

“Well, so I know in recent months, or in the past year, there have been various groups publishing their various principles, and those look like great efforts too. But this is still an emerging area to conceptualize these principles. I think what Future of Life had done, as I would put it, is something like a meta-analysis of these various proposals. It seemed to me that you guys weren’t just reinventing the wheel, you’re not just putting out another set of principles like Stanford did or IEEE or the White House.

“What’s needed now is a meta-analysis, someone to consolidate these principles and arrive at a best-of-breed set of principles. That’s why I support it. I also think they are top level, ambitious principles, and it’s going to take work to clarify them, but at least at the top level, they seem headed to the right direction.”

Q: As a non-AI researcher, why do you think AI researchers should weigh in on issues like those that were brought up at the conference and in the principles document?

“At the conference, I met Lord Martin Rees. Before I met him and for a couple years now, I’ve usually included a quote from him in my talks about technology ethics. This came from an op-ed he wrote in the Guardian about 10 years ago. Martin Rees was talking about the responsibilities that scientists have. He says, ‘Scientists surely have a special responsibility. It is their ideas that form the basis of new technology. They should not be indifferent to the fruits of their ideas. They should forego experiments that are risky or unethical.’

“I think he gets it right, that scientists and engineers, they’re responsible for creating these products that could have good uses and bad uses. It’s not just causal responsibility, but it’s also moral responsibility. If it weren’t for them, these products and services and effects wouldn’t have happened. In a real way, they are responsible for these outcomes, and that responsibility means that they need to be engaged as early as possible in steering their inventions or products or services in the right direction.

“You know, maybe this isn’t a popular opinion in engineering, but a lot of engineers and scientists want to say, ‘Look, we’re just doing pure research here, and that’s why we have the lawyers and the ethicists and other folks to sort it out.’ I think that’s right and wrong. Something works about a division of labor: it’s efficient and you get to focus on your competitive advantage and your skillsets, but we can’t be totally divorced from our responsibility. We can’t fully handoff a responsibility or punt it to other people. I do think that technologists have a responsibility to weigh in. It’s a moral responsibility.

“A lot of them don’t do it, and that’s why it’s all too rare for a gathering like what you guys had, where you had socially-minded technologists who understood that their work is not just groundbreaking, but it could have some serious positive and negative impacts on society, and they’re worried about that, which is good, because they should be. I see this as a natural move for technologists to feel responsibility and to be engaged, but unfortunately, not everyone sees it that way, and they’re not aware of the limits of their design.

“That could get them in trouble if their programming decisions lead to some bad crash that the public is outraged. They can’t just say, ‘Hey, we’re not ethicists. We’re just doing what the data says, or we’re just doing what maximizes the good.’ Well, that means you’re doing ethics, implicitly. A lot of scientists are engineers, whether they know it or not, they’re already engaged in ethics and values. It’s already baked into a lot of their work. You could bake ethics into design. A lot of people think technology is amoral or neutral, but I don’t believe that. I think ethics can be baked into the design of something.

“In most cases, it’s subtle, it might not make a difference; but in other cases, it’s pretty clear. For instance, there have been health apps coming out of Silicon Valley that fail to track say, women’s periods. A health issue or body issue for half the population is just totally ignored because the man programming it didn’t think about these use-cases. I think it’s wrong, it’s a myth to say that technology is neutral. Sure, in most cases, it’s too subtle to tell, but there’s definitely ethics built in, or can be built into the design of technology.”

Q. Explain what you think of the following principles:

21) Risks: Risks posed by AI systems, especially catastrophic or existential risks, must be subject to planning and mitigation efforts commensurate with their expected impact.

“This sounds great in ‘principle’ but you need to work it out. For instance, it could be that there’s this catastrophic risk that’s going to affect everyone in the world – it could be AI or an asteroid or something, but it’s a risk that will affect everyone – but the probabilities are tiny, 0.000001 percent, let’s say. Now if you do an expected utility calculation, these large numbers are going to break the formula every time. There could be some AI risk that’s truly catastrophic, but so remote that if you do an expected utility calculation, you might be misled by the numbers.

“I agree with it in general, but part of my issue with this particular phrasing is the word ‘commensurate.’ Commensurate meaning an appropriate level that correlates to its severity. So I think how we define commensurate is going to be important. Are we looking at the probabilities? Are we looking at the level of damage? Or are we looking at expected utility? The different ways you look at risk might point you to different conclusions. I’d be worried about that. We can imagine all sorts of catastrophic risks from AI or robotics or genetic engineering, but if the odds are really tiny, and you still want to stick with this expected utility framework, these large numbers might break the math. It’s not always clear what the right way is to think about risk and a proper response to it.”

1) Research Goal: The goal of AI research should be to create not undirected intelligence, but beneficial intelligence.

“Yeah I think I generally agree with this research goal. Given the potential of AI to be misused or abused, it’s important to have a specific positive goal in mind. Now again, I think where it might get hung up is what this word ‘beneficial’ means. If we’re directing it towards beneficial intelligence, we’ve got to define our terms; we’ve got to define what beneficial means, and that to me isn’t clear. It means different things to different people, and it’s rare that you could benefit everybody.

“Most of the times, you might have to hurt or go against the interest of some groups in order to maximize benefits. Is that what we’re talking about? If we are, then again, we’re implicitly adopting this consequentialist framework. It’s important to understand that, because if you don’t know you’re a consequentialist, then you don’t know the limits of consequentialism, and there are important limits there. That means you don’t really understand whether it’s appropriate to use consequentialism in this area.”

4) Research Culture: A culture of cooperation, trust, and transparency should be fostered among researchers and developers of AI.

“I think building a cohesive culture of cooperation is going to help in a lot of things. It’s going to help accelerate research and avoid a race, but the big problem I see for the AI community is that there is no AI community, it’s fragmented, it’s a Frankenstein-stitching together of various communities. You have programmers, engineers, roboticists; you have data scientists, and it’s not even clear what a data scientist is. Are they people who work in statistics or economics, or are they engineers, are they programmers?

“Data science has an identity problem, and I think AI definitely has an identity problem. Compare let’s say an AI programmer (or designer or whatever) with a civil engineer or an electrical engineer or architect or teacher or lawyer — these other fields, they have a cohesive professional identity. There’s required professional training, there’s required educational components, there’s licensing requirements, there’s certification requirements. You don’t have that in software engineering or in AI. Anyone who can write code is effectively a programmer. You don’t have to have graduated from high school, or you could have graduated from Oxford or Stanford, you could be working out of the basement in your mother’s house, or you could be working at a fancy Google or Facebook lab.

“The profession is already so fragmented. There’s no cohesive identity, and that’s going to be super challenging to creating a cohesive culture that cooperates and trusts and is transparent, but it is a worthy goal. I think it’s an immense challenge, especially for AI because there’s no set educational or professional requirements. That also means it’s going to be really hard to impose or to set a professional code of ethics for the industry if there’s no clear way of defining the industry. Architects, teachers, doctors, lawyers: they all have their professional codes of ethics, but that’s easy because they have a well-delineated professional culture.”

ARIEL: “Do you think some of that is just because these other professions are so much older? I mean, AI and computer science are relatively new.”

PATRICK: “Yeah, I think that’s part of it. Over time, we might see educational and professional standards imposed on programmers, but also AI and programming, or at least AI and data science draws from different fields inherently. It’s not just about programming, it’s about learning algorithms, and that involves data. Once you get data, you have this interpretation problem, you have a statistics problem, and then once you worry about impacts, then you have social, political, psychological issues to attend to. Just by the nature of AI in particular, it draws from such diverse fields that, number one, you have to have these diverse participants in order to have a comprehensive discussion, but also, number two, because they’re so diverse, they’re hard to pull together into a cohesive culture.”

5) Race Avoidance: Teams developing AI systems should actively cooperate to avoid corner-cutting on safety standards.

“I would lump race avoidance into the research culture. Yeah, it’s probably good to avoid an arms race. Competition is good, and an arms race is bad, but how do you get people to cooperate to avoid arms race? Well, you’ve got to develop the culture first, but developing the culture is hard because of the reasons I already talked about.”

6) Safety: AI systems should be safe and secure throughout their operational lifetime, and verifiably so where applicable and feasible.

“Who can object to the safety one? Again, I think it’s not clear that this principle recognizes that safety is not just a technical problem. I mentioned Ray Kurzweil and Sam Harris talking about this onstage that even with perfect AI, I think Sam Harris said this, ‘Even if God dropped down perfect AI to us today, what does that mean? Does that really solve our problems and worries about AI? No. It could still be misused and abused in a number of ways.’ When Stuart Russell talks about aligning AI to values, it seems to be this big, open question, ‘Well, what are the right values?’

“If you just make AI that can align perfectly with whatever values you set it to, well the problem is, people can have a range of values, and some of them are bad. Just merely matching AI, aligning it to whatever value you specify I think is not good enough. It’s a good start, it’s a good big picture goal to make AI safe, and the technical element is a big part of it; but again, I think safety also means policy and norm-setting.”

14) Shared Benefit: AI technologies should benefit and empower as many people as possible.

“Shared benefit is interesting, because again, this is a principle that implies consequentialism, that we should think about ethics as satisfying the preferences or benefiting as many people as possible. That approach to ethics isn’t always right. I mean, yeah, a lot of our decisions are based on consequentialism: we make policy decisions based on pros and cons and numbers, and we might make personal decisions based on that. Should I have the pizza or the hot dog? Which one’s going to make me happier? Well, maybe I’m lactose intolerant, and maybe even though pizza’s really yummy and makes me happy now, I’ve got to think of my happiness later on.

“Consequentialism often makes sense, so weighing these pros and cons makes sense, but that’s not the only way of thinking about ethics. Consequentialism could fail you in many cases. For instance, consequentialism might green-light torturing or severely harming a small group of people if it gives rise to a net increase in overall happiness to the greater community. If you then look at things like the Bill of Rights, if you think that we have human rights or we have duties and obligations, these are things that aren’t so much about quantifiable numbers. These are things that can’t easily fit into a consequentialist framework.

“That’s why I worry about the Research Goal Principle and Shared Benefit Principle. They make sense, but they implicitly adopt a consequentialist framework, which by the way is very natural for engineers and technologists to use, so they’re very numbers-oriented and tend to think of things in black and white and pros and cons, but ethics is often squishy. You deal with these squishy, abstract concepts like rights and duties and obligations, and it’s hard to reduce those into algorithms or numbers that could be weighed and traded off.”

16) Human Control: Humans should choose how and whether to delegate decisions to AI systems, to accomplish human-chosen objectives.

“Yeah, this is tough. I wouldn’t say that you must always have meaningful human control in everything you do. I mean, it depends on the decision, but also I think this gives rise to new challenges. I say this when I give my talks on robot car ethics; say for instance that you get in a bad car accident, there’s a wreck in front of you or something jumps out in front of you and you just swerve reflexively into another car, or you swerve reflexively off into the shoulder.

“Now, no one can blame you for that; it’s not premeditated, there’s no malice, there’s no forethought, it’s just a bad reaction. Now imagine an AI driver doing the exact same thing, and as reasonable as it may be for the AI to do the exact same thing, something feels different. The AI decision is scripted, it’s programmed deliberately, or if it’s learned from a neural net, it’s already predetermined what the outcome is going to be. If an AI driver hurts somebody, this now seems to be a premeditated harm, and there’s a big legal difference between an innocent accident where I harm someone, and a premeditated injury.

“This is related to the idea of human control and responsibility. If you don’t have human control, it could be unclear who’s responsible for it, but the context matters. It really does depend on what kind of decisions we’re talking about, that will help determine how much human control there needs to be.”

23) Common Good: Superintelligence should only be developed in the service of widely shared ethical ideals, and for the benefit of all humanity rather than one state or organization.

“This might be the most ambitious of all principles. It’s good, but it’s broad. I mean, it’s good to have it down on paper, even though it’s broad and ambitious, but this is going to be a hard one.

“The trick is to figure out what our common set of ethical values are. If we don’t have a common set, then we have to make a judgment of whose ethics we should be following. Of course everyone’s going to say, ‘Well, you know our ethics is the best, and follow us.’ You’re going to run into some problems here. More generally, since the elections I’ve been really just pessimistic about the prospect of building ethics into anything. If the rule of law is eroding, if the rule of law no longer matters, then what the hell are we talking about ethics for? I think we’ve got bigger problems in the next few years than this, given the general trend is. But ethics and norms are still important. You don’t need to have bright-line laws to influence behavior. Again, think about Hollywood.

“Norms and principles are much better than nothing, but if they are our primary line of defense, then we’re going to be in for a rough ride. Principles and ethics alone aren’t going to stop a lot of people from doing bad things. It could help forestall it, it could help postpone a disaster, but we need to see a lot more humanity and just social awareness and consciousness globally in order for us to really reign in this AI genie.”

ARIEL: “What would your take on this had been if we were discussing this a year ago today?”

PATRICK: “I’d be more optimistic. I’d definitely be more optimistic. For instance, if you look at the United Nations—say, Heather Roff and Peter Asaro’s work on killer robots—they’ve been making progress. It’s slow, but they’ve been making real progress. Just in December, the United Nations finally made it official to convene some meetings to figure out if we’ve got to regulate killer robots.

“Internationally, I think we’re making progress, but if the US is a technology leader in the world, and we are, what happens in the US is going to be important — it’s going to set the tone for a lot of AI research around the world. Ethics and principles, all this comes top-down. If your company has a bad CEO, then naturally your employees are going to do bad things. It’s worse when you talk about political leaders or leaders of state. So if you don’t have ethical, moral leaders, then a lot of bad things are going to flow from that. Yeah, a year ago I think we’d be having a different conversation, unfortunately.”

Join the discussion about the Asilomar AI Principles!

Bart Selman Interview

The following is an interview with Bart Selman about the Beneficial AI 2017 conference and The Asilomar Principles that it produced. Selman is a Professor of Computer Science at Cornell University, a Fellow of the American Association for Artificial Intelligence (AAAI) and a Fellow of the American Association for the Advancement of Science (AAAS).

Q: From your perspective, what were the highlights of the conference?

“I think the overall highlight for me was to see how much progress has been made on the whole AI safety question and beneficial AI since the Puerto Rico meeting. It was really exciting to see the workshop program where there was serious technical research into AI safety issues, and interesting progress. And at the conference I think what we saw was much more agreement among the participants and panelists about AI safety issues, the importance of it, the risk we have to address and the responsibility of AI researchers and AI companies that have large AI projects. So it was really about the progress in two years, it was very dramatic and very positive.”

Q: Why did you decide to sign the Asilomar AI Principles document?

“I think that if you look at the evolution since Puerto Rico, there’s a lot more agreement among the researchers that we have to think about these issues and that we have to have some general principles to guide research projects and development projects. I remember the lunch time discussions about the principles – most principles we would say, ‘well of course this sounds good.’ We had some where we had some modifications but most of them sounded very reasonable, very good principles to have.

“Society as a whole has never dealt with these kinds of principles, and I always viewed it as, ‘AI is a fairly academic discipline, there’s no direct societal impact.’ As that is changing so rapidly now, we need to have AI researchers and companies have a set of guidelines and principles to work by.”

Q: Why do you think AI researchers should weigh in on issues like those that were brought up at the conference and in the principles document?

“I think we need to weigh in because we are more aware of the deeper underlying technical issues. I actually saw – recently I was at a different conference on data science – the responsibility of data scientists on basic issues of privacy and bias, and all those kinds of things that can creep into machine learning systems. And an undergraduate student of ours asked at the end, ‘well the responsibility must lie with – say I work for Google – the responsibility must lie with Google. Why would I care as an undergraduate student about these ethical issues?’ But then the speaker made a very good point. He said, ‘well the person who writes the code, who develops the code, is often the only person who knows exactly what’s in the code, and knows exactly what is implemented in the machine learning system or the data mining system. So even though maybe in some abstract legal sense the employer will have final responsibility – financial or legal – it’s really the person building the system.’

“This was actually done in the context of issues with automation of drive-by-wire cars, and he showed an example of a piece of software that was really developed with all kinds of safety risks in it. But he said the only person who knew those were safety risks was the person who was building the system. So he said, one very important reason for AI researchers and implementers to be involved is because they’re often the only ones who really know what goes into the code. And I thought that was a very good point.”

Q. Explain what you think of the following principles:

19) Capability Caution: There being no consensus, we should avoid strong assumptions regarding upper limits on future AI capabilities.

“My take was that’s a reasonable assumption. I think there’s quite a spectrum of opinions of when superhuman AI emerges or how broad it will be – will it be of certain capabilities or of all capabilities – so there’s a lot of discussion about that and no consensus. But to assume that we will reach it or to assume that there are some limits on this kind of machine intelligence seems incorrect. So I thought it was completely reasonable to say we should not think this cannot be done, and based on current progress I should say we’ve seen various points in AI where there was very exciting progress and where it did not continue. So I’m not saying that it’s definitely going to happen that the kind of progress we’re seeing in the last 2-3 years will continue without interruption, but we should not assume that it won’t.”

20) Importance: Advanced AI could represent a profound change in the history of life on Earth, and should be planned for and managed with commensurate care and resources.

“I think I also agreed with that one. I thought the discussions at the meetings were interesting. There are many different risks that society faces – climate change, and other kinds of inequality, and other kinds of risks – so advanced AI is definitely one big issue, and I think some group of people needs to study the problem. Maybe not every person on Earth should be concerned about it, but there should be among scientists a discussion about these issues and a plan – can you build safety guidelines to work with value alignment work? What can you actually do to make sure that the developments are beneficial in the end?”

ARIEL: “So I have a side question. We’re asking about advanced AI, but even if we just continue at our current rate for a couple years, how profound of an effect do you think AI will have on life on Earth?”

BART: “I think the effect will be quite dramatic. This is another interesting point – sometimes AI scientists say, well it might not be advanced AI that will do us in, but dumb AI. B y shifting responsibilities to machines, and in many ways it’s beneficial, but the fear is that these digital systems are being integrated in our society – be it automatic trading, we saw talks about decision making on parole issues, mortgage approvals – all kinds of systems are now being integrated into our society and an interesting point is that they’re actually very limited in some sense. They’re autonomous and they’re somewhat intelligent but actually also quite dumb in many ways.

“For example, they lack common sense, and that’s sort of the big thing. So the example is always the self-driving car has no idea it’s driving you anywhere. It doesn’t even know what driving is. So we look at these systems and we think the car must have some idea of what it is to drive but it actually doesn’t have any idea. It’s a little bit of short term risk. Say the self-driving car comes at you but it doesn’t brake. We may assume, ‘well of course it’s going to brake at the end – it’s just trying to scare me,’ while the car vision may literally not see you or have decided that you are not really there, or something like that. Some of these things that – actually if you looked at the videos of an accident that’s going to happen, people are so surprised that the car doesn’t hit the brakes at all, and that’s because the car works quite differently than humans. So I think there is some short term risk in that there are going to be systems and we actually misjudge them, we actually think they’re smarter than they are. And I think that will actually go away when the machines become smarter, but for now…”

20) Risks: Risks posed by AI systems, especially catastrophic or existential risks, must be subject to planning and mitigation efforts commensurate with their expected impact.

“Again, a very reasonable principle. And again I sort of refer to some of the discussions between AI scientists who might differ in how big they think that risk is. I’m quite certain it’s not zero, and the impact could be very high. So it’s one of those things that even if these things are still far off and we’re not clear if we’ll ever reach them, even with a small probability of a very high consequence we should be serious about these issues. And again, not everybody, but the subcommunity should.”

12) Personal Privacy: People should have the right to access, manage and control the data they generate, given AI systems’ power to analyze and utilize that data.

“I liked that as a principle because I’d like for policymakers to start thinking about these issues. It’s incredibly difficult, I think, because by putting together data from so many different sources – we’ve seen examples of medical data that’s anonymized and handed up. And with Yahoo, I think, data sets were released and researchers were quite convinced that they were anonymized, but in very short time researchers would find ways to identify single individuals based on some obscure pattern of behavior that they could find in the data.

“So this whole question of we should have control of our own data, I like it as a principle. I think what will happen to it, hopefully, is that people will become aware that this is a real issue. And there’s not such a simple solution to it, because it’s really the combining of data sources that gives a lot of power. If Amazon knows all of your shopping behavior it’s probably able to figure out what kind of disease you might have, just to give an example. So this is one of the things we have to find out how to manage in a reasonable way.”

10) Value Alignment: Highly autonomous AI systems should be designed so that their goals and behaviors can be assured to align with human values throughout their operation.

“Yes, again I think that’s a very good principle to say explicitly. It’s something that AI researchers have not worried about much before. Stuart Russell is probably the first one who really brought this out to the forefront. I think there’s a general agreement that we have to really pay attention to that issue of Value Alignment. How will we achieve it? Well, there are different approaches and not everybody agrees on that. I think actually we can learn from the ethics and the moral ethics community. Some of these fields where people who have thought for centuries about ethics and moral behavior, will become relevant because these are deep issues, and it’s just great to see that now we’re actually saying let’s try to build systems that have moral principles, even if we don’t know quite what they are.”

18) AI Arms Race: An arms race in lethal autonomous weapons should be avoided.

“So it should be avoided. Again I think this is a little similar to the privacy issue because policymakers should start to think about this issue, and there’s always a difference between it ‘should’ be avoided and ‘can’ it be avoided – that may be a much harder question. But we need to get policymakers aware of it and get the same kinds of discussions as there were around atomic weapons or biological weapons, where people actually start to look at the tradeoffs and the risks of an arms race. That discussion has to be had, and it may actually bring people together in a positive way. Countries could get together and say this is not a good development and we should limit it and avoid it. So to bring it out as a principle, I think the main value there is that we need to have the discussion as a society and with other countries.”

15) Shared Prosperity: The economic prosperity created by AI should be shared broadly, to benefit all of humanity.

“I think that’s a good principle, and again I think Silicon Valley people are already starting to think about this issue. I think economic disparity is accelerating, and I would say more generally that it’s technology that is behind that, and not just AI, but AI is a big factor of it. I think it’s almost a moral principle that we should share benefits among more people in society. I think it’s now down to 8 people who have as much as half of humanity. These are incredible numbers, and of course if you look at that list it’s often technology pioneers that own that half. So we have to go into a mode where we are first educating the people about what’s causing this inequality and acknowledging that technology is part of that cost, and then society has to decide how to proceed. Again it’s more of a political and societal question of how to proceed. From my personal perspective I would like to see the benefits shared. Some people might argue against it but at least I would like to see a discussion about it at a societal level.”

ARIEL: “One thing I’m getting from you is that these principles seem like a good way to start the discussion.”

BART: “Yes, a meeting as we had in the Puerto Rico meeting and the Asilomar meeting, it’s largely among people – academics, technology pioneers – and I think it’s our responsibility to get these issues out to the wider audience and educate the public. It’s not an easy thing to do. During the recent election, technology and unemployment were barely mentioned and people were barely aware of it. The same thing with data privacy issues – people are not quite aware of it.

“There are good reasons why we share data and there are good reasons why we’ll benefit to a large extent from having shared data, but it also is good to have discussions about to what extent people want that and to what extent they want to put limits on some of these capabilities. But it’s something that ultimately, policymakers and the public have to decide.

“As a technology community, we should start making people aware of these issues. I think one of the things that struck me at the meeting is someone gave the example – we may not be far off from generating video of someone saying something. And we see it with the fake news, we can generate text that pretty much sounds like a certain person. We start generating videos that sound like some person. We have to educate people about that. People will start wondering, ‘well is this real or not.’ They have to at least be aware that it could be done.

“So there are some dramatic examples. The other thing I remember is somebody talked about measuring pupil dilation, so basically some neural net learns on physiological responses, and you could detect that with these high-precision cameras. Then you can find out whether somebody is lying or whether somebody likes you when they talk to you, and do the kind of things that now we assume that nobody knows. But that may not be so long anymore, in 5-10 years you might be in an interview and the person on the screen sees exactly what you’re thinking. So these are major changes and we shouldn’t scare everybody, but I think we should tell people and give them some idea of what’s happening and that we have to think about that.”

Q: If all goes well and we’re able to create an advanced, beneficial AI, what will the world look like?

“I think it would be a good world. Automation will replace a lot of work, but work that we might not actually enjoy doing so much. Hopefully we’ll find, and I’m confident we can find other ways for people to feel useful and to have fulfilling lives. But there may be more lives of leisure, of creativity, or arts, or even science. Things that people love doing. So I think if it’s managed well it can be hugely beneficial. I think we already see how our capabilities are extended with smartphones and Google searches and clouds, so people I think in general enjoy the new capabilities that we have. It’s just that the process has to be managed carefully so that people who want to do harm are not taking advantage of it. But overall I can see it working out very nicely, and disease is cured, and all kinds of misery could potentially be eliminated with much better AI, even policymaking – smarter policymaking, smarter decision-making, smarter managing of large numbers of people and systems. So if you think of the upside – there are great upsides. It’s not something we should stop. It’s something we should embrace, but in a well thought out manner.”

Join the discussion about the Asilomar AI Principles!

Dan Weld Interview

The following is an interview with Dan Weld about the Beneficial AI 2017 conference and The Asilomar Principles that it produced. Weld is Professor of Computer Science & Engineering and Entrepreneurial Faculty Fellow at the University of Washington.

Q: From your perspective what were the highlights of the conference?

“One of the highlights was having the chance to interact with such a diverse group of people, including economists and lawyers as well as the technical folks. I also really enjoyed Yann LeCun’s talk, because I hadn’t previously heard his vision for taking a deep neural-network architecture and extending it to learn full agent capabilities.”

Q: Why did you choose to sign the AI principles that emerged from discussions at the conference?

“To be honest, I was a little bit torn, because I had concerns about the wording of many of the principles. Furthermore, some of the proposed principles seemed much more important than others. As a result, the current set was a bit unsatisfying, since it doesn’t reflect the prioritization that I think is important. Overall, however, I thought that the spirit underlying the set of principles was right-on, and it was important to move forward with them – even if imperfect.
“One other comment – I should note that many of the principles hold, in my opinion, even if you replace ‘artificial intelligence’ with any other advanced technology – biotech or big data or pesticides or anything, really. And so specifying “AI” implicitly suggests that the concern is much bigger for AI than it is for these other technologies. However, for many of the principles I don’t think that that’s true.”

Q: Why do you think that AI researchers should weigh in on such issues as opposed to simply doing technical work?

“I think that this type of advocacy is important for any scientist who’s working on any technology, not just for AI researchers. Since we understand the technology better than many other people, we have one important perspective to bring to the table. And so it’s incumbent on us to take that seriously.”

Q: Explain what you think of the following principles:

Shared Prosperity: The economic prosperity created by AI should be shared broadly, to benefit all of humanity.
“AI is having incredible successes and becoming widely deployed. But this success also leads to a big challenge – its impending potential to increase productivity to the point where many people may lose their jobs. As a result, AI is likely to dramatically increase income disparity, perhaps more so than other technologies that have come about recently. If a significant percentage of the populace loses employment, that’s going to create severe problems, right? We need to be thinking about ways to cope with these issues, very seriously and soon. I actually wrote an editorial about this issue.”

AI Arms Race: An arms race in lethal autonomous weapons should be avoided.
“I fervently hope we don’t see an arms race in lethal autonomous weapons. That said, this principle bothered me, because it doesn’t seem to have any operational form. Specifically, an arms race is a dynamic phenomenon that happens when you’ve got multiple agents interacting. It takes two people to race. So whose fault is it if there is a race? I’m worried that both participants will point a finger at the other and say, “Hey, I’m not racing! Let’s not have a race, but I’m going to make my weapons more accurate and we can avoid a race if you just relax.” So what force does the principle have? That said, I think any kind of arms race is dangerous, whether or not AI is involved.”

Capability Caution: There being no consensus, we should avoid strong assumptions regarding upper limits on future AI capabilities.
“I agree! As a scientist, I’m against making strong or unjustified assumptions about anything, so of course I agree. Yet this principle bothers me a bit, because it seems to be implicitly saying that there is an immediate danger that AI is going to become superhumanly, generally intelligent very soon, and we need to worry about this issue. This assertion, which was held by a number of people at the workshop, concerns me because I think it’s a distraction from what are likely to be much bigger, more important, more near term, potentially devastating problems. I’m much more worried about job loss and the need for some kind of guaranteed healthcare, education and basic income than I am about Skynet. And I’m much more worried about some terrorist taking an AI system and trying to program it to kill all Americans than I am about an AI system suddenly waking up and deciding that it should do that on its own.”

ARIEL: “That was along the lines of something Yoshua Bengio talked about, which was that a lot of the conference ended up focusing on how AI design could go wrong and that we need to watch out for that, but he’s also worried about misuse, which sounds like that would be the terrorist stuff that you’re worried about.”

DAN: “Yes, that’s another way of saying it, and I think that’s a much bigger, more immediate concern. I have really nothing against what Nick and Stuart are talking about, I just think other problems are much more urgent.”

Risks: Risks posed by AI systems, especially catastrophic or existential risks, must be subject to planning and mitigation efforts commensurate with their expected impact.
“How could I disagree? Should we ignore the risks of any technology and not take precautions? Of course not. So I’m happy to endorse this one. But it did make me uneasy, because there is again an implicit premise that AI systems have a significant probability of posing an existential risk. I don’t believe that this is something that we should be spending very much time worrying about. Why not? Because they distract us from more important problems such as employment (which I mentioned above).
“Can I insert something here? There is a point that I wanted to make during the workshop but never really found the right time. At the workshop there was a lot of discussion about superhuman-level, artificial general intelligence, but I think what’s going to happen is – long before we get superhuman AGI – we’re going to get superhuman artificial *specific* intelligence. Indeed, we already have! Computers multiply numbers much faster than people can. They play Chess, Jeopardy and Go much better than people can. Computers detect credit card fraud much better than people can. So there’s an increasing number of things that AI computers can do better than people. It probably won’t be very long until computers can drive better than most people as well. Radiology and many types of medical diagnosis may come soon as well.
“These narrower kinds of intelligence are going to be at the superhuman level long before a *general* intelligence is developed, and there are many challenges that accompany these more narrowly described intelligences, even ignoring the fact that maybe someday in the distant future AI systems will be able to build other AI systems. So one thing that I thought was missing from the conference was a discussion more about these nearer-term risks.
“One technology, for example, that I wish had been discussed more is explainable machine learning. Since machine learning is at the core of pretty much every AI success story, it’s really important for us to be able to understand *what* it is that the machine learned. And, of course, with deep neural networks it is notoriously difficult to understand what they learned. I think it’s really important for us to develop techniques so machines can explain what they learned so humans can validate that understanding. For example, an explanation capability will be essential to ensure that a robot has correctly induced our utility function, before it can be trusted with minimal supervision. Of course, we’ll need explanations before we can trust an AGI, but we’ll need it long before we achieve general intelligence, as we deploy much more limited intelligent systems. For example, if a medical expert system recommends a treatment, we want to be able to ask ‘Why?'”

ARIEL: “So do you think that’s something that is technically possible? That’s something that I’ve heard other people comment on – when we bring up issues of transparency or explainability they say, well this won’t be technically possible.”

DAN: “I think it’s still an open question the degree to which we can make our systems explainable and transparent, but I do think it’s possible. In general, any explanation is built atop a number of simplifying assumptions. That’s what makes it comprehensible. And there’s a tricky judgement question about what simplifying assumptions are OK for me to make when I’m trying to explain something to you. Different audiences want different levels of detail, and the listener’s objectives and interests also affect whether an explanation is appropriate. Furthermore, an explanation shouldn’t be one-shot; the AI system needs to answer follow-up questions as well. So there are many challenges there, and that’s why I had hoped that they would get more attention at the workshop.”

ARIEL: “One of the things we tend to focus on at FLI is the idea of existential risks. Do you foresee the possibility that some of these superhuman narrow AI could also become existential risks?”

DAN: “I’m less concerned by existential risks than by catastrophic risks. And narrow AI systems, foolishly deployed, could be catastrophic. I think the immediate risk is less a function of the intelligence of the system than it is about the system’s autonomy, specifically the power of its effectors and the type of constraints on its behavior. Knight Capital’s automated trading system is much less intelligent than Google Deepmind’s AlphaGo, but the former lost $440 million in just forty-five minutes. AlphaGo hasn’t and can’t hurt anyone. If we deploy autonomous systems with powerful effectors we had better have constraints on their behavior regardless of their intelligence. But specifying these constraints is extremely hard, leading to deep questions about utility alignment. I think we need to solve challenges well before we have AGI. In summary, I think that these issues are important even short of the existential risk. … And don’t get me wrong – I think it’s important to have some people thinking about problems surrounding AGI; I applaud supporting that research. But I do worry that it distracts us from some other situations which seem like they’re going to hit us much sooner and potentially cause calamitous harm.
“That said, besides being a challenge in their own right, these superhuman narrow AI systems can be a significant counterbalance to any AGI that gets introduced. See for example, the proposals made by Oren Etzioni about AI guardians – AI systems to monitor other AI systems.”

Personal Privacy: People should have the right to access, manage and control the data they generate, given AI systems’ power to analyze and utilize that data.
“I support that principle very strongly! I’m really quite worried about the loss of privacy. The number of sensors is increasing and combined with advanced machine learning, there are few limits to what companies and governments can learn about us. Now is the time to insist on the ability to control our own data.”

Q: Assuming all goes well, what do you think a world with advanced beneficial AI would look like? What are you striving for with your AI work?

“It’s tricky predicting the future, but there are myriad ways that AI can improve our lives. In the near term I see greater prosperity and reduced mortality due to things like highway accidents and medical errors, where there’s a huge loss of life today.
“In the longer term, I’m excited to create machines that can do the work that is dangerous or that people don’t find fulfilling. This should lower the costs of all services and let people be happier… by doing the things that humans do best – most of which involve social and interpersonal interaction. By automating rote work, people can focus on creative and community-oriented activities. Artificial Intelligence and robotics should provide enough prosperity for everyone to live comfortably – as long as we find a way to distribute the resulting wealth equitably.”

Join the discussion about the Asilomar AI Principles!

Toby Walsh Interview

The following is an interview with Toby Walsh about the Beneficial AI 2017 conference and The Asilomar Principles that it produced. Walsh is Guest Professor at Technical University of Berlin, Professor of Artificial Intelligence at the University of New South Wales, and leads the Algorithmic Decision Theory group at Data61, Australia’s Centre of Excellence for ICT Research.

Q: From your perspective, what were the highlights of the conference?

“I think there’s a real community starting to build and some consensus around some of the interesting questions. These are really important topics and it’s good to see that a lot of people, both within and on the fringes of AI, are starting to take these issues seriously. It’s a very pleasant change from when people would say AI would never succeed, and criticized us for trying to attempt to do it. And now it’s quite pleasant to have people say, well what if you succeed, we may have to worry about this. It’s quite pleasant to be on the other side of people’s criticisms.”

Q: Why did you decide to sign the Asilomar AI Principles document?

“I don’t think it’s perfect, but broadly speaking, it’s hard to disagree with many of the principles. And as a scientist I think we have a responsibility for the beneficial outcomes of our research, and many of the principles were ones that we really, as scientists, should be worrying about. I do think, though, that there’s a lot to be said for less is more … I wonder if perhaps we’ve got a little too many principles down. Some of them were principles I don’t disagree with but I think they applied to research in general or to anything that science does, not just AI, and perhaps we should be focusing more on the ones that are particular to AI.”

Q: Why do you think AI researchers should weigh in on issues like those that were brought up at the conference and in the principles document?

“Because as a scientist you have a responsibility, and I think it’s particularly challenging in an area like AI because the technology can always be used for good or for bad. But that’s true of almost all technologies. It’s hard to think of a technology that doesn’t have both good and bad uses. And so that’s particularly true here, and it seems clear that AI is going to have a large impact upon society, and it’s going to happen relatively quickly – certainly when compared to the Industrial Revolution. In the Industrial Revolution we had to build machines, you had to invest money into building large steam engines, and it didn’t scale as quickly as computing technologies do. So it’s likely that this next revolution is going to happen that much quicker than the Industrial Revolution, which took maybe 50 years to start having real impact. This, when it gets going, is going to perhaps take less time than that, potentially.”

ARIEL: “Do you think it’s already gotten going?”

TOBY: “Well, you can already see the shoots of it happening in things like autonomous cars that are less than a decade away. There are dozens of trials starting up this year around the planet. Trials on autonomous taxis, autonomous trucks, autonomous cars. So if you’re a truck driver or a taxi driver you have been given less than a decade’s warning that your job is at severe risk. And so I do think there are going to be interesting consequences, and that’s just in the short-term. In the long-term there are going to be much more profound changes. But we should really start thinking about some of these issues quite soon because there isn’t a lot of time to solve some of them – if you’re a taxi driver or a truck driver, for sure. But society tends to change very slowly, it’s always catching up on technology.”

Q. Explain what you think of the following principles:

15) Shared Prosperity: The economic prosperity created by AI should be shared broadly, to benefit all of humanity.
“I think that’s a very laudable principle. It’s a real fundamental problem facing our society today, which is the increasing inequality and the fact that prosperity is not being shared around. This is fracturing our societies and we see this in many places, in Brexit, in Trump. A lot of dissatisfaction within our societies. So it’s something that we really have to fundamentally address. But again, this doesn’t seem to me something that’s really particular to AI. I think really you could say this about most technologies. Many scientists like myself are funded by the public purse for the public good, and so we have a responsibility to ensure that we work on technologies and we ensure technologies do benefit all. So that’s something that doesn’t seem special about AI, although AI is going to amplify some of these increasing inequalities. If it takes away people’s jobs and only leaves wealth in the hands of those people owning the robots, then that’s going to exacerbate some trends that are already happening.”

18) AI Arms Race: An arms race in lethal autonomous weapons should be avoided.
“One reason that I got involved in these discussions is that there are some topics I think are very relevant today, and one of them is the arms race that’s happening amongst militaries around the world already, today. This is going to be very destabilizing. It’s going to upset the current world order when people get their hands on these sorts of technologies. It’s actually stupid AI that they’re going to be fielding in this arms race to begin with and that’s actually quite worrying – that it’s technologies that aren’t going to be able to distinguish between combatants and civilians, and aren’t able to act in accordance with international humanitarian law, and will be used by despots and terrorists and hacked to behave in ways that are completely undesirable. And that’s something that’s happening today. You have to see the recent segment on 60 Minutes to see the terrifying swarms of robot UAVs that the American military is now experimenting with.”

ARIEL: “I talked to Yoshua Bengio a couple days ago, too, and one of the things he commented on about the conference was that our big focus seems to be on value alignment and us creating AI that doesn’t do what we wanted it to do, but he said another concern that he’s realizing is the idea of people misusing AI. What’s your take on that? It seemed like something that connects to what you were saying here with the arms race.”

TOBY: “I’m very worried that people will misuse AI. These technologies can be used for good or for evil, and we literally get to choose which of those it’s going to be. And if we go ahead unchecked, I think it’s clear that there are certain groups and individuals who will use it for evil and not for good. And to go back to his observation that perhaps we’re focusing too much on longer-term things like value alignment, I actually suspect that people will start to realize that value alignment is a problem that we already have today, that we’re already building systems that implicitly or explicitly have values or display values that are not aligned with ours. And we can already see this. The Tay chatbot didn’t share our community values about racism or sexism or misogyny, or the freedom of speech that we would like to have. The Compass program that is discriminating in its recommendations for sentencing in 20 of the 52 states in the US, doesn’t share our values about racism – it discriminates against black people. One of the impressions I came away from the conference with is that people think issues like value alignment are long-term research issues. I think they’re not just short-term research issues, they’re problems that already plague us. We don’t have solutions for them already. And they’re already impacting upon people’s lives. And that’s a value alignment problem we face today. It’s not a problem for superintelligent systems; it’s a problem for stupid AI systems that we build today. And, of course, if we build superintelligent AI systems we’ve only amplified the challenges, but I think we have many of those challenges today. But actually I think that’s a hopeful observation, because if we can solve it for these simple systems, for narrow-focused domains, for narrow-focused values, then hopefully the sorts of solutions we come up with will be things that will be components of value alignment for more intelligent systems in the future. Hopefully we’ll get our training wheels on the primitive AI systems we have today and will have tomorrow to help us solve the problem for much more intelligent systems decades or centuries away from today.”

10) Value Alignment: Highly autonomous AI systems should be designed so that their goals and behaviors can be assured to align with human values throughout their operation.
“First of all, I don’t know why we have the word highly. I think any autonomous system, even a lowly autonomous system, should be aligned with human values. I’d wordsmith away the ‘high’. Other than that, I think that we have to worry about enforcing that principle today. As I’ve said, I think that will be helpful in solving the more challenging value alignment problem as systems get more sophisticated.”

20) Importance: Advanced AI could represent a profound change in the history of life on Earth, and should be planned for and managed with commensurate care and resources.
“Yes, and again this is one of those principles where I think you could put any society-changing technology in place of advanced AI. So this is not a principle that’s true of AI, it’s true of any groundbreaking technology. It would be true of the steam engine, in some sense it’s true of social media and we’ve failed at that one, it could be true of the Internet but we failed at planning that well. It could be true of fire too, but we failed on that one as well and used it for war. But to get back to the observation that some of them are things that are not particular to AI – once you realize that AI is going to be groundbreaking, then all of the things that should apply to any groundbreaking technology should apply.”

12) Personal Privacy: People should have the right to access, manage and control the data they generate, given AI systems’ power to analyze and utilize that data.
“Yes, this is a great one, and actually I’m really surprised how little discussion we have around AI and privacy. I thought there was going to be much more fallout from Snowden and some of the revelations that happened, and AI, of course, is enabling technology. If you’re collecting all of this data, the only way to make sense of it is to use AI, so I’ve been surprised that there hasn’t been more discussion and more concern amongst the public around these sorts of issues.”

Q: If all goes well and we’re able to create an advanced, beneficial AI, what will the world look like?

“I always say to people, OK, there are all of these risks and challenges that we have to solve, but also, the technology is about our only hope to solve not only these risks but all of the problems facing the planet like global warming, global financial crisis, the global terrorism problem, the global refugee problem, all of these problems. If our children are going to have better lives than ours, this is about the only technology in play to solve them.
“If we follow the good path, the world would be a lot happier. But we get to choose. There are good paths and bad paths to be followed, and we literally get to choose the path we get to follow. And it doesn’t even require us to get to artificial general intelligence. Even if we continue to only be able to build focused, specialized AIs, again it’s the only hope to solve the problems facing society. The last seventy-five years of economic growth have come from information technology. There are also a few other technologies like biotech and nanotech in play today. A large part of our economic prosperity is going to come from IT. The world is getting more digital, and so if our children are going to live as good lives as ours, it’s going to come from technology, and it’s going to largely come from IT. The immense benefits, the fact that most of us in the first world do live such comfortable lives, and the fact that many people in the third world are being lifted out of poverty – sometimes we forget all of this, but the third world is also getting better- despite the fact inequality is increasing, but technology has brought that and a lot of that is from IT. So it’s our only hope for the future.”

Q: Is there anything else that you wanted to add that you think is important?

“I guess I wanted to say one more thing about the principles, about the idea that there are perhaps too many principles. One reason why I prefer to have fewer principles is because if you can have a few simple, very general principles, then just like the Founding Fathers couldn’t foresee the future and all of the challenges that face the US going forward, but by having some fundamental general principles you have more hope that they’re still going to be applicable in 50, 100, 200 years time. And so I just worry that some of them are a little too specific to the things that we can see in front of us today, and that there will be other issues that come up.
“Of course, it’s just a start. And to remember the historical analogy even with the US Constitution, there have been a number of amendments made to it, and amendments made to the amendments. It’s always a work in progress.”

Join the discussion about the Asilomar AI Principles!

Stefano Ermon Interview

The following is an interview with Stefano Ermon about the Beneficial AI 2017 conference and The Asilomar Principles that it produced. Ermon is an Assistant Professor in the Department of Computer Science at Stanford University, where he is affiliated with the Artificial Intelligence Laboratory.

Q: From your perspective what were highlights of the conference?

“I really liked the technical part at the beginning. I saw a lot of really concrete research problems that people can start working on, and I thought that people had made a lot of interesting progress in the last year or so. It was really nice to see all these smart people working on these problems and coming up with questions and partial solutions – it’s like the beginning of a new research area.”

Q: Why did you choose to sign the AI principles that emerged from discussions at the conference?

“It seemed balanced. The only worry is that you don’t want it to be too extreme, but I thought that [FLI] did a very good job of coming up with principles that I think lots of people can potentially agree on. It identifies some important issues that people should be thinking about more, and if by signing that letter we can get slightly more attention to the problem, then I think that’s a good thing to do.”

Q: Why do you think that AI researchers should weigh in on such issues as opposed to simply doing technical work?

“Because there might actually be a technical solution to some of these problems, but not to all of them. There are some inherent tradeoffs that people will have to discuss and we will have to come up with the right ways to balance everybody’s needs, and the different instabilities of different problems. But on some of the issues I think we should try to do as much as possible by trying to find technological solutions, and I think that would make the discussion more scientific. In this way it’s not purely based on speculation and we don’t leave it to non-experts, but it becomes more grounded on what AI really is.”

Q: Explain what you think of the following principles.

Shared Prosperity: The economic prosperity created by AI should be shared broadly, to benefit all of humanity.
“I think it’s very important that we make sure that AI is really for everybody’s benefit – that it’s not just going to be benefitting a small fraction of the world’s population, or just a few large corporations. And I think there is a lot that can be done by AI researchers just by working on very concrete research problems where AI can have a huge impact. I’d really like to see more of that research work done.”

18) AI Arms Race: An arms race in lethal autonomous weapons should be avoided.
“I’m not a fan of wars, and I think it could be extremely dangerous. Obviously I think that the technology has a huge potential, and even just with the capabilities we have today it’s not hard to imagine how it could be used in very harmful ways. I don’t want my contributions to the field and any kind of techniques that we’re all developing to do harm to other humans or to develop weapons or to start wars or to be even more deadly than what we already have.”

19) Capability Caution: There being no consensus, we should avoid strong assumptions regarding upper limits on future AI capabilities.
“I think that it’s always hard to predict the future. At the moment I don’t think there is any consensus on the limits of what AI can do, so it’s better not to make any assumption on what we will not be able to achieve. Think about what people were imagining a hundred years ago, about what the future would look like. At the beginning of the last century they were saying- how will the future look in 100 years? And I think it would’ve been very hard for them to imagine what we have today. I think we should take a similar, very cautious view, about making predictions about the future. If it’s extremely hard, then it’s better to play it safe.”

20) Importance: Advanced AI could represent a profound change in the history of life on Earth, and should be planned for and managed with commensurate care and resources.
“It’s an incredibly powerful technology. I think it’s even hard to imagine what one could do if we are able to develop a strong AI, but even before that, well before that, the capabilities are really huge. We’ve seen the kind of computers and information technologies we have today, the way they’ve revolutionized our society, our economy, our everyday lives. And my guess is that AI technologies would have the potential to be even more impactful and even more revolutionary on our lives. And so I think it’s going to be a big change and it’s worth thinking very carefully about, although it’s hard to plan for it.”

12) Personal Privacy: People should have the right to access, manage and control the data they generate, given AI systems’ power to analyze and utilize that data.
“I think that’s a big immediate issue. I think when the general public thinks about AI safety, maybe they think about killer robots or these kind of apocalyptic scenarios, but there are big concrete issues like privacy, fairness, and accountability. The more we delegate decisions to AI systems, the more we’re going to run into these issues. Privacy is definitely a big one, and one of the most valuable things that these large corporations have is the data they are collecting from us, so we should think about that carefully.”

10) Value Alignment: Highly autonomous AI systems should be designed so that their goals and behaviors can be assured to align with human values throughout their operation.
“It seems like a natural thing to do. What else would you do? It’s hard to imagine not to try to achieve this goal. Why would you ever want to develop a highly intelligent system that is designed to harm us? It is something that I think the majority of people would agree on, but the issue, of course, is to define what exactly these values are, because people have different cultures, come from different parts of the world, and have different socioeconomic backgrounds,  so they will have very different opinions on what those values are. That’s really the challenge. But assuming it’s possible to agree on a set of values, then I think it makes sense to strive for those and develop technology that will allow us to get closer to those goals.”

Q: Assuming all goes well, what do you think a world with advanced beneficial AI would look like? What are you striving for with your AI work?

“I think there’s hopefully going to be greater prosperity. Hopefully we’re going to be able to achieve a more sustainable society, we’re going to speed up the scientific discovery process dramatically, we might be able to discover new sources of clean energy, and we might find ways to manage the planet in a more sustainable way. It’s hard to imagine but the potential could be huge to really create a better society.”

Join the discussion about the Asilomar AI Principles!

Kay Firth-Butterfield Interview

The following is an interview with Kay Firth-Butterfield about the Beneficial AI 2017 conference and The Asilomar Principles that it produced. Firth-Butterfield is the Executive Director of AI-Austin.org, and an adjunct Professor of Law at the University of Texas at Austin.

Q. From your perspective what were the highlights of the conference?

“The opportunity to meet old friends and colleagues but also to meet and hear the views of new people of who are making important contributions to our work. It was a super interdisciplinary gathering. Also, it was a very interesting and valuable choice of speaking and panel topics.”

Q. Why did you choose to sign the AI principles that emerged from discussions at the conference?

“As Vice-Chair of the IEEE Global Initiative for Ethical Considerations in Artificial Intelligence and Autonomous Systems and Executive Director of AI-Austin I am looking at ways of using the important discussions we are having in practical ways, for example creating standards through our work at IEEE or, at AI-Austin, practical use cases in our community. Thus, signing the principles is very important to me. It is vital that this interdisciplinary group representing academia, business and society starts setting out principles and showing how much we are doing, and need to do, to create safe, beneficial AI.”

Q. Why do you think that AI researchers should weigh in on such issues as opposed to simply doing technical work?

“AI will change everything and will do so at a fast pace. Those who research in the discipline of AI are well suited to inform the discussion and help shape it so that responsible beneficial design is at the forefront of policy decisions at international, national, business and community levels.”

Q. Explain what you think of the following principles:

15) Shared Prosperity: The economic prosperity created by AI should be shared broadly, to benefit all of humanity.
“My background is in law and international relations. AI is a technology with such great capacity to benefit all of humanity but also the chance of simply exacerbating the divides between the developed and developing world, and the haves and have nots in our society. To my mind that is unacceptable and so we need to ensure, as Elon Musk said, that AI is truly democratic and its benefits are available to all.”
18) AI Arms Race: An arms race in lethal autonomous weapons should be avoided.
“I believe that any arms race should be avoided but particularly this one where the stakes are so high and the possibility of such weaponry, if developed, being used within domestic policing is so terrifying.”
20) Importance: Advanced AI could represent a profound change in the history of life on Earth, and should be planned for and managed with commensurate care and resources.
“I believe that AI will create profound change even before it is ‘advanced’ and thus we need to plan and manage growth of the technology. As humans we are not good at long-term planning because our civil systems don’t encourage it, however, this is an area in which we must develop our abilities to ensure a responsible and beneficial partnership between man and machine.”
12) Personal Privacy: People should have the right to access, manage and control the data they generate, given AI systems’ power to analyze and utilize that data.
“I would re-write this principle as “Given AI systems’ power to analyze and utilize data, people should have the right to access, manage and control the data they generate. I agree with this principle on a number of levels.

  • a. for the reasons expounded by the IEEE Initiative’s privacy committee
  • b. as AI becomes more powerful we need to take steps to ensure that it cannot use our personal data against us if it falls into the wrong hands
  • c. Data is worth money and as individuals we should be able to choose when and how to monetize our own data whilst being encouraged to share data for public health and other benefits.”

10) Value Alignment: Highly autonomous AI systems should be designed so that their goals and behaviors can be assured to align with human values throughout their operation.
“Yes!”

Q. Assuming all goes well, what do you think a world with advanced beneficial AI would look like? What are you striving for with your AI work?

“I would like to see a world in which all can equally benefit from the use of beneficial, ethically and responsibly designed AI. I am working to help make that a reality and create an environment where AI enables humans to get the best out of ourselves, create the greatest good for humanity as a whole and achieve excellent outcomes for the flora and fauna with which we share our planet.”

Join the discussion about the Asilomar AI Principles!

Francesca Rossi Interview

The following is an interview with Francesca Rossi about the Beneficial AI 2017 conference and The Asilomar Principles that it produced. Rossi is a research scientist at the IBM T.J. Watson Research Centre, and a professor of computer science at the University of Padova, Italy, currently on leave.

Q: From your perspective, what were the highlights of the conference?

“In general, what was great about this conference was the very diverse mix of people. It’s important that this was done on a very multidisciplinary level with experts of many different disciplines – not just AI people.
“But, of course, the days of the conference and workshop that I liked the most were the ones devoted to the research agenda – what people have been doing to make AI more ethical and to have an even more beneficial impact on the real world.
“I think that it’s very important that more and more people in AI and in general understand that these issues of beneficial AI, of ethical implications, and of moral agency are not just the subject of discussion, but are really the subject of concrete research efforts.
“And then, I think the first day of the conference – the one devoted to economics – it was great. They had great talks and great panels. And that’s, of course, a very important issue that everybody is reflecting on.”

Q: And why did you choose to sign the AI Principles?

“I think that was a very interesting and very big step forward to put together all these principles. Then during the conference, people had the chance to say which ones they agreed or didn’t agree on. And we realized how much consensus there is among the people in so many different disciplines, with so many different backgrounds, and on a large set of principles.
“So signing at least the principles that basically everybody unanimously agreed on and giving support to that is very important in trying to guide the community. The principles were already there, but they were not – until now – explicitly written down. I think it is very good that people can read them.
“In Puerto Rico, two years ago, the discussion was not happening around these issues, and things were not as clear as now. We could not have done that kind of effort with the principles, but now I think it’s the right time. And it’s a big step forward in the discussion.”

Q: And then, why do you think it’s important for AI researchers to weigh in on issues like these?

“Like I said, I think it’s important that AI research is the center of discussion. But I would not say that these are two contradictory things: discussing these issues as opposed to doing technical work. Because I think one of the main accomplishments of FLI is that it makes clear that these subjects are also subject to concrete technical research efforts. And before, this was not clear.
“With the FLI grant program, we realize that actually addressing those issues and possibly moving forward to resolving them is also a matter of doing concrete, technical work.
“So, I think that we should make it clear to AI researchers that they can do research and publish scientific papers on these issues as well.”

Q: And then, going into some of the individual principles, were there a few that you were specifically interested in?

15) Shared Prosperity: The economic prosperity created by AI should be shared broadly, to benefit all of humanity.
“I think it’s very important. And it also ties in with the general effort and commitment by IBM to work a lot on education and re-skilling people to be able to engage with the new technologies in the best way. In that way people will be more able to take advantage of all the potential benefits of AI technology.
“That also ties in with the impact of AI on the job market and all the other things that are being discussed. And they are very dear to IBM as well, in really helping people to benefit the most out of the AI technology and all the applications.”

19) Capability Caution: There being no consensus, we should avoid strong assumptions regarding upper limits on future AI capabilities.
“I personally am for building AI systems that augment human intelligence instead of replacing human intelligence. And I think that in that space of augmenting human intelligence there really is a huge potential for AI in making the personal and professional lives of everybody much better. I don’t think that there are upper limits of the future AI capabilities in that respect.
“I think more and more AI systems together with humans will enhance our kind of intelligence, which is complementary to the kind of intelligence that machines have, and will help us make better decisions, and live better, and solve problems that we don’t know how to solve right now. I don’t see any upper limit to that.”

12) Personal Privacy: People should have the right to access, manage and control the data they generate, given AI systems’ power to analyze and utilize that data.
“That’s definitely a principle that I think is very important. People should really have the right to own their privacy, and companies like IBM or any other that provide AI capabilities and systems should protect the data of their clients.
“The quality and amount of data is essential for many AI systems to work well, especially in machine learning. But the developers and providers of AI capabilities should really take care of this data in the best way. This is fundamental in order to build this trust between users of AI systems and those that develop and deploy the AI systems, like IBM or any other company.
“It’s also very important that these companies don’t just assure that they are taking care of the data, but that they are transparent about the use of the data. Without this transparency and trust, people will resist giving their data, which would be detrimental to the AI capabilities and the help AI can offer in solving their health problems, or whatever the AI is designed to solve.”

10) Value Alignment: Highly autonomous AI systems should be designed so that their goals and behaviors can be assured to align with human values throughout their operation.
“The one closest to my heart. I definitely agree with this principle. AI systems should behave in a way that is aligned with human values.
“But actually, I would be even more general than what you’ve written in this principle. Because this principle has to do not only with autonomous AI systems, but I think this is very important and essential also for systems that work tightly with humans in the loop, and also where the human is the final decision maker. Because when you have human and machine tightly working together, you want this to be a real team. So you want the human to be really sure that the AI system works with values aligned to that person. It takes a lot of discussion to understand those values.
“For every job, for every task, we need to write down the values or the principles we want to focus on in that scenario. And then, again, there is scientific research that can be undertaken to actually understand how to go from these values that we all agree on to embedding them into the AI system that’s working with humans. So this principle – value alignment – is a very important principle and a big technical challenge as well.”

ARIEL: “I have a quick follow-up question on that one. I was talking to Toby Walsh earlier today, and we were talking about value alignment. And one of his comments was that the way we’ve been approaching the principles is that this is something we have to deal with in the future, and that value alignment is something we need to worry about as machines get more intelligent and we get closer to a superintelligent AI or something like that. But his comment was that it’s an issue now; that we’re having these issues today.”

FRANCESCA: I agree completely that value alignment is something to be solved not because we think that there will be some sort of superintelligence, but right now. With the machines and AI systems that work right now with doctors or other professionals – we want these machines to behave in a way that is aligned with what we would expect for a human. So I agree that value alignment is a very big challenge and should be solved as soon as possible.

Q: Assuming that all goes well and we achieve the advanced beneficial AI that we’re hoping for, how do you envision that world? What does that look like to you?

“Well, first I think that we don’t have to wait for long to actually see a new world. I mean, the new world is already here, with the AI systems being more and more pervasive in the professional and private lives of people.
“I don’t know if we are able to predict the far future, but I think in a very short time there will be a world where the human-machine relationship will be tighter and tighter. And hopefully in the most beneficial way, so that the quality of work and life of people will be much higher, and that people will be able to trust these machines at the correct level.
“And with the effort that we make in making sure these machines really get the best capabilities in interacting with humans and explaining what they do to humans… all this will really help in making this relationship better and better.”

Q: And was there anything else that you thought was important that you wanted to add, that wasn’t part of the questions?

“I think that at this stage of the conversation and discussion it is very important that everybody engages in the discussion and also engages in educational efforts. I don’t just mean graduate and undergraduate curriculum, but also to selected target groups like business people or policymakers or the media – telling them and educating them about what AI really is, what the state of the art is, what the current capabilities and limitations are, what the potential is, what the concerns are. I think this will be very helpful in shaping the discussion and guiding it in the right way.”

Join the discussion about the Asilomar AI Principles!

Roman Yampolskiy Interview

The following is an interview with Roman Yampolskiy about the Beneficial AI 2017 conference and The Asilomar Principles that it produced. Yampolskiy is an Associate Professor of Computer Engineering & Computer Science at the University of Louisville and the Founding Director of the Cyber Security Lab.

Q. From your perspective what were the highlights of the conference?

“The Conference brought together an unprecedented number of thought leaders from industry, academia, non-profits, NGOs, US government, the UN and charities. It was wonderful to see so much cognitive diversity with dozens of different domains represented – philosophy, computer science, economics, security, policy, physics, law, political science, to name just a few. Participants were able to share their ideas in an atmosphere of freedom guaranteed by the Chatham House Rule.

“I personally benefited the most from insider information I got from AI industry leaders, which will certainly help me guide and improve my future work. It was also great to see that AI Safety is no longer a fringe domain of computer science but a major area of research in AI which is now recognized as important by people who have the potential to impact the future of AI as a transformative technology and by extension the future of humanity.

“I suspect that in a decade this Asilomar conference will be considered as important as the Asilomar Recombinant DNA conference from 1975.”

Q. Why did you choose to sign the AI principles that emerged from discussions at the conference?

“The principles address issues of fundamental importance, and while we are far from offering any settled scientific solutions to most of them, it is important to show the world that they are taken seriously by the leaders in the field. The suggested principles also do a great job of articulating important directions for future research. It was an honor for me to endorse them along with many distinguished colleagues.”

Q. Why do you think that AI researchers should weigh in on such issues as opposed to simply doing technical work?

“In my opinion, the principles outlined at Asilomar contain the most important unfinished technical work. For example “Value Alignment” is just a fancy way of saying that we actually should retain control of our robots and digital assistants, instead of them deciding what to do with us. Similarly, maintaining “Personal Privacy” requires sophisticated filtering software to work alongside our data-mining algorithms.”

Q. Explain what you think of the following principles:

15) Shared Prosperity: The economic prosperity created by AI should be shared broadly, to benefit all of humanity.

“Given that all the jobs (physical and mental) will be gone, it is the only chance we have to be provided for.”

18) AI Arms Race: An arms race in lethal autonomous weapons should be avoided.

“Weaponized AI is a weapon of mass destruction and an AI Arms Race is likely to lead to an existential catastrophe for humanity.”

19) Capability Caution: There being no consensus, we should avoid strong assumptions regarding upper limits on future AI capabilities.

“In many areas of computer science such as complexity or cryptography the default assumption is that we deal with the worst case scenario. Similarly, in AI Safety we should assume that AI will become maximally capable and prepare accordingly. If we are wrong we will still be in great shape.”

20) Importance: Advanced AI could represent a profound change in the history of life on Earth, and should be planned for and managed with commensurate care and resources.

“Design of human-level AI will be the most impactful event in the history of humankind. It is impossible to over-prepare for it.”

21) Risks: Risks posed by AI systems, especially catastrophic or existential risks, must be subject to planning and mitigation efforts commensurate with their expected impact.

“Even a small probability of existential risk becomes very impactful once multiplied by all the people it will impact. Nothing could be more important than avoiding the extermination of humanity.”

12) Personal Privacy: People should have the right to access, manage and control the data they generate, given AI systems’ power to analyze and utilize that data.

“The world’s dictatorships are looking forward to opportunities to target their citizenry with extreme levels of precision. The tech we will develop will most certainly become available throughout the world and so we have a responsibility to make privacy a fundamental cornerstone of any data analysis.”

10) Value Alignment: Highly autonomous AI systems should be designed so that their goals and behaviors can be assured to align with human values throughout their operation.

“It is very difficult to encode human values in a programming language, but the problem is made more difficult by the fact that we as humanity do not agree on common values and even parts we do agree on change with time.”

Q. Assuming all goes well, what do you think a world with advanced beneficial AI would look like? What are you striving for with your AI work?

“I am skeptical about our chances to create beneficial AI. We may succeed in the short term with narrow domain systems but intelligence and control are inversely related and as superintelligent systems appear we will lose all control over them. The future may not have a meaningful place for us. In my work I am trying to determine if the control problem is solvable, what obstacles are in our way and perhaps buy us a bit more time to look for a solution if one exists. In my book Artificial Superintelligence: a Futuristic Approach I talk about a number of important problems we would have to solve to have any chance for success, such as wire-heading, boxing, and utility function security.”

Join the discussion about the Asilomar AI Principles!

Guruduth Banavar Interview

The following is an interview with Guruduth Banavar about the Beneficial AI 2017 conference and The Asilomar Principles that it produced. Banavar works for IBM as the VP of IBM Research and the Chief Science Officer of Cognitive Computing.

Q. From your perspective what were the highlights of the conference?

“Absolutely the best thing was meeting people,” Banavar explained, saying that he had many “transformative conversations.”

Q. Why did you choose to sign the AI principles that emerged from discussions at the conference?

“The general principles, as they’re laid out, are important for the community to rally around and to use to dig deeper into their research.”

He explained that some are obvious and “naturally based on principles of human rights and social principles.” But some we still need to educate the public and the community about, and some still need more research.

That’s why he signed – he felt that the principles do three things:
1) They resonate with our fundamental rights and liberties.
2) They require education and open discussion in the community.
3) And several require deep research.

Q. Explain what you think of the following principles:

15) Shared Prosperity: The economic prosperity created by AI should be shared broadly, to benefit all of humanity.

“I agreed but it needs rephrasing. This is broader than AI work. Any AI prosperity should be available for the broad population. Everyone should benefit and everyone should find their lives changed for the better. This should apply to all technology – nanotechnology, biotech – it should all help to make life better. But I’d write it as ‘prosperity created by AI should be available as an opportunity to the broadest population.’”

19) Capability Caution: There being no consensus, we should avoid strong assumptions regarding upper limits on future AI capabilities.

“Makes sense. I think about the long-term issue off and on, and the general idea is that intelligence as we understand it today is ultimately the ability to process information from all possible sources and to use that to predict the future and to adapt to the future. It is entirely in the realm of possibility that machines can do that. … I do think we should avoid assumptions of upper limits on machine intelligence because I don’t want artificial limits on how advanced AI can be.”

20) Importance: Advanced AI could represent a profound change in the history of life on Earth, and should be planned for and managed with commensurate care and resources.

“I strongly believe this. I think this goes back to evolution. From the evolutionary point of view, humans have reached their current level of power and control over the world because of intelligence. … AI is augmented intelligence – it’s a combination of humans and AI working together. And this will produce a more productive and realistic future than autonomous AI, which is too far out. In the foreseeable future, augmented AI – AI working with people – will transform life on the planet. It will help us solve the big problems like those related to the environment, health, and education.”

12) Personal Privacy: People should have the right to access, manage and control the data they generate, given AI systems’ power to analyze and utilize that data.

“It’s absolutely crucial that individuals should have the right to manage access to the data they generate. … AI does open new insight to individuals and institutions. It creates a persona for the individual or institution – personality traits, emotional make-up, lots of the things we learn when we meet each other. AI will do that too and it’s very personal. I want to control how [my] persona is created. A persona is a fundamental right.”

9) Responsibility: Designers and builders of advanced AI systems are stakeholders in the moral implications of their use, misuse, and actions, with a responsibility and opportunity to shape those implications.

“This one I particularly care about. The community of AI researchers and developers carries a significant responsibility to think about, incorporate, and not compromise our values. The community needs to take this more seriously. It’s a meta-level principle.

“In all cases, we should take more responsibility in incorporating the right principles into AI activities.”

Q. Assuming all goes well, what do you think a world with advanced beneficial AI would look like? What are you striving for with your AI work?

“I look at the future of the world as a place where AI redefines industry, professions, and experts, and it does so in every field. If one looks at the impact from AI on different fields, each one will be redefined. We will be better equipped to solve the hardest problems, like those of global warming, health, and education.”

Join the discussion about the Asilomar AI Principles!

Yoshua Bengio Interview

The following is an interview with Yoshua Bengio about the Beneficial AI 2017 conference and The Asilomar Principles that it produced. Bengio is a Professor of Computer Science and Operations Research at the University of Montreal and Head of the Montreal Institute for Learning Algorithms (MILA).

Q. From your perspective what were the highlights of the conference?

“The first day, the economists, I really enjoyed.” Especially Jeffrey Sachs’s presentation – Bengio says he tends to agree with a lot of the things Sachs said. Specifically, he liked how the data presented during the economics talks illustrated the effects of automation and how that gives us a hint of what could happen in the future.

“One thing I came with is also … this subject of safe AI came [up] in many discussions, and I would say that these discussions left a strong impression on me. And there was not just discussion about fear, but a lot of interesting technical things, like the presentation by Stuart Russell. He had pretty well thought out proposals. I found that pretty inspiring. In general, the debates about safe AI gave me ideas.”

Paraphrased: One issue that was raised, but not enough, was the question of misuse of AI technology in the future. As the technology becomes more powerful in the future, that may become a bigger issue for the safety of everyone. AI safety can be looked at from two angles – and both were discussed – but most people, like Stuart Russell, talked about the first. That is they looked at how we could be in danger of an AI misusing itself – an AI not doing what we meant. But there’s the other safety issue, which is people not using AI in ethical ways.

“I found, in particular, the discussion about the military use very interesting. Heather [Roff’s] tone was quite different from the other presentations, which was good because I think we do need a wake up call. In particular, I hadn’t realized that military were already playing a kind of play of words to obfuscate the actual use of AI in weapons. So that got me concerned.

“How do we model or even talk about human values or human ethics in a way that we can get computers to follow those human morals? I think this is a hard question, and there are different approaches to it.

“Wendell Wallach had some interesting things to say, but he came at the issue from a different perspective. The way these issues will be dealt with through machine learning approaches and deep learning, at least in my group, is probably going to bring a very different color.”

ARIEL: “It’s sounding like before the conference, you weren’t thinking about AI safety quite as much?”

YOSHUA: “True. [The conference] has been very useful for me in the context of doing this grant, and understanding better the community – it’s not a community I really knew before, so it has been very useful.”

Q. Why did you choose to sign the AI principles that emerged from discussions at the conference?

“I think it is important to send a message, even if I didn’t agree with all the wordings as they were. Overall, I think they were quite aligned with what I think. I like the idea of having a collective statement because, overall, our group – I don’t think it’s represented by what the media or decision-makers understand of the issues. So I think it is important to send these kinds of messages, and that we have some authority that we should be using.”

Q. Why do you think that AI researchers should weigh in on such issues as opposed to simply doing technical work?

“I feel very strongly that I’m going to be much more comfortable with my own technical work if I know that I’m acting as a responsible person, in general, with respect to this work. And that means talking to politicians, talking to the media, thinking about these issues, even if they’re not the usual technical things. We can’t leave it in the hands of just the usual people. I think we need to be part of the discussion. So I feel compelled to participate. … I’m really happy to see all these young people caring about these issues.”

Q. Explain what you think of the following principles:

15) Shared Prosperity: The economic prosperity created by AI should be shared broadly, to benefit all of humanity.

“I’m a very progressive person so I feel very strongly that dignity and justice mean wealth is redistributed. And I’m really concerned about AI worsening the effects and concentration of power and wealth that we’ve seen in the last 30 years. So this is pretty important for me.

“I consider that one of the greatest dangers is that people either deal with AI in an irresponsible way or maliciously – I mean for their personal gain. And by having a more egalitarian society, throughout the world, I think we can reduce those dangers. In a society where there’s a lot of violence, a lot of inequality, the risk of misusing AI or having people use it irresponsibly in general is much greater. Making AI beneficial for all is very central to the safety question.”

18) AI Arms Race: An arms race in lethal autonomous weapons should be avoided.

Bengio signed the open letter on autonomous weapons, so that says it all.

12) Personal Privacy: People should have the right to access, manage and control the data they generate, given AI systems’ power to analyze and utilize that data.

“I agree with that.”

Bengio had reservations about the explainability or justifiability of decisions based on AI because he thinks that may not be technically feasible in the way that some people would like. “We have to be careful with that because we may end up barring machine learning from publicly used systems, if we’re not careful.” But he agrees with the underlying principle which is that “we should be careful that the complexity of AI systems doesn’t become a tool for abusing minorities or individuals who don’t have access to understand how it works.

“I think this is a serious social rights issue, but the solution may not be as simple as saying ‘it has to be explainable,’ because it won’t be [explainable].”

10) Value Alignment: Highly autonomous AI systems should be designed so that their goals and behaviors can be assured to align with human values throughout their operation.

“I agree, except ‘assured’ is maybe strong. It may not be possible to be completely aligned. There are a lot of things that are innate, which we won’t be able to get by machine learning, and that may be difficult to get by philosophy or introspection, so it’s not totally clear we’ll be able to perfectly align. I think the wording should be something along the lines of ‘we’ll do our best.’ Otherwise, I totally agree.”

Q. Assuming all goes well, what do you think a world with advanced beneficial AI would look like? What are you striving for with your AI work?

“In the last few years, the vast majority of AI research and machine learning research has been very, very much influenced by the IT industry to build the next gadget, better phone, better search engines, better advertising, etc. This has been useful because we can now recognize images much better and understand languages much better. But I believe it’s high time that – and I see it happening – that researchers in both academia and industry should look at applications of machine learning that are not necessarily going to make a profit, but the main selling point for doing the research is you can have a really positive impact for a lot of people. … I would be delighted if we found applications in other areas like the environment or fighting poverty.”

Join the discussion about the Asilomar AI Principles!

Anca Dragan Interview

The following is an interview with Anca Dragan about the Beneficial AI 2017 conference and The Asilomar Principles that it produced. Dragan is an Assistant Professor in the Electrical Engineering & Computer Sciences Department at UC Berkeley and co-PI for the Center for Human Compatible AI.

Q. From your perspective what were the highlights of the conference?

“I think a big highlight was the mixture of people – the fact that we had people from industry and academia, and people from diverse fields: computer science, economics, law, philosophy.

“I like that there was a variety of topics: not just what would happen with a superintelligence, but also economic and policy issues around jobs, for instance.”

Q. Why did you choose to sign the AI principles that emerged from discussions at the conference?

“I signed the principles to show my support for thinking more carefully about the technology we are developing and its role in and impact on society.

“Some principles seemed great on the surface, but I worry about turning them into policies. I think policies would have to be much more nuanced. Much like it is difficult to write down a utility function for a robot, it is difficult to write down a law: we can’t imagine all the different scenarios – especially for a future world – and be certain that the law would be true to our intent.”

Q. Why do you think that AI researchers should weigh in on such issues as opposed to simply doing technical work?

“I think it’s very important for AI researchers to weigh in on AI safety issues. Otherwise, we are putting the conversation in the hands of people who are well-intentioned, but unfamiliar with how the technology actually works. The scenarios portrayed in science fiction are far from the real risks, but that doesn’t mean that real risks don’t exist. We should all work together to identify and mediate them.”

Q. Explain what you think of the following principles:

18) AI Arms Race: An arms race in lethal autonomous weapons should be avoided.
“I fully agree. It’s scary to think of an AI arms race happening. It’d be the equivalent of very cheap and easily accessible nuclear weapons, and that would not fare well for us. My main concern is what to do about it and how to avoid it. I am not qualified to make such judgements, but I assume international treaties would have to occur here.”

20) Importance: Advanced AI could represent a profound change in the history of life on Earth, and should be planned for and managed with commensurate care and resources.
“Ultimately, we work on AI because we believe it can have a strong positive impact on the world. But the more capable the technology becomes, the easier it becomes to misuse it – or perhaps, the effects of misusing it become more drastic. That is why it is so important, as we make progress, to start thinking more strongly about what role AI will play.

“As the AI capabilities advance, we have to take a step back and ask ourselves: are we solving the right problem? Is there a better problem definition that will more likely result in benefits to humanity?

“For instance, we have always defined AI agents as rational. That means they maximize expected utility. Thus far, utility is assumed to be known. But if you think about it, there is no gospel specifying utility. We are assuming that some *person* somewhere will know exactly what utility to specify for their agent. Well, it turns out, we don’t work like that: it is really hard for people, including AI experts, to specify utility functions. We try our best, but when the system goes ahead and optimizes for what we inputted, the result is sometimes surprising, and not in a good way. This suggests that our definition of an AI agent is predicated on a wrong assumption. We’ve already started seeing that in robotics – the definition of how a robot should move didn’t account for people, the definition of how a robot should learn from demonstration assumed that people can provide perfect demonstrations to a robot, etc. – I assume we are going to see this more and more in AI as a whole. We have to stop making implicit assumptions about people and end-users of AI, and rigorously tackle that head-on, putting people into the equation.”

21) Risks: Risks posed by AI systems, especially catastrophic or existential risks, must be subject to planning and mitigation efforts commensurate with their expected impact.
“An immediate risk is agents producing unwanted, surprising behavior. Even if we plan to use AI for good, things can go wrong, precisely because we are bad at specifying objectives and constraints for AI agents. Their solutions are often not what we had in mind.”

10) Value Alignment: Highly autonomous AI systems should be designed so that their goals and behaviors can be assured to align with human values throughout their operation.
“This is one step toward helping AI figure out what it should do, and continuously refining the goals should be an ongoing process between humans and AI.

“At Berkeley, we think the key to value alignment is agents not taking their objectives or utility functions for granted: we already know that people are bad at specifying them. Instead, we think that the agent should collaboratively work with people to understand and verify its utility function. We think it’s important for agents to have uncertainty about their objectives, rather than assuming they are perfectly specified, and treat human input as valuable observations about the true, underlying, desired objective.”

6) Safety: AI systems should be safe and secure throughout their operational lifetime, and verifiably so where applicable and feasible.

“We all agree that we want our systems to be safe. More interesting is what do we mean by “safe”, and what are acceptable ways of verifying safety.

“Traditional methods for formal verification that prove (under certain assumptions) that a system will satisfy desired constraints seem difficult to scale to more complex and even learned behavior. Moreover, as AI advances, it becomes less clear what these constraints should be, and it becomes easier to forget important constraints. We might write down that we want the robot to always obey us when we tell it to turn off, but a sufficiently intelligent robot might find ways to even prevent us from telling it so. This sounds far off and abstract, but even right now it every so often happens that we write down one constraint, when what we really have in mind is something more general. All of these issues make it difficult to prove safety. And while that’s true, does it mean that we just give up and go home? Probably not — it probably means that we need to rethink what we mean by safe, perhaps building in safety from the get-go as opposed to designing a capable system and adding safety after. It opens up difficult but important areas of research that we’re very excited about here at the Center for Human-Compatible AI.”

Q. Are there other principles you want to comment on?

“Value alignment is a big one. Robots aren’t going to try to revolt against humanity, but they’ll just try to optimize whatever we tell them to do. So we need to make sure to tell them to optimize for the world we actually want.”

Shared Prosperity – Paraphrased: We have to make progress on addressing impacts on the economy and developing policies for a world in which more and more of our resource production is going to be automated. We need to address the overall impacts of more and more being automated. It seems good to give people a chance to do what they want, but there’s an income distribution problem: “if all the resources are automated, then who actually controls the automation? Is it everyone or is it a few select people?”

Q. Assuming all goes well, what do you think a world with advanced beneficial AI would look like? What are you striving for with your AI work?

“I envision a future in which AI can eliminate resource scarcity, such as food, healthcare, and even education. A world in which people are not limited by their physical impairments. And ultimately, a world in which machines augment our intelligence and enable us to do things that are difficult to achieve on our own.”

Final message:

“In robotics, and in AI as a whole, we’ve been defining the problem by leaving people out. As we get improvements in function and capability, it is time to think about how AI will be used, and put people back into the equation.”

Join the discussion about the Asilomar AI Principles!