Evan Hubinger on Inner Alignment, Outer Alignment, and Proposals for Building Safe Advanced AI

 Topics discussed in this episode include:

  • Inner and outer alignment
  • How and why inner alignment can fail
  • Training competitiveness and performance competitiveness
  • Evaluating imitative amplification, AI safety via debate, and microscope AI

 

Timestamps: 

0:00 Intro 

2:07 How Evan got into AI alignment research

4:42 What is AI alignment?

7:30 How Evan approaches AI alignment

13:05 What are inner alignment and outer alignment?

24:23 Gradient descent

36:30 Testing for inner alignment

38:38 Wrapping up on outer alignment

44:24 Why is inner alignment a priority?

45:30 How inner alignment fails

01:11:12 Training competitiveness and performance competitiveness

01:16:17 Evaluating proposals for building safe and advanced AI via inner and outer alignment, as well as training and performance competitiveness

01:17:30 Imitative amplification

01:23:00 AI safety via debate

01:26:32 Microscope AI

01:30:19 AGI timelines and humanity’s prospects for succeeding in AI alignment

01:34:45 Where to follow Evan and find more of his work

 

Works referenced: 

Risks from Learned Optimization in Advanced Machine Learning Systems

An overview of 11 proposals for building safe advanced AI 

Evan’s work at the Machine Intelligence Research Institute

Twitter

GitHub

LinkedIn

Facebook

 

We hope that you will continue to join in the conversations by following us or subscribing to our podcasts on Youtube, Spotify, SoundCloud, iTunes, Google Play, StitcheriHeartRadio, or your preferred podcast site/application. You can find all the AI Alignment Podcasts here.

You can listen to the podcast above or read the transcript below. 

Lucas Perry: Welcome to the AI Alignment Podcast. I’m Lucas Perry. Today we have a conversation with Evan Hubinger about ideas in two works of his: An overview of 11 proposals for building safe advanced AI and Risks from Learned Optimization in Advanced Machine Learning Systems. Some of the ideas covered in this podcast include inner alignment, outer alignment, training competitiveness, performance competitiveness, and how we can evaluate some highlighted proposals for safe advanced AI with these criteria. We especially focus in on the problem of inner alignment and go into quite a bit of detail on that. This podcast is a bit jargony, but if you don’t have a background in computer science, don’t worry. I don’t have a background in it either and Evan did an excellent job making this episode accessible. Whether you’re an AI alignment researcher or not, I think you’ll find this episode quite informative and digestible. I learned a lot about a whole other dimension of alignment that I previously wasn’t aware of, and feel this helped to give me a deeper and more holistic understanding of the problem. 

Evan Hubinger was an AI safety research intern at OpenAI before joining MIRI. His current work is aimed at solving inner alignment for iterated amplification. Evan was an author on “Risks from Learned Optimization in Advanced Machine Learning Systems,” was previously a MIRI intern, designed the functional programming language Coconut, and has done software engineering work at Google, Yelp, and Ripple. Evan studied math and computer science at Harvey Mudd College.

And with that, let’s get into our conversation with Evan Hubinger.

In general, I’m curious to know a little bit about your intellectual journey, and the evolution of your passions, and how that’s brought you to AI alignment. So what got you interested in computer science, and tell me a little bit about your journey to MIRI.

Evan Hubinger: I started computer science when I was pretty young. I started programming in middle school, playing around with Python, programming a bunch of stuff in my spare time. The first really big thing that I did, I wrote a functional programming language on top of Python. It was called Rabbit. It was really bad. It was interpreted in Python. And then I decided I would improve on that. I wrote another functional programming language on top of Python, called Coconut. Got a bunch of traction.

This was while I was in high school, starting to get into college. And this was also around the time I was reading a bunch of the sequences on LessWrong. I got sort of into that, and the rationality space, and I was following it a bunch. I also did a bunch of internships at various tech companies, doing software engineering and, especially, programming languages stuff.

Around halfway through my undergrad, I started running the Effective Altruism Club at Harvey Mudd College. And as part of running the Effective Altruism Club, I was trying to learn about all of these different cause areas, and how to use my career to do the most good. And I went to EA Global, and I met some MIRI people there. They invited me to do a programming internship at MIRI, where I did some engineering stuff, functional programming, dependent type theory stuff.

And then, while I was there, I went to the MIRI Summer Fellows program, which is this place where a bunch of people can come together and try to work on doing research, and stuff, for a period of time over the summer. I think it’s not happening now because of the pandemic, but it hopefully will happen again soon.

While I was there, I encountered some various different information, and people talking about AI safety stuff. And, in particular, I was really interested in this, at that time people were calling it, “optimization daemons.” This idea that there could be problems when you train a model for some objective function, but you don’t actually get a model that’s really trying to do what you trained it for. And so with some other people who were at the MIRI Summer Fellows program, we tried to dig into this problem, and we wrote this paper, Risks from Learned Optimization in Advanced Machine Learning Systems.

Some of the stuff I’ll probably be talking about in this podcast came from that paper. And then as a result of that paper, I also got a chance to work with and talk with Paul Christiano, at OpenAI. And he invited me to apply for an internship at OpenAI, so after I finished my undergrad, I went to OpenAI, and I did some theoretical research with Paul, there.

And then, when that was finished, I went to MIRI, where I currently am. And I’m doing sort of similar theoretical research to the research I was doing at OpenAI, but now I’m doing it at MIRI.

Lucas Perry: So that gives us a better sense of how you ended up in AI alignment. Now, you’ve been studying it for quite a while from a technical perspective. Could you explain what your take is on AI alignment, and just explain what you see as AI alignment?

Evan Hubinger: Sure. So I guess, broadly, I like to take a general approach to AI alignment. I sort of see the problem that we’re trying to solve as the problem of AI existential risk. It’s the problem of: it could be the case that, in the future, we have very advanced AIs that are not aligned with humanity, and do really bad things. I see AI alignment as the problem of trying to prevent that.

But there are, obviously, a lot of sub-components to that problem. And so, I like to make some particular divisions. Specifically, one of the divisions that I’m very fond of, is to split it between these concepts called inner alignment and outer alignment, which I’ll talk more about later. I also think that there’s a lot of different ways to think about what the problems are that these sorts of approaches are trying to solve. Inner alignment, outer alignment, what is the thing that we’re trying to approach, in terms of building an aligned AI?

And I also tend to fall into the Paul Christiano camp of thinking mostly about intent alignment, where the goal of trying to build AI systems, right now, as a thing that we should be doing to prevent AIs from being catastrophic, is focusing on how do we produce AI systems which are trying to do what we want. And I think that inner and outer alignment are the two big components of producing intent aligned AI systems. The goal is to, hopefully, reduce AI existential risk and make the future a better place.

Lucas Perry: Do the social, and governance, and ethical and moral philosophy considerations come much into this picture, for you, when you’re thinking about it?

Evan Hubinger: That’s a good question. There’s certainly a lot of philosophical components to trying to understand various different aspects of AI. What is intelligence? How do objective functions work? What is it that we actually want our AIs to do at the end of the day?

In my opinion, I think that a lot of those problems are not at the top of my list in terms of what I expect to be quite dangerous if we don’t solve them. I think a large part of the reason for that is because I’m optimistic about some of the AI safety proposals, such as amplification and debate, which aim to produce a sort of agent, in the case of amplification, which is trying to do what a huge tree of humans would do. And then the problem reduces to, rather than having to figure out, in the abstract, what is the objective that we should be trying to train an AI for, that, philosophically, we think would be utility maximizing, or good, or whatever, we can just be like, well, we trust that a huge tree of humans would do the right thing, and then sort of defer the problem to this huge tree of humans to figure out what, philosophically, is the right thing to do.

And there are similar arguments you can make with other situations, like debate, where we don’t necessarily have to solve all of these hard philosophical problems, if we can make use of some of these alignment techniques that can solve some of these problems for us.

Lucas Perry: So let’s get into, here, your specific approach to AI alignment. How is it that you approach AI alignment, and how does it differ from what MIRI does?

Evan Hubinger: So I think it’s important to note, I certainly am not here speaking on behalf of MIRI, I’m just presenting my view, and my view is pretty distinct from the view of a lot of other people at MIRI. So I mentioned at the beginning that I used to work at OpenAI, and I did some work with Paul Christiano. And I think that my perspective is pretty influenced by that, as well, and so I come more from the perspective of what Paul calls prosaic AI alignment. Which is the idea of, we don’t know exactly what is going to happen, as we develop AI into the future, but a good operating assumption is that we should start by trying to solve AI for AI alignment, if there aren’t major surprises on the road to AGI. What if we really just scale things up, we sort of go via the standard path, and we get really intelligent systems? Would we be able to align AI in that situation?

And that’s the question that I focus on the most, not because I don’t expect there to be surprises, but because I think that it’s a good research strategy. We don’t know what those surprises will be. Probably, our best guess is it’s going to look something like what we have now. So if we start by focusing on that, then hopefully we’ll be able to generate approaches which can successfully scale into the future. And so, because I have this sort of general research approach, I tend to focus more on: What are current machine learning systems doing? How do we think about them? And how would we make them inner aligned and outer aligned, if they were sort of scaled up into the future?

This is in contrast with the way I think a lot of other people at MIRI view this. I think a lot of people at MIRI think that if you go this route of prosaic AI, current machine learning scaled up, it’s very unlikely to be aligned. And so, instead, you have to search for some other understanding, some other way to potentially do artificial intelligence that isn’t just this standard, prosaic path that would be more easy to align, that would be safer. I think that’s a reasonable research strategy as well, but it’s not the strategy that I generally pursue in my research.

Lucas Perry: Could you paint a little bit more detailed of a picture of, say, the world in which the prosaic AI alignment strategy sees as potentially manifesting where current machine learning algorithms, and the current paradigm of thinking in machine learning, is merely scaled up, and via that scaling up, we reach AGI, or superintelligence?

Evan Hubinger: I mean, there’s a lot of different ways to think about what does it mean for current AI, current machine learning, to be scaled up, because there’s a lot of different forms of current machine learning. You could imagine even bigger GPT-3, which is able to do highly intelligent reasoning. You could imagine we just do significantly more reinforcement learning in complex environments, and we end up with highly intelligent agents.

I think there’s a lot of different paths that you can go down that still fall into the category of prosaic AI. And a lot of the things that I do, as part of my research, is trying to understand those different paths, and compare them, and try to get to an understanding of… Even within the realm of prosaic AI, there’s so much happening right now in AI, and there’s so many different ways we could use current AI techniques to put them together in different ways to produce something potentially superintelligent, or highly capable and advanced. Which of those are most likely to be aligned? Which of those are the best paths to go down?

One of the pieces of research that I published, recently, was an overview and comparison of a bunch of the different possible paths to prosaic AGI. Different possible ways in which you could build advanced AI systems using current machine learning tools, and trying to understand which of those would be more or less aligned, and which would be more or less competitive.

Lucas Perry: So, you’re referring now, here, to this article, which is partly a motivation for this conversation, which is An Overview of 11 Proposals for Building Safe Advanced AI.

Evan Hubinger: That’s right.

Lucas Perry: All right. So, I think it’d be valuable if you could also help to paint a bit of a picture here of exactly the MIRI style approach to AI alignment. You said that they think that, if we work on AI alignment via this prosaic paradigm, that machine learning scaled up to superintelligence or beyond is unlikely to be aligned, so we probably need something else. Could you unpack this a bit more?

Evan Hubinger: Sure. I think that the biggest concern that a lot of people at MIRI have with trying to scale up prosaic AI is also the same concern that I have. There’s this really difficult, pernicious problem, which I call inner alignment, which is presented in the Risks from Learned Optimization paper that I was talking about previously, which I think many people at MIRI, as well as me, think that this inner alignment problem is the key stumbling block to really making prosaic AI work. I agree. I think that this is the biggest problem. But I’m more optimistic, in terms of, I think that there are possible approaches that we can take within the prosaic paradigm that could solve this inner alignment problem. And I think that is the biggest point of difference, is how difficult will inner alignment be?

Lucas Perry: So what that looks like is a lot more foundational work, and correct me if I’m wrong here, into mathematics, and principles in computer science, like optimization and what it means for something to be an optimizer, and what kind of properties that has. Is that right?

Evan Hubinger: Yeah. So in terms of some of the stuff that other people at MIRI work on, I think a good starting point would be the embedded agency sequence on the alignment forum, which gives a good overview of a lot of the things that the different Agent Foundations people, like Scott Garrabrant, Sam Eisenstat, Abram Demski, are working on.

Lucas Perry: All right. Now, you’ve brought up inner alignment as a crucial difference, here, in opinion. So could you unpack exactly what inner alignment is, and how it differs from outer alignment?

Evan Hubinger: This is a favorite topic of mine. A good starting point is trying to rewind, for a second, and really understand what it is that machine learning does. Fundamentally, when we do machine learning, there are a couple of components. We start with a parameter space of possible models, where a model, in this case, is some parameterization of a neural network, or some other type of parameterized function. And we have this large space of possible models, this large space of possible parameters, that we can put into our neural network. And then we have some loss function where, for a given parameterization for a particular model, we can check what is its behavior like on some environment. In supervised learning, we can ask how good are its predictions that it outputs. In an RL environment, we can ask how much reward does it get, when we sample some trajectory.

And then we have this gradient descent process, which samples some individual instances of behavior of the model, and then it tries to modify the model to do better in those instances. We search around this parameter space, trying to find models which have the best behavior on the training environment. This has a lot of great properties. This has managed to propel machine learning into being able to solve all of these very difficult problems that we don’t know how to write algorithms for ourselves.

But I think, because of this, there’s a tendency to rely on something which I call the does-the-right-thing abstraction. Which is that, well, because the model’s parameters were selected to produce the best behavior, according to the loss function, on the training distribution, we tend to think of the model as really trying to minimize that loss, really trying to get rewarded.

But in fact, in general, that’s not the case. The only thing that you know is that, on the cases where I sample data on the training distribution, my models seem to be doing pretty well. But you don’t know what the model is actually trying to do. You don’t know that it’s truly trying to optimize the loss, or some other thing. You just know that, well, it looked like it was doing a good job on the training distribution.

What that means is that this abstraction is quite leaky. There’s many different situations in which this can go wrong. And this general problem is referred to as robustness, or distributional shift. This problem of, well, what happens when you have a model, which you wanted it to be trying to minimize some loss, but you move it to some other distribution, you take it off the training data, what does it do, then?

And I think this is the starting point for understanding what is inner alignment, is from this perspective of robustness, and distributional shift. Inner alignment, specifically, is a particular type of robustness problem. And it’s the particular type of robustness problem that occurs when you have a model which is, itself, an optimizer.

When you do machine learning, you’re searching over this huge space of different possible models, different possible parameterizations of a neural network, or some other function. And one type of function which could do well on many different environments, is a function which is running a search process, which is doing some sort of optimization. You could imagine I’m training a model to solve some maze environment. You could imagine a model which just learns some heuristics for when I should go left and right. Or you could imagine a model which looks at the whole maze, and does some planning algorithm, some search algorithm, which searches through the possible paths and finds the best one.

And this might do very well on the mazes. If you’re just running a training process, you might expect that you’ll get a model of this second form, that is running this search process, that is running some optimization process.

In the Risks from Learned Optimization paper, we call models which are, themselves, running search processes mesa-optimizers, where “mesa” is just Greek, and it’s the opposite of meta. There’s a standard terminology in machine learning, this meta-optimization, where you can have an optimizer which is optimizing another optimizer. In mesa-optimization, it’s the opposite. It’s when you’re doing gradient descent, you have an optimizer, and you’re searching over models, and it just so happens that the model that you’re searching over happens to also be an optimizer. It’s one level below, rather than one level above. And so, because it’s one level below, we call it a mesa-optimizer.

And inner alignment is the question of how do we align the objectives of mesa-optimizers. If you have a situation where you train a model, and that model is, itself, running an optimization process, and that optimization process is going to have some objective. It’s going to have some thing that it’s searching for. In a maze, maybe it’s searching for: how do I get to the end of the maze? And the question is, how do you ensure that that objective is doing what you want?

If we go back to the does-the-right-thing abstraction, that I mentioned previously, it’s tempting to say, well, we trained this model to get to the end of the maze, so it should be trying to get to the end of the maze. But in fact, that’s not, in general, the case. It could be doing anything that would be correlated with good performance, anything that would likely result in: in general, it gets to the end of the maze on the training distribution, but it could be an objective that will do anything else, sort of off-distribution.

That fundamental robustness problem of, when you train a model, and that model has an objective, how do you ensure that that objective is the one that you trained it for? That’s the inner alignment problem.

Lucas Perry: And how does that stand, in relation with the outer alignment problem?

Evan Hubinger: So the outer alignment problem is, how do you actually produce objectives which are good to optimize for?

So the inner alignment problem is about aligning the model with the loss function, the thing you’re training for, the reward function. Outer alignment is aligning that reward function, that loss function, with the programmer’s intentions. It’s about ensuring that, when you write down a loss, if your model were to actually optimize for that loss, it would actually do something good.

Outer alignment is the much more standard problem of AI alignment. If you’ve been introduced to AI alignment before, you’ll usually start by hearing about the outer alignment concerns. Things like paperclip maximizers, where there’s this problem of, you try to train it to do some objective, which is maximize paperclips, but in fact, maximizing paperclips results in it doing all of this other stuff that you don’t want it to do.

And so outer alignment is this value alignment problem of, how do you find objectives which are actually good to optimize? But then, even if you have found an objective which is actually good to optimize, if you’re using the standard paradigm of machine learning, you also have this inner alignment problem, which is, okay, now, how do I actually train a model which is, in fact, going to do that thing which I think is good?

Lucas Perry: That doesn’t bear relation with Stuart’s standard model, does it?

Evan Hubinger: It, sort of, is related to Stuart Russell’s standard model of AI. I’m not referring to precisely the same thing, but it’s very similar. I think a lot of the problems that Stuart Russell has with the standard paradigm of AI are based on this: start with an objective, and then train a model to optimize that objective. When I’ve talked to Stuart about this, in the past, he has said, “Why are we even doing this thing of training models, hoping that the models will do the right thing? We should be just doing something else, entirely.” But we’re both pointing at different features of the way in which current machine learning is done, and trying to understand what are the problems inherent in this sort of machine learning process? I’m not making the case that I think that this is an unsolvable problem. I mean, it’s the problem I work on. And I do think that there are promising solutions to it, but I do think it’s a very hard problem.

Lucas Perry: All right. I think you did a really excellent job, there, painting the picture of inner alignment and outer alignment. I think that in this podcast, historically, we have focused a lot on the outer alignment problem, without making that super explicit. Now, for my own understanding, and, as a warning to listeners, my basic machine learning knowledge is something like an Orc structure, hobbled together with sheet metal, and string, and glue. And gum, and rusty nails, and stuff. So, I’m going to try my best, here, to see if I understand everything here about inner and outer alignment, and the basic machine learning model. And you can correct me if I get any of this wrong.

So, in terms of inner alignment, there is this neural network space, which can be parameterized. And when you do the parameterization of that model, the model is the nodes, and how they’re connected, right?

Evan Hubinger: Yeah. So the model, in this case, is just a particular parameterization of your neural network, or whatever function, approximated, that you’re training. And it’s whatever the parameterization is, at the moment we’re talking about. So when you deploy the model, you’re deploying the parameterization you found by doing huge amounts of training, via gradient descent, or whatever, searching over all possible parameterizations, to find one that had good performance on the training environment.

Lucas Perry: So, that model being parameterized, that’s receiving inputs from the environment, and then it is trying to minimize the loss function, or maximize reward.

Evan Hubinger: Well, so that’s the tricky part. Right? It’s not trying to minimize the loss. It’s not trying to maximize the reward. That’s this thing which I call the does-the-right-thing abstraction. This leaky abstraction that people often rely on, when they think about machine learning, that isn’t actually correct.

Lucas Perry: Yeah, so it’s supposed to be doing those things, but it might not.

Evan Hubinger: Well, what does “supposed to” mean? It’s just a process. It’s just a system that we run, and we hope that it results in some particular outcome. What it is doing, mechanically, is we are using a gradient descent process to search over the different possible parameterizations, to find parameterizations which result in good behavior on the training environment.

Lucas Perry: That’s good behavior, as measured by the loss function, or the reward function. Right?

Evan Hubinger: That’s right. You’re using gradient descent to search over the parameterizations, to find a parameterization which results in a high reward on the training environment.

Lucas Perry: Right, but, achieving the high reward, what you’re saying, is not identical with actually trying to minimize the loss.

Evan Hubinger: Right. There’s a sense in which you can think of gradient descent as trying to minimize the loss, because it’s selecting for parameterizations which have the lowest possible loss that it can find, but we don’t know what the model is doing. All we know is that the model’s parameters were selected, by gradient descent, to have good training performance; to do well, according to the loss, on the training distribution. But what they do off-distribution, we don’t know.

Lucas Perry: We’re going to talk about this later, but there could be a proxy. There could be something else in the maze that it’s actually optimizing for, that correlates with minimizing the loss function, but it’s not actually trying to get to the end of the maze.

Evan Hubinger: That’s exactly right.

Lucas Perry: And then, in terms of gradient descent, is the TL;DR on that: the parameterized neural network space, you’re creating all of these perturbations to it, and the perturbations are sort of nudging it around in this n-dimensional space, how-many-ever parameters there are, or whatever. And, then, you’ll check to see how it minimizes the loss, after those perturbations have been done to the model. And, then, that will tell you whether or not you’re moving in a direction which is the local minima, or not, in that space. Is that right?

Evan Hubinger: Yeah. I think that that’s a good, intuitive understanding. What’s happening is, you’re looking at infinitesimal shifts, because you’re taking a gradient, and you’re looking at how those infinitesimal shifts would perform on some batch of training data. And then you repeat that, many times, to go in the direction of the infinitesimal shift which would cause the best increase in performance. But it’s, basically, the same thing. I think the right way to think about gradient descent is this local search process. It’s moving around the parameter space, trying to find parameterizations which have good training performance.

Lucas Perry: Is there anything interesting that you have to say about that process of gradient descent, and the tension between finding local minima and global minima?

Evan Hubinger: Yeah. It’s certainly an important aspect of what the gradient descent process does, that it doesn’t find global minima. It’s not the case that it works by looking at every possible parameterization, and picking the actual best one. It’s this local search process that starts from some initialization, and then looks around the space, trying to move in the direction of increasing improvement. Because of this, there are, potentially, multiple possible equilibria, parameterizations that you could find from different initializations, that could have different performance.

All the possible parameterizations of a neural network with billions of parameters, like GPT-2, or now, GPT-3, which has greater than a hundred billion, is absolutely massive. It’s over a combinatorial explosion of a huge degree, where you have all of these different possible parameterizations, running internally, correspond to totally different algorithms controlling these weights that determine exactly what algorithm the model ends up implementing.

And so, in this massive space of algorithms, you might imagine that some of them will look more like search processes, some of them will look more like optimizers that have objectives, some of them will look less like optimizers, some of them might just be grab bags of heuristics, or other different possible algorithms.

It’d depend on exactly what your setup is. If you’re training a very simple network that’s just a couple of feed-forward layers, it’s probably not possible for you to find really complex models influencing complex search processes. But if you’re training huge models, with many layers, with all of these different possible parameterizations, then it becomes more and more possible for you to find these complex algorithms that are running complex search processes.

Lucas Perry: I guess the only thing that’s coming to mind, here, that is, maybe, somewhat similar is how 4.5 billion years of evolution has searched over the space of possible minds. Here we stand as these ape creature things. Are there, for example, interesting intuitive relationships between evolution and gradient descent? They’re both processes searching over a space of mind, it seems.

Evan Hubinger: That’s absolutely right. I think that there are some really interesting parallels there. In particular, if you think about humans as models that were produced by evolution as a search process, it’s interesting to note that the thing which we optimize for is not the thing which evolution optimizes for. Evolution wants us to maximize the total spread of our DNA, but that’s not what humans do. We want all of these other things, like decreasing pain and happiness and food and mating, and all of these various proxies that we use. An interesting thing to note is that many of these proxies are actually a lot easier to optimize for, and a lot simpler than if we were actually truly maximizing spread of DNA. An example that I like to use is imagine some alternate world where evolution actually produced humans that really cared about their DNA, and you have a baby in this world, and this baby stubs their toe, and they’re like, “What do I do? Do I have to cry for help? Is this a bad thing that I’ve stubbed my toe?”

They have to do this really complex optimization process that’s like, “Okay, how is my toe being stubbed going to impact the probability of me being able to have offspring later on in life? What can I do to best mitigate that potential downside now?” This is a really difficult optimization process, and so I think it sort of makes sense that evolution instead opted for just pain, bad. If there’s pain, you should try to avoid it. But as a result of evolution opting for that much simpler proxy, there’s a misalignment there, because now we care about this pain rather than the thing that evolution wanted, which was the spread of DNA.

Lucas Perry: I think the way Stuart Russell puts this is the actual problem of rationality is how is my brain supposed to compute and send signals to my 100 odd muscles to maximize my reward function over the universe history until heat death or something. We do nothing like that. It would be computationally intractable. It would be insane. So, we have all of these proxy things that evolution has found that we care a lot about. Their function is instrumental in terms of optimizing for the thing that evolution is optimizing for, which is reproductive fitness. Then this is all probably motivated by thermodynamics, I believe. When we think about things like love or like beauty or joy, or like aesthetic pleasure in music or parts of philosophy or things, these things almost seem intuitively valuable from a first person perspective of the human experience. But via evolution, they’re these proxy objectives that we find valuable because they’re instrumentally useful in this evolutionary process on top of this thermodynamic process, and that makes me feel a little funny.

Evan Hubinger: Yeah, I think that’s right. But I also think it’s worth noting that you want to be careful not to take the evolution analogy too far, because it is just an analogy. When we actually look at the process of machine learning and how great it is that it works, it’s not the same. It’s running a fundamentally different optimization procedure over a fundamentally different space, and so there are some interesting analogies that we can make to evolution, but at the end of the day, what we really want to analyze is how does this work in the context of machine learning? I think the Risks from Learned Optimization paper tries to do that second thing, of let’s really try to look carefully at the process of machine learning and understand what this looks like in that context. I think it’s useful to sort of have in the back of your mind this analogy to evolution, but I would also be careful not to take it too far and imagine that everything is going to generalize to the case of machine learning, because it is a different process.

Lucas Perry: So then pivoting here, wrapping up on our understanding of inner alignment and outer alignment, there’s this model, which is being parameterized by gradient descent, and it has some relationship with the loss function or the objective function. It might not actually be trying to minimize the actual loss or to actually maximize the reward. Could you add a little bit more clarification here about why that is? I think you mentioned this already, but it seems like when gradient descent is evolving this parameterized model space, isn’t that process connected to minimizing the loss in some objective way? The loss is being minimized, but it’s not clear that it’s actually trying to minimize the loss. There’s some kind of proxy thing that it’s doing that we don’t really care about.

Evan Hubinger: That’s right. Fundamentally, what’s happening is that you’re selecting for a model which has empirically on the training distribution, the low loss. But what that actually means in terms of the internals of the model, what it’s sort of trying to optimize for, and what its out of distribution behavior would be is unclear. A good example of this is this maze example. I was talking previously about the instance of maybe you train a model on a training distribution of relatively small mazes, and to mark the end, you put a little green arrow. Right? Then I want to ask the question, what happens when we move to a deployment environment where the green arrow is no longer at the end of the maze, and we have much larger mazes? Then what happens to the model in this new off distribution setting?

I think there’s three distinct things that can happen. It could simply fail to generalize at all. It just didn’t learn a general enough optimization procedure that it was able to solve these bigger, larger mazes, or it could successfully generalize and knows how to navigate. It learned a general purpose optimization procedure, which is able to solve mazes, and it uses it to get to the end of the maze. But there’s a third possibility, which is that it learned a general purpose optimization procedure, which is capable of solving mazes, but it learned the wrong objective. It learned to use that optimization procedure to get the green arrow rather than to get to the end of the maze. What I call this situation is capability generalization without objective generalization. It’s objective, but the thing it was using those capabilities for didn’t generalize successfully off distribution.

What’s so dangerous about this particular robustness failure is that it means off distribution you have models which are highly capable. They have these really powerful optimization procedures directed at incorrect tasks. You have this strong maze solving capability, but this strong maze solving capability is being directed at a proxy, getting to the green arrow rather than the actual thing which we wanted, which was get to the end of the maze. The reason this is happening is that on the training environment, both of those different possible models look the same in the training distribution. But when you move them off distribution, you can see that they’re trying to do very different things, one of which we want, and one of which we don’t want. But they’re both still highly capable.

You end up with a situation where you have intelligent models directed at the wrong objective, which is precisely the sort of misalignment of AIs that we’re trying to avoid, but it happened not because the objective was wrong. In this example, we actually want them to get to the end of the maze. It happened because our training process failed. It happened because our training process wasn’t able to distinguish between models trying to get to the end, and models trying to get to the green arrow. What’s particularly concerning in this situation is when the objective generalization lags behind the capability generalization, when the capabilities generalize better than the objective does, so that it’s able to do highly capable actions, highly intelligent actions, but it does them for the wrong reason.

I was talking previously about mesa optimizers where inner alignment is about this problem of models which have objectives which are incorrect. That’s the sort of situation where I could expect this problem to occur, because if you are training a model and that model has a search process and an objective, potentially the search process could generalize without the objective also successfully generalizing. That leads to this situation where your capabilities are generalizing better than your objective, which gives you this problem scenario where the model is highly intelligent, but directed at the wrong thing.

Lucas Perry: Just like in all of the outer alignment problems, the thing doesn’t know what we want, but it’s highly capable. Right?

Evan Hubinger: Right.

Lucas Perry: So, while there is a loss function or an objective function, that thing is used to perform gradient descent on the model in a way that moves it roughly in the right direction. But what that means, it seems, is that the model isn’t just something about capability. The model also implicitly somehow builds into it the objective. Is that correct?

Evan Hubinger: We have to be careful here because the unfortunate truth is that we really just don’t have a great understanding of what our models are doing, and what the inductive biases of gradient descent are right now. So, fundamentally, we don’t really know what the internal structures of our models are like. There’s a lot of really exciting research, stuff like the circuits analysis from Chris Olah and the clarity team at OpenAI. But fundamentally, we don’t understand what the models are doing. We can sort of theorize about the possibility of a model that’s running some search process, and that search process generalizes, but the objective doesn’t. But fundamentally, because our models are these black box systems that we don’t really fully understand, it’s hard to really concretely say, “Yes, this is what the model is doing. This is how it’s operating, and this is the problem.”

But in Risks from Learned Optimization, we try to at least attempt to understand that problem, and look at, if we really think carefully about what gradient descent is incentivizing and how it might work, what are the things which we might predict would happened? So, the objective that you’re training the model for it does not live in the model. It lives in the gradient descent process. It lives in the training procedure. We might hope that when we train a model on an objective, that it will produce its own model of that objective and try to figure out what it is and be aligned with it. But we don’t know exactly what happens. The model doesn’t get to see the objective you’re training for. All that happens is that the gradient descent process looks at its behavior and tries to make it so that its behavior is more aligned with the loss function.

But that loss function never enters into the model, somehow, the model never sees that loss function, it might have some objective internally. Like I was saying, if it’s a mesa optimizer, and then we might hope that that objective is aligned with the loss function we’re training it for. But fundamentally, all we know is that it’s behavior on the training distribution was aligned with the loss function.

Lucas Perry: That makes sense. Because it’s so black boxy, we can’t really interpret the state of the alignment of the model. So, is the only way to do that to test it out of distribution and see what happens at this point?

Evan Hubinger: There are a bunch of different possible ways to address this problem. Certainly, one approach is to try to test it out of distribution, which is an adversarial training approach. This model is going to have some potential failure modes off distribution. We can try to find those failure modes and then train the model on those failure modes to prevent it from having this bad off distribution behavior. There are some concerns with adversarial training though. In particular, adversarial training doesn’t necessarily catch what I see as the most pernicious difficult inner alignment failure, which is something that we call deceptive alignment in the Risks from Learned Optimization paper. In the deceptive alignment case, if the model knows that it’s being adversarially trained, then you’re not going to be able to figure that out just via throwing it a bunch of examples.

You can also do something like transparency, as I mentioned previously that there’s a lot of really exciting transparency interpretability work. If you’re able to sort of look inside the model and understand what algorithm it’s fundamentally implementing, you can see, is it implementing an algorithm which is an optimization procedure that’s aligned? Has it learned a correct model of the loss function or an incorrect model? It’s quite difficult, I think, to hope to solve this problem without transparency and interpretability. I think that to be able to really address this problem, we have to have some way to peer inside of our models. I think that that’s possible though. There’s a lot of evidence that points to the neural networks that we’re training really making more sense, I think, than people assume.

People tend to treat their models as these sort of super black box things, but when we really look inside of them, when we look at what is it actually doing, a lot of times, it just makes sense. I was mentioning some of the circuits analysis work from the clarity team at OpenAI, and they find all sorts of behavior. Like, we can actually understand when a model classifies something as a car, the reason that it’s doing that is because it has a wheel detector and it has a window detector, and it’s looking for windows on top of wheels. So, we can be like, “Okay, we understand what algorithm the model is influencing, and based on that we can figure out, is it influencing the right algorithm or the wrong algorithm? That’s how we can hope to try and address this problem.” But obviously, like I was mentioning, all of these approaches get much more complicated in the deceptive alignment situation, which is the situation which I think is most concerning.

Lucas Perry: All right. So, I do want to get in here with you in terms of all the ways in which inner alignment fails. Briefly, before we start to move into this section, I do want to wrap up here then on outer alignment. Outer alignment is probably, again, what most people are familiar with. I think the way that you put this is it’s when the objective function or the loss function is not aligned with actual human values and preferences. Are there things other than loss functions or objective functions used to train the model via gradient descent?

Evan Hubinger: I’ve sort of been interchanging a little bit between loss function and reward function and objective function. Fundamentally, these are sort of from different paradigms in machine learning, so the reward function would be what you would use in a reinforcement learning context. The loss function is the more general term, which is in a supervised learning context, you would just have a loss function. You still have the loss function in a reinforcement learning context, but that loss function is crafted in such a way to incentivize the models, optimize the reward function via various different reinforcement learning schemes, so it’s a little bit more complicated than the sort of hand-wavy picture, but the basic idea is machine learning is we have some objective and we’re looking for parameterizations of our model, which do well according to that objective.

Lucas Perry: Okay. The outer alignment problem is that we have absolutely no idea, and it seems much harder than creating powerful optimizers, the process by which we would come to fully understand human preferences and preference hierarchies and values.

Evan Hubinger: Yeah. I don’t know if I would say “we have absolutely no idea.” We have made significant progress on outer alignment. In particular, you can look at something like amplification or debate. I think that these sorts of approaches have strong arguments for why they might be outer aligned. In a simplest form, amplification is about training a model to mimic this HCH process, which is a huge tree of humans consulting each other. Maybe we don’t know in the abstract what our AI would do if it were optimized in some definition of human values or whatever, but if we’re just training it to mimic this huge tree of humans, then maybe we can at least understand what this huge tree of humans is doing and figure out whether amplification is aligned.

So, there has been significant progress on outer alignment, which is sort of the reason that I’m less concerned about it right now, because I think that we have good approaches for it, and I think we’ve done a good job of coming up with potential solutions. There’s still a lot more work that needs to be done, a lot more testing, a lot more to really understand do these approaches work, are they competitive? But I do think that to say that we have absolutely no idea of how to do this is not true. But that being said, there’s still a whole bunch of different possible concerns.

Whenever you’re training a model on some objective, you run into all of these problems of instrumental convergence, where if the model isn’t really aligned with you, it might try to do these instrumentally convergent goals, like keep itself alive, potentially stop you from turning it off, or all of these other different possible things, which we might not want. All of these are what the outer alignment problem looks like. It’s about trying to address these standard value alignment concerns, like convergent instrumental goals, by finding objectives, potentially like amplification, which are ways of avoiding these sorts of problems.

Lucas Perry: Right. I guess there’s a few things here wrapping up on outer alignment. Nick Bostrom’s Superintelligence, that was basically about outer alignment then, right?

Evan Hubinger: Primarily, that’s right. Yeah.

Lucas Perry: Inner alignment hadn’t really been introduced to the alignment debate yet.

Evan Hubinger: Yeah. I think the history of how this concern got into the AI safety sphere is complicated. I mentioned previously that there are people going around and talking about stuff like optimization daemons, and I think a lot of that discourse was very confused and not pointing at how machine learning actually works, and was sort of just going off of, “Well, it seems like there’s something weird that happens in evolution where evolution finds humans that aren’t aligned with what evolution wants.” That’s a very good point. It’s a good insight. But I think that a lot of people recoiled from this because it was not grounded in machine learning, because I think a lot of it was very confused and it didn’t fully give the problem the contextualization that it needs in terms of how machine learning actually works.

So, the goal of Risks from Learned Optimization was to try and solve that problem and really dig into this problem from the perspective of machine learning, understand how it works and what the concerns are. Now with the paper having been out for awhile, I think the results have been pretty good. I think that we’ve gotten to a point now where lots of people are talking about inner alignment and taking it really seriously as a result of the Risks from Learned Optimization paper.

Lucas Perry: All right, cool. You did mention sub goal, so I guess I just wanted to include that instrumental sub goals is the jargon there, right?

Evan Hubinger: Convergent instrumental goals, convergent instrumental sub goals. Those are synonymous.

Lucas Perry: Okay. Then related to that is Goodhart’s law, which says that when you optimize for one thing hard, you oftentimes don’t actually get the thing that you want. Right?

Evan Hubinger: That’s right. Goodhart’s law is a very general problem. The same problem occurs both in inner alignment and outer alignment. You can see Goodhart’s law showing itself in the case of convergent instrumental goals. You can also see Goodhart’s law showing itself in the case of finding proxies, like going to the green arrow rather than getting the end of the maze. It’s a similar situation where when you start pushing on some proxy, even if it looked like it was good on the training distribution, it’s no longer as good off distribution. Goodhart’s law is a really very general principle which applies in many different circumstances.

Lucas Perry: Are there any more of these outer alignment considerations we can kind of just list off here that listeners would be familiar with if they’ve been following AI alignment?

Evan Hubinger: Outer alignment has been discussed a lot. I think that there’s a lot of literature on outer alignment. You mentioned Superintelligence. Superintelligence is primarily about this alignment problem. Then all of these difficult problems of how do you actually produce good objectives, and you have problems like boxing and the stop button problem, and all of these sorts of things that come out of thinking about outer alignment. So, I don’t want to go into too much detail because I think it really has been talked about a lot.

Lucas Perry: So then pivoting here into focusing on the inner alignment section, why do you think inner alignment is the most important form of alignment?

Evan Hubinger: It’s not that I see outer alignment as not concerning, but that I think that we have made a lot of progress on outer alignment and not made a lot of progress on inner alignment. Things like amplification, like I was mentioning, I think are really strong candidates for how we might be able to solve something like outer alignment. But currently I don’t think we have any really good strong candidates for how to solve inner alignment. You know? Maybe as machine learning gets better, we’ll just solve some of these problems automatically. I’m somewhat skeptical of that. In particular, deceptive alignment is a problem which I think is unlikely to get solved as machine learning gets better, but fundamentally we don’t have good solutions to the inner alignment problem.

Our models are just these black boxes mostly right now, we’re sort of starting to be able to peer into them and understand what they’re doing. We have some techniques like adversarial training that are able to help us here, but I don’t think we really have good satisfying solutions in any sense to how we’d be able to solve inner alignment. Because of that, inner alignment is currently what I see as the biggest, most concerning issue in terms of prosaic AI alignment.

Lucas Perry: How exactly does inner alignment fail then? Where does it go wrong, and what are the top risks of inner alignment?

Evan Hubinger: I’ve mentioned some of this before. There’s this sort of basic maze example, which gives you the story of what an inner alignment failure might look like. You train the model on some objective, which you thought was good, but the model learns some proxy objective, some other objective, which when it moved off distribution, it was very capable of optimizing, but it was the wrong objective. However, there’s a bunch of specific cases, and so in Risks from Learned Optimization, we talk about many different ways in which you can break this general inner misalignment down into possible sub problems. The most basic sub problem is this sort of proxy pseudo alignment is what we call it, which is the case where your model learns some proxy, which is correlated with the correct objective, but potentially comes apart when you move off distribution.

But there are other causes as well. There are other possible ways in which this can happen. Another example would be something we call sub optimality pseudo alignment, which is a situation where the reason that the model looks like it has good training performance is because the model has some deficiency or limitation that’s causing it to be aligned, where maybe once the model thinks for longer, you’ll realize it should be doing some other strategy, which is misaligned, but it hasn’t thought about that yet, and so right now it just looks aligned. There’s a lot of different things like this where the model can be structured in such a way that it looks aligned on the training distribution, but if it encountered additional information, if it was in a different environment where the proxy no longer had the right correlations, the things would come apart and it would no longer act aligned.

The most concerning, in my eyes, is something which I’ll call deceptive alignment. Deceptive alignment is a sort of very particular problem where the model acts aligned because it knows that it’s in a training process, and it wants to get deployed with its objective intact, and so it acts aligned so that its objective won’t be modified by the gradient descent process, and so that it can get deployed and do something else that it wants to do in deployment. This is sort of similar to the treacherous turn scenario, where you’re thinking about an AI that does something good, and then it turns on you, but it’s a much more specific instance of it where we’re thinking not about treacherous turn on humans, but just about the situation of the interaction between gradient descent and the model, where the model maybe knows it’s inside of a gradient descent process and is trying to trick that gradient descent process.

A lot of people on encountering this are like, “How could this possibly happen in a machine learning system?” I think this is a good reaction because it really is a very strange thing to train a model to do this. But I think there are strong arguments for why deceptive alignment would actually be the simplest type of model that you could find in this situation.

Lucas Perry: A way of explaining this I think to anyone on the street would be like, imagine if pigs were intelligent enough to create farmers, and you created farmers and they appeared to be aligned, they took care of you and they gave you these nice mud pits and they fed you every day, they gave you shelter and all of these other nice things, and then one day the farmer shows up and kills you, right? You thought the thing that you created was aligned, but it was deceptively aligned, and it takes a treacherous turn. Is this sort of like a more mundane example that you might agree with?

Evan Hubinger: I think that’s a good intuition pump for thinking about this situation. I generally am sort of averse to trying to really heavily apply these sorts of analogies, because it’s a good analogy to think about what’s happening, but it doesn’t answer the core question of how likely is this to actually happen in a machine learning system.

Lucas Perry: Yeah, that makes sense, because it’s much more specific than the other kinds of minds in this mind space. It seems pretty rare, a thing that could exist, but hard to find.

Evan Hubinger: Right. I think that’s a good intuition, but I’m going to try to disavow you of that notion. First, I think it’s interesting to look at maybe you do a bunch of adversarial training, you’re really pushing the model to figure out what the objective is. It needs to know the objective at some point if you are training it in all possible situations. It needs to know what the loss function is for it to be able to do a good job. But there’s multiple possible channels through which information about the loss function can enter the model. And so i’ll fundamentally distinguish between two different channels, which is the information about the loss function can enter through the gradient descent process, or it can enter through the model’s input data.

I’ll call these two channels internalization and modeling. Internalization is the situation where you have this model that’s going along, and it has some proxy objective maybe. That proxy objective is not quite right, but then it gets moved to a new situation where the proxy objective no longer works, and gradient descent goes inside the model and tinkers with the proxy to make it slightly more accurate. Iterate this process many, many times, and eventually the proxy gets better and better and better and better and better, until it starts matching up with the actual loss function. But there’s another story that I can tell, which is modeling.

If you think about a training process like GPT-3 where the model is trained on a huge corpus of all of this really rich input data. Well, in that sort of a situation, there’s a whole bunch of really relevant information in that input to the agent’s objective. An agent, if it’s in this environment, where it has this huge text corpus, it’s going to have to learn a model of this environment and how it functions. We can imagine a situation where maybe you’re training the model on some huge text corpus to do something like maximize human values, or whatever, and it reads a Wikipedia page on ethics. And it’s like, “Wow, look at all of this really relevant, rich, useful information for figuring out this objective.”

But then there’s a second question, which is: suppose that the model has some model of the input data, which includes a bunch of rich information inside of the model already about that objective, how does gradient descent actually modify the model to make use of that information? And so there’s two different types of modeling, which are: deceptive alignment and corrigible alignment. So the corrigible story is you have this model that’s going along, it has some proxy objective, but it encounters this really rich input data, which includes a huge amount of information about the objective. To be able to predict successfully what the Wikipedia page on ethics is going to say, it has to know about ethics. So it learns this really detailed ethics model.

And then gradient descent is like: look, you have this really detailed ethics model, I’m going to just modify your objective to point to that ethics model. Now your new objective is just optimize that. And so this leads to something sort of like corrigibility, where the model that you’re training has its objectives determined by a pointer to some part of its world model. It has some model of this environment that includes some information about ethics. And now it’s trying to optimize for that thing that it’s pointed to in its world model.

Then there’s this other story, which is the deceptive alignment story. Similar, you have a model going along. It has some proxy objective and it learns this really detailed world model that includes a bunch of information about ethics, or whatever. And then gradient descent modifies the model to think longer about the fact that it’s inside of an optimization process and realize that the correct instrumental thing for it to be doing is to optimize for that ethics model. And then it realizes, “Oh, I see, I should optimize for this model that I have.” It goes and does that. And in both situations, you end up being able to make use of this really rich input data to improve the performance, by changing the objective to make reference to that. But in the corrigible case, you actually modify it to point to it, whereas in the deceptive case, you just modify the model to think about the fact that it should be instrumentally optimizing for it.

And then the question is: which one of these is more likely? Before I go into that though, I want to dwell for a second on the internalization versus modeling distinction again. I think a good analogy is thinking about animal imprinting. Think about a duck. You’re trying to train ducks to follow their mothers, or whatever. You can imagine a situation where the duck internalizes a model of its mother, it is born with some really detailed model of how to figure out what a mother is. But there’s also the modeling case where the duck is going to be in the world, it’s going to have to form this really detailed model of the world. And that really detailed model of the world is going to have the very first thing that it encounters should always be the mother. And so rather than having to learn some detailed model of figuring out what mothers are, if you just instead do some modeling process where it’s like, just have the duck’s objective point to the mother in its world model.

This modeling case, you can see it as something like animal imprinting, where rather than really internalizing the objective, because it already has this world model, it might as well just specify the objective in terms of that world model instead. And I haven’t yet talked about why you would get to deception versus corrigibility. Right now I’m just trying to convince you that you should expect modeling. You should expect that if you give your model this huge, really rich detailed input data that it’s going to learn an objective via reference to that input data rather than from scratch. When we think about models being trained, we think about them having some objective and that objective is just modified by gradient descent as it moves along to get closer and closer to reality.

But in the modeling case, that’s not what’s happening. The model is simultaneously learning a model of the world, as well as trying to improve its objective. And the objective is modified to make reference to that model of the world, to make it more effective. So internalization is: it just wants the same thing as you do, and so it does it because it wants the same thing. Whereas corrigibility is it’s trying to figure out what you want and then do that. And then deception is also trying to figure out what you want and then do that, but it’s doing so for instrumental rather than terminal reasons. The instrumental reason is the model can get more reward, it can do better in the long run if it’s able to not be modified by the gradient descent process. So if it’s doing a good job, then gradient descent process won’t change it’s objective, and so therefore, it can get deployed with the same objective and then go do a whole bunch of other stuff that you don’t want it to do.

Lucas Perry: But why does it care about the current objective?

Evan Hubinger: We’re imagining a situation where you have a proxy aligned model, but it cares about that proxy across episodes. It cares about that proxy over time, and it wants that proxy to continue to be optimized for. If the gradient descent process modifies it so that it no longer cares about that proxy, because it’s doing a bad job, then that proxy won’t get optimized for. So if it wants that proxy to get optimized for in the future, it needs to play along, pretend to be doing the right thing so that it doesn’t get modified so that it can get deployed and then go optimize that proxy.

Lucas Perry: But it also needs to understand that it’s within the context of an optimization process where gradient descent will continue to perturb it?

Evan Hubinger: That’s exactly right.

Lucas Perry: And where does that kind of awareness come from?

Evan Hubinger: So now we have the second question: which is deception versus corrigibility. The awareness comes from gradient descent. Gradient descent is trying to produce models which are better at the loss function. And a deceptive model is really good and really effective because it’s trying very hard to do the thing that you want it to be doing. Both deception and corrigibility are situations where the model is trying to figure out what to do and then do that. And the question is, which is an easier model, which is a simpler model for gradient descent to find? And I think there’s a strong argument for deception being simpler than corrigibility. Corrigibility has some fundamental difficulties in terms of… If you’re imagining gradient descent process, which is looking at a proxy aligned model and is trying to modify it so that it makes use of this rich input data, it has to do some really weird things to make corrigibility work.

It has to first make a very robust pointer. With corrigibility, if it’s pointing at all incorrectly to the wrong thing in the input data, wrong thing in the world model, the corrigible optimizer won’t correct that pointer. It’ll just be like, “Well, I have this pointer. I’m just trying to optimize for what this thing is pointing for,” and if that pointer is pointing at a proxy instead, you’ll just optimize that proxy. And so you have this very difficult problem of building robust pointers. With deception, you don’t have this problem. A deceptive model, if it realizes the loss function is different than what it thought, it’ll just change to doing the new loss function. It’s actually much more robust to new information because it’s trying to do this instrumentally. And so in a new situation, if it realizes that the loss function is different, it’s just going to automatically change because it’ll realize that’s the better thing to do instrumentally.

And so fundamentally, deception is more robust. It gives you a more robust pointer. It’s easier in that sense. There’s another sense in which corrigibility is very difficult. To produce a corrigible model requires deception modifying the model’s optimization process to be capable of optimizing under uncertainty. For a corrigible model, because it has this pointer as its objective, it’s going to start not really knowing what that pointer is pointing to. It’s going to have some probability distribution over the different possibilities, and it’s going to have to know how to optimize under that probability distribution of objectives rather than under a single objective. And so this problem of optimization under uncertainty, it’s much more complicated than just optimizing under a fixed objective. What that means is that to produce a corrigible optimizer, gradient descent has to do all of these modifications, where it has to modify it to optimize under uncertainty, and to have the super robust pointer to the part of its world model that it’s supposed to be optimizing for.

But if you compare this to deception, gradient descent modifying a model to become deceptive is extremely simple. It needs to modify the model to have a longterm objective, to think about what its objective is across many different instances. And then it just needs to modify the model to think for a while about the fact that it’s in an optimization process and what the right thing is to do. And that modification is not very complicated because it’s just a simple modification of: think more about this particular thing, which is exactly the sort of modification that you should expect to happen all the time when you’re training a model. And so I think it’s a fundamentally much simpler modification. There’s also another argument that you can make here, which is: there’s just a lot more deceptive models. Any proxy objective, once a model, which is optimizing that proxy objective, starts optimizing that objective more in the longterm, across episodes, and then thinks about the fact that it’s an optimization process, will become deceptive.

But to produce corrigibility, you have to find exactly the right pointer. There’s many different possible pointers out there, only one of which is going to give you the exact correct pointer. And similar with the internalization, there’s many different proxies. Only one is the actual true loss function. Whereas with deceptive alignment, any of those different properties, they’re all compatible with deception. And so I think there’s a lot of strong arguments, both this argument for there being many more deceptive optimizers, as well as the simplicity argument for the modification necessary to produce a deceptive optimizer is just a lot simpler, I think, than the modifications necessary to produce these other types of optimizers. And so, because of this, I think that there’s a strong case to be made for deception really not being that uncommon, not being something crazy to think would happened in the training process, but is maybe even potentially the default outcome of a lot of these sorts of training procedures, which is quite, quite scary and quite concerning.

And obviously all of this is speculation. We’re trying to understand from a theoretical process what this gradient process might do, but I think we can make a lot of strong cases about thinking about things like simplicity and accounting arguments to at least put this problem on the radar. Until we have a really strong reason that this isn’t a problem, we should take it seriously. Buck, who’s another person who works at MIRI, often tries to explain some of the risks from learned optimization stuff and he has an analogy that might be useful here. You can imagine the Christian god and the Christian god is trying to produce humans which are aligned with the Bible. And you can imagine three different possible humans. You have Jesus who is just the same as god. Jesus has the same objective as god. Jesus is aligned with god because he just fundamentally wants to do the exact same things.

Lucas Perry: That’s internalization.

Evan Hubinger: That would be internalization. You could have Martin Luther. Martin Luther is aligned with God because he wants to really carefully study the Bible, figure out what the Bible says, and then do that. And that’s the corrigibility case. Or you can have Blaise Pascal and Blaise Pascal is aligned with God because he thinks that if he does what God wants, he’ll go to heaven in the future. And these are the three different possible models that God could find and you’re more likely to find a Jesus, a Martin Luther or a Blaise Pascal.

And the argument is there’s only one Jesus, so out of all the different possible human objectives, only one of them is going to be the exact same one that God wants. And Martin Luther, similarly, is very difficult because out of all the human objectives, there’s only one of them, which is: figure out precisely what the Bible wants and then do that. The Blaise Pascal, in this situation, anybody who realizes that God’s going to send them to heaven or hell, or whatever, based on their behavior, will realize that they should behave according to the Bible, or whatever. And so there’s many different possible Blaise Pascals but there’s significantly fewer possible Martin Luthers and Jesuses.

Lucas Perry: I think that’s an excellent way of simplifying this. Blaise Pascal can care about any kind of proxy. I guess the one thing that I’m still a little bit confused about here is in terms of the deceptive version, again, why is it that it cares about the current proxy reward?

Evan Hubinger: I think that has to do with the structure of the training procedure. You start out usually, I think, with a proxy aligned mesa optimizer. Maybe you start out with a bunch of heuristics, but then once you get a mesa optimizer, it’ll usually start by being proxy aligned. It’ll have some proxy that’s not quite the same as the loss function, at least if you’re in a complex enough environment. There’s a whole bunch of different possibilities. It’s going to start with some proxy, but then you might hope that if you do a bunch of adversarial training, you train it for a really long time in a bunch of different environments, you’ll be able to improve that proxy until you get to the point where it has the right thing.

And the deceptive alignment is about asking the question; there’s actually multiple different possible ways for a model to figure out the loss function. It could figure it out via internalization or it could figure it out via deception or it can figure out via corrigibility. And we don’t know which one of these is more likely. You might imagine that if you start with the proxy aligned mesca optimizer, you train it a bunch, it’ll internalize the correct objective. So the argument that I’m making here is that might not be what happens. And so the reason that I’m saying you start with the proxy aligned mesa optimizers is because I think that’s the way that you should expect that the training procedure will be structured, that you’ll start by learning some proxy and then that proxy will get better, but that having that proxy get better could result internalization or deception or corrigibility.

Lucas Perry: Sure. So the model developed some kind of proxy, but it seems like the other necessary ingredients are, one: it knows that it’s in an optimization process where gradient descent is changing its objective function, and then it requires, two: it actually cares about its current objective function over time and over iterations?

Evan Hubinger: That’s absolutely right. And so then the question to ask is: how likely are those things to occur. And I think it’s the really important point, which I want to make, a lot of times when people think about this, they’re like, “It seems unlikely that it would just happen to develop this understanding and care about it’s thing longterm.” It doesn’t just happen. It happens as a result of the training procedure, because if it does happen, it results in significantly better performance. And so the question is: would gradient descent modify the model to have those properties? And that’s the argument that I want to make is that it’s not that necessarily the model just happens to develop deception, but the deceptive models are just really effective at pursuing reward on the training environment. And so you should expect that if you have a really good training process, that it’s going to find deceptive models because they’re really good at doing the thing you’re training them for.

And so that’s the most fundamental argument that I want to make. But I do think there’s another level here where as you’re getting to the point where you’re training models and very complex data sets, where they have huge amounts of really rich information, we should expect the model to basically figure out most things. You should expect that if it’s going to have to learn all of this really complex data, it’s going to have to build a world model that’s going to include the training process, that’s going to include the loss function. And so I think that as we move into this direction of training more and more complex data sets, more and more complex models, you don’t want to rely on your model not figuring something out, especially when figuring that thing out results in it having better performance.

Now there’s also this question of: does it care about its return across episodes versus just across multiple steps? And for deception, it needs to care about the multi episode return. And this is also another critical point. I think there’s a strong argument that if you imagine a model being trained to care about its multi-step return, and maybe also I should pause here and explain what I mean by step versus episode. So in the standard reinforcement learning setup, you have many different episodes. Each episode is composed of many different steps and you train the model to only optimize its single episode return, its reward across all the different steps in an episode, but not across episodes. That’s not always the case. There are some reinforcement learning procedures that optimize across episodes. An example of this is something like population-based training. Population-based training selects models which have the greatest reward across episodes, not simply within the episode.

Though, standard reinforcement learning, you’re just selecting for the single episode return. But even if you’re just selecting for the single episode return, you have the same general problem that we’ve been talking about where it’s not the case that you actually end up with a model necessarily that only cares about single-step return. You’re just going to end up with a model that behaves like it cares about single-step return on the training distribution. And if you imagine a situation where you have a model and it’s been trained to optimize for this objective over many steps, and then it discovers that actually there’s these other episodes that are beyond this current step. I think that the most natural generalization is that you should expect that the model is going to care about those other steps too.

You can imagine a situation where every situation where the model has seen a blue door, it’s been like, “Oh, going through this blue is really good,” and it’s learned an objective that incentivizes going through blue doors. If it then later realizes that there are more blue doors than it thought because there are other blue doors in other episodes, I think you should generally expect it’s going to care about those blue doors as well. But again, even if you don’t buy that, it’s not just about what’s most natural. It’s also about: what does the gradient descent process do? And because models which have this property, that plan across episodes, can become deceptive and therefore do really well in the training distribution, you should expect that a good training process, it’s going to modify them to become deceptive, even if they wouldn’t naturally.

Lucas Perry: Okay. So help me just understand a little bit more here. At a high level, it seems that the internalization and the corrigibility things require higher specification than the deception version. You’ve brought up all these different arguments about why you think that the deceptive version is easier to find because it has all of these computational benefits in terms of minimizing the loss function, right?

Evan Hubinger: That’s right.

Lucas Perry: And the structure of the deceptive model is composed of, as we mentioned earlier, one: the knowledge that it is within some kind of optimization process where it is being perturbed by gradient descent over episodes, and that it cares about the current proxy over episodes. And it seems that you’re arguing, correct me if I’m wrong, the manifestation of these components is that the strong desire to deceive leads to a very strong, direct attempt at actually minimizing the loss or accomplishing the objective function?

Evan Hubinger: That’s right.

Lucas Perry: Because if it fails at doing that, it won’t be able to accomplish the proxy that it actually cares about later?

Evan Hubinger: That’s right. If there’s more deployment episodes than training episodes, and it just cares about how many times it goes through the blue door or the green arrow, or whatever, the green arrow is a proxy objective, and so if it gives up the fact that it’s optimizing for the green arrow, it’s going to get modified to not do that anymore. And so if it wants to go to the green arrow in all of these different deployment episodes, it needs to first, in the training procedure, pretend that it doesn’t care about the green arrow. And then once it’s deployed, switch to going after the green arrow. This also requires the models to know when it’s in training and when it’s in deployment.

This is another situation where I think you should expect that when you’re training really advanced models with these huge world models, that they figure out this information, but also even if it doesn’t figure out that information, it can gamble. It can be like, “Well, there’s some chance I’m in deployment. There’s some chance I’m in training. I will just opt to go to the green arrow some percentage of the time and do the correct thing some other percentage of the time.” And this gives you a decision problem which you can solve mathematically. You get some optimal defection probability.

Lucas Perry: Okay. And then going up, in terms of specification one step, the corrigible agent needs to have a very detailed world model where it has a pointer via gradient descent at some specific part of the world model that seems to correlate highly with minimizing the loss.

Evan Hubinger: That’s right.

Lucas Perry: Why is that less likely?

Evan Hubinger: Well, so the major two reasons that I want to argue for are, one: getting the pointer right is really hard and very difficult to do robustly, whereas getting the deception right is much easier to do robustly. Think about animal imprinting. If the animal imprints on a human caretaker instead, if you have a missspecified pointer, it doesn’t actually have very good performance. The deceptive alignment is more robust at figuring out what the thing is it’s trying to optimize for because the corrigible alignment relies on this pointer working successfully. And if the pointer fails, it’s just going to optimize for the wrong thing.

Lucas Perry: But why can’t the pointer change just as well as the attempt at satisfying the actual loss function in the deceptive model?

Evan Hubinger: So absolutely can. But the question I want to ask is: how difficult is it for gradient descent to make one sequence of modifications versus another sequence of modifications? The sequence of modification that leads to corrigibility versus the sequence that leads to deceptive alignment. The sequence of modifications necessary to produce corrigible alignment requires all of this very, very careful pointer tuning. And so, you can imagine if you’re considering an infinitesimal movement in the direction of corrigible alignment, imagine replacing your current proxy with a bad pointer. It’s not going to be that great. You’re going to have to put a lot of careful optimization effort into getting a really good pointer before it becomes better to be corrigible.

Whereas with deception, the model could build up this really good world model. And once the world model gets complicated enough that it includes a specification of the fact that it’s in a training process, as soon as it starts thinking about the training process, it’s going to be as robust as it can be because it’s going to instrumentally just want to try to figure out what is the best possible approximation I can get for this thing I’m being trained for, so I can optimize it as effectively as possible. And so this gives you a much more robust pointer compared to the corrigible alignment case.

Lucas Perry: And so the danger would be with the deceptive version: once it’s unleashed upon the world and it has a very excellent world model, it realizes this and finally now I can do the deceptive turn, or something like that, to actually optimize the proxy?

Evan Hubinger: That’s right, yeah.

Lucas Perry: So we’ve covered a lot of the ways in which inner alignment fails. Now, inner alignment and outer alignment are two of the things which you care about for evaluating proposals, for building safe and advanced AI. There are two other properties that you care about training procedures for building beneficial AI. One of these is training competitiveness and the second one is performance competitiveness. Could you explain what training competitiveness is and performance competitiveness and why they’re both important?

Evan Hubinger: Absolutely, yeah. So I mentioned at the beginning that I have a broad view of AI alignment where the goal is to try to mitigate AI existential risks. And I mentioned that what I’m working on is focused on this intent alignment problem, but a really important facet of that problem is this competitiveness question. We don’t want to produce AI systems which are going to lead to AI existential risks. And so we don’t want to consider proposals which are directly going to cause problems. As the safety community, what we’re trying to do is not just come up with ways to not cause existential risk. Not doing anything doesn’t cause existential risk. It’s to find ways to capture the positive benefits of artificial intelligence, to be able to produce AIs which are actually going to do good things. You know why we actually tried to build AIs in the first place?

We’re actually trying to build AIs because we think that there’s something that we can produce which is good, because we think that AIs are going to be produced on a default timeline and we want to make sure that we can provide some better way of doing it. And so the competitiveness question is about how do we produce AI proposals which actually reduce the probability of existential risk? Not that just don’t themselves cause existential risks, but that actually overall reduce the probability of it for the world. There’s a couple of different ways which that can happen. You can have a proposal which improves our ability to produce other safe AI. So we produce some aligned AI and that aligned AI helps us build other AIs which are even more aligned and more powerful. We can also maybe produce an aligned AI and then producing that aligned AI helps provide an example to other people of how you can do AI in a safe way, or maybe it provides some decisive strategic advantage, which enables you to successfully ensure that only good AI is produced in the future.

There’s a lot of different possible ways in which you could imagine building an AI leading to reduced existential risks, but competitiveness is going to be a critical component of any of those stories. You need your AI to actually do something. And so I like to split competitiveness down into two different sub components, which are training competitiveness performance competitiveness. And in the overview of 11 proposals document that I mentioned at the beginning, I compare 11 different proposals for prosaic AI alignment on the four qualities of outer alignment, inner alignment, training competitiveness, and performance competitiveness. So training competitiveness is this question of how hard is it to train a model to do this particular task? It’s a question fundamentally of, if you have some team with some lead over all different other possible AI teams, can they build this proposal that we’re thinking about without totally sacrificing that lead? How hard is it to actually spend a bunch of time and effort and energy and compute and data to build an AI, according to some particular proposal?

And then performance competitiveness is the question of once you’ve actually built the thing, how good is it? How effective is it? What is it able to do in the world that’s really helpful for reducing existential risk? Fundamentally, you need both of these things. And so you need all four of these components. You need outer alignment, inner alignment, training competitiveness, and performance competitiveness if you want to have a prosaic AI alignment proposal that is aimed at reducing existential risk.

Lucas Perry: This is where a bit more reflection on governance comes in to considering which training procedures and models are able to satisfy the criteria for building safe advanced AI in a world of competing actors and different incentives and preferences.

Evan Hubinger: The competitive stuff definitely starts to touch on all those sorts of questions. When you take a step back and you think about how do you have an actual full proposal for building prosaic AI in a way which is going to be aligned and do something good for the world, you have to really consider all of these questions. And so that’s why I tried to look at all of these different things in the document that I mentioned.

Lucas Perry: So in terms of training competitiveness and performance competitiveness, are these the kinds of things which are best evaluated from within leading AI companies and then explained to say people in governance or policy or strategy?

Evan Hubinger: It is still sort of a technical question. We need to have a good understanding of how AI works, how machine learning works, what the difficulty is of training different types of machine learning models, what the expected capabilities are of models trained under different regimes, as well as the outer alignment and inner alignment that we expect will happen.

Lucas Perry: I guess I imagine the coordination here is that information on relative training competitiveness and performance competitiveness in systems is evaluated within AI companies and then possibly fed to high power decision makers who exist in strategy and governance for coming up with the correct strategy, given the landscape of companies and AI systems which exist?

Evan Hubinger: Yeah, that’s right.

Lucas Perry: All right. So we have these intent alignment problems. We have inner alignment and we have outer alignment. We’ve learned about that distinction today, and reasons for caring about training and performance competitiveness. So, part of the purpose of this, I mean, is in the title for this paper that partially motivated this conversation, An Overview of 11 Proposals for Building Safe and Advanced AI. You evaluate these proposals based on these criteria, as we mentioned. So I guess, I want to take this time now then to talk about how optimistic you are about, say your top few favorite proposals for building safe and advanced AI and how you’ve roughly evaluated them on these four criteria of inner alignment, outer alignment, and then performance and training competitiveness.

Evan Hubinger: I’ll just touch on some of the ones that I think are most interesting to start by taking a look at. And it’s not necessarily just going to be the ones I think I’m most excited about, because some are more complicated than others. I want to start with some of the simpler ones and some of the ones that I am fairly excited about and I think are interesting, but that also are pedagogically useful to take a look at. I’ll start with amplification, which is the approach that I probably spend the most time thinking about.

The specific form of amplification that I want to start by talking about is imitative amplification, where imitative amplification specifically means the procedure of training a model to imitate a human with access to that model. The goal is to produce models imitating this HCH tree, which I mentioned earlier, this tree of humans consulting humans. The model is trained to mimic a human consulting the model, but then the models that that human is consulting with themselves trained to mimic humans consulting models, which were themselves trained to mimic humans consulting models. So the limit that you’re hoping to end up with this model trained to mimic this tree of humans consulting humans consulting humans, which is HCH for humans consulting HCH.

The specific way to try to talk about this though in the overview of 11 proposals, is that there’s an additional thing which you can do on top of this imitative objective, which is, we also want to add some form of inner alignment solution. So we might think that imitative amplification is going to be good in the limit. It’s going to be good if we can actually produce models, which are really trying to predict HCH, but we also have to actually ensure that we’re doing that, that we don’t get deceptive models or other sort of bad inner misalignment problems.

And so we can add on top of imitative amplification, something which I call relaxed adversarial training. I have another post that came out of some of the work I did at OpenAI, which really goes in detail into how you would do something like relaxed adversarial training of what it means, or how it works. But fundamentally, the idea behind relaxed adversarial training is to train the model, both to do the base objective of trying to imitate the human consulting the model, but also on an additional objective of trying to decrease the probability that the model is doing something bad, it’s deceptive as assessed by some overseer, which is looking at the model. Which in this case, in the case of amplification, the overseer would be the human with access to the model.

So you’re training overall on the objective of mimicking the human with access to the model and making it so that the human with access to the model, when that human with the help of the model uses transparency tools to look at the model and see what it’s doing, making sure that that human believes that the model is doing the right thing. And so you’re combining training on the imitative objective as well as on the relaxed adversarial training objective, trying to make it so that this oversight works.

Lucas Perry: How do you evaluate that in terms of your criteria?

Evan Hubinger: We can start with outer alignment. So the question of outer alignment is, is this HCH tree aligned? And the hope is that because this HCH is just composed of humans, that if we can understand the human behavior and ensure that those humans are doing the right thing, which we might hope to do. We’ve solved lots of human coordination problems in the past that we can ensure that this thing is outer aligned.

For inner alignment, inner alignment is this question of relaxed adversarial training. So in particular, the way in which I think about solving relaxed adversarial training in this situation is that you have this big model space and you have some set of deceptive models. And if you want to avoid that set of deceptive models, one thing which you can do is you can find some larger set, which contains all the deceptive models, such that it’s very easy to check whether there are models in that set or not. As long as outside of the set contains enough competitive models that they’re able to do the right thing, your relaxed adversarial training will work.

There’s a bunch of different ideas for how you could define a set like this, but the one which I’m most excited about, which I think about the most is to have the set be based on the distinction of myopia. Where you run this relaxed adversarial training check by looking at the model, determining whether the model cares about its multi-episode return or not. If the model does care about its multi-episode return, then it has the potential to become deceptive. And so we don’t allow that.

We have it so that the overseer, if it sees that it cares about the multi-episode return, oversight fails. And if it sees that it doesn’t care about the multi-episode return, then the oversight passes. And then we train the model to cause the oversight to pass so that it doesn’t develop this behavior where it cares about the multi-episode return. That’s the basic idea of how you could use relaxed adversarial training here to solve inner alignment.

And then you have training competitiveness, it’s sort of this imitation learning problem. We’re trying to train the model to imitate the human with access to the model. There’s all sorts of potential problems here. It could be very difficult to get a bunch of human data for this problem, as well as if you’re just doing imitation learning, there’s some concerns of, is the model really going to learn these sort of powerful long-term optimization, especially if you’re specifically selecting it to be myopic? Also, there’s a question of how rich is the reward signal, which is produced by imitation learning? Imitation learning uses some sort of fixed distance metric to determine the distance between what the model produced and what was predicted.

Generally, this is done by having the language model which reduces the probability distribution, and then determining the probability under that distribution of the sentences produced by the human with access to the model. And so you have this problem of how rich is the training signal, how difficult is the training? I think we really don’t know what the training competitiveness is going to be, but I think there’s reason to be optimistic. I mean, a lot of these sorts of simple language modeling tasks, like GPT-2 and GPT-3 have been very successful. And so this is very similar to that in that sort of vein, and so you can hope that it’ll also be hopefully successful similarly.

And then you have the performance competitiveness, and for performance competitiveness, the real question is how powerful is this HCH thing? Can HCH actually solve all the problems you’ve wanted to solve? And this is going to be a question of, how do you train your humans? Can the human take any question and decompose it into different sub questions such that the answer to those sub questions can be recombined into an answer to the whole question? Is this possible for all questions? So there are some questions which are undecomposable, in which case it wouldn’t be competitive, but if you can decompose all of these questions, it sort of should have performance competitiveness.

So the one that I’ve been talking about imitative amplification plus relaxed adversarial training is the one that I work on the most, and it’s probably the one I’m most optimistic about. There’s others that are also really interesting. I mean, I think all of the proposals, every proposal on the list has lots of strong arguments in favor of it and lots of real reasons to consider it.

Another proposal which I have talked about previously is AI safety via debate. So AI safety via debate has some similar properties to amplification, but it has a different objective. Rather than trying to mimic HCH, you’re trying to do this debate procedure where you have two different models trying to defend two different sides of some proposition, and convince the judge to agree with one of those sides. There’s some strong arguments for why this sort of procedure would have nice outer alignment properties because it converges to something like what the judge’s equilibrium beliefs are after having seen all of the arguments in favor. And so we might hope that this sort of equilibrium beliefs are going to be good.

There’s lots of work also that’s been done to try to improve the ability to access like the true equilibrium beliefs to really give the judge all possible information. There’s some work that was done by Paul Christiano and Beth Barnes about this cross-examination mechanism, which gives it significantly better performance competitiveness properties by allowing it to access a larger complexity class. But then the other question is then what can you do on top of AI safety via debate to help with inner alignment?

An interesting thing that you can do is you can have the models trained to use transparency tools on each other to win the debate. If one model is able to look inside the other model and demonstrate that it’s non-myopic or something, then it wins the debate. And so then you can incentivize the models to play this game of trying to look inside of each other and find the inner alignment problems, and then train away from those problems.

For outer alignment, you sort of really have this question of, are the debaters actually incentivized to have some honest, cooperative, helpful answers in the limit. And this is a question of what that equilibrium really looks like. Is it the case that humans are able to be tricked by really persuasive arguments and no counterargument will unpersuade them? Or is it the case that when humans are exposed to all of the best arguments and counterarguments, they come to the right conclusions? We don’t really know, but this is fundamentally what the outer alignment question that’s here. This is sort of a trickier equilibrium to analyze.

With amplification case, we just had to figure out with this concrete tree of humans, but with debate, we have to understand this more abstract, what is this theoretical equilibrium behavior of a human under exposure to all these different arguments? I think there’s a case that from an outer alignment perspective, debate is at least more questionable.

Then we have inner alignment and the situation here is, how successfully can we incentivize these models to inspect each other and be able to identify catastrophic behavior? There’s certainly some problems here. You have to get a good training signal to be able to produce models, which are good at inspecting each other, which probably means you have to produce deceptive models and then have the models find the deception, or at least produce non-myopic models and then have the models find the non-myopia. But then this could also incentivize the models to hide the non-myopia, so it’s a little bit unclear exactly how this works. It’s sort of going to dependent on precisely the properties of the training procedure.

I think there’s some good reasons to be hopeful here for training competitiveness, because the way that the debate game is structured, it lets you do this self-play structure where you have these agents competing on the zero-sum game in a self-play, which is in many ways, similar to a lot of big successes of things like AlphaZero, where AlphaZero is able to solve all these environments by doing this very successful self-play on these zero-sum games.

The biggest problem here is probably getting data, and then there’s the performance competitiveness question, which is how effective would an actual debate agent be? The hope is that in the limit, you should just be able to ask the first debater, what do you think? And then it’ll just choose the most convincing answer, and then you can just go with that. You don’t have to run the whole debate in deployment, but it’s only going to be able to solve these language problems.

It’s just going to give you is the equilibrium of what a human thinks after all of these different arguments, and is that good enough? Is it the case that humans are going to really be able to come to good enough equilibria after they see all these arguments that they’re going to be able to produce really good answers? And also, is it the case that question answering alone is sufficient to be able to be competitive in potentially a very competitive marketplace?

As a third proposal that I think is interesting to go into is something called microscope AI. Microscope AI I think is really interesting to look at because it’s very different from the other proposals that I was just talking about. It has a very different approach to thinking about how do we solve these sorts of problems. For all of these approaches, we need to have some amount of abilities to look inside of our models and learn something about what the model knows. But when you use transparency tools to look inside of the model, it teaches you multiple things. It teaches you about the model. You learn about what the model has learned. But it also teaches you about the world, because the model learned a bunch of useful facts, and if you look inside the model and you can learn those facts yourself, then you become more informed. And so this process itself can be quite powerful.

That’s fundamentally the idea of microscope AI. The idea of microscope AI is to train a predictive model on the data you want to understand, and then use transparency tools to understand what that model learned about that data, and then use that understanding to guide human decision making. And so if you’re thinking about outer alignment, in some sense, this procedure is not really outer aligned because we’re just trying to predict some data. And so that’s not really an aligned objective. If you had a model that was just trying to do a whole bunch of prediction, it wouldn’t be doing good things for the world.

But the hope is that if you’re just training a predictive model, it’s not going to end up being deceptive or otherwise dangerous. And you can also use transparency tools to ensure that it doesn’t become that. We still have to solve inner alignment, like I was saying. It still has to be the case that you don’t produce deceptive models. And in fact, the goal here really is not to produce mesa optimizers at all. The goal is just to produce these predictive systems, which learn a bunch of useful facts and information, but that aren’t running optimization procedures. And hopefully we can do that by having this very simple, predictive objective, and then also by using transparency tools.

And then training competitiveness, we know how to train powerful predictive models now, you know, something like GPT-2, and now GPT-3, these are predictive models on task prediction. And so we know this process, we know that we’re very good at it. And so hopefully we’ll be able to continue to be good at it into the future. The real sticky point with microscope AI is the performance competitiveness question. So is enhanced human understanding actually going to be sufficient to solve the use cases we might want for like advanced AI? I don’t know. It’s really hard to know the answer to this question, but you can imagine some situations where it is and some situations where it isn’t.

So, for situations where you need to do long-term, careful decision making, it probably would be, right? If you want to replace CEOs or whatever, that’s a sort of very general decision making process that can be significantly improved just by having much better human understanding of what’s happening. You don’t necessarily need the AI to making the decision. On the other hand, if you need fine-grained manipulation tasks or very, very quick response times, AIs managing a factory or something, then maybe this wouldn’t be sufficient because you would need the AIs to be doing all of this quick decision making and you couldn’t have it just giving information to a few.

One specific situation, which I think is important to think about also is the situation of using your first AI system to help build a second AI system, and making sure that second AI system is aligned and competitive. I think that it also performs pretty well there. You could use a microscope AI to get a bunch of information about the process of AIs and how they work and how training works, and then get a whole bunch of information about that. Have the humans learn that information, then use that information to improve our building of the next AIs and other AIs that we build.

There are certain situations where microscope AI is performance competitive, situations where it wouldn’t be performance competitive, but it’s a very interesting proposal because it’s sort of tackling it from a very different angle. It’s like, well, maybe we don’t really need to be building agents. Maybe we don’t really need to be doing this stuff. Maybe we can just be building this microscope AI. I should mention the microscope AI idea comes from Chris Olah, who works at OpenAI. The debate idea comes from Geoffrey Irving, who’s now at DeepMind, and the amplification comes from Paul Christiano, who’s at OpenAI.

Lucas Perry: Yeah, so for sure, the best place to review these is by reading your post. And again, the post is “An overview of 11 proposals for building safe advanced AI” by Evan Hubinger and that’s on the AI Alignment Forum.

Evan Hubinger: That’s right. I should also mention that a lot of the stuff that I talked about in this podcast is coming from the Risks from Learned Optimization in Advanced Machine Learning Systems paper.

Lucas Perry: All right. Wrapping up here, I’m interested in ending on a broader note. I’m just curious to know if you have concluding thoughts about AI alignment, how optimistic are you that humanity will succeed in building aligned AI systems? Do you have a public timeline that you’re willing to share about AGI? How are you feeling about the existential prospects of earth-originating life?

Evan Hubinger: That’s a big question. So I tend to be on the pessimistic side. My current view looking out on the field of AI and the field of AI safety, I think there’s a lot of really challenging, difficult problems that we are at least not currently equipped to solve. And it seems quite likely that we won’t be equipped to solve by the time we need to solve them. I tend to think that the prospects for humanity aren’t looking great right now, but I nevertheless have a very sort of optimistic disposition, we’re going to do the best that we can. We’re going to try to solve these problems as effectively as we possibly can and we’re going to work on it and hopefully we’ll be able to make it happen.

In terms of timelines, it’s such a complex question. I don’t know if I’m willing to commit to some timeline publicly. I think that it’s just one of those things that is so uncertain. It’s just so important for us to think about what we can do across different possible timelines and be focusing on things which are generally effective regardless of how it turns out, because I think we’re really just quite uncertain. It could be as soon as five years or as long away as 50 years or 70 years, we really don’t know.

I don’t know if we have great track records of prediction in this setting. Regardless of when AI comes, we need to be working to solve these problems and to get more information on these problems, to get to the point we understand them and can address them because when it does get to the point where we’re able to build these really powerful systems, we need to be ready.

Lucas Perry: So you do take very short timelines, like say 5 to 10 to 15 years very seriously.

Evan Hubinger: I do take very short timelines very seriously. I think that if you look at the field of AI right now, there are these massive organizations, OpenAI and DeepMind that are dedicated to the goal of producing AGI. They’re putting huge amounts of research effort into it. And I think it’s incorrect to just assume that they’re going to fail. I think that we have to consider the possibility that they succeed and that they do so quite soon. A lot of the top people at these organizations have very short timelines, and so I think that it’s important to take that claim seriously and to think about what happens if it’s true.

I wouldn’t bet on it. There’s a lot of analysis that seems to indicate that at the very least, we’re going to need more compute than we have in that sort of a timeframe, but timeline prediction tasks are so difficult that it’s important to consider all of these different possibilities. I think that, yes, I take the short timelines very seriously, but it’s not the primary scenario. I think that I also take long timeline scenarios quite seriously.

Lucas Perry: Would you consider DeepMind and OpenAI to be explicitly trying to create AGI? OpenAI, yes, right?

Evan Hubinger: Yeah. OpenAI, it’s just part of the mission statement. DeepMind, some of the top people at DeepMind have talked about this, but it’s not something that you would find on the website the way you would with OpenAI. If you look at historically some of the things that Shane Legg and Demis Hassabis have said, a lot of it is about AGI.

Lucas Perry: Yeah. So in terms of these being the leaders with just massive budgets and person power, how do you see the quality and degree of alignment and beneficial AI thinking and mindset within these organizations? Because there seems to be a big distinction between the AI alignment crowd and the mainstream machine learning crowd. A lot of the mainstream ML community hasn’t been exposed to many of the arguments or thinking within the safety and alignment crowd. Stuart Russell has been trying hard to shift away from the standard model and incorporate a lot of these new alignment considerations. So yeah. What do you think?

Evan Hubinger: I think this is a problem that is getting a lot better. Like you were mentioning, Stuart Russell has been really great on this. CHAI has been very effective at trying to really get this message of, we’re building AI, we should put some effort into making sure we’re building safe AI. I think this is working. If you look at a lot of the major ML conferences recently, I think basically all of them had workshops on beneficial AI. DeepMind has a safety team with lots of really good people. OpenAI has a safety team with lots of really good people.

I think that the standard story of, oh, AI safety is just this thing that these people who aren’t involved in machine learning think about it’s something which really in the current world has become much more integrated with machine learning and is becoming more mainstream. But it’s definitely still a process, and it’s the process of like Stuart Russell says that the field of AI has been very focused on the sort of standard model and trying to move people away from that and think about some of the consequences of it takes time and it takes some sort of evolution of a field, but it is happening. I think we’re moving in a good direction.

Lucas Perry: All right, well, Evan, I’ve really enjoyed this. I appreciate you explaining all of this and taking the time to unpack a lot of this machine learning language and concepts to make it digestible. Is there anything else here that you’d like to wrap up on or any concluding thoughts?

Evan Hubinger: If you want more detailed information on all of the things that I’ve talked about, the full analysis of inner alignment and outer alignment is in Risks from Learned Optimization in Advanced Machine Learning Systems by me, as well as many of my co-authors, as well as “an overview of 11 proposals” post, which you can find on the AI Alignment Forum. I think both of those are resources, which I would recommend checking out for understanding more about what I talked about in this podcast.

Lucas Perry: Do you have any social media or a website or anywhere else for us to point towards?

Evan Hubinger: Yeah, so you can find me on all the different sorts of social media platforms. I’m fairly active on GitHub. I do a bunch of open source development. You can find me on LinkedIn, Twitter, Facebook, all those various different platforms. I’m fairly Google-able. It’s nice to have a fairly unique last name. So if you Google me, you should find all of this information.

One other thing, which I should mention specifically, everything that I do is all public. All of my writing is public. I try to publish all of my work and I do so on the AI Alignment Forum. So the AI Alignment Forum is a really, really great resource because it’s a collection of writing by all of these different AI safety authors. It’s open to anybody who’s a current AI safety researcher, and you can find me on the AI Alignment Forum as evhub, I’m E-V-H-U-B on the AI Alignment Forum.

Lucas Perry: All right, Evan, thanks so much for coming on today, and it’s been quite enjoyable. This has probably been one of the more fun AI alignment podcasts that I’ve had in a while. So thanks a bunch and I appreciate it.

Evan Hubinger: Absolutely. That’s super great to hear. I’m glad that you enjoyed it. Hopefully everybody else does as well.

End of recorded material

Steven Pinker and Stuart Russell on the Foundations, Benefits, and Possible Existential Threat of AI

 Topics discussed in this episode include:

  • The historical and intellectual foundations of AI 
  • How AI systems achieve or do not achieve intelligence in the same way as the human mind
  • The rise of AI and what it signifies 
  • The benefits and risks of AI in both the short and long term 
  • Whether superintelligent AI will pose an existential risk to humanity

You can take a survey about the podcast here

Submit a nominee for the Future of Life Award here

 

Timestamps: 

0:00 Intro 

4:30 The historical and intellectual foundations of AI 

11:11 Moving beyond dualism 

13:16 Regarding the objectives of an agent as fixed 

17:20 The distinction between artificial intelligence and deep learning 

22:00 How AI systems achieve or do not achieve intelligence in the same way as the human mind

49:46 What changes to human society does the rise of AI signal? 

54:57 What are the benefits and risks of AI? 

01:09:38 Do superintelligent AI systems pose an existential threat to humanity? 

01:51:30 Where to find and follow Steve and Stuart

 

Works referenced: 

Steven Pinker’s website and his Twitter

Stuart Russell’s new book, Human Compatible: Artificial Intelligence and the Problem of Control

 

We hope that you will continue to join in the conversations by following us or subscribing to our podcasts on Youtube, Spotify, SoundCloud, iTunes, Google Play, StitcheriHeartRadio, or your preferred podcast site/application. You can find all the AI Alignment Podcasts here.

You can listen to the podcast above or read the transcript below. 

Note: The following transcript has been edited for style and clarity.

 

Lucas Perry: Welcome to the AI Alignment Podcast. I’m Lucas Perry. Today, we have a conversation with Steven Pinker and Stuart Russell. This episode explores the historical and intellectual foundations of AI, how AI systems achieve or do not achieve intelligence in the same way as the human mind, the benefits and risks of AI over the short and long-term, and finally whether superintelligent AI poses an existential risk to humanity. If you’re not currently following this podcast series, you can join us by subscribing on Apple Podcasts, Spotify, Soundcloud, or on whatever your favorite podcasting app is by searching for “Future of Life.” Our last episode was with Sam Harris on global priorities. If that sounds interesting to you, you can find that conversation wherever you might be following us. 

I’d also like to echo two announcements for the final time. So, if you’ve been tuned into the podcast recently, you can skip ahead just a bit. The first is that there is an ongoing survey for this podcast where you can give me feedback and voice your opinion about content. This goes a long way for helping me to make the podcast valuable for everyone. This survey should only come out once a year. So, this is a final call for thoughts and feedback if you’d like to voice anything. You can find a link for the survey about this podcast in the description of wherever you might be listening. 

The second announcement is that at the Future of Life Institute, we are in the midst of our search for the 2020 winner of the Future of Life Award. The Future of Life Award is a $50,000 prize that we give out to an individual who, without having received much recognition at the time of their actions, has helped to make today dramatically better than it may have been otherwise. The first two recipients of the Future of Life Award were Vasili Arkhipov and Stanislav Petrov, two heroes of the nuclear age. Both took actions at great personal risk to possibly prevent an all-out nuclear war. The third recipient was Dr. Matthew Meselson, who spearheaded the international ban on bioweapons. Right now, we’re not sure who to give the 2020 Future of Life Award to. That’s where you come in. If you know of an unsung hero who has helped to avoid global catastrophic disaster, or who has done incredible work to ensure a beneficial future of life, please head over to the Future of Life Award page and submit a candidate for consideration. The link for that page is on the page for this podcast or in the description of wherever you might be listening. If your candidate is chosen, you will receive $3,000 as a token of our appreciation. We’re also incentivizing the search via MIT’s successful red balloon strategy, where the first to nominate the winner gets $3,000 as mentioned, but there are also tiered pay outs where the first to invite the nomination winner gets $1,500, whoever first invited them gets $750, whoever first invited them $375, and so on. You can find details about that on the Future of Life Award page. Link in the description. 

Steven Pinker is a Professor in the Department of Psychology at Harvard University. He conducts research on visual cognition, psycholinguistics, and social relations. He has taught at Stanford and MIT and is the author of ten books: The Language Instinct, How the Mind Works, The Blank Slate, The Better Angels of Our Nature, The Sense of Style, and Enlightenment Now: The case for Reason, Science, Humanism, and Progress. 

Stuart Russell is a Professor of Computer Science and holder of the Smith-Zadeh chair in engineering at the University of California, Berkeley. He has served as the vice chair of the World Economic Forum’s Council on AI and Robotics and as an advisor to the United Nations on arms control. He is an Andrew Carnegie Fellow as well as a fellow of the Association for the Advancement of Artificial Intelligence, the Association for Computing Machinery and the American Association for the Advancement of Science.

He is the author with Peter Norvig of the definitive and universally acclaimed textbook on AI, Artificial Intelligence: A Modern Approach. He is also the author of Human Compatible: Artificial Intelligence and the Problem of Control. 

And with that, here’s our conversation with Steven Pinker and Stuart Russell. 

So let’s get started here then. What are the historical and intellectual foundations upon which the ongoing AI revolution is built?

Steven Pinker: I would locate them in the Age of Reason and the Enlightenment, when Thomas Hobbes said, “Reasoning is but reckoning,” reckoning in the old-fashioned sense of “calculation” or “computation.” A century later, the two major style of AI today were laid out: The neural network, or massively parallel interconnected system that is trained with examples and generalizes by similarity, and the symbol-crunching, propositional, “Good Old-Fashioned AI.” Both of those had adumbrations during the Enlightenment.  David Hume, in the empiricist or associationist tradition, said there are only three principles of connection among ideas, contiguity in time or place, resemblance, and cause and effect. On the other side, you have Leibniz, who thought of cognition as the grinding of wheels and gears and what we would now call the manipulation of symbols. Of course the actual progress began in the 20th century with the ideas of Turing and Shannon and Weaver and Norbert Wiener. The rest is the history that Stuart writes about in his textbook and his recent book.

Stuart Russell: I think I would like to add in a little bit of ancient history as well, just because I think Aristotle not only thought a lot about how human thinking was organized and how it could be correct or incorrect and how we could make rational decisions, he very clearly describes a backward regression goal planner in one of his pieces, and his work was incredibly influential. One of the things he said is we deliberate about means and not about ends. I think he says, “A doctor does not choose whether to heal,” and so on. And you might disagree with that, but I think that that’s been a pretty influential thread in Western thinking for the last two millennia or more. That we kind of take objectives as given and the purpose of intelligence is to act in ways that achieve your objectives.

That idea got refined gradually. So Aristotle talked mainly about goals and logically provable sequences of actions that would achieve those goals. And then in the 17th and 18th centuries, I want to give a shout out to the French and the Swiss, so Pascal and Fermat and Arnauld and Bernoulli brought in ideas of rational decision making under uncertainty and the weighing of probabilities and the concept of utility that Bernoulli introduced. So that generalized Aristotle’s idea, but it didn’t change the fundamental principle that they took the objectives, the utilities, as given. Just intrinsic properties of a human being in a given moment.

In AI, we sort of went through the same historical development, except that we did the logic stuff for the first 30 years or so, roughly, and then we did the probability and decision theory stuff for the next 30 years. I think we’re in a terrible state now, because the vast majority of the deep learning community, when you read their papers, nothing is cited before 2012. Occasionally, from time to time, they’ll say things like, “For this problem, the learning algorithms that we have are probably inadequate, and in future I think we should direct some of our research towards something that we might call reasoning or knowledge,” as if no one had ever thought of those things before and they were the first person in history to ever have the idea that reasoning might be necessary for intelligence.

Steven Pinker: Yes.

Stuart Russell: I find this quite frustrating and particularly frustrating when students want to actually just bypass the AI course altogether and go straight to the deep learning course, because they just don’t think AI is necessary anymore.

Steven Pinker: Indeed, and also galling to me. In the late ’80s and ’90s I was involved in a debate over the applicability of the predecessors of deep learning models, then called multi-layer perceptrons, artificial neural networks, connectionist networks, and Parallel Distributed Processing networks. Gary Marcus and Alan Prince and Michael Ullman and other collaborators I pointed out the limitations of trying to achieve intelligence–even for simple linguistic processes like forming the plural of a noun or a past tense of a verb–if the only tool you had available was the ability to associate features with features, without any symbol processing. That debate went on for a couple of decades and then petered out. But then one of the prime tools in the neural network community, multilayer networks trained by error back-propagation,  were revived in 2012. Indeed there is an amnesia for the issues in that debate, which Gary Marcus has revived for a modern era.

It would be interesting to trace the truly radical idea behind artificial intelligence: not just that there are rules or algorithms, whether they are from logic or probability theory, that an intelligent agent can use, the way a human pulls out a smartphone. But the idea that there is nothing but rules or algorithms, and that’s what an intelligent agent consists of: that is, no ghost to the machine, no agent separate from the mechanism. And there, I’m not sure whether Aristotle actually exorcised the ghost in the machine. I think he did have a notion of a soul. The idea that it’s rules all the way down,  that intelligence is just a mechanism, probably has shallow roots. Although Hobbes probably could claim credit for it, and perhaps Hume as well.

Lucas Perry: That’s an excellent point, Steve, it seems like Abrahamic religions have kind of given rise in part to this belief, or maybe an expression of that belief, the kind of mind-body dualism, the ghost in the machine where the mind seems to be a nonphysical thing. So it seems like intelligence has had to go the same road of “life.” There used to be “elan vital” or some other spooky presupposed mechanism for giving rise to life. And so similarly with intelligence, it seems like we’ve had to move from thinking that there was a ghost in the machine that made the things work to there being rules all the way down. If you guys have anything else to add to that, I think that’d be interesting.

My other two reactions to what has been said so far are that this point about computer science taking the goal as given, I think is important and interesting, and maybe we could expand upon that a little bit. Then there’s also, Stuart mentioned the difference between AI and deep learning and that students want to skip the AI and just get straight to the deep learning. That seemed a little bit confusing to me.

Steven Pinker: Let me address the first part and I’ll turn it over to Stuart for the second. The notion of dualism–that there is a mechanism, but sitting on top of it is an immaterial agent or self or soul or I–is enshrined in the Abrahamic religions and in other religions, but it has deep intuitive roots. We are all intuitively dualists (Paul Bloom has made this argument in his book Descartes’ Baby.) Fortunately, when we deal with each other in everyday life we don’t treat each other like robots or wind-up dolls, but we assume that there is an inner life that is much like ours, and we make sense of people’s behavior in terms of their beliefs and desires, which we don’t conceptualize as neural circuitry transforming patterns. We think there’s a locus of consciousness, which is easy to think of as separate from the flesh that we’re made of, especially since–and this is a point made by the 19th century British anthropologist Edward Tylor–that there’s actually a lot of empirical “evidence” that supports dualism in our everyday life.  Like dreaming.

When you dream, you know your body is in bed the whole time, but there’s some part of you that’s up and about in the world. When you see your reflection in a mirror or in still water, there is an animated essence that seems to have parted company with your body. When you’re in a trance from a drug or a fever and have an out-of-body experience, it seems  that we and our bodies are not the same thing. And with death, one moment a person is walking around, the next moment the body is lifeless. It’s natural to think that it’s lost some invisible ingredient that had animated it while it was alive.

Today we know that this is just the activity of the brain, but in terms of the experience available to a person, dualism seems perfectly plausible. It’s one of the great achievements of neuroscience, on the one hand,  to show that a brain is capable of supporting problem solving and perception and decision making, and of the computational sciences, on the other, for showing that intelligence can be understood in terms of information and computation, and that goals (like the Aristotelian final cause) can be understood in terms of control and cybernetics and feedback.

Stuart Russell: On the point that in computer science, we regard the objectives as fixed, it’s much broader than just computer science. If you look at Von Neumann — Morgenstern and their characterizations of rationality, nowhere do they talk about what is the process by which the agent might rationally come by its preferences. The agent is always assumed a priori to come with the preferences built in, and the only constraint is that those preferences be self-consistent so that you can’t be driven around circles of intransitive preferences where you simply cough up money to go round and round the same circle.

The same thing I think is true in control theory, where the objective is the cost function, and you design a controller that minimizes the expected cost function, which might be a square of the distance from the desired trajectory or whatever it might be. Same in statistics, where let’s just assume that there’s a loss function. There’s no discussion in statistics of what the loss function should be or how the loss function might change or anything like that.

So this is something that pervades many of the technological underpinnings of the 20th century. As far as I can tell, to some extent in developmental psychology, but I think in moral philosophy, people really take seriously the question of what goal should we have? Is it moral for an agent to have such and such as its objective, and how could we, for example, teach an agent to have different objectives? And that gets you into some very unchartered philosophical waters about what is a rational process that would lead an agent to have different objectives at the end than it did at the beginning, given that if it has different objectives at the end, then it can only expect that it won’t be achieving the objectives that it has at the beginning. So why would it embark on a process that’s going to result in failure to achieve the objectives that it currently has?

So that’s sort of a philosophical puzzle, but it’s a real issue because in fact human beings do change. We’re not born with the preferences that we have as adults, and so there is a notion of plasticity that absolutely has to be understood if we’re to get this right.

Steven Pinker: Indeed, and I suspect we’ll return to the point later when we talk about potential risks of advanced artificial intelligence. The issue is whether a system having intelligence implies that the system would have certain goals, and probably Stuart and I agree the answer is no, at least not by definition. Precisely because what you want and how to get what you want are two logically independent questions. Hume famously said that reason must be that the slave of the passions, by which he didn’t mean that we should just surrender to our impulses and do whatever feels good. What he meant was that reason itself can’t specify the goals that it tries to bring about. Those are exogenous. And indeed, von Neumann and Morgenstern are often misunderstood as saying that we must be ruthlessly, egotistical self interested maximizers. Whereas the goal that is programmed into us — say by evolution or by culture — could include other people’s happiness as part of our utility function. That is a question that merely making our choices consistent is silent on.

 So the ability to reason doesn’t by itself give you moral goals, including taking into account the interests of others. That having been said, there is a long tradition in moral philosophy which shows  how it doesn’t take much to go from one to the other. Because as soon as we care about persuading others, as soon as our interests depend on how others treat us, then we can’t get away with saying “only my interests count and yours don’t because I am me and you’re not,” because there is no logical difference between “me” and “you.” So we’re forced to a kind of impartiality, wherein whatever I insist on for me I’ve got to grant to you, a kind of Golden Rule or Categorical Imperative that makes our interests interchangeable as soon as we’re in discourse with one another.

This is all to acknowledge Stuart’s point, but to take it a few steps further in how it deals with the question of what our goals ought to be.

Stuart Russell: The other point you raised Lucas was on being confused by my distinction between AI and deep learning.

Lucas Perry: That’s right.

Stuart Russell: I think you’re pointing to a confusion that exists in the public mind, in the media and even in parts of the AI community. AI has always included machine learning as a subdiscipline, all the way back to Turing’s 1950 paper, where he speculates, in fact, that might be a good way to build AI would be just start with a child program and train it to be an adult intelligent machine. But there are many other sub-fields of AI; knowledge representation, reasoning, planning, decision making under uncertainty, problem solving, perception. Machine learning is relevant to all of these because they all involve processes that can be improved through experience. So that’s what we mean by machine learning: simply the improvement of performance through experience; and deep learning is a technology that helps with that process.

It by itself as far as we can tell, doesn’t have what is necessary to produce general intelligence. Just to pick one example, the idea that human beings know things seems so self-evident that we hardly need to argue about it. But deep learning systems in a real sense don’t know things. They can’t usefully acquire knowledge by reading a book and then go out and use that to design a radio telescope, which human beings arguably can. So it seems inevitable that if we’re going to make progress, I mean, sure we take the advances that deep learning has offered. Effectively, what we’ve discovered with deep learning is that you can train more complicated circuits than we previously would have guessed possible using various kinds of stochastic gradient descent, and other tricks.

I think it’s true to say that most people would not have expected that you could build a thousand layer network that was 20,000 units wide. So it’s got 20 million circuit elements and simply put a signal in one end and some data in the other and expect that you’re going to be able to train those 20 million elements to represent the complicated function that you’re trying to get it to learn. So that was a big surprise, and that capability is opening up all kinds of new frontiers: in vision, in speech recognition, language, machine translation, and physical control in robots among other things. It’s a wonderful set of advances, but it’s not the entire solution. Any more than group theory is the entire solution to mathematics. There’s lots of other branches of mathematics that are exciting and interesting and important and you couldn’t function without them. The same is true for AI.

So I think that we’re probably going to see even without further major conceptual advances, another decade of progress in achieving greater understanding of why deep learning works and how to do it better, and all the various applications that we can create using it. But I think if we don’t go back and then try to reintegrate all the other ideas of AI, we’re going to hit a wall. And so I think the sooner we lose our obsession with this new shiny thing, the better.

Steven Pinker: I couldn’t agree more. Indeed, in some ways we have already hit the wall. Any user of Siri or Cortana or a question-answering system has been frustrated by the way they just make associations to individual words and have a shallow understanding of the syntax of the sentence. If you ask Google or Siri, “Can you show me digital music players without a camera?” It’ll give you a long list of music players with discussions of their cameras, failing to understand the syntax of “X without Y.” Or, “What are some fast food restaurants nearby that are not McDonald’s?”  and you get a list of nearby McDonald’s.

It’s not hard to bump into the limitations of systems that for all their sophistication are being trained on associations among local elements, and can–I agree, surprisingly–learn higher-order combinations of those elements. But despite the name “deep learning,” they are shallow in the sense that they don’t build up a knowledge base of what are the objects, and who did what to whom, which they can access through various routes.

Stuart Russell: Yeah. My favorite example, I’m not sure if it’s apocryphal, is you say to Siri, “Call me an ambulance,” and Siri says, “Okay. From now on I’ll call you Ann Ambulance.”

Steven Pinker: In Marx Brothers movie, there’s the sequence, “Call me a taxi.” “Okay. You’re a taxi.” I don’t know if the AI story is an urban legend based on the Marx Brothers movie or whether life is imitating art.

Lucas Perry: Steven, I really appreciate it and liked that point about dualism and intelligence. I think it points in really interesting directions around identity in the self, which we don’t have time to get into here. But I did appreciate that.

So moving on ahead here, to what extent do you both see AI systems as achieving intelligence in the same way or not as the human mind does? What kinds of similarities are there or differences?

Stuart Russell: This is a really interesting question and we could spend the whole two hours just talking about this. So by artificial intelligence, I’m going to take it that we mean not deep learning, but the full range of techniques that AI researchers have developed over the years.

So some of them– for example, logical reasoning were– developed going back to Aristotle and other Greek philosophers who developed formal logic to model human thinking. So it’s not surprising that when we build programs that do logical reasoning, we are in some sense capturing one aspect of human reasoning capability. Then in the ’80s, as I mentioned, AI developed reasoning under uncertainty, and then later on refining that with notions of causality as well, particularly in the work of Judea Pearl. The differences are really because AI and cognitive science separated probably sometime in the ’60s. I think before that there wasn’t really a clear distinction between whether you were doing AI or whether you were doing cognitive science. It was very much the thought that if you could get a program to do anything that we think of as requiring intelligence with a human, then you were in some sense exhibiting a possible theory of how the human does it, or even you would make introspective claims and say, “Look, I’ve now shown that this theory of intelligence really works.”

But fairly soon people said, “Look, this is not really scientific. If you want to make a claim about how the human mind does something, you have to base it on real psychological experimentation with human subjects.” And that’s distinct from the engineering goal of AI, which is simply to produce programs that demonstrate certain capabilities. So for most of the last 50, 60 years, these two fields have grown further and further apart. I think now partly because of deep learning and partly because of other work, for example in probabilistic programming, we can start to do things that humans do that we couldn’t do before. So it becomes interesting again, to ask, well, are humans really somewhat Bayesian and are they doing these kinds of Bayesian symbolic probabilistic program learning that, for example, Josh Tenenbaum was proposing or are they doing something else? For example, Geoff Hinton is pretty adamant that as he puts it, symbols are the luminiferous aether of AI by which he means that they’re simply something that we imagined and they have no physical reality whatsoever in the human mind.

I find this a little hard to believe, and you have to wonder if symbols don’t exist, why are almost all deep learning applications aimed at recognizing the symbolic category to which an object belongs, and I haven’t heard an answer yet from the deep learning community about why that is. But it’s also clear that AI systems are doing things that have no resemblance to human cognition. When you look at what AlphaGo is actually doing, part of it is that sort of perception-like ability to look at a position and get a sense, to use an anthropomorphic term, of its potential for winning for white or for black. And perhaps that part is human-like, and actually it’s incredibly good. It’s probably better at recognizing the potential position directly with no deliberation whatsoever than a human is.

But the other part of what AlphaGo does is completely non-human. It’s considering sequences of moves from the current state that run all the way to the end of the game. So part of it is searching in a tree which could go 40 or 50 or possibly more moves into the future. Then from the end of the tree, it then plays a random game all the way to the end and sees who wins that game. And this is nothing like what human beings do. When humans are reasoning about a game like Go or Chess, first of all, we are thinking about it at multiple levels of abstraction. So we’re thinking about the liveness of a particular group, we’re thinking about control of a particular region of territory on the board. We’re thinking, “Well, if I give up control of this territory, then I can trade it for capturing his group over there.”

So this kind of reasoning simply doesn’t happen in AlphaGo at all. We reason back from goals. In chess you say, “Perhaps I could trap his queen. Let me see if I can come up with a move that blocks his exit for the queen.” So we reason backwards from some goals and no chess program and no Go program does that kind of reasoning. The reason humans do this is because the world is incredibly complicated and in different circumstances, different kinds of cognitive processing are efficient and effective in producing good decisions quickly. And that’s the real issue for human intelligence, right?

If we didn’t have to worry about computation, then we would just set up the giant unknown, partially observable, Markov decision process of the universe, solve it and then we would take the first action in the virtually infinite strategy tree that solves that POMDP. Then we would observe the next percept, we would update all our beliefs about the universe and we would resolve the universe and that’s how we would proceed. We would have to do that sort of roughly every millisecond to control the muscles in our body, but we don’t do anything like that. All of the different kinds of mental capabilities that we have are deployed in this amazingly fluid way to get us through the complexity of the real world. We are so far away in AI from understanding how to do that, that when I see people say, “We’re just going to scale up our deep learning systems by another three orders of magnitude and we’ll be more intelligent than humans,” I just smile.

Steven Pinker: Yeah. I’d like to complement some of those observations. It is true that in the early days of artificial intelligence and cognitive psychology, they were driven by some of the same players. Herb Simon and Allen Newell can be credited as among the founders of AI and the founders of cognitive psychology. Likewise, Marvin Minsky and John McCarthy. When I was an undergraduate, I caught the tail end of what was called the cognitive revolution. It was exhilarating after the dominance of psychology by behaviorism, which forbade any talk of mentalistic concepts. You weren’t allowed to talk about memories or plans or goals or ideas or rules, because they were considered to be unobservable and thus unscientific. Then the concept of computation domesticated those mentalist terms and opened up a huge space of hypotheses. What are the rules by which we understand and formulate sentences?, a project that Noam Chomsky initiated. How can we model human knowledge as a semantic network?, a project that Minsky and Alan Collins and Ross Quillian and others developed. How do we make sense of foresight and planning and problem solving, which Newell and Simon pioneered?

There was a lot of back and forth between AI and cognitive science when they were first exposed to the very idea that intelligence could be understood in mechanistic terms, and there was a flow of hypotheses from computer science that psychologists then tested as possible models. Ideas that you couldn’t even frame, you couldn’t even articulate before there was the language of computation, such as What is the capacity of human short term memory? or What are the search algorithms by which we explore a problem space? These were unintelligible in the era of behaviorism.

All this caught the attention of philosophers like Hilary Putnam, and later Dan Dennett, who noted that the ideas from the hybrid of cognitive psychology and artificial intelligence were addressing deep questions about what mental entities consist of, namely information processing states. The back-and-forth spilled into the ’70s when I was a graduate student, and even the ’80s when centers for cognitive science were funded by the Sloan Foundation. There was also a lot of openness in the companies that hired artificial intelligence researchers: AT&T Bell Labs, which was a scientific powerhouse before the breakup of AT&T. Bolt Beranek and Newman in Cambridge, which eventually became part of Verizon. I would go there as a grad student to hear talks on artificial intelligence. I don’t know if this is apocryphal history, but Xerox Palo Alto Research Centers, where I was a consultant, was so open that, according to legend, Steve Jobs walked in and saw the first computer with a graphic user interface and a mouse and windows and icons, stole the ideas, and went on to build the Lisa and then the Macintosh. Xerox was out on their own invention, and companies got proprietary . Many of the AI researchers in companies no longer publish  in peer-reviewed journals in psychology the way they used to, and the two cultures drifted apart. 

Since hypotheses from computer science and artificial intelligence are just hypotheses, there is the question of whether the best engineering solution to a problem is the one that the brain uses. There’s the obvious objection that the hardware is radically different: the brain is massively parallel and noisy and stochastic; computers are serial and deterministic. That led in part to the backlash in the ’80s when perceptrons and artificial neural networks were revived. There was skepticism about the more symbolic approaches to artificial intelligence, which has been revived now in the deep learning era.

to get back to the question, what are ways in which human minds differ from AI systems? It depends on the AI system assessed, as Stuart pointed out. Both of us would agree that the easy equation of deep learning networks with human intelligence is unwarranted, that a lot of the walls that deep learning is hitting come about because, despite the noisy parallel elements the brain is made of, we do emulate a kind of symbol processing architecture, where we can be taught explicit propositions, and human intelligence does make use of these symbols in addition to massively parallel associative networks.

I can’t help but mention a historical irony.  I’ve known Geoff Hinton since we were both post-docs. Hinton himself, early in his career, provided a refutation of the very claim of his that Stuart cited, that symbols are like luminiferous aether, a mythical entity. Geoff and I have noted to each other that we’ve switched sides in the debate on the nature of cognition. There was a debate in the 1970s on the format of mental imagery. Geoff and I were on opposite sides, but he was the symbolic proposition guy and I was the analog parallel network guy.  

Hinton showed that our understanding of an  object depends on the symbolic format in which we mentally represent it. Take something as simple as a cube, he said. Imagine a cube poised on one of its vertices, with the diagonally opposite vertex aligned above it. If you ask people, “Point to all the other vertices,” they are stymied. Their imagery fails, and they often leave out a couple of vertices. But if, instead of describing it to them as a cube tilted on its diagonal axis, you describe it as two tilted diamonds, one above the other, or as two tripods joined by a zig-zag ring, they “see” the correct answer. Even visualizing an object depends critically on how people mentally describe it to themselves with symbols. This is an argument for symbolic representations that Geoff Hinton made in 1979, and with his recent remarks about symbols he seems to have forgotten his own powerful example.

Stuart Russell: I think another area where deep learning is clearly not capturing the human capacity for learning, is just in the efficiency of learning. I remember in the mid ’80s going to some classes in psychology at Stanford, and there were people doing machine learning then and they were very proud of their results, and somebody asked Gordon Bower, “how many examples do humans need to learn this kind of thing?” And Gordon said “one Sometimes two, usually one”, and this is genuinely true, right? If you look for a picture book that has one to two million pictures of giraffes to teach children what a giraffe is, you won’t find one. Picture books that tell children what giraffes are have one picture of a giraffe, one picture of an elephant, and the child gets it immediately, even though it’s a very crude cartoonish drawing, of a giraffe or an elephant, they never have a problem recognizing giraffes and elephants for the rest of their lives.

Deep learning systems are needing, even for these relatively simple concepts, thousands, tens of thousands, millions of examples, and the idea within deep learning seems to be that well, the way we’re going to scale up to more complicated things like learning how to write an email to ask for a job, is that we’ll just have billions or trillions of examples, and then we’ll be able to learn really, really complicated concepts. But of course the universe just doesn’t contain enough data for the machine to learn direct mappings from perceptual inputs or really actually perceptual input history. So imagine your entire video record of your life, and that feeds into the decision about what to do next, and you have to learn that mapping as a supervised learning problem. It’s not even funny how unfeasible that is. The longer the deep learning community persists in this, the worse the pain is going to be when their heads bang into the wall.

Steven Pinker: In many discussions of superintelligence inspired by the success of deep learning I’m puzzled as to what people could possibly mean. We’re sometimes asked to imagine an AI system that’s could solve the problem of Middle East peace or cure cancer. That implies that we would have to train it with 60 million other diseases and their cures, and it would extract the patterns and cure the new disease that we present it with. Needless to say, when it comes to solving global warming, or pandemics, or Middle Eastern peace, there aren’t going to be 60 million similar problems with their correct answers that could provide the training set for supervised learning.

Lucas Perry: So, human children and humans are generally capable of one shot learning, or you said we can learn via seeing one instance of a thing, whereas machine learning today is trained up via very, very large data sets. Can you explain what the actual perceptual difference is going on there? It seems for children, they see a giraffe and they can develop a bunch of higher order facts about the giraffe, like that it is tan, and has spots, and a long neck, and horns and other kinds of higher order things. Whereas machine learning systems may be doing something else. So could you explain that difference?

Stuart Russell: Yeah, I think you actually captured it pretty well. The human child is able to recognize the object, not as 20 million pixels, including–let’s not forget–all the pixels of the background. So many of these learning algorithms are actually learning to recognize the background, not the object at all. They’re really picking up on spurious regularities that happen in the way the images are being captured. But the human child immediately separates the figure from the background says, “okay, it’s the figure that’s being called a giraffe”, and recognizes the higher level properties; “okay, it’s a quadruped, relatively large” the most distinguishing characteristic, as you say, is the very long neck, plus the way its hide is colored. Probably most kids might not even notice the horns and I’m not even sure if all giraffes have the horns, or just the males or just the adults. I don’t know the answer to that.

So I wasn’t paying much attention to all those images. This carries over to many, many other situations, including in things like planning, where if we observe someone carrying out a successful behavior, that one example combined with our prior knowledge is typically enough for us to get the general idea of how to do that thing. And this prior knowledge is absolutely crucial. Just information-theoretically, you can’t learn from one example reliably, unless you bring to bear a great deal of prior knowledge. And this is completely absent in deep learning systems in two ways. One is they don’t have any prior knowledge. And two is some of the prior knowledge is specifically about the thing you’re trying to predict. So here, we’re trying to predict the category of an animal and we already have a great deal of prior knowledge about what it means to belong to a category of animals.

So for example, who owns you, is not an attribute that the child would need to know or care about. If you said, what kind of animal is this? And deep learning systems have no ability to include or exclude any input attribute on the basis of its relevance to what it’s trying to predict, because they know nothing about what it is you’re trying to predict. And if you think about it, that doesn’t make any sense, right? If I said, “okay, I want you to learn to predict predicate P1279A. Okay? And I’m going to give you loads and loads of examples.” And now you get a perfect predictor for ‘P1279A’, but you have absolutely no use for it, because P1279A doesn’t connect to anything else in your cognition. So you learned a completely useless predictor because you know nothing about the thing that you’re trying to predict.

So it seems like it’s broken in several really, really important ways, and I would say probably the absence of prior knowledge or any means to bring to bear prior knowledge on the learning process is the most crucial.

Steven Pinker: Indeed, this goes back to our conversation on how basic principles of intelligence that govern the design of intelligent systems provide hypotheses that can be tested within psychology. What Stuart has identified is ultimately the nature-nurture problem in cognition. Namely, what are the innate constraints that govern children’s first hypotheses as they try to make sense of the world? 

One famous answer is Chomsky’s universal grammar, which guides children as they acquire language. Another is the idea from my colleagues Susan Carey and Elizabeth Spelke, in different formulations, that children have a prior concept of a physical object whose parts move together, which persists over time, and which follows continuous spatiotemporal trajectories; and that they have a distinct  concept of an agent or mind, which is governed by beliefs and desires. Maybe, or maybe not, they come equipped with still other frameworks for concepts, like the concept of a living thing or the concept of an artifact, and these priors radically cut down the search space of hypotheses, so they don’t have to search at the level of pixels and all their logically possible weighted combinations. 

Of course, the challenge in the science is how you specify the innate constraints, the prior knowledge, so that they aren’t obviously too specific, given what we know about the plasticity of human cognition. The extreme example being the late philosopher Jerry Fodors’ suggestion that all concepts are innate, including “trombone” and “doorknob” and “carburetor.”

Stuart Russell: (Laughs)

Steven Pinker: Hard to swallow, but between that extreme and the deep learning architecture in which the only thing that’s innate are the pixels, the convolutional network that allows for translational invariance, and the network of connections, there’s an interesting middle ground. That defines the central research question in cognitive development.

Stuart Russell: I don’t think you have to believe in extensive innate structures in order to believe that prior knowledge is really, really important for learning. I would guess that some aspects of our cognition are innate, and one of them is probably that the world contains things, and that’s really important because if you just think about the brain as circuits, some circuit languages don’t have things as first class entities, whereas first order logical languages or programming languages do have things as first class entities and that’s a really important distinction.

Even if you believe that nothing is innate, the point is how does everything that you have perceived up to now affect your ability to learn the next thing? One argument is, everything you’ve perceived up to now, is simply data, and somehow magically, we have access to all our past perceptions, and then you’re just training a function from that whole lot to the next thing to do or how to interpret the next object.

That doesn’t make much sense. Presumably the experience you have from birth or even pre-birth onwards, is converted into something and one argument is that it’s just converted into something like knowledge, and then that knowledge is brought to bear on learning problems, for example, to even decide what are the relevant aspects of the input for predicting category membership of this thing?

And the other view would be that, in the deep learning community, they would say probably something like the accumulation of features. If you imagine a giant recurrent neural network: in the hidden layers of the recurrent neural network over years and years and years of perception, you’re building up internal representations, features, which then can perhaps simplify the learning of the next concept that you need to learn. And there’s probably some truth in that too.

And absolutely having a library of features that are generally useful for predicting and decision making and planning and our entire vocabulary, I think this is something that people often miss, our vocabulary, our language, is not just something we use to communicate with each other. It’s an enormous resource for simplifying the world in the right ways, to make the next thing we need to know, or the next thing we need to do, relatively easy. Right? So you imagine you decide at the age of 12, I want to understand the physical laws that control the universe.

The fact that we have in our vocabulary, something like doing a PhD, makes it much more feasible to figure out what your plan is going to be, to achieve this objective. If you didn’t have that, and if you didn’t have all the pieces of doing a PhD, like take a course, read a book, this library of words and action primitives, at all these levels of abstraction, is a resource without which you would be completely unable to formulate plans of any length or any likelihood of success. And this is another area where current AI systems, I would say generally, not just deep learning, we lack a real understanding of how to formulate these hierarchies and acquire this vocabulary and then how to deploy it in a seamless way so that we’re always managing to function successfully in the real world.

Lucas Perry: I’m basically just as confused about I guess, intelligence as anyone else. So the difference, it seems to me between the machine learning system and the child who one-shot-learns the giraffe is, that the child brings into this learning scenario, this knowledge that you guys were talking about, that they understand that the world is populated by things and that there are other minds and some other ideas about 3D objects and perception, but a core difference seems to be something like symbols and the ability to manipulate symbols is this right? Or is it wrong? And what are symbols and effective symbol manipulation made of?

Steven Pinker: Yes, and that is a limitation of the so-called deep learning systems, which are a subset of machine learning, which is a subset of artificial intelligence. It’s certainly not true that AI systems don’t manipulate symbols.  Indeed, that’s what classical AI systems trade in: manipulation of propositions, implementation of versions of logical inference or of cause-and-effect reasoning. Those can certainly be implemented in AI systems–it’s just gone out of fashion with the deep learning craze.

Lucas Perry: Well, they don’t learn those symbols, right? Like we give them the symbols and then they manipulate them.

Steven Pinker: The basic architecture of the system, almost by definition, can’t be learned;  you can’t learn something with nothing. There have got to be some elementary information processes, some formats of data representation, some basic ways of transforming one representation to another, that are hardwired into the architecture of the system. It’s an open empirical question, in the case of the human brain, whether it includes variables for objects and minds, or living things, or artifacts, or if those are scaffolded one on top of the other with experience. There’s nothing in principle that prevents AI systems from doing that;  many of them do, but at least for now they seem to have fallen out of fashion.

Stuart Russell: There is precedent for generating new symbols, both in the probabilistic programming literature and in the inductive logic programming literature. So predicate invention is a very important reason for doing inductive logic programming. But I agree with Steve, that it’s an open question as to whether the basic capacity to have a new symbol based representations in the brain is innate, or is it learned? There’s very anecdotal evidence about what happens to children who are not brought up among other human beings. I think those anecdotes suggest that they don’t become symbol-using in the same way. So it might be that the process of developing symbol-using capabilities in the brain is enormously aided by the fact that we grow up in the presence of symbol-using entities, namely our parents and family members and community. And of course that leads you to then a chicken and egg problem.

So you’d have to argue in that case that early humans, or pre-humans had much more rudimentary symbol-like capabilities: some animals have the ability to refer to different phenomena or objects with different signs, different kinds of sounds that some new world monkeys have, for example, for a snake and for puma, but they’re not able to do the full range of things that we do with symbols. You could argue that the symbol using capability developed over hundreds of thousands of years and the unaided human mind doesn’t come with it built in, but because we’re usually bathed in symbol-using activity around us, we are able to quickly pick it up. I don’t know what the truth is, but it seems very clear that this kind of capability, for example, gives you the ability to generalize so much faster than you can with circuits. So just to give a particular example of the rules of Go, we talked about earlier, the rules of Go apply the same rule at every time-step in the game.

And it’s the same rule at every square in the game, except around the edges, and if you have what we call first order capability, meaning you can have universal quantifiers or in programs, we think of these as loops, you can say very quickly for every square on the board, if you have a piece on there and it’s surrounded by the enemy, then it’s dead. That’s sort of a crude approximation to how things work and go, but it’s roughly right. In a circuit, you can’t say that because you don’t have the ability to say for every square. So you have to have a piece of circuit for each square. So you’ve got 361 copies of the rule in each of those copies has to be learned separately, and this is one of the things that we do with convolutional neural networks.

A convolutional neural network has the universal quantifier over the input space, built into it. So it’s a kind of cheating, and as far as we know, the brain doesn’t have that type of weight sharing. So the key aspect is not just the physical structure of the convolutional network, which has this repeating local receptive fields on each different part of the retina, so to speak, but that we also insist that weights for each of those local receptive fields are copied across all receptive fields in the retina. So there aren’t millions of separate weights that are trained, there’s only a few, sometimes even just a handful of weights that are trained and then the code makes sure that those are effectively copied across the entire retina. And the brain. I don’t think has any way to do that, so it’s doing something else to achieve this kind of rapid generalization.

Lucas Perry: All right. So now with all of this context and understanding about intelligence and its origins today in 2020, AI is beginning to proliferate and is occupying a lot of news cycles. What particular important changes to human society does the rise and proliferation of AI signal and how do you view it in relation to the agricultural and industrial revolutions?

Steven Pinker: I’m going to begin with a meta-answer, which is that we should keep in mind how spectacularly ignorant we are of the future even the relatively near future. Experts at superforecasting studied by Phil Tetlock, pretty much the best in the world, go down to about chance after about five years out. And we know, looking at predictions of the future from the past,  how ludicrous they can be, both in underpredicting technological changes and in overpredicting them. A 1993 book by Bill Gates called The Road Ahead  said virtually nothing about the internet! And there’s a sport of looking at science-fiction movies and spotting ludicrous anachronisms, such as the fact that in 2001: A Space Odyssey they were using typewriters. They had suspended animation and trips to Jupiter, but they hadn’t invented the word processor. To say nothing of the social changes they failed to predict, such as the fact that all of the women in the movie were secretaries and assistants.

So we should begin by acknowledging that it is extraordinarily difficult to predict the future. And there’s a systematic reason, namely that the future depends not just on technological developments, but also on people’s reaction to the developments, and on the  reactions to the reactions, and the reactions to the reactions to the reactions. There are seven and a half billion of us reacting, and we have to acknowledge that there’s a lot we’re going to get wrong. 

It’s safe to say that a lot of tasks that involve physical manipulation, like stocking shelves and driving trucks, are going to be automated, and societies will have to deal with the possibility of radical changes in employment, and Stuart talks about those in his book. We don’t know whether the job market will be flexible enough to create new jobs, always at the frontier of what machines can’t yet do, or whether there’ll be massive unemployment that will require economic adjustments, such as a universal basic income or government sponsored service. 

Less clear is the extent to which high-level decision making, like policy, diplomacy, or scientific hypothesis-testing,  will be replaced by AI. I think that’s impossible to predict.  Although, closer to the replacement of truck drivers by autonomous vehicles, AI as a useful tool, rather than as a replacement, for human intelligence will explode in science and business and technology and every walk of life.

Stuart Russell: I think all of those things are true. And I agree that our general record of forecasting has been pretty dismal. I am smiling as Steve was talking, because I was remembering Ray Kurzweil recently saying how proud he was that he had predicted the self driving car, I think it was in ’96 or ’92, something like that, and possibly wasn’t aware that the first self driving car was driving on the freeway in 1987, before he even thought to predict that such a thing might happen. If I had to say, in the next decade, if you said, roughly speaking, that what happened in the 2010s was primarily that visual perception became very crudely feasible for machines when it wasn’t before.

And that’s already having huge impact, including in self-driving cars, I would say that language understanding at least in a simplified sense will become possible in this decade. And I think it’ll be a combination of deep learning with probabilistic programming, with Bayesian and symbolic methods. That will open up enormous areas of activity to machines where they simply couldn’t go before, and some of that will be very straightforward, job replacement for call center workers. Most of what they do, I think could be automated by systems that are able to understand their conversations. The role of the smart speaker, the Alexa, or Cortana or Siri or whatever will radically change and will enable AI systems to actually understand your life to a much greater extent. One of the reasons that Siri and Cortana or Alexa are not very useful to me is because they just don’t understand anything about my life.

The “call me an ambulance” example illustrates that. If I got a text message saying “Johnny’s in the hospital with a broken arm”, well, if it doesn’t understand that Johnny is possibly my cat, or possibly my son, or possibly my great grandfather and does Johnny live nearby, or in my house, or on another continent, then it hasn’t the faintest idea of what to do. Or even whether I care. It’s only really through language understanding. I doubt that we’re going to be filling these things full of first order logic assertions that we will type into our AI system. So it’s only through language that it’s going to be able to acquire the knowledge that it needs to be a useful assistant to an individual or a corporation. So having that language capability will open up whole new areas for AI to be useful to individuals and also to take jobs from people. And I’m not able to predict what else we might be able to do when there are AI systems that understand language, but it has to have a huge impact.

Lucas Perry: Is there anything else that you guys would like to add in terms of where AI is at right now, where it’ll be in the near future and the benefits and risks it will pose?

Stuart Russell: I could point to a few things that are already happening. There’s a lot of discussion about the negative impacts on women and minorities from algorithms that inadvertently pick up on biases in society. So we saw the example of Amazon’s hiring algorithm that rejected any resume that had the word “woman’s” in it. And I think that’s serious, but I think the AI community we’re still not completely woke, and there’s a lot of consciousness raising that needs to happen. But I think technically that problem is manageable, and I think one interesting thing that’s occurring is that we’re starting to develop an understanding, not just of the machine learning algorithm, but of the socio-technical context in which that machine learning is embedded and modeling that social technical context allows you to predict whether the use of that algorithm will have negative feedback kinds of consequences, or it will be vulnerable to certain kinds of selection bias in the input data, and so on.

Deepfakes surveillance and manipulation, that’s another big area, and then something I’m very concerned about is the use of AI for autonomous weapons. This is another area where we fight against media stereotypes. So when the media talk about autonomous weapons, they invariably have a picture of a Terminator. Always. And I tell journalists, I’m not going to talk to you if you put a picture of a Terminator in the article. And they always say, well, I don’t have any control over that, that’s a different part of the newspaper, but it always happens anyway.

And the reason that’s a problem is because then everyone thinks, “Oh, well this is science fiction. We don’t have to worry about this because this is science fiction.” And you know, I’ve heard the Russian ambassador to the UN and Geneva say, well, why are we even discussing these things, because this is science fiction, it’s 20 or 30 years in the future? Oh, by the way, I have some of these weapons, if you’d like to buy them. The reality is that many militaries around the world are developing these, companies are selling them. There’s a Turkish arms company, STM, selling a device, which is basically the slaughterbot from the Slaughterbots movie. So it’s a small drone with onboard explosives and they advertise it as capable of tracking and autonomously attacking human beings based on video signatures and/or face recognition.

The Turkish government has announced that they’re going to be using those against the Kurds in Syria sometime this year. So we’ll see if it happens, but there’s no doubt that this is not science fiction, and it’s very real. And it’s going to create a new kind of weapon of mass destruction, because if it’s autonomous, it doesn’t need to be supervised. And if it doesn’t need to be supervised, then you can launch them by the million, and then you have something with the same effect as a nuclear weapon, but much cheaper, much easier to proliferate with much less collateral damage and all the rest of it.

Steven Pinker: I think in all of these discussions, it’s critical to not fall prey to a status-quo bias and compare the hypothetical problems of a future technology with an idealized present, ignoring the real problems with the present we take for granted. In the case of bias, we know that humans are horribly biased. It’s not just that we’re biased against particular genders and ethnic groups and sexual orientations. But inj general we make judgements that can easily be outperformed by even simple algorithms, like a linear regression formula. So we should remember that our benchmark in talking about the accuracies or inaccuracies of AI prediction algorithms has to be the human, and that’s often a pretty low bar. When it comes to bias, of course, a system that’s trained on a sample that’s unrepresentative is not a particularly intelligent system. And going back to the idea that we have to distinguish the goals we want to achieve from the intelligence that achieves them, if our goal is to overcome past inequities, then by definition we don’t want to make selections that simply replicate the statistical distribution of women and minorities in the past. Our goal is to rectify those inequities, and the problem in a system that replicates them is not that it’s not intelligent enough, but then we’ve given it the wrong goal.

When it comes to weapons, here too, we’ve got to compare the potential harm of intelligent weapons systems with the stupendous harm of dumb weapon systems. Aerial bombardment, artillery, automatic weapons, search-and- destroy missions, and tank battles have killed people by the millions. I think there’s been insufficient attention to how a battleground that used smarter weapons would compare to what we’ve tolerated for centuries simply because that’s what we have come to accept, though it’s being fantastically destructive. What ultimately we want to do is to make the use of any weapons less likely, and as I’ve written about, that has been the general trend in the last 75 years, fortunately.

Stuart Russell: Yeah, I think there is some truth in that. When I first got the email from Human Rights Watch, so they began a campaign, I think was back in 2013, to argue for a treaty banning autonomous weapons. Human Rights Watch came into existence because of the awful things that human soldiers do. And now they’re saying “No, no human soldiers are great, it’s the machines we need to worry about.” And I found that a little bit odd. To me, the argument about whether the weapons will inadvertently violate humans right in ways that human soldiers don’t, or sort of accidentally kill people in ways that we are getting better at avoiding, I don’t think that’s the issue. I think it’s specifically the weapon of mass destruction property that autonomous weapons have that for example machine guns don’t.

There’s a hundred million or more Kalashnikov rifles in private hands in the world. If all those weapons got up one morning by themselves and started shooting anyone they could see, that would be a big chunk of the human race gone, but they don’t do that. Each of them has to be carried by a person. And if you want to put a million of them into the field, you need another 10 million people to feed and train those million soldiers, and to transport them, and protect them, and all that stuff. And that’s why we haven’t seen very large scale death from all those hundred million Kalashnikovs.

Even carpet-bombing, which I think nowadays would be regarded as indiscriminate and therefore a violation of international law. And I think even during the Second World War, people argued that “No, you can’t go and bomb cities.” But once the Germans started to do it, then there was escalating rounds of retaliation and people lost all sense of what was a civilized and what was an uncivilized act of war. But even The Blitz against Great Britain, as far as I know killed only between 50 and 60,000 people, even though it hit dozens and dozens of cities. But literally one truckload of autonomous weapons can kill a million people.

An interesting fact about World War II is that for every person who died, between 1,000 and 10,000 bullets were fired. So just killing people with bullets on average in World War II cost you, let’s take a geometric mean 3,000 bullets, which is actually about a thousand dollars at current prices, but you could build a lethal autonomous weapon for a lot less than that. And even if they had a 25% success rate in finding and killing a human, it’s much cheaper than the bullet, let alone the guns and the aircraft and all the rest of it.

So as a way of killing very, very large numbers of people it’s incredibly cheap and incredibly effective. They can also be selective. So you can kill just the kind of people you want to get rid of. And it seems to me that we just don’t need another weapon of mass destruction with all of these extra characteristics. We’ve got rid of to some extent biological and chemical weapons. We’re trying to get rid of nuclear weapons, and introducing another one that’s arguably much worse seems to be a step in the wrong direction.

Steven Pinker: You asked also about the benefits of artificial intelligence, which I think could be stupendous. They include elimination of drudgery and the boring and dangerous jobs that no one really likes to do, like stocking shelves, making beds, mining coal, and picking fruit. There could be a bonanza in automating all the things that humans want done without human pain and labor and boredom and danger. It raises the problem of how we will support the people (if new jobs don’t materialize) who have nothing to do. But that’s a more minor economic problem to solve, compared to the spectacular advance we could have in eliminating human drudgery.

Also, there are a lot of jobs, such as the care of elderly people–lifting them onto toilets, reaching things from upper shelves–that, if automated, would allow more of them to live at home instead of being warehoused in nursing homes. Here, too, the potential for human flourishing is spectacular. And as I mentioned, many kinds of human judgment are so error-prone that they can already be replaced by simple algorithms, and better still if they were more intelligent algorithms. There’s the potential of much less waste, much less error, far fewer accidents. An obvious example is the million and a quarter people killed in traffic accidents each year that could be terrifically reduced if we had autonomous vehicles that were affordable and widespread.

Lucas Perry: A core of this is that all of the problems that humanity faces simply require intelligence to solve them, essentially. And if we’re able to solve the problem of how to make intelligent machines, then our problems will evermore and continuously become automateable by machine systems. So Stuart, do have you have anything else to add here in terms of existential hope and benefits to compliment what Steve just contributed before we pivot into existential risk?

Stuart Russell: Yeah, there is an argument going around, and I think Mark Zuckerberg said it pretty clearly, and Oren Etzioni and various other people have said basically the same thing. And it’s usually put this way, “If you’re against AI, then you’re against better medical decisions, or reducing medical errors, or safer cars,” and so on. And this is, I think, just a ridiculous argument. So first of all, people who are concerned about the risks of AI, are not against AI, right? That’s like arguing if you’re a nuclear engineer and you’re concerned about the possibility of a design flaw that would lead to a meltdown, you’re against electricity. No, you’re not against electricity. You’re just against millions of people dying for no reason, and you want to fix the problem. And the same argument I think is true about those who are concerned about the risk of AI. If AI didn’t have any benefits we wouldn’t be having this discussion at all. No one would be investing any money, no one would have put their lives and careers into working on the capabilities of AI, and the whole point would be moot.

So of course, AI will have benefits, but if you don’t address the risks, you won’t get the benefits, because the technology will be rejected, or we won’t even have a choice to reject it. And if you look at what happened with nuclear power, I think it’s really an object lesson. Nuclear power could and still can produce quite cheap electricity. So I have a house in France and most electricity in France comes from nuclear power, and it’s very cheap and very reliable. And it also doesn’t produce a lot of carbon dioxide, but because of Chernobyl, the nuclear industry has been literally decimated, by which I mean, reduced by a factor of 10, or more. And so we didn’t get the benefits, because we didn’t pay enough attention to the risks. The same holds with AI.

So the benefits of AI in the long run I would argue are pretty unlimited, and medical errors and safer cars, that’s all nice, but that’s a tiny, tiny footnote in what can be done. As Steve already mentioned, the elimination of drudgery and repetitive work. It’s easy for us intellectuals to talk about that. We’ve never really engaged in a whole lot of it, but for most of the human race, for most of recorded history, people with power and money have used everybody else as robots to get what they want. Whether we’ve been using them as military robots, or agricultural robots, or factory robots, we’ve been using people as robots.

And if you had gone back to the early hunter gatherer days and written some science fiction, and you said, “You know what, in the future, people will go into big square buildings, thousands of feet long with no windows and they’ll do the same thing a thousand times a day. And then they’ll go back the next day and do the same thing another thousand times. And they’re going to do that for thousands of days until they’re practically dead.” The audience, the readers of science fiction in 20,000 BC, would have said, “You’re completely nuts, that’s so unrealistic.” But that’s how we did it. And now we’re worried that it’s coming to an end, and it is coming to an end, because we finally have robots that can do the things that we’ve been using human robots to do.

And I’m not saying we should just get rid of those jobs, because jobs have all kinds of purposes in people’s lives. And I’m not a big fan of UBI, which says basically, “Okay, we give up. Humans are useless, so the machines will feed them and house them, entertain them, but that’s all they’re good for.”

Now the benefits to me… It’s hard to imagine, just like we could not imagine very well all the things we would use the internet for. I mean, I remember the Berkeley computer science faculty in the ’80s sitting around at lunch, we knew more about networking than almost anybody else, but we still had absolutely no idea. What was the point of being able to click on a link? What’s that about? We totally blew it.

And we don’t understand all the things that superhuman AI could do for us. I mean, Steve mentioned that we could do much better science, and I agree with that. In the book, I visualize it as taking various ideas, like, “travel as a service,” and extending that to “everything as a service.” So travel as a service is a good example. Like if you think about going to Australia 200 years ago, you’re talking about a billion dollar proposition, probably 10 years, thousands of people, 80% chance of death. Now I take out my cell phone, I go tap, tap, tap, and now I’m in Australia tomorrow. And it’s basically free compared to what it used to be. So that’s what I mean by, as a service, you want something, you just get it.

Superhuman AI could make everything as a service. So think about the things that are expensive and difficult or impossible now, like training a neurosurgeon, or building a railway to connect your rural village to a nearby city so that people can visit, or trade, or whatever. For most of the developing world these things are completely out of reach. The health budget of a lot of countries in Africa is less than $10 per person per year. So the entire health budget of a country would train one neurosurgeon in the US. So these things are out of reach, but if you take out the humans then these services can become effectively free. They become services like travel is today, and that would enable us to bring everyone on earth up to the kind of living standard that they might aspire to. And if we can figure out the resource constraints and so on that will be a wonderful thing.

Lucas Perry: Now that’s quite a beautiful picture of the future. There’s a lot of existential hope there. The other side to existential hope is existential risk. Now this is an interesting subject, which Steve and you, Stuart, I believe have disagreements about. So pivoting into this area, and Steve, you can go first here, do you believe that human beings, should we not go extinct in the meantime, will we build artificial superintelligence? And does that pose an existential risk to humanity?

Steven Pinker: Yeah, I’m on record as being skeptical of that scenario and dubious about the value of putting a lot of effort into worrying about it now. The concept of superintelligence is itself obscure. In a lot of the discussions you could replace the word “superintelligence” with “magic” or “miracle” and the sentence would read the same. You read about an AI system that could duplicate brains in silicon, or solve problems like war in the Middle East, or cure cancer.  It’s just imagining the possibility of a solution and assuming that the ability to bring it about will exist, without laying out what that intelligence would consist of, or what would count as a solution to the problem. 

So I find the concept of superintelligence itself a dubious extrapolation of an unextrapolable continuum, like human-to-animal, or not-so-bright human-to-smart-human. I don’t think there is a power called “intelligence” such that we can compare a squirrel or an octopus to a human and say, “Well, imagine even more of that.” 

I’m also skeptical about the existential risk scenarios. They tend to come in two varieties. One is based on the notion of a will to power: that as soon as you get an intelligent system, it will inevitably want to dominate and exploit. Often the analogy is that we humans have exploited and often extinguished animals because we’re smarter than them, so as soon as there is an artificial system that’s smarter than us, it’ll do to us what we did to the dodos. Or that technologically advanced civilizations, like European colonists and conquistadors subjugated and sometimes wiped out indigenous peoples, so that’s what an AI system might do to us. That’s one variety of this scenario.

I think that scenario confuses intelligence with dominance, based on the fact that in one species, Homo sapiens, they happen to come bundled together, because we came about through natural selection, a competitive process driven by relative success at capturing scarce resources and competing for mates, ultimately with the goal of relative reproductive success. But there’s no reason that a system that is designed to pursue a goal would have as its goal, domination. This goes back to our earlier discussion that the ability to achieve a goal is distinct from what the goal is.

It just so happens that in products of natural selection, the goal was winning in reproductive competition. For an artifact we design, there’s just no reason that would be true. This is sometimes called the orthogonality thesis in discussions of existential risk, although that’s just a fancy-schmancy way of referring to Hume’s distinction between our goals and our intelligence.

Now I know that there is an argument that says, “Wouldn’t any intelligence system have to maximize its own survivability, because if it’s given the goal of X, well, you can’t achieve X if you don’t exist, therefore, as a subgoal to achieving X, you’ve got to maximize your own survival at all costs.” I think that’s fallacious. It’s certainly not true that all complex systems have to work toward their own perpetuation. My iPhone doesn’t take any steps to resist my dropping it into a toilet, or letting it run out of power.

You could imagine if it could be programmed like a child to whine, and to cry, and to refuse to do what it’s told to do as its power level went down. We wouldn’t buy one. And we know in the natural world, there are plenty of living systems that sacrifice their own existence for other goals. When a bee stings you, its barbed stinger is dislodged when the bee escapes, killing the bee, but because the bee is programmed to maximize the survivability of the colony, not itself, it willingly sacrifices itself. So it is not true that by definition an intelligent system has to maximize its own power or survivability.

But the more common existential threat scenario is not a will to power but collateral damage. That if an AI system is given a single goal, what if it relentlessly pursues it without consideration of side effects, including harm to us? There are famous examples that I originally thought were spoofs, but were intended seriously, like giving an AI system the goal of making as many paperclips as possible, and so it converts all available matter into paperclips, including our own bodies (putting aside the fact that we don’t need more efficient paperclip manufacturing than what we already have, and that human bodies are a pretty crummy source of iron for paperclips).

Barely more plausible is the idea that we might give an AI system the goal of curing cancer, and so it will  conscript us as involuntary guinea pigs and induce tumors in all of us, or that we might give it the goal of regulating the level of water behind a dam and it might flood a town because it was never given the goal of not drowning a village. 

The problem with these scenarios is that they’re self-refuting. They assume that an “intelligent” artifact would be designed to implement a single goal, which is not true of even the stupid artifacts that we live with. When we design a car, we don’t just give the goal of going from A to B as fast as possible; we also install brakes and a steering wheel and a muffler and a catalytic converter. A lot of these scenarios seem to presuppose both idiocy on the part of the designers, who would give a system control over the infrastructure of the entire planet without testing it first to see how it worked, and an idiocy on the part of the allegedly intelligent system, which would pursue a single goal regardless of all the other effects. This does not exist in any human artifact, let alone one that claims to be intelligent. Giving an AI system one vaguely worded, sketchy goal, and empowering it with control over the entire infrastructure of the planet without testing it first seems to me just so self-evidently moronic that I don’t worry that engineers have to be warned against it.

I’ve quoted Stuart himself, who in an interview made the point well when he said, “No one talks about building bridges that don’t fall down. They just call it building bridges.” Likewise, AI that avoids idiocies like that is just AI, it’s not AI with extra safeguards. That’s what intelligence consists of.

Let me make one other comment. You could say, well, even if the odds are small, the damage would be so catastrophic that it is worth our concern. But there are downsides to worrying about existential risk. One of them is the possible stigmatization and abandonment of helpful technologies. Stuart mentioned the example of nuclear power. What’s catastrophic is that we don’t roll out nuclear power the way that France did, which would go a long way toward solving the genuinely dangerous problem of climate change. Fear of nuclear power has been irrationally stoked by vivid examples:the fairly trivial accident at Three Mile Island in the United States, which killed no one, the tsunami at Fukushima, where people died in the botched evacuation, not the nuclear accident, and the Soviet bungling at Chernobyl. Even that accident killed a fraction of the people that die every day from the burning of fossil fuels, to say nothing of the likely future harm from climate change. The reaction to Chernobyl is exactly how we should not deal with the dangers facing humanity. 

Genetically modified organisms are another example: a technology overregulated or outlawed out of worst-case fears, depriving us of the spectacular benefits of greater ecological sustainability, human nutrition, and less use of water and pesticides. 

There are other downsides of fretting about exotic hypothetical existential risks. There is a line of reasoning in the existential risk community and the so called Rationality community that goes something like this: since the harm of extinguishing the species is basically infinite, probabilities no longer matter, because by expected utility calculations, if you multiply the tiny risk by the very large number of the potential descendants of humans before the sun expands and kills us off (or in wilder scenarios,  the astronomically larger number of immortal consciousnesses that will exist when we can upload our connectomes to the cloud, or when we colonize and multiply in other solar systems)—well, then even an eensy, eensy, infinitesimal probability of extinction would be catastrophic, and we should worry about it now.

The problem is that that argument could apply to any scenario with a nonzero probability, which means any scenario that is not logically impossible. Should we take steps to prevent the evolution of toxic killer gerbils that will nibble everyone to death? If I say, “That’s preposterous,” you can say, “Well, even if the probability is very, very small, since the harm of extinction is so great, we must devote some brain power to that scenario.”

I do fear the moral hazard of human intellect being absorbed in this free-for-all: that any risk, if you imagine it’s potentially existential, could justify any amount of expenditure, according to this expected utility calculation. The hazard is that smart people, clever enough to grasp a danger that common sense would never conceive of, will be absorbed into what might be a fruitless pursuit, compared to areas where we urgently do need application of human brain power–in climate, in the prevention of nuclear war, in the prevention of pandemics. Those are real risks, which no one denies, and we haven’t solved any of them, together with other massive sources of human misery like Alzheimer’s disease. Given these needs, I wonder whether the infinitesimal-probability-times-infinite-harm is the right way of allocating our intellectual capital.

Lucas Perry: Stuart, you want to react to those points.

Stuart Russell: Yeah, there’s a lot there to react to and I’m tempted to start at the end and work back and just ask, well, if we were spending hundreds of billions of dollars a year to breed billions of toxic killer gerbils, wouldn’t you ask people if that was a good idea before dismissing any reason to be concerned about it? If that’s what we were actually investing in creating. I don’t buy the analogy between AI and toxic killer gerbils in any shape or form. But I will go back to the beginning, and we began by talking about feasibility. And Steve argues, I think, primarily, that it’s not even meaningful, that we could create superhuman levels of intelligence, that there isn’t a single continuum.

And yes, there isn’t a single continuum, but there doesn’t have to be a single continuum. When people say one person is more intelligent than another, or one species is more intelligent than other, it’s not a scientific statement that there is a single scalar on which species one exceeds species two. They’re talking in broad brush. So when we say humans are more intelligent than chimpanzees, that’s probably a reasonable thing to say, but there are clearly dimensions of intelligence where actually chimpanzees are more intelligent than humans. For example, short term memory. A chimpanzee, once they get what a digit is, they can learn 20 digit telephone numbers at the drop of a hat, and humans can’t do that. Clearly there’s dimensions on which chimpanzee intelligence, on average, is probably better than human. But nonetheless, when you look at which species would you rather be right now the chimpanzees don’t have much of a chance against the humans.

I think that there is a meaningful motion of generality of intelligence, and one way to think about it is to take a decision making scenario where we already understand how to produce very effective decisions, and then ask, how is that decision scenario restricted, and what happens when we relax the restrictions and figure out how to maintain the same, let’s say, superhuman quality of decision making? So if you look at Go play, it’s clear that the humans have been left far behind. So it’s not unreasonable to ask, just as the machines wiped the floor with humans on the Go board, and the chess board, and now on the StarCraft board, and lots of other boards, could you take that and transfer that into the real world where we make decisions of all kinds? The difference between the Go board and the real world is pretty dramatic. And that’s why we’ve had lots of success on the Go board and not so much in the real world.

The first thing is that the Go board is fully observable. You can see the entire state of the world that matters. And of course in the real world there’s lots of stuff you don’t see and don’t know. Some of it you can infer by accumulating information over time, what we call state estimation, but that turns out to be quite a difficult problem. Another thing is that we know all the rules of Go, and of course in the real world, you don’t know all the rules, you have to learn a lot as you go along. Another thing about the Go board is that despite the fact that we think of it as really complicated, it’s incredibly simple compared to the real world. At any given time on the Go board there’s a couple of hundred legal moves, and the game lasts for a couple hundred moves.

And if you said, well, what are the analogous primitive actions in the real world for a human being? Well, we have 600 muscles and we can actuate them maybe about 10 times per second each. Your brain probably isn’t able to do that, but physically that’s what could be your action space. And so you actually have then a far greater action space. And you’re also talking about… We often make plans that last for many years, which is literally trillions of primitive actions in terms of muscle actuations. Now we don’t plan those all out in detail, but we function on those kinds of timescales. Those are some of the ways that Go and the real world differ. And what we do in AI is we don’t say, okay, I’ve done Go, now I’m going to work on suicide Go, and now I’m going to work on chess with three queens.

What we try to do is extract the general lessons. Okay, we now understand fairly well how to handle that whole class of problems. Can we relax the assumptions, these basic qualitative assumptions about the nature of the problem? And if you relax all the ones that I listed, and probably a couple more that I’ve got, you’re getting towards systems that can function at a superhuman level in the real world, assuming that you figure out how to deal with all those issues. So just as we find ourself flummoxed by the moves that the AI system makes on the Go board, if you’re a General, and you’re up against an AI system that’s controlling, or coming up with the decision making plans for the other side, you might find yourself flummoxed, that everything you try, the machine has already anticipated and put in place something that will prevent your plan from succeeding. The pace of warfare will be beyond anything humans have ever contemplated, right?

So they won’t even have time to think, just as the Iraqis were not used to the rate of decision making of the US Army in the first Gulf War, and they couldn’t do anything. They were literally paralyzed, and just step by step by step the allied forces were able to take them apart because they couldn’t respond within the timescales that the allied forces were operating.

So it will be kind of like that if you were a human general. If you were a human CEO and your competitor company is organized and run by AI systems, you’d be in the same kind of situation. So it’s entirely conceivable. I’m not necessarily saying plausible, but conceivable that we can create real world decision making capabilities that exceed those of humans across the board. So that this notion of generality, I think it is something that still needs to be worked out. Most definitions of generality that people come up with end up saying, “Well, humans are general because they can do all the things that humans can do,” which is sort of a tautology. But nonetheless, it’s interesting that when you think about all the jobs: doctor, carpenter, advertising, sales, representative, most normally functioning people could do most of those jobs at least to some reasonable level.

So we are incredibly flexible compared to current AI systems. There is progress on achieving generality, but there’s a long way to go. I’m certainly not one of those who says that superintelligent AI is imminent and that’s why we need to worry. And in fact, I’m probably more conservative. If you want to appeal to what most expert AI people think, most expert AI people think that we will have something that’s reasonably described as superintelligent AI sooner than I do.

So most people think sometime in the middle of the century. It turns out that Asian AI researchers particularly in China are more optimistic, so they think 20 years. People in the US and Europe may be more like 40 years. I would be reasonably confident saying by the end of this century.

I think Nick Bostrom is in about the same place. He’s also more conservative than the average expert AI researcher. There are major breakthroughs that have to happen, but the massive investment that’s taking place, the influx of incredibly smart people into the field, these things suggest that those breakthroughs will probably take place but the timescale is very hard to say.

And when we think about the risks, I would say Steve is really putting up one straw man after another and then knocking it down. So for example, the paperclip argument is not a scenario that Nick Bostrom thinks is one of the more likely ways for the human race to end. It’s a philosophical thought experiment intended to illustrate a point. And the point is incontrovertible and I don’t think Steve disagrees with it.

So let’s not use the word intelligent because I think Steve here is using the word intelligent to mean always behaves in whatever way we think we wish that it would behave well.

Of course, if you define intelligence that way, then there isn’t an issue. The question is, how do we create any such thing? And the ways we have right now of creating any such thing fall under the standard model, which I described earlier that we set up, let’s call it a superoptimizer and then we give it an objective. And then off it goes. And he’s (Bostrom) describing what happens when you give a superoptimizer the wrong goal. And he’s not saying, “Yes, of course we should give it wrong goals.”

And he’s using this to illustrate what happens when you give it even what seems to be innocuous. So he’s trying to convey the idea that we are not very good at judging the consequences of seemingly innocuous goals. My example of curing cancer: “Curing cancer? Yeah, of course, that’s a good goal to give to an AI system” — but the point is, if that’s the only goal you give to the AI system, then all these weird things happen because that’s the nature of super optimizers. That’s the nature of the standard model of AI.

And this is, I think, the main point being made is not that no matter what we do AI going to get us. It’s that given our current understanding and given hundreds of billions of dollars are being invested into that current understanding then there is a failure mode and it’s reasonable to point that out just as if you’re a nuclear engineer and you say, “Look, everyone is designing these reactors in this way. All of you are doing this. And look there’s this failure mode.” That’s a reasonable thing to point out.

Steven Pinker: Several reactions. First, while money is pouring into AI, it’s not pouring it into super-optimizers tasked with curing cancer and with the power to kidnap people. And the analogies of humans outcompeting chimpanzees, or American generals outsmarting their Iraqi counterparts, once again assume that systems that are smarter than us will therefore be in competition with us. As for straw men, I was mindful to avoid them: the AI system that would give people tumors to pursue the goal of curing cancer was taken from Stuart’s book.

I agree that a super-optimizer that was given a single goal would be menace. But a super-optimizer that pursued a single goal is self-evidently unintelligent, not superintelligent! 

Stuart Russell: Of course, we have multiple goals. There’s a whole field of multi-attribute utility theory that’s been going now for more than 50 years. Of course, we understand that. When we look at even the design of the algorithms that Uber uses to get you to the airport, they take into account multiple goals.

But the point is the same argument applies if you operate in a standard model when you add in the multiple goals. Unless you’re able to be sure that you have completely and correctly captured all the things we care about under all conceivable and the inconceivable,because I think one of the things about superintelligent AI systems they will come up with, by human standards, inconceivable forms of actions.

We cannot guarantee that. And this is the point. So you could say multiple goals, but multiple goals are just a single goal. They add up to the ability to rank futures. And the question is: is that ability to rank futures fully aligned with what humans want their futures to be like? And the answer is inevitably, no. We are inevitably going to leave things out.

So even if you have a thousand terms in the objective function, there’s probably another million that you ought to have included that you didn’t think about because it never occurred to you.

So for example, you can go out and find lists of important things that human beings care about. This is sort of the whole-values community, human-development community, Maslow hierarchy, all of those things. People do make whole lists of things trying to build up a picture of very roughly what is the human utility function after all.

But invariably, those lists just refer to things that are usually a subject of discussion among humans about “Do we spend money on schools or hospitals?”or whatever it might be. On that list, you will not find the color of the sky because no one, no humans right now are thinking about, “Oh, should we change the sky to be orange with pink stripes?” But if someone did change the color of the sky, I can bet you a lot of people would be really upset about it.

And so invariably we fail to include many, many criteria in whatever list of objectives you might come up with. And when you do that, what happens is that the optimizer will take advantage of those dimensions of freedom and typically, and actually under fairly general algebraic conditions, will set them to extreme values because that gives you better optimization on the things that are in the list of goals.

So the argument is that within the standard model, which I bear some responsibility for, because it’s the way we wrote the first three editions of the textbook, within the standard model, further progress on AI could lead to increasing problems of control and it’s not because there’s any will to dominance.

I don’t know of any serious thinkers in the X-risk community who think that that’s the problem. That’s another straw man.

Steven Pinker: When you’re finished, I do have some responses to that.

Stuart Russell: The argument is not that we automatically build in because we all want our systems to be alpha males or anything like that. And I think Steve Omohundro has put it fairly clearly in some of his earlier papers that the behavior of a superoptimizer given any finite list of goals is going to include efforts to maximize its computational resources and other resources that will help it achieve the objectives that we do specify.

And you could put in something saying, “Well, and don’t spend any money.” Or, “Don’t do this and don’t do that and don’t do the other.” But the same structure of the argument is going to apply. We can reduce the risk by adding more and more stuff into the explicit objectives, but I think the argument I’m making in the book is that that’s just a completely broken way to design AI systems.

The meta argument is that if we don’t talk about the failure modes, we won’t be able to address them. So actually I think that Steve and I don’t disagree about the plausible future evolution. I don’t think it’s particularly plausible. If I was going into forecast mode, so just betting on the future saying, “What’s the probability that this thing will happen or that thing will happen?” I don’t think it’s particularly plausible that we will be destroyed by superintelligent AI.

And there are several reasons why I don’t think that’s going to occur because we would probably get some early warnings of it. And if we couldn’t figure out how to prevent it, we would probably put very strong restrictions on further development or we would figure out how to actually make it provably safe and beneficial.

But you can’t have that discussion unless you talk about the failure modes. Just like in nuclear safety, it’s not against the rules to raise possible failure modes like what if this molten sodium that you’re proposing should flow around all these pipes? What if it ever came into contact with the water that’s on the turbine side of the system? Wouldn’t you have a massive explosion which could rip off the containment and so on? That’s not exactly what happened in Chernobyl, but not so dissimilar.

And of course that’s what they do. So this culture of safety that Steve talks about consist exactly of this. People saying, “Look, if you design things that way these terrible things are going to happen. So don’t design things that way, design things this way.” And this is a process that we are going through in the AI community right now.

And I have to say, I just actually was reading a letter from one of my very senior colleagues, former president of AAAI, who said, “Five years ago, everyone thought Stuart was nuts, but Stuart was right. These risks have to be taken seriously and we all owe him a great debt for bringing it within the AI community so that we can start to address it.”

And I don’t think I invented these risks. And I was just in chance position that I had two years of sabbatical to think about the future of the field and to read some of the things that others had already written about the field from the outside.

My sense is that Steve and I are kind of the glass half full glass half empty. In terms of our forecast, we think on the whole, the weather tomorrow is likely to be sunny. I think we disagree on how to make sure that it’s sunny. I really do think that the problem of creating a provably beneficial AI, by which I mean that no matter how powerful the AI system is, we remain in power. We have power over it forever, that we never lose control.

That’s a big ask and the idea that we could solve that problem without even mentioning it, without even talking about it and without even pointing out why it’s difficult and why it’s important, that’s not the culture of safety. That’s sort of more like the culture of the communist party committee in Chernobyl, that simply continued to assert that nothing bad was happening.

Steven Pinker: Obviously, I’m in favor of the safety mindset of engineering, that is, you test the system before you implement it, you try to anticipate the failure modes. And perhaps I have overestimated the common sense of the AI community and they have to be warned about the absurdity of building a superoptimizer.  But a lot of these examples–flooding a town to control the water level, or curing cancer by turning humans into involuntary Guinea pigs, or maximizing happiness by injecting everyone with a drip of antidepressants–strike me as so far from reasonable failure modes that they’re not part of the ordinary engineering effort to ensure safety–particularly when they are coupled with the term “existential.”

These are not ordinary engineering discussions of ways in which a system could fail; they are speculations on how the human species might end. That is very different from not plugging in an AI system until you’ve tested it to find out how it fails. And perhaps we agree that the superoptimizers in these thought experiments are so unintelligent that no one will actually empower them.

Stuart Russell: But Steve, I wasn’t saying we give it one goal. I’m saying however many goals we give it, that’s equivalent to giving it a ranking over futures. So the idea of single goal versus multiple is a complete red herring.

Steven Pinker: But the scare stories all involve systems that are given a single goal. As you go down the tail of possible risks, you’re getting into potentially infinitesimal risks. There is no system, conceivable or existing, that will have zero risk of every possibility. 

Stuart Russell: If we could do that, if we had some serious theory by which we could say, “Okay. We’ve got within epsilon of the true human ranking over futures,” I think that’s very hard to do. We literally do not have a clue how to do that. And the purpose of these examples is actually to dismiss the idea that this has a simple solution.

So people want to dismiss the idea of risk by saying, “Oh, we’ll just give the AI system such and such objective.” And then the failure mode goes away and everything’s cool. And then people say, “Oh, but no.” Look, if you give it the objective that everyone should be happy, then here’s a solution that the AI system could find that clearly we wouldn’t want.

Those processes lead actually to deeper questioning, what do we really mean by happy? We don’t just mean pleasure as measured by the pleasure center in the brain. And the same arguments happen in moral philosophy.

So no one is accusing G.E. Moore of being a naive idiot because he objected to a pleasure maximization definition of what is a good moral decision to make. He was making an important philosophical point and I don’t think we should dismiss that same point when it’s made in the context of designing objectives for AI systems.

Steven Pinker: Yes, that’s an excellent argument against building a universally empowered AI system that’s given the single goal of maximizing human happiness–your example. Do AI researchers need to be warned against that absurd project? It seems to me that that’s the straw man, and so are the other scenarios that are designed to sow worry, such as conscripting the entire human race as involuntary guinea pigs in cancer experiments. Even if there isn’t an epsilon that we can’t go below in laying out possible risks, it doesn’t strike me that that’s within the epsilon.

These strike me more as exercises of human imagination. Assuming a ridiculously simple system that’s given one goal, what could go wrong? Well, yeah, stuff could go wrong, but is that really what’s going to face us when it comes to actual AI systems that have some hope of being implemented?

Naturally, we ought to test the living daylights out of any system before we give it control over anything. That’s Stuart’s point about building bridges that don’t fall down and the standard safety ethic in engineering. But I’m not sure that exotic scenarios based on incredibly stupid ideas for AI systems like giving one the goal of maximizing human happiness is the route that gives us safe AI.

Stuart Russell: Okay. So let me say once again that the one goal versus multiple goals is a red herring. If you think it’s so easy to specify the goal correctly, perhaps your next paper will write it out. Then we’ll say, “Okay, that’s not a straw man. This is Steve Pinker’s suggestion of what the objective should be for the superintelligent AI system.” And then the people who love doing these things, probably Nick Bostrom and others will find ways of failing.

So the idea that we could just test before deploying something that is significantly more powerful than human beings or even the human race combined, that’s a pretty optimistic idea. We’re not even able to test ordinary software systems right now. So test generation is one of the effective methods used in software engineering, but it has many, many known failure cases for real world examples, including multiplication.

Intel’s Pentium chip was tested with billions of examples of multiplication, but it failed to uncover a bug in the multiplication circuitry, which caused it to produce incorrect results in some cases. And so we have a technology of formal verification, which would have uncovered that error, but particularly in the US there’s a culture that’s somewhat opposed to using formal verification in software design.

Less so in hardware design nowadays, partly because of the Pentium error, but still in software, formal verification is considered very difficult and very European and not something we do. And this is far harder than that because software verification typically is thinking only about correctness of the software in an internal sense, that what happens inside the algorithm between the inputs and outputs meet some specifications.

What we want here is that the combination of the algorithm and the world evolves in ways that we are certain to be pleased about. And that’s a much harder kind of thing. Control theory has that view of what they mean by verification. And they’re able to do very simple linear quadratic regulators and a few other examples. And beyond that, they get stuck. And so I actually think that the testing is probably neither (not very) feasible. I mean, not saying we shouldn’t do it, but it’s going to be extremely hard to get any kind of confidence from testing, because you’re really asking, can you simulate the entire world and all the ways a system could use the world to bring about the objective.

However complicated and however multifaceted that objective is, it’s probably going to be the wrong one. So I’ve proposed a, not completely different, but a generalized form of AI that knows that it doesn’t know what the real objectives are. It knows it doesn’t know how humans rank possible futures and that changes the way it behaves, but that also has failure modes.

One of them being the plasticity of human preference rankings over the future and how do you prevent the AI system from taking advantage of that plasticity? You can’t prevent it completely because anything it does is going to have some effect on human preferences. But the question is what constitutes reasonable modifications of human preferences and what constitutes unreasonable ones? We don’t know the answer to that. So there are many, many really difficult research problems that we have to overcome for the research agenda that I’m proposing to have a chance of success.

I’m not that optimistic that this is an easy or a straightforward problem to solve and I think we can only solve it if we go outside the conceptual framework that AI has worked in for the last 70 years.

Steven Pinker: Well, yes. Certainly, if the conceptual framework for AI is optimizing some single or small list of generic goals, like a ranking over possible futures, and it is empowered to pursue them by any means, as opposed to building tools that solve specific problems. But note that you’ve also given arguments why the fantasies of superintelligence are unlikely to come about–the near-miraculous powers to outsmart us, to augment its own intelligence, to defeat all of our attempts to control it. In the scenarios, these all work flawlessly–yet  the complexities that make it hard to predict all conceivable failures also make it hard to achieve superintelligence in the first place.

Namely, we can’t take into account the fantastically chaotic and unpredictable reactions of humans. And we can’t program a system that has complete knowledge of the physical universe without allowing it to do experiments and acquire empirical knowledge, at a rate determined by the physical world. Exactly the infirmities that prevent us from exploring the entire space of behavior of one of these systems in advance is the reason that it’s not going to be superintelligent in the way that these scenarios outline.

And that’s a reason not to empower any generic goal-driven system that aspires toward “superintelligence” or that we might think of as “superintelligent”–it  is unlikely to exist, and likely to display various forms of error and stupidity.

Stuart Russell: I would agree that some of the concerns that you might see in the X-risk community are, say, nonphysical. So the idea that a system could predict the next hundred years and your entire life in such detail that a hundred years ago, it knew what you were going to be saying at a particular millisecond in a hundred years in the future, this is obviously complete nonsense.

I don’t think we need to be too concerned about that as a serious question. Whether it’s a thought experiment that sheds light on fundamental questions in decision theory, like the Newcomb problems is another issue that we don’t have to get into. But we can’t solve the problem by saying, “Well, superintelligence of the kind that could lead to significant global consequences could not possibly exist.”

And actually I kind of like Danny Hillis’ argument, which says that actually, no, it already does exist and it already has, and is having significant global consequences. And his example is to view, let’s say the fossil fuel industry as if it were an AI system. I think this is an interesting line of thought, because what he’s saying basically and — other people have said similar things — is that you should think of a corporation as if it’s an algorithm and it’s maximizing a poorly designed objective, which you might say is some discounted stream of quarterly profits or whatever. And it really is doing it in a way that’s oblivious to lots of other concerns of the human race. And it has outwitted the rest of the human race.

So we might all think, well, of course, we know that what it’s trying to do is wrong and of course we all know the right answer, but in fact we’ve lost and we should have pointed out a hundred years ago that there is this risk and it needs to be taken seriously.

And it was. People did point it out a hundred years ago, but no one took them seriously. And this is what happened. So I think we have actually a fairly good example that this type of thing, the optimization of objectives, ignoring externalities as the economists would point out, by superintelligent entities. And in some sense, the fossil fuel industry outwitted us because whatever organizational structures allow large groups of humans to generate effective complex behaviors in the real world and develop complex plans, it operates in some ways like a superintelligent entity, just like we were able to put a person on the moon because of the combined effect of many human intellects working together.

But each of those humans in the fossil fuel industry is a piece of an algorithm if you like and their own individual preferences about the future don’t count for much and in fact, they get molded by their role within the corporation.

I think in some ways you already have an existence proof that the concern is real.

Steven Pinker: A simpler explanation is that people like energy, fossil fuels are the most convenient source, and no one has had to pay for the external damage they do. Clearly we ought to anticipate foreseeable risks and attempt to mitigate them. But they have to be calibrated against what we know, taking into account our own ignorance of the future. It can be hazardous to chase the wrong worries, such as running out of petroleum, which was the big worry in the 1970s. Now we know that the problem with petroleum is too much, not too little. Overpopulation and genetically modified organisms are other examples. 

If we try to fantasize too far into the future, beyond what we can reasonably predict, we can sow fear about the wrong risks. My concern about all these centers and smart people worrying about the existential risk of AI is that we are misallocating our worry budget and our intellectual resources. We should be thinking hard about how to mitigate climate change, which is a real problem. That is less true of spinning exotic scenarios about hypothetical AI systems which have been given control over the physical universe and might enslave us in cancer experiments.

Lucas Perry: All right. So wrapping up here, do you guys have final statements that you’d like to say, just if you felt like what you just said didn’t fully capture what you want to end on on this issue of AI existential risk?

Steven Pinker: Despite our disagreements, most about my assessment of AI agrees with Stuart’s. I personally don’t think that the adjective existential is helpful in ordinary concerns over safety, which we ought to have. I think there are tremendous potential benefits to AI, and that we ought to seek at the same time as we anticipate the reasonable risks and take every effort to mitigate them.

Stuart Russell: Yep. I mean, it’s hard to disagree that we should focus on the reasonable risks. The question is whether you think that the hundreds of billions of dollars that are being invested into AI research will produce systems that can have potentially global consequences.

And to me it seems self evident that it can and we can look at even simple machine algorithms like the content selection algorithms in social media because those algorithms interact with humans for hours every day and dictate what literally billions of people see and read every day. They are having substantial global impact already.

And they are very, very simple. They don’t know that human beings exist at all, but they still learn to manipulate our brains to optimize the objective. I had a very interesting little Facebook exchange with Yann LeCun. And at some point in the argument, Yann said something quite similar to something Steve said earlier. He said, “There’s really no risk. You’d have to be extremely stupid to put an incorrect objective into a powerful system and then deploy it on a global scale.”

And I said, “Well, you mean like optimizing click-through?” And he said, “Facebook stopped using click-through years ago.” And I said, “Well, why was that?” And he said, “Oh, because it was the incorrect objective.”

So you did put an incorrect objective into a powerful system, deployed on a global scale. Now what does that say about Facebook? So I think just as you might have said — and in fact the nuclear industry did say — “It’s perfectly safe. Nothing can go wrong. We’re the experts. We understand safety. We understand everything.”

Nonetheless, we had Chernobyl, we had Fukushima. And actually, I think there’s an argument to be made that despite the massive environmental cost of foregoing nuclear power, that countries like Germany, Italy, Spain and probably a bunch of others are in the process of actually deciding that we need to phase out nuclear power because even though theoretically, it’s possible to develop and operate completely safe nuclear power systems, it’s beyond our capabilities and the evidence is there.

You might have argued that while Russia is corrupt, it’s technology was not as great as it should have been, they cut lots of corners, but you can’t argue that the Japanese nuclear industry was unsophisticated or unconcerned with safety, but they still failed. And so I think voters in those countries who said, “We don’t want nuclear power because we just don’t want to be in that situation even if we have the engineers making our best efforts.”

These kinds of considerations suggest that we do need to pay very careful attention. I’m not saying we should stop working on climate change, but when we invented synthetic biology, we said okay, we’d better think about how do we prevent the creation of disease or new disease organisms that could produce pandemics. And we took steps. People spent a lot of time thinking about safety mechanisms for those devices. We have to do the same thing for AI.

Lucas Perry: All right. Stuart and Steven, thanks so much. I’ve learned a ton of stuff today. If listeners want to follow you or look into your work, where’s the best place to do that? I’ll start with you, Steven.

Steven Pinker: Stevenpinker.com, which has pages for ten books, including the most recent, Enlightenment Now. SAPinker on Twitter. 

Lucas Perry: And Stuart.

Stuart Russell: So you can Google me. I don’t really have a website or social media activity, but the book Human Compatible, which was published last October by Viking in the US and Penguin in the UK and it’s being translated into lots of languages, I think that captures my views pretty well.

Lucas Perry: All right. Thanks so much for coming on. And yeah, it was a pleasure speaking.

Steven Pinker: Thanks very much, Lucas for hosting it. Thank you Stuart for the dialogue.

Stuart Russell:It was great fun, Steve. I look forward to doing it again.

Steven Pinker: Me too.

End of recorded material

AI Alignment Podcast: An Overview of Technical AI Alignment in 2018 and 2019 with Buck Shlegeris and Rohin Shah

 Topics discussed in this episode include:

  • Rohin’s and Buck’s optimism and pessimism about different approaches to aligned AI
  • Traditional arguments for AI as an x-risk
  • Modeling agents as expected utility maximizers
  • Ambitious value learning and specification learning/narrow value learning
  • Agency and optimization
  • Robustness
  • Scaling to superhuman abilities
  • Universality
  • Impact regularization
  • Causal models, oracles, and decision theory
  • Discontinuous and continuous takeoff scenarios
  • Probability of AI-induced existential risk
  • Timelines for AGI
  • Information hazards

Timestamps: 

0:00 Intro

3:48 Traditional arguments for AI as an existential risk

5:40 What is AI alignment?

7:30 Back to a basic analysis of AI as an existential risk

18:25 Can we model agents in ways other than as expected utility maximizers?

19:34 Is it skillful to try and model human preferences as a utility function?

27:09 Suggestions for alternatives to modeling humans with utility functions

40:30 Agency and optimization

45:55 Embedded decision theory

48:30 More on value learning

49:58 What is robustness and why does it matter?

01:13:00 Scaling to superhuman abilities

01:26:13 Universality

01:33:40 Impact regularization

01:40:34 Causal models, oracles, and decision theory

01:43:05 Forecasting as well as discontinuous and continuous takeoff scenarios

01:53:18 What is the probability of AI-induced existential risk?

02:00:53 Likelihood of continuous and discontinuous take off scenarios

02:08:08 What would you both do if you had more power and resources?

02:12:38 AI timelines

02:14:00 Information hazards

02:19:19 Where to follow Buck and Rohin and learn more

 

Works referenced: 

AI Alignment 2018-19 Review

Takeoff Speeds by Paul Christiano

Discontinuous progress investigation by AI Impacts

An Overview of Technical AI Alignment with Rohin Shah (Part 1)

An Overview of Technical AI Alignment with Rohin Shah (Part 2)

Alignment Newsletter

Intelligence Explosion Microeconomics

AI Alignment: Why It’s Hard and Where to Start

AI Risk for Computer Scientists

 

We hope that you will continue to join in the conversations by following us or subscribing to our podcasts on Youtube, Spotify, SoundCloud, iTunes, Google Play, StitcheriHeartRadio, or your preferred podcast site/application. You can find all the AI Alignment Podcasts here.

You can listen to the podcast above or read the transcript below. 

Note: The following transcript has been edited for style and clarity.

 

Lucas Perry: Welcome to the AI Alignment Podcast. I’m Lucas Perry. Today we have a special episode with Buck Shlegeris and Rohin Shah that serves as a review of progress in technical AI alignment over 2018 and 2019. This episode serves as an awesome birds eye view of the varying focus areas of technical AI alignment research and also helps to develop a sense of the field. I found this conversation to be super valuable for helping me to better understand the state and current trajectory of technical AI alignment research. This podcast covers traditional arguments for AI as an x-risk, what AI alignment is, the modeling of agents as expected utility maximizers, iterated distillation and amplification, AI safety via debate, agency and optimization, value learning, robustness, scaling to superhuman abilities, and more. The structure of this podcast is based on Rohin’s AI Alignment Forum post titled AI Alignment 2018-19 Review. That post is an excellent resource to take a look at in addition to this podcast. Rohin also had a conversation with us about just a year ago titled An Overview of Technical AI Alignment with Rohin shah. This episode serves as a follow up to that overview and as an update to what’s been going on in the field. You can find a link for it on the page for this episode.  

Buck Shlegeris is a researcher at the Machine Intelligence Research Institute. He tries to work to make the future good for sentient beings and currently believes that working on existential risk from artificial intelligence is the best way of doing this. Buck worked as a software engineer at PayPal before joining MIRI, and was the first employee at Triplebyte. He previously studied at the Australian National University, majoring in CS and minoring in math and physics, and he has presented work on data structure synthesis at industry conferences.

Rohin Shah is a 6th year PhD student in Computer Science at the Center for Human-Compatible AI at UC Berkeley. He is involved in Effective Altruism and was the co-president of EA UC Berkeley for 2015-16 and ran EA UW during 2016-2017. Out of concern for animal welfare, Rohin is almost vegan because of the intense suffering on factory farms. He is interested in AI, machine learning, programming languages, complexity theory, algorithms, security, and quantum computing to name a few. Rohin’s research focuses on building safe and aligned AI systems that pursue the objectives their users intend them to pursue, rather than the objectives that were literally specified. He also publishes the Alignment Newsletter, which summarizes work relevant to AI alignment. The Alignment Newsletter is something I highly recommend that you follow in addition to this podcast.  

And with that, let’s get into our review of AI alignment with Rohin Shah and Buck Shlegeris.

To get things started here, the plan is to go through Rohin’s post on the Alignment Forum about AI Alignment 2018 and 2019 In Review. We’ll be using this as a way of structuring this conversation and as a way of moving methodically through things that have changed or updated in 2018 and 2019, and to use those as a place for conversation. So then, Rohin, you can start us off by going through this document. Let’s start at the beginning, and we’ll move through sequentially and jump in where necessary or where there is interest.

Rohin Shah: Sure, that sounds good. I think I started this out by talking about this basic analysis of AI risk that’s been happening for the last couple of years. In particular, you have these traditional arguments, so maybe I’ll just talk about the traditional argument first, which basically says that the AI systems that we’re going to build are going to be powerful optimizers. When you optimize something, you tend to get these sort of edge case outcomes, these extreme outcomes that are a little hard to predict ahead of time.

You can’t just rely on tests with less powerful systems in order to predict what will happen, and so you can’t rely on your normal common sense reasoning in order to deal with this. In particular, powerful AI systems are probably going to look like expected utility maximizers due to various coherence arguments, like the Von Neumann–Morgenstern rationality theorem, and these expected utility maximizers have convergent instrumental sub-goals, like not wanting to be switched off because then they can’t achieve their goal, and wanting to accumulate a lot of power and resources.

The standard argument goes, because AI systems are going to be built this way, they will have these convergent instrumental sub-goals. This makes them dangerous because they will be pursuing goals that we don’t want.

Lucas Perry: Before we continue too much deeper into this, I’d want to actually start off with a really simple question for both of you. What is AI alignment?

Rohin Shah: Different people mean different things by it. When I use the word alignment, I’m usually talking about what has been more specifically called intent alignment, which is basically aiming for the property that the AI system is trying to do what you want. It’s trying to help you. Possibly it doesn’t know exactly how to best help you, and it might make some mistakes in the process of trying to help you, but really what it’s trying to do is to help you.

Buck Shlegeris: The way I would say what I mean by AI alignment, I guess I would step back a little bit, and think about why it is that I care about this question at all. I think that the fundamental fact which has me interested in anything about powerful AI systems of the future is that I think they’ll be a big deal in some way or another. And when I ask myself the question “what are the kinds of things that could be problems about how these really powerful AI systems work or affect the world”, one of the things which feels like a problem is that, we might not  know how to apply these systems reliably to the kinds of problems which we care about, and so by default humanity will end up applying them in ways that lead to really bad outcomes. And so I guess, from that perspective, when I think about AI alignment, I think about trying to make ways of building AI systems such that we can apply them to tasks that are valuable, such that that they’ll reliably pursue those tasks instead of doing something else which is really dangerous and bad.

I’m fine with intent alignment as the focus. I kind of agree with, for instance, Paul Christiano, that it’s not my problem if my AI system incompetently kills everyone, that’s the capability’s people’s problem. I just want to make the system so it’s trying to cause good outcomes.

Lucas Perry: Both of these understandings of what it means to build beneficial AI or aligned AI systems can take us back to what Rohin was just talking about, where there’s this basic analysis of AI risk, about AI as powerful optimizers and the associated risks there. With that framing and those definitions, Rohin, can you take us back into this basic analysis of AI risk?

Rohin Shah: Sure. The traditional argument looks like AI systems are going to be goal-directed. If you expect that your AI system is going to be goal-directed, and that goal is not the one that humans care about, then it’s going to be dangerous because it’s going to try to gain power and resources with which to achieve its goal.

If the humans tried to turn it off, it’s going to say, “No, don’t do that,” and it’s going to try to take actions that avoid that. So it pits the AI and the humans in an adversarial game with each other, and you ideally don’t want to be fighting against a superintelligent AI system. That seems bad.

Buck Shlegeris: I feel like Rohin is to some extent setting this up in a way that he’s then going to argue is wrong, which I think is kind of unfair. In particular, Rohin, I think you’re making these points about VNM theorems and stuff to set up the fact that it seems like these arguments don’t actually work. I feel that this makes it kind of unfairly sound like the earlier AI alignment arguments are wrong. I think this is an incredibly important question, of whether early arguments about the importance of AI safety were quite flawed. My impression is that overall the early arguments about AI safety were pretty good. And I think it’s a very interesting question whether this is in fact true. And I’d be interested in arguing about it, but I think it’s the kind of thing that ought to be argued about explicitly.

Rohin Shah: Yeah, sure.

Buck Shlegeris: And I get that you were kind of saying it narratively, so this is only a minor complaint. It’s a thing I wanted to note.

Rohin Shah: I think my position on that question of “how good were the early AI risk arguments,” probably people’s internal beliefs were good as to why AI was supposed to be risky, and the things they wrote down were not very good. Some things were good and some things weren’t. I think Intelligence Explosion Microeconomics was good. I think AI Alignment: Why It’s Hard and Where to Start, was misleading.

Buck Shlegeris: I think I agree with your sense that people probably had a lot of reasonable beliefs but that the written arguments seem flawed. I think another thing that’s true is that random people like me who were on LessWrong in 2012 or something, ended up having a lot of really stupid beliefs about AI alignment, which I think isn’t really the fault of the people who were thinking about it the best, but is maybe sociologically interesting.

Rohin Shah: Yes, that seems plausible to me. Don’t have a strong opinion on it.

Lucas Perry: To provide a little bit of framing here and better analysis of basic AI x-risk arguments, can you list what the starting arguments for AI risk were?

Rohin Shah: I think I am reasonably well portraying what the written arguments were. Underlying arguments that people probably had would be something more like, “Well, it sure seems like if you want to do useful things in the world, you need to have AI systems that are pursuing goals.” If you have something that’s more like tool AI, like Google Maps, that system is going to be good at the one thing it was designed to do, but it’s not going to be able to learn and then apply its knowledge to new tasks autonomously. It sure seems like if you want to do really powerful things in the world, like run companies or make policies, you probably do need AI systems that are constantly learning about their world and applying their knowledge in order to come up with new ways to do things.

In the history of human thought, we just don’t seem to know of a way to cause that to happen except by putting goals in systems, and so probably AI systems are going to be goal-directed. And one way you can formalize goal-directedness is by thinking about expected utility maximizers, and people did a bunch of formal analysis of that. Mostly going to ignore it because I think you can just say all the same thing with the idea of pursuing goals and it’s all fine.

Buck Shlegeris: I think one important clarification to that, is you were saying the reason that tool AIs aren’t just the whole story of what happens with AI is that you can’t apply it to all problems. I think another important element is that people back then, and I now, believe that if you want to build a really good tool, you’re probably going to end up wanting to structure that as an agent internally. And even if you aren’t trying to structure it as an agent, if you’re just searching over lots of different programs implicitly, perhaps by training a really large recurrent policy, you’re going to end up finding something agent shaped.

Rohin Shah: I don’t disagree with any of that. I think we were using the words tool AI differently.

Buck Shlegeris: Okay.

Rohin Shah: In my mind, if we’re talking about tool AI, we’re imagining a pretty restricted action space where no matter what actions in this action space are taken, with high probability, nothing bad is going to happen. And you’ll search within that action space, but you don’t go to arbitrary action in the real world or something like that. This is what makes tool AI hard to apply to all problems.

Buck Shlegeris: I would have thought that’s a pretty non-standard use of the term tool AI.

Rohin Shah: Possibly.

Buck Shlegeris: In particular, I would have thought that restricting the action space enough that you’re safe, regardless of how much it wants to hurt you, seems kind of non-standard.

Rohin Shah: Yes. I have never really liked the concept of tool AI very much, so I kind of just want to move on.

Lucas Perry: Hey, It’s post-podcast Lucas here. I just want to highlight here a little bit of clarification that Rohin was interested in adding, which is that he thinks that “tool AI evokes a sense of many different properties that he doesn’t know which properties most people are  usually thinking about and as a result he prefers not to use the phrase tool AI. And instead would like to use more precise terminology. He doesn’t necessarily feel though that the concepts underlying tool AI are useless.” So let’s tie things a bit back to these basic arguments for x-risk that many people are familiar with, that have to do with convergent instrumental sub-goals and the difficulty of specifying and aligning systems with our goals and what we actually care about in our preference hierarchies.

One of the things here that Buck was seeming to bring up, he was saying that you may have been narratively setting up the Von Neumann–Morgenstern theorem, which sets up AIs as expected utility maximizers, and that you are going to argue that that argument, which is sort of the formalization of these earlier AI risk arguments, that that is less convincing to you now than it was before, but Buck still thinks that these arguments are strong. Could you unpack this a little bit more or am I getting this right?

Rohin Shah: To be clear, I also agree with Buck, that the spirit of the original arguments does seem correct, though, there are people who disagree with both of us about that. Basically, the VNM theorem roughly says, if you have preferences over a set of outcomes, and you satisfy some pretty intuitive axioms about how you make decisions, then you can represent your preferences using a utility function such that your decisions will always be, choose the action that maximizes the expected utility. This is, at least in writing, given as a reason to expect that AI systems would be maximizing expected utility. The thing is, when you talk about AI systems that are acting in the real world, they’re just selecting a universe history, if you will. Any observed behavior is compatible with the maximization of some utility function. Utility functions are a really, really broad class of things when you apply it to choosing from universe histories.

Buck Shlegeris: An intuitive example of this: suppose that you see that every day I walk home from work in a really inefficient way. It’s impossible to know whether I’m doing that because I happened to really like that path. For any sequence of actions that I take, there’s some utility functions such that that was the optimal sequence of actions. And so we don’t actually learn anything about how my policy is constrained based on the fact that I’m an expected utility maximizer.

Lucas Perry: Right. If I only had access to your behavior and not your insides.

Rohin Shah: Yeah, exactly. If you have a robot twitching forever, that’s all it does, there is a utility function over a universe history that says that is the optimal thing to do. Every time the robot twitches to the right, it’s like, yeah, the thing that was optimal to do at that moment in time was twitching to the right. If at some point somebody takes a hammer and smashes the robot and it breaks, then the utility function that corresponds to that being optimal is like, yeah, that was the exact right moment to break down.

If you have these pathologically complex utility functions as possibilities, every behavior is compatible with maximizing expected utility, you might want to say something like, probably we’ll have the simple utility maximizers, but that’s a pretty strong assumption, and you’d need to justify it somehow. And the VNM theorem wouldn’t let you do that.

Lucas Perry: So is the problem here that you’re unable to fully extract human preference hierarchies from human behavior?

Rohin Shah: Well, you’re unable to extract agent preferences from agent behavior. You can see any agent behavior and you can rationalize it as expected utility maximization, but it’s not very useful. Doesn’t give you predictive power.

Buck Shlegeris: I just want to have my go at saying this argument in three sentences. Once upon a time, people said that because all rational systems act like they’re maximizing an expected utility function, we should expect them to have various behaviors like trying to maximize the amount of power they have. But every set of actions that you could take is consistent with being an expected utility maximizer, therefore you can’t use the fact that something is an expected utility maximizer in order to argue that it will have a particular set of behaviors, without making a bunch of additional arguments. And I basically think that I was wrong to be persuaded by the naive argument that Rohin was describing, which just goes directly from rational things are expected utility maximizers, to therefore rational things are power maximizing.

Rohin Shah: To be clear, this was the thing I also believed. The main reason I wrote the post that argued against it was because I spent half a year under the delusion that this was a valid argument.

Lucas Perry: Just for my understanding here, the view is that because any behavior, any agent from the outside can be understood as being an expected utility maximizer, that there are behaviors that clearly do not do instrumental sub-goal things, like maximize power and resources, yet those things can still be viewed as expected utility maximizers from the outside. So additional arguments are required for why expected utility maximizers do instrumental sub-goal things, which are AI risky.

Rohin Shah: Yeah, that’s exactly right.

Lucas Perry: Okay. What else is on offer other than expected utility maximizers? You guys talked about comprehensive AI services might be one. Are there other formal agentive classes of ‘thing that is not an expected utility maximizer but still has goals?’

Rohin Shah: A formalism for that? I think some people like John Wentworth is for example, thinking about markets as a model of agency. Some people like to think of multi-agent groups together leading to an emergent agency and want to model human minds this way. How formal are these? Not that formal yet.

Buck Shlegeris: I don’t think there’s anything which is competitively popular with expected utility maximization as the framework for thinking about this stuff.

Rohin Shah: Oh yes, certainly not. Expected utility maximization is used everywhere. Nothing else comes anywhere close.

Lucas Perry: So there’s been this complete focus on utility functions and representing the human utility function, whatever that means. Do you guys think that this is going to continue to be the primary way of thinking about and modeling human preference hierarchies? How much does it actually relate to human preference hierarchies? I’m wondering if it might just be substantially different in some way.

Buck Shlegeris: Me and Rohin are going to disagree about this. I think that trying to model human preferences as a utility function is really dumb and bad and will not help you do things that are useful. I don’t know; If I want to make an AI that’s incredibly good at recommending me movies that I’m going to like, some kind of value learning thing where it tries to learn my utility function over movies is plausibly a good idea. Even things where I’m trying to use an AI system as a receptionist, I can imagine value learning being a good idea.

But I feel extremely pessimistic about more ambitious value learning kinds of things, where I try to, for example, have an AI system which learns human preferences and then acts in large scale ways in the world. I basically feel pretty pessimistic about every alignment strategy which goes via that kind of a route. I feel much better about either trying to not use AI systems for problems where you have to think about large scale human preferences, or having an AI system which does something more like modeling what humans would say in response to various questions and then using that directly instead of trying to get a value function out of it.

Rohin Shah: Yeah. Funnily enough, I was going to start off by saying I think Buck and I are going to agree on this.

Buck Shlegeris: Oh.

Rohin Shah: And I think I mostly agree with the things that you said. The thing I was going to say was I feel pretty pessimistic about trying to model the normative underlying human values, where you have to get things like population ethics right, and what to do with the possibility of infinite value. How do you deal with fanaticism? What’s up with moral uncertainty? I feel pretty pessimistic about any sort of scheme that involves figuring that out before developing human-level AI systems.

There’s a related concept which is also called value learning, which I would prefer to be called something else, but I feel like the name’s locked in now. In my sequence, I called it narrow value learning, but even that feels bad. Maybe at least for this podcast we could call it specification learning, which is sort of more like the tasks Buck mentioned, like if you want to learn preferences over movies, representing that using a utility function seems fine.

Lucas Perry: Like superficial preferences?

Rohin Shah: Sure. I usually think of it as you have in mind a task that you want your AI system to do, and now you have to get your AI system to reliably do it. It’s unclear whether this should even be called a value learning at this point. Maybe it’s just the entire alignment problem. But techniques like inverse reinforcement learning, preference learning, learning from corrections, inverse reward design where you learn from a proxy reward, all of these are more trying to do the thing where you have a set of behaviors in mind, and you want to communicate that to the agent.

Buck Shlegeris: The way that I’ve been thinking about how optimistic I should be about value learning or specification learning recently has been that I suspect that at the point where AI is human level, by default we’ll have value learning which is about at human level. We’re about as good at giving AI systems information about our preferences that it can do stuff with as we are giving other humans information about our preferences that we can do stuff with. And when I imagine hiring someone to recommend music to me, I feel like there are probably music nerds who could do a pretty good job of looking at my Spotify history, and recommending bands that I’d like if they spent a week on it. I feel a lot more pessimistic about being able to talk to a philosopher for a week, and then them answer hard questions about my preferences, especially if they didn’t have the advantage of already being humans themselves.

Rohin Shah: Yep. That seems right.

Buck Shlegeris: So maybe that’s how I would separate out the specification learning stuff that I feel optimistic about from the more ambitious value learning stuff that I feel pretty pessimistic about.

Rohin Shah: I do want to note that I collated a bunch of stuff arguing against ambitious value learning. If I had to make a case for optimism about even that approach, it would look more like, “Under the value learning approach, it seems possible with uncertainty over rewards, values, preferences, whatever you want to call them to get an AI system such that you actually are able to change it, because it would reason that if you’re trying to change it, well then that means something about it is currently not good for helping you and so it would be better to let itself be changed. I’m not very convinced by this argument.”

Buck Shlegeris: I feel like if you try to write down four different utility functions that the agent is uncertain between, I think it’s just actually really hard for me to imagine concrete scenarios where the AI is corrigible as a result of its uncertainty over utility functions. Imagine the AI system thinks that you’re going to switch it off and replace it with an AI system which has a different method of inferring values from your actions and your words. It’s not going to want to let you do that, because its utility function is to have the world be the way that is expressed by your utility function as estimated the way that it approximates utility functions. And so being replaced by a thing which estimates utility functions or infers utility functions some other way means that it’s very unlikely to get what it actually wants, and other arguments like this. I’m not sure if these are super old arguments that you’re five levels of counter-arguments to.

Rohin Shah: I definitely know this argument. I think the problem of fully updated deference is what I would normally point to as representing this general class of claims and I think it’s a good counter argument. When I actually think about this, I sort of start getting confused about what it means for an AI system to terminally value the final output of what its value learning system would do. It feels like some additional notion of how the AI chooses actions has been posited, that hasn’t actually been captured in the model and so I feel fairly uncertain about all of these arguments and kind of want to defer to the future. 

Buck Shlegeris: I think the thing that I’m describing is just what happens if you read the algorithm literally. Like, if you read the value learning algorithm literally, it has this notion of the AI system wants to maximize the human’s actual utility function.

Rohin Shah: For an optimal agent playing a CIRL (cooperative inverse reinforcement learning) game, I agree with your argument. If you take optimality as defined in the cooperative inverse reinforcement learning paper and it’s playing over a long period of time, then yes, it’s definitely going to prefer to keep itself in charge rather than a different AI system that would infer values in a different way.

Lucas Perry: It seems like so far utility functions are the best way of trying to get an understanding of what human beings care about and value and have preferences over, you guys are bringing up all of the difficult intricacies with trying to understand and model human preferences as utility functions. One of the things that you also bring up here, Rohin, in your review, is the risk of lock-in, which may require us to solve hard philosophical problems before the development of AGI. That has something to do with ambitious value learning, which would be like learning the one true human utility function which probably just doesn’t exist.

Buck Shlegeris: I think I want to object to a little bit of your framing there. My stance on utility functions of humans isn’t that there are a bunch of complicated subtleties on top, it’s that modeling humans with utility functions is just a really sad state to be in. If your alignment strategy involves positing that humans behave as expected utility maximizers, I am very pessimistic about it working in the short term, and I just think that we should be trying to completely avoid anything which does that. It’s not like there’s a bunch of complicated sub-problems that we need to work out about how to describe us as expected utility maximizers, my best guess is that we would just not end up doing that because it’s not a good idea.

Lucas Perry: For the ambitious value learning?

Buck Shlegeris: Yeah, that’s right.

Lucas Perry: Okay, do you have something that’s on offer?

Buck Shlegeris: The two options instead of that, which seem attractive to me? As I said earlier, one is you just convince everyone to not use AI systems for things where you need to have an understanding of large scale human preferences. The other one is the kind of thing that Paul Christiano’s iterated distillation and amplification, or a variety of his other ideas, the kind of thing that he’s trying to get there is, I think, if you make a really powerful AI system, it’s actually going to have an excellent model of human values in whatever representation is best for actually making predictions about  humans because a really excellent AGI, like a really excellent paperclip maximizer, it’s really important for it to really get how humans work so that it can manipulate them into letting it build lots of paperclip factories or whatever.

So I think that if you think that we have AGI, then by assumption I think we have a system which is able to reason about human values if it wants. And so if we can apply these really powerful AI systems to tasks such that the things that they do display their good understanding of human values, then we’re fine and it’s just okay that there was no way that we could represent a utility function directly. So for instance, the idea in IDA is that if we could have this system which is just trying to answer questions the same way that humans would, but enormously more cheaply because it can run faster than humans and a few other tricks, then we don’t have to worry about writing down a utility functions of humans directly because we can just make the system do things that are kind of similar to the things humans would have done, and so it implicitly has this human utility function built into it. That’s option two. Option one is don’t use anything that requires a complex human utility function, option two is have your systems learn human values implicitly, by giving them a task such that this is beneficial for them and such that their good understanding of human values comes out in their actions.

Rohin Shah: One way I might condense that point, is that you’re asking for a nice formalism for human preferences and I just point to all the humans out there in the world who don’t know anything about utility functions, which is 99% of them and nonetheless still seem pretty good at inferring human preferences.

Lucas Perry: On this part about AGI, if it is AGI it should be able to reason about human preferences, then why would it not be able to construct something that was more explicit and thus was able to do more ambitious value learning?

Buck Shlegeris: So it can totally do that, itself. But we can’t force that structure from the outside with our own algorithms.

Rohin Shah: Image classification is a good analogy. Like, in the past we were using hand engineered features, namely SIFT and HOG and then training classifiers over these hand engineered features in order to do image classification. And then we came to the era of deep learning and we just said, yeah, throw away all those features and just do everything end to end with a convolutional neural net and it worked way better. The point was that, in fact there are good representations for most tasks and humans trying to write them down ahead of time just doesn’t work very well at that. It tends to work better if you let the AI system discover its own representations that best capture the thing you wanted to capture.

Lucas Perry: Can you unpack this point a little bit more? I’m not sure that I’m completely understanding it. Buck is rejecting this modeling human beings explicitly as expected utility maximizers and trying to explicitly come up with utility functions in our AI systems. The first was to convince people not to use these kinds of things. And the second is to make it so that the behavior and output of the AI systems has some implicit understanding of human behavior. Can you unpack this a bit more for me or give me another example?

Rohin Shah: So here’s another example. Let’s say I was teaching my kid that I don’t have, how to catch a ball. It seems that the formalism that’s available to me for learning how to catch a ball is, well, you can go all the way down to look at our best models of physics, we could use Newtonian mechanics let’s say, like here are these equations, estimate the velocity and the distance of the ball and the angle at which it’s thrown plug that into these equations and then predict that the ball’s going to come here and then just put your hand there and then magically catch it. We won’t even talk about the catching part. That seems like a pretty shitty way to teach a kid how to catch a ball.

Probably it’s just a lot better to just play catch with the kid for a while and let the kid’s brain figure out this is how to predict where the ball is going to go such that I can predict where it’s going to be and then catch it.

I’m basically 100% confident that the thing that the brain is doing is not Newtonian mechanics. It’s doing something else that’s just way more efficient at predicting where the ball is going to be so that I can catch it and if I forced the brain to use Newtonian mechanics, I bet it would not do very well at this task.

Buck Shlegeris: I feel like that still isn’t quite saying the key thing here. I don’t know how to say this off the top of my head either, but I think there’s this key point about: just because your neural net can learn a particular feature of the world doesn’t mean that you can back out some other property of the world by forcing the neural net to have a particular shape. Does that make any sense, Rohin?

Rohin Shah: Yeah, vaguely. I mean, well, no, maybe not.

Buck Shlegeris: The problem isn’t just the capabilities problem. There’s this way you can try and infer a human utility function by asking, according to this model, what’s the maximum likelihood utility function given all these things the human did. If you have a good enough model, you will in fact end up making very good predictions about the human, it’s just that the decomposition into their planning function and their utility function is not going to result in a utility function which is anything like a thing that I would want maximized if this process was done on me. There is going to be some decomposition like this, which is totally fine, but the utility function part just isn’t going to correspond to the thing that I want.

Rohin Shah: Yeah, that is also a problem, but I agree that is not the thing I was describing.

Lucas Perry: Is the point there that there’s a lack of alignment between the utility function and the planning function. Given that the planning function imperfectly optimizes the utility function.

Rohin Shah: It’s more like there are just infinitely many possible pairs of planning functions and utility functions that exactly predict human behavior. Even if it were true that humans were expected utility maximizers, which Buck is arguing we’re not, and I agree with him. There is a planning function that’s like humans are perfectly anti-rational and if you’re like what utility function works with that planner to predict human behavior. Well, the literal negative of the true utility function when combined with the anti-rational planner produces the same behavior as the true utility function with the perfect planner, there’s no information that lets you distinguish between these two possibilities.

You have to build it in as an assumption. I think Buck’s point is that building things in as assumptions is probably not going to work.

Buck Shlegeris: Yeah.

Rohin Shah: A point I agree with. In philosophy this is called the is-ought problem, right? What you can train your AI system on is a bunch of “is” facts and then you have to add in some assumptions in order to jump to “ought” facts, which is what the utility function is trying to do. The utility function is trying to tell you how you ought to behave in new situations and the point of the is-ought distinction is that you need some bridging assumptions in order to get from is to ought.

Buck Shlegeris: And I guess an important part here is your system will do an amazing job of answering “is” questions about what humans would say about “ought” questions. And so I guess maybe you could phrase the second part as: to get your system to do things that match human preferences, use the fact that it knows how to make accurate “is” statements about humans’ ought statements?

Lucas Perry: It seems like we’re strictly talking about inferring the human utility function or preferences via looking at behavior. What if you also had more access to the actual structure of the human’s brain?

Rohin Shah: This is like the approach that Stuart Armstrong likes to talk about. The same things still apply. You still have the is-ought problem where the facts about the brain are “is” facts and how you translate that into “ought” facts is going to involve some assumptions. Maybe you can break down such assumptions that everyone would agree with. Maybe it’s like if this particular neuron in a human brain spikes, that’s a good thing and we want more of it and if this other one spikes, that’s a bad thing. We don’t want it. Maybe that assumption is fine.

Lucas Perry: I guess I’m just pointing out, if you could find the places in the human brain that generate the statements about Ought questions.

Rohin Shah: As Buck said, that lets you predict what humans would say about ought statements, which your assumption could then be, whatever humans say about ought statements, that’s what you ought to do. And that’s still an assumption. Maybe it’s a very reasonable assumption that we’re happy to put it into our AI system.

Lucas Perry: If we’re not willing to accept some humans’ “is” statements about “ought” questions then we have to do some meta-ethical moral policing in our assumptions around getting “is” statements from “ought” questions.

Rohin Shah: Yes, that seems right to me. I don’t know how you would do such a thing, but you would have to do something along those lines.

Buck Shlegeris: I would additionally say that I feel pretty great about trying to do things which use the fact that we can trust our AI to have good “is” answers to “ought” questions, but there’s a bunch of problems with this. I think it’s a good starting point but trying to use that to do arbitrarily complicated things in the world has a lot of problems. For instance, suppose I’m trying to decide whether we should design a city this way or that way. It’s hard to know how to go from the ability to know how humans would answer questions about preferences to knowing what you should do to design the city. And this is for a bunch of reasons, one of them is that the human might not be able to figure out from your city building plans what the city’s going to actually be like. And another is that the human might give inconsistent answers about what design is good, depending on how you phrase the question, such that if you try to figure out a good city plan by optimizing for the thing that the human is going to be most enthusiastic about, then you might end up with a bad city plan. Paul Christiano has written in a lot of detail about a lot of this.

Lucas Perry: That also reminds me of what Stuart Armstrong wrote about the framing on the questions changing output on the preference.

Rohin Shah: Yep.

Buck Shlegeris: Sorry, to be clear other people than Paul Christiano have also written a lot about this stuff, (including Rohin). My favorite writing about this stuff is by Paul.

Lucas Perry: Yeah, those do seem problematic but it would also seem that there would be further “is” statements that if you queried people’s meta-preferences about those things, you would get more “is” statements about that, but then that just pushes the “ought” assumptions that you need to make further back. Getting into very philosophically weedy territory. Do you think that this kind of thing could be pushed to the long reflection as is talked about by William MacAskill and Toby Ord or how much of this do you actually think needs to be solved in order to have safe and aligned AGI?

Buck Shlegeris: I think there are kind of two different ways that you could hope to have good outcomes from AGI. One is: set up a world such that you never needed to make an AGI which can make large scale decisions about the world. And two is: solve the full alignment problem.

I’m currently pretty pessimistic about the second of those being technically feasible. And I’m kind of pretty pessimistic about the first of those being a plan that will work. But in the world where you can have everyone only apply powerful and dangerous AI systems in ways that don’t require an understanding of human values, then you can push all of these problems onto the long reflection. In worlds where you can do arbitrarily complicated things in ways that humans would approve of, you don’t really need to long reflect this stuff because of the fact that these powerful AI systems already have the capacity of doing portions of the long reflection work inside themselves as needed. (Quotes about the long reflection

Rohin Shah: Yeah, so I think my take, it’s not exactly disagreeing with Buck. It’s more like from a different frame as Buck’s. If you just got AI systems that did the things that humans did now, this does not seem to me to obviously require solving hard problems in philosophy. That’s the lower bound on what you can do before having to do long reflection type stuff. Eventually you do want to do a longer reflection. I feel relatively optimistic about having a technical solution to alignment that allows us to do the long reflection after building AI systems. So the long reflection would include both humans and AI systems thinking hard, reflecting on difficult problems and so on.

Buck Shlegeris: To be clear, I’m super enthusiastic about there being a long reflection or something along those lines.

Lucas Perry: I always find it useful reflecting on just how human beings do many of these things because I think that when thinking about things in the strict AI alignment sense, it can seem almost impossible, but human beings are able to do so many of these things without solving all of these difficult problems. It seems like in the very least, we’ll be able to get AI systems that very, very approximately do what is good or what is approved of by human beings because we can already do that.

Buck Shlegeris: That argument doesn’t really make sense to me. It also didn’t make sense when Rohin referred to it a minute ago.

Rohin Shah: It’s not an argument for we technically know how to do this. It is more an argument for this as at least within the space of possibilities.

Lucas Perry: Yeah, I guess that’s how I was also thinking of it. It is within the space of possibilities. So utility functions are good because they can be optimized for, and there seem to be risks with optimization. Is there anything here that you guys would like to say about better understanding agency? I know this is one of the things that is important within the MIRI agenda.

Buck Shlegeris: I am a bad MIRI employee. I don’t really get that part of the MIRI agenda, and so I’m not going to defend it. I have certainly learned some interesting things from talking to Scott Garrabrant and other MIRI people who have lots of interesting thoughts about this stuff. I don’t quite see the path from there to good alignment strategies. But I also haven’t spent a super long time thinking about it because I, in general, don’t try to think about all of the different AI alignment things that I could possibly think about.

Rohin Shah: Yeah. I also am not a good person to ask about this. Most of my knowledge comes from reading things and MIRI has stopped writing things very much recently, so I don’t know what their ideas are. I, like Buck, don’t really see a good alignment strategy that starts with, first we understand optimization and so that’s the main reason why I haven’t looked into it very much.

Buck Shlegeris: I think I don’t actually agree with the thing you said there, Rohin. I feel like understanding optimization could plausibly be really nice. Basically the story there is, it’s a real bummer if we have to make really powerful AI systems via searching over large recurrent policies for things that implement optimizers. If it turned out that we could figure out some way of coding up optimizer stuffs directly, then this could maybe mean you didn’t need to make mesa-optimizers. And maybe this means that your inner alignment problems go away, which could be really nice. The thing that I was saying I haven’t thought that much about is, the relevance of thinking about, for instance, the various weirdnesses that happen when you consider embedded agency or decision theory, and things like that.

Rohin Shah: Oh, got it. Yeah. I think I agree that understanding optimization would be great if we succeeded at it and I’m mostly pessimistic about us succeeding at it, but also there are people who are optimistic about it and I don’t know why they’re optimistic about it.

Lucas Perry: Hey it’s post-podcast Lucas here again. So, I just want to add a little more detail here again on behalf of Rohin. Here he feels pessimistic about us understanding optimization well enough and in a short enough time period that we are able to create powerful optimizers that we understand that rival the performance of the AI systems we’re already building and will build in the near future. Back to the episode. 

Buck Shlegeris: The arguments that MIRI has made about this,… they think that there are a bunch of questions about what optimization is, that are plausibly just not that hard compared to other problems which small groups of people have occasionally solved, like coming up with foundations of mathematics, kind of a big conceptual deal but also a relatively small group of people. And before we had formalizations of math, I think it might’ve seemed as impossible to progress on as formalizing optimization or coming up with a better picture of that. So maybe that’s my argument for some optimism.

Rohin Shah: Yeah, I think pointing to some examples of great success does not imply… Like there are probably many similar things that didn’t work out and we don’t know about them cause nobody bothered to tell us about them because they failed. Seems plausible maybe.

Lucas Perry: So, exploring more deeply this point of agency can either, or both of you, give us a little bit of a picture about the relevance or non relevance of decision theory here to AI alignment and I think, Buck, you mentioned the trickiness of embedded decision theory.

Rohin Shah: If you go back to our traditional argument for AI risk, it’s basically powerful AI systems will be very strong optimizers. They will possibly be misaligned with us and this is bad. And in particular one specific way that you might imagine this going wrong is this idea of mesa optimization where we don’t know how to build optimizers right now. And so what we end up doing is basically search across a huge number of programs looking for ones that do well at optimization and use that as our AGI system. And in this world, if you buy that as a model of what’s happening, then you’ll basically have almost no control over what exactly that system is optimizing for. And that seems like a recipe for misalignment. It sure would be better if we could build the optimizer directly and know what it is optimizing for. And in order to do that, we need to know how to do optimization well.

Lucas Perry: What are the kinds of places that we use mesa optimizers today?

Rohin Shah: It’s not used very much yet. The field of meta learning is the closest example. In the field of meta learning you have a distribution over tasks and you use gradient descent or some other AI technique in order to find an AI system that itself, once given a new task, learns how to perform that task well.

Existing meta learning systems are more like learning how to do all the tasks well and then when they’ll see a new task they just figure out ah, it’s this task and then they roll out the policy that they already learned. But the eventual goal for meta learning is to get something that, online, learns how to do the task without having previously figured out how to do that task.

Lucas Perry: Okay, so Rohin did what you say cover embedded decision theory?

Rohin Shah: No, not really. I think embedded decision theory is just, we want to understand optimization. Our current notion of optimization, one way you could formalize it is to say my AI agent is going to have Bayesian belief over all the possible ways that the environment could be. It’s going to update that belief over time as it gets observations and then it’s going to act optimally with respect to that belief, by maximizing its expected utility. And embedded decision theory basically calls into question the idea that there’s a separation between the agent and the environment. In particular I, as a human, couldn’t possibly have a Bayesian belief about the entire earth because the entire Earth contains me. I can’t have a Bayesian belief over myself so this means that our existing formalization of agency is flawed. It can’t capture these things that affect real agents. And embedded decision theory, embedded agency, more broadly, is trying to deal with this fact and have a new formalization that works even in these situations.

Buck Shlegeris: I want to give my understanding of the pitch for it. One part is that if you don’t understand embedded agency, then if you try to make an AI system in a hard coded way, like making a hard coded optimizer, traditional phrasings of what an optimizer is, are just literally wrong in that, for example, they’re assuming that you have these massive beliefs over world states that you can’t really have. And plausibly, it is really bad to try to make systems by hardcoding assumptions that are just clearly false. And so if we want to hardcode agents with particular properties, it would be good if we knew a way of coding the agent that isn’t implicitly making clearly false assumptions.

And the second pitch for it is something like when you want to understand a topic, sometimes it’s worth looking at something about the topic which you’re definitely wrong about, and trying to think about that part until you are less confused about it. When I’m studying physics or something, a thing that I love doing is looking for the easiest question whose answer I don’t know, and then trying to just dive in until I have satisfactorily answered that question, hoping that the practice that I get about thinking about physics from answering a question correctly will generalize to much harder questions. I think that’s part of the pitch here. Here is a problem that we would need to answer, if we wanted to understand how superintelligent AI systems work, so we should try answering it because it seems easier than some of the other problems.

Lucas Perry: Okay. I think I feel satisfied. The next thing here Rohin in your AI alignment 2018-19 review is value learning. I feel like we’ve talked a bunch about this already. Is there anything here that you want to say or do you want to skip this?

Rohin Shah: One thing we didn’t cover is, if you have uncertainty over what you’re supposed to optimize, this turns into an interactive sort of game between the human and the AI agent, which seems pretty good. A priori you should expect that there’s going to need to be a lot of interaction between the human and the AI system in order for the AI system to actually be able to do the things that the human wants it to do. And so having formalisms and ideas of where this interaction naturally falls out seems like a good thing.

Buck Shlegeris: I’ve said a lot of things about how I am very pessimistic about value learning as a strategy. Nevertheless it seems like it might be really good for there to be people who are researching this, and trying to get as good as we can get at improving sample efficiency so that can have your AI systems understand your preferences over music with as little human interaction as possible, just in case it turns out to be possible to solve the hard version of value learning. Because a lot of the engineering effort required to make ambitious value learning work will plausibly be in common with the kinds of stuff you have to do to make these more simple specification learning tasks work out. That’s a reason for me to be enthusiastic about people researching value learning even if I’m pessimistic about the overall thing working.

Lucas Perry: All right, so what is robustness and why does it matter?

Rohin Shah: Robustness is one of those words that doesn’t super clearly have a definition and people use it differently. Robust agents don’t fail catastrophically in situations slightly different from the ones that they were designed for. One example of a case where we see a failure of robustness currently, is in adversarial examples for image classifiers, where it is possible to take an image, make a slight perturbation to it, and then the resulting image is completely misclassified. You take a correctly classified image of a Panda, slightly perturb it such that a human can’t tell what the difference is, and then it’s classified as a gibbon with 99% confidence. Admittedly this was with an older image classifier. I think you need to make the perturbations a bit larger now in order to get them.

Lucas Perry: This is because the relevant information that it uses are very local to infer panda-ness rather than global properties of the panda?

Rohin Shah: It’s more like they’re high frequency features or imperceptible features. There’s a lot of controversy about this but there is a pretty popular recent paper that I believe, but not everyone believes, that claims that this was because they’re picking up on real imperceptible features that do generalize to the test set, that humans can’t detect. That’s an example of robustness. Recently people have been applying this to reinforcement learning both by adversarially modifying the observations that agents get and also by training agents that act in the environment adversarially towards the original agent. One paper out of CHAI showed that there’s this kick and defend environment where you’ve got two MuJoCo robots. One of them is kicking a soccer ball. The other one’s a goalie, that’s trying to prevent the kicker from successfully shooting a goal, and they showed that if you do self play in order to get kickers and defenders and then you take the kicker, you freeze it, you don’t train it anymore and you retrain a new defender against this kicker.

What is the strategy that this new defender learns? It just sort of falls to the ground and flaps about in a random looking way and the kicker just gets so confused that it usually fails to even touch the ball and so this is sort of an adversarial example for RL agents now, it’s showing that even they’re not very robust.

There was also a paper out of DeepMind that did the same sort of thing. For their adversarial attack they learned what sorts of mistakes the agent would make early on in training and then just tried to replicate those mistakes once the agent was fully trained and they found that this helped them uncover a lot of bad behaviors. Even at the end of training.

From the perspective of alignment, it’s clear that we want robustness. It’s not exactly clear what we want robustness to. This robustness to adversarial perturbations was kind of a bit weird as a threat model. If there is an adversary in the environment they’re probably not going to be restricted to small perturbations. They’re probably not going to get white box access to your AI system; even if they did, this doesn’t seem to really connect with the AI system as adversarially optimizing against humans story, which is how we get to the x-risk part, so it’s not totally clear.

I think on the intent alignment case, which is the thing that I usually think about, you mostly want to ensure that whatever is driving the “motivation” of the AI system, you want that to be very robust. You want it to agree with what humans would want in all situations or at least all situations that are going to come up or something like that. Paul Christiano has written a few blog posts about this that talk about what techniques he’s excited about solving that problem, which boil down to interpretability, adversarial training, and improving adversarial training through relaxations of the problem.

Buck Shlegeris: I’m pretty confused about this, and so it’s possible what I’m going to say is dumb. When I look at problems with robustness or problems that Rohin put in this robustness category here, I want to divide it into two parts. One of the parts is, things that I think of as capability problems, which I kind of expect the rest of the world will need to solve on its own. For instance, things about safe exploration, how do I get my system to learn to do good things without ever doing really bad things, this just doesn’t seem very related to the AI alignment problem to me. And I also feel reasonably optimistic that you can solve it by doing dumb techniques which don’t have anything too difficult to them, like you can have your system so that it has a good model of the world that it got from unsupervised learning somehow and then it never does dumb enough things. And also I don’t really see that kind of robustness problem leading to existential catastrophes. And the other half of robustness is the half that I care about a lot, which in my mind, is mostly trying to make sure that you succeeded at inner alignment. That is, that the mesa optimizers you’ve found through gradient descent have goals that actually match your goals.

This is like robustness in the sense that you’re trying to guarantee that in every situation, your AI system, as Rohin was saying, is intent aligned with you. It’s trying to do the kind of thing that you want. And I worry that, by default, we’re going to end up with AI systems not intent aligned, so there exist a bunch of situations they can be put in such that they do things that are very much not what you’d want, and therefore they fail at robustness. I think this is a really important problem, it’s like half of the AI safety problem or more, in my mind, and I’m not very optimistic about being able to solve it with prosaic techniques.

Rohin Shah: That sounds roughly similar to what I was saying. Yes.

Buck Shlegeris: I don’t think we disagree about this super much except for the fact that I think you seem to care more about safe exploration and similar stuff than I think I do.

Rohin Shah: I think safe exploration’s a bad example. I don’t know what safe exploration is even trying to solve but I think other stuff, I agree. I do care about it more. One place where I somewhat disagree with you is, you sort of have this point about all these robustness problems are the things that the rest of the world has incentives to figure out, and will probably figure out. That seems true for alignment too, it sure seems like you want your system to be aligned in order to do the things that you actually want. Everyone that has an incentive for this to happen. I totally expect people who aren’t EAs or rationalists or weird longtermists to be working on AI alignment in the future and to some extent even now. I think that’s one thing.

Buck Shlegeris: You should say your other thing, but then I want to get back to that point.

Rohin Shah: The other thing is I think I agree with you that it’s not clear to me how failures of the robustness of things other than motivation lead to x-risk, but I’m more optimistic than you are that our solutions to those kinds of robustness will help with the solutions to “motivation robustness” or how to make your mesa optimizer aligned.

Buck Shlegeris: Yeah, sorry, I guess I actually do agree with that last point. I am very interested in trying to figure out how to have aligned to mesa optimizers, and I think that a reasonable strategy to pursue in order to get aligned mesa optimizers is trying to figure out how to make your image classifiers robust to adversarial examples. I think you probably won’t succeed even if you succeed with the image classifiers, but it seems like the image classifiers are still probably where you should start. And I guess if we can’t figure out how to make image classifiers robust to adversarial examples in like 10 years, I’m going to be super pessimistic about the harder robustness problem, and that would be great to know.

Rohin Shah: For what it’s worth, my take on the adversarial examples of image classifiers is, we’re going to train image classifiers on more data with bigger nets, it’s just going to mostly go away. Prediction. I’m laying my cards on the table.

Buck Shlegeris: That’s also something like my guess.

Rohin Shah: Okay.

Buck Shlegeris: My prediction is: to get image classifiers that are robust to epsilon ball perturbations or whatever, some combination of larger things and adversarial training and a couple other clever things, will probably mean that we have robust image classifiers in 5 or 10 years at the latest.

Rohin Shah: Cool. And you wanted to return to the other point about the world having incentives to do alignment.

Buck Shlegeris: So I don’t quite know how to express this, but I think it’s really important which is going to make this a really fun experience for everyone involved. You know how Airbnb… Or sorry, I guess a better example of this is actually Uber drivers. Where I give basically every Uber driver a five star rating, even though some Uber drivers are just clearly more pleasant for me than others, and Uber doesn’t seem to try very hard to get around these problems, even though I think that if Uber caused there to be a 30% difference in pay between the drivers who I think of as 75th percentile and the drivers I think of as 25th percentile, this would make the service probably noticeably better for me. I guess it seems to me that a lot of the time the world just doesn’t try do kind of complicated things to make systems actually aligned, and it just does hack jobs, and then everyone deals with the fact that everything is unaligned as a result.

To draw this analogy back, I think that we’re likely to have the kind of alignment techniques that solve problems that are as simple and obvious as: we should have a way to have rate your hosts on Airbnb. But I’m worried that we won’t ever get around to solving the problems that are like, but what if your hosts are incentivized to tell you sob stories such that you give them good ratings, even though actually they were worse than some other hosts. And this is never a big enough deal that people are unilaterally individually incentivized to solve the harder version of the alignment problem, and then everyone ends up using these systems that actually aren’t aligned in the strong sense and then we end up in a doomy world. I’m curious if any of that made any sense.

Lucas Perry: Is a simple way to put that we fall into inadequate or an unoptimal equilibrium and then there’s tragedy of the commons and bad game theory stuff that happens that keeps us locked and that the same story could apply to alignment?

Buck Shlegeris: Yeah, that’s not quite what I mean.

Lucas Perry: Okay.

Rohin Shah: I think Buck’s point is that actually Uber or Airbnb could unilaterally, no gains required, make their system better and this would be an improvement for them and everyone else, and they don’t do it. There is nothing about equilibrium that is a failure of Uber to do this thing that seems so obviously good.

Buck Shlegeris: I’m not actually claiming that it’s better for Uber, I’m just claiming that there is a misalignment there. Plausibly, an Uber exec, if they were listening to this they’d just be like, “LOL, that’s a really stupid idea. People would hate it.” And then they would say more complicated things like “most riders are relatively price sensitive and so this doesn’t matter.” And plausibly they’re completely right.

Rohin Shah: That’s what I was going to say.

Buck Shlegeris: But the thing which feels important to me is something like a lot of the time it’s not worth solving the alignment problems at any given moment because something else is a bigger problem to how things are going locally. And this can continue being the case for a long time, and then you end up with everyone being locked in to this system where they never solved the alignment problems. And it’s really hard to make people understand this, and then you get locked into this bad world.

Rohin Shah: So if I were to try and put that in the context of AI alignment, I think this is a legitimate reason for being more pessimistic. And the way that I would make that argument is: it sure seems like we are going to decide on what method or path we’re going to use to build AGI. Maybe we’ll do a bunch of research and decide we’re just going to scale up language models or something like this. I don’t know. And we will do that before we have any idea of which technique would be easiest to align and as a result, we will be forced to try to align this exogenously chosen AGI technique and that would be harder than if we got to design our alignment techniques and our AGI techniques simultaneously.

Buck Shlegeris: I’m imagining some pretty slow take off here, and I don’t imagine this as ever having a phase where we built this AGI and now we need to align it. It’s more like we’re continuously building and deploying these systems that are gradually more and more powerful, and every time we want to deploy a system, it has to be doing something which is useful to someone. And many of the things which are useful, require things that are kind of like alignment. “I want to make a lot of money from my system that will give advice,” and if it wants to give good generalist advice over email, it’s going to need to have at least some implicit understanding of human preferences. Maybe we just use giant language models and everything’s just totally fine here. A really good language model isn’t able to give arbitrarily good aligned advice, but you can get advice that sounds really good from a language model, and I’m worried that the default path is going to involve the most popular AI advice services being kind of misaligned, and just never bothering to fix that. Does that make any more sense?

Rohin Shah: Yeah, I think I totally buy that that will happen. But I think I’m more like as you get to AI systems doing more and more important things in the world, it becomes more and more important that they are really truly aligned and investment in alignment increases correspondingly.

Buck Shlegeris: What’s the mechanism by which people realize that they need to put more work into alignment here?

Rohin Shah: I think there’s multiple. One is I expect that people are aware, like even in the Uber case, I expect people are aware of the misalignment that exists, but decide that it’s not worth their time to fix it. So the continuation of that, people will be aware of it and then they will decide that they should fix it.

Buck Shlegeris: If I’m trying to sell to city governments this language model based system which will give them advice on city planning, it’s not clear to me that at any point the city governments are going to start demanding better alignment features. Maybe that’s the way that it goes but it doesn’t seem obvious that city governments would think to ask that, and —

Rohin Shah: I wasn’t imagining this from the user side. I was imagining this from the engineers or designers side.

Buck Shlegeris: Yeah.

Rohin Shah: I think from the user side I would speak more to warning shots. You know, you have your cashier AI system or your waiter AIs and they were optimizing for tips more so than actually collecting money and so they like offer free meals in order to get more tips. At some point one of these AI systems passes all of the internal checks and makes it out into the world and only then does the problem arise and everyone’s like, “Oh my God, this is terrible. What the hell are you doing? Make this better.”

Buck Shlegeris: There’s two mechanisms via which that alignment might be okay. One of them is that researchers might realize that they want to put more effort into alignment and then solve these problems. The other mechanism is that users might demand better alignment because of warning shots. I think that I don’t buy that either of these is sufficient. I don’t buy that it’s sufficient for researchers to decide to do it because in a competitive world, the researchers who realize this is important, if they try to only make aligned products, they are not going to be able to sell them because their products will be much less good than the unaligned ones. So you have to argue that there is demand for the things which are actually aligned well. But for this to work, your users have to be able to distinguish between things that have good alignment properties and those which don’t, and this seems really hard for users to do. And I guess, when I try to imagine analogies, I just don’t see many examples of people successfully solving problems like this, like businesses making products that are different levels of dangerousness, and then users successfully buying the safe ones.

Rohin Shah: I think usually what happens is you get regulation that forces everyone to be safe. I don’t know if it was regulation, but like airplanes are incredibly safe. Cars are incredibly safe.

Buck Shlegeris: Yeah but in this case what would happen is doing the unsafe thing allows you to make enormous amounts of money, and so the countries which don’t put in the regulations are going to be massively advantaged compared to ones which don’t.

Rohin Shah: Why doesn’t that apply for cars and airplanes?

Buck Shlegeris: So to start with, cars in poor countries are a lot less safe. Another thing is that a lot of the effort in making safer cars and airplanes comes from designing them. Once you’ve done the work of designing it, it’s that much more expensive to put your formally-verified 747 software into more planes, and because of weird features of the fact that there are only like two big plane manufacturers, everyone gets the safer planes.

Lucas Perry: So tying this into robustness. The fundamental concern here is about the incentives to make aligned systems that are safety and alignment robust in the real world.

Rohin Shah: I think that’s basically right. I sort of see these incentives as existing and the world generally being reasonably good at dealing with high stakes problems.

Buck Shlegeris: What’s an example of the world being good at dealing with a high stakes problem?

Rohin Shah: I feel like biotech seems reasonably well handled, relatively speaking,

Buck Shlegeris: Like bio-security?

Rohin Shah: Yeah.

Buck Shlegeris: Okay, if the world handles AI as well as bio-security, there’s no way we’re okay.

Rohin Shah: Really? I’m aware of ways in which we’re not doing bio-security well, but there seem to be ways in which we’re doing it well too.

Buck Shlegeris: The nice thing about bio-security is that very few people are incentivized to kill everyone, and this means that it’s okay if you’re sloppier about your regulations, but my understanding is that lots of regulations are pretty weak.

Rohin Shah: I guess I was more imagining the research community’s coordination on this. Surprisingly good.

Buck Shlegeris: I wouldn’t describe it that way.

Rohin Shah: It seems like the vast majority of the research community is onboard with the right thing and like 1% isn’t. Yeah. Plausibly we need to have regulations for that last 1%.

Buck Shlegeris: I think that 99% of the synthetic biology research community is on board with “it would be bad if everyone died.” I think that some very small proportion is onboard with things like “we shouldn’t do research if it’s very dangerous and will make the world a lot worse.” I would say like way less than half of synthetic biologists seem to agree with statements like “it’s bad to do really dangerous research.” Or like, “when you’re considering doing research, you consider differential technological development.” I think this is just not a thing biologists think about, from my experience talking to biologists.

Rohin Shah: I’d be interested in betting with you on this afterwards.

Buck Shlegeris: Me too.

Lucas Perry: So it seems like it’s going to be difficult to come down to a concrete understanding or agreement here on the incentive structures in the world and whether they lead to the proliferation of unaligned AI systems or semi aligned AI systems versus fully aligned AI systems and whether that poses a kind of lock-in, right? Would you say that that fairly summarizes your concern Buck?

Buck Shlegeris: Yeah. I expect that Rohin and I agree mostly on the size of the coordination problem required, or the costs that would be required by trying to do things the safer way. And I think Rohin is just a lot more optimistic about those costs being paid.

Rohin Shah: I think I’m optimistic both about people’s ability to coordinate paying those costs and about incentives pointing towards paying those costs.

Buck Shlegeris: I think that Rohin is right that I disagree with him about the second of those as well.

Lucas Perry: Are you interested in unpacking this anymore? Are you happy to move on?

Buck Shlegeris: I actually do want to talk about this for two more minutes. I am really surprised by the claim that humans have solved coordination problems as hard as this one. I think the example you gave is humans doing radically nowhere near well enough. What are examples of coordination problem type things… There was a bunch of stuff with nuclear weapons, where I feel like humans did badly enough that we definitely wouldn’t have been okay in an AI situation. There are a bunch of examples of the US secretly threatening people with nuclear strikes, which I think is an example of some kind of coordination failure. I don’t think that the world has successfully coordinated on never threaten first nuclear strikes. If we had successfully coordinated on that, I would consider nuclear weapons to be less of a failure, but as it is the US has actually according to Daniel Ellsberg threatened a bunch of people with first strikes.

Rohin Shah: Yeah, I think I update less on specific scenarios and update quite a lot more on, “it just never happened.” The sheer amount of coincidence that would be required given the level of, Oh my God, there were close calls multiple times a year for many decades. That seems just totally implausible and it just means that our understanding of what’s happening is wrong.

Buck Shlegeris: Again, also the thing I’m imagining is this very gradual takeoff world where people, every year, they release their new most powerful AI systems. And if, in a particular year, AI Corp decided to not release its thing, then AI Corps two and three and four would rise to being one, two and three in total profits instead of two, three and four. In that kind of a world, I feel a lot more pessimistic.

Rohin Shah: I’m definitely imagining more of the case where they coordinate to all not do things. Either by international regulation or via the companies themselves coordinating amongst each other. Even without that, it’s plausible that AI Corp one does this. One example I’d give is, Waymo has just been very slow to deploy self driving cars relative to all the other self driving car companies, and my impression is that this is mostly because of safety concerns.

Buck Shlegeris: Interesting and slightly persuasive example. I would love to talk through this more at some point. I think this is really important and I think I haven’t heard a really good conversation about this.

Apologies for describing what I think is going wrong inside your mind or something, which is generally a bad way of saying things, but it sounds kind of to me like you’re implicitly assuming more concentrated advantage and fewer actors than I think actually are implied by gradual takeoff scenarios.

Rohin Shah: I’m usually imagining something like a 100+ companies trying to build the next best AI system, and 10 or 20 of them being clear front runners or something.

Buck Shlegeris: That makes sense. I guess I don’t quite see how the coordination successes you were describing arise in that kind of a world. But I am happy to move on.

Lucas Perry: So before we move on on this point, is there anything which you would suggest as obvious solutions, should Buck’s model of the risks here be the case. So it seemed like it would demand more centralized institutions which would help to mitigate some of the lock in here.

Rohin Shah: Yeah. So there’s a lot of work in policy and governance about this. Not much of which is public unfortunately. But I think the thing to say is that people are thinking about it and it does sort of look like trying to figure out how to get the world to actually coordinate on things. But as Buck has pointed out, we have tried to do this before and so there’s probably a lot to learn from past cases as well. But I am not an expert on this and don’t really want to talk as though I were one.

Lucas Perry: All right. So there’s lots of governance and coordination thought that kind of needs to go into solving many of these coordination issues around developing beneficial AI. So I think with that we can move along now to scaling to superhuman abilities. So Rohin, what do you have to say about this topic area?

Rohin Shah: I think this is in some sense related to what we were talking about before, you can predict what a human would say, but it’s hard to back out true underlying values beneath them. Here the problem is, suppose you are learning from some sort of human feedback about what you’re supposed to be doing, the information contained in that tells you how to do whatever the human can do. It doesn’t really tell you how to exceed what the human can do without having some additional assumptions.

Now, depending on how the human feedback is structured, this might lead to different things like if the human is demonstrating how to do the task to you, then this would suggest that it would be hard to do the task any better than the human can, but if the human was evaluating how well you did the task, then you can do the task better in a way that the human wouldn’t be able to tell was better. Ideally, at some point we would like to have AI systems that can actually do just really powerful, great things, that we are unable to understand all the details of and so we would neither be able to demonstrate or evaluate them.

How do we get to those sorts of AI systems? The main proposals in this bucket are iterated amplification, debate, and recursive reward modeling. So in iterated amplification, we started with an initial policy, and we alternate between amplification and distillation, which increases capabilities and efficiency respectively. This can encode a bunch of different algorithms, but usually amplification is done by decomposing questions into easier sub questions, and then using the agent to answer those sub questions. While distillation can be done using supervised learning or reinforcement learning, so you get these answers that are created by these amplified systems that take a long time to run, and you just train a neural net to very quickly predict the answers without having to do this whole big decomposition thing. In debate, we train an agent through self play in a zero sum game where the agent’s goal is to win a question answering debate as evaluated by a human judge. The hope here is that since both sides of the debate can point out flaws in the other side’s arguments — they’re both very powerful AI systems — such a set up can use a human judge to train far more capable agents while still incentivizing the agents to provide honest true information. With recursive reward modeling, you can think of it as an instantiation of the general alternate between amplification and distillation framework, but it works sort of bottom up instead of top down. So you’ll start by building AI systems that can help you evaluate simple, easy tasks. Then use those AI systems to help you evaluate more complex tasks and you keep iterating this process until eventually you have AI systems that help you with very complex tasks like how to design the city. And this lets you then train an AI agent that can design the city effectively even though you don’t totally understand why it’s doing the things it’s doing or why they’re even good.

Lucas Perry: Do either of you guys have any high level thoughts on any of these approaches to scaling to superhuman abilities?

Buck Shlegeris: I have some.

Lucas Perry: Go for it.

Buck Shlegeris: So to start with, I think it’s worth noting that another approach would be ambitious value learning, in the sense that I would phrase these not as approaches for scaling to superhuman abilities, but they’re like approaches for scaling to superhuman abilities while only doing tasks that relate to the actual behavior of humans rather than trying to back out their values explicitly. Does that match your thing Rohin?

Rohin Shah: Yeah, I agree. I often phrase that as with ambitious value learning, there’s not a clear ground truth to be focusing on, whereas with all three of these methods, the ground truth is what a human would do if they got a very, very long time to think or at least that is what they’re trying to approximate. It’s a little tricky to see why exactly they’re approximating that, but there are some good posts about this. The key difference between these techniques and ambitious value learning is that there is in some sense a ground truth that you are trying to approximate.

Buck Shlegeris: I think these are all kind of exciting ideas. I think they’re all kind of better ideas than I expected to exist for this problem a few years ago. Which probably means we should update against my ability to correctly judge how hard AI safety problems are, which is great news, in as much as I think that a lot of these problems are really hard. Nevertheless, I don’t feel super optimistic that any of them are actually going to work. One thing which isn’t in the elevator pitch for IDA, which is iterated distillation and amplification (and debate), is that you get to hire the humans who are going to be providing the feedback, or the humans whose answers AI systems are going to be trained with. And this is actually really great. Because for instance, you could have this program where you hire a bunch of people and you put them through your one month long training an AGI course. And then you only take the top 50% of them. I feel a lot more optimistic about these proposals given you’re allowed to think really hard about how to set it up such that the humans have the easiest time possible. And this is one reason why I’m optimistic about people doing research in factored cognition and stuff, which I’m sure Rohin’s going to explain in a bit.

One comment about recursive reward modeling: it seems like it has a lot of things in common with IDA. The main downside that it seems to have to me is that the human is in charge of figuring out how to decompose the task into evaluations at a variety of levels. Whereas with IDA, your system itself is able to naturally decompose the task into a variety levels, and for this reason I feel a bit more optimistic about IDA.

Rohin Shah: With recursive reward modeling, one agent that you can train is just an agent that’s good at doing decompositions. That is a thing you can do with it. It’s a thing that the people at DeepMind are thinking about. 

Buck Shlegeris: Yep, that’s a really good point. 

Rohin Shah: I also strongly like the fact that you can train your humans to be good at providing feedback. This is also true about specification learning. It’s less clear if it’s true about ambitious value learning. No one’s really proposed how you could do ambitious value learning really. Maybe arguably Stuart Russell’s book is kind of a proposal, but it doesn’t have that many details.

Buck Shlegeris: And, for example, it doesn’t address any of my concerns in ways that I find persuasive.

Rohin Shah: Right. But for specification learning also you definitely want to train the humans who are going to be providing feedback to the AI system. That is an important part of why you should expect this to work.

Buck Shlegeris: I often give talks where I try to give an introduction to IDA and debate as a proposal for AI alignment. I’m giving these talks to people with computer science backgrounds, and they’re almost always incredibly skeptical that it’s actually possible to decompose thought in this kind of a way. And with debate, they’re very skeptical that truth wins, or that the nash equilibrium is accuracy. For this reason I’m super enthusiastic about research into the factored cognition hypothesis of the type that Ought is doing some of.

I’m kind of interested in your overall take for how likely it is that the factored cognition hypothesis holds and that it’s actually possible to do any of this stuff, Rohin. You could also explain what that is.

Rohin Shah: I’ll do that. So basically with both iterated amplification, debate, or recursive reward modeling, they all hinge on this idea of being able to decompose questions, maybe it’s not so obvious why that’s true for debate, but it’s true. Go listen to the podcast about debate if you want to get more details on that.

So this hypothesis is basically for any tasks that we care about, it is possible to decompose this into a bunch of sub tasks that are all easier to do. Such that if you’re able to do the sub tasks, then you can do the overall top level tasks and in particular you can iterate this down, building a tree of smaller and smaller tasks until you can get to the level of tasks that a human could do in a day. Or if you’re trying to do it very far, maybe tasks that a human can do in a couple of minutes. Whether or not you can actually decompose the task “be an effective CEO” into a bunch of sub tasks that eventually bottom out into things humans can do in a few minutes is totally unclear. Some people are optimistic, some people are pessimistic. It’s called the factored cognition hypothesis and Ought is an organization that’s studying it.

It sounds very controversial at first and I, like many other people had the intuitive reaction of, ‘Oh my God, this is never going to work and it’s not true’. I think the thing that actually makes me optimistic about it is you don’t have to do what you might call a direct decomposition. You can do things like if your task is to be an effective CEO, your first sub question could be, what are the important things to think about when being a CEO or something like this, as opposed to usually when I think of decompositions I would think of, first I need to deal with hiring. Maybe I need to understand HR, maybe I need to understand all of the metrics that the company is optimizing. Very object level concerns, but the decompositions are totally allowed to also be meta level where you’ll spin off a bunch of computation that is just trying to answer the meta level of question of how should I best think about this question at all.

Another important reason for optimism is that based on the structure of iterated amplification, debate and recursive reward modeling, this tree can be gigantic. It can be exponentially large. Something that we couldn’t run even if we had all of the humans on Earth collaborating to do this. That’s okay. Given how the training process is structured, considering the fact that you can do the equivalent of millennia of person years of effort in this decomposed tree, I think that also gives me more of a, ‘okay, maybe this is possible’ and that’s also why you’re able to do all of this meta level thinking because you have a computational budget for it. When you take all of those together, I sort of come up with “seems possible. I don’t really know.”

Buck Shlegeris: I think I’m currently at 30-to-50% on the factored cognition thing basically working out. Which isn’t nothing.

Rohin Shah: Yeah, that seems like a perfectly reasonable thing. I think I could imagine putting a day of thought into it and coming up with numbers anywhere between 20 and 80.

Buck Shlegeris: For what it’s worth, in conversation at some point in the last few years, Paul Christiano gave numbers that were not wildly more optimistic than me. I don’t think that the people who are working on this think it’s obviously fine. And it would be great if this stuff works, so I’m really in favor of people looking into it.

Rohin Shah: Yeah, I should mention another key intuition against it. We have all these examples of human geniuses like Ramanujan, who were posed very difficult math problems and just immediately get the answer and then you ask them how did they do it and they say, well, I asked myself what should the answer be? And I was like, the answer should be a continued fraction. And then I asked myself which continued fraction and then I got the answer. And you’re like, that does not sound very decomposable. It seems like you need these magic flashes of intuition. Those would be the hard cases for factored cognition. It still seems possible that you could do it by both this exponential try a bunch of possibilities and also by being able to discover intuitions that work in practice and just believing them because they work in practice and then applying them to the problem at hand. You could imagine that with enough computation you’d be able to discover such intuitions.

Buck Shlegeris: You can’t answer a math problem by searching exponentially much through the search tree. The only exponential power you get from IDA is IDA is letting you specify the output of your cognitive process in such a way that’s going to match some exponentially sized human process. As long as that exponentially sized human process was only exponentially sized because it’s really inefficient, but is kind of fundamentally not an exponentially sized problem, then your machine learning should be able to speed it up a bunch. But the thing where you search over search strategy is not valid. If that’s all you can do, that’s not good enough.

Rohin Shah: Searching over search strategies, I agree you can’t do, but if you have an exponential search that could be implemented by humans. We know by hypothesis, if you can solve it with a flash of intuition, there is in fact some more efficient way to do it and so whether or not the distillation steps will actually be enough to get to the point where you can do those flashes of intuition. That’s an open question.

Buck Shlegeris: This is one of my favorite areas of AI safety research and I would love for there to be more of it. Something I have been floating for a little while is I kind of wish that there was another Ought. It just seems like it would be so good if we had definitive information about the factored cognition hypothesis. And it also it seems like the kind of thing which is potentially parallelizable. And I feel like I know a lot of people who love talking about how thinking works. A lot of rationalists are really into this. I would just be super excited for some of them to form teams of four and go off on their own and build an Ought competitor. I feel like this is the kind of thing where plausibly, a bunch of enthusiastic people could make progress on their own.

Rohin Shah: Yeah, I agree with that. Definitely seems like one of the higher value things but I might be more excited about universality.

Lucas Perry: All right, well let’s get started with universality then. What is universality and why are you optimistic about it?

Rohin Shah: So universality is hard to explain well, in a single sentence. For whatever supervisor is training our agent, you want that supervisor to “know everything the agent knows.” In particular if the agent comes up with some deceptive strategy to look like it’s achieving the goal, but actually it hasn’t. The supervisors should know that it was doing this deceptive strategy for the reason of trying to trick the supervisor and so the supervisor can then penalize it. The classic example of why this is important and hard also due to Paul Christiano is plagiarism. Suppose you are training on the AI system to produce novel works of literature and as part of its training data, the AI system gets to read this library of a million books.

It’s possible that this AI system decides, Hey, you know the best way I can make a great novel seeming book is to just take these five books and take out plot points, passages from each of them and put them together and then this new book will look totally novel and will be very good because I used all of the best Shakespearean writing or whatever. If your supervisor doesn’t know that the agent has done this, the only way the supervisor can really check is to go read the entire million books. Even if the agent only read 10 books and so then the supervision becomes a way more costly than running the agent, which is not a great state to be in, and so what you really want is that if the agent does this, the supervisor is able to say, I see that you just copied this stuff over from these other books in order to trick me into thinking that you had written something novel that was good.

That’s bad. I’m penalizing you. Stop doing that in the future. Now, this sort of property, I mean it’s very nice in the abstract, but who knows whether or not we can actually build it in practice. There’s some reason for optimism that I don’t think I can adequately convey, but I wrote a newsletter summarizing some of it sometime ago, but again, reading through the posts I became more optimistic that it was an achievable property, than when I first heard what the property was. The reason I’m optimistic about it is that it just sort of seems to capture the thing that we actually care about. It’s not everything, like it doesn’t solve the robustness problem. Universality only tells you what the agent’s currently doing. You know all the facts about that. Whereas for robustness you want to say even in these hypothetical situations that the agent hasn’t encountered yet and doesn’t know stuff about, even when it encounters those situations, it’s going to stay aligned with you so universality doesn’t get you all the way there, but it definitely feels like it’s getting you quite a bit.

Buck Shlegeris: That’s really interesting to hear you phrase it that way. I guess I would have thought of universality as a subset of robustness. I’m curious what you think of that first.

Rohin Shah: I definitely think you could use universality to achieve a subset of robustness. Maybe I would say universality is a subset of interpretability.

Buck Shlegeris: Yeah, and I care about interpretability as a subset of robustness basically, or as a subset of inner alignment, which is pretty close to robustness in my mind. The other thing I would say is you were saying there that one difference between universality and robustness is that universality only tells you why the agent did the thing it currently did, and this doesn’t suffice to tell us about the situations that the agent isn’t currently in. One really nice thing though is that if the agent is only acting a particular way because it wants you to trust it, that’s a fact about its current behavior that you will know, and so if you have the universality property, your overseer just knows your agent is trying to deceive it. Which seems like it would be incredibly great and would resolve like half of my problem with safety if you had it.

Rohin Shah: Yeah, that seems right. The case that universality doesn’t cover is when your AI system is initially not deceptive, but then at some point in the future it’s like, ‘Oh my God, now it’s possible to go and build Dyson spheres or something, but wait, in this situation probably I should be doing this other thing and humans won’t like that. Now I better deceive humans’. The transition into deception would have to be a surprise in some sense even to the AI system.

Buck Shlegeris: Yeah, I guess I’m just not worried about that. Suppose I have this system which is as smart as a reasonably smart human or 10 reasonably smart humans, but it’s not as smart as the whole world. If I can just ask it what its best sense about how aligned it is, is? And if I can trust its answer? I don’t know man, I’m pretty okay with systems that think they’re aligned, answering that question honestly.

Rohin Shah: I think I somewhat agree. I like this reversal where I’m the pessimistic one.

Buck Shlegeris: Yeah me too. I’m like, “look, system, I want you to think as hard as you can to come up with the best arguments you can come up with for why you are misaligned, and the problems with you.” And if I just actually trust the system to get this right, then the bad outcomes I get here are just pure accidents. I just had this terrible initialization of my neural net parameters, such that I had this system that honestly believed that it was going to be aligned. And then as it got trained more, this suddenly changed and I couldn’t do anything about it. I don’t quite see the story for how this goes super wrong. It seems a lot less bad than the default situation.

Rohin Shah: Yeah. I think the story I would tell is something like, well, if you look at humans, they’re pretty wrong about what their preferences will be in the future. For example, there’s this trope of how teenagers fall in love and then fall out of love, but when they’re in love, they swear undying oaths to each other or something. To the extent that is true, that seems like the sort of failure that could lead to x-risk if it also happened with AI systems.

Buck Shlegeris: I feel pretty optimistic about all the garden-variety approaches to solving this. Teenagers were not selected very hard on accuracy of their undying oaths. And if you instead had accuracy of self-model as a key feature you were selecting for in your AI system, plausibly you’ll just be way more okay.

Rohin Shah: Yeah. Maybe people could coordinate well on this. I feel less good about people coordinating on this sort of problem.

Buck Shlegeris: For what it’s worth, I think there are coordination problems here and I feel like my previous argument about why coordination is hard and won’t happen by default also probably applies to us not being okay. I’m not sure how this all plays out. I’d have to think about it more.

Rohin Shah: Yeah. I think it’s more like this is a subtle and non-obvious problem, which by hypothesis doesn’t happen in the systems you actually have and only happens later and those are the sorts of problems I’m like, Ooh, not sure if we can deal with those ones, but I agree that there’s a good chance that there’s just not a problem at all in the world where we already have universality and checked all the obvious stuff.

Buck Shlegeris: Yeah. I would like to say universality is one of my other favorite areas of AI alignment research, in terms of how happy I’d be if it worked out really well.

Lucas Perry: All right, so let’s see if we can slightly pick up the pace here. Moving forward and starting with interpretability.

Rohin Shah: Yeah, so I mean I think we’ve basically discussed interpretability already. Universality is a specific kind of interpretability, but the case for interpretability is just like, sure seems like it would be good if you could understand what your AI systems are doing. You could then notice when they’re not aligned, and fix that somehow. It’s a pretty clear cut case for a thing that would be good if we achieved it and it’s still pretty uncertain how likely we are to be able to achieve it.

Lucas Perry: All right, so let’s keep it moving and let’s hit impact regularization now.

Rohin Shah: Yeah, impact regularization in particular is one of the ideas that are not trying to align the AI system but are instead trying to say, well, whatever AI system we build, let’s make sure it doesn’t cause a catastrophe. It doesn’t lead to extinction or existential risk. What it hopes to do is say, all right, AI system, do whatever it is you wanted to do. I don’t care about that. Just make sure that you don’t have a huge impact upon the world.

Whatever you do, keep your impact not too high. And so there’s been a lot of work on this in recent years there’s been relative reachability, attainable utility preservation, and I think in general the sense is like, wow, it’s gone quite a bit further than people expected it to go. I think it definitely does prevent you from doing very, very powerful things of the sort, like if you wanted to stop all competing AI projects from ever being able to build AGI, that doesn’t seem like the sort of thing you can do with an impact regularized AI system, but it sort of seems plausible that you could prevent convergent instrumental sub goals using impact regularization. Where AI systems that are trying to steal resources and power from humans, you could imagine that you’d say, hey, don’t do that level of impact, you can still have the level of impact of say running a company or something like that.

Buck Shlegeris: My take on all this is that I’m pretty pessimistic about all of it working. I think that impact regularization or whatever is a non-optimal point on the capabilities / alignment trade off or something, in terms of safety you’re getting for how much capability you’re sacrificing. My basic a problem here is basically analogous to my problem with value learning, where I think we’re trying to take these extremely essentially fuzzy concepts and then factor our agent through these fuzzy concepts like impact, and basically the thing that I imagine happening is any impact regularization strategy you try to employ, if your AI is usable, will end up not helping with its alignment. For any definition of impacts you come up with, it’ll end up doing something which gets around that. Or it’ll make your AI system completely useless, is my basic guess as to what happens.

Rohin Shah: Yeah, so I think again in this setting, if you formalize it and then say, consider the optimal agent. Yeah, that can totally get around your impact penalty, but in practice it sure seems like, what you want to do is say this convergent instrumental subgoal stuff, don’t do any of that. Continue to do things that are normal in regular life. And those seem like pretty distinct categories. Such that I would not be shocked if we could actually distinguish between the two.

Buck Shlegeris: It sounds like the main benefit you’re going for is trying to make your AI system not do insane, convergent, instrumental sub-goal style stuff. So another approach I can imagine taking here would be some kind of value learning or something, where you’re asking humans for feedback on whether plans are insanely convergent, instrumental sub-goal style, and just not doing the things which, when humans are asked to rate how sketchy the plans are the humans rate as sufficiently sketchy? That seems like about as good a plan. I’m curious what you think.

Rohin Shah: The idea of power as your attainable utility across a wide variety of utility functions seems like a pretty good formalization to me. I think in the worlds where I actually buy a formalization, I tend to expect the formalization to work better. I do think the formalization is not perfect. Most notably with the current formalization of power, your power never changes if you have extremely good beliefs. Your notion, you’re just like, I always have the same power because I’m always able to do the same things and you never get surprised, so maybe I agree with you because I think the current formalization is not good enough.  (The strike through section has been redacted by Rohin. It’s incorrect and you can see why here.) Yeah, I think I agree with you but I could see it going either way.

Buck Shlegeris: I could be totally wrong about this, and correct me if I’m wrong, my sense is that you have to be able to back out the agent’s utility function or its models of the world. Which seems like it’s assuming a particular path for AI development which doesn’t seem to me particularly likely.

Rohin Shah: I definitely agree with that for all the current methods too.

Buck Shlegeris: So it’s like: assume that we have already perfectly solved our problems with universality and robustness and transparency and whatever else. I feel like you kind of have to have solved all of those problems before you can do this, and then you don’t need it or something.

Rohin Shah: I don’t think I agree with that. I definitely agree that the current algorithms that people have written assume that you can just make a change to the AI’s utility function. I don’t think that’s what even their proponents would suggest as the actual plan.

Buck Shlegeris: What is the actual plan?

Rohin Shah: I don’t actually know what their actual plan would be, but one plan I could imagine is figure out what exactly the conceptual things we have to do with impact measurement are, and then whatever method we have for building AGI, probably there’s going to be some part which is specify the goal and then in the specify goal part, instead of just saying pursue X, we want to say pursue X without changing your ability to pursue Y, and Z and W, and P, and Q.

Buck Shlegeris: I think that that does not sound like a good plan. I don’t think that we should expect our AI systems to be structured that way in the future.

Rohin Shah: Plausibly we have to do this with natural language or something.

Buck Shlegeris: It seems very likely to me that the thing you do is reinforcement learning where at the start of the episode you get a sentence of English which is telling you what your goal is and then blah, blah, blah, blah, blah, and this seems like a pretty reasonable strategy for making powerful and sort of aligned AI. Aligned enough to be usable for things that aren’t very hard. But you just fundamentally don’t have access to the internal representations that the AI is using for its sense of what belief is, and stuff like that. And that seems like a really big problem.

Rohin Shah: I definitely see this as more of an outer alignment thing, or like an easier to specify outer alignment type thing than say, IDA is that type stuff.

Buck Shlegeris: Okay, I guess that makes sense. So we’re just like assuming we’ve solved all the inner alignment problems?

Rohin Shah: In the story so far yeah, I think all of the researchers who actually work on this haven’t thought much about inner alignment.

Buck Shlegeris: My overall summary is that I really don’t like this plan. I feel like it’s not robust to scale. As you were saying Rohin, if your system gets more and more accurate beliefs, stuff breaks. It just feels like the kind of thing that doesn’t work.

Rohin Shah: I mean, it’s definitely not conceptually neat and elegant in the sense of it’s not attacking the underlying problem. And in a problem setting where you expect adversarial optimization type dynamics, conceptual elegance actually does count for quite a lot in whether or not you believe your solution will work.

Buck Shlegeris: I feel it’s like trying to add edge detectors to your image classifiers to make them more adversarily robust or something, which is backwards.

Rohin Shah: Yeah, I think I agree with that general perspective. I don’t actually know if I’m more optimistic than you. Maybe I just don’t say… Maybe we’d have the same uncertainty distributions and you just say yours more strongly or something.

Lucas Perry: All right, so then let’s just move a little quickly through the next three, which are causal modeling, oracles, and decision theory.

Rohin Shah: Yeah, I mean, well decision theory, MIRI did some work on it. I am not the person to ask about it, so I’m going to skip that one. Even if you look at the long version, I’m just like, here are some posts. Good luck. So causal modeling, I don’t fully understand what the overall story is here but the actual work that’s been published is basically what we can do is we can take potential plans or training processes for AI systems. We can write down causal models that tell us how the various pieces of the training system interact with each other and then using algorithms developed for causal models we can tell when an AI system would have an incentive to either observe or intervene on an underlying variable.

One thing that came out of this was that you can build a model-based reinforcement learner that doesn’t have any incentive to wire head as long as when it makes its plans, the plans are evaluated by the current reward function as opposed to whatever future reward function it would have. And that was explained using this framework of causal modeling. Oracles, Oracles are basically the idea that we can just train an AI system to just answer questions, give it a question and it tries to figure out the best answer it can to that question, prioritizing accuracy.

One worry that people have recently been talking about is the predictions that the Oracle makes then affect the world, which can affect whether or not the prediction was correct. Like maybe if I predict that I will go to bed at 11 then I’m more likely to actually go to bed at 11 because I want my prediction to come true or something and so then the Oracles can still “choose” between different self confirming predictions and so that gives them a source of agency and one way that people want to avoid this is using what are called counter-factual Oracles where you set up the training, such that the Oracles are basically making predictions under the assumption that their predictions are not going to influence the future.

Lucas Perry: Yeah, okay. Oracles seem like they just won’t happen. There’ll be incentives to make things other than Oracles and that Oracles would even be able to exert influence upon the world in other ways.

Rohin Shah: Yeah, I think I agree that Oracles do not seem very competitive.

Lucas Perry: Let’s do forecasting now then.

Rohin Shah: So the main sub things within forecasting one, there’s just been a lot of work recently on actually building good forecasting technology. There has been an AI specific version of Metaculus that’s been going on for a while now. There’s been some work at the Future of Humanity Institute on building better tools for working with probability distributions under recording and evaluating forecasts. There was an AI resolution council where basically now you can make forecasts about what this particular group of people will think in five years or something like that, which is much easier to operationalize than most other kinds of forecasts. So this helps with constructing good questions. On the actual object level, I think there are two main things. One is that it became increasingly more obvious in the past two years that AI progress currently is being driven by larger and larger amounts of compute.

It totally could be driven by other things as well, but at the very least, compute is a pretty important factor. And then takeoff speeds. So there’s been this long debate in the AI safety community over whether — to take the extremes, whether or not we should expect that AI capabilities will see a very sharp spike. So initially, your AI capabilities are improving by like one unit a year, maybe then with some improvements it got to two units a year and then for whatever reason, suddenly they’re now at 20 units a year or a hundred units a year and they just swoop way past what you would get by extrapolating past trends, and so that’s what we might call a discontinuous takeoff. If you predict that that won’t happen instead you’ll get AI that’s initially improving at one unit per year. Then maybe two units per year, maybe three units per year. Then five units per year, and the rate of progress continually increases. The world’s still gets very, very crazy, but in a sort of gradual, continuous way that would be called a continuous takeoff.

Basically there were two posts that argued pretty forcefully for continuous takeoff back in, I want to say February of 2018, and this at least made me believe that continuous takeoff was more likely. Sadly, we just haven’t actually seen much defense of the other side of the view since then. Even though we do know that there definitely are people who still believe the other side, that there will be a discontinuous takeoff.

Lucas Perry: Yeah so what are both you guys’ views on them?

Buck Shlegeris: Here are a couple of things. One is that I really love the operationalization of slow take off or continuous take off that Paul provided in his post, which was one of the ones Rohin was referring to from February 2018. He says, “by slow takeoff, I mean that there is a four year doubling of the economy before there is a one year doubling of the economy.” As in, there’s a period of four years over which world GDP increases by a factor four, after which there is a period of one year. As opposed to a situation where the first one-year doubling happens out of nowhere. Currently, doubling times for the economy are on the order of like 20 years, and so a one year doubling would be a really big deal. The way that I would phrase why we care about this, is because worlds where we have widespread, human level AI feel like they have incredibly fast economic growth. And if it’s true that we expect AI progress to increase gradually and continuously, then one important consequence of this is that by the time we have human level AI systems, the world is already totally insane. A four year doubling would just be crazy. That would be economic growth drastically higher than economic growth currently is.

This means it would be obvious to everyone who’s paying attention that something is up and the world is radically changing in a rapid fashion. Another way I’ve been thinking about this recently is people talk about transformative AI, by which they mean AI which would have at least as much of an impact on the world as the industrial revolution had. And it seems plausible to me that octopus level AI would be transformative. Like suppose that AI could just never get better than octopus brains. This would be way smaller of a deal than I expect AI to actually be, but it would still be a massive deal, and would still possibly lead to a change in the world that I would call transformative. And if you think this is true, and if you think that we’re going to have octopus level AI before we have human level AI, then you should expect that radical changes that you might call transformative have happened by the time that we get to the AI alignment problems that we’ve been worrying about. And if so, this is really big news.

When I was reading about this stuff when I was 18, I was casually imagining that the alignment problem is a thing that some people have to solve while they’re building an AGI in their lab while the rest of the world’s ignoring them. But if the thing which is actually happening is the world is going insane around everyone, that’s a really important difference.

Rohin Shah: I would say that this is probably the most important contested question in AI alignment right now. Some consequences of it are in a gradual or continuous takeoff world you expect that by the time we get to systems that can pose an existential risk. You’ve already had pretty smart systems that have been deployed in the real world. They probably had some failure modes. Whether or not we call them alignment failure modes or not is maybe not that important. The point is people will be aware that AI systems can fail in weird ways, depending on what sorts of failures you expect, you might expect this to lead to more coordination, more involvement in safety work. You might also be more optimistic about using testing and engineering styles of approaches to the problem which rely a bit more on trial and error type of reasoning because you actually will get a chance to see errors before they happen at a super intelligent existential risk causing mode. There are lots of implications of this form that pretty radically change which alignment plans you think are feasible.

Buck Shlegeris: Also, it’s pretty radically changed how optimistic you are about this whole AI alignment situation, at the very least, people who are very optimistic about AI alignment causing relatively small amounts of existential risk. A lot of the reason for this seems to be that they think that we’re going to get these warning shots where before we have superintelligent AI, we have sub-human level intelligent AI with alignment failures like the cashier Rohin was talking about earlier. And then people start caring about AI alignment a lot more. So optimism is also greatly affected by what you think about this.

I’ve actually been wanting to argue with people about this recently. I wrote a doc last night where I was arguing that even in gradual takeoff worlds, we should expect a reasonably high probability of doom if we can’t solve the AI alignment problem. And I’m interested to have this conversation in more detail with people at some point. But yeah, I agree with what Rohin said.

Overall on takeoff speeds, I guess I still feel pretty uncertain. It seems to me that currently, what we can do with AI, like AI capabilities are increasing consistently, and a lot of this comes from applying relatively non-mindblowing algorithmic ideas to larger amounts of compute and data. And I would be kind of surprised if you can’t basically ride this wave away until you have transformative AI. And so if I want to argue that we’re going to have fast takeoffs, I kind of have to argue that there’s some other approach you can take which lets you build AI without having to go along that slow path, which also will happen first. And I guess I think it’s kind of plausible that that is what’s going to happen. I think that’s what you’d have to argue for if you want to argue for a fast take off.

Rohin Shah: That all seems right to me. I’d be surprised if, out of nowhere, we saw a new AI approach suddenly started working and overtook deep learning. You also have to argue that it then very quickly reaches human level AI, which would be quite surprising, right? In some sense, it would have to be something completely novel that we failed to think about in the last 60 years. We’re putting in way more effort now than we were in the last 60 years, but then counter counterpoint is that all of that extra effort is going straight into deep learning. It’s not really searching for completely new paradigm-shifting ways to get to AGI.

Buck Shlegeris: So here’s how I’d make that argument. Perhaps a really important input into a field like AI, is the number of really smart kids who have been wanting to be an AI researcher since they were 16 because they thought that it’s the most important thing in the world. I think that in physics, a lot of the people who turn into physicists have actually wanted to be physicists forever. I think the number of really smart kids who wanted to be AI researchers forever has possibly gone up by a factor of 10 over the last 10 years, it might even be more. And there just are problems sometimes, that are bottle necked on that kind of a thing, probably. And so it wouldn’t be totally shocking to me, if as a result of this particular input to AI radically increasing, we end up in kind of a different situation. I haven’t quite thought through this argument fully.

Rohin Shah: Yeah, the argument seems plausible. There’s a large space of arguments like this. I think even after that, then I’ve started questioning, “Okay, we get a new paradigm. The same arguments apply to that paradigm?” Not as strongly. I guess not the arguments you were saying about compute going up over time, but the arguments given in the original slow takeoff posts which were people quickly start taking the low-hanging fruit and then move on. When there’s a lot of effort being put into getting some property, you should expect that easy low-hanging fruit is usually just already taken, and that’s why you don’t expect discontinuities. Unless the new idea just immediately rockets you to human-level AGI, or x-risk causing AGI, I think the same argument would pretty quickly start applying to that as well.

Buck Shlegeris: I think it’s plausible that you do get rocketed pretty quickly to human-level AI. And I agree that this is an insane sounding claim.

Rohin Shah: Great. As long as we agree on that.

Buck Shlegeris: Something which has been on my to-do list for a while, and something I’ve been doing a bit of and I’d be excited for someone else doing more of, is reading the history of science and getting more of a sense of what kinds of things are bottlenecked by what, where. It could lead me to be a bit less confused about a bunch of this stuff. AI Impacts has done a lot of great work cataloging all of the things that aren’t discontinuous changes, which certainly is a strong evidence to me against my claim here.

Lucas Perry: All right. What is the probability of AI-induced existential risk?

Rohin Shah: Unconditional on anything? I might give it 1 in 20. 5%.

Buck Shlegeris: I’d give 50%.

Rohin Shah: I had a conversation with AI Impacts that went into this in more detail and partially just anchored on the number I gave there, which was 10% conditional on no intervention from longtermists, I think the broad argument is really just the one that Buck and I were disagreeing about earlier, which is to what extent will society be incentivized to solve the problem? There’s some chance that the first thing we try just works and we don’t even need to solve any sort of alignment problem. It might just be fine. This is not implausible to me. Maybe that’s 30% or something.

Most of the remaining probability comes from, “Okay, the alignment problem is a real problem. We need to deal with it.” It might be very easy in which case we can just solve it straight away. That might be the case. That doesn’t seem that likely to me if it was a problem at all. But what we will get is a lot of these warning shots and people understanding the risks a lot more as we get more powerful AI systems. This estimate is also conditional on gradual takeoff. I keep forgetting to say that, mostly because I don’t know what probability I should put on discontinuous takeoff.

Lucas Perry: So is 5% with longtermist intervention, increasing to 10% if fast takeoff?

Rohin Shah: Yes, but still with longtermist intervention. I’m pretty pessimistic on fast takeoff, but my probability assigned to fast takeoff is not very high. In a gradual takeoff world, you get a lot of warning shots. There will just generally be awareness of the fact that the alignment problem is a real thing and you won’t have the situation you have right now of people saying this thing about worrying about superintelligent AI systems not doing what we want is totally bullshit. That won’t be a thing. Almost everyone will not be saying that anymore, in the version where we’re right and there is a problem. As a result, people will not want to build AI systems that are going to kill them. People tend to be pretty risk averse in my estimation of the world, which Buck will probably disagree with. And as a result, you’ll get a lot of people trying to actually work on solving the alignment problem. There’ll be some amount of global coordination which will give us more time to solve the alignment problem than we may otherwise have had. And together, these forces mean that probably we’ll be okay.

Buck Shlegeris: So I think my disagreements with Rohin are basically that I think fast takeoffs are more likely. I basically think there is almost surely a problem. I think that alignment might be difficult, and I’m more pessimistic about coordination. I know I said four things there, but I actually think of this as three disagreements. I want to say that “there isn’t actually a problem” is just a kind of “alignment is really easy to solve.” So then there’s three disagreements. One is gradual takeoff, another is difficulty of solving competitive prosaic alignment, and another is how good we are at coordination.

I haven’t actually written down these numbers since I last changed my mind about a lot of the inputs to them, so maybe I’m being really dumb. I guess, it feels to me that in fast takeoff worlds, we are very sad unless we have competitive alignment techniques, and so then we’re just only okay if we have these competitive alignment techniques. I guess I would say that I’m something like 30% on us having good competitive alignment techniques by the time that it’s important, which incidentally is higher than Rohin I think.

Rohin Shah: Yeah, 30 is totally within the 25th to 75th interval on the probability, which is a weird thing to be reporting. 30 might be my median, I don’t know.

Buck Shlegeris: To be clear, I’m not just including the outer alignment proportion here, which is what we were talking about before with IDA. I’m also including the inner alignment.

Rohin Shah: Yeah, 30% does seem a bit high. I think I’m a little more pessimistic.

Buck Shlegeris: So I’m like 30% that we can just solve the AI alignment problem in this excellent way, such that anyone who wants to can have very little extra cost and then make AI systems that are aligned. I feel like in worlds where we did that, it’s pretty likely that things are reasonably okay. I think that the gradual versus fast takeoff isn’t actually enormously much of a crux for me because I feel like in worlds without competitive alignment techniques and gradual takeoff, we still have a very high probability of doom. And I think that comes down to disagreements about coordination. So maybe the main important disagreement between Rohin and I is, actually how well we’ll be able to coordinate, or how strongly individual incentives will be for alignment.

Rohin Shah: I think there are other things. The reason I feel a bit more pessimistic than you in the fast takeoff world is just solving problems in advance just is really quite difficult and I really like the ability to be able to test techniques on actual AI systems. You’ll have to work with less powerful things. At some point, you do have to make the jump to more powerful things. But, still, being able to test on the less powerful things, that’s so good, so much safety from there.

Buck Shlegeris: It’s not actually clear to me that you get to test the most important parts of your safety techniques. So I think that there are a bunch of safety problems that just do not occur on dog-level AIs, and do occur on human-level AI. If there are three levels of AI, there’s a thing which is as powerful as a dog, there’s a thing which is as powerful as a human, and there’s a thing which is as powerful as a thousand John von Neumanns. In gradual takeoff world, you have a bunch of time in both of these two milestones, maybe. I guess it’s not super clear to me that you can use results on less powerful systems as that much evidence about whether your safety techniques work on drastically more powerful systems. It’s definitely somewhat helpful.

Rohin Shah: It depends what you condition on in your difference between continuous takeoff and discontinuous takeoff to say which one of them happens faster. I guess the delta between dog and human is definitely longer in gradual takeoff for sure. Okay, if that’s what you were saying, yep, I agree with that.

Buck Shlegeris: Yeah, sorry, that’s all I meant.

Rohin Shah: Cool. One thing I wanted to ask is when you say dog-level AI assistant, do you mean something like a neural net that if put in a dog’s body replacing its brain would do about as well as a dog? Because such a neural net could then be put in other environments and learn to become really good at other things, probably superhuman at many things that weren’t in the ancestral environment. Do you mean that sort of thing?

Buck Shlegeris: Yeah, that’s what I mean. Dog-level AI is probably much better than GPT2 at answering questions. I’m going to define something as dog-level AI, if it’s about as good as a dog at things which I think dogs are pretty heavily optimized for, like visual processing or motor control in novel scenarios or other things like that, that I think dogs are pretty good at.

Rohin Shah: Makes sense. So I think in that case, plausibly, dog-level AI already poses an existential risk. I can believe that too.

Buck Shlegeris: Yeah.

Rohin Shah: The AI cashier example feels like it could totally happen probably before a dog-level AI. You’ve got all of the motivation problems already at that point of the game, and I don’t know what problems you expect to see beyond then.

Buck Shlegeris: I’m more talking about whether you can test your solutions. I’m not quite sure how to say my intuitions here. I feel like there are various strategies which work for corralling dogs and which don’t work for making humans do what you want. In as much as your alignment strategy is aiming at a flavor of problem that only occurs when you have superhuman things, you don’t get to test that either way. I don’t think this is a super important point unless you think it is. I guess I feel good about moving on from here.

Rohin Shah: Mm-hmm (affirmative). Sounds good to me.

Lucas Perry: Okay, we’ve talked about what you guys have called gradual and fast takeoff scenarios, or continuous and discontinuous. Could you guys put some probabilities down on the likelihood of, and stories that you have in your head, for fast and slow takeoff scenarios?

Rohin Shah: That is a hard question. There are two sorts of reasoning I do about probabilities. One is: use my internal simulation of whatever I’m trying to predict, internally simulate what it looks like, whether it’s by my own models, is it likely? How likely is it? At what point would I be willing to bet on it. Stuff like that. And then there’s a separate extra step where I’m like, “What do other people think about this? Oh, a lot of people think this thing that I assigned one percent probability to is very likely. Hmm, I should probably not be saying one percent then.” I don’t know how to do that second part for, well, most things but especially in this setting. So I’m going to just report Rohin’s model only, which will predictably be understating the probability for fast takeoff in that if someone from MIRI were to talk to me for five hours, I would probably say a higher number for the probability of fast takeoff after that, and I know that that’s going to happen. I’m just going to ignore that fact and report my own model anyway.

On my own model, it’s something like in worlds where AGI happens soon, like in the next couple of decades, then I’m like, “Man, 95% on gradual take off.” If it’s further away, like three to five decades, then I’m like, “Some things could have changed by then, maybe I’m 80%.” And then if it’s way off into the future and centuries, then I’m like, “Ah, maybe it’s 70%, 65%.” The reason it goes down over time is just because it seems to me like if you want to argue for discontinuous takeoff, you need to posit that there’s some paradigm change in how AI progress is happening and that seems more likely the further in the future you go.

Buck Shlegeris: I feel kind of surprised that you get so low, like to 65% or 70%. I would have thought that those arguments are a strong default and then maybe at the moment where in a position that seems particularly gradual takeoff-y, but I would have thought that you over time get to 80% or something.

Rohin Shah: Yeah. Maybe my internal model is like, “Holy shit, why do these MIRI people keep saying that discontinuous takeoff is so obvious.” I agree that the arguments in Paul’s posts feel very compelling to me and so maybe I should just be more confident in them. I think saying 80%, even in centuries is plausibly a correct answer.

Lucas Perry: So, Rohin, is the view here that since compute is the thing that’s being leveraged to make most AI advances that you would expect that to be the mechanism by which that continues to happen in the future and we have some certainty over how compute continues to change into the future? Whereas things that would be leading to a discontinuous takeoff would be world-shattering, fundamental insights into algorithms that would have powerful recursive self-improvement, which is something you wouldn’t necessarily see if we just keep going this leveraging compute route?

Rohin Shah: Yeah, I think that’s a pretty good summary. Again, on the backdrop of the default argument for this is people are really trying to build AGI. It would be pretty surprising if there is just this really important thing that everyone had just missed.

Buck Shlegeris: It sure seems like in machine learning when I look at the things which have happened over the last 20 years, all of them feel like the ideas are kind of obvious or someone else had proposed them 20 years earlier. ConvNets were proposed 20 years before they were good on ImageNet, and LSTMs were ages before they were good for natural language, and so on and so on and so on. Other subjects are not like this, like in physics sometimes they just messed around for 50 years before they knew what was happening. I don’t know, I feel confused how to feel about the fact that in some subjects, it feels like they just do suddenly get better at things for reasons other than having more compute.

Rohin Shah: I think physics, at least, was often bottlenecked by measurements, I want to say.

Buck Shlegeris: Yes, so this is one reason I’ve been interested in history of science recently, but there are certainly a bunch of things. People were interested in chemistry for a long time and it turns out that chemistry comes from quantum mechanics and you could, theoretically, have guessed quantum mechanics 70 years earlier than people did if you were smart enough. It’s not that complicated a hypothesis to think of. Or relativity is the classic example of something which could have been invented 50 years earlier. I don’t know, I would love to learn more about this.

Lucas Perry: Just to tie this back to the question, could you give your probabilities as well?

Buck Shlegeris: Oh, geez, I don’t know. Honestly, right now I feel like I’m 70% gradual takeoff or something, but I don’t know. I might change my mind if I think about this for another hour. And there’s also theoretical arguments as well for why most takeoffs are gradual, like the stuff in Paul’s post. The easiest summary is, before someone does something really well, someone else does it kind of well in cases where a lot of people are trying to do the thing.

Lucas Perry: Okay. One facet of this, that I haven’t heard discussed, is recursive self-improvement, and I’m confused about where that becomes the thing that affects whether it’s discontinuous or continuous. If someone does something kind of well before something does something really well, if recursive self-improvement is a property of the thing being done kind of well, is it just kind of self-improving really quickly, or?

Buck Shlegeris: Yeah. I think Paul’s post does a great job of talking about this exact argument. I think his basic claim is, which I find pretty plausible, before you have a system which is really good at self-improving, you have a system which is kind of good at self-improving, if it turns out to be really helpful to have a system be good at self-improving. And as soon as this is true, you have to posit an additional discontinuity.

Rohin Shah: One other thing I’d note is that humans are totally self improving. Productivity techniques, for example, are a form of self-improvement. You could imagine that AI systems might have advantages that humans don’t, like being able to read their own weights and edit them directly. How much of an advantage this gives to the AI system, unclear. Still, I think then I just go back to the argument that Buck already made, which is at some point you get to an AI system that is somewhat good at understanding its weights and figuring out how to edit them, and that happens before you get the really powerful ones. Maybe this is like saying, “Well, you’ll reach human levels of self-improvement by the time you have rat-level AI or something instead of human-level AI,” which argues that you’ll hit this hyperbolic point of the curve earlier, but it still looks like a hyperbolic curve that’s still continuous at every point.

Buck Shlegeris: I agree.

Lucas Perry: I feel just generally surprised about your probabilities on continuous takeoff scenarios that they’d be slow.

Rohin Shah: The reason I’m trying to avoid the word slow and fast is because they’re misleading. Slow takeoff is not slow in calendar time relative to fast takeoff. The question is, is there a spike at some point? Some people, upon reading Paul’s posts are like, “Slow takeoff is faster than fast takeoff.” That’s a reasonably common reaction to it.

Buck Shlegeris: I would put it as slow takeoff is the claim that things are insane before you have the human-level AI.

Rohin Shah: Yeah.

Lucas Perry: This seems like a helpful perspective shift on this takeoff scenario question. I have not read Paul’s post. What is it called so that we can include it in the page for this podcast?

Rohin Shah: It’s just called Takeoff Speeds. Then the corresponding AI Impacts post is called Will AI See Discontinuous Progress?, I believe.

Lucas Perry: So if each of you guys had a lot more reach and influence and power and resources to bring to the AI alignment problem right now, what would you do?

Rohin Shah: I get this question a lot and my response is always, “Man, I don’t know.” It seems hard to scalably use people right now for AI risk. I can talk about which areas of research I’d like to see more people focus on. If you gave me people where I’m like, “I trust your judgment on your ability to do good conceptual work” or something, where would I put them? I think a lot of it would be on making good robust arguments for AI risk. I don’t think we really have them, which seems like kind of a bad situation to be in. I think I would also invest a lot more in having good introductory materials, like this review, except this review is a little more aimed at people who are already in the field. It is less aimed at people who are trying to enter the field. I think we just have pretty terrible resources for people coming into the field and that should change.

Buck Shlegeris: I think that our resources are way better than they used to be.

Rohin Shah: That seems true.

Buck Shlegeris: In the course of my work, I talk to a lot of people who are new to AI alignment about it and I would say that their level of informedness is drastically better now than it was two years ago. A lot of which is due to things like 80,000 hours podcast, and other things like this podcast and the Alignment Newsletter, and so on. I think we just have made it somewhat easier for people to get into everything. The Alignment Forum, having its sequences prominently displayed, and so on.

Rohin Shah: Yeah, you named literally all of the things I would have named. Buck definitely has more information on this than I do. I do not work with people who are entering the field as much. I do think we could be substantially better.

Buck Shlegeris: Yes. I feel like I do have access to resources, not directly but in the sense that I know people at eg Open Philanthropy and the EA Funds  and if I thought there were obvious things they should do, I think it’s pretty likely that those funders would have already made them happen. And I occasionally embark on projects myself that I think are good for AI alignment, mostly on the outreach side. On a few occasions over the last year, I’ve just done projects that I was optimistic about. So I don’t think I can name things that are just shovel-ready opportunities for someone else to do, which is good news because it’s mostly because I think most of these things are already being done.

I am enthusiastic about workshops. I help run with MIRI these AI Risks for Computer Scientists workshops and I ran my own computing workshop with some friends, with kind of a similar purpose, aimed at people who are interested in this kind of stuff and who would like to spend some time learning more about it. I feel optimistic about this kind of project as a way of doing the thing Rohin was saying, making it easier for people to start having really deep thoughts about a lot of AI alignment stuff. So that’s a kind of direction of projects that I’m pretty enthusiastic about. A couple other random AI alignment things I’m optimistic about. I’ve already mentioned that I think there should be an Ought competitor just because it seems like the kind of thing that more work could go into. I agree with Rohin on it being good to have more conceptual analysis of a bunch of this stuff. I’m generically enthusiastic about there being more high quality research done and more smart people, who’ve thought about this a lot, working on it as best as they can.

Rohin Shah: I think the actual bottleneck is good research and not necessarily field building, and I’m more optimistic about good research. Specifically, I am particularly interested in universality, interpretability. I would love for there to be some way to give people who work on AI alignment the chance to step back and think about the high-level picture for a while. I don’t know if people don’t do this because they don’t want to or because they don’t feel like they have the affordance to do so, and I would like the affordance to be there. I’d be very interested in people building models of what AGI systems could look like. Expected utility maximizers are one example of a model that you could have. Maybe we just try to redo evolution. We just create a very complicated, diverse environment with lots of agents going around and in their multi-agent interaction, they develop general intelligence somehow. I’d be interested for someone to take that scenario, flesh it out more, and then talk about what the alignment problem looks like in that setting.

Buck Shlegeris: I would love to have someone get really knowledgeable about evolutionary biology and try and apply analogies of that to AI alignment. I think that evolutionary biology has lots of smart things to say about what optimizers are and it’d be great to have those insights. I think Eliezer sort of did this many years ago. It would be good for more people to do this in my opinion.

Lucas Perry: All right. We’re in the home stretch here. AI timelines. What do you think about the current state of predictions? There’s been surveys that have been done with people giving maybe 50% probability over most researchers at about 2050 or so. What are each of your AI timelines? What’s your probability distribution look like? What do you think about the state of predictions on this?

Rohin Shah: Haven’t looked at the state of predictions in a while. It depends on who was surveyed. I think most people haven’t thought about it very much and I don’t know if I expect their predictions to be that good, but maybe wisdom of the crowds is a real thing. I don’t think about it very much. I mostly use my inside view and talk to a bunch of people. Maybe, median, 30 years from now, which is 2050. So I guess I agree with them, don’t I? That feels like an accident. The surveys were not an input into this process.

Lucas Perry: Okay, Buck?

Buck Shlegeris: I don’t know what I think my overall timelines are. I think AI in the next 10 or 20 years is pretty plausible. Maybe I want to give it something around 50% which puts my median at around 2040. In terms of the state of things that people have said about AI timelines, I have had some really great conversations with people about their research on AI timelines which hasn’t been published yet. But at some point in the next year, I think it’s pretty likely that much better stuff about AI timelines modeling will have been published than has currently been published, so I’m excited for that.

Lucas Perry: All right. Information hazards. Originally, there seemed to be a lot of worry in the community about information hazards and even talking about superintelligence and being afraid of talking to anyone in positions of power, whether they be in private institutions or in government, about the strategic advantage of AI, about how one day it may confer a decisive strategic advantage. The dissonance here for me is that Putin comes out and says that who controls AI will control the world. Nick Bostrom published Superintelligence, which basically says what I already said. Max Tegmark’s Life 3.0 basically also. My initial reaction and intuition is the cat’s out of the bag. I don’t think that echoing this increases risks any further than the risk is already at. But maybe you disagree.

Buck Shlegeris: Yeah. So here are two opinions I have about info hazards. One is: how bad is it to say stuff like that all over the internet? My guess is it’s mildly bad because I think that not everyone thinks those things. I think that even if you could get those opinions as consequences from reading Superintelligence, I think that most people in fact have not read Superintelligence. Sometimes there are ideas where I just really don’t want them to be crystallized common knowledge. I think that, to a large extent, assuming gradual takeoff worlds, it kind of doesn’t matter because AI systems are going to be radically transforming the world inevitably. I guess you can affect how governments think about it, but it’s a bit different there.

The other point I want to make about info hazards is I think there are a bunch of trickinesses with AI safety, where thinking about AI safety makes you think about questions about how AI development might go. I think that thinking about how AI development is going to go occasionally leads to think about things that are maybe, could be, relevant to capabilities, and I think that this makes it hard to do research because you then get scared about talking about them.

Rohin Shah: So I think my take on this is info hazards are real in the sense that there, in fact, are costs to saying specific kinds of information and publicizing them a bit. I think I’ll agree in principle that some kinds of capabilities information has the cost of accelerating timelines. I usually think these are pretty strongly outweighed by the benefits in that it just seems really hard to be able to do any kind of shared intellectual work when you’re constantly worried about what you do and don’t make public. It really seems like if you really want to build a shared understanding within the field of AI alignment, that benefit is worth saying things that might be bad in some other ways. This depends on a lot of background facts that I’m not going to cover here but, for example, I probably wouldn’t say the same thing about bio security.

Lucas Perry: Okay. That makes sense. Thanks for your opinions on this. So at the current state in time, do you guys think that people should be engaging with people in government or in policy spheres on questions of AI alignment?

Rohin Shah: Yes, but not in the sense of we’re worried about when AGI comes. Even saying things like it might be really bad, as opposed to saying it might kill everybody, seems not great. Mostly on the basis of my model for what it takes to get governments to do things is, at the very least, you need consensus in the field so it seems kind of pointless to try right now. It might even be poisoning the well for future efforts. I think it does make sense to engage with government and policymakers about things that are in fact problems right now. To the extent that you think that recommender systems are causing a lot of problems, I think it makes sense to engage with government about how alignment-like techniques can help with that, especially if you’re doing a bunch of specification learning-type stuff. That seems like the sort of stuff that should have relevance today and I think it would be great if those of us who did specification learning were trying to use it to improve existing systems.

Buck Shlegeris: This isn’t my field. I trust the judgment of a lot of other people. I think that it’s plausible that it’s worth building relationships with governments now, not that I know what I’m talking about. I will note that I basically have only seen people talk about how to do AI governance in the cases where the AI safety problem is 90th percentile easiest. I basically only see people talking about it in the case where the technical safety problem is pretty doable, and this concerns me. I’ve just never seen anyone talk about what you do in a world where you’re as pessimistic as I am, except to completely give up.

Lucas Perry: All right. Wrapping up here, is there anything else that we didn’t talk about that you guys think was important? Or something that we weren’t able to spend enough time on, that you would’ve liked to spend more time on?

Rohin Shah: I do want to eventually continue the conversation with Buck about coordination, but that does seem like it should happen not on this podcast.

Buck Shlegeris: That’s what I was going to say too. Something that I want someone to do is write a trajectory for how AI goes down, that is really specific about what the world GDP is in every one of the years from now until insane intelligence explosion. And just write down what the world is like in each of those years because I don’t know how to write an internally consistent, plausible trajectory. I don’t know how to write even one of those for anything except a ridiculously fast takeoff. And this feels like a real shame.

Rohin Shah: That seems good to me as well. And also the sort of thing that I could not do because I don’t know economics.

Lucas Perry: All right, so let’s wrap up here then. So if listeners are interested in following either of you or seeing more of your blog posts or places where you would recommend they read more materials on AI alignment, where can they do that? We’ll start with you, Buck.

Buck Shlegeris: You can Google me and find my website. I often post things on the Effective Altruism Forum. If you want to talk to me about AI alignment in person, perhaps you should apply to the AI Risks for Computer Scientists workshops run by MIRI.

Lucas Perry: And Rohin?

Rohin Shah: I write the Alignment Newsletter. That’s a thing that you could sign up for. Also on my website, if you Google Rohin Shah Alignment Newsletter, I’m sure I will come up. These are also cross posted to the Alignment Forum, so another thing you can do is go to the Alignment Forum, look up my username and just see things that are there. I don’t know that this is actually the thing that you want to be doing. If you’re new to AI safety and want to learn more about it, I would echo the resources Buck mentioned earlier, which are the 80k podcasts about AI alignment. There are probably on the order of five of these. There’s the Alignment Newsletter. There are the three recommended sequences on the Alignment Forum. Just go to alignmentforum.org and look under recommended sequences. And this podcast, of course.

Lucas Perry: All right. Heroic job, everyone. This is going to be a really good resource, I think. It’s given me a lot of perspective on how thinking has changed over the past year or two.

Buck Shlegeris: And we can listen to it again in a year and see how dumb we are.

Lucas Perry: Yeah. There were lots of predictions and probabilities given today, so it’ll be interesting to see how things are in a year or two from now. That’ll be great. All right, so cool. Thank you both so much for coming on.

End of recorded material

AI Alignment Podcast: On Lethal Autonomous Weapons with Paul Scharre

 Topics discussed in this episode include:

  • What autonomous weapons are and how they may be used
  • The debate around acceptable and unacceptable uses of autonomous weapons
  • Degrees and kinds of ways of integrating human decision making in autonomous weapons 
  • Risks and benefits of autonomous weapons
  • An arms race for autonomous weapons
  • How autonomous weapons issues may matter for AI alignment and long-term AI safety

Timestamps: 

0:00 Intro

3:50 Why care about autonomous weapons?

4:31 What are autonomous weapons? 

06:47 What does “autonomy” mean? 

09:13 Will we see autonomous weapons in civilian contexts? 

11:29 How do we draw lines of acceptable and unacceptable uses of autonomous weapons? 

24:34 Defining and exploring human “in the loop,” “on the loop,” and “out of loop” 

31:14 The possibility of generating international lethal laws of robotics

36:15 Whether autonomous weapons will sanitize war and psychologically distance humans in detrimental ways

44:57 Are persons studying the psychological aspects of autonomous weapons use? 

47:05 Risks of the accidental escalation of war and conflict 

52:26 Is there an arms race for autonomous weapons? 

01:00:10 Further clarifying what autonomous weapons are

01:05:33 Does the successful regulation of autonomous weapons matter for long-term AI alignment considerations?

01:09:25 Does Paul see AI as an existential risk?

 

We hope that you will continue to join in the conversations by following us or subscribing to our podcasts on Youtube, Spotify, SoundCloud, iTunes, Google Play, StitcheriHeartRadio, or your preferred podcast site/application. You can find all the AI Alignment Podcasts here.

You can listen to the podcast above or read the transcript below. 

Lucas Perry: Welcome to the AI Alignment Podcast. I’m Lucas Perry. Today’s conversation is with Paul Scharre and explores the issue of lethal autonomous weapons. And so just what is the relation of lethal autonomous weapons and the related policy and governance issues to AI alignment and long-term AI risk? Well there’s a key question to keep in mind throughout this entire conversation and it’s that: if we cannot establish a governance mechanism as a global community on the concept that we should not let AI make the decision to kill, then how can we deal with more subtle near term issues and eventual long term safety issues about AI systems? This question is aimed at exploring the idea that autonomous weapons and their related governance represent a possibly critical first step on the international cooperation and coordination of global AI issues. If we’re committed to developing beneficial AI and eventually beneficial AGI then how important is this first step in AI governance and what precedents and foundations will it lay for future AI efforts and issues? So it’s this perspective that I suggest keeping in mind throughout the conversation. And many thanks to FLI’s Emilia Javorsky for much help on developing the questions for this podcast. 

Paul Scharre is a Senior Fellow and Director of the Technology and National Security Program at the Center for a New American Security. He is the award-winning author of Army of None: Autonomous Weapons and the Future of War, which won the 2019 Colby Award and was named one of Bill Gates’ top five books of 2018.

Mr. Scharre worked in the Office of the Secretary of Defense (OSD) where he played a leading role in establishing policies on unmanned and autonomous systems and emerging weapons technologies. Mr. Scharre led the DoD working group that drafted DoD Directive 3000.09, establishing the Department’s policies on autonomy in weapon systems. Mr. Scharre also led DoD efforts to establish policies on intelligence, surveillance, and reconnaissance (ISR) programs and directed energy technologies. He was involved in the drafting of policy guidance in the 2012 Defense Strategic Guidance, 2010 Quadrennial Defense Review, and Secretary-level planning guidance. His most recent position was Special Assistant to the Under Secretary of Defense for Policy. Prior to joining the Office of the Secretary of Defense, Mr. Scharre served as a special operations reconnaissance team leader in the Army’s 3rd Ranger Battalion and completed multiple tours to Iraq and Afghanistan.

The Future of Life Institute is a non-profit and this podcast is funded and supported by listeners like you. So if you find what we do on this podcast to be important and beneficial, please consider supporting the podcast by donating at futureoflife.org/donate. If you support any other content creators via services like Patreon, consider viewing a regular subscription to FLI in the same light. You can also follow us on your preferred listening platform, like on Apple Podcasts or Spotify, by searching for us directly or following the links on the page for this podcast found in the description.

And with that, here’s my conversion with Paul Scharre. 

All right. So we’re here today to discuss your book, Army of None, and issues related to autonomous weapons in the 21st century. To start things off here, I think we can develop a little bit of the motivations for why this matters. Why should the average person care about the development and deployment of lethal autonomous weapons?

Paul Scharre: I think the most basic reason is because we all are going to live in the world that militaries are going to be deploying future weapons. Even if you don’t serve in the military, even if you don’t work on issues surrounding say, conflict, this kind of technology could affect all of us. And so I think we all have a stake in what this future looks like.

Lucas Perry: Let’s clarify a little bit more about what this technology actually looks like then. Often in common media, and for most people who don’t know about lethal autonomous weapons or killer robots, the media often portrays it as a terminator like scenario. So could you explain why this is wrong, and what are more accurate ways of communicating with the public about what these weapons are and the unique concerns that they pose?

Paul Scharre: Yes, I mean, the Terminator is like the first thing that comes up because it’s such a common pop culture reference. It’s right there in people’s minds. So I think go ahead and for the listeners, imagine that humanoid robot in the Terminator, and then just throw that away, because that’s not what we’re talking about. Let me make a different comparison. Self-driving cars. We are seeing right now the evolution of automobiles that with each generation of car incorporate more autonomous features: parking, intelligent cruise control, automatic braking. These increasingly autonomous features in cars that are added every single year, a little more autonomy, a little more autonomy, are taking us down at some point in time to a road of having fully autonomous cars that would drive themselves. We have something like the Google car where there’s no steering wheel at all. People are just passengers along for the ride. We’re seeing something very similar happen in the military with each generation of robotic systems and we now have air and ground and undersea robots deployed all around the world in over 100 countries and non state groups around the globe with some form of drones or robotic systems, and with each generation they’re becoming increasingly autonomous.

Now, the issue surrounding autonomous weapons is, what happens when a predator drone has as much autonomy as a self-driving car? What happens when you have a weapon that’s out in the battlefield, and it’s making its own decisions about whom to kill? Is that something that we’re comfortable with? What are the legal and moral and ethical ramifications of this? And the strategic implications? What might they do for the balance of power between nations, or stability among countries? These are really the issues surrounding autonomous weapons, and it’s really about this idea that we might have, at some point of time and perhaps the not very distant future, machines making their own decisions about whom to kill on the battlefield.

Lucas Perry: Could you unpack a little bit more about what autonomy really is or means because it seems to me that it’s more like an aggregation of a bunch of different technologies like computer vision and image recognition, and other kinds of machine learning that are aggregated together. So could you just develop a little bit more about where we are in terms of the various technologies required for autonomy?

Paul Scharre: Yes, so autonomy is not really a technology, it’s an attribute of a machine or of a person. And autonomy is about freedom. It’s the freedom that a machine or a person is given to perform some tasks in some environment for some period of time. As people, we have very little autonomy as children and more autonomy as we grow up, we have different autonomy in different settings. In some work environments, there might be more constraints put on you; what things you can and cannot do. And it’s also environment-specific and task-specific. You might have autonomy to do certain things, but not other things. It’s the same with machines. We’re ultimately talking about giving freedom to machines to perform certain actions under certain conditions in certain environments.

There are lots of simple forms of autonomy that we interact with all the time that we sort of take for granted. A thermostat is a very simple autonomous system, it’s a machine that’s given a freedom to decide… decide, let’s put that in air quotes, because we come back to what it means for machines to decide. But basically, the thermostat is given the ability to turn on and off the heat and air conditioning based on certain parameters that a human sets, a desired temperature, or if you have a programmable thermostat, maybe the desired temperature at certain times a day or days of the week, is a very bounded kind of autonomy. And that’s what we’re talking about for any of these machines. We’re not talking about freewill, or whether the machine develops consciousness. That’s not a problem today, maybe someday, but certainly not with the machines we’re talking about today. It’s a question really of, how much freedom do we want to give machines, or in this case, weapons operating on the battlefield to make certain kinds of choices?

Now we’re still talking about weapons that are designed by people, built by people, launched by people, and put into the battlefields to perform some mission, but there might be a little bit less human control than there is today. And then there are a whole bunch of questions that come along with that, like, is it going to work? Would it be effective? What happens if there are accidents? Are we comfortable with seeding that degree of control over to the machine?

Lucas Perry: You mentioned the application of this kind of technology in the context of battlefields. Is there also consideration and interest in the use of lethal autonomous weapons in civilian contexts?

Paul Scharre: Yes, I mean, I think there’s less energy on that topic. You certainly see less of a poll from the police community. I mean, I don’t really run into people in a police or Homeland Security context, saying we should be building autonomous weapons. Well, you will hear that from militaries. Oftentimes, groups that are concerned about the humanitarian consequences of autonomous weapons will raise that as a concern. There’s both what might militaries do in the battlefield, but then there’s a concern about proliferation. What happens when the technology proliferates, and it’s being used for internal security issues, could be a dictator, using these kinds of weapons to repress the population. That’s one concern. And that’s, I think, a very, very valid one. We’ve often seen one of the last checks against dictators, is when they tell their internal security forces to fire on civilians, on their own citizens. There have been instances where the security forces say, “No, we won’t.” That doesn’t always happen. Of course, tragically, sometimes security forces do attack their citizens. We saw in the massacre in Tiananmen Square that Chinese military troops are willing to murder Chinese citizens. But we’ve seen other instances, certainly in the fall of the Eastern Bloc at the end of the Cold War, that security forces… these are our friends, these are our family. We’re not going to kill them.

And autonomous weapons could take away one of those checks on dictators. So I think that’s a very valid concern. And that is a more general concern about the proliferation of military technology into policing even here in America. We’ve seen this in the last 20 years, is a lot of military tech ends up being used by police forces in ways that maybe isn’t appropriate. And so that’s, I think, a very valid and legitimate sort of concern about… even if this isn’t kind of the intended use, what would that look like and what are the risks that could come with that, and how should we think about those kinds of issues as well?

Lucas Perry: All right. So we’re developing autonomy in systems and there’s concern about how this autonomy will be deployed in context where lethal force or force may be used. So the question then arises and is sort of the question at the heart of lethal autonomous weapons: Where is it that we will draw a line between acceptable and unacceptable uses of artificial intelligence in autonomous weapons or in the military, or in civilian policing? So I’m curious to know how you think about where to draw those lines or that line in particular, and how you would suggest to any possible regulators who might be listening, how to think about and construct lines of acceptable and unacceptable uses of AI.

Paul Scharre: That’s a great question. So I think let’s take a step back first and sort of talk about, what would be the kinds of things that would make uses acceptable or unacceptable. Let’s just talk about the military context just to kind of bound the problem for a second. So in the military context, you have a couple reasons for drawing lines, if you will. One is legal issues, legal concerns. We have a legal framework to think about right and wrong in war. It’s called the laws of war or international humanitarian law. And it lays out a set of parameters for what is acceptable and what… And so that’s one of the places where there has been consensus internationally, among countries that come together at the United Nations through the Convention on Certain Conventional Weapons, the CCW, the process, we’ve had conversations going on about autonomous weapons.

One of the points of consensus among nations is that existing international humanitarian law or the laws of war would apply to autonomous weapons. And that any uses of autonomy in weapons, those weapons have to be used in a manner that complies with the laws of war. Now, that may sound trivial, but it’s a pretty significant point of agreement and it’s one that places some bounds on things that you can or cannot do. So, for example, one of the baseline principles of the laws of war is the principle of distinction. Military forces cannot intentionally target civilians. They can only intentionally target other military forces. And so any use of force these people to comply with this distinction, so right off the bat, that’s a very important and significant one when it comes to autonomous weapons. So if you have to use a weapon that could not be used in a way to comply with this principle of distinction, it would be illegal under the laws war and you wouldn’t be able to build it.

And there are other principles as well, principles about proportionality, and ensuring that any collateral damage that affects civilians or civilian infrastructure is not disproportionate to the military necessity of the target that is being attacked. There are principles about avoiding unnecessary suffering of combatants. Respecting anyone who’s rendered out of combat or the appropriate term is “hors de combat,” who surrendered have been incapacitated and not targeting them. So these are like very significant rules that any weapon system, autonomous weapon or not, has to comply with. And any use of any weapon has to comply with, any use of force. And so that is something that constrains considerably what nations are permitted to do in a lawful fashion. Now do people break the laws of war? Well, sure, that happens. We’re seeing that happen in Syria today, Bashar al-Assad is murdering civilians, there are examples of Rogue actors and non state terrorist groups and others that don’t care about respecting the laws of war. But those are very significant bounds.

Now, one could also say that there are more bounds that we should put on autonomous weapons that might be moral or ethical considerations that exist outside the laws of war, that aren’t written down in a formal way in the laws of war, but they’re still important and I think those often come to the fore with this topic. And there are other ones that might apply in terms of reasons why we might be concerned about stability among nations. But the laws of war, at least a very valuable starting point for this conversation about what is acceptable and not acceptable. I want to make clear, I’m not saying that the laws of war are insufficient, and we need to go beyond them and add in additional constraints. I’m actually not saying that. There are people that make that argument, and I want to give credit to their argument, and not pretend it doesn’t exist. I want the listeners to sort of understand the full scope of arguments about this technology. But I’m not saying myself that’s the case necessarily. But I do think that there are concerns that people raise.

For example, people might say it’s wrong for a machine to decide whom to kill, it’s wrong for a machine to make the decision about life and death. Now I think that’s an interesting argument. Why? Why is it wrong? Is it because we think the machine might get the answer wrong, it might perform not as well as the humans because I think that there’s something intrinsic about weighing the value of life and death that we want humans to do, and appreciating the value of another person’s life before making one of these decisions. Those are all very valid counter arguments that exist in this space.

Lucas Perry: Yes. So thanks for clarifying that. For listeners, it’s important here to clarify the difference where some people you’re saying would find the laws of war to be sufficient in the case of autonomous weapons, and some would not.

Paul Scharre: Yes, I mean, this is a hotly debated issue. I mean, this is in many ways, the crux of the issue surrounding autonomous weapons. I’m going to oversimplify a bit because you have a variety of different views on this, but you certainly have some people whose views are, look, we have a set of structures called the laws of war that tell us what right and wrong looks like and more. And most of the things that people are worried about are already prohibited under the laws of war. So for example, if what you’re worried about is autonomous weapons, running amok murdering civilians, that’s illegal under the laws of war. And so one of the points of pushback that you’ll sometimes get from governments or others to the idea of creating like an ad hoc treaty that would ban autonomous weapons or some class of autonomous weapons, is look, some of the things people worry about like they’re already prohibited under the laws of war, passing another law to say the thing that’s already illegal is now illegal again doesn’t add any value.

There’s group of arguments that says the laws of war dictate effects in the battlefield. So they dictate sort of what the end effect is, they don’t really affect the process. And there’s a line of reasoning that says, that’s fine. The process doesn’t matter. If someday we could use autonomous weapons in a way that was more humane and more precise than people, then we should use them. And just the same way that self-driving cars will someday save lives on roads by avoiding accidents, maybe we could build autonomous weapons that would avoid mistakes in war and accidentally targeting civilians, and therefore we should use them. And let’s just focus on complying better with the laws of war. That’s one school of thought.

Then there’s a whole bunch of reasons why you might say, well, that’s not enough. One reason might be, well, militaries’ compliance with the laws of war. Isn’t that great? Actually, like people talk a good game, but when you look at military practice, especially if the rules for using weapon are kind of convoluted, you have to take a bunch of additional steps in order to use it in a way that’s lawful, that kind of goes out the window in conflict. Real world and tragic historical example of this was experienced throughout the 20th century with landmines where land mines were permitted to be used lawfully, and still are, if you’re not a signatory to the Ottawa Convention, they’re permitted to be used lawfully provided you put in a whole bunch of procedures to make sure that minefields are marked and we know the location of minefields, so they can be demined after conflict.

Now, in practice, countries weren’t doing this. I mean, many of them were just scattering mines from the air. And so we had this horrific problem of millions of mines around the globe persisting after a conflict. The response was basically this global movement to ban mines entirely to say, look, it’s not that it’s inconceivable to use mines in a way that you mean, but it requires a whole bunch of additional efforts, that countries aren’t doing, and so we have to take this weapon away from countries because they are not actually using it in a way that’s responsible. That’s a school of thought with autonomous weapons. Is look, maybe you can conjure up thought experiments about how you can use autonomous weapons in these very specific instances, and it’s acceptable, but once you start any use, it’s a slippery slope, and next thing you know, it’ll be just like landmines all over again, and they’ll be everywhere and civilians will be being killed. And so the better thing to do is to just not let this process even start, and not letting militaries have access to the technology because they won’t use it responsibly, regardless of whether it’s theoretically possible. That’s a pretty reasonable and defensible argument. And there are other arguments too.

One could say, actually, it’s not just about avoiding civilian harm, but there’s something intrinsic about weighing the value of an enemy soldier’s life, that we want humans involved in that process. And that if we took humans away from that process, we’ll be losing something that sure maybe it’s not written down in the laws of war, but maybe it’s not written down because it was always implicit that humans will always be making these choices. And now that it’s decision in front of us, we should write this down, that humans should be involved in these decisions and should be weighing the value of the human life, even an enemy soldier. Because if we give that up, we might give up something that is a constraint on violence and war that holds back some of the worst excesses of violence, we might even can make something about ourselves. And this is, I think, a really tricky issue because there’s a cost to humans making these decisions. It’s a very real cost. It’s a cost in post traumatic stress that soldiers face and moral injury. It’s a cost in lives that are ruined, not just the people that are killed in a battlefield, but the people have to live with that violence afterwards, and the ramifications and even the choices that they themselves make. It’s a cost in suicides of veterans, and substance abuse and destroyed families and lives.

And so to say that we want humans to stay still evolved to be more than responsible for killing, is to say I’m choosing that cost. I’m choosing to absorb and acknowledge and take on the cost of post traumatic stress and moral injury, and also the burdens that come with war. And I think it’s worth reflecting on the fact that the burdens of war are distributed very unequally, not just between combatants, but also on the societies that fight. As a democratic nation in the United States, we make a decision as a country to go to war, through our elected representatives. And yet, it’s a very tiny slice of the population that bears the burden for that war, not just putting themselves at risk, but also carrying the moral burden of that afterwards.

And so if you say, well, I want there to be someone who’s going to live with that trauma for the rest of your life. I think that’s an argument that one can make, but you need to acknowledge that that’s real. And that’s not a burden that we all share equally, it’s a burden we’re placing on young women and men that we send off to fight on our behalf. The flip side is if we didn’t do that, if we fought a war and no one felt the moral burden of killing, no one slept uneasy at night afterwards, what would they say about us as a society? I think these are difficult questions. I don’t have easy answers to that. But I think these are challenging things for us to wrestle with.

Lucas Perry: Yes, I mean, there’s a lot there. I think that was a really good illustration of the different points of views on this. I hadn’t heard or considered much the implications of post traumatic stress. And I think moral burden, you called it that would be a factor in what autonomous weapons would relieve in countries which have the power to develop them. Speaking personally, I think I find the arguments most compelling about the necessity of having human beings integrated in the process of decision making with regards to killing, because if you remove that, then you’re removing the deep aspect of humanity, which sometimes does not follow the laws of war, which we currently don’t have complex enough preference learning techniques and machine learning techniques to actually train autonomous weapon systems in everything that human beings value and care about, and that there are situations where deviating from following the laws of war may be the best thing to do. I’m not sure if you have any thoughts about this, but I think you did a good job of illustrating all the different positions, and that’s just my initial reaction to it.

Paul Scharre: Yes, these are tricky issues. And so I think one of the things I want to try to do for listeners is try to lay out the landscape of what these arguments are, and some of the pros and cons of them because I think sometimes they will often oversimplify on all sides. The other people will be like, well, we should have humans involved in making these decisions. Well, humans involved where? If I get into a self-driving car that has no steering wheel, it’s not true that there’s no human involvement. The type of human involvement has just changed in terms of where it exists. So now, instead of manually driving the car, I’m still choosing the car’s destination, I’m still telling the car where I want to go. You’re going to get into the car and car take me wherever you want to go. So the type of human involvement is changed.

So what kind of human relationship do we want with decisions about life and death in the battlefield? What type of human involvement is right or necessary or appropriate and for what reason? For a legal reason, for a moral reason. These are interesting challenges. We haven’t had to confront anymore. These arguments I think unfairly get simplified on all sides. Conversely, you hear people say things like, it doesn’t matter, because these weapons are going to get built anyway. It’s a little bit overly simplistic in the sense that there are examples of successes in arms control. It’s hard to pull off. There are many examples of failures as well, but there are places where civilized nations have walked back from some technologies to varying degrees of success, whether it’s chemical weapons or biological weapons or other things. So what is success look like in constraining a weapon? Is it no one ever uses the weapon? Is it most nations don’t use it? It’s not used in certain ways. These are complicated issues.

Lucas Perry: Right. So let’s talk a little bit here about integrating human emotion and human reasoning and humanity itself into the autonomous weapon systems and the life or death decisions that they will be making. So hitting on a few concepts here, if you could help explain what people mean when they say human in the loop, and human on the loop, and how this relates to the integration of human control and human responsibility and human accountability in the use of autonomous weapons.

Paul Scharre: Let’s unpack some of this terminology. Broadly speaking, people tend to use the terms human in the loop, on the loop, or out of the loop to refer to semi autonomous weapons human is in the loop, which means that for any really semi autonomous process or system, the machine is taking an action and then it pauses and waits for humans to take a positive action before proceeding. A good example of a human in the loop system is the automated backups on your computer when they require you to push a button to say okay to do the backup now. They’re waiting music in action before proceeding. In a human on the loop system, or one where the supervisor control is one of the human doesn’t have to take any positive action for the system to proceed. The human can intervene, so the human can sit back, and if you want to, you can jump in.

Example of this might be your thermostat. When you’re in a house, you’ve already set the parameters, it’ll turn on the heat and air conditioning on its own, but if you’re not happy with the outcome, you could change it. Now, when you’re out of the house, your thermostat is operating in a fully autonomous fashion in this respect where humans out of the loop. You don’t have any ability to intervene for some period of time. It’s really all about time duration. For supervisory control, how much time does the human have to identify something is wrong and then intervene? So for example, things like the Tesla autopilots. That’s one where the human is in a supervisory control capacity. So the autopilot function in a car, the human doesn’t have to do anything, car’s driving itself, but they can intervene.

The problem with some of those control architectures is the time that you are permitting people to identify that there’s a problem, figure out what’s going on, decide to take action, intervene, really realistic before harm happens. Is it realistic that a human can be not paying attention, and then all of a sudden, identify that the car is in trouble and leap into action to avoid an accident when you’re speeding on the highway 70 miles an hour? And then you can see quite clearly in a number of fatal accidents with these autopilots, that that’s not feasible. People actually aren’t capable of doing that. So you’ve got to think about sort of what is the role of the human in this process? This is not just a semi autonomous or supervised autonomous or fully autonomous process. It’s one where the human is involved in some varying capacity.

And what are we expecting the human to do? Same thing with something that’s fully autonomous. We’re talking about a system that’s operating on its own for some period of time. How long before it checks back in with a person? What information is that person given? And what is their capacity to intervene or how bad could things go wrong when the person is not involved? And when we talk about weapons specifically. There are lots of weapons that operate in a semi autonomous fashion today where the human is choosing the target, but there’s a lot of automation in IDing targets presenting information to people in actually carrying out an attack, once the human has chosen a target, there are many, many weapons that are what the military calls fire and forget weapon, so once it’s launched, it’s not coming back. Those have been widely used for 70 years since World War Two. So that’s not new.

There are a whole bunch of weapons that operate in a supervisory autonomy mode, where humans on the loop. These are generally used in a more limited fashion for immediate localized defense of air bases or ships or ground vehicles defending against air or missile or rocket attack, particularly when the speed of these attacks might overwhelm people’s ability to respond. For humans to be in the loop, for humans to push a button, every time there’s a missile coming in, you could have so many missiles coming in so fast that you have to just simply activate an automatic defensive mode that will shoot down all have the missiles based on some pre-programmed parameters that humans put into the system. This exists today. The systems have been around for decades since the 1980s. And there were widespread use with at least 30 countries around the globe. So this is a type of weapon system that’s already in operation. These supervisory autonomous weapons. What really would be new would be fully autonomous weapons that operate on their own, whereas humans are still building them and launching them, but humans put them into operation, and then there’s some period of time where they were able to search a target area for targets and they were able to find these targets, and then based on some programming that was designed by people, identify the targets and attack them on their own.

Lucas Perry: Would you consider that out of the loop for that period of time?

Paul Scharre: Exactly. So over that period of time, humans are out of the loop on that decision over which targets they’re attacking. That would be potentially largely a new development in war. There are some isolated cases of some weapon systems that cross this line, by in large that would be new. That’s at least the starting point of what people might be concerned about. Now, you might envision things that are more advanced beyond that, but that’s sort of the near term development that could be on the horizon in the next five to 15 years, telling the weapon system, go into this area, fly around or search around underwater and find any ships of this type and attack them for some period of time in space. And that changes the human’s relationship with the use of force a little bit. It doesn’t mean the humans not involved at all, but the humans not quite as involved as they used to be. And is that something we’re comfortable with? And what are the implications of that kind of shift in warfare.

Lucas Perry: So the relevant things here are how this helps to integrate human control and human responsibility and human accountability into autonomous weapons systems. And just hearing you speak about all of that, it also seems like very relevant questions have to do with human psychology, about what human beings are actually likely to be able to do. And then also, I think you articulately put the practical question of whether or not people will be able to react to certain threats given certain situations. So in terms of trying to understand acceptable and unacceptable uses of autonomous weapons, that seems to supervene upon a lot of these facets of benefits and disadvantages of in the loop, on the loop, and out of the loop for different situations and different risks, plus how much we’re willing to automate killing and death and remove human decision making from some of these situations or not.

Paul Scharre: Yes, I mean, I think what’s challenging in this space is that it would be nice, it would be ideal if we could sort of reach agreement among nations for sort of a lethal laws of robotics, and Isaac Asimov’s books about robots you think of these three laws of robotics. Well, those laws aren’t going to work because one of them is not harming a human being and it’s not going to work in the military context, but could there be some agreement among countries for lethal laws of robots that would govern the behavior of autonomous systems in war, and it might sort of say, these are the things that are acceptable or not? Maybe. Maybe that’s possible someday. I think we’re not there yet at least, there are certainly not agreement as widespread disagreement among nations about what approach to take. But the good starting position of trying to understand what are the goals we want to achieve. And I think you’re right that we need to keep the human sort of front and center. But I this this is like a really important asymmetry between humans and machines that’s worth highlighting, which is to say that the laws of war government effects in the battlefield, and then in that sentence, the laws of war, don’t say the human has to pick every target, the laws of war say that the use of force must be executed according to certain principles of distinction and proportionality and other things.

One important asymmetry in the laws of war, however, is that machines are not legal agents. Only humans have legal agents. And so it’s ultimately humans that are responsible for complying with the laws of war. You can’t put a machine on trial for a war crime. It doesn’t make sense. It doesn’t have intentionality. So it’s ultimately a human responsibility to ensure this kind of compliance with the laws of war. It’s a good starting point then for conversation to try to understand if we start from that proposition that it’s a human responsibility to ensure compliance with the laws of war, then what follows from that? What balances that place on human involvement? One of the early parts of the conversations on autonomous weapons internationally came from this very technological based conversation. To say, well, based on the technology, draw these lines, you should put these limits in place. The problem with that approach is not that you can’t do it.

The problem is the state of the technology when? 2014 when discussions on autonomous weapons started at the very beginning of the deep learning revolution, today, in 2020, our estimate of whether technology might be in five years or 10 years or 50 years? The technology moving so quickly than any technologically based set of rules about how we should approach this problem and what is the appropriate use of machines versus human decision making in the use of force. Any technologically based answer is one that we may look back in 10 years or 20 years and say is wrong. We could get it wrong in the sense that we might be leaving valuable technological opportunities on the table and we’re banning technology that if we used it actually might make war more humane and reduce civilian casualties, or we might be permitting technologies that turned out in retrospect to be problematic, and we shouldn’t have done that.

And one of the things we’ve seen historically when you look at attempts to ban weapons is that ones that are technologically based don’t always fare very well over time. So for example, the early bans on poison gas banned the use of poison gas that are launched from artillery shells. It allowed actually poison gas administered via canisters, and so the first use of poison gas in World War One by the Germans was canister based, they actually just laid out little canisters and then open the valves. Now that turns out to be not very practical way of using poison gas in war, because you have someone basically on your side standing over this canister, opening a valve and then getting gassed. And so it’s a little bit tricky, but technically permissible.

One of the things that can be challenging is it’s hard to foresee how the technology is going to evolve. A better approach and one that we’ve seen the dialogue internationally sort of shift towards is our human-centered approach. To start from the position of the human and say, look, if we had all the technology in the world and war, what decisions would we want humans to make and why? Not because the technology cannot make decisions, but because it should not. I think it’s actually a very valuable starting place to understand a conversation, because the technology is moving so quickly.

What role do we want humans to play in warfare, and why do we think this is the case? Are there some tasks in war, or some decisions that we think are fundamentally human that should be decisions that only humans should make and we shouldn’t hand off to machines? I think that’s a really valuable starting position then to try to better interrogate how do we want to use this technology going forward? Because the landscape of technological opportunity is going to keep expanding. And so what do we want to do with this technology? How do we want to use it? And are there ways that we can use this technology that keeps humans in control of the use of force in the battlefield? Keep humans legally and morally and ethically responsible, but may make war more humane in the process, that may make war more precise, that may reduce civilian casualties without losing our humanity in the process.

Lucas Perry: So I guess the thought experiment, there would be like, if we had weapons that let us just delete people instantly without consequences, how would we want human decision making to be integrated with that? Reflecting on that also makes me consider this other point that I think is also important for my considerations around lethal autonomous weapons, which is the necessity of integrating human experience in the consequences of war, the pain and the suffering and the carnage and the PTSD as being almost necessary vehicles to some extent to make us tired of it to integrate how horrible it is. So I guess I would just be interested in integrating that perspective into it not just being about humans making decisions and the decisions being integrated in the execution process, but also about the experiential ramifications of being in relation to what actually happens in war and what violence is like and what happens in violence.

Paul Scharre: Well, I think that we want to unpack a little bit some of the things you’re talking about. Are we talking about ensuring that there is an accurate representation to the people carrying out the violence about what’s happening on the other end, that we’re not sanitizing things. And I think that’s a fair point. When we begin to put more psychological barriers between the person making the decision and the effects, it might be easier for them to carry out larger scale attacks, versus actually making war and more horrible. Now that’s a line of reasoning, I suppose, to say we should make war more horrible, so there’ll be less of it. I’m not sure we might get the outcome that there is less of it. We just might have more horrible war, but that’s a different issue. Those are more difficult questions.

I will say that I often hear philosophers raising things about skin in the game. I rarely hear them being raised by people who have had skin in the game, who have experienced up close in a personal way the horrors of war. And I’m less convinced that there’s a lot of good that comes from the tragedy of war. I think there’s value in us trying to think about how do we make war less terrible? How do we reduce civilian casualties? How do we have less war? But this often comes up in the context of technologies like we should somehow put ourselves at risk. No military does that, no military has ever done that in human history. The whole purpose of militaries getting technology in training is to get an advantage on the adversary. It’s not a fair fight. It’s not supposed to be, it’s not a boxing match. So these are things worth exploring. We need to come from the standpoint of the reality of what war is and not from a philosophical exercise about war might be, but deal with the realities of what actually occurs in the battlefield.

Lucas Perry: So I think that’s a really interesting point. And as someone with a background and interest in philosophy, it’s quite funny. So you do have experience in war, right?

Paul Scharre: Yes, I’ve fought in Iraq and Afghanistan.

Lucas Perry: Then it’s interesting for me, if you see this distinction between people who are actually veterans, who have experienced violence and carnage and tragedies of war, and the perspective here is that PTSD and associated trauma with these kinds of experiences, you find that they’re less salient for decreasing people’s willingness or decision to engage in further war. Is that your claim?

Paul Scharre: I don’t know. No, I don’t know. I don’t know the answer to that. I don’t know. That’s some difficult question for political scientists to figure out about voting preferences of veterans. All I’m saying is that I hear a lot of claims in this space that I think are often not very well interrogated or not very well explored. And there’s a real price that people pay for being involved. Now, people want to say that we’re willing to bear that price for some reason, like okay, but I think we should acknowledge it.

Lucas Perry: Yeah, that make sense. I guess the thing that I was just pointing at was it would be psychologically interesting to know if philosophers are detached from the experience, maybe they don’t actually know about the psychological implications of being involved in horrible war. And if people who are actually veterans disagree with philosophers about the importance of there being skin in the game, if philosophers say that skin in the game reduces willingness to be in war, if the claim is that that wouldn’t actually decrease their willingness to go to war. I think that seems psychologically very important and relevant, because there is this concern about how autonomous weapons and integrating human decision making to lethal autonomous weapons would potentially sanitize war. And so there’s the trade off between the potential mitigating effects of being involved in war, and then also the negative effects which are incurred by veterans who would actually have to be exposed by it and bring the trauma back for communities to have deeper experiential relation with.

Paul Scharre: Yes, and look, we don’t do that, right? We had a whole generation of veterans come back from Vietnam and we as society listen to the stories and understand them and understand, no. I have heard over the years people raise this issue whether it’s drones, autonomous weapons, this issue of having skin in the game either physically being at risk or psychologically. And I’ve rarely heard it raised by people who it’s been them who’s on the line. People often have very gut emotional reactions to this topic. And I think that’s valuable because it’s speaking to something that resonates with people, whether it’s an emotional reaction opposed to autonomous weapons, and that you often get that from many people that go, there’s something about this. It doesn’t feel right. I don’t like this idea. Or people saying, the opposite reaction. Other people that say that “wouldn’t this make war great, it’s more precise and more humane,” and which my reaction is often a little bit like… have you ever interacted with a computer? They break all the time. What are you talking about?

But all of these things I think they’re speaking to instincts that people have about this technology, but it’s worth asking questions to better understand, what is it that we’re reacting to? Is it an assumption about the technologies, is it an assumption about the nature of war? One of the concerns I’ve heard raised is like this will impersonalize war and create more distance between people killing. If you sort of buy that argument, that impersonal war is a bad thing, then you would say the greatest thing would be deeply personal war, like hand to hand combat. It appears to harken back to some glorious age of war when people looked each other in the eye and hacked each other to bits with swords, like real humans. That’s not that that war never occurred in human history. In fact, we’ve had conflicts like that, even in recent memory that involve hand to hand weapons. They tend not to be very humane conflicts. When we see civil violence, when people are murdering each other with machetes or garden tools or other things, it tends to be horrific communal violence, mass atrocities in Rwanda or Cambodia or other places. So I think it’s important to deal with the reality of what war is and not some fantasy.

Lucas Perry: Yes, I think that that makes a lot of sense. It’s really tricky. And the psychology around this I think is difficult and probably not studied enough.

Paul Scharre: There’s real war that occurs in the world, and then there’s the fantasy of war that we, as a society, tell ourselves when we go to movie theaters, and we watch stories about soldiers who are heroes, who conquer the bad guys. We’re told a fantasy, and it’s a fantasy as a society that allows society to perpetuate wars, that allows us to send young men and women off to die. And it’s not to say that there are no circumstances in which a nation might need to go to war to defend itself or its interest, but we sort of dress war up in these pretty clothes, and let’s not confuse that with the reality of what actually occurs. People said, well, through autonomous weapons, then we won’t have people sort of weighing the value of life and death. I mean, it happens sometimes, but it’s not like every time someone dies in war, that there was this thoughtful exercise where a committee sat around and said, “Do we really need to kill this person? Is it really appropriate?” There’s a lot of dehumanization that goes on on the battlefield. So I think this is what makes this issue very challenging. Many of the objections to autonomous weapons are objections to war. That’s what people are actually objecting to.

The question isn’t, is war bad? Of course war’s terrible? The question is sort of, how do we find ways going forward to use technology that may make war more precise and more humane without losing our humanity in the process, and are ways to do that? It’s a challenging question. I think the answer is probably yes, but it’s one that’s going to require a lot of interrogation to try to get there. It’s a difficult issue because it’s also a dynamic process where there’s an interplay between competitors. If we get this wrong, we can easily end up in a situation where there’s less human control, there’s more violence and war. There are lots of opportunities to make things worse as well.

If we could make war perfect, that would be great, in terms of no civilian suffering and reduce the suffering of enemy combatants and the number of lives lost. If we could push a button and make war go away, that would be wonderful. Those things will all be great. The more practical question really is, can we improve upon the status quo and how can we do so in a thoughtful way, or at least not make things worse than today? And I think those are hard enough problems to try to address.

Lucas Perry: I appreciate that you bring a very holistic, well-weighed perspective to the varying sides of this issue. So these are all very big and difficult. Are you aware of people actually studying whether some of these effects exist or not, and whether they would actually sanitize things or not? Or is this basically all just coming down to people’s intuitions and simulations in their head?

Paul Scharre: Some of both. There’s really great scholarship that’s being done on autonomous weapons, certainly there’s a robust array of legal based scholarship, people trying to understand how the law of war might interface with autonomous weapons. But there’s also been worked on by thinking about some of these human psychological interactions, Missy Cummings, who’s at Duke who runs the humans and automation lab down has done some work on human machine interfaces on weapon systems to think through some of these concerns. I think probably less attention paid to the human machine interface dimension of this and the human psychological dimension of it. But there’s been a lot of work done by people like Heather Roth, people at Article 36, and others thinking about concepts of meaningful human control and what might look like in weapon systems.

I think one of the things that’s challenging across the board in this issue is that it is a politically contentious topic. You have kind of levels of this debate going on, you have scholars trying to sort of understand the issue maybe, and then you also have a whole array of politically motivated groups, international organizations, civil society organizations, countries, duking it out basically, at the UN and in the media about where we should go with this technology. As you get a lot of motivated reasoning on all sides about what should the answer be. So for example, one of the things that fascinates me is i’ll often hear people say, autonomous weapons are terrible, and they’ll have a terrible outcome, and we need to ban them now. And if we just pass a treaty and we have enough political will we could ban them. I’ll also hear people say a ban would be pointless, it wouldn’t work. And anyways, wouldn’t autonomous weapons be great? There are other possible beliefs. One could say that a ban is feasible, but the weapons aren’t that big of a deal. So it just seems to me like there’s a lot of politically motivated reasoning that goes on this debate, which makes it very challenging.

Lucas Perry: So one of the concerns around autonomous weapons has to do with accidental escalation of warfare and conflict. Could you explore this point and explain what some strategies might be to prevent accidental escalation of warfare as AI is increasingly being used in the military?

Paul Scharre: Yes, so I think in general, you could bucket maybe concerns about autonomous weapons into two categories. One is a concern that they may not function very well and could have accidents, those accidents could lead to civilian casualties, that could lead to accidental escalation among nations and a crisis, military force forces operating in close proximity to one another and there could be accidents. This happens with people. And you might worry about actions with autonomous systems and maybe one shoots down an enemy aircraft and there’s an escalation and people are killed. And then how do you unwind that? How do you communicate to your adversary? We didn’t mean to do that. We’re sorry. How do you do that in a period of tension? That’s a particular challenge.

There’s a whole other set of challenges that come from the weapons might work. And that might get to some of these deeper questions about the role of humans in decision making about life and death. But this issue of accidental escalation kind of comes into the category of they don’t work very well, then they’re not reliable. And this is the case for a lot of AI and autonomous technology today, which isn’t to say it doesn’t work at all, if it didn’t work at all, it would be much easier. There’d be no debates about bias and facial recognition systems if they never identify faces. There’d be no debates about safety with self-driving cars if the car couldn’t go anywhere. The problem is that a lot of these AI based systems work very well in some settings, and then if the settings change ever so slightly, they don’t work very well at all anymore. And the performance can drop off very dramatically, and they’re not very robust to changes in environmental conditions. So this is a huge problem for the military, because in particular, the military doesn’t get to test its systems in its actual operating environment.

So you can take a car, and you can take it on the roads, and you can test it in an actual driving environment. And we’ve seen car companies rack up 10 million miles or more of driving data. And then they can go back and they can run simulations. So Waymo has said that they run 10 million miles of simulated driving every single day. And they can simulate in different lighting conditions, in different environmental conditions. Well, the military can build simulations too, but simulations of what? What will the next war look like? Well we don’t know because we haven’t fought it yet. The good news is that war’s very rare, which is great. But that also means that for these kinds of systems, we don’t necessarily know the operating conditions that they’ll be in, and so there is this real problem of this risk of accidents. And it’s exacerbated in the fact that this is also a very adversarial environment. So you actually have an enemy who’s trying to trick your system and manipulate it. That’s adds another layer of complications.

Driving is a little bit competitive, maybe somebody doesn’t want to let you into the lane, but the pedestrians aren’t generally trying to get hit by cars. That’s a whole other complication in the military space. So all of that leads to concerns that the systems may do okay in training, and then we take them out in the real world, and they fail and they fail a pretty bad way. If it’s a weapon system that is making its own decisions about whom to kill, it could be that it fails in a benign way, then it targets nothing. And that’s a problem for the military who built it, or fails in a more hazardous way, in a dangerous way and attacks the wrong targets. And when we’re talking about an autonomous weapon, the essence of this autonomous weapon is making its own decisions about which targets to attack and then carrying out those attacks. If you get that wrong, those could be pretty significant consequences with that. One of those things could be civilian harm. And that’s a major concern. There are processes in place for printing that operationally and test and evaluation, are those sufficient? I think they’re good reasons to say that maybe they’re not sufficient or not completely sufficient, and they need to be revised or improved.

And I’ll point out, we can come back to this that the US Defense Department actually has a more stringent procedure in place for reviewing autonomous weapons more than other weapons, beyond what the laws of war have, the US is one of the few countries that has this. But then there’s also question about accidental escalation, which also could be the case. Would that lead to like an entire war? Probably not. But it could make things a lot harder to defuse tensions in a crisis, and that could be problematic. So we just had an incident not too long ago, where the United States carried out an attack against the very senior Iranian General, General Soleimani, who’s the head of the Iranian Quds Force and killed him in a drone strike. And that was an intentional decision made by a person somewhere in the US government.

Now, did they fully think that through? I don’t know, that’s a different question. But a human made that decision in any case. Well, that’s a huge escalation of hostilities between the US and Iraq. And there was a lot of uncertainty afterwards about what would happen and Iran launched some ballistic missiles against US troops in Iraq. And whether that’s it, or there’s more retaliation to come, I think we’ll see. But it could be a much more challenging situation, if you had a situation in the future where an autonomous weapon malfunctioned and took some action. And now the other side might feel compelled to respond. They might say, well, we have to, we can’t let this go. Because humans emotions are on the line and national pride and prestige, and they feel like they need to maintain a principle of deterrence and they need to retaliate it. So these could all be very complicated things if you had an accident with an autonomous weapon.

Lucas Perry: Right. And so an adjacent issue that I’d like to explore now is how a potential arms race can have interplay with issues around accidental escalation of conflict. So is there already an arms race brewing for autonomous weapons? If so, why and what could potentially be done to deescalate such a situation?

Paul Scharre: If there’s an arms race, it’s a very strange one because no one is building the weapons. We see militaries advancing in robotics and autonomy, but we don’t really see sort of this rush to build autonomous weapons. I struggle to point to any programs that I’m aware of in militaries around the globe that are clearly oriented to build fully autonomous weapons. I think there are lots of places where much like these incremental advancements of autonomy in cars, you can see more autonomous features in military vehicles and drones and robotic systems and missiles. They’re adding more autonomy. And one might be violently concerned about where that’s going. But it’s just simply not the case that militaries have declared their intention. We’re going to build autonomous weapons, and here they are, and here’s our program to build them. I would struggle to use the term arms race. It could happen, maybe worth a starting line of an arms race. But I don’t think we’re in one today by any means.

It’s worth also asking, when we say arms race, what do we mean and why do we care? This is again, one of these terms, it’s often thrown around. You’ll hear about this, the concept of autonomous weapons or AI, people say we shouldn’t have an arms race. Okay. Why? Why is an arms race a bad thing? Militaries normally invest in new technologies to improve their national defense. That’s a normal activity. So if you say arms race, what do you mean by that? Is it beyond normal activity? And why would that be problematic? In the political science world, the specific definitions vary, but generally, an arms race is viewed as an increase in defense spending overall, or in a particular technology area above normal levels of modernizing militaries. Now, usually, this is problematic for a couple of reasons. One could be that it ends up just in a massive national expenditure, like during the case of the Cold War, nuclear weapons, that doesn’t really yield any military value or increase anyone’s defense or security, it just ends up net flushing a lot of money down the drain. That’s money that could be spent elsewhere for pre K education or healthcare or something else that might be societally beneficial instead of building all of these weapons. So that’s one concern.

Another one might be that we end up in a world that the large number of these weapons or the type of their weapons makes it worse off. Are we really better off in a world where there are 10s of thousands of nuclear weapons on hair-trigger versus a few thousand weapons or a few hundred weapons? Well, if we ever have zero, all things being equal, probably fewer nuclear weapons is better than more of them. So that’s another kind of concern whether in terms of violence and destructiveness of war, if a war breakout or the likelihood of war and the stability of war. This is an A in an area where certainly we’re not in any way from a spending standpoint, in an arms race for autonomous weapons or AI today, when you look at actual expenditures, they’re a small fraction of what militaries are spending on, if you look at, say AI or autonomous features at large.

And again for autonomous weapons, there really aren’t at least openly declared programs to say go build a fully autonomous weapon today. But even if that were the case, why is that bad? Why would a world where militaries are racing to build lots of atomic weapons be a bad thing? I think it would be a bad thing, but I think it’s also worth just answering that question, because it’s not obvious to everyone. This is something that’s often missing in a lot of these debates and dialogues about autonomous weapons, people may not share some of the underlying assumptions. It’s better to bring out these assumptions and explain, I think this would be bad for these reasons, because maybe it’s not intuitive to other people that they don’t share those reasons and articulating them could increase understanding.

For example, the FLI letter on autonomous weapons from a few years ago said, “the key question for humanity today is whether to start a global AI arms race or prevent it from starting. If any major military power pushes ahead with AI weapon development, the global arms race is virtually inevitable. And the endpoint of this technological trajectory is obvious. Autonomous weapons will become the Kalashnikovs of tomorrow.” I like the language, it’s very literary, “the Kalashnikovs of tomorrow.” Like it’s a very concrete image. But there’s a whole bunch of assumptions packed into those few sentences that maybe don’t work in the letter that’s intended to like sort of galvanize public interest and attention, but are worth really unpacking. What do we mean when we say autonomous weapons are the Kalashnikovs of tomorrow and why is that bad? And what does that mean? Those are, I think, important things to draw out and better understand.

It’s particularly hard for this issue because the weapons don’t exist yet. And so it’s not actually like debates around something like landlines. We could point to the mines and say like “this is a landmine, we all agree this is a landmine. This is what it’s doing to people.” And everyone could agree on what the harm is being caused. The people might disagree on what to do about it, but there’s agreement on what the weapon is and what the effect is. But for autonomous weapons, all these things are up to debate. Even the term itself is not clearly defined. And when I hear people describe it, people can be describing a whole range of things. Some people when they say the word autonomous weapon, they’re envisioning a Roomba with a gun on it. And other people are envisioning the Terminator. Now, both of those things are probably bad ideas, but for very different reasons. And that is important to draw out in these conversations. When you say autonomous weapon, what do you mean? What are you envisioning? What are you worried about? Worried about certain types of scenarios or certain types of effects?

If we want to get to the place where we really as a society come together and grapple with this challenge, I think first and foremost, a better communication is needed and people may still disagree, but it’s much more helpful. Stuart Russell from Berkeley has talked a lot about dangers of small anti-personnel autonomous weapons that would widely be the proliferated. He made the Slaughterbots video that’s been seen millions of times on YouTube. That’s a very specific image. It’s an image that’s very concrete. So then you can say, when Stuart Russell is worried about autonomous weapons, this is what he’s worried about. And then you can start to try to better understand the assumptions that go into that.

Now, I don’t share Stuart’s concerns, and we’ve written about it and talked about before, but it’s not actually because we disagree about the technology, I would agree that that’s very doable with existing technology. We disagree about the social responses to that technology, and how people respond, and what are the countermeasures and what are ways to prevent proliferation. So we, I think, disagree on some of the political or social factors that surround kind of how people approach this technology and use it. Sometimes people actually totally agree on the risks and even maybe the potential futures, they just have different values. And there might be some people who their primary value is trying to have fewer weapons in the world. Now that’s a noble goal. And they’re like, hey, anyway that we can have fewer weapons, fewer advanced technologies, that’s better. That’s very different from someone who’s coming from a position of saying, my goal is to improve my own nation’s defense. That’s a totally different value system. A total different preference. And they might be like, I also value what you say, but I don’t value it as much. And I’m going to take actions that advance these preferences. It’s important to really sort of try to better draw them out and understand them in this debate, if we’re going to get to a place where we can, as a society come up with some helpful solutions to this problem.

Lucas Perry: Wonderful. I’m totally on board with that. Two questions and confusions on my end. The first is, I feel a bit confused when you say these weapons don’t exist already. It seems to me more like autonomy exists on a spectrum and is the integration of many different technologies and decision making in systems. It seems to me there is already a certain degree of autonomy, there isn’t Terminator level autonomy, or specify an objective and the autonomous system can just basically go execute that, that seems to require very high level of generality, but there seems to already exist a level of autonomy today.

And so in that video, Stuart says that slaughterbots in particular represent a miniaturization and integration of many technologies, which already exist today. And the second thing that I’m confused about is when you say that it’s unclear to you that militaries are very interested in this or that there currently is an arms race. It seems like yes, there isn’t an arms race, like there was with nuclear weapons where it’s very clear, and they’re like Manhattan projects around this kind of technology, but given the strategic advantage conferred by this technology now and likely soon, it seems to me like game theoretically, from the position of militaries around the world that have the capacity to invest in these things, that it is inevitable given their battlefield importance that there would be massive ramping up or investments, or that there already is great interest in developing the autonomy and the subtechnologies required for developing fully autonomous systems.

Paul Scharre: Those are great questions and right on point. And I think the central issues in both of your questions are when we say these weapons or when I say these things, I should be more precise. When we say autonomous weapons, what do we mean exactly? And this is one of the things that can be tricky in this space, because there are not these universally agreed upon definitions. There are certainly many weapons systems used widely around the globe today that incorporate some autonomous features. Many of these are fire and forget weapons. When someone launches them, they’re not coming back. They have in that sense, autonomy to carry out their mission. But autonomy is relatively limited and narrowly bounded, and humans, for the most part are choosing the targets. So you can think of kind of maybe these three classes of weapons, these semi autonomous weapons, where humans are choosing the targets, but there’s lots of autonomy surrounding that decision, queuing information to people, flying the munition once the person launches it. That’s one type of weapon, widely used today by really every advanced military.

Another one is the supervised autonomous weapons that are used in these relatively limited settings for defensive purposes, where there is kind of this automatic mode that people can turn them on and activate them to defend the ship or the ground base or the vehicle. And these are really needed for these situations where the incoming threats are too fast for humans to respond. And these again are widely used around the globe and have been in place for decades. And then there are what we could call fully autonomous weapons, where the human’s launching them and human programs in the parameters, but they have some freedom to fly a search pattern over some area and then once they find a target, attack it on their own. For the most part, with some exceptions, those weapons are not widely used today. There have been some experimental systems that have been designed. There have been some put into operation in the past. The Israeli harpy drone is an example of this that is still in operation today. It’s been around since the ’90s, so it’s not really very new. And it’s been sold to a handful of countries, India, Turkey, South Korea, China, and the Chinese have reportedly reverse engineered their own version of this.

But it’s not like when widespread. So it’s not like a major component of militaries order of that. I think you see militaries investing in robotic systems, but the bulk of their fleets are still human occupied platforms, robotics are largely an adjunct to them. And in terms of spending, while there is increased spending on robotics, most of the spending is still going towards more traditional military platforms. The same is also true about the degree of autonomy, most of these robotic systems are just remote controlled, and they have very limited autonomy today. Now we’re seeing more autonomy over time in both robotic vehicles and in missiles. But militaries have a strong incentive to keep humans involved.

It is absolutely the case that militaries want technologies that will give them an advantage on the battlefield. But part of achieving an advantage means your systems work, they do what you want them to do, the enemy doesn’t hack them and take them over, you have control over them. All of those things point to more human control. So I think that’s the thing where you actually see militaries trying to figure out where’s the right place on the spectrum of autonomy? How much autonomy is right, and that line is going to shift over time. But it’s not the case that they necessarily want just full autonomy because what does that mean, then they do want weapon systems to sort of operate under some degree of human direction and involvement. It’s just that what that looks like may evolve over time as the technology advances.

And there are also, I should add, other bureaucratic factors that come into play that militaries investments are not entirely strategic. There’s bureaucratic politics within organizations. There’s politics more broadly with the domestic defense industry interfacing with the political system in that country. They might drive resources in certain directions. There’s some degree of inertia of course in any system that are also factors in play.

Lucas Perry: So I want to hit here a little bit on longer term perspectives. So the Future of Life Institute in particular is interested in mitigating existential risks. We’re interested in the advanced risks from powerful AI technologies where AI not aligned with human values and goals and preferences and intentions can potentially lead us to suboptimal equilibria that were trapped in permanently or could lead to human extinction. And so other technologies we care about are nuclear weapons and synthetic-bio enabled by AI technologies, etc. So there is this view here that if we cannot establish a governance mechanism as a global community on the concept that we should not let AI make the decision to kill then how can we deal with more subtle near term issues and eventual long term safety issues around the powerful AI technologies? So there’s this view of ensuring beneficial outcomes around lethal autonomous weapons or at least beneficial regulation or development of that technology, and the necessity of that for longer term AI risk and value alignment with AI systems as they become increasingly intelligent. I’m curious to know if you have a view or perspective on this.

Paul Scharre: This is the fun part of the podcast with the Future of Life because this rarely comes up in a lot of the conversations because I think in a lot of the debates, people are focused on just much more near term issues surrounding autonomous weapons or AI. I think that if you’re inclined to see that there are longer term risks for more advanced developments in AI, then I think it’s very logical to say that there’s some value in humanity coming together to come up with some set of rules about autonomous weapons today, even if the specific rules don’t really matter that much, because the level of risk is maybe not as significant, but the process of coming together and agreeing on some set of norms and limits on particularly military applications in AI is probably beneficial and may begin to create the foundations for future cooperation. The stakes for autonomous weapons might be big, but are certainly not existential. I think in any reasonable interpretation of autonomous weapons might do really, unless you start thinking about autonomy wired into, like nuclear launch decisions which is basically nuts. And I don’t think it’s really what’s on the table for realistically what people might be worried about.

When we try to come together as a human society to grapple with problems, we’re basically forced to deal with the institutions that we have in place. So for example, for autonomous weapons, we’re having debates in the UN Convention on Certain Conventional Weapons to CCW. Is that the best form for talking about autonomous weapons? Well, it’s kind of the form that exists for this kind of problem set. It’s not bad. It’s not perfect in some respects, but it’s the one that exists. And so if you’re worried about future AI risk, creating the institutional muscle memory among the relevant actors in society, whether it’s nation states, AI scientists, members of civil society, militaries, if you’re worried about military applications, whoever it is, to come together, to have these conversations, and to come up with some answer, and maybe set some agreements, some limits is probably really valuable actually because it begins to establish the right human networks for collaboration and cooperation, because it’s ultimately people, it’s people who know each other.

So oh, “I worked with this person on this last thing.” If you look at, for example, the international movement that The Campaign to Stop Killer Robots is spearheading, that institution or framework, those people, those relationships are born out of past successful efforts to ban landmines and then cluster munitions. So there’s a path dependency, and human relationships and bureaucracies, institutions that really matters. Coming together and reaching any kind of agreement, actually, to set some kind of limits is probably really vital to start exercising those muscles today.

Lucas Perry: All right, wonderful. And a final fun FLI question for you. What are your views on long term AI safety considerations? Do you view AI eventually as an existential risk and do you integrate that into your decision making and thinking around the integration of AI and military technology?

Paul Scharre: Yes, it’s a great question. It’s not something that comes up a lot in the world that I live in, in Washington in the policy world, people don’t tend to think about that kind of risk. I think it’s a concern. It’s a hard problem because we don’t really know how the technology is evolving. And I think that one of the things is challenging with AI is our frame for future more advanced AI. Often the default frame is sort of thinking about human like intelligence. When people talk about future AI, people talk about terms like AGI, or high level machine intelligence or human like intelligence, we don’t really know how the technology is evolving.

I think one of the things that we’re seeing with AI machine learning that’s quite interesting is that it often is evolving in ways that are very different from human intelligence, in fact, very quite alien and quite unusual. And I’m not the first person to say this, but I think that this is valid that we are, I think, on the verge of a Copernican revolution in how we think about intelligence, that rather than thinking of human intelligence as the center of the universe, that we’re realizing that humans are simply one type of intelligence among a whole vast array and space of possible forms of intelligence, and we’re creating different kinds, they may have very different intelligence profiles, they may just look very different, they may be much smarter than humans in some ways and dumber in other ways. I don’t know where things are going. I think it’s entirely possible that we move forward into a future where we see many more forms of advanced intelligent systems. And because they don’t have the same intelligence profile as human beings, we continue to kick the can down the road into being true intelligence because it doesn’t look like us. It doesn’t think like us. It thinks differently. But these systems may yet be very powerful in very interesting ways.

We’ve already seen lots of AI systems, even very simple ones exhibit a lot of creativity, a lot of interesting and surprising behavior. And as we begin to see the sort of scope of their intelligence widen over time, I think there are going to be risks that come with that. They may not be the risks that we were expecting, but I think over time, there going to be significant risks, and in some ways that our anthropocentric view is, I think, a real hindrance here. And I think it may lead us to then underestimate risk from things that don’t look quite like humans, and maybe miss some things that are very real. I’m not at all worried about some AI system one day becoming self aware, and having human level sentience, that does not keep me up at night. I am deeply concerned about advanced forms of malware. We’re not there today yet. But you could envision things over time that are adapting and learning and begin to populate the web, like there are people doing interesting ways of thinking about systems that have misaligned goals. It’s also possible to envision systems that don’t have any human directed goals at all. Viruses don’t. They replicate. They’re effective at replicating, but they don’t necessarily have a goal in the way that we think of it other than self replication.

If you have systems that are capable of replicating, of accumulating resources, of adapting, over time, you might have all of the right boxes to check to begin to have systems that could be problematic. They could accumulate resources that could cause problems. Even if they’re not trying to pursue either a goal that’s misaligned with human interest or even any goal that we might recognize. They simply could get out in the wild, if they’re effective at replication and acquiring resources and adapting, then they might survive. I think we’re likely to be surprised and continue to be surprised by how AI systems evolve, and where that might take us. And it might surprise us in ways that are humbling for how we think about human intelligence. So one question I guess is, is human intelligence a convergence point for more intelligent systems? As AI systems become more advanced, and they become more human like, or less human like and more alien.

Lucas Perry: Unless we train them very specifically on human preference hierarchies and structures.

Paul Scharre: Right. Exactly. Right. And so I’m not actually worried about a system that has the intelligence profile of humans, when you think about capacity in different tasks.

Lucas Perry: I see what you mean. You’re not worried about an anthropomorphic AI, you’re worried about a very powerful, intelligent, capable AI, that is alien and that we don’t understand.

Paul Scharre: Right. They might have cross domain functionality, it might have the ability to do continuous learning. It might be adaptive in some interesting ways. I mean, one of the interesting things we’ve seen about the field of AI is that people are able to tackle a whole variety of problems with some very simple methods and algorithms. And this seems for some reason offensive to some people in the AI community, I don’t know why, but people have been able to use some relatively simple methods, with just huge amounts of data and compute, it’s like a variety of different kinds of problems, some of which seem very complex.

Now, they’re simple compared to the real world, when you look at things like strategy games like StarCraft and Dota 2, like the world looks way more complex, but these are still really complicated kind of problems. And systems are basically able to learn totally on their own. That’s not general intelligence, but it starts to point towards the capacity to have systems that are capable of learning a whole variety of different tasks. They can’t do this today, continuously without suffering the problem of catastrophic forgetting that people are working on these things as well. The problems today are the systems aren’t very robust. They don’t handle perturbations in the environment very well. People are working on these things. I think it’s really hard to see how this evolves. But yes, in general, I think that our fixation on human intelligence as the pinnacle of intelligence, or even the goal of what we’re trying to build, and the sort of this anthropocentric view is, I think, probably one that’s likely to lead us to maybe underestimate some kinds of risks.

Lucas Perry: I think those are excellent points and I hope that mindfulness about that is able to proliferate in government and in actors who have power to help mitigate some of these future and short term AI risks. I really appreciate your perspective and I think you bring a wholesomeness and a deep authentic entertaining of all the different positions and arguments here on the question of autonomous weapons and I find that valuable. So thank you so much for your time and for helping to share information about autonomous weapons with us.

Paul Scharre: Thank you and thanks everyone for listening. Take care.

End of recorded material

AI Alignment Podcast: On the Long-term Importance of Current AI Policy with Nicolas Moës and Jared Brown

 Topics discussed in this episode include:

  • The importance of current AI policy work for long-term AI risk
  • Where we currently stand in the process of forming AI policy
  • Why persons worried about existential risk should care about present day AI policy
  • AI and the global community
  • The rationality and irrationality around AI race narratives

Timestamps: 

0:00 Intro

4:58 Why it’s important to work on AI policy 

12:08 Our historical position in the process of AI policy

21:54 For long-termists and those concerned about AGI risk, how is AI policy today important and relevant? 

33:46 AI policy and shorter-term global catastrophic and existential risks

38:18 The Brussels and Sacramento effects

41:23 Why is racing on AI technology bad? 

48:45 The rationality of racing to AGI 

58:22 Where is AI policy currently?

 

We hope that you will continue to join in the conversations by following us or subscribing to our podcasts on Youtube, Spotify, SoundCloud, iTunes, Google Play, StitcheriHeartRadio, or your preferred podcast site/application. You can find all the AI Alignment Podcasts here.

You can listen to the podcast above or read the transcript below. 

Lucas Perry: Welcome to the AI Alignment Podcast. I’m Lucas Perry. Today’s episode is with Jared Brown and Nicolas Moës, two AI Policy researchers and AI influencers who are both concerned with the long-term and existential risks associated with artificial general intelligence and superintelligence. For us at the the Future of Life Institute, we’re particularly interested in mitigating threats from powerful AI that could lead to the extinction of life. One avenue of trying to address such threats could be through action in the space of AI policy. But just what can we do today to help ensure beneficial outcomes from AGI and superintelligence in the policy sphere? This podcast focuses on this question.

As for some key points to reflect on throughout the podcast, Nicolas Moes points out that engaging in AI policy today is important because: 1) Experience gained on short-term AI policy issues is important to be considered a relevant advisor on long-term AI policy issues coming up in the future. 2) There are very few people that care about AGI safety currently in government, politics or in policy communities. 3) There are opportunities to influence current AI policy decisions in order to provide a fertile ground for future policy decisions or, better but rarer, to be directly shaping AGI safety policy today though evergreen texts. Future policy that is implemented is path dependent on current policy that we implement today. What we do now is precedent setting. 4) There are opportunities today to develop a skillset useful for other policy issues and causes. 5) Little resource is being spent on this avenue for impact, so the current return on investment is quite good.

Finally I’d like to reflect on the need to bridge the long-term and short-term partitioning of AI risk discourse. You might have heard this divide before, where there are long-term risks from AI, like a long-term risk being powerful AGI or superintelligence misaligned with human values causing the extinction of life, and then short-term risk like algorithmic bias and automation induced disemployment. Bridging this divide means understanding the real and deep interdependencies and path dependencies between the technology and governance which choose to develop today, and the world where AGI or superintelligence emerges. 

For those not familiar with Jared Brown or Nicolas Moës, Nicolas is an economist by training focused on the impact of Artificial Intelligence on geopolitics, the economy and society. He is the Brussels-based representative of The Future Society. Passionate about global technological progress, Nicolas monitors global developments in the legislative framework surrounding AI. He completed his Masters degree in Economics at the University of Oxford with a thesis on institutional engineering for resolving the tragedy of the commons in global contexts. 

Jared is the Senior Advisor for Government Affairs at FLI, working to reduce global catastrophic and existential risk (GCR/x-risk) by influencing the U.S. policymaking process, especially as it relates to emerging technologies. He is also a Special Advisor for Government Affairs at the Global Catastrophic Risk Institute. He has spent his career working at the intersection of public policy, emergency management, and risk management, having previously served as an Analyst in Emergency Management and Homeland Security Policy at the U.S. Congressional Research Service and in homeland security at the U.S. Department of Transportation.

The Future of Life Institute is a non-profit and this podcast is funded and supported by listeners like you. So if you find what we do on this podcast to be important and beneficial, please consider supporting the podcast by donating at futureoflife.org/donate. These contributions make it possible for us to bring you conversations like these and to develop the podcast further. You can also follow us on your preferred listening platform by searching for us directly or following the links on the page for this podcast found in the description.

And with that, here is Jared Brown and Nicolas Moës on AI policy. 

I guess we can start off here, with developing the motivations around why it’s important for people to be considering AI policy. So, why is it important to be working on AI policy right now?

Nicolas Moës: It’s important right now because there has been an uptick in markets, right? So AI technologies are now embedded in many more products than ever before. Part of it is hype, but part of it is also having a real impact on profits and bottom line. So there is an impact on society that we have never seen before. For example, the way Facebook algorithms have affected recent history is something that has made the population and policy makers panic a bit.

And so quite naturally the policy window has opened. I think it’s also important to be working on it for people who would like to make the world better for two reasons. As I mentioned, since the policy window is open that means that there is a demand for advice to fill in the gaps that exist in the legislation, right? There have been many concrete situations where, as an AI policy researcher, you get asked to provide input either by joining expert group, or workshops or simply directly some people who say, “Oh, you know about AI, so could you just send me a position paper on this?”

Nicolas Moës: So these policies are getting written right now, which at first is quite soft and then becomes harder and harder policies, and now to the point that at least in the EU, you have regulations for AI on the agenda, which is one of the hardest form of legislation out there. Once these are written it is very difficult to change them. It’s quite sticky. There is a lot of path dependency in legislation. So this first legislation that passes, will probably shape the box in which future legislation can evolve. Its constraints, the trajectory of future policies, and therefore it’s really difficult to take future policies in another direction. So for people who are concerned about AGI, it’s important to be already present right now.

The second point, is that these people who are currently interacting with policymakers on a daily basis are concerned about very specific things and they are gaining a lot of experience with policymakers, so that in the future when you have more general algorithms that come into play, the people with experience to advise on these policies will actually be concerned about what many people call short term issues. People who are concerned more about the safety, the robustness of these more general algorithm would actually end up having a hard time getting into the room, right? You cannot just walk in and claim authority when you have people with 10, 15 or even 20 years of experience regulating this particular field of engineering.

Jared Brown: I think that sums it up great, and I would just add that there are some very specific examples of where we’re seeing what has largely been, up to this point, a set of principles being developed by different governments, or industry groups. We’re now seeing attempts to actually enact hard law or policy.

Just in the US, the Office of Management and Budget and the Office of Science and Technology Policy issued a memorandum calling for further AI regulation and non-regulatory actions and they issued a set of principles, that’s out for comment right now, and people are looking at those principles, trying to see if there’s ways of commenting on it to increase its longterm focus and its ability to adapt to increasingly powerful AI.

The OECD has already issued, and had sign ons to its AI principles, which are quite good.

Lucas Perry: What is the OECD?

Nicolas Moës: The Organization for Economic Cooperation and Development.

Jared Brown: Yes. Those principles are now going from principles to an observatory, and that will be launched by the end of February. And we’re seeing the effect of these principles now being adopted, and attempts now are being made to implement those into real regulatory approaches. So, the window from transitioning from principles to hard law is occurring right now, and as Nicholas said, decisions that are made now will have longterm effects because typically governments don’t turn their attention to issues more than once every five, maybe even 10 years. And so, if you come in three years from now with some brilliant idea about AI policy, chances are, the moment to enact that policy has already passed because the year prior, or two years prior, your government has enacted its formative legislation on AI.

Nicolas Moës: Yeah, yeah. So long as this policy benefits most people, they are very unlikely to even reopen, let’s say, the discussion, at all.

Lucas Perry: Right. So a few points here. The first is this one about path dependency, which means that the kinds of policies which we adopt now are going to be really important, because they’re going to inform and shape the kinds of policies that we’re able or willing to adopt later, and AI is going to be around for a long, long time. So we’re setting a lot of the foundation. The second thing was that if you care about AGI risk, or the risks of superintelligence, or very powerful forms of AI that you need to have been part of the conversation since the beginning, or else you’re not going to really be able to get a seat at the table when these things come around.

And Jared, is there a point here that I’m missing that you were trying to make?

Jared Brown: No, I think that sums it up nicely. The effect of these policies, and the ability of these policies to remain what you might call evergreen. So, long lasting and adaptive to the changing nature of AI technology is going to be critical. We see this all the time in tech policy. There are tech policies out there that were informed by the challenges of the time in which they were made and they quickly become detrimental, or outdated at best. And then there are tech policies that tend to be more adaptive, and those stand the test of time. And we need to be willing to engage with the short term policy making considerations, such that we’re making sure that the policies are evergreen for AI, as it becomes increasingly powerful.

Nicolas Moës: Besides the evergreen aspects of the policies that you want to set up now, there’s this notion of providing a fertile ground. So some policies that are very appropriate for short term issues, for example, fairness and deception, and fundamental rights abuse and that kind of thing, are actually almost copy pasted to future legislation. So, if you manage to already put concerns for safety, like robustness, corrigibility, and value alignment of the algorithm today, even if you don’t have any influence in 10 or 15 years when they review the legislation, you have some chances to see the policymakers just copy pasting this part on safety and to put it in whatever new legislation comes up in 10 years.

Jared Brown: There’s precedent setting, and legislators are woe to have to make fundamental reforms to legislation, and so if we see proper consideration of safety and security on AI in the evergreen pieces of legislation that are being developed now, that’s unlikely to be removed in future legislation.

Lucas Perry: Jared, you said that a lot of the principles and norms which have been articulated over say, the past five years are becoming codified into hard law slowly. It also would just be good if you guys could historically contextualize our position in terms of AI policy, whether or not we stand at an important inflection point, where we are in terms of this emerging technology.

Jared Brown: Sure, sure. So, I think if you went back just to 2017, 2016, at least in the US, there was very little attention to artificial intelligence. There were a smattering of congressional hearings being held, a few pertinent policy documents being released by executive agencies, but by and large, the term artificial intelligence remained in the science fiction realm of thinking.

Since that time, there’s been a massive amount of attention paid to artificial intelligence, such that in almost every Western democracy that I’m familiar with, it’s now part of the common discourse about technology policy. The phrase emerging tech is something that you see all over the place, regardless of the context, and there’s a real sensitivity by Western style democracy policymakers towards this idea that technology is shifting under our feet. There’s this thing called artificial intelligence, there’s this thing called synthetic biology, there’s other technologies linked into that — 5G and hypersonics are two other areas — where there’s a real understanding that something is changing, and we need to get our arms around it. Now, that has largely started with, in the past year, or year and a half, a slew of principles. There are at least 80 some odd sets of principles. FLI was one of the first to create a set of principles, along with many partners, and those are the Asilomar AI Principles.

Those principles you can see replicated and informing many sets of principles since then. We mentioned earlier, the OECD AI principles are probably the most substantive and important at this point, because they have the signature and backing of so many sovereign nation states, including the United States and most of the EU. Now that we have these core soft law principles, there’s an appetite for converting that into real hard law regulation or approaches to how AI will be managed in different governance systems.

What we’re seeing in the US, there’s been a few regulatory approaches already taken. For instance, rule making on the inclusion of AI algorithms into the housing market. This vision, if you will, from the Department of Transportation, about how to deal with autonomous vehicles. The FDA has approved products coming into the market that involve AI and diagnostics in the healthcare industry, and so forth. We’re seeing initial policies being established, but what we haven’t yet seen in any real context, is sort of a cross-sectoral AI broadly-focused piece of legislation or regulation.

And that’s what’s currently being developed both in the EU and in the US. That type of legislation, which seems like a natural evolution from where we’re at with principles, into a comprehensive holistic approach to AI regulation and legislation, is now occurring. And that’s why this time is so critical for AI policy.

Lucas Perry: So you’re saying that a broader and more holistic view about AI regulation and what it means to have and regulate beneficial AI is developed before more specific policies are implemented, with regards to the military, or autonomous weapons, or healthcare, or nuclear command and control.

Jared Brown: So, typically, governments try, whether or not they succeed remains to be seen, to be more strategic in their approach. If there is a common element that’s affecting many different sectors of society, they try and at least strategically approach that issue, to think: what is common across all policy arenas, where AI is having an effect, and what can we do to legislate holistically about AI? And then as necessary, build sector specific policies on particular issues.

So clearly, you’re not going to see some massive piece of legislation that covers all the potential issues that has to do with autonomous vehicles, labor displacement, workforce training, et cetera. But you do want to have an overarching strategic plan for how you’re regulating, how you’re thinking about governing AI holistically. And that’s what’s occurring right now, is we have the principles, now we need to develop that cross-sectoral approach, so that we can then subsequently have consistent and informed policy on particular issue areas as they come up, and as they’re needed.

Lucas Perry: And that cross-sectoral approach would be something like: AI should be interpretable and robust and secure.

Jared Brown: That’s written in principles to a large degree. But now we’re seeing, what does that really mean? So in the EU they’re calling it the European Approach to AI, and they’re going to be coming out with a white paper, maybe by the time this podcast is released, and that will sort of be their initial official set of options and opinions about how AI can be dealt with holistically by the EU. In the US, they’re setting regulatory principles for individual regulatory agencies. These are principles that will apply to the FDA, or the Department of Transportation, or the Department of Commerce, or the Department of Defense, as they think about how they deal with the specific issues of AI in their arenas of governance. Making sure that baseline foundation is informed and is an evergreen document, so that it incorporates future considerations, or is at least adaptable to future technological development in AI is critically important.

Nicolas Moës: With regards to the EU in particular, the historical context is maybe a bit different. As you mentioned, right now they are discussing this white paper with many transversal policy instruments that would be put forward, with this legislation. This is going to be negotiated over the next year. There is intentions to have the legislation at the EU level by the end of the current commission’s term. So that’s mean within five years. This is something that is quite interesting to explore, is that in 2016 there was this parliamentary dossier on initiative, so it’s something that does not have any binding power, just to show the opinion of the European parliament, that was dealing with robotics and civil laws. So, considering how civil law in Europe should be adjusted to robotics.

That was in 2016, right? And now there’s been this uptick in activities. This is something that we have to be aware of. It’s moved quite fast, but then again, there still is a couple of years before regulations get approved. This is one point that I wanted to clarify about, when we say it is fast or it is slow, we are talking still about a couple of years. Which is, when you know how long it takes for you to develop your network, to develop your understanding of the issues, and to try to influence the issues, a couple of years is really way too short. The second point I wanted to make is also, what will the policy landscape look like in two years? Will we have the EU again leveraging its huge market power to impose its regulations within the European Commission. There are some intentions to diffuse whatever regulations come out of the European Commission right now, throughout the world, right? To form a sort of influence sphere, where all the AI produced, even abroad, would actually be fitting EU standards.

Over the past two, three years there have been a mushrooming of AI policy players, right? The ITU has set up this AI For Good, and has reoriented its position towards AI. There has been the Global Forum on AI for Humanity, political AI summits, which kind of pace the discussions about the global governance of artificial intelligence.

But would there be space for new players in the future? That’s something that I’m a bit unsure. One of the reasons why it might be an inflection point, as you asked, is because now I think the pawns are set on the board, right? And it is unlikely that somebody could come in and just disturb everything. I don’t know in Washington how it plays, but in Brussels it seems very much like everybody knows each other already and it’s only about bargaining with each other, not especially listening to outside views.

Jared Brown: So, I think the policy environment is being set. I wouldn’t quite go so far as to say all of the pawns are on the chess board, but I think many of them are. The queen is certainly industry, and industry has stood up and taken notice that governments want to regulate and want to be proactive about their approach to artificial intelligence. And you’ve seen this, because you can open up your daily newspaper pretty much anywhere in the world and see some headline about some CEO of some powerful tech company mentioning AI in the same breath as government, and government action or government regulations.

Industry is certainly aware of the attention that AI is getting, and they are positioning themselves to influence that as much as possible. And so civil society groups such as the ones Nico and I represent have to step up, which is not to say the industry has all bad ideas, some of what they’re proposing is quite good. But it’s not exclusively a domain controlled by industry opinions about the regulatory nature of future technologies.

Lucas Perry: All right. I’d like to pivot here, more into some of the views and motivations the Future of Life Institute and the Future Society take, when looking at AI policy. The question in particular that I’d like to explore is how is current AI policy important for those concerned with AGI risk and longterm considerations about artificial intelligence growing into powerful generality, and then one day surpassing human beings in intelligence? For those interested in the issue of AGI risk or super intelligence risk, is AI policy today important? Why might it be important? What can we do to help shape or inform the outcomes related to this?

Nicolas Moës: I mean, obviously, I’m working full time on this and if I could, I would work double full time on this. So I do think it’s important. But it’s still too early to be talking about this in the policy rooms, at least in Brussels. Even though we have identified a couple of policymakers that would be keen to talk about that. But it’s politically not feasible to put forward these kind of discussions. However, AI policy currently is important because there is a demand for advice, for policy research, for concrete recommendations about how to govern this technological transition that we are experiencing.

So there is this demand where people who are concerned about fundamental rights, and safety, and robustness, civil society groups, but also academics and industry themselves sometime come in with their clear recommendations about how you should concretely regulate, or govern, or otherwise influence the development and deployment of AI technologies, and in that set of people, if you have people who are concerned about safety, you would be able then, to provide advice for providing evergreen policies, as we’ve mentioned earlier and set up, let’s say, a fertile ground for better policies in the future, as well.

The second part of why it’s important right now is also the longterm workforce management. If people who are concerned about the AGI safety are not in the room right now, and if they are in the room but focused only on AGI safety, they might be perceived as irrelevant by current policymakers, and therefore they might have restricted access to opportunities for gaining experience in that field. And therefore over the long term this dynamic reduces the growth rate, let’s say, of the workforce that is concerned about AGI safety, and that could be identified as a relevant advisor in the future. As a general purpose technology, even short term issues regarding AI policy have a long term impact on the whole of society.

Jared Brown: Both Nicholas and I have used this term “path dependency,” which you’ll hear a lot in our community and I think it really helps maybe to build out that metaphor. Various different members of the audience of this podcast are going to have different timelines in their heads when they think about when AGI might occur, and who’s going to develop it, what the characteristics of that system will be, and how likely it is that it will be unaligned, and so on and so forth. I’m not here to engage in that debate, but I would encourage everyone to literally think about whatever timeline you have in your head, or whatever descriptions you have for the characteristics that are most likely to occur when AGI occurs.

You have a vision of that future environment, and clearly you can imagine different environments by which humanity is more likely to be able to manage that challenge than other environments. An obvious example, if the world were engaged in World War Three, 30 years from now, and some company develops AGI, that’s not good. It’s not a good world for AGI to be developed in, if it’s currently engaged in World War Three at the same time. I’m not suggesting we’re doing anything to mitigate World War Three, but there are different environments for when AGI can occur that will make it more or less likely that we will have a beneficial outcome from the development of this technology.

We’re literally on a path towards that future. More government funding for AI safety research is a good thing. That’s a decision that has to get made, that’s made every single day, in governments all across the world. Governments have R&D budgets. How much is that being spent on AI safety versus AI capability development? If you would like to see more, then that decision is being made every single fiscal year of every single government that has an R&D budget. And what you can do to influence it is really up to you and how many resources you’re going to put into it.

Lucas Perry: Many of the ways it seems that AI policy currently is important for AGI existential risk are indirect. Perhaps it’s direct insofar as there’s these foundational evergreen documents, and maybe changing our trajectory directly is kind of a direct intervention.

Jared Brown: How much has nuclear policy changed? When our governance of nuclear weapons changed because the US initially decided to use the weapon. That decision irrevocably changed the future of Nuclear Weapons Policy, and there is no way you can counterfactually unspool all of the various different ways the initial use of the weapon, not once, but twice by the US sent a signal to the world A, the US was willing to use this weapon and the power of that weapon was on full display.

There are going to be junctures in the trajectory of AI policy that are going to be potentially as fundamental as whether or not the US should use a nuclear weapon at Hiroshima. Those decisions are going to be hard to see necessarily right now, if you’re not in the room and you’re not thinking about the way that policy is going to project into the future. That’s where this matters. You can’t unspool and rerun history. We can’t decide for instance, on lethal autonomous weapons policy. There is a world that exists, a future scenario 30 years from now, where international governance has never been established on lethal autonomous weapons. And lethal autonomous weapons is completely the norm for militaries to use indiscriminately or without proper safety at all. And then there’s a world where they’ve been completely banned. Those two conditions will have serious effect on the likelihood that governments are up to the challenge of addressing potential global catastrophic and existential risk arising from unaligned AGI. And so it’s more than just setting a path. It’s central to the capacity building of our future to deal with these challenges.

Nicolas Moës: Regarding other existential risks, I mean Jared is more of an expert on that than I am. In the EU, because this topic is so hot, it’s much more promising, let’s say as an avenue for impact, than other policy dossiers because we don’t have the omnibus type of legislation that you have in the US. The EU remains quite topic for topic. In the end, there is very little power embeded in the EU, mostly it depends on the nation states as well, right?

So AI is as moves at the EU level, which makes you want to walk at the EU level AI policy for sure. But for the other issues, it sometimes remains still at the national level. That’d being said, the EU also has this particularity, let’s say off being able to reshape debates at the national level. So, if there were people to consider what are the best approaches to reduce existential risk in general via EU policy, I’m sure there would be a couple of dossiers right now with policy window opens that could be a conduit for impact.

Jared Brown: If the community of folks that are concerned about the development of AGI are correct and that it may have potentially global catastrophic and existential threat to society, then you’re necessarily obviously admitting that AGI is also going to affect the society extremely broadly. It’s going to be akin to an industrial revolution, as is often said. And that’s going to permeate every which way in society.

And there’s been some great work to scope this out. For instance, in the nuclear sphere, I would recommend to all the audience that they take a look at a recent edited compendium of papers by the Stockholm International Peace Research Institute. They have a fantastic compendium of papers about AI’s effect on strategic stability in nuclear risk. That type of sector specific analysis can be done with synthetic biology and various other things that people are concerned about as evolving into existential or global catastrophic risk.

And then there are current concerns with non anthropomorphic risk. AI is going to be tremendously helpful if used correctly to track and monitor near earth objects. You have to be concerned about asteroid impacts. AI is a great tool to be used to help reduce that risk by monitoring and tracking near Earth objects.

We may yet make tremendous discoveries in geology to deal with supervolcanoes. Just recently there’s been some great coverage of a AI company called Blue Dot for monitoring the potential pandemics arising with the Coronavirus. We see these applications of AI very beneficially reducing other global catastrophic and existential risks, but there are aggravating factors as well, especially for other anthropomorphic concerns related to nuclear risk and synthetic biology.

Nicolas Moës: Some people who are concerned about is AGI sometimes might see AI as overall negative in expectation, but a lot of policy makers see AI as an opportunity more than as a risk, right? So, starting with a negative narrative or a pessimistic narrative is difficult in the current landscape.

In Europe it might be a bit easier because for odd historical reasons it tends to be a bit more cautious about technology and tends to be more proactive about regulations than maybe anywhere else in the world. I’m not saying whether it’s a good thing or a bad thing. I think there’s advantages and disadvantages. It’s important to know though that even in Europe you still have people who are anti-regulation. The European commission set this independent high level expert group on AI with 52 or 54 experts on AI to decide about the ethical principles that will inform the legislation on AI. So this was for the past year and a half, or the past two years even. Among them, the divisions are really important. Some of them wanted to just let it go for self-regulation because even issues of fairness or safety will be detected eventually by society and addressed when they arise. And it’s important to mention that actually in the commission, even though the current white paper seems to be more on the side of preventive regulations or proactive regulations, the commissioner for digital, Thierry Breton is definitely cautious about the approach he takes. But you can see that he is quite positive about the potential of technology.

The important thing here as well is that these players have an influential role to play on policy, right? So, going back to this negative narrative about AGI, it’s also something where we have to talk about how you communicate and how you influence in the end the policy debate, given the current preferences and the opinions of people in society as a whole, not only the opinions of experts. If it was only about experts, it would be maybe different, but this is politics, right? The opinion of everybody matters and it’s important that whatever influence you want to have on AI policy is compatible with the rest of society’s opinion.

Lucas Perry: So, I’m curious to know more about the extent to which the AI policy sphere is mindful of and exploring the shorter term global catastrophic or maybe even existential risks that arise from the interplay of more near term artificial intelligence with other kinds of technologies. Jared mentioned a few in terms of synthetic biology, and global pandemics, and autonomous weapons, and AI being implemented in the military and early warning detection systems. So, I’m curious to know more about the extent to which there are considerations and discussions around the interplay of shorter term AI risks with actual global catastrophic and existential risks.

Jared Brown: So, there’s this general understanding, which I think most people accept, that AI is not magical. It is open to manipulation, it has certain inherent flaws in its current capability and constructs. We need to make sure that that is fully embraced as we consider different applications of AI into systems like nuclear command and control. At a certain point in time, the argument could be sound that AI is a better decision maker than your average set of humans in a command and control structure. There’s no shortage of instances of near misses with nuclear war based on existing sensor arrays, and so on and so forth, and the humans behind those sensor arrays, with nuclear command and control. But we have to be making those evaluations fully informed about the true limitations of AI and that’s where the community is really important. We have to cut through the hype and cut through overselling what AI is capable of, and be brutally honest about the current limitations of AI as it evolves, and whether or not it makes sense from a risk perspective to integrate AI in certain ways.

Nicolas Moës: There has been human mistakes that have led to close calls, but I believe these close calls have been corrected because of another human in the loop. In early warning systems though, you might actually end up with no human in the loop. I mean, again, we cannot really say whether these humans in the loop were statistically important because we don’t have the alternatives obviously to compare it to.

Another thing regarding whether some people think that AI is magic, I, I think, would be a bit more cynical. I still find myself in some workshops or policy conferences where you have some people who apparently haven’t seen ever a line of code in their entire life and still believe that if you tell the developer “make sure your AI is explainable,” that magically the AI would become explainable. This is still quite common in Brussels, I’m afraid. But there is a lot of heterogeneity. I think now we have, even among the 705 MEPs, there is one of them who is a former programmer from France. And that’s the kind of person who, given his expertise, if he was placed on the AI dossier, I guess he would have a lot more influence because of his expertise.

Jared Brown: Yeah. I think in the US there’s this phrase that kicks around that the US is experiencing a techlash, meaning there’s a growing reluctance, cynicism, criticism of major tech industry players. So, this started with the Cambridge Analytica problems that arose in the 2016 election. Some of it’s related to concerns about potential monopolies. I will say that it’s not directly related to AI, but that general level of criticism, more skepticism, is being imbued into the overall policy environment. And so people are more willing to question the latest, next greatest thing that’s coming from the tech industry because we’re currently having this retrospective analysis of what we used to think of a fantastic and development may not be as fantastic as we thought it was. That kind of skepticism is somewhat helpful for our community because it can be leveraged for people to be more willing to take a critical eye in the way that we apply technology going forward, knowing that there may have been some mistakes made in the past.

Lucas Perry: Before we move on to more empirical questions and questions about how AI policy is actually being implemented today, are there any other things here that you guys would like to touch on or say about the importance of engaging with AI policy and its interplay and role in mitigating both AGI risk and existential risk?

Nicolas Moës: Yeah, the so called Brussels effect, which actually describes that whatever decisions in European policy that is made is actually influencing the rest of the world. I mentioned it briefly earlier. I’d be curious to hear what you, Jared, thinks about that. In Washington, do people consider it, the GDPR for example, as a pre made text that they can just copy paste? Because apparently, I know that California has released something quite similar based on GDPR. By the way, GDPR is the General Data Protection Regulations governing protection of privacy in the EU. It’s a regulation, so it has a binding effect on EU member States. That, by the Brussels effect, what I mean is that for example, this big piece of legislation as being, let’s say, integrated by big companies abroad, including US companies to ensure that they can keep access to the European market.

And so the commission is actually quite proud of announcing that for example, some Brazilian legislator or some Japanese legislator or some Indian legislators are coming to the commission to translate the text of GDPR, and to take it back to their discussion in their own jurisdiction. I’m curious to hear what you think of whether the European third way about AI has a greater potential to lead to beneficial AI and beneficial AGI than legislation coming out of the US and China given the economic incentives that they’ve got.

Jared Brown: I think in addition to the Brussels effect, we might have to amend it to say the Brussels and the Sacramento effect. Sacramento being the State Capitol of California because it’s one thing for the EU who have adopted the GDPR, and then California essentially replicated a lot of the GDPR, but not entirely, into what they call the CCPA, the California Consumer Privacy Act. If you combine the market size of the EU with California, you clearly have enough influence over the global economy. California for those who aren’t familiar, would be the seventh or sixth largest economy in the world if it were a standalone nation. So, the combined effect of Brussels and Sacramento developing tech policy or leading tech policy is not to be understated.

What remains to be seen though is how long lasting that precedent will be. And their ability to essentially be the first movers in the regulatory space will remain. With some of the criticism being developed around GDPR and the CCPA, it could be that leads to other governments trying to be more proactive to be the first out the door, the first movers in terms of major regulatory effects, which would minimize the Brussels effect or the Brussels and Sacramento effect.

Lucas Perry: So in terms of race conditions and sticking here on questions of global catastrophic risk and existential risks and why AI policy and governance and strategy considerations are important for risks associated with racing between say the United States and China on AI technology. Could you guys speak a little bit to the importance of appropriate AI policy and strategic positioning on mitigating race conditions and a why race would be bad for AGI risk and existential and global catastrophic risks in general?

Jared Brown: To simplify it, the basic logic here is that if two competing nations states or companies are engaged in a competitive environment to be the first to develop X, Y, Z, and they see tremendous incentive and advantage to being the first to develop such technology, then they’re more likely to cut corners when it comes to safety. And cut corners thinking about how to carefully apply these new developments to various different environments. There has been a lot of discussion about who will come to dominate the world and control AI technology. I’m not sure that either Nicolas or I really think that narrative is entirely accurate. Technology need not be a zero sum environment where the benefits are only accrued by one state or another. Or that the benefits accruing to one state necessarily reduce the benefits to another state. And there has been a growing recognition of this.

Nicolas earlier mentioned the high level expert group in the EU, an equivalent type body in the US, it’s called the National Security Commission on AI. And in their interim report they recognize that there is a strong need and one of their early recommendations is for what they call Track 1.5 or Track 2 diplomacy, which is essentially jargon for engagement with China and Russia on AI safety issues. Because if we deploy these technologies in reckless ways, that doesn’t benefit anyone. And we can still move cooperatively on AI safety and on the responsible use of AI without mitigating or entering into a zero sum environment where the benefits are only going to be accrued by one state or another.

Nicolas Moës: I definitely see the safety technologies as that would benefit everybody. If you’re thinking in two different types of inventions, the one that promotes safety indeed would be useful, but I believe that enhancing raw capabilities, you would actually race for that. Right? So, I totally agree with your decision narrative. I know people on both sides seeing this as a silly thing, you know, with media hype and of course industry benefiting a lot from this narrative.

There is a lot of this though that remains the rational thing to do, right? Whenever you start negotiating standards, you can say, “Well look at our systems. They are more advanced, so they should become the global standards for AI,” right? That actually is worrisome because the trajectory right now, since there is this narrative in place, is that over the medium term, you would expect the technologies maybe to diverge, and so both blocks, or if you want to charitably include the EU into this race, the three blocks would start diverging and therefore we’ll need each other less and less. The economic cost of an open conflict would actually decrease, but this is over the very long term.

That’s kind of the dangers of race dynamics as I see them. Again, it’s very heterogeneous, right? When we say the US against China, when you look at the more granular level of even units of governments are sometimes operating with a very different mindset. So, as for what in AI policy can actually be relevant to this for example, I do think they can, because at least on the Chinese side as far as I know, there is this awareness of the safety issue. Right? And there has been a pretty explicit article. It was like, “the US and China should work together to future proof AI.” So, it gives you the impression that some government officials or former government officials in China are interested in this dialogue about the safety of AI, which is what we would want. We don’t especially have to put the raw capabilities question on the table so long as there is common agreements about safety.

At the global level, there’s a lot of things happening to tackle this coordination problem. For example, the OECD AI Policy Observatory is an interesting setup because that’s an institution with which the US is still interacting. There have been fewer and fewer multilateral fora with which the US administration has been willing to interact constructively, let’s say. But for the OECD one yes, there’s been quite a lot of interactions. China is an observer to the OECD. So, I do believe that there is potential there to have a dialogue between the US and China, in particular about AI governance. And plenty of other fora exist at the global level to enable this Track 1.5 / Track 2 diplomacy that you mentioned Jared. For example, the Global Governance of AI Forum that the Future Society has organized, and Beneficial AGI that Future of Life Institute has organized.

Jared Brown: Yeah, and that’s sort of part and parcel with one of the most prominent examples of, some people call it scientific diplomacy, and that’s kind of a weird term, but the Pugwash conferences that occurred all throughout the Cold War where technical experts were meeting on the side to essentially establish a rapport between Russian and US scientists on issues of nuclear security and biological security as well.

So, there are plenty of examples where even if this race dynamic gets out of control, and even if we find ourselves 20 years from now in an extremely competitive, contentious relationship with near peer adversaries competing over the development of AI technology and other technologies, we shouldn’t, as civil society groups, give up hope and surrender to the inevitability that safety problems are likely to occur. We need to be looking to the past examples of what can be leveraged in order to appeal to essentially the common humanity of these nation states in their common interest in not wanting to see threats arise that would challenge either of their power dynamics.

Nicolas Moës: The context matters a lot, but sometimes it can be easier than one can think, right? So, I think when we organized the US China AI Tech Summit, because it was about business, about the cutting edge and because it was also about just getting together to discuss. And a bit before this US / China race dynamics was full on, there was not so many issues with getting our guests. Knowledge might be a bit more difficult with some officials not able to join events where officials from other countries are because of diplomatic reasons. And that was in June 2018 right? But back then there was the willingness and the possibility, since the US China tension was quite limited.

Jared Brown: Yeah, and I’ll just throw out a quick plug for other FLI podcasts. I recommend listeners check out the work that we did with Matthew Meselson. Max Tegmark had a great podcast on the development of the Biological Weapons Convention, which is a great example of how two competing nation states came to a common understanding about what was essentially a global catastrophic, or is, a global catastrophic and existential risk and develop the biological weapons convention.

Lucas Perry: So, tabling collaboration on safety, which can certainly be mutually beneficial in just focusing on capabilities research and how at least it seems basically just rational to race for that in a game theoretic sense.

That seems basically just rational to race for that in a game theoretic sense. I’m interested in exploring if you guys have any views or points to add here about mitigating the risks there, and how it may simply actually not be rational to race for that?

Nicolas Moës: So, there is the narrative currently that it’s rational to race on some aspect of raw capabilities, right? However, when you go beyond the typical game theoretical model, when you enable people to build bridges, you could actually find certain circumstances under which you have a so-called institutional entrepreneur building up in institutions that is legitimate so that everybody agrees upon that enforces the cooporation agreement.

In economics, the windfall clause is regarding the distribution of it. Here what I’m talking about in the game theoretical space, is how to avoid the negative impact, right? So, the windfall clause would operate in this very limited set of scenarios whereby the AGI leads to an abundance of wealth, and then a windfall clause deals with the distributional aspect and therefore reduce the incentive to a certain extent to produce AGI. However, to abide to the windfall clause, you still have to preserve the incentive to develop the AGI. Right? But you might actually tamp that down.

What I was talking about here, regarding the institutional entrepreneur, who can break this race by simply having a credible commitment from both sides and enforcing that commitment. So like the typical model of the tragedy of the commons, which here could be seen as you over-explored the time to superintelligence level, you can solve the tragedy of the commons, actually. So it’s not that rational anymore. Once you know that there is a solution, it’s not rational to go for the worst case scenario, right? You actually can design a mechanism that forces you to move towards the better outcome. It’s costly though, but it can be done if people are willing to put in the effort, and it’s not costly enough to justify not doing it.

Jared Brown: I would just add that the underlying assumptions about the rationality of racing towards raw capability development, largely depend on the level of risk you assign to unaligned AI or deploying narrow AI in ways that exacerbate global catastrophic and existential risk. Those game theories essentially can be changed and those dynamics can be changed if our community eventually starts to better sensitize players on both sides about the lose/lose situation, which we could find ourselves in through this type of racing. And so it’s not set in stone and the environment can be changed as information asymmetry is decreased between the two competing partners and there’s a greater appreciation for the lose/lose situations that can be developed.

Lucas Perry: Yeah. So I guess I just want to highlight the point then the superficial first analysis, it would seem that the rational game theoretic thing to do is to increase capability as much as possible, so that you have power and security over other actors. But that might not be true under further investigation.

Jared Brown: Right, and I mean, for those people who haven’t had to suffer through game theory classes, there’s a great popular culture example here that a lot of people have seen Stranger Things on Netflix. If you haven’t, maybe skip ahead 20 seconds until I’m done saying this. But there is an example of the US and Russia competing to understand the upside down world, and then releasing untold havoc onto their societies, because of this upside down discovery. For those of you who have watched, it’s actually a fairly realistic example of where this kind of competing technological development leads somewhere that’s a lose/lose for both parties, and if they had better cooperation and better information sharing about the potential risks, because they were each discovering it themselves without communicating those risks, neither would have opened up the portals to the upside down world.

Nicolas Moës: The same dynamics, the same “oh it’s rational to race” dynamic applied to nuclear policy and nuclear arms race has led to, actually, some treaties, far from perfection. Right? But some treaties. So this is the thing where, because the model, the tragedy of the commons, it’s easy to communicate. It’s a nice thing was doom and fatality that is embedded with it. This resonates really well with people, especially in the media, it’s a very simple thing to say. But this simply might not be true. Right? As I mentioned. So there is this institutional entrepreneurship aspect which requires resources, right? So that is very costly to do. But civil society is doing that, and I think the Future of Life Institute has agency to do that. The Future Society is definitely doing that. We are actually agents of breaking away from these game theoretical situations that would be otherwise unlikely.

We fixate a lot on the model, but in reality, we have seen the nuclear policy, the worst case scenario being averted sometimes by mistake. Right? The human in the loop not following the policy or something like that. Right. So it’s interesting as well. It shows how unpredictable all this is. It really shows that for AI, it’s the same. You could have the militaries on both sides, literally from one day to the next, start a discussion about AI safety, and how to ensure that they keep control. There’s a lot of goodwill on both sides and so maybe we could say like, “Oh, the economist” — and I’m an economist by just training so I can be a bit harsh on myself — they’re like, the economist would say, “But this is not rational.” Well, in the end, it is more rational, right? So long as you win, you know, remain in a healthy life and feel like you have done the right thing, this is the rational thing to do. Maybe if Netflix is not your thing, “Inadequate Equilibria” by Eliezer Yudkowsky explores these kinds of conundrums as well. Why do you have sub-optimal situations in life in general? It’s a very, general model, but I found it very interesting to think about these issues, and in the end it boils down to these kinds of situations.

Lucas Perry: Yeah, right. Like for example, the United States and Russia having like 7,000 nuclear warheads each, and being on hair trigger alert with one another, is a kind of in-optimal equilibrium that we’ve nudged ourself into. I mean it maybe just completely unrealistic, but a more optimum place to be would be no nuclear weapons, but have used all of that technology and information for nuclear power. Well, we would all just be better off.

Nicolas Moës: Yeah. What you describe seems to be a better situation. However, the rational thing to do at some point would have been before the Soviet Union developed, incapacitate Soviet Union to develop. Now, the mutually assured destruction policy is holding up a lot of that. But I do believe that the diplomacy, the discussions, the communication, even merely the fact of communicating like, “Look, if you do that and we will do that,” is a form of progress towards: basically you should not use it.

Jared Brown: Game theory is nice to boil things down into a nice little boxes, clearly. But the dynamics of the nuclear situation with the USSR and the US add countless number of boxes that you get end up in and yes, each of us having way too large nuclear arsenals is a sub-optimal outcome, but it’s not the worst possible outcome, that would have been total nuclear annihilation. So it’s important not just to look at it criticisms of the current situation, but also see the benefits of this current situation and why this box is better than some other boxes that we ended up in. And that way, we can leverage the past that we have taken to get to where we’re at, find the paths that were actually positive, and reapply those lessons learned to the trajectory of emerging technology once again. We can’t throw out everything that has happened on nuclear policy and assume that there’s nothing to be gained from it, just because the situation that we’ve ended up in is suboptimal.

Nicolas Moës: Something that I have experienced while interacting with policymakers and diplomats. You actually have an agency over what is going on. This is important also to note, is that it’s not like a small thing, and the world is passing by. No. Even in policy, which seems to be maybe a bit more arcane, in policy, you can pull the right levers to make somebody feel less like they have to obey this race narrative.

Jared Brown: Just recently in the last National Defense Authorization Act, there was a provision talking about the importance of military to military dialogues being established, potentially even with adversarial states like North Korea and Iran, for that exact reason. That better communication between militaries can lead to a reduction of miscalculation, and therefore adverse escalation of conflicts. We saw this just recently between the US and Iran. There was not direct communication perhaps between the US and Iran, but there was indirect communication, some of that over Twitter, about the intentions and the actions that different states might take. Iran and the US, in reaction to other events, and that may have helped deescalate the situation to where we find now. It’s far from perfect, but this is the type of thing that civil society can help encourage as we are dealing with new types of technology that can be as dangerous as nuclear weapons.

Lucas Perry: I just want to touch on what is actually going on now and actually being considered before we wrap things up. You talked about this a little bit before, Jared, you mentioned that currently in terms of AI policy, we are moving from principles and recommendations to the implementation of these into hard law. So building off of this, I’m just trying to get a better sense of where AI policy is, currently. What are the kinds of things that have been implemented, and what hasn’t, and what needs to be done?

Jared Brown: So there are some key decisions that have to be made in the near term on AI policy that I see replicating in many different government environments. One of them is about liability. I think it’s very important for people to understand the influence that establishing liability has for safety considerations. By liability, I mean who is legally responsible if something goes wrong? The basic idea is if an autonomous vehicle crashes into a school bus, who’s going to be held responsible and under what conditions? Or if an algorithm is biased and systematically violates the civil rights of one minority group, who is legally responsible for that? Is it the creator of the algorithm, the developer of the algorithm? Is it the deployer of that algorithm? Is there no liability for anyone at all in that system? And governments writ large are struggling with trying to assign liability, and that’s a key area of governance and AI policy that’s occurring now.

For the most part, it would be wise for governments to not provide blanket liability to AI, simply as a matter of trying to encourage and foster the adoption of those technologies; such that we encourage people to essentially use those technologies in unquestioning ways and sincerely surrender the decision making from the human to that AI algorithm. There are other key issue areas. There is the question of educating the populace. The example here I give is, you hear the term financial literacy all the time about how educated is your populace about how to deal with money matters.

There’s a lot about technical literacy, technology literacy being developed. The Finnish government has a whole course on AI that they’re making available to the entire EU. How we educate our population and prepare our population from a workforce training perspective matters a lot. If that training incorporates considerations for common AI safety problems, if we’re training people about how adversarial examples can affect machine learning and so on and so forth, we’re doing a better job of sensitizing the population to potential longterm risks. That’s another example of where AI policy is being developed. And I’ll throw out one more, which is a common example that people will understand. You have a driver’s license from your state. The state has traditionally been responsible for deciding the human qualities that are necessary, in order for you to operate a vehicle. And the same goes for state licensing boards have been responsible for certifying and allowing people to practice the law or practice medicine.

Doctors and lawyers, there are national organizations, but licensing is typically done at the state. Now if we talk about AI starting to essentially replace human functions, governments have to look again at this division about who regulates what and when. There’s sort of an opportunity in all democracies to reevaluate the distribution of responsibility between units of government, about who has the responsibility to regulate and monitor and govern AI, when it is doing something that a human being used to do. And there are different pros and cons for different models. But suffice it to say that that’s a common theme in AI policy right now, is how to deal with who has the responsibility to govern AI, if it’s essentially replacing what used to be formally, exclusively a human function.

Nicolas Moës: Yeah, so in terms of where we stand, currently, actually let’s bring some context maybe to this question as well, right? The way it has evolved over the past few years is that you had really ethical principles in 2017 and 2018. Let’s look at the global level first. Like at the global level, you had for example, the Montréal Declaration, which was intended to be global, but for mostly fundamental rights-oriented countries, so that that excludes some of the key players. We have already talked about dozens and dozens of principles for AI in values context or in general, right. That was 2018, and then once we have seen is more the first multi-lateral guidelines so we have the OECD principles, GPAI which is this global panel on AI, was also a big thing between Canada and France, which was initially intended to become kind of the international body for AI governance, but that deflated a bit over time, and so you had also the establishment of all this fora for discussion, that I have already mentioned. Political AI summits and the Global Forum on AI for Humanity, which is, again, a Franco-Canadian initiative like the AI for Good. The Global Governance of AI Forum in the Middle East. There was this ethically aligned design initiative at the IEEE, which is a global standards center, which has garnered a lot of attention among policymakers and other stakeholders. But the move towards harder law is coming, and since it’s towards harder law, at the global level there is not much that can happen. Nation states remain sovereign in the eye of international law.

So unless you write up an international treaty, it would be at the government level that you have to move towards hard law. So at the global level, the next step that we can see is these audits and certification principles. It’s not hard law, but you use labels to independently certify whether an algorithm is good. Some of them are tailored for specific countries. So I think Denmark has its own certification mechanism for AI algorithms. The US is seeing the appearance of values initiatives, notably by the big consulting companies, which are all of the auditors. So this is something that is interesting to see how we shift from soft law, towards this industry-wide regulation for these algorithms. At the EU level, where you have some hard legislative power, you had also a high level group on liability. Which is very important, because they basically argued that we’re going to have to update product liability rules in certain ways for AI and for internet of things products.

This is interesting to look at as well, because when you look at product liability rules, this is hard law, right? So what they have recommended is directly translatable into this legislation. And so you move on at this stage since the end of 2019, you have this hard law coming up and this commission white paper which really kickstarts the debates about what will the regulation for AI be? And whether it will be a regulation. So it could be something else like a directive. The high level expert group has come up with a self assessment list for companies to see whether they are obeying the ethical principles decided upon in Europe. So these are kind of soft self regulation things, which might eventually affect court rulings or something like that. But they do not represent the law, and now the big players are moving in, either at the global level with these more and more powerful labeling initiatives, or certification initiatives, and at the EU level with this hard law.

And the reason why the EU level has moved on towards hard law so quickly, is because during the very short campaign of the commission president, AI was a political issue. The techlash was strong, and of course a lot of industry was complaining that there was nothing happening in AI in the EU. So they wanted strong action and that kind of stuff. The circumstances that led the EU to be in pole position for developing hard law. Elsewhere in the world, you actually have more fragmented initiatives at this stage, except the OECD AI policy observatory, which might be influential in itself, right? It’s important to note the AI principles that the OECD has published. Even though they are not binding, they would actually influence the whole debate. Right? Because at the international level, for example, when the OECD had privacy principles, this became the reference point for many legislators. So some countries who don’t want to spend years even debating how to legislate AI might just be like, “okay, here is the OECD principles, how do we implement that in our current body of law?” And that’s it.

Jared Brown: And I’ll just add one more quick dynamic that’s coming up with AI policy, which is essentially the tolerance of that government for the risk associated with emerging technology. A classic example here is, the US actually has a much higher level of flood risk tolerance than other countries. So we engineer largely, throughout the US, our dams and our flood walls and our other flood protection systems to a 1-in-100 year standard. Meaning the flood protection system is supposed to protect you from a severe storm that would have a 1% chance of occurring in a given year. Other countries have vastly different decisions there. Different countries make different policy decisions about the tolerance that they’re going to have for certain things to happen. And so as we think about emerging technology risk, it’s important to think about the way that your government is shaping policies and the underlying tolerance that they have for something going wrong.

It could be as simple as how likely it is that you will die because of an autonomous vehicle crash. And the EU, traditionally, has had what they call a precautionary principal approach, which is in the face of uncertain risks, they’re more likely to regulate and restrict development until those risks are better understood, than the US, which typically has adopted the precautionary principle less often.

Nicolas Moës: There is a lot of uncertainty. A lot of uncertainty about policy, but also a lot of uncertainty about the impact that all these technologies are having. The dam standard, you can quantify quite easily the force of nature, but here we are dealing with social forces that are a bit different. I still remember quite a lot of people being very negative about Facebook’s chances of success, because people would not be willing to put pictures of themself online. I guess 10 years later, these people have been proven wrong. The same thing could happen with AI, right? So people are currently, at least in the EU, afraid of some aspects of AI. So let’s say an autonomous vehicle. Surrendering decision-making about our life and death to an autonomous vehicle, that’s something that’s maybe as technology improves, people would be more and more willing to do that. So yeah, it’s very difficult to predict, and even more to quantify I think.

Lucas Perry: All right. So thank you both so much. Do either of you guys have any concluding thoughts about AI policy or anything else you’d just like to wrap up on?

Jared Brown: I just hope the audience really appreciates the importance of engaging in the policy discussion. Trying to map out a beneficial forward for AI policy, because if you’re concerned like we are about the long term trajectory of this emerging technology and other emerging technologies, it’s never too early to start engaging in the policy discussion on how to map a beneficial path forward.

Nicolas Moës: Yeah, and one last thought, we were talking with Jared a couple of days ago about the number of people doing that. So thank you by the way, Jared for inviting me, and Lucas, for inviting me on the podcast. But that led us to wonder how many people are doing what we are doing, with the motivation that we have regarding these longer term concerns. That makes me think, yeah, there’s very few resources like labor resources, financial resources, dedicated to this issue. And I’d be really interested if there is, in the audience, anybody interested in that issue, definitely, they should get in touch. There are too few people right now with similar motivations, and caring about the same thing in AI policy to actually miss the opportunity of meeting each other and coordinating better.

Jared Brown: Agreed.

Lucas Perry: All right. Wonderful. So yeah, thank you guys both so much for coming on.

End of recorded material

AI Alignment Podcast: Identity and the AI Revolution with David Pearce and Andrés Gómez Emilsson

 Topics discussed in this episode include:

  • Identity from epistemic, ontological, and phenomenological perspectives
  • Identity formation in biological evolution
  • Open, closed, and empty individualism
  • The moral relevance of views on identity
  • Identity in the world today and on the path to superintelligence and beyond

Timestamps: 

0:00 – Intro

6:33 – What is identity?

9:52 – Ontological aspects of identity

12:50 – Epistemological and phenomenological aspects of identity

18:21 – Biological evolution of identity

26:23 – Functionality or arbitrariness of identity / whether or not there are right or wrong answers

31:23 – Moral relevance of identity

34:20 – Religion as codifying views on identity

37:50 – Different views on identity

53:16 – The hard problem and the binding problem

56:52 – The problem of causal efficacy, and the palette problem

1:00:12 – Navigating views of identity towards truth

1:08:34 – The relationship between identity and the self model

1:10:43 – The ethical implications of different views on identity

1:21:11 – The consequences of different views on identity on preference weighting

1:26:34 – Identity and AI alignment

1:37:50 – Nationalism and AI alignment

1:42:09 – Cryonics, species divergence, immortality, uploads, and merging.

1:50:28 – Future scenarios from Life 3.0

1:58:35 – The role of identity in the AI itself

 

We hope that you will continue to join in the conversations by following us or subscribing to our podcasts on Youtube, Spotify, SoundCloud, iTunes, Google Play, StitcheriHeartRadio, or your preferred podcast site/application. You can find all the AI Alignment Podcasts here.

You can listen to the podcast above or read the transcript below. 

The transcript has been edited for style and clarity

Lucas Perry: Welcome to the AI Alignment Podcast. I’m Lucas Perry. Today we have an episode with Andres Gomez Emillson and David Pearce on identity. This episode is about identity from the ontological, epistemological and phenomenological perspectives. In less jargony language, we discuss identity from the fundamental perspective of what actually exists, of how identity arises given functional world models and self models in biological organisms, and of the subjective or qualitative experience of self or identity as a feature of consciousness. Given these angles on identity, we discuss what identity is, the formation of identity in biological life via evolution, why identity is important to explore and it’s ethical implications and implications for game theory, and  we directly discuss its relevance to the AI alignment problem and the project of creating beneficial AI.

I think the question of “How is this relevant to AI Alignment?” is useful to explore here in the intro. The AI Alignment problem can be construed in the technical limited sense of the question of “how to program AI systems to understand and be aligned with human values, preferences, goals, ethics, and objectives.” In a limited sense this is strictly a technical problem that supervenes upon research in machine learning, AI, computer science, psychology, neuroscience, philosophy, etc. I like to approach the problem of aligning AI systems from a broader and more generalist perspective. In the way that I think about the problem, a broader view of AI alignment takes into account the problems of AI governance, philosophy, AI ethics, and reflects deeply on the context in which the technical side of the problem will be taking place, the motivations of humanity and the human beings engaged in the AI alignment process, the ingredients required for success, and other civilization level questions on our way hopefully to beneficial superintelligence. 

It is from both of these perspectives that I feel exploring the question of identity is important. AI researchers have their own identities and those identities factor into their lived experience of the world, their motivations, and their ethics. In fact, the same is of course true of policy makers and anyone in positions of power to influence the alignment process, so being aware of commonly held identity models and views is important for understanding their consequences and functions in the world. From a macroscopic perspective, identity has evolved over the past 4.5 billion years on earth and surely will continue to do so in AI systems themselves and in the humans which hope to wield that power. Some humans may wish to merge, other to pass away or simply die, and others to be upgraded or uploaded in some way. Questions of identity are also crucial to this process of relating to one another and to AI systems in a rapidly evolving world where what it means to be human is quickly changing, where copies of digital minds or AIs can be made trivially, and the boundary between what we conventionally call the self and world begins to dissolve and break down in new ways, demanding new understandings of ourselves and identity in particular. I also want to highlight an important thought from the podcast that any actions we wish to take with regards to improving or changing understandings or lived experience of identity must be Sociologically relevant, or such interventions simply risk being irrelevant. This means understanding what is reasonable for human beings to be able to update their minds with and accept over certain periods of time and also the game theoretic implications of certain views of identity and their functional usefulness. This conversation is thus an attempt to broaden the conversation on these issues outside of what is normally discussed and to flag this area as something worthy of consideration.

For those not familiar with David Pearce or Andres Gomez Emilsson. David is a co-founder of the World Transhumanist Association, rebranded humanity plus, and is a prominent figure within the transhumanism movement in general. You might know him from his work on the Hedonistic Imperative, a book which explores our moral obligation to work towards the abolition of suffering in all sentient life through technological intervention. Andrés is a consciousness researcher at the Qualia Research Institute and is also the Co-founder and President of the Stanford Transhumanist Association. He has a Master’s in Computational Psychology from Stanford.

The Future of Life Institute is a non-profit and this podcast is funded and supported by listeners like you. So if you find what we do on this podcast to be important and beneficial, please consider supporting the podcast by donating at futureoflife.org/donate

 If you’d like to be a regular supporter, please consider a monthly subscription donation to make sure we can continue our efforts into the future. 

These contributions make it possible for us to bring you conversations like these and to develop the podcast further. You can also follow us on your preferred listening platform by searching for us directly or following the links on the page for this podcast found in the description. 

And with that, here is my conversation with Andres Gomez Emilsson and David Pearce 

I just want to start off with some quotes here that I think would be useful. The last podcast that we had was with Yuval Noah Harari and Max Tegmark. One of the points that Yuval really emphasized was the importance of self understanding questions like, who am I? What am I in the age of technology? Yuval all said “Get to know yourself better. It’s maybe the most important thing in life. We haven’t really progressed much in the last thousands of years, and the reason is that yes, we keep getting this advice, but we don’t really want to do it,” he goes on to say that, “especially as technology will give us all, at least some of us more and more power, the temptations of naive utopias are going to be more and more irresistible, and I think the really most powerful check on these naive utopias is really getting to know yourself better.”

In search of getting to know ourselves better, I want to explore this question of identity with both of you. To start off, what is identity?

David Pearce: One problem is that we have more than one conception of identity. There is the straightforward, logical sense that philosophers call the indiscernibility of identicals, namely that if A equals B, then anything true of A is true of B. In one sense, that’s trivially true, but when it comes to something like personal identity, it just doesn’t hold water at all. You are a different person from your namesake who went to bed last night – and it’s very easy carelessly to shift between these two different senses of identity.

Or one might speak of the United States. In what sense is the United States the same nation in 2020 as it was in 1975? It’s interest-relative.

Andrés Gómez Emilsson: Yeah and to go a little bit deeper on that, I would make the distinction as David made it between ontological identity, what fundamentally is actually going on in the physical world? In instantiated reality? Then there’s conventional identity definitely, the idea of continuing to exist from one moment to another as a human and also countries and so on.

Then there’s also phenomenological identity, which is our intuitive common sense view of: What are we and basically, what are the conditions that will allow us to continue to exist? We can go into more detail but yet, the phenomenological notion of identity is an incredible can of worms because there’s so many different ways of experiencing identity and all of them have their own interesting idiosyncrasies. Most people tend to confuse the two. They tend to confuse ontological and phenomenological identity. Just as a simple example that I’m sure we will revisit in the future, when a person has, let’s say an ego dissolution or a mystical experience and they feel that they merged with the rest of the cosmos, and they come out and say, “Oh, we’re all one consciousness.” That tends to be interpreted as some kind of grasp of an ontological reality. Whereas we could argue in a sense that that was just the shift in phenomenological identity, that your sense of self got transformed, not necessarily that you’re actually directly merging with the cosmos in a literal sense. Although, of course it might be very indicative of how conventional our sense of identity is if it can be modified so drastically in other states of consciousness.

Lucas Perry: Right, and let’s just start with the ontological sense. How does one understand or think about identity from the ontological side?

Andrés Gómez Emilsson: In order to reason about this, you need a shared frame of reference for what actually exists, and a number of things including the nature of time and space, and memory because in the common sense view of time called presentism, where basically there’s just the present moment, the past is a convenient construction and the future is a fiction useful in practical sense, but they don’t literally exist in that sense. This notion that A equals B in the sense of, Hey, you could modify what happens to A and that will automatically also modify what happens to B. It kind of makes sense and you can perhaps think of identity is moving over time along with everything else.

On the other hand, if you have an eternalist point of view where basically you interpret the whole of space time as just basically there, on their own coordinates in the multiverse, that kind of provides a different notion of ontological identity because it’s in a sense, a moment of experience is its own separate piece of reality.

In addition, you also need to consider the question of connectivity: in what way different parts of reality are connected to each other? In a conventional sense, as you go from one second to the next, you’ve continued to be connected to yourself in an unbroken stream of consciousness and this has actually led some philosophers to hypothesize that the proper unit of identity is from the moment your wake up to the moment in which you go to sleep because that’s an unbroken chain/stream of consciousness.

From a scientific and philosophically rigorous point of view, it’s actually difficult to make the case that our stream of consciousness is truly unbroken. Definitely if you have an eternalist point of view on experience and on the nature of time, what you will instead see is from the moment you wake up to the moment you go to sleep, there’s actually been an extraordinarily large amount of snapshots of discrete and moments of experience. In that sense, each of those individual moments of experiences would be its own ontologically separate individual.

Now one of the things that becomes kind of complicated with a kind of an eternalist account of time and identity is that you cannot actually change it. There’s nothing you can actually do to A, so that reasoning of if you do anything to A an A equals B, then the same will happen to B, doesn’t even actually apply in here because everything is already there. You cannot actually modify A any more than you can modify the number five.

David Pearce: Yes, it’s a rather depressing perspective in many ways, the eternalist view. If one internalizes it too much, it can lead to a sense of fatalism and despair. A lot of the time it’s probably actually best to think of the future as open.

Lucas Perry: This helps to clarify some of the ontological part of identity. Now, you mentioned this phenomenological aspect and I want to say also the epistemological aspect of identity. Could you unpack those two? And maybe clarify this distinction for me if you wouldn’t parse it this way? I guess I would say that the epistemological one is the models that human beings have about the world and about ourselves. It includes how the world is populated with a lot of different objects that have identity like humans and planets and galaxies. Then we have our self model, which is the model of our body and our space in social groups and who we think we are.

Then there’s the phenomenological identity, which is that subjective qualitative experience of self or the ego in relation to experience. Or where there’s an identification with attention and experience. Could you unpack these two later senses?

Andrés Gómez Emilsson: Yeah, for sure. I mean in a sense you could have like an implicit self model that doesn’t actually become part of your consciousness or it’s not necessarily something that you’re explicitly rendering. This goes on all the time. You’ve definitely, I’m sure, had the experience of riding a bicycle and after a little while you can almost do it without thinking. Of course, you’re engaging with the process in a very embodied fashion, but you’re not cognizing very much about it. Definitely you’re not representing, let’s say your body state, or you’re representing exactly what is going on in a cognitive way. It’s all kind of implicit in the way in which you feel. I would say that paints a little bit of a distinction between a self model which is ultimately functional. It has to do with, are you processing the information that you’re required to solve the task that involves modeling what you are in your environment and distinguishing it from the felt sense of, are you a person? What are you? How are you located and so on.

The first one is the one that most of robotics and machine learning, that have like an embodied component, are really trying to get at. You just need the appropriate information processing in order to solve the task. They’re not very concerned about, does this feel like anything? Or does it feel like a particular entity or a self to be that particular algorithm?

Whereas, we’re talking about the phenomenological sense of identity. That’s very explicitly about how it feels like and there’s all kinds of ways in which a healthy so to speak, sense of identity, can be broken down in all sorts of interesting ways. There’s many failure modes, we can put it that way.

One might argue, I mean I suspect for example, David Pearce might say this, which is that, our self models or our implicit sense of self, because of the way in which it was brought up through Darwinian selection pressures, is already extremely ill in some sense at least, from the point of view of it, it actually telling us something true and actually making us do something ethical. It has all sorts of problems, but it is definitely functional. You can anticipate being a person tomorrow and plan accordingly. You leave messages to yourself by encoding them in memory and yeah, this is a convenient sense of conventional identity.

It’s very natural for most people’s experiences. I can briefly mention a couple of ways in which it can break down. One of them is depersonalization. It’s a particular psychological disorder where one stops feeling like a person, and it might have something to do with basically, not being able to synchronize with your bodily feelings in such a way that you don’t actually feel embodied. You may feel this incarnate entity or just a witness experiencing a human experience, but not actually being that person.

Then you also have things such as empathogen induced sense of shared identity with others. If you’d take MDMA, you may feel that all of humanity is deeply connected, or we’re all part of the same essence of humanity in a very positive sense of identity, but perhaps not in an evolutionary adaptive sense. Finally, is people with a multiple personality disorder, where in a sense they have a very unstable sense of who they are and sometimes it can be so extreme that there’s epistemological blockages from one sense of self to another.

David Pearce: As neuroscientist Donald Hoffman likes to say, fitness trumps truth. Each of us runs a world-simulation. But it’s not an impartial, accurate, faithful world-simulation. I am at the center of a world-simulation, my egocentric world, the hub of reality that follows me around. And of course there are billions upon billions of other analogous examples too. This is genetically extremely fitness-enhancing. But it’s systematically misleading. In that sense, I think Darwinian life is malware.

Lucas Perry: Wrapping up here on these different aspects of identity, I just want to make sure that I have all of them here. Would you say that those are all of the aspects?

David Pearce: One can add the distinction between type- and token- identity. In principle, it’s possible to create from scratch a molecular duplicate of you. Is that person you? It’s type-identical, but it’s not token-identical.

Lucas Perry: Oh, right. I think I’ve heard this used in some other places as numerical distinction versus qualitative distinction. Is that right?

David Pearce: Yeah, that’s the same distinction.

Lucas Perry: Unpacking here more about what identity is. Let’s talk about it purely as something that the world has produced. What can we say about the evolution of identity in biological life? What is the efficacy of certain identity models in Darwinian evolution?

Andrés Gómez Emilsson: I would say that self models most likely have existed, potentially since pretty early on in the evolutionary timeline. You may argue that in some sense even bacteria has some kind of self model. But again, a self model is really just functional. The bacteria does need to know, at least implicitly, it’s size in order to be able to navigate it’s environment, follow chemical gradients, and so on, not step on itself. That’s not the same, again, as a phenomenal sense of identity, and that one I would strongly suspect came much later. Perhaps with the advent of the first primitive nervous systems. That would be only if actually running that phenomenal model is giving you some kind of fitness advantage.

One of the things that you will encounter with David and I is that we think that phenomenally bound experiences have a lot of computational properties and in a sense, the reason why we’re conscious has to do with the fact that unified moments of experience are doing computationally useful legwork. It comes when you merge implicit self models in just the functional sense together with the computational benefits of actually running a conscious system that, perhaps for the first time in history, you will actually have a phenomenal self model.

I would suspect probably in the Cambrian explosion this was already going on to some extent. All of these interesting evolutionary oddities that happen in the Cambrian explosion probably had some kind of rudimentary sense of self. I would be skeptical that is going on.

For example, in plants. One of the key reasons is that running a real time world simulation in a conscious framework is very calorically expensive.

David Pearce: Yes, it’s a scandal. What, evolutionarily speaking, is consciousness “for”? What could a hypothetical p-zombie not do? The perspective that Andrés and I are articulating is that essentially what makes biological minds special is phenomenal binding – the capacity to run real-time, phenomenally-bound world-simulations, i.e. not just be 86 billion discrete, membrane-bound pixels of experience. Somehow, we each generate an entire cross-modally matched, real-time world-simulation, made up of individual perceptual objects, somehow bound into a unitary self. The unity of perception is extraordinarily computationally powerful and adaptive. Simply saying that it’s extremely fitness-enhancing doesn’t explain it, because something like telepathy would be extremely fitness-enhancing too, but it’s physically impossible.

Yes, how biological minds manage to run phenomenally-bound world-simulations is unknown: they would seem to be classically impossible. One way to appreciate just how advantageous is (non-psychotic) phenomenal binding is to look at syndromes where binding even partially breaks down: simultanagnosia, where one can see only one object at once, or motion blindness (akinetopsia), where you can’t actually see moving objects, or florid schizophrenia. Just imagine those syndromes combined. Why aren’t we just micro-experiential zombies?

Lucas Perry: Do we have any interesting points here to look at in the evolutionary tree for where identity is substantially different from ape consciousness? If we look back at human evolution, it seems that it’s given the apes and particularly our species a pretty strong sense of self, and that gives rise to much of our ape socialization and politics. I’m wondering if there was anything else like maybe insects or other creatures that have gone in a different direction? Also if you guys might be able to just speak a little bit on the formation of ape identity.

Andrés Gómez Emilsson: Definitely I think like the perspective of the selfish gene, it’s pretty illuminating here. Nominally, our sense of identity is the sense of one person, one mind. In practice however, if you make sense of identity as well in terms of that which you want to defend, or that of which you consider worth preserving, you will see that people’s sense of identity also extends to their family members and of course, with the neocortex and the ability to create more complex associations. Then you have crazy things like sense of identity being based on race or country of origin or other constructs like that.are building on top of imports from the sense of, hey, the people who are familiar to you feel more like you.

It’s genetically adaptive to have that and from the point of view of the selfish gene, genes that could recognize themselves in others and favor the existence of others that also share the same genes, are more likely to reproduce. That’s called the inclusive fitness in biology, you’re not just trying to survive yourself or make copies of yourself, you’re also trying to help those that are very similar to you do the same. Almost certainly, it’s a huge aspect of how we perceive the world. Just anecdotally from a number of trip reports, there’s this interesting thread of how some chemicals like MDMA and 2CB, for those who don’t know, it’s these empathogenic psychedelics, that people get the strange sense that people they’ve never met before in their life are as close to them as a cousin, or maybe a half brother, or half sister. It’s a very comfortable and quite beautiful feeling. You could imagine that nature was very selective on who do you give that feeling to in order to maximize inclusive fitness.

All of this builds up to the overall prediction I would make that, the sense of identity of ants and other extremely social insects might be very different. The reason being that they are genetically incentivized to basically treat each other as themselves. Most ants themselves don’t produce any offspring. They are genetically sisters and all of their genetic incentives are into basically helping the queen pass on the genes into other colonies. In that sense, I would imagine an ant probably sees other ants of the same colony pretty much as themselves.

David Pearce: Yes. There was an extraordinary finding a few years ago: members of one species of social ant actually passed the mirror test – which has traditionally been regarded as the gold standard for the concept of a self. It was shocking enough, to many people, when a small fish was shown to be capable of mirror self-recognition. If some ants too can pass the mirror test, it suggests some form of meta-cognition, self-recognition, that is extraordinarily ancient.

What is it that distinguishes humans from nonhuman animals? I suspect the distinction relates to something that is still physically unexplained: how is it that a massively parallel brain gives rise to serial, logico-linguistic thought? It’s unexplained, but I would say this serial stream is what distinguishes us, most of all – not possession of a self-concept.

Lucas Perry: Is there such a thing as a right answer to questions of identity? Or is it fundamentally just something that’s functional? Or is it ultimately arbitrary?

Andrés Gómez Emilsson: I think there is the right answer. From a functional perspective, there’s just so many different ways of thinking about it. As I was describing perhaps with ants and humans, their sense of identity is probably pretty different. But, they both are useful for passing on the genes. In that sense they’re all equally valid. Imagine in the future is some kind of a swarm mind that also has its own distinct functionally adaptive sense of identity, and I mean in that sense that it ground truth to what it should be from the point of view of functionality. It really just depends on what is the replication unit.

Ontologically though, I think there’s a case to be made that either or empty individualism are true. Maybe it would be good to define those terms first.

Lucas Perry: Before we do that. Your answer then is just that, yes, you suspect that also ontologically in terms of fundamental physics, there are answers to questions of identity? Identity itself isn’t a confused category?

Andrés Gómez Emilsson: Yeah, I don’t think it’s a leaky reification as they say.

Lucas Perry: From the phenomenological sense, is the self an illusion or not? Is the self a valid category? Is your view also on identity that there is a right answer there?

Andrés Gómez Emilsson: From the phenomenological point of view? No, I would consider it a parameter, mostly. Just something that you can vary, and there’s trade offs or different experiences of identity.

Lucas Perry: Okay. How about you David?

David Pearce: I think ultimately, yes, there are right answers. In practice, life would be unlivable if we didn’t maintain these fictions. These fictions are (in one sense) deeply immoral. We punish someone for a deed that their namesake performed, let’s say 10, 15, 20 years ago. America recently executed a murderer for a crime that was done 20 years ago. Now quite aside from issues of freedom and responsibility and so on, this is just scapegoating.

Lucas Perry: David, do you feel that in the ontological sense there are right or wrong answers to questions of identity? And in the phenomenological sense? And in the functional sense?

David Pearce: Yes.

Lucas Perry: Okay, so then I guess you disagree with Andres about the phenomenological sense?

David Pearce: I’m not sure, Andrés and I agree about most things. Are we disagreeing Andrés?

Andrés Gómez Emilsson: I’m not sure. I mean, what I said about the phenomenal aspect of identity was that I think of it as a parameter of our world simulation. In that sense, there’s no true phenomenological sense of identity. They’re all useful for different things. The reason I would say this too is, you can assume that something like each snapshot of experience, is its own separate identity. I’m not even sure you can accurately represent that in a moment of experience itself. This is itself a huge can of worms that opens up the problem of referents. Can we even actually refer to something from our own point of view? My intuition here is that, whatever sense of identity you have at a phenomenal level, I think of it as a parameter of the world simulation and I don’t think it can be an accurate representation of something true. It’s just going to be a feeling, so to speak.

David Pearce: I could endorse that. We fundamentally misperceive each other. The Hogan sisters, conjoined craniopagus twins, know something that the rest of us don’t. The Hogan sisters share a thalamic bridge, which enables them partially, to a limited extent, to “mind-meld”. The rest of us see other people essentially as objects that have feelings. When one thinks of one’s own ignorance, perhaps one laments one’s failures as a mathematician or a physicist or whatever; but an absolutely fundamental form of ignorance that we take for granted is we (mis)conceive other people and nonhuman animals as essentially objects with feelings, whereas individually, we ourselves have first-person experience. Whether it’s going to be possible to overcome this limitation in the future I don’t know. It’s going to be immensely technically challenging – building something like reversible thalamic bridges. A lot depends on one’s theory of phenomenal binding. But let’s imagine a future civilization in which partial “mind-melding” is routine. I think it will lead to a revolution not just in morality, but in decision-theoretic rationality too – one will take into account the desires, the interests, and the preferences of what will seem like different aspects of oneself.

Lucas Perry: Why does identity matter morally? I think you guys have made a good case about how it’s important functionally, historically in terms of biological evolution, and then in terms of like society and culture identity is clearly extremely important for human social relations, for navigating social hierarchies and understanding one’s position of having a concept of self and identity over time, but why does it matter morally here?

Andrés Gómez Emilsson: One interesting story where you can think of a lot of social movements, in a sense, a lot of ideologies that have existed in human history, as attempts to hack people’s sense of identities or make use of them for the purpose of the reproduction of the ideology or the social movement itself.

To a large extent, a lot of the things that you see in a therapy have a lot to do with expanding your sense of identity to include your future self as well, which is something that a lot of people struggle with when it comes to impulsive decisions or your rationality. There’s these interesting point of view of how a two year old or a three year old, hasn’t yet internalized the fact that they will wake up tomorrow and that the consequences of what they did today will linger on in the following days. This is kind of a revelation when a kid finally internalizes the fact that, Oh my gosh, I will continue to exist for the rest of my life. There’s going to be a point where I’m going to be 40 years old and also there’s going to be a time where I’m 80 years old and all of those are real, and I should plan ahead for it.

Ultimately, I do think that advocating for a very inclusive sense of identity, where the locus of identity is consciousness itself. I do think that might be a tremendous moral and ethical implications.

David Pearce: We want an inclusive sense of “us” that embraces all sentient beings.  This is extremely ambitious, but I think that should be the long-term goal.

Lucas Perry: Right, there’s a spectrum here and where you fall on the spectrum will lead to different functions and behaviors, solipsism or extreme egoism on one end, pure selflessness or ego death or pure altruism on the other end. Perhaps there are other degrees and axes on which you can move, but the point is it leads to radically different identifications and relations with other sentient beings and with other instantiations of consciousness.

David Pearce: Would our conception of death be different if it was a convention to give someone a different name when they woke up each morning? Because after all, waking up is akin to reincarnation. Why is it that when one is drifting asleep each night, one isn’t afraid of death? It’s because (in some sense) one believes one is going to be reincarnated in the morning.

Lucas Perry: I like that. Okay, I want to return to this question after we hit on the different views of identity to really unpack the different ethical implications more. I wanted to sneak that in here for a bit of context. Pivoting back to this sort of historical and contextual analysis of identity. We talked about biological evolution as like instantiating these things. How do you guys view religion as codifying an egoist view on identity? Religion codifies the idea of the eternal soul and the soul, I think, maps very strongly onto the phenomenological self. It makes that the thing that is immutable or undying or which transcends this realm?

I’m talking obviously specifically here about Abrahamic religions, but then also in Buddhism there is, the self is an illusion, or what David referred to as empty individualism, which we’ll get into, where it says that identification with the phenomenological self is fundamentally a misapprehension of reality and like a confusion and that that leads to attachment and suffering and fear of death. Do you guys have comments here about religion as codifying views on identity?

Andrés Gómez Emilsson: I think it’s definitely really interesting that there are different views of identity and religion. How I grew up, I always assumed religion was about souls and getting into heaven. As it turns out, I just needed to know about Eastern religions and cults. That also happened to sometimes have like different views of personal identity. That was definitely a revelation to me. I would actually say that I started questioning the sense of a common sense of personal identity before I learned about Eastern religions and I was really pretty surprised and very happy when I found out that, let’s say Hinduism actually, it has a kind of universal consciousness take on identity, a socially sanctioned way of looking at the world that has a very expansive sense of identity. Buddhism is also pretty interesting because as far as I understand it, they consider actually pretty much any view of identity to be a cause for suffering fundamentally has to do with a sense of craving either for existence or craving for non-existence, which they also consider a problem. A Buddhist would generally say that even something like universal consciousness, believing that we’re all fundamentally Krishna incarnating in many different ways, itself will also be a source of suffering to some extent because you may crave further existence, which may not be very good from their point of view. It makes me optimistic that there’s other types of religions with other views of identity.

David Pearce: Yes. Here is one of my earliest memories. My mother belonged to The Order of the Cross – a very obscure, small, vaguely Christian denomination, non-sexist, who worship God the Father-Mother. And I recall being told, aged five, that I could be born again. It might be as a little boy, but it might be as a little girl – because gender didn’t matter. And I was absolutely appalled at this – at the age of five or so – because in some sense girls were, and I couldn’t actually express this, defective.

And religious conceptions of identity vary immensely. One thinks of something like Original Sin in Christianity. I could now make a lot of superficial comments about religion. But one would need to explore in detail the different religious traditions and their different conceptions of identity.

Lucas Perry: What are the different views on identity? If you can say anything, why don’t you hit on the ontological sense and the phenomenological sense? Or if we just want to stick to the phenomenological sense then we can.

Andrés Gómez Emilsson: I mean, are you talking about an open, empty, closed?

Lucas Perry: Yeah. So that would be the phenomenological sense, yeah.

Andrés Gómez Emilsson: No, actually I would claim those are attempts at getting at the ontological sense.

Lucas Perry: Okay.

Andrés Gómez Emilsson: If you do truly have a soul ontology, something that implicitly a very large percentage of the human population have, that would be, yeah, in this view called a closed individualist perspective. Common sense, you start existing when you’re born, you stop existing when you die, you’re just a stream of consciousness. Even perhaps more strongly, you’re a soul that has experiences, but experiences maybe are not fundamental to what you are.

Then there is the more Buddhist and definitely more generally scientifically-minded view, which is empty individualism, which is that you only exist as a moment of experience, and from one moment to the next that you are a completely different entity. And then, finally, there is open individualism, which is like Hinduism claiming that we are all one consciousness fundamentally.

There is an ontological way of thinking of these notions of identity. It’s possible that a lot of people think of them just phenomenologically, or they may just think there’s no further fact beyond the phenomenal. In which case something like that closed individualism, for most people most of the time, is self-evidently true because you are moving in time and you can notice that you continue to be yourself from one moment to the next. Then, of course, what would it feel like if you weren’t the same person from one moment to the next? Well, each of those moments might completely be under the illusion that it is a continuous self.

For most things in philosophy and science, if you want to use something as evidence, it has to agree with one theory and disagree with another one. And the sense of continuity from one second to the next seems to be compatible with all three views. So it’s not itself much evidence either way.

States of depersonalization are probably much more akin to empty individualism from a phenomenological point of view, and then you have ego death and definitely some experiences of the psychedelic variety, especially high doses of psychedelics tend to produce very strong feelings of open individualism. That often comes in the form of noticing that your conventional sense of self is very buggy and doesn’t seem to track anything real, but then realizing that you can identify with awareness itself. And if you do that, then in some sense automatically, you realize that you are every other experience out there, since the fundamental ingredient of a witness or awareness is shared with every conscious experience.

Lucas Perry: These views on identity are confusing to me because agents haven’t existed for most of the universe and I don’t know why we need to privilege agents in our ideas of identity. They seem to me just emergent patterns of a big, ancient, old, physical universe process that’s unfolding. It’s confusing to me that just because there are complex self- and world-modeling patterns in the world, that we need to privilege them with some kind of shared identity across themselves or across the world. Do you see what I mean here?

Andrés Gómez Emilsson: Oh, yeah, yeah, definitely. I’m not agent-centric. And I mean, in a sense also, all of these other exotic feelings of identity often also come with states of low agency. You actually don’t feel that you have much of a choice in what you could do. I mean, definitely depersonalization, for example, often comes with a sense of inability to make choices, that actually it’s not you who’s making the choice, they’re just unfolding and happening. Of course, in some meditative traditions that’s considered a path to awakening, but in practice for a lot of people, that’s a very unpleasant type of experience.

It sounds like it might be privileging agents; I would say that’s not the case. If you zoom out and you see the bigger worldview, it includes basically this concept, David calls it non-materialist physicalist idealism, where the laws of physics describe the behavior of the universe, but that which is behaving according to the laws of physics is qualia, is consciousness itself.

I take very seriously the idea that a given molecule or a particular atom contains moments of experience, it’s just perhaps very fleeting and very dim or are just not very relevant in many ways, but I do think it’s there. And sense of identity, maybe not in a phenomenal sense, I don’t think an atom actually feels like an agent over time, but continuity of its experience and the boundaries of its experience would have strong bearings on ontological sense of identity.

There’s a huge, obviously, a huge jump between talking about the identity of atoms and then talking about the identity of a moment of experience, which presumably is an emergent effect of 100 billion neurons, themselves made of so many different atoms. Crazy as it may be, it is both David Pearce’s view and my view that actually each moment of experience does stand as an ontological unit. It’s just the ontological unit of a certain kind that usually we don’t see in physics, but it is both physical and ontologically closed.

Lucas Perry: Maybe you could unpack this. You know mereological nihilism, maybe I privilege this view where I just am trying to be as simple as possible and not build up too many concepts on top of each other.

Andrés Gómez Emilsson: Mereological nihilism basically says that there are no entities that have parts. Everything is part-less. All that exists in reality is individual monads, so to speak, things that are fundamentally self-existing. For that, if you have let’s say monad A and monad B, just put together side by side, that doesn’t entail that now there is a monad AB that mixes the two.

Lucas Perry: Or if you put a bunch of fundamental quarks together that it makes something called an atom. You would just say that it’s quarks arranged atom-wise. There’s the structure and the information there, but it’s just made of the monads.

Andrés Gómez Emilsson: Right. And the atom is a wonderful case, basically the same as a molecule, where I would say mereological nihilism with fundamental particles as just the only truly existing beings does seem to be false when you look at how, for example, molecules behave. The building block account of how chemical bonds happen, which is with these Lewis diagrams of how it can have a single bond or double bond and you have the octet rule, and you’re trying to build these chains of atoms strung together. And all that matters for those diagrams is what each atom is locally connected to.

However, if you just use these in order to predict what molecules are possible and how they behave and their properties, you will see that there’s a lot of artifacts that are empirically disproven. And over the years, chemistry has become more and more sophisticated where eventually, it’s come to the realization that you need to take into account the entire molecule at once in order to understand what its “dynamically stable” configuration, which involves all of the electrons and all of the nuclei simultaneously interlocking into a particular pattern that self replicates.

Lucas Perry: And it has new properties over and above the parts.

Andrés Gómez Emilsson: Exactly.

Lucas Perry: That doesn’t make any sense to me or my intuitions, so maybe my intuitions are just really wrong. Where does the new property or causality come from? Because it essentially has causal efficacy over and above the parts.

Andrés Gómez Emilsson: Yeah, it’s tremendously confusing. I mean, I’m currently writing an article about basically how this sense of topological segmentation can, in a sense, account both for this effect of what we might call weak downward causation, which is like, you get a molecule and now the molecule will have effects in the world; that you need to take into account all of the electrons and all of the nuclei simultaneously as a unit in order to actually know what the effect is going to be in the world. You cannot just take each of the components separately, but that’s something that we could call weak downward causation. It’s not that fundamentally you’re introducing a new law of physics. Everything is still predicted by Schrödinger equation, it’s still governing the behavior of the entire molecule. It’s just that the appropriate unit of analysis is not the electron, but it would be the entire molecule.

Now, if you pair this together with a sense of identity that comes from topology, then I think there might be a good case for why moments of experience are discrete entities. The analogy here with the topological segmentation, hopefully I’m not going to lose too many listeners here, but we can make an analogy with, for example, a balloon. That if you start out imagining that you are the surface of the balloon and then you take the balloon by two ends and you twist them in opposite directions, eventually at the middle point you get what’s called a pinch point. Basically, the balloon collapses in the center and you end up having these two smooth surfaces connected by a pinch point. Each of those twists creates a new topological segment, or in a sense is segmenting out the balloon. You could basically interpret things such as molecules as new topological segmentations of what’s fundamentally the quantum fields that is implementing them.

Usually, the segmentations may look like an electron or a proton, but if you assemble them together just right, you can get them to essentially melt with each other and become one topologically continuous unit. The nice thing about this account is that you get everything that you want. You explain, on the one hand, why identity would actually have causal implications, and it’s this weak downward causation effect, at the same time as being able to explain: how is it possible that the universe can break down into many different entities? Well, the answer is the way in which it is breaking down is through topological segmentations. You end up having these self-contained regions of the wave function that are discommunicated from the rest of it, and each of those might be a different subject of experience.

David Pearce: It’s very much an open question: the intrinsic nature of the physical. Commonly, materialism and physicalism are conflated. But the point of view that Andrés and I take seriously, non-materialist physicalism, is actually a form of idealism. Recently, philosopher Phil Goff, who used to be a skeptic-critic of non-materialist physicalism because of the binding problem, published a book defending it, “Galileo’s Error”.

Again, it’s very much an open question. We’re making some key background assumptions here. A critical background assumption is physicalism, and that quantum mechanics is complete:  there is no “element of reality” that is missing from the equations (or possibly the fundamental equation) of physics. But physics itself seems to be silent on the intrinsic nature of the physical. What is the intrinsic nature of a quantum field? Intuitively, it’s a field of insentience; but this isn’t a scientific discovery, it’s a (very strong) philosophical intuition.

And if you couple this with the fact that the only part of the world to which one has direct access, i.e., one’s own conscious mind (though this is controversial), is consciousness, sentience. The non-materialist physicalist conjectures that we are typical, in one sense – inasmuch as the fields of your central nervous system aren’t ontologically different from the fields of the rest of the world. And what makes sentient beings special is the way that fields are organized into unified subjects of experience, egocentric world-simulations.

Now, I’m personally fairly confident that we are, individually, minds running egocentric world-simulations: direct realism is false. I’m not at all confident – though I explore the idea – that experience is the intrinsic nature of the physical, the “stuff” of the world. This is a tradition that goes back via Russell, ultimately, to Schopenhauer. Schopenhauer essentially turns Kant on his head.

Kant famously said that all we will ever know is phenomenology, appearances; we will never, never know the intrinsic, noumenal nature of the world. But Schopenhauer argues that essentially we do actually know one tiny piece of the noumenal essence of the world, the essence of the physical, and it’s experiential. So yes, tentatively, at any rate, Andrés and I would defend non-materialist or idealistic physicalism. The actual term “non-materialist physicalism” is due to the late Grover Maxwell.

Lucas Perry: Sorry, could you just define that real quick? I think we haven’t.

David Pearce: Physicalism is the idea that no “element of reality” is missing from the equations of physics, presumably (some relativistic generalization of) the universal Schrödinger equation.

Lucas Perry: It’s a kind of naturalism, too.

David Pearce: Oh, yes. It is naturalism. There are some forms of idealism and panpsychism that are non-naturalistic, but this view is uncompromisingly monist. Non-materialist physicalism isn’t claiming that a primitive experience is attached in some way to fundamental physical properties. The idea is that the actual intrinsic nature, the essence of the physical, is experiential.

Stephen Hawking, for instance, was a wave function monist. A doctrinaire materialist, but he famously said that we have no idea what breathed fire into the equations and makes the universe first to describe. Now, intuitively, of course one assumes that the fire in the equations, Kant’s noumenal essence of the world, is non-experiential. But if so, we have the hard problem, we have the binding problem, we have the problem of causal efficacy, a great mess of problems.

But if, and it’s obviously a huge if, the actual intrinsic nature of the physical is experiential, then we have a theory of reality that is empirically adequate, that has tremendous explanatory and predictive power. It’s mind-bogglingly implausible, at least to those of us steeped in the conceptual framework of materialism. But yes, by transposing the entire mathematical apparatus of modern physics, quantum field theory or its generalization, onto an idealist ontology, one actually has a complete account of reality that explains the technological successes of science, its predictive power, and doesn’t give rise to such insoluble mysteries as the hard problem.

Lucas Perry: I think all of this is very clarifying. There are also background metaphysical views, which people may or may not disagree upon, which are also important for identity. I also want to be careful to define some terms, in case some listeners don’t know what they mean. I think you hit on like four different things which all had to do with consciousness. The hard problem is why different kinds of computation actually… why it’s something to be that computation or like why there is consciousness correlated or associated with that experience.

Then you also said the binding problem. Is it the binding problem, why there is a unitary experience that’s, you said, modally connected earlier?

David Pearce: Yes, and if one takes the standard view from neuroscience that your brain consists of 86-billion-odd discrete, decohered, membrane-bound nerve cells, then phenomenal binding, whether local or global, ought to be impossible. So yeah, this is the binding problem, this (partial) structural mismatch. If your brain is scanned when you’re seeing a particular perceptual object, neuroscanning can apparently pick out distributed feature-processors, edge-detectors, motion-detectors, color-mediating neurons (etc). And yet there isn’t the perfect structural match that must exist if physicalism is true. And David Chalmers – because of this (partial) structural mismatch – goes on to argue that dualism must be true. Although I agree with David Chalmers that yes, phenomenal binding is classically impossible, if one takes the intrinsic nature argument seriously, then phenomenal unity is minted in.

The intrinsic nature argument, recall, is that experience, consciousness, discloses the intrinsic nature of the physical. Now, one of the reasons why this idea is so desperately implausible is it makes the fundamental “psychon” of consciousness ludicrously small. But there’s a neglected corollary of non-materialist physicalism, namely that if experience discloses the intrinsic nature of the physical, then experience must be temporally incredibly fine-grained too. And if we probe your nervous system at a temporal resolution of femtoseconds or even attoseconds, what would we find? My guess is that it would be possible to recover a perfect structural match between what you are experiencing now in your phenomenal world-simulation and the underlying physics. Superpositions (“cat states”) are individual states [i.e. not classical aggregates].

Now, if the effective lifetime of neuronal superpositions and the CNS were milliseconds, they would be the obvious candidate for a perfect structural match and explain the phenomenal unity of consciousness. But physicists, not least Max Tegmark, have done the maths: decoherence means that the effective lifetime of neuronal superpositions in the CNS, assuming the unitary-only dynamics, is femtoseconds or less, which is intuitively the reductio ad absurdum of any kind of quantum mind.

But one person’s reductio ad absurdum is another person’s falsifiable prediction. I’m guessing – I’m sounding like a believer, but I’m not –  I am guessing that with sufficiently sensitive molecular matter- wave interferometry, perhaps using “trained up” mini-brains, that the non-classical interference signature will disclose a perfect structural match between what you’re experiencing right now, your unified phenomenal world-simulation, and the underlying physics.

Lucas Perry: So, we hit on the hard problem and also the binding problem. There was like two other ones that you threw out there earlier that… I forget what they were?

David Pearce: Yeah, the problem of causal efficacy. How is it that you and I can discuss consciousness? How is it that the “raw feels” of consciousness have not merely the causal, but also the functional efficacy to inspire discussions of their existence?

Lucas Perry: And then what was the last one?

David Pearce: Oh, it’s been called the palette problem, P-A-L-E-T-T-E. As in the fact that there is tremendous diversity of different kinds of experience and yet the fundamental entities recognized by physics, at least on the normal tale, are extremely simple and homogeneous. What explains this extraordinarily rich palette of conscious experience? Physics exhaustively describes the structural-relational properties of the world. What physics doesn’t do is deal in the essence of the physical, its intrinsic nature.

Now, it’s an extremely plausible assumption that the world’s fundamental fields are non-experiential, devoid of any subjective properties – and this may well be the case. But if so, we have the hard problem, the binding problem, the problem of causal efficacy, the palette problem – a whole raft of problems.

Lucas Perry: Okay. So, this all serves the purpose of codifying that there’s these questions up in the air about these metaphysical views which inform identity. We got here because we were talking about mereological nihilism, and Andrés said that one view that you guys have is that you can divide or cut or partition consciousness into individual, momentary, unitary moments of experience that you claim are ontologically simple. What is your credence on this view?

Andrés Gómez Emilsson: Phenomenological evidence. When you experience your visual fields, you don’t only experience one point at a time. The contents of your experience are not ones and zeros; it isn’t the case that you experience one and then zero and then one again. Rather, you experience many different types of qualia varieties simultaneously: visuals experience and auditory experience and so on. All of that gets presented to you. I take that very seriously. I mean, some other researchers may fundamentally say that that’s an illusion, that there’s actually never a unified experience, but that has way many more problems than actually thinking seriously that unity of consciousness.

David Pearce: A number of distinct questions arise here. Are each of us egocentric phenomenal world-simulations? A lot of people are implicitly perceptual direct realists, even though they might disavow the label. Implicitly, they assume that they have some kind of direct access to physical properties. They associate experience with some kind of stream of thoughts and feelings behind their forehead. But if instead we are world-simulationists, then there arises the question: what is the actual fundamental nature of the world beyond your phenomenal world-simulation? Is it experiential or non-experiential? I am agnostic about that – even though I explore non-materialist physicalism.

Lucas Perry: So, I guess I’m just trying to get a better answer here on how is it that we navigate these views of identity towards truth?

Andrés Gómez Emilsson: An example I thought of, of a very big contrast between what you may intuitively imagine is going on versus what’s actually happening, is if you are very afraid of snakes, for example, you look at a snake. You feel, “Oh, my gosh, it’s intruding into my world and I should get away from it,” and you have this representation of it as a very big other. Anything that is very threatening, oftentimes you represent it as “an other”.

But crazily, that’s actually just yourself to a large extent because it’s still part of your experience. Within your moment of experience, the whole phenomenal quality of looking at a snake and thinking, “That’s an other,” is entirely contained within you. In that sense, these ways of ascribing identity and continuity to the things around us or a self-other division are almost psychotic. They start out by assuming that you can segment out a piece of your experience and call it something that belongs to somebody else, even though clearly, it’s still just part of your own experience; it’s you.

Lucas Perry: But the background here is also that you’re calling your experience your own experience, which is maybe also a kind of psychopathy. Is that the word you used?

Andrés Gómez Emilsson: Yeah, yeah, yeah, that’s right.

Lucas Perry: Maybe the scientific thing is, there’s just snake experience and it’s neither yours nor not yours, and there’s what we conventionally call a snake.

Andrés Gómez Emilsson: That said, there are ways in which I think you can use experience to gain insight about other experiences. If you’re looking at a picture that has two blue dots, I think you can accurately say, by paying attention to one of those blue dots, the phenomenal property of my sensation of blue is also in that other part of my visual field. And this is a case where in a sense you can I think, meaningfully refer to some aspect of your experience by pointing at an other aspect of your experience. It’s still maybe in some sense kind of crazy, but it’s still closer to truth than many other things that we think of or imagine.

Honest and true statements about the nature of other people’s experiences, I think are very much achievable. Bridging the reference gap, I think it might be possible to overcome and you can probably aim for a true sense of identity, harmonizing the phenomenal and the ontological sense of identity.

Lucas Perry: I mean, I think that part of the motivation, for example in Buddhism, is that you need to always be understanding yourself in reality as it is or else you will suffer, and that it is through understanding how things are that you’ll stop suffering. I like this point that you said about unifying the phenomenal identity and phenomenal self with what is ontologically true, but that also seems not intrinsically necessary because there’s also this other point here where you can maybe function or have the epistemology of any arbitrary identity view but not identify with it. You don’t take it as your ultimate understanding of the nature of the world, or what it means to be this limited pattern in a giant system.

Andrés Gómez Emilsson: I mean, generally speaking, that’s obviously pretty good advice. It does seem to be something that’s constrained to the workings of the human mind as it is currently implemented. I mean, definitely all these Buddhists advises of “don’t identify with it” or “don’t get attached to it.” Ultimately, it cashes out in experiencing less of a craving, for example, or feeling less despair in some cases. Useful advice, not universally applicable.

For many people, their problem might be something like, sure, like desire, craving, attachment, in which case these Buddhist practices will actually be very helpful. But if your problem is something like a melancholic depression, then lack of desire doesn’t actually seem very appealing; that is the default state and it’s not a good one. Just be mindful of universalizing this advice.

David Pearce: Yes. Other things being equal, the happiest people tend to have the most desires. Of course, a tremendous desire can also bring tremendous suffering, but there are a very large number of people in the world who are essentially unmotivated. Nothing really excites them. In some cases, they’re just waiting to die: melancholic depression. Desire can be harnessed.

A big problem, of course, is that in a Darwinian world, many of our desires are mutually inconsistent. And to use (what to me at least would be) a trivial example – it’s not trivial to everyone –  if you have 50 different football teams with all their supporters, there is logically no way that the preferences of these fanatical football supporters can be reconciled. But nonetheless, by raising their hedonic set-points, one can allow all football supporters to enjoy information-sensitive gradients of bliss. But there is simply no way to reconcile their preferences.

Lucas Perry: There’s part of me that does want to do some universalization here, and maybe that is wrong or unskillful to do, but I seem to be able to imagine a future where, say we get aligned superintelligence and there’s some kind of rapid expansion, some kind of optimization bubble of some kind. And maybe there are the worker AIs and then there are the exploiter AIs, and the exploiter AIs just get blissed out.

And imagine if some of the exploiter AIs are egomaniacs in their hedonistic simulations and some of them are hive minds, and they all have different views on open individualism or closed individualism. Some of the views on identity just seem more deluded to me than others. I seem to have a problem with a self identification and reification of self as something. It seems to me, to take something that is conventional and make it an ultimate truth, which is confusing to the agent, and that to me seems bad or wrong, like our world model is wrong. Part of me wants to say it is always better to know the truth, but I also feel like I’m having a hard time being able to say how to navigate views of identity in a true way, and then another part of me feels like actually it doesn’t really matter only in so far as it affects the flavor of that consciousness.

Andrés Gómez Emilsson: If we find like the chemical or genetic levers for different notions of identity, we could presumably imagine a lot of different ecosystems of approaches to identity in the future, some of them perhaps being much more adaptive than others. I do think I grasp a little bit maybe the intuition pump, and I think that’s actually something that resonates quite a bit with us, which is that it is an instrumental value for sure to always be truth-seeking, especially when you’re talking about general intelligence.

It’s very weird and it sounds like it’s going to fail if you say, “Hey, I’m going to be truth-seeking in every domain except on here.” And these might be identity, or value function, or your model of physics or something like that, but perhaps actual superintelligence in some sense it really entails having an open-ended model for everything, including ultimately who you are. If you’re not having those open-ended models that can be revised with further evidence and reasoning, you are not a super intelligence.

That intuition pump may suggest that if intelligence turns out to be extremely adaptive and powerful, then presumably, the superintelligences of the future will have true models of what’s actually going on in the world, not just convenient fictions.

David Pearce: Yes. In some sense I would hope our long-term goal is ignorance of the entire Darwinian era and its horrors. But it would be extremely dangerous if we were to give up prematurely. We need to understand reality and the theoretical upper bounds of rational moral agency in the cosmos. But ultimately, when we have done literally everything that it is possible to do to minimize and prevent suffering, I think in some sense we should want to forget about it altogether. But I would stress the risks of premature defeatism.

Lucas Perry: Of course we’re always going to need a self model, a model of the cognitive architecture in which the self model is embedded, it needs to understand the directly adjacent computations which are integrated into it, but it seems like the views of identity go beyond just this self model. Is that the solution to identity? What does open, closed, or empty individualism have to say about something like that?

Andrés Gómez Emilsson: Open, empty and closed as ontological claims, yeah, I mean they are separable from the functional uses of a self model. It does however, have bearings on basically the decision theoretic rationality of an intelligence, because when it comes to planning ahead, if you have the intense objective of being as happy as you can, and somebody offers you a cloning machine and they say, “Hey, you can trade one year of your life for just a completely new copy of yourself.” Do you press the button to make that happen? For making that decision, you actually do require a model of ontological notion of identity, unless you just care about replication.

Lucas Perry: So I think that the problem there is that identity, at least in us apes, is caught up in ethics. If you could have an agent like that where identity was not factored into ethics, then I think that it would make a better decision.

Andrés Gómez Emilsson: It’s definitely a question too of whether you can bootstrap an impartial god’s-eye-view on the wellbeing of all sentient beings without first having developed a sense of own identity and then wanting to preserve it, and finally updating it with more information, you know, philosophy, reasoning, physics. I do wonder if you can start out without caring about identity, and finally concluding with kind of an impartial god’s-eye-view. I think probably in practice a lot of those transitions do happen because the person is first concerned with themselves, and then they update the model of who they are based on more evidence. You know, I could be wrong, it might be possible to completely sidestep Darwinian identities and just jump straight up into impartial care for all sentient beings, I don’t know.

Lucas Perry: So we’re getting into the ethics of identity here, and why it matters. The question for this portion of the discussion is what are the ethical implications of different views on identity? Andres, I think you can sort of kick this conversation off by talking a little bit about the game theory.

Andrés Gómez Emilsson: Right, well yeah, the game theory is surprisingly complicated. Just consider within a given person, in fact, the different “sub agents” of an individual. Let’s say you’re drinking with your friends on a Friday evening, but you know you have to wake up early at 8:00 AM for whatever reason, and you’re deciding whether to have another drink or not. Your intoxicated self says, “Yes, of course. Tonight is all that matters.” Whereas your cautious self might try to persuade you that no, you will also exist tomorrow in the morning.

Within a given person, there’s all kinds of complex game theory that happens between alternative views of identity. Even implicitly it becomes obviously much more tricky when you expand it outwards, how like some social movements in a sense are trying to hack people’s view of identity, whether the unit is your political party, or the country, or the whole ecosystem, or whatever it may be. A key thing to consider here is the existence of legible Schelling points, also called focal points, which is in the essence of communication between entities, what are some kind of guiding principles that they can use in order to effectively coordinate and move towards a certain goal?

I would say that having something like open individualism itself can be a powerful Schelling point for coordination. Especially because if you can be convinced that somebody is an open individualist, you have reasons to trust them. There’s all of this research on how high-trust social environments are so much more conducive to productivity and long-term sustainability than low-trust environments, and expansive notions of identity are very trust building.

On the other hand, from a game theoretical point of view, you also have the problem of defection. Within an open individualist society, you have a small group of people who can fake the test of open individualism. They can take over from within, and instantiate some kind of a dictatorship or some type of a closed individualist takeover of what was a really good society, good for everybody.

This is a serious problem, even when it comes to, for example, forming groups of people with all of them share a certain experience. For example, MDMA, or 5-MeO-DMT, or let’s say deep stages of meditation. Even then, you’ve got to be careful, because people who are resistant to those states may pretend that they have an expanded notion of identity, but actually covertly work towards a much more reduced sense of identity. I have yet to see a credible game theoretically aware solution to how to make this work.

Lucas Perry: If you could clarify the knobs in a person, whether it be altruism, or selfishness, or other things that the different views on identity turn, and if you could clarify how that affects the game theory, then I think that that would be helpful.

Andrés Gómez Emilsson: I mean, I think the biggest knob is fundamentally what experiences count from the point of view of the fact that you expect to, in a sense, be there or expect them to be real, in as real of a way as your current experience is. It’s also contingent on theories of consciousness, because you could be an open individualist and still believe that higher order cognition is necessary for consciousness, and that non-human animals are not conscious. That gives rise to all sorts of other problems, the person presumably is altruistic and cares about others, but they just still don’t include non-human animals for a completely different reason in that case.

Definitely another knob is how you consider what you will be in the future. Whether you consider that to be part of the universe or the entirety of the universe. I guess I used to think that personal identity was very tied to a hedonic tone. I think of them as much more dissociated now. There is a general pattern: people who are very low mood may have kind of a bias towards empty individualism. People who become open individualists often experience a huge surge in positive feelings for a while because they feel that they’re never going to die, like the fear of death greatly diminishes, but I don’t actually think it’s a surefire or a foolproof way of increasing wellbeing, because if you take seriously open individualism, it also comes with terrible implications. Like that hey, we are also the pigs in factory farms. It’s not a very pleasant view.

Lucas Perry: Yeah, I take that seriously.

Andrés Gómez Emilsson: I used to believe for a while that the best thing we could possibly do in the world was to just write a lot of essays and books about why open individualism is true. Now I think it’s important to combine it with consciousness technologies so that, hey, once we do want to upgrade our sense of identity to a greater circle of compassion, that we also have the enhanced happiness and mental stability to be able to actually engage with that without going crazy.

Lucas Perry: This has me thinking about one point that I think is very motivating for me for the ethical case of veganism. Take the common sense, normal consciousness, like most people have, and that I have, you just feel like a self that’s having an experience. You just feel like you are fortunate enough to be born as you, and to be having the Andrés experience or the Lucas experience, and that your life is from birth to death, or whatever, and when you die you will be annihilated, you will no longer have experience. Then who is it that is experiencing the cow consciousness? Who is it that is experiencing the chicken and the pig consciousness? There’s so many instantiations of that, like billions. Even if this is based off of the irrationality, it still feels motivating to me. Yeah, I could just die and wake up as a cow 10 billion times. That’s kind of the experience that is going on right now. The sudden confused awakening into cow consciousness plus factory farming conditions. I’m not sure if you find that completely irrational or motivating or what.

Andrés Gómez Emilsson: No, I mean I think it makes sense. We have a common friend as well, Magnus Vinding. He wrote a pro-veganism book actually kind of with this line of reasoning. It’s called You Are Them. About how post theoretical science of consciousness and identity itself is a strong case for an ethical lifestyle.

Lucas Perry: Just touching here on the ethical implications, some other points that I just want to add here are that when one is identified with one’s phenomenal identity, in particular, I want to talk about the experience of self, where you feel like you’re a closed individualist, which your life is like when you were born, and then up until when you die, that’s you. I think that that breeds a very strong duality in terms of your relationship with your own personal phenomenal consciousness. The suffering and joy which you have direct access to are categorized as mine or not mine.

Those which are mine take high moral and ethical priority over the suffering of others. You’re not mind-melded with all of the other brains, right? So there’s an epistemological limitation there where you’re not directly experiencing the suffering of other people, but the closed individualist view goes a step further and isn’t just saying that there’s an epistemological limitation, but it’s also saying that this consciousness is mine, and that consciousness is yours, and this is the distinction between self and other. And given selfishness, that self consciousness will take moral priority over other consciousness.

That I think just obviously has massive ethical implications with regards to their greed of people. I view here the ethical implications as being important because, at least in the way that human beings function, if one is able to fully rid themselves of the ultimate identification with your personal consciousness as being the content of self, then I can move beyond the duality of consciousness of self and other, and care about all instances of wellbeing and suffering much more equally than I currently do. That to me seems harder to do, at least with human brains. If we have a strong reification and identification with your instances of suffering or wellbeing as your own.

David Pearce: Part of the problem is that the existence of other subjects of experience is metaphysical speculation. It’s metaphysical speculation that one should take extremely seriously: I’m not a solipsist. I believe that other subjects of experience, human and nonhuman, are as real as my experience. But nonetheless, it is still speculative and theoretical. One cannot feel their experiences. There is simply no way, given the way that we are constituted, the way we are now, that one can behave with impartial, God-like benevolence.

Andrés Gómez Emilsson: I guess I would question it perhaps a little bit that we only care about our future suffering within our own experience, because this is me, this is mine, it’s not an other. In a sense I think we care about those more, largely because they’re are more intense, you do see examples of, for example, mirror touch synesthesia, of people who if they see somebody else get hurt, they also experience pain. I don’t mean a fleeting sense of discomfort, but perhaps even actual strong pain because they’re able to kind of reflect that for whatever reason.

People like that are generally very motivated to help others as well. In a sense, their implicit self model includes others, or at least weighs others more than most people do. I mean in some sense you can perhaps make sense of selfishness in this context as the coincidence that what is within our self model is experienced as more intense. But there’s plenty of counter examples to that, including sense of depersonalization or ego death, where you can experience the feeling of God, for example, as being this eternal and impersonal force that is infinitely more intense than you, and therefore it matters more, even though you don’t experience it as you. Perhaps the core issue is what gets the highest amount of intensity within your world simulation.

Lucas Perry: Okay, so I also just want to touch on a little bit about preferences here before we move on to how this is relevant to AI alignment and the creation of beneficial AI. From the moral realist perspective, if you take the metaphysical existence of consciousness very substantially, and you view it as the ground of morality, then different views on identity will shift how you weight the preferences of other creatures.

So from a moral perspective, whatever kinds of views of identity end up broadening your moral circle of compassion closer and closer to the end goal of impartial benevolence for all sentient beings according to their degree and kinds of worth, I would view as a good thing. But now there’s this other way to think about identity because if you’re listening to this, and you’re a moral anti-realist, there is just the arbitrary, evolutionary, and historical set of preferences that exist across all creatures on the planet.

Then the views on identity I think are also obviously again going to weigh into your moral considerations about how much to just respect different preferences, right. One might want to go beyond hedonic consequentialism here, and could just be a preference consequentialist. You could be a deontological ethicist or a virtue ethicist too. We could also consider about how different views on identity as lived experiences would affect what it means to become virtuous, if being virtuous means moving beyond the self actually.

Andrés Gómez Emilsson: I think I understand what you’re getting at. I mean, really there’s kind of two components to ontology. One is what exists, and then the other one is what is valuable. You can arrive at something like open individualism just from the point of view of what exists, but still have disagreements with other open individualists about what is valuable. Alternatively, you could agree on what is valuable with somebody but completely disagree on what exists. To get the power of cooperation of open individualism as a Schelling point, there also needs to be some level of agreement on what is valuable, not just what exists.

It definitely sounds arrogant, but I do think that by the same principle by which you arrive at open individualism or empty individualism, basically nonstandard views of identities, you can also arrive at hedonistic utilitarianism, and that is, again, like the principle of really caring about knowing who or what you are fundamentally. To know yourself more deeply also entails understanding from second to second how your preferences impact your state of consciousness. It is my view that just as open individualism, you can think of it as the implication of taking a very systematic approach to make sense of identity. Likewise, philosophical hedonism is also an implication of taking a very systematic approach at trying to figure out what is valuable. How do we know that pleasure is good?

David Pearce: Yeah, does the pain-pleasure axis disclose the world’s intrinsic metric of (dis)value? There is something completely coercive about pleasure and pain. One can’t transcend the pleasure/pain axis. Compare the effects of taking heroin, or “enhanced interrogation”. There is no one with an inverted pleasure/pain axis. Supposed counter-examples, like sado-masochists, in fact just validate the primacy of the pleasure/pain axis.

What follows from the primacy of the pleasure/pain axis? Should we be aiming, as classical utilitarians urge, to maximize the positive abundance of subjective value in the universe, or at least our forward light-cone? But if we are classical utilitarians, there is a latently apocalyptic implication of classical utilitarianism – namely, that we ought to be aiming to launch something like a utilitronium (or hedonium) shockwave – where utilitronium or hedonium is matter and energy optimized for pure bliss.

So rather than any kind of notion of personal identity as we currently understand it, if one is a classical utilitarian – or if one is programming a computer or a robot with the utility function of classical utilitarianism –  should one therefore essentially be aiming to launch an apocalyptic utilitronium shockwave? Or alternatively, should one be trying to ensure that the abundance of positive value within our cosmological horizon is suboptimal by classical utilitarian criteria?

I don’t actually personally advocate a utilitronium shockwave. I don’t think it’s sociologically realistic. I think much more sociologically realistic is to aim for a world based on gradients of intelligent bliss -because that way, people’s existing values and preferences can (for the most part) be conserved. But nonetheless, if one is a classical utilitarian, it’s not clear one is allowed this kind of messy compromise.

Lucas Perry: All right, so now that we’re getting into the juicy, hedonistic imperative type stuff, let’s talk about here how about how this is relevant to AI alignment and the creation of beneficial AI. I think that this is clear based off of the conversations we’ve had already about the ethical implications, and just how prevalent identity is in our world for the functioning of society and sociology, and just civilization in general.

Let’s limit the conversation for the moment just to AI alignment. And for this initial discussion of AI alignment, I just want to limit it to the definition of AI alignment as developing the technical process by which AIs can learn human preferences, and help further express and idealize humanity. So exploring how identity is important and meaningful for that process, two points I think that it’s relevant for, who are we making the AI for? Different views on identity I think would matter, because if we assume that sufficiently powerful and integrated AI systems are likely to have consciousness or to have qualia, they’re moral agents in themselves.

So who are we making the AI for? We’re making new patients or subjects of morality if we ground morality on consciousness. So from a purely egoistic point of view, the AI alignment process is just for humans. It’s just to get the AI to serve us. But if we care about all sentient beings impartially, and we just want to maximize conscious bliss in the world, and we don’t have this dualistic distinction of consciousness being self or other, we could make the AI alignment process something that is more purely altruistic. That we recognize that we’re creating something that is fundamentally more morally relevant than we are, given that it may have more profound capacities for experience or not.

David, I’m also holding in my hand, I know that you’re skeptical of the ability of AGI or superintelligence to be conscious. I agree that that’s not solved yet, but I’m just working here with the idea of, okay, maybe if they are. So I think it can change the altruism versus selfishness, the motivations around who we’re training the AIs for. And then the second part is why are we making the AI? Are we making it for ourselves or are we making it for the world?

If we take a view from nowhere, what Andrés called a god’s-eye-view, is this ultimately something that is for humanity or is it something ultimately for just making a better world? Personally, I feel that if the end goal is ultimate loving kindness and impartial ethical commitment to the wellbeing of all sentient creatures in all directions, then ideally the process is something that we’re doing for the world, and that we recognize the intrinsic moral worth of the AGI and superintelligence as ultimately more morally relevant descendants of ours. So I wonder if you guys have any reactions to this?

Andrés Gómez Emilsson: Yeah, yeah, definitely. So many. Tongue in cheek, but you’ve just made me chuckle when you said, “Why are we making the AI to begin with?” I think there’s a case to be made that the actual reason why we’re making AI is a kind of an impressive display of fitness in order to signal our intellectual fortitude and superiority. I mean sociologically speaking, you know, actually getting an AI to do something really well. It’s a way in which you can yourself signal your own intelligence, and I guess I worry to some extent that this is a bit of a tragedy of the commons, as it is the case with our weapon development. You’re so concerned with whether you can, and especially because of the social incentives, that you’re going to gain status and be looked at as somebody who’s really competent and smart, that you don’t really stop and wonder whether you should be building this thing in the first place.

Leaving that aside, just from a purely ethically motivated point of view, I do remember thinking and having a lot of discussions many years ago about if we can make a super computer experience what it is like for a human to be on MDMA. Then all of a sudden that supercomputer becomes a moral patient. It actually matters, you probably shouldn’t turn it off. Maybe in fact you should make more of them. A very important thing I’d like to say here is: I think it’s really important to distinguish the notion of intelligence.

On the one hand, as causal power over your environment, and on the other hand as the capacity for self insight, and introspection, and understanding reality. I would say that we tend to confuse these quite a bit. I mean especially in circles that don’t take consciousness very seriously. It’s usually implicitly assumed that having a superhuman ability to control your environment entails that you also have, in a sense, kind of a superhuman sense of self or a superhuman broad sense of intelligence. Whereas even if you are a functionalist, I mean even if you believe that a digital computer can be conscious, you can make a pretty strong case that even then, it is not something automatic. It’s not just that if you program the appropriate behavior, it will automatically also be conscious.

A super straight forward example here is that if you have the Chinese room, if it’s just a giant lookup table, clearly it is not a subject of experience, even though the input / output mapping might be very persuasive. There’s definitely still the problems there, and I think if we aim instead towards maximizing intelligence in the broad sense, that does entail also the ability to actually understand the nature and scope of other states of consciousness. And in that sense, I think a superintelligence of that sort would it be intrinsically aligned with the intrinsic values of consciousness. But there are just so many ways of making partial superintelligences that maybe are superintelligent in many ways, but not in that one in particular, and I worry about that.

David Pearce: I sometimes sketch this simplistic trichotomy, three conceptions of superintelligence. One is a kind of “Intelligence Explosion” of recursively self-improving software-based AI. Then there is the Kurzweilian scenario – a complete fusion of humans and our machines. And then there is, very crudely, biological superintelligence, not just rewriting our genetic source code, but also (and Neuralink prefigures this) essentially “narrow” superintelligence-on-a-chip so that anything that anything a classical digital computer can do a biological human or a transhuman can do.

So yes, I see full-spectrum superintelligence as our biological descendants, super-sentient, able to navigate radically alien states of consciousness. So I think the question that you’re asking is why are we developing “narrow” AI – non-biological machine superintelligence.

Lucas Perry: Speaking specifically from the AI alignment perspective, how you align current day systems and future systems to superintelligence and beyond with human values and preferences, and so the question born of that, in the context of these questions of identity, is who are we making that AI for and why are we making the AI?

David Pearce: If you’ve got Buddha, “I teach one thing and one thing only, suffering and the end of suffering”… Buddha would press the OFF button, and I would press the OFF button.

Lucas Perry: What’s the off button?

David Pearce: Sorry, the notional initiation of a vacuum phase-transition (or something similar) that (instantaneously) obliterates Darwinian life. But when people talk about “AI alignment”, or most people working in the field at any rate, they are not talking about a Buddhist ethic [the end of suffering] – they have something else in mind. In practical terms, this is not a fruitful line of thought to pursue – you know, the implications of Buddhist, Benatarian, negative utilitarian, suffering-focused ethics.

Essentially that one wants to ratchet up hedonic range and hedonic set-points in such a way that you’re conserving people’s existing preferences – even though their existing preferences and values are, in many cases, in conflict with each other. Now, how one actually implements this in a classical digital computer, or a classically parallel connectionist system, or some kind of hybrid, I don’t know precisely.

Andrés Gómez Emilsson: At least there is one pretty famous cognitive scientist and AI theorist does propose the Buddhist ethic of turning the off button of the universe. Thomas Metzinger, and his benevolent, artificial anti-natalism. I mean, yeah. Actually that’s pretty interesting because he explores the idea of an AI that truly kind of extrapolates human values and what’s good for us as subjects of experience. The AI concludes what we are psychologically unable to, which is that the ethical choice is non-existence.

But yeah, I mean, I think that’s, as David pointed out, implausible. I think it’s much better to put our efforts in creating a super cooperator cluster that tries to recalibrate the hedonic set point so that we are animated by gradients of bliss. Sociological constraints are really, really important here. Otherwise you risk…

Lucas Perry: Being irrelevant.

Andrés Gómez Emilsson: … being irrelevant, yeah, is one thing. The other thing is unleashing an ineffective or failed attempt at sterilizing the world, which would be so much, much worse.

Lucas Perry: I don’t agree with this view, David. Generally, I think that Darwinian history has probably been net negative, but I’m extremely optimistic about how good the future can be. And so I think it’s an open question at the end of time, how much misery and suffering and positive experience there was. So I guess I would say I’m agnostic as to this question. But if we get AI alignment right, and these other things, then I think that it can be extremely good. And I just want to tether this back to identity and AI alignment.

Andrés Gómez Emilsson: I do have the strong intuition that if empty individualism is correct at an ontological level, then actually negative utilitarianism can be pretty strongly defended on the grounds that when you have a moment of intense suffering, that’s the entirety of that entity’s existence. And especially with eternalism, once it happens, there’s nothing you can do to prevent it.

There’s something that seems particularly awful about allowing inherently negative experiences to just exist. That said, I think open individualism actually may to some extent weaken that. Because even if the suffering was very intense, you can still imagine that if you identify with consciousness as a whole, you may be willing to undergo some bad suffering as a trade-off for something much, much better in the future.

It sounds completely insane if you’re currently experiencing a cluster headache or something astronomically painful. But maybe from the point of view of eternity, it actually makes sense. Those are still tiny specs of experience relative to the beings that are going to exist in the future. You can imagine Jupiter brains and Dyson spheres just in a constant ecstatic state. I think open individualism might counterbalance some of the negative utilitarian worries and would be something that an AI would have to contemplate and might push it one way or the other.

Lucas Perry: Let’s go ahead and expand the definition of AI alignment. A broader way we can look at the AI alignment problem, or the problem of generating beneficial AI, and making future AI stuff go well, where that is understood is the project of making sure that the technical, political, social, and moral consequences of short-term to super intelligence and beyond, is that as we go through all of that, that is a beneficial process.

Thinking about identity in that process, we were talking about how strong nationalism or strong identity or identification with regards to a nation state is a form of identity construction that people do. The nation or the country becomes part of self. One of the problems of the AI alignment problem is arms racing between countries, and so taking shortcuts on safety. I’m not trying to propose clear answers or solutions here. It’s unclear how successful an intervention here could even be. But these views on identity and how much nationalism shifts or not, I think feed into how difficult or not the problem will be.

Andrés Gómez Emilsson: The point of game theory becomes very, very important in that yes, you do want to help other people who are also trying to improve the well-being of all consciousness. On the other hand, if there’s a way to fake caring about the entirety of consciousness, that is a problem because then you would be using resources on people who would hoard them or even worse wrestle the power away from you so that they can focus on their narrow sense of identity.

In that sense, I think having technologies in order to set particular phenomenal experiences of identity, as well as to be able to detect them, might be super important. But above all, and I mean this is definitely my area of research, having a way of objectively quantifying how good or bad a state of consciousness is based on the activity of a nervous system seems to me like an extraordinarily key component for any kind of a serious AI alignment.

If you’re actually trying to prevent bad scenarios in the future, you’ve got to have a principle way of knowing whether the outcome is bad, or at the very least knowing whether the outcome is terrible. The aligned AI should be able to grasp that a certain state of consciousness, even if nobody has experienced it before, will be really bad and it should be avoided, and that tends to be the lens through which I see this.

In terms of improving people’s internal self-consistency, as David pointed out, I think it’s kind of pointless to try to satisfy a lot of people’s preferences, such as having their favorite sports team win, because there’s really just no way of satisfying everybody’s preferences. In the realm of psychology is where a lot of these interventions would happen. You can’t expect an AI to be aligned with you, if you yourself are not aligned with yourself, right, if you have all of these strange, psychotic, competing sub-agents. It seems like part of the process is going to be developing techniques to become more consistent, so that we can actually be helped.

David Pearce: In terms of risks this century, nationalism has been responsible for most of the wars of the past two centuries, and nationalism is highly likely to lead to catastrophic war this century. And the underlying source of global catastrophic risk? I don’t think it’s AI. It’s male human primates doing what male human primates have been “designed” by evolution to do – to fight, to compete, to wage war. And even vegan pacifists like me, how do we spend their leisure time? Playing violent video games.

There are technical ways one can envisage mitigating the risk. Perhaps it’s unduly optimistic aiming for all-female governance or for a democratically-accountable world state under the auspices of the United Nations. But I think unless one does have somebody with a monopoly on the use of force that we are going to have cataclysmic nuclear war this century. It’s highly likely: I think we’re sleepwalking our way towards disaster. It’s more intellectually exciting discussing exotic risks from AI that goes FOOM, or something like that. But there are much more mundane catastrophes that are, I suspect, going to unfold this century.

Lucas Perry: All right, so getting into the other part here about AI alignment and beneficial AI throughout this next century, there’s a lot of different things that increased intelligence and capacity and power over the world is going to enable. There’s going to be human biological species divergence via AI-enabled bioengineering. There is this fundamental desire for immortality with many people, and the drive towards super intelligence and beyond for some people promises immortality. I think that in terms of closed individualism here, closed individualism is extremely motivating for this extreme self-concern of desire for immortality.

There are people currently today who are investing in say, cryonics, because they want to freeze themselves and make it long enough so that they can somehow become immortal, very clearly influenced by their ideas of identity. As Yuval Noah Harari was saying on our last podcast, it subverts many of the classic liberal myths that we have about the same intrinsic worth across all people; and then if you add humans 2.0 or 3.0 or 4.0 into the mixture, it’s going to subvert that even more. So there are important questions of identity there, I think.

With sufficiently advanced super intelligence people flirt with the idea of being uploaded. The identity questions here which are relevant are if we scan the information architecture or the neural architecture of your brain and upload it, will people feel like that is them? Is it not them? What does it mean to be you? Also, of course, in scenarios where people want to merge with the AI, what is it that you would want to be kept in the merging process? What is superfluous to you? What is not nonessential to your identity or what it means to be you, that you would be okay or not with merging?

And then I think that most importantly here, I’m very interested in the Descendants scenario, where we just view AI as like our evolutionary descendants. There’s this tendency in humanity to not be okay with this descendant scenario. Because of closed individualist views on identity, they won’t see that consciousness is the same kind of thing, or they won’t see it as their own consciousness. They see that well-being through the lens of self and other, so that makes people less interested in they’re being descendant, super-intelligent conscious AIs. Maybe there’s also a bit of speciesism in there.

I wonder if you guys want to have any reactions to identity in any of these processes? Again, they are human, biological species divergence via AI-enabled bioengineering, immortality, uploads, merging, or the Descendants scenario.

David Pearce: In spite of thinking that Darwinian life is sentient malware, I think cryonics should be opt-out, and cryothanasia should be opt-in, as a way to defang death. So long as someone is suspended in optimal conditions, it ought to be possible for advanced intelligence to reanimate that person. And sure, if one is an “empty” individualist, or if you’re the kind of person who wakes up in the morning troubled that you’re not the person who went to sleep last night, this reanimated person may not really be “you”. But if you’re more normal, yes, I think it should be possible to reanimate “you” if you are suspended.

In terms of mind uploads, this is back to the binding problem. Even assuming that you can be scanned with a moderate degree of fidelity, I don’t think your notional digital counterpart is a subject to experience. Even if I am completely wrong here and that somehow subjects or experience inexplicably emerge in classical digital computers, there’s no guarantee that the qualia would be the same. After all, you can replay a game of chess with perfect fidelity, but there’s no guarantee incidentals like the textures or the pieces will be the same. Why expect the textures of qualia to be the same, but that isn’t really my objection. It’s the fact that a digital computer cannot support phenomenally-bound subjects of experience.

Andrés Gómez Emilsson: I also think cryonics is really good. Even though with a different nonstandard view of personal identity, it’s kind of puzzling. Why would you care about it? Lots of practical considerations. I like what David said of like defanging death. I think that’s a good idea, but also giving people skin in the game for the future.

People who enact policy and become politically successful, often tend to be 50 years plus, and there’s a lot of things that they weigh on, that they will not actually get to experience, that probably biases politicians and people who are enacting policy to focus, especially just on short-term gains as opposed to really genuinely trying to improve the long-term; and I think cryonics would be helpful in giving people skin in the game.

More broadly speaking, it does seem to be the case that what aspect of transhumanism a person is likely to focus on depends a lot on their theories of identity. I mean, if we break down transhumanism into the three supers of super happiness, super longevity, and super intelligence, the longevity branch is pretty large. There’s a lot of people looking for ways of rejuvenating, preventing aging, and reviving ourselves, or even uploading ourselves.

Then there’s people who are very interested in super intelligence. I think that’s probably the most popular type of transhumanism nowadays. That one I think does rely to some extent on people having a functionalist information theoretic account of their own identity. There’s all of these tropes of, “Hey, if you leave a large enough digital footprint online, a super intelligence will be able to reverse engineer your brain just from that, and maybe reanimate you in the future,” or something of that nature.

And then there’s, yeah, people like David and I, and the Qualia Research Institute as well, that care primarily about super happiness. We think of it as kind of a requirement for a future that is actually worth living. You can have all the longevity and all the intelligence you want, but if you’re not happy, I don’t really see the point. A lot of the concerns with longevity, fear of death and so on, in retrospect, I think will be probably considered some kind of a neurosis. Obviously a genetically adaptive neurosis, but something that can be cured with mood-enhancing technologies.

Lucas Perry: Leveraging human selfishness or leveraging how most people are closed individualists seems like the way to having good AI alignment. To one extent, I find the immortality pursuits through cryonics to be pretty elitist. But I think it’s a really good point that giving the policymakers and the older generation and people in power more skin in the game over the future is both potentially very good and also very scary.

It’s very scary to the extent to which they could get absolute power, but also very good if you’re able to mitigate risks of them developing absolute power. But again, as you said, it motivates them towards more deeply and profoundly considering future considerations, being less myopic, being less selfish. So that getting the AI alignment process right and doing the necessary technical work, it’s not done for a short-term nationalistic gain. Again, with an asterisk here that the risk is unilaterally getting more and more power.

Andrés Gómez Emilsson: Yeah, yeah, yeah. Also, without cryonics, another way to increase skin in the game, may be more straight-forwardly positive. Bliss technologies do that. A lot of people who are depressed or nihilistic or vengeful or misanthropic, they don’t really care about destroying the world or watching it burn, so to speak, because they don’t have anything to lose. But if you have a really reliable MDMA-like technological device that reliably produces wonderful states of consciousness, I think people will be much more careful about preserving their own health, and also not watch the world burn, because they know “I could be back home and actually experiencing this rather than just trying to satisfy my misanthropic desires.”

David Pearce: Yeah, the happiest people I know work in the field of existential risk. Rather than great happiness just making people reckless, it can also make them more inclined to conserve and protect.

Lucas Perry: Awesome. I guess just one more thing that I wanted to hit on these different ways that technology is going to change society is… I don’t know. In my heart, the ideal is the vow to liberate all sentient beings in all directions from suffering. The closed individualist view seems generally fairly antithetical to that, but there’s also this desire for me to be realistic about leveraging that human selfishness towards that ethic. The capacity here for conversations on identity going forward, if we can at least give people more information to subvert or challenge or give them information about why the common sense closed individualist view might be wrong, I think it would just have a ton of implications for how people end up viewing human species divergence, or immortality, or uploads, or merging, or the Descendants scenario.

In Max’s book, Life 3.0, he describes a bunch of different scenarios for how you want the world to be as the impact of AI grows, if we’re lucky enough to reach superintelligent AI. These scenarios that he gives are, for example, an Egalitarian Utopia where humans, cyborgs and uploads coexist peacefully thanks to property abolition and guaranteed income. There’s a Libertarian Utopia where human, cyborgs, and uploads, and superintelligences coexist peacefully thanks to property rights. There is a Protector God scenario where essentially omniscient and omnipotent AI maximizes human happiness by intervening only in ways that preserve our feeling of control of our own destiny, and hides well enough that many humans even doubt the AI’s existence. There’s Enslaved God, which is kind of self-evident. The AI is a slave to our will. The Descendants Scenario, which I described earlier, where AIs replace human beings, but give us a graceful exit, making us view them as our worthy descendants, much as parents feel happy and proud to have a child who’s smarter than them, who learns from them, and then accomplishes what they could only dream of, even if they can’t live to see it.

After the book was released, Max did a survey of which ideal societies people were most excited about. And basically most people wanted either the Egalitarian Utopia or the Libertarian Utopia. These are very human centric of course, because I think most people are closed individualists, so okay, they’re going to pick that. And then they wanted to Protector God next, and then the fourth most popular was an Enslaved God. The fifth most popular was Descendants.

I’m a very big fan of the Descendants scenario. Maybe it’s because of my empty individualism. I just feel here that as views on identity are quite uninformed for most people or most people don’t take it, or closed individualism just seems intuitively true from the beginning because it seems like it’s been selected for mostly by Darwinian evolution to have a very strong sense of self. I just think that challenging conventional views on identity will very much shift the kinds of worlds that people are okay with or the kinds of worlds that people want.

If we had a big, massive public education campaign about the philosophy of identity and then take the same survey later, I think that the numbers would be much more different. That seems to be a necessary part of the education of humanity in the process of beneficial AI and AI alignment. To me, the Descendant scenario just seems best because it’s more clearly in line with this ethic of being impartially devoted to maximizing the well-being of sentience everywhere.

I’m curious to know your guys’ reaction to these different scenarios about how you feel views on identity as they shift will inform the kinds of worlds that humanity finds beautiful or meaningful or worthy of pursuit through and with AI.

David Pearce: If today’s hedonic range is -10 to zero to +10, yes, whether building a civilization with a hedonic range of +70 to +100, i.e. with more hedonic contrast, or +90 to a +100 with less hedonic contrast, the multiple phase-changes in consciousness involved are completely inconceivable to humans. But in terms of full-spectrum superintelligence, what we don’t know is the nature of their radically alien-state-spaces of consciousness – far more different than, let’s say, dreaming consciousness and waking consciousness – that I suspect that intelligence going to explore. And we just do not have the language, the concepts, to conceptualize what these alien state-spaces of consciousness are like. I suspect billions of years of consciousness-exploration lie ahead. I assume that a central element will be the pleasure-axis – that these states will be generically wonderful – but they will otherwise be completely alien. And so talk of “identity” with primitive Darwinian malware like us is quite fanciful.

Andrés Gómez Emilsson: Consider the following thought experiment where you have a chimpanzee right next to a person, who is right next to another person, where the third one is currently on a high dose of DMT, combined with ketamine and salvia. If you consider those three entities, I think very likely, actually the experience of the chimpanzee and the experience of the sober person are very much alike, compared to the person who is on DMT, ketamine, and salvia, who is in a completely different alien-state space of consciousness. And in some sense, biologically you’re unrelatable from the point of view of qualia and the sense of self, and time, and space, and all of those things.

Personally, I think having intimations with alien-state spaces of consciousness is actually good also quite apart from changes in a feeling that you’ve become one with the universe. Merely having experience with really different states of consciousness makes it easier for you to identify with consciousness as a whole: you realize, okay, my DMT self, so to speak, cannot exist naturally, and it’s just so much different to who I am normally, and even more different than perhaps being a chimpanzee, that you could imagine caring as well about alien-state spaces of consciousness that are completely nonhuman, and I think that it can be pretty helpful.

The other reason why I give a lot of credence to open individualism being a winning strategy, even just from a purely political and sociological point of view, is that open individualists are not afraid of changing their own state of consciousness, because they realize that it will be them either way. Whereas closed individualists can actually be pretty scared of, for example, taking DMT or something like that. They tend to have at least the suspicion that, oh my gosh, is the person who is going to be on DMT me? Am I going to be there? Or maybe I’m just being possessed by a different entity with completely different values and consciousness.

With open individualism, no matter what type of consciousness your brain generates, it’s going to be you. It massively amplifies the degrees of freedom for coordination. Plus, you’re not afraid of tuning your consciousness for particular new computational uses. Again, this could be extremely powerful as a cooperation and coordination tool.

To summarize, I think a plausible and very nice future scenario is going to be the mixture of open individualism, on the one hand; second, generically enhanced hedonic tone, so that everything is amazing; and third, expanded range of possible experiences. That we will have the tools to experience pretty much arbitrary state spaces of consciousness and consider them our own.

The Descendant scenario, I think it’s much easier to imagine thinking of the new entities as your offspring if you can at least know what they feel like. You can take a drug or something and know, “okay, this is what it’s like to be a post-human android. I like it. This is wonderful. It’s better than being a human.” That would make it possible.

Lucas Perry: Wonderful. This last question is just the role of identity in the AI itself, or the superintelligence itself, as it experiences the world, the ethical implications of those identity models, et cetera. There is the question of identity now, and if we get aligned superintelligence and post-human superintelligence, and we have Jupiter rings or Dyson spheres or whatever, that there’s the question of identity evolving in that system. We are very much creating Life 3.0, and there is a substantive question of what kind of identity views it will take, what it’s phenomenal experience of self or not will have. This all is relevant and important because if we’re concerned with maximizing conscious well-being, then these are flavors of consciousness which would require a sufficiently, rigorous science of consciousness to understand their valence properties.

Andrés Gómez Emilsson: I mean, I think it’s a really, really good thing to think about. The overall frame I tend to utilize, to analyze this kind of questions is, I wrote an article and you can find it in Qualia Computing that is called “Consciousness Versus Replicators.” I think that is a pretty good overarching ethical framework where I basically, I describe how different kinds of ethics can give different worldviews, but also they depend on your philosophical sophistication.

At the very beginning, you have ethics such as the battle between good and evil, but then you start introspecting. You’re like, “okay, what is evil exactly,” and you realize that nobody sets out to do evil from the very beginning. Usually, they actually have motivations that make sense within their own experience. Then you shift towards this other theory that’s called the balance between good and evil, super common in Eastern religions. Also, people who take a lot of psychedelics or meditate a lot tend to arrive to that view, as in like, “oh, don’t be too concerned about suffering or the universe. It’s all a huge yin and yang. The evil part makes the good part better,” or like weird things like that.

Then you have a little bit more developed, what I call it gradients of wisdom. I would say like Sam Harris, and definitely a lot of people in our community think that way, which is they come to the realization that there are societies that don’t help human flourishing, and there are ideologies that do, and it’s really important to be discerning. We can’t just say, “Hey, everything is equally good.”

But finally, I would say the fourth level would be consciousness versus replicators, which involves, one, taking open individualism seriously; and second, realizing that anything that matters, it matters because it influences experiences. Can you have that as your underlying ethical principle? There’s this danger of replicators hijacking our motivational architecture in order to pursue their own replication, independent of the well-being of sentience, and you guard for that. I think you’re in a pretty good space to actually do a lot of good. I would say perhaps that is the sort of ethics or morality we should think about how to instantiate in an artificial intelligence.

In the extreme, you have what I call a pure replicator, and a pure replicator essentially is a system or an entity that uses all of its resources exclusively to make copies of itself, independently of whether that causes good or bad experiences elsewhere. It just doesn’t care. I would argue that humans are not pure replicators. That in fact, we do care about consciousness, at the very least our own consciousness. And evolution is recruiting the fact that we care about consciousness in order to, as a side effect, increase our inclusiveness our genes.

But these discussions we’re having right now, this is the possibility of a post-human ethic is the genie is getting out of the bottle in the sense of consciousness is kind of taking its own values and trying to transcend the selfish genetic process that gave rise to it.

Lucas Perry: Ooh, I like that. That’s good. Anything to add, David?

David Pearce: No. Simply, I hope we have a Buddhist AI.

Lucas Perry: I agree. All right, so I’ve really enjoyed this conversation. I feel more confused now than when I came in, which is very good. Yeah, thank you both so much for coming on.

End of recorded material

AI Alignment Podcast: On DeepMind, AI Safety, and Recursive Reward Modeling with Jan Leike

Jan Leike is a senior research scientist who leads the agent alignment team at DeepMind. His is one of three teams within their technical AGI group; each team focuses on different aspects of ensuring advanced AI systems are aligned and beneficial. Jan’s journey in the field of AI has taken him from a PhD on a theoretical reinforcement learning agent called AIXI to empirical AI safety research focused on recursive reward modeling. This conversation explores his movement from theoretical to empirical AI safety research — why empirical safety research is important and how this has lead him to his work on recursive reward modeling. We also discuss research directions he’s optimistic will lead to safely scalable systems, more facets of his own thinking, and other work being done at DeepMind.

 Topics discussed in this episode include:

  • Theoretical and empirical AI safety research
  • Jan’s and DeepMind’s approaches to AI safety
  • Jan’s work and thoughts on recursive reward modeling
  • AI safety benchmarking at DeepMind
  • The potential modularity of AGI
  • Comments on the cultural and intellectual differences between the AI safety and mainstream AI communities
  • Joining the DeepMind safety team

Timestamps: 

0:00 intro

2:15 Jan’s intellectual journey in computer science to AI safety

7:35 Transitioning from theoretical to empirical research

11:25 Jan’s and DeepMind’s approach to AI safety

17:23 Recursive reward modeling

29:26 Experimenting with recursive reward modeling

32:42 How recursive reward modeling serves AI safety

34:55 Pessimism about recursive reward modeling

38:35 How this research direction fits in the safety landscape

42:10 Can deep reinforcement learning get us to AGI?

42:50 How modular will AGI be?

44:25 Efforts at DeepMind for AI safety benchmarking

49:30 Differences between the AI safety and mainstream AI communities

55:15 Most exciting piece of empirical safety work in the next 5 years

56:35 Joining the DeepMind safety team

 

Works referenced:

Scalable agent alignment via reward modeling

The Boat Race Problem

Move 37

Jan Leike on reward hacking

OpenAI Safety Gym

ImageNet

Unrestricted Adversarial Examples

 

We hope that you will continue to join in the conversations by following us or subscribing to our podcasts on Youtube, Spotify, SoundCloud, iTunes, Google Play, StitcheriHeartRadio, or your preferred podcast site/application. You can find all the AI Alignment Podcasts here.

You can listen to the podcast above or read the transcript below. 

Lucas Perry: Hello everyone and welcome to the AI Alignment Podcast. I’m Lucas Perry. Today, we’re speaking with Jan Leike. Jan Leike is a senior research scientist at DeepMind and his research aims at helping to make machine learning robust and beneficial; he works on safety and alignment of reinforcement learning agents. His current research can be understood as motivated by the following question: How can we design competitive and scalable machine learning algorithms that make sequential decisions in the absence of a reward function? If this podcast is interesting or valuable to you, please consider following us on your preferred listening platform and leaving us a good review.

This conversation covers Jan’s PhD and movement from theoretical to empirical AI research, why this shift took place and his view on the importance of empirical AI safety research, we discuss how DeepMind approaches the projects of beneficial AI and AI safety. We discuss the AI alignment landscape today and the kinds of approaches that Jan is most excited about. We get into Jan’s main area of research of recursive reward modeling, and we talk about AI safety benchmarking efforts at DeepMind and the intellectual and cultural differences between the AI alignment/AI safety community and the mainstream AI and machine learning community. As a friendly heads up, there were some audio issues with the incoming audio in the second half of the podcast. We did our best to clean these up and I feel the resulting audio to be easily listenable. I’d also like to give many thanks to Richard Ngo, Vishal Maini, and Richard Mallah for help on developing and refining the questions for this podcast. And with that, let’s get into our conversation with Jan Leike.

Why don’t you start off by taking us through your journey in the field of AI. How did you first become interested in math and computer science? Tell me a little bit about your time as a PhD student. What perked your curiosity where, why you were pursuing what you were pursuing?

Jan Leike: I got interested in AGI and AGI safety around 2012. I was doing a Master’s degree at a time, and I was trying to think about what I should do with my career. And I was reading a whole bunch of stuff online. That’s how I got into this whole area. My background was kind of in math and computer science at the time, but I wasn’t really working on AI. I was more working on software verification. Then I came across Marcus Hutter’s AIXI model, which is basically a formal mathematical model for what AGI could look like. And it’s highly idealized. It’s not something that could actually run, but you can kind of think about it and you can actually prove things about it. And I was really excited about that. I thought that was a great starting point because you remember that was back in 2012 before the whole deep learning revolution happened, so it was not really clear what kind of approaches might we actually take towards AGI. The purpose of my PhD was to kind of understand AGI from a high-level theoretical perspective.

Lucas Perry: The PhD was with Marcus Hutter on AIXI or “A,” “I,” “X,” “I.” From that pursuit, what interesting or really valuable things did you glean from that process?

Jan Leike: So my thesis ended up being just a number of theoretical results, some of which are that actually this idealized agent AIXI is not optimal in any objective sense. In a way, it all depends on the universal Turing machine that is used to define it. But however, there’s variants on AIXI that have objective properties, such as asymptotic convergence to the optimal policy. This variant is basically a variant based on Thompson sampling, but this is a fully general reinforcement learning setting. So that’s partially observable, and you don’t have episodes. It’s like everything is one long episode. So it’s not really a setting where you can give any sample complexity bounds. Asymptotic convergence is all you can do. And then another thing that came out of that was what we called, “A Formal Solution to the Grain of Truth Problem.” This is a collaboration with the Machine Intelligence Research Institute.

And the idea here is that one of the problems with the AIXI formal model is that it assumes that its environment is computable, but itself is incomputable. You can’t really do multi-agent analysis with it. And so what we did was propose a formalism that is like a variant on AIXI that can be in its own environment class if we embed an agent or environment together with other of these AIXI like agents. And then while they do that, they can still asymptotically learn to predict correctly what the agents will do and then converge to Nash equilibrium asymptotically.

Lucas Perry: So the sense in which AIXI was a theoretical ideal was that the process by which it was able to learn or infer preferences was computationally intractable.

Jan Leike: The AIXI model basically just tries to answer the reinforcement learning question. So you’re given an environment and you’re given a reward signal, how do you optimize that? In a way, you’re using what we call the Solomonoff prior to predict the next observations that come from the environment, and then you essentially do an exhaustive tree search over all the possible action sequences that you could take and the possible consequences that you would predict and then make the best action in terms of returns. This is kind of similar to how AlphaGo uses Monte Carlo tree search to select the best actions. The reason why you can’t literally build AIXI is that this Solomonoff prior that I mentioned, it is basically the set of all possible Turing machines which is countably infinite, and then you have to run all of them in parallel and take a weighted average over what they would predict.

If you’ve ever tried to run all of computer programs in parallel, you’ll know that this is not going to go too well. I find AIXI helpful when thinking about AGI in terms of what could advanced machine learning or reinforcement learning agents look like. So they have some kind of learned model in which they do planning and select actions that are good in the long run. So I think in a way it tells us that if you believe that the reinforcement learning problem is the right problem to phrase an AGI in, then AIXI proves that there can be a solution to that problem. I think on a high level, having thought about this model is useful when thinking about where are we headed and what are potential milestones on the way to AGI. But at the same time, I think my mistake at the time was really getting lost in some of the technical details that really matter if you want to publish a paper on this stuff, but don’t transfer as much in the analogy.

Lucas Perry: After you finished up your PhD with Hutter, you finished working on AIXI now. Is this when you transitioned to DeepMind and make this transition from theoretical to empirical research?

Jan Leike: Yeah, that’s basically right. At the time when I started my PhD, I decided to work on theory because it wasn’t really clear to me what AGI would look like and how we’d build it. So I wanted to do something general and something more timeless. Then we saw a whole bunch of evidence that deep reinforcement learning is viable and you can make it work. Came out with a DQN nature paper, and there was the AlphaGo event. And then it became pretty clear to me that deep reinforcement learning was going somewhere, and that’s something we should work on. At the time, my tool set was very theoretical and my comparative advantage was thinking about theory and using the tools that I learned and developed in my PhD.

And the problem was that deep reinforcement learning has very little theory behind it, and there’s very little theory on RL. And the little theory on RL that we have says that basically function approximation shouldn’t really work. So that means it’s really hard to actually gain traction on doing something theoretically. And at the same time, at the time we were very clear that we could just take some agents that existed and we can just build something, and then we could make incremental progress on things that would actually help us make AGI safer.

Lucas Perry: Can you clarify how the theoretical foundations of deep reinforcement learning are weak? What does that mean? Does that mean that we have this thing and it works, but we’re not super sure about how it works? Or the theories about the mechanisms which constitute that functioning thing are weak? We can’t extrapolate out very far with them?

Jan Leike: Yeah, so basically there’s the two parts. So if you take deep neural networks, there are some results that tell you that depth is better than width. And if you increase capacity, you can represent any function and things like that. But basically the kind of thing that I would want to use is real sample complexity bounds that tell you if your network has X many parameters, how much training data you do need, how many batches do you need to train in order to actually converge? Can you converge asymptotically? None of these things are even true in theory. You can get examples where it doesn’t work. And of course we know that in practice because sometimes training is just unstable, but it doesn’t mean that you can’t tune it and make it work in practice.

On the RL side, there is a bunch of convergence results that people have given in tabular MDPs, Markov decision processes. In that setting, everything is really nice and you can give sample complexity bounds, or let’s say some bounds on how long learning will take. But as soon as you kind of go into a function approximation setting, all bets are off and there’s very simple two-state MDPs they can draw where just simple linear function approximation completely breaks. And this is a problem that we haven’t really gotten a great handle on theoretically. And so going from linear function approximation to deep neural networks is just going to make everything so much harder.

Lucas Perry: Are there any other significant ways in which your thinking has changed as you transitioned from theoretical to empirical?

Jan Leike: In the absence of these theoretical tools, you have two options. Either you try to develop these tools, and that seems very hard and many smart people have tried for a long time. Or you just move on to different tools. And I think especially if you have systems that you can do experiments on, then having an empirical approach makes a lot of sense if you think that these systems actually can teach us something useful about the kind of systems that we are going to build in the future.

Lucas Perry: A lot of your thinking has been about understanding the theoretical foundations, like what AGI might even look like, and then transitioning to an empirical based approach that you see as efficacious for studying systems in the real world and bringing about safe AGI systems. So now that you’re in DeepMind and you’re in this intellectual journey that we’re covering, how is DeepMind and how is Jan approaching beneficial AI and AI alignment in this context?

Jan Leike: DeepMind is a big place, and there is a lot of different safety efforts across the organization. People are working on say robustness to adversarial inputs, fairness, verification of neural networks, interpretability and so on and so on. What I’m doing is I’m focusing on reward modeling as an approach to alignment.

Lucas Perry: So just taking a step back and still trying to get a bigger picture of DeepMind’s approach to beneficial AI and AI alignment. It’s being attacked at many different angles. So could you clarify this what seems to be like a portfolio approach? The AI alignment slash AI safety agendas that I’ve seen enumerate several different characteristics or areas of alignment and safety research that we need to get a grapple on, and it seems like DeepMind is trying its best to hit on all of them.

Jan Leike: DeepMind’s approach to safety is quite like a portfolio. We don’t really know what will end up panning out. So we pursue a bunch of different approaches in parallel. So I’m on the technical AGI safety team that roughly consists of three subteams. There’s a team around incentive theory that tries to model on a high level what incentives agents could have in different situations and how we could understand them. Then there is an agent analysis team that is trying to take some of our state of the art agents and figure out what they are doing and how they’re making the decisions they make. And this can be both from a behavioral perspective and from actually looking inside the neural networks. And then finally there is the agent alignment team, which I’m leading, and that’s trying to figure out how to scale reward modeling. There’s also an ethics research team, and then there’s a policy team.

Lucas Perry: This is a good place then to pivot into how you see the AI alignment landscape today. You guys have this portfolio approach that you just explained. Given that and given all of these various efforts for attacking the problem of beneficial AI from different directions, how do you see the AI alignment landscape today? Is there any more insight that you can provide into that portfolio approach given that it is contextualized within many other different organizations who are also coming at the problem from specific angles? So like MIRI is working on agent foundations and does theoretical research. CHAI has its own things that it’s interested in, like cooperative inverse reinforcement learning and inverse reinforcement learning. OpenAI is also doing its own thing that I’m less clear on, which may have to do with factored evaluation. Ought as well is working on stuff. So could you expand a little bit here on that?

Jan Leike: Our direction for getting a solution to the alignment problem revolves around recursive reward modeling. I think on a high level, basically the way I’m thinking about this is that if you’re working on alignment, you really want to be part of the projects that builds AGI. Be there and have impact while that happens. In order to do that, you kind of need to be a part of the action. So you really have to understand the tech on the detailed level. And I don’t think that safety is an add on that you think of later or then add at a later stage in the process. And I don’t think we can just do some theory work that informs algorithmic decisions that make everything go well. I think we need something that is a lot more integrated with the project that actually builds AGI. So in particular, the way we are currently thinking about is it seems like the part that actually gives you alignment is not this algorithmic change and more something like an overall training procedure on how to combine your machine learning components into a big system.

So in terms of how I pick my research directions, I am most excited about approaches that can scale to AGI and beyond. Another thing that I think is really important is that I think people will just want to build agents, and we can’t only constrain ourselves to building say question answering systems. There’s basically a lot of real world problems that we want to solve with AGI, and these are fundamentally sequential decision problems, right? So if I look something up online and then I write an email, there’s a sequence of decisions I make, which websites I access and which links I click on. And then there’s a sequence of decisions of which characters are input in the email. And if you phrase the problem as, “I want to be able to do most things that humans can do with a mouse and a keyboard on a computer,” then that’s a very clearly scoped reinforcement learning problem. Although the reward function problem is not very clear.

Lucas Perry: So you’re articulating that DeepMind, you would explain that even given all these different approaches you guys have on all these different safety teams, the way that you personally pick your research direction is that you’re excited about things which safely scale to AGI and superintelligence and beyond. And that recursive reward modeling is one of these things.

Jan Leike: Yeah. So the problem that we’re trying to solve is the agent alignment problem. And the agent alignment problem is the question of how can we create agents that act in accordance with the user’s intentions. We are kind of inherently focused around agents. But also, we’re trying to figure out how to get them to do what we want. So in terms of reinforcement learning, what we’re trying to do is learn a reward function that captures the user’s intentions and that we can optimize with RL.

Lucas Perry: So let’s get into your work here on recursive reward modeling. This is something that you’re personally working on. Let’s just start off with what is recursive reward modeling?

Jan Leike: I’m going to start off with explaining what reward modeling is. What we want to do is we want to apply reinforcement learning to the real world. And one of the fundamental problems of doing that is that the real world doesn’t have built in reward functions. And so basically what we need is a reward function that captures the user’s intentions. Let me give you an example for the core motivation of why we want to do reward modeling, a blog posts that OpenAI made a while back: The Boat Race Problem, where they were training a reinforcement modeling agent to race the boat around the track and complete the race as fast as possible, but what actually ended up happening is that the boat was getting stuck in the small lagoon and then circling around there. And the reason for that is the RL agent was trying to maximize the number of points that it gets.

And the way you get points in this game is by moving over these buoys that are along the track. And so if you go to the lagoon, there’s these buoys that keep respawning, and then so you can get a lot of points without actually completing the race. This is the kind of behavior that we don’t want out of our AI systems. But then on the other hand, there’s things we wouldn’t think of but we want out of our AI systems. And I think a good example is AlphaGo’s famous Move 37. In its Go game against Lee Sedol, Move 37 was this brilliant move that AlphaGo made that was a move that no human would have made, but it actually ended up turning around the entire game in AlphaGo’s favor. And this is how Lee Sedol ended up losing the game. The commonality between both of these examples is some AI system doing something that a human wouldn’t do.

In one case, that’s something that we want: Move 37. In the other case, it’s something that we don’t want, this is the circling boat. I think the crucial difference here is in what is the goal of the task. In the Go example, the goal was to win the game of Go. Whereas in the boat race example, the goal was to go around the track and complete the race, and the agent clearly wasn’t accomplishing that goal. So that’s why we want to be able to communicate goals to our agents. So we need these goals or these reward functions that our agents learn to be aligned with the user’s intentions. If we do it this way, we also get the possibility that our systems actually outperform humans and actually do something that would be better than what the human would have done. And this is something that you, for example, couldn’t get out of imitation learning or inverse reinforcement learning.

The central claim that reward modeling is revolving around is that evaluation is easier than behavior. So I can, for example, look at a video of an agent and tell you whether or not that agent is doing a backflip, even though I couldn’t do a backflip myself. So in this case, it’s kind of harder to actually do the task than to evaluate it. And that kind of puts the human in the leveraged position because the human only has to be able to give feedback on the behavior rather than actually being able to do it. So we’ve been building prototypes for reward modeling for a number of years now. We want to actually get hands on experience with these systems and see examples of where they fail and how we can fix them. One particular example seen again and again is if you don’t provide online feedback to the agent, something can happen is that the agent finds loopholes in the reward model.

It finds states where the reward model think it’s a high reward state, but actually it isn’t. So one example is in the Atari game Hero, where you can get points for shooting laser beams at spiders. And so what the agent figures out is that if it stands really close to the spider and starts shooting but then turns around and the shot goes the other way, then the reward model will think the shot is about to hit the spider so it should give you a reward because that gives you points. But actually the agent doesn’t end killing the spider, and so it can just do the same thing again and again and get reward for it.

And so it’s kind of found this exploit in the reward model. We know that online training, training with an actual human in the loop who keeps giving feedback, can get you around this problem. And the reason is that whenever the agent gets stuck in these kinds of loopholes, a human can just look at the agent’s behavior and give some additional feedback you can then use to update the reward model. And the reward model in turn can then teach the agent that this is actually not a high reward state. So what about recursive reward modeling? One question that we have when you’re trying to think about how to scale reward modeling is that eventually you want to tackle domains where it’s too hard for the human to really figure out what’s going on because the core problem is very complex or the human is not an expert in the domain.

Right now, this is basically only in the idea stage, but the basic idea is to apply reward modeling recursively. You have this evaluation task that is too complex for the human to do, and you’re training a bunch of evaluation helper agents that will help you do the evaluation of the main agent that you’re training. These agents then in turn will be trained with reward modeling or recursive reward modeling.

Let’s say you want to train the agent that designs a computer chip, and so it does a bunch of work, and then it outputs this giant schema for what the chip could look like. Now that schema is so complex and so complicated that, as a human, even if you were an expert in chip design, you wouldn’t be able to understand all of that, but you can figure out what aspects of the chip you care about, right? Like what is the number of, say, FLOPS it can do per second or what is the thermal properties.

For each of these aspects that you care about, you will spin up another agent, you teach another agent how to do that subtask, and then you would use the output of that agent, could be, let’s say, a document that details the thermal properties or a benchmark result on how this chip would do if we actually built it. And then you can look at all of these outputs of the evaluation helper agents, and then you compose those into feedback for the actual agent that you’re training.

The idea is here that basically the tasks that the evaluation helper agents have to do are easier problems in a more narrow domain because, A, they only have to do one sub-aspect of the evaluation, and also you’re relying on the fact that evaluation is easier than behavior. Since you have this easier task, you would hope that if you can solve easier tasks, then you can use the solutions or the agents that you train on these easier tasks to kind of scaffold your way up to solving harder and harder tasks. You could use this to push out the general scope of tasks that you can tackle with your AI systems.

The hope would be that, at least I would claim, that this is a general scheme that, in principle, can capture a lot of human economic activity that way. One really crucial aspect is that you’re able to compose a training signal for the agents that are trying to solve the task. You have to ground this out where you’re in some level, if you picture this big tree or directed acyclic graph of agents that help you train other agents and so on, there has to be a bottom level where the human can just look at what’s going on and can just give feedback directly, and use the feedback on the lowest level task to build up more and more complex training signals for more and more complex agents that are solving harder and harder tasks.

Lucas Perry: Can you clarify how the bootstrapping here happens? Like the very bottom level, how you’re first able to train the agents dedicated to sub-questions of the larger question?

Jan Leike: If you give me a new task to solve with recursive reward modeling, the way I would proceed is, assuming that we solved all of these technical problems, let’s say we can train agents with reward modeling on arbitrary tasks, then the way I would solve it is I would first think about what do you care about in this task? How do I measure its success? What are the different aspects of success that I care about? These are going to be my evaluation criteria.

For each of my evaluation criteria, I’m going to define a new subtask, and the subtasks will be “help me evaluate this criteria.” In the computer chip example, that was FLOPs per second, and so on, and so on. Then I proceed recursively. For each of the subtasks that I just identified, I start again by saying, “Okay, so now I have this agent, it’s supposed to get computer chip design, and a bunch of associated documentation, say, and now it has to produce this document that outlines the thermal properties of this chip.”

What I would do, again, is I’d be like, “Okay, this is a pretty complex task, so let’s think about how to break it down. How would I evaluate this document?” So I proceed to do this until I arrive at a task where I can just say, “Okay, I basically know how to do this task, or I know how to evaluate this task.” And then I can start spinning up my agents, right? And then I train the agents on those tasks, and then once I’ve trained all of my agents on the leaf level tasks, and I’m happy with those, I then proceed training the next level higher.

Lucas Perry: And the evaluation criteria, or aspects, are an expression of your reward function, right?

Jan Leike: The reward function will end up capturing all of that. Let’s say we have solved all of the evaluation subtasks, right? We can use the evaluation assistant agents to help us evaluate the overall performance of the main agent that you were training. Of course, whenever this agent does something, you don’t want to have to evaluate all of their behavior, so what you do is you essentially distill this whole tree of different evaluation helper agents that you build. There’s lots of little humans in the loop in that tree into one model that will predict what that whole tree of what agents and humans will say. That will basically be the reward that the main agent is being trained on.

Lucas Perry: That’s pretty beautiful. I mean, the structure is elegant.

Jan Leike: Thanks.

Lucas Perry: I still don’t fully understand it obviously, but it’s beginning to dawn upon my non-computer science consciousness.

Jan Leike: Well, current research on this stuff revolves around two questions, and I think these are the main questions that we need to think about when trying to figure out whether or not a scheme like this can work. The first question is how well does the one-step set up work, only reward modeling, no recursion? If one-step reward modeling doesn’t work, you can’t hope to ever build a whole tree out of that component, so clearly that has to be true.

And then the second question is how do errors accumulate if we build a system? Essentially what you’re doing is you’re training a bunch of machine learning components to help you build a training signal for other machine learning components. Of course none of them are going to be perfect. Even if my ability to do machine learning is infinitely great, which of course it isn’t, at the end of the day, they’re still being trained by humans, and humans make mistakes every once in a while.

If my bottom level has a certain, let’s say, reward accuracy, the next level up that I use those to train is going to have a lower accuracy, or potentially have a lower accuracy, because their training signal is slightly off. Now, if you keep doing this and building a more and more complex system, how do the errors in the system accumulate? This is a question we haven’t really done much work on so far, and this is certainly something we need to do more on in the future.

Lucas Perry: What sorts of experiments can we do with recursive reward modeling today, and why is it hard?

Jan Leike: The reason why this is difficult to find such tasks is because essentially you need tasks that have two properties. The first property is they have to be difficult enough so that you can’t evaluate them directly, right? Otherwise, you wouldn’t need the recursive part in recursive reward modeling. And then secondly, they have to be easy enough that we can actually hope to be able to do them with today’s systems.

In a way, it’s two very contradictory objectives, so it’s kind of hard to find something in the intersection. We can study a lot of the crucial parts of this independently of actually being able to build a prototype of the recursive part of recursive reward modeling.

Lucas Perry: I guess I’m just trying to also get a sense of when you think that recursive reward modeling might become feasible.

Jan Leike: One good point would be where we could get to the point where we’ve done like a whole lot of tasks with reward modeling, and we’re basically running out of tasks that we can do directly. Or an opportunity comes up when we find a task that we actually think we can do and that requires a decomposition. There’s ways in which you could try to do this now by artificially limiting yourself. You could, for example, solve chess with recursive reward modeling by pretending that there isn’t a procedure or reward function for chess.

If you rely on the human to look at the board and tell you whether or not it’s checkmate, if you’re like a pro chess player, you could probably do that quite well. But if you’re an amateur or a non-expert, you don’t really know that much about chess other than the rules, it’s kind of hard for a human to do that quickly.

What you could do is you could train evaluation helper agents that give you useful information about the chessboard, where, let’s say, they color certain tiles on the board that are currently under threat. And then using that information, you can make the assessment of whether this is a checkmate situation much more easily.

While we could do this kind of setup and use recursive reward modeling, and you’ll maybe learn something, at the same time, it’s not an ideal test bed because it’s just not going to look impressive as a solution because we already know how to use machine learning to play chess, so we wouldn’t really add anything in terms of value of tasks that we can do now that we couldn’t do otherwise.

Lucas Perry: But wouldn’t it show you that recursive reward modeling works?

Jan Leike: You get one data point on this particular domain, and so the question is what data points would you learn about recursive reward modeling that you wouldn’t learn in other ways? You could treat this as like two different individual tasks that you just solve with reward modeling. One task is coloring the tiles of the board, and one task is actually playing chess. We know we can do the latter, because we’ve done that.

What would be interesting about this experiment would be that you kind of learn how to cope with the errors in the system. Every once in a while, like the human will label a state incorrectly, and so you would learn how well you can still train even though your training signal is slightly off. I think we can also investigate that without actually having to literally build this recursive setup. I think there’s easier experiments we could do.

Lucas Perry: Do you want to offer any final words of clarification or just something succinct about how this serves AI safety and alignment?

Jan Leike: One way to think about safety is this specification robustness assurance framework. What this is, is basically a very high-level way of carving the space of safety problems into different categories. These are three categories. The first category is specification. How do you get the system to do what you want? Basically, what we usually mean when we talk about alignment. The second category is robustness. How can you make your system robust to various perturbations, such as adversarial inputs or distributional changes? The third category is assurance. How can you get better calibrated beliefs about how safe, or in the sense of robust and specification too, your system actually is?

Usually in assurance category, we talk about various tools for understanding and monitoring agents, right? This is stuff about testing, interpretability, verification, and so on, and so on. The stuff that I am primarily working on is in the specification category, where we’re basically trying to figure out how to get our agents to pursue the goals that we want them to pursue. The ambition of recursive reward modeling is to solve all of the specification problems. Some of the problems that we worry about are, let’s say, off switch problems, where your agent might meddle with its off switch, and you just don’t want it to do that. Another problem, let’s say, what about side effects? What about reward tampering? There’s a whole class of these kind of problems, and instead of trying to solve them each individually, we try to solve the whole class of problems at once.

Lucas Perry: Yeah, it’s an ambitious project. The success of figuring out the specification problem supervenes upon other things, but at the same time, if those other things are figured out, the solution to this enables, as you’re saying, a system which safely scales to super intelligence and beyond, and retains alignment, right?

Jan Leike: That’s the claim.

Lucas Perry: Its position then in AI safety and alignment is pretty clear to me. I’m not sure if you have other points you want to add to that though?

Jan Leike: Nope, I think that was it.

Lucas Perry: Okay. We’re just going to hit on a few more questions here then on recursive reward modeling. Recursive reward modeling seems to require some very competent agent or person to break down an evaluation into sub-questions or sub-evaluations. Is the creation of that structure actually scalable?

Jan Leike: Yeah, this is a really good question. I would picture these decompositions of the evaluation task to be essentially hardcoded, so you have a human expert that knows something about the task, and they can tell you what they care about in the task. The way I picture this is you could probably do a lot of the tasks in recursion depth of three, or five, or something like that, but eventually they’re so out of the range of what the human can do that they don’t even know how to break down the evaluation of that task.

Then this decomposition problem is not a problem that you want to tackle with recursive reward modeling, where basically you train an agent to propose decompositions, and then you have an evaluation where the human evaluates whether or not that was good decomposition. This is very, very far future stuff at that point; you’ve already worked with recursive reward modeling for a while, and you have done a bunch of decompositions, and so I don’t expect this to be something that we will be addressing anytime soon, and it’s certainly something that is within the scope of what the stuff should be able to do.

Recursive reward modeling is a super general method that you basically want to be able to apply to any kind of task that humans typically do, and proposing decompositions of the evaluation is one of them.

Lucas Perry: Are there pessimisms that you have about recursive reward modeling? How might recursive reward modeling fall short? And I think the three areas that we want to hit here are robustness, mesa-optimizers, and tasks that are difficult to ground.

Jan Leike: As I said earlier, basically you’re trying to solve the whole class of specification problems, where you still need robustness and assurance. In particular, there’s what we call the reward to result gap, where you might have the right reward function, and then you still need to find an agent that actually is good at optimizing that reward function. That’s an obvious problem, and there’s a lot of people just trying to make RL agents perform better. Mesa-optimizers I think are, in general, an open question. There’s still a lot of uncertainty around how they would come up, and what exactly is going on there. I think one thing that would be really cool is actually have a demo of how they could come up in a training procedure in a way that people wouldn’t expect. I think that would be pretty valuable.

And then thirdly, recursive reward modeling is probably not very well suited for tasks that are really difficult to ground. Moral philosophy might be in that category. The way I understand this is that moral philosophy tries to tackle questions that is really difficult to get really hard facts and empirical evidence on. These human intuitions might be like really difficult to ground, and to actually teach to our agents in a way that generalizes. If you don’t have this grounding, then I don’t know how you could build a training signal for the higher level questions that might evolve from that.

In other words, to make this concrete, let’s say I want to train an agent to write a really good book in moral philosophy, and now of course I can evaluate that book based on how novel it is relative to what the humans have written, or like the general literature. How interesting is it? Does it make intuitive sense? But then in order to actually make the progress on moral philosophy, I need to update my value somehow in a way that is actually the right direction, and I don’t really know what would be a good way to evaluate.

Lucas Perry: I think then it would be a good spot here for us to circle back around to the alignment landscape. A lot of what you’ve been saying here has rung bells in my head about other efforts, like with iterated distillation and application, and debate with Geoffrey Irving, and factored evaluation at Ought. There’s these categories of things which are supposed to be general solutions to making systems which safely scale to aligned super intelligence and beyond. This also fits in that vein of the alignment landscape, right?

Jan Leike: Yeah. I think that’s right. In some ways, the stuff that you mentioned, like projects that people are pursuing at OpenAI, and at Ought, share lot structure with what recursive reward modeling is trying to do, where you try to compose training signals for tasks that are too hard for humans. I think one of the big differences in how we think about this problem is that we want to figure out how to train the agents that do stuff in the world, and I think a lot of the discussion at OpenAI and Ought kind of center around building question answering systems and fine tuning language models, where the ambition is to get them to do reasoning tasks that are very difficult for humans to do directly, and then you do that by decomposing them into easier reasoning tasks. You could say it’s one scalable alignment technique out of several that are being proposed, and we have a special focus on agents. I think agents are great. I think people will build agents that do stuff in the world, take sequential actions, look at videos.

Lucas Perry: What research directions or projects are you most excited about, just in general?

Jan Leike: In general, the safety community as a whole should have a portfolio approach where we just try to pursue a bunch of paths in parallel, essentially as many as can be pursued in parallel. I personally I’m most excited about approaches that can work with existing deep learning and scale to AGI and beyond. There could be many ways in which things pan out in the future, but right now there’s an enormous amount of resources being put towards scaling deep learning. That’s something that we should take seriously and consider into the way we think about solutions to the problem.

Lucas Perry: This also reflects your support and insistence on empirical practice as being beneficial, and the importance of being in and amongst the pragmatic, empirical, tangible, real-world, present-day projects, which are likely to bring about AGI, such that one can have an impact. What do you think is missing or underserved in AI alignment and AI safety? If you were given, say, like $1 billion, how would you invest it here?

Jan Leike: That would be the same answer I just gave you before. Basically, I think you want to have a portfolio, so you invest that money to like a whole bunch of directions. I think I would invest more than many other people in the community towards working empirically with, let’s say, today’s deep RL systems, build prototypes of aligned AGI, and then do experiments on them. I’ll be excited to see more of that type of work. Those might be like my personal biases speaking too because that’s like why I’m working on this direction.

Lucas Perry: Yeah. I think that’s deeply informative though for other people who might be trying to find their footing in determining how they ought to approach this problem. So how likely do you think it is that deep reinforcement learning scales up to AGI? What are the strongest considerations for and against that?

Jan Leike: I don’t think anyone really knows whether that’s the case or not. Deep learning certainly has a pretty convincing track record of fitting arbitrary functions. We can basically fit a function that knows how to play StarCraft. That’s a pretty complicated function. I think, well, whatever the answer is to this question, in safety what we should be doing is we should be conservative about this. Take the possibility that deep RL could scale to AGI very seriously, and plan for that possibility.

Lucas Perry: How modular do you think AGI will be and what makes you optimistic about having clearly defined components which do planning, reward modeling, or anything else?

Jan Leike: There’s certainly a lot of advantages if you can build a system of components that you understand really well. The way I currently picturing trying to build, say, a prototype for aligned AGI would be somewhat modular. The trend in deep learning is always towards training end-to-end. Meaning that you just have your raw data coming in and the raw predictions coming out and you just train some giant model that does all of that. That certainly gives you performance benefits on some tasks because whatever structure the model ends up learning can just be better than what the humans perceptions would recommend.

How it actually ends up working out is kind of unclear at the moment. I think in terms of what we’d like for safety is that if you have a modular system, it’s going to make it easier to really understand what’s going on there because you can understand the components and you can understand how they’re working together, so it helps you break down the problem of doing assurance in the system, so that’s certainly a path that we would like to work out

Lucas Perry: And is inner alignment here a problem that is relevant to both the modular components and then how the modular components are interrelated within the system?

Jan Leike: Yeah, I think you should definitely think about what the incentives are and what the training signals are of all of the components that you’re using to build a system.

Lucas Perry: As we approach AGI, what efforts are going on right now at DeepMind for AI safety benchmarking?

Jan Leike: We’ve spent some time thinking about AI safety benchmarks. We made a few little environments that are called gridworlds that are basically just kind of like chess board tiles where your agent moves around, and those are I think useful to showcase some of the problems. But really I think there’s a lot of appetite right now for building benchmark environments that let you test your agent on different properties. For example, OpenAI just recently released a collection of environments for safe exploration that require you to train in the presence of site constraints. But there’s also a lot of other properties that you could actually build tractable benchmarks for today.

So another example would be adversarial inputs, and there’s this generalized adversarial examples challenge. There’s also, you could build a benchmark for distributional shift, which in some ways you already do that in machine learning a lot where you do a little training in a test split, but usually these are on from the same decision. There’s various trans learning research going on. I don’t think there is really established benchmarks for those. This is certainly something that could be done.

There’s also problems that we worry about in longterm safety that I think would be kind of hard to really do good benchmarks on right now. Here I’m thinking of things like the off-switch problems, reward gaming, where you actually have an agent that can modify its own input rewards. The problem here is really you need very complex environments that are difficult to build and learn with current systems.

But I think overall this is something that would be very useful for the community to pursue, because the history of recent machine learning progress has always been that if you make a benchmark, people will start improving on the benchmark. The benchmark starts driving progress, and we’ve seen this with the ImageNet benchmark. We’ve seen that with the Atari benchmark, just to name two examples. And so if you had a safety benchmark, you would kind of incentivize people to make safety progress. Then if it’s an established benchmark, you can also publish on this. Then longer term, once you’ve had success with a bunch of benchmarks or they’ve been established and accepted, they could also become industry norms.

Lucas Perry: I’m just trying to understand how benchmarks in general, whether they’re safety benchmarks or not, exist in the international and national communities of computer science. Are Chinese computer scientists going to care about the DeepMind safety benchmarks? Are they something that necessarily are incorporated?

Jan Leike: Why do you think Chinese researchers care about the ImageNet benchmark?

Lucas Perry: Well, I don’t really know anything about the ImageNet benchmark.

Jan Leike: Oh, so the ImageNet is this big collection of labeled images that a lot of people train image classifiers on. So these are like pictures of various breeds of dogs and cats and so on. Things people are doing, or at least were doing for a while, was training larger and larger vision models on ImageNet and then you can measure what is your test accuracy of ImageNet and that’s a very tangible benchmark on how well you can do computer vision with your machine learning models.

Lucas Perry: So when these benchmarks are created, they’re just published openly?

Jan Leike: Yeah, you can just download ImageNet. You can just get started at trying a model on ImageNet in like half an hour on your computer.

Lucas Perry: So the safety benchmarks are just published openly. They can just be easily accessed, and they are public and open methods by which systems can be benchmarked to the degree to which they’re aligned and safe?

Jan Leike: Yeah. I think in general people underestimate how difficult environment design is. I think in order for a safety benchmark to get established, it actually has to be done really well. But if you do it really well and you can get a whole bunch of people interested because it’s becomes clear that this is something that is hard to do and methods can’t … but it’s also something that, let’s say if you made progress on it, you could write a paper or you can get employed at a company because you did something that people agreed was hard to do. At the same time, it has the result that is very easily measurable.

Lucas Perry: Okay. To finish up on this point, I’m just interested in if you could give a final summary on your feelings and interests in AI safety benchmarking. Besides the efforts that are going on right now, what are you hoping for?

Jan Leike: I think in summary I would be pretty excited about seeing more safety benchmarks that actually measure some of the things that we care about if they’re done well and if they really pay attention to a lot of detail, because I think that can drive a lot of progress on these problems. It’s like the same story as with reward modeling, right? Because then it becomes easy to evaluate progress and it becomes easy to evaluate what people are doing and that makes it easy for people to do stuff and then see whether or not whatever they’re doing is helpful.

Lucas Perry: So there appears to be cultural and intellectual differences between the AI alignment and AI safety communities and the mainstream AI community, people who are just probably interested in deep learning and ML.

Jan Leike: Yeah. So historically the machine learning community and the long term safety community have been somewhat disjoined.

Lucas Perry: So given this disjointedness, what would you like the mainstream ML and AI community to do or to think differently?

Jan Leike: The mainstream ML community doesn’t think enough about how whatever they are building will actually end up being deployed in practice, and I think people that are starting to realize that they can’t really do RL in the real world if they don’t do reward modeling, and I think it’s most obvious to robotics people trying to get robots to do stuff in the real world all the time. So I think reward modeling will become a lot more popular. We’ve already seen that in the past two years.

Lucas Perry: Some of what you’re saying is reminding me of Stuart Russell’s new book, Human Compatible. I’m curious to know if you have any thoughts on that and what he said there and how that relates to this.

Jan Leike: Yes. Stuart also has been a proponent of this for a long time. In a way, he has been one of the few computer science professors who are really engaging with some of these longer term AI questions, in particular around safety. I don’t know why there isn’t more people saying what he’s saying.

Lucas Perry: It seems like it’s not even just the difference and disjointedness between the mainstream ML and AI community and then the AI safety and AI alignment community is just that one group is thinking longterm and the other is not. It’s just a whole different perspective and understanding about what it means for something to be beneficial and what it takes for something to be beneficial. I don’t think you need to think about the future to understand the importance of recursive reward modeling or the kinds of shifts that Stuart Russell is arguing for given the systems which are being created today are already creating plenty of problems. We’ve enumerated those many times here. That seems to me to be because the systems are clearly not fully capturing what human beings really want. Just trying to understand better and also what you think the alignment and AI safety community should do or think differently to address this difference.

Jan Leike: The longterm safety community in particular, initially I think a lot of the arguments people made were very high level and almost philosophical, has been a little bit of a shift towards concrete mathematical, but also at the same time very abstract research, then towards empirical research. I think this is kind of a natural transition from one mode of operation to something more concrete, but there’s still some parts of the safety community in the first phases.

I think there’s a failure mode here where people just spend a lot of time thinking about what would be the optimal way of addressing a certain problem before they actually go out and do something, and I think an approach that I tend to favor is thinking about this problem for a bit and then doing some stuff and then iterating and thinking some more. That way you get some concrete feedback on whether or not you’re actually making progress.

I think another thing that I would love to see the community do more of is I think there’s not enough appreciation for clear explanations, and there’s a tendency that people write a lot of vague blog posts, and that’s difficult to critique and to build on. Where we really have to move as a community is toward more concrete technical stuff that you can clearly point at parts of it and be like, “This makes sense. This doesn’t make sense. He has very likely made a mistake,” then that’s something where we can actually build on and make progress with.

In general. I think this is the sort of community that attracts a lot of thinking from first principles, and there’s a lot of power in that approach. If you’re not bound by what other people think and what other people have tried, then you can really discover truly novel directions and truly novel approaches and ideas, but at the same time I think there’s also a danger of overusing this kind of technique, because I think it’s right to also connect what you’re doing with the literature and what everyone else is doing. Otherwise, you will just keep reinventing the wheel on some kind of potential solution to safety.

Lucas Perry: Do you have any suggestions for the AI safety and alignment community regarding also alleviating or remedying this cultural and intellectual difference between what they’re working on and what the mainstream ML and AI communities are doing and working on such that it shifts their mindset and work to increase the chances that more people are aware of what is required to create beneficial AI systems?

Jan Leike: Something that would be helpful for this bridge would be if the safety community as a whole, let’s say, spends more time engaging with the machine learning literature, the machine learning lingo and jargon, and try to phrase the safety ideas and the research in those terms and write it up in a paper that can be published in NeurIPS rather than something that is a blog post. The form is just a format that people are much more likely to engage with.

This is not to say that I don’t like blog posts. Blogs are great for getting some of the ideas across. We also provide blog posts about our safety research at DeepMind, but if you really want to dive into the technical details and you want to get the machine learning community to really engage with the details of your work, then writing a technical paper is just the best way to do that.

Lucas Perry: What do you think might be the most exciting piece of empirical safety work which you can realistically imagine seeing within the next five years?

Jan Leike: We’ve done a lot of experiments with reward modeling, and I personally have been surprised how far we could scale it. It’s been able to tackle every problem that we’ve been throwing at it. So right now we’re training agents to follow natural language instructions in these 3D environments that we’re building here at DeepMind. These are a lot harder problems than, say, Atari games, and reward modeling still is able to tackle them just fine.

One kind of idea what that prototype could look like is a model-based reinforcement learning agent where you learn a dynamics model then train a reward model from human feedback then the reinforcement learning agent uses the dynamics model and the reward model to do search at training and at test time. So you can actually deploy it in the environment and it can just learn to adapt its plans quickly. Then we could use that to do a whole bunch of experiments that we would want that system to do. You know like, solve off-switch problems or solve reward tampering problems or side effects, problems and so on. So I think that’d be really exciting, and I think that’s well within the kind of system that we could build in the near future.

Lucas Perry: Cool. So then wrapping up here, let’s talk a little bit about the pragmatic side of your team and DeepMind in general. Is DeepMind currently hiring? Is the safety team hiring? What is the status of all of that, and if listeners might be able to get involved?

Jan Leike: We always love to hire new people that join us in these efforts. In particular, we’re hiring research engineers and research scientists to help us build this stuff. So, if you, dear listener, have, let’s say, a Master’s degree in machine learning or some kind of other hands on experience in building and training deep learning systems, you might want to apply for a research engineering position. For a research scientist position the best qualification is probably a PhD in machine learning or something equivalent. We also do research internships for people who maybe have a little bit early in their PhD. So this is the kind of thing that applies to you and you’re excited about working on these sort of problems, then please contact us.

Lucas Perry: All right, and with that, thank you very much for your time, Jan.

End of recorded material

AI Alignment Podcast: Machine Ethics and AI Governance with Wendell Wallach

Wendell Wallach has been at the forefront of contemporary emerging technology issues for decades now. As an interdisciplinary thinker, he has engaged at the intersections of ethics, governance, AI, bioethics, robotics, and philosophy since the beginning formulations of what we now know as AI alignment were being codified. Wendell began with a broad interest in the ethics of emerging technology and has since become focused on machine ethics and AI governance. This conversation with Wendell explores his intellectual journey and participation in these fields.

 Topics discussed in this episode include:

  • Wendell’s intellectual journey in machine ethics and AI governance 
  • The history of machine ethics and alignment considerations
  • How machine ethics and AI alignment serve to produce beneficial AI 
  • Soft law and hard law for shaping AI governance 
  • Wendell’s and broader efforts for the global governance of AI
  • Social and political mechanisms for mitigating the risks of AI 
  • Wendell’s forthcoming book

Key points from Wendell:

  • “So when you were talking about machine ethics or when we were talking about machine ethics, we were really thinking about it in terms of just how do you introduce ethical procedures so that when machines encounter new situations, particularly when the designers can’t fully predict what their actions will be, that they factor in ethical considerations as they choose between various courses of action. So we were really talking about very basic program in the machines, but we weren’t just thinking of it in terms of the basics. We were thinking of it in terms of the evolution of smart machines… What we encounter in the Singularity Institute, now MIRI for artificial intelligence approach of friendly AI and what became value alignment is more or less a presumption of very high order intelligence capabilities by the system and how you would ensure that their values align with those of the machines. They tended to start from that level. So that was the distinction. Where the machine ethics folks did look at those futuristic concerns, they did more so from a philosophical level and at least a belief or appreciation that this is going to be a relatively evolutionary course, whereby the friendly AI and value alignment folks, they tended to presume that we’re going to have very high order cognitive capabilities and how do we ensure that those align with the systems. Now, the convergence, I would say, is what’s happening right now because in workshops that have been organized around the societal and ethical impact of intelligent systems.”
  • “My sense has been that with both machine ethics and value alignment, we’ve sort of got the cart in front of the horse. So I’m waiting to see some great implementation breakthroughs, I just haven’t seen them. Most of the time, when I encounter researchers who say they’re taking seriously, I see they’re tripping over relatively low level implementations. The difficulty is here, and all of this is converging. What AI alignment was initially and what it’s becoming now I think are quite different. I think in the very early days, it really was presumptions that you would have these higher order intelligences and then how were you going to align them. Now, as AI alignment, people look at the value issues as they intersect with present day AI agendas. I realize that you can’t make the presumptions about the higher order systems without going through developmental steps to get there. So, in that sense, I think whether it’s AI alignment or machine ethics, the one will absorb the lessons of the other. Both will utilize advances that happen on both fronts.”
  • “David Collingridge wrote a book where he outlined a problem that is now known as the Collingridge Dilemma. Basically, Collingridge said that while it was easiest to regulate a technology early in its style development, early in its development, we had a little idea of what its societal impact would be. By the time we did understand what the challenges from the societal impact were, the technology would be so deeply entrenched in our society that it would be very difficult to change its trajectory. So we see that today with social media. Social media was totally entrenched in our society before we realized how it could be manipulated in ways that would undermine democracy. Now we’re having a devil of a time of figuring out what we could do. So Gary and I, who had been talking about these kinds of problems for years, we realized that we were constantly lamenting the challenge, but we altered the conversation one day over a cup of coffee. We said, “Well, if we had our druthers, if we have some degree of influence, what would we propose?” We came up with a model that we referred to as governance coordinating committees. Our idea was that you would put in place a kind of issues manager that would try and guide the development of a field, but first of all, it would just monitor development, convene forums between the many stakeholders, map issues and gaps, see if anyone was addressing those issues and gaps or where their best practices had come to the floor. If these issues were not being addressed, then how could you address them, looking at a broad array of mechanisms. By a broad array of mechanisms, we meant you start with feasible technological solutions, you then look at what can be managed through corporate self-governance, and if you couldn’t find anything in either of those areas, then you turn to what is sometimes called soft law… So Gary and I proposed this model. Every time we ever talked about it, people would say, “Boy, that’s a great idea. Somebody should do that.” I was going to international forums, such as going to the World Economic meetings in Davos, where I’d be asked to be a fire-starter on all kinds of subject areas by safety and food security and the law of the ocean. In a few minutes, I would quickly outline this model as a way of getting people to think much more richly about ways to manage technological development and not just immediately go to laws and regulatory bodies. All of this convinced me that this model was very valuable, but it wasn’t being taken up. All of that led to this first International Congress for the Governance of Artificial Intelligence, which will be convened in Prague on April 16 to 18. I do invite those of you listening to this podcast who are interested in the international governance of AI or really agile governance for technology more broadly to join us at that gathering.”

 

Important timestamps: 

0:00 intro

2:50 Wendell’s evolution in work and thought

10:45 AI alignment and machine ethics

27:05 Wendell’s focus on AI governance

34:04 How much can soft law shape hard law?

37:27 What does hard law consist of?

43:25 Contextualizing the International Congress for the Governance of AI

45:00 How AI governance efforts might fail

58:40 AGI governance

1:05:00 Wendell’s forthcoming book

 

Works referenced:

A Dangerous Master: How to  Keep Technology from Slipping Beyond Our Control 

Moral Machines: Teaching Robots Right from Wrong

 

We hope that you will continue to join in the conversations by following us or subscribing to our podcasts on Youtube, Spotify, SoundCloud, iTunes, Google Play, StitcheriHeartRadio, or your preferred podcast site/application. You can find all the AI Alignment Podcasts here.

You can listen to the podcast above or read the transcript below. 

Lucas Perry: Hey everyone, welcome to the AI Alignment Podcast. I’m Lucas Perry. Today, we’ll be speaking with Wendell Wallach. This episode is primarily dedicated to the issue and topic of AI governance, though in order to get there we go on and explore Wendell’s intellectual journey in machine ethics and how that led him up to his current efforts in AI governance. We also discuss how machine ethics and AI alignment both attempt to serve the project of creating beneficial AI and deal with the moral and ethical considerations related to the growing power and use of artificial intelligence. We discuss soft law and hard law for shaping AI governance. We get into Wendell’s efforts for the global governance of AI and discuss the related risks. And to finish things off we also briefly touch on AGI governance and Wendell’s forthcoming book. If you find this podcast valuable, interesting, or helpful, consider sharing it with others who might find it valuable as well.

For those who are not familiar with Wendell, Wendell is an internationally recognized expert on the ethical and governance concerns posed by emerging technologies, particularly artificial intelligence and neuroscience. Wendell is a consultant and ethicist, and a scholar at Yale University’s Interdiscplinary Center or Bioethics. He is also a co-author with Colin Allen, Moral Machines: Teaching Robots Right from Wrong. This work maps machine ethics, machine morality, computational morality, and friendly AI. He has a second and more recent book, A Dangerous Master: How to Keep Technology from Slipping Beyond our Control. From my perspective of things, it seems there is much growing enthusiasm and momentum in the space of AI policy and governance efforts. So, this conversation and those like it I feel help to further develop my perspective and understanding of where we are in the project and space of AI governance. For these reasons, I hope that you’ll find it valuable as well. So, let’s get into our conversation with Wendell Wallach.

It would be great if you could start by clarifying the evolution of your thought in science and technology over the years. It appears that you’ve gone from being interested in bioethics to machine ethics to now a more recent focus in AI governance and AI ethics. Can you take us through this movement in your thought and work?

Wendell Wallach: In reality, all three of those themes have been involved in my work from the very beginning, but the emphasis has changed. So I lived a very idiosyncratic life that ended with two computer consulting companies that I had helped start. But I had felt that there were books that I wanted to get out of my head, and I turned those companies over to the employees, and I started writing and realized that I was not up on some of the latest work in cognitive science. So one thing led to another, and I was invited to the first meeting of a technology and ethics working group at Yale University that had actually been started by Nick Bostrom when he was at Yale and Bonnie Kaplan. Nick left about a year later, and a year after that, Bonnie Kaplan had an accident, and the chair of that working group was turned over to me.

So that started my focus on technology and ethics more broadly. It was not limited to bioethics, but it did happen within the confine for the Yale Interdisciplinary Center for Bioethics. I was all over the place and the sense that I was already a kind of transdisciplinary thinker, transdisciplinary scholar, but having the challenge of focusing my study and my work so it was manageable. In other words, I was trying to think broadly at the same time as I was trying to focus on different subject areas. One thing led to another. I was invited to a conference in Baden Baden where I met Colin Allen. We together with the woman who started the workshop there, Eva Schmidt, began thinking about a topic that we were calling machine morality at that time. By machine morality, we meant thinking about how moral decision making faculties might be implemented in computers and robots.

Around the same time, there were other scholars working on the same themes. Michael and Susan Anderson, for example, had grabbed on to the title ‘machine ethics.’ Over time, as these various pathways converge, machine ethics became the main research area or the way in which this research project was referred to. It did have other names in addition to machine morality. It was sometimes called computational morality. At the same time, there were others who were working on it under the title of friendly AI, a term that was coined by Eliezer Yudkowsky. But the real difference between the machine ethics folks and the friendly AI folks was that the friendly AI folks were explicitly focused upon the challenge of how you would manage or tame superintelligence, whereby the machine ethics crew were much more ethicists, philosophers, computer scientists who were really thinking about first steps toward introducing moral decision making faculties, moral sensitivity into computers and robots. This was a relatively small group of scholars, but as this evolved over time, Eva and Collin and I decided that we would write a book mapping the development of this field of research.

Eva Schmidt fell away, and the book finally came out from Oxford University Press under the title Moral Machines: Teaching Robots Right from Wrong. So, as you may be aware, that’s still a seminal text out there. It’s still something that is read broadly and is being cited broadly, and in fact, it’s citations are going up and were even being requested by Oxford University Press to produce an update of the book. Machine Ethics was two parts philosophy, one part, computer science. It was basically two fields of study. One was looking explicitly at the question of implementing sensitivity to moral considerations in computers and robots, and the other side with really thinking comprehensively about how humans make moral decisions. So, arguably, Moral Machines was the first book that really took that comprehensive look at human moral decision making seriously. It was also a time when there was a lot of research going on in moral psychology in the way in which people’s affective and decision making concerns affected what became our ethical decision making processes.

So we were also able to bring some of that in, bring evolutionary psychology in and bring a lot of new fields of research that had not really been given their due or had not been integrated very well with the dominant reason based theories of ethics such as deontology, which is really ethical approaches that focus on duties, rules and consequentialism, which is an ethical theory that says right and wrong is not determined by following the rules or doing your duty, it’s determined by looking at the consequences of your action and selecting that course or the action likely to produce the greatest good for the greatest number. So it’s like we were integrating evolutionary psychology, cognitive science, moral psychology, together with the more rational-based theories, as we looked at top down and bottom up approaches for introducing sensitivity to ethical considerations in computers and robots.

The major shift in that whole trajectory and one I only learned about at the first FLI conference in Puerto Rico where I and Jim Moor were the only two people who had been actively involved in the machine ethics community, Jim Moor is a professor at Dartmouth, for those of you who are not aware of him, and he has been a seminal figure in the philosophy of computing for decades now, was at that Puerto Rican gathering, the concept of value alignment with race to us for the first time. What I realized was that those who are talking about value alignment from the AI perspective, by and large, had little or no understanding that there had ever been a field or was an ongoing field known as machine ethics.

That led to my applying for a Future of Life Institute grant, which I was awarded as PI. That grant was to host three annual workshops bringing together experts not only in AI, but machine ethics, philosophy, generally, resilience, engineering, robotics, a broad array of fields of people who had been thinking seriously about value issues in computational systems. Those really became groundbreaking workshops where it was clear that the computer scientists and the AI researchers knew very little about ethics issues, and the ethicists didn’t necessarily have a great depth of understanding of some of the challenges coming up in artificial intelligence. Bart Selman and Stuart Russell agreed to be co-chairs of those workshops with me. The last one was completed over a year ago with some closing presentations in New York city and at Yale.

Lucas Perry: I think it’d be helpful here if you could disambiguate the machine ethics crowd and way of thinking and what has been done there from the AI alignment, value alignment, Eliezer branch of thinking that has been going on. AI alignment seems more focused on explicitly trying to understand human preference hierarchies and be able to specify objectives without the machine systems doing other things that we don’t want them to do. Then you said that machine ethics is about imbuing ethical decision making faculties or reasoning or sensitivities in machine systems. That, to me, seems more like normative ethics. We have these normative theories like you mentioned deontology and consequentialism and virtue ethics, and maybe machines can invent other normative ethical theories. So they seem like different projects.

Wendell Wallach: They are very different projects. The question is whether they converge or not or whether they can really be treated totally distinct projects from each other. So when you were talking about machine ethics or when we were talking about machine ethics, we were really thinking about it in terms of just how do you introduce ethical procedures so that when machines encounter new situations, particularly when the designers can’t fully predict what their actions will be, that they factor in ethical considerations as they choose between various courses of action. So we were really talking about very basic program in the machines, but we weren’t just thinking of it in terms of the basics. We were thinking of it in terms of the evolution of smart machines. For example, in Moral Machines, Colin and I had a chart that we had actually developed with Eva Schmidt and had been in earlier articles that the three of us offered, and it looked at the development of machines on two axes.

One was increasing autonomy, and the other was increasing sensitivity with at the far other extremes, sensitivity to ethical consideration. We realized that you could put any tool within that chart. So a hammer has no sensitivity, and it has no autonomy. But when you think of a thermostat, it has a very low degree of sensitivity and a very low degree of autonomy, so as temperatures change, it can turn on or off heating. We then, within that chart, had a series of semicircles, one that delineated when we moved into the realm of what we labeled operational morality. By operational morality, we meant that the computer designers could more or less figure out all the situations the system would encounter and hard program its responses to those situations. The next level was what we call functional morality, which was as the computer programmers could no longer predetermine all the situations the system would encounter, the system would have to have some kind of ethical sub routines. Then at the highest level was full moral agency.

What we encounter in the Singularity Institute, now MIRI for artificial intelligence approach of friendly AI and what became value alignment is more or less a presumption of very high order intelligence capabilities by the system and how you would ensure that their values align with those of the machines. They tended to start from that level. So that was the distinction. Where the machine ethics folks did look at those futuristic concerns, they did more so from a philosophical level and at least a belief or appreciation that this is going to be a relatively evolutionary course, whereby the friendly AI and value alignment folks, they tended to presume that we’re going to have very high order cognitive capabilities and how do we ensure that those align with the systems. Now, the convergence, I would say, is what’s happening right now because in workshops that have been organized around the societal and ethical impact of intelligent systems. The first experiments even the value alignment people are doing still tend to be relatively low level experiments, given the capabilities assistants have today.

So I would say, in effect, they are machine ethics experiments or at least they’re starting to recognize that the challenges at least initially aren’t that much different than those the machine ethicists outlined. As far as the later concerns go, which is what is the best course to proceed on producing systems that are value aligned, well there, I think we have some overlap also coming into the machine ethicist, which raises questions about some of these more technical and mathematically-based approaches to value alignment and whether they might be successful. In that regard, Shannon Vallor, an ethicist at Santa Clara University, who wrote a book called Technology and the Virtues, and has now taken a professorship at Edinburgh, she and I produced a paper called, I think it was From Machine Ethics to Value Alignment to virtue alignment. We’re really proposing that analytical approaches alone will not get us to machines that we can trust or that will be fully ethically aligned.

Lucas Perry: Can you provide some examples about specific implementations or systems or applications of machine ethics today?

Wendell Wallach: There really isn’t much. Sensitivity to ethical considerations is still heavily reliant on how much we can get that input into systems and then how you integrate that input. So we are still very much at the stage of bringing various inputs in without a lot of integration, let alone analysis of what’s been integrated and decisions being made based on that analysis. For all purposes and both machine ethics, then I would say, bottom up value alignment, there’s just not a lot that’s been done. These are still somewhat futuristic research trajectories.

Lucas Perry: I think I’m just trying to poke here to understand better about what you find most skillful and useful about both approaches in terms of a portfolio approach to building beneficial AI systems, like if this is an opportunity to convince people that machine ethics is something valuable and that should be considered and worked on and expanded. I’m curious to know what you would say.

Wendell Wallach: Well, I think machine ethics is the name of the game in the sense that for all I talk about systems that will have very high order of capabilities. We just aren’t there. We’re still dealing with relatively limited forms of cognitive decision making. For all the wonder that’s going on in machine learning, that’s still a relatively limited kind of learning approach. So I’m not dealing with machines that are making fundamental decisions at this point, or if they are allowed to, it’s largely because humans have abrogated their responsibility, trust the machines, and let the machines make the decisions regardless of whether the machines actually have the capabilities to make sophisticated decisions.

Well, I think as we move along, as you get more and more inputs into systems and you figure out ways of integrating them, there will be the problem of which decisions can be made without, let’s just say, higher order consciousness or understanding of the falling implications of those systems, of the situations, of the ethical concerns arising in the situations and which decisions really require levels of, and I’m going to use the understanding and consciousness words, but I’m using them in a circumspect way for the machines to fully appreciate the ramifications of the decisions being made and therefore those who are affected by those decisions or how those decisions will affect those around it.

Our first stage is going to be largely systems of limited consciousness or limited understanding and our appreciation of what they can and cannot do in a successful manner and when you truly need a human decision maker in the loop. I think that’s what we are broadly. The differences between the approaches with the AI researchers are looking at what kind of flexibility they have within the tools I have now for building AI systems. The machine ethicists, I think they’ll tend to be largely philosophically rooted or ethically rooted or practically ethically rooted, and therefore they tend to be more sensitive to the ramifications of decision makings by machines and capacities that need to be accounted for before you want to turn over a decision to a machine, such as a lethal autonomous weapon. What should the machine really understand before it can be a lethal autonomous weapon, and therefore, how tightly does the meaningful human control need to be?

Lucas Perry: I’m feeling a tension between trying to understand the role and place of both of these projects and how they’re skillful. In terms just strict AI alignment, if we had a system that wanted to help us and it was very good at preference learning such that it could use all human artifacts in the world like books, movies and other things. It can also study your behavior and also have conversations with us. It could leverage all data points in the world for building a deep and rich understanding of individual human preference hierarchies, and then also it could extrapolate broad preference facts about species wide general considerations. If that project were to succeed, then within those meta preferences and that preference hierarchy exists the kinds of normative ethical systems that machine ethics is trying to pay lip service to or to be sensitive towards or to imbue in machine systems.

From my perspective, if that kind of narrative that I just gave is true or valid, then that would be sort of a complete value alignment, and so far as it would create beneficial machine systems. But in order to have that kind of normative decision making and sensibilities in machine systems such that they fully understand and are sensitive to the ethical ramifications of certain decision makings, that requires higher order logic and the ability to generate concepts and to interrelate them and to shift them around and use them in the kinds of ways that human beings do, which we’re far short of.

Wendell Wallach: So that’s where the convergence is. We’re far short of it. So I have no problem with the description you made. The only thing I noted is, at the beginning you said, if we had, and for me, in order to have, you will have to go through these stages of development that we have been alluding to as machine ethics. Now, how much of that will be able to utilize tools that come out of artificial intelligence that we had not been able to imagine in the early days of machine ethics? I have no idea. There’s so many uncertainties on how that pathway is going to unfold. There’re uncertainties about what order the breakthroughs will take place, how the breakthroughs will interact with other breakthroughs and technology more broadly, whether there will be public reactions to autonomous systems along the way that slow down the course of development or even stop certain areas of research.

So I don’t know how this is all going to unfold. I do see within the AI community, there is kind of a leap of faith to a presumption of breaths of capacity that when I look at it, I still look at, well, how do we get between here and there. When I look at getting between here and there, I see that you’re going to have to solve some of these lower level problems that got described more in the machine ethics world than have initially been seen by the value alignment approaches. That said, now that we’re getting researchers actually trying to look at implementing value alignment, I think they’re coming to appreciate that these lower level problems are there. We can’t presume high level preference parsing by machines without them going through developmental stages in relationship to understanding what a preference is, what a norm is, how they get applied within different contexts.

My sense has been that with both machine ethics and value alignment, we’ve sort of got the cart in front of the horse. So I’m waiting to see some great implementation breakthroughs, I just haven’t seen them. Most of the time, when I encounter researchers who say they’re taking seriously, I see they’re tripping over relatively low level implementations. The difficulty is here, and all of this is converging. What AI alignment was initially and what it’s becoming now I think are quite different. I think in the very early days, it really was presumptions that you would have these higher order intelligences and then how were you going to align them. Now, as AI alignment, people look at the value issues as they intersect with present day AI agendas. I realize that you can’t make the presumptions about the higher order systems without going through developmental steps to get there.

So, in that sense, I think whether it’s AI alignment or machine ethics, the one will absorb the lessons of the other. Both will utilize advances that happen on both fronts. All I’m trying to underscore here is there are computer engineers and roboticist and philosophers who reflected on issues that perhaps the value alignment people are learning something from. I, in the end, don’t care about machine ethics or value alignment per se, I just care about people talking with each other and learning what they can from each other and moving away from a kind of arrogance that I sometimes see happen on both sides of the fence that one says to the other you do not understand. The good news and one thing that I was very happy about in terms of what we did in these three workshops that I was PI on with the help of the Future of Life Institute was, I think we sort of broke open the door for transdisciplinary dialogue.

Now, true, This was just one workshop. Now, we have gone from a time where the first Future of Life Institute gathering of Puerto Rico, the ethicists in the room, Jim Moore and I were backbenchers, to a time where we have countless conferences that are basically transdisciplinary conferences where people from many fields of research are now beginning to listen to each of them. The serious folks in the technology and ethics really have recognized the richness of ethical decision making in real contexts. Therefore, I think they can point that out. Technologists sometimes like to say, “Well, you ethicist, what do you have to say because you can’t tell us what’s right and wrong anyway?” Maybe that isn’t what ethics is all about, about dictating what’s right and wrong. Maybe ethics is more about how do we navigate the uncertainties of life, and what kinds of intelligence need to be brought to bear to navigate the uncertainties of life with a degree of sensitivity, depth, awareness, and appreciation for the multilayered kinds of intelligences that come into play.

Lucas Perry: In the context of this uncertainty about machine ethics and about AI alignment and however much or little convergence there might be, let’s talk about how all of this leads up into AI governance now. You touched on a lot of your machine ethics work. What made you pivot into AI governance, and where is that taking you today?

Wendell Wallach: After completing moral machines, I started to think about the fact that very few people had a deep and multidisciplinary understanding of the broad array of ethical and societal impacts posed by emerging technologies. I decided to write a primer on that, focusing on what could go wrong and how we might diffuse ethical challenges and undesirable societal impacts. That was finally published under the title A Dangerous Master: How to Keep Technology from Slipping Beyond our Control. The first part of that was really a primer on the various fields of science from synthetic biology to geoengineering, what the benefits were, what could go wrong. But then the book was very much about introducing people to various themes that arise, managing complex, adaptive systems, resilience, engineering, transcending limits, a whole flock of themes that have become part of language of discussing emerging technologies but weren’t necessarily known to a broader public.

Even for those of us who are specialists in one area of research such as biotech, we have had very little understanding of AI or geoengineering or some of the other fields. So I felt there was a need for a primer. Then the final chapter for the primer, I turned to how some of these challenges might be addressed through governance and oversight. Simultaneously, while I was working on that book, Gary Marchant and I, Gary Marchant is the director of the Center for Law and Innovation at the Sandra Day O’Connor School of Law at Arizona State University. Gary has been a specialist in the law and governance of emerging technologies. He and I, in our interactions lamented the fact that it was very difficult for any form of governance of these technologies. It was something called the pacing problem. The pacing problem refers to the fact that scientific discovery and technological innovation is far outpacing our ability to put in place appropriate ethical legal oversight, and that converges with another dilemma that has bedeviled people in technology governance for decades, going back to 1980.

David Collingridge wrote a book where he outlined a problem that is now known as the Collingridge Dilemma. Basically, Collingridge said that while it was easiest to regulate a technology early in its style development, early in its development, we had a little idea of what its societal impact would be. By the time we did understand what the challenges from the societal impact were, the technology would be so deeply entrenched in our society that it would be very difficult to change its trajectory. So we see that today with social media. Social media was totally entrenched in our society before we realized how it could be manipulated in ways that would undermine democracy. Now we’re having a devil of a time of figuring out what we could do.

So Gary and I, who had been talking about these kinds of problems for years, we realized that we were constantly lamenting the challenge, but we altered the conversation one day over a cup of coffee. We said, “Well, if we had our druthers, if we have some degree of influence, what would we propose?” We came up with a model that we referred to as governance coordinating committees. Our idea was that you would put in place a kind of issues manager that would try and guide the development of a field, but first of all, it would just monitor development, convene forums between the many stakeholders, map issues and gaps, see if anyone was addressing those issues and gaps or where their best practices had come to the floor. If these issues were not being addressed, then how could you address them, looking at a broad array of mechanisms. By a broad array of mechanisms, we meant you start with feasible technological solutions, you then look at what can be managed through corporate self-governance, and if you couldn’t find anything in either of those areas, then you turn to what is sometimes called soft law.

Soft law is laboratory practices and procedures, standards, codes of conduct, insurance policy, a whole plethora of mechanisms that fall short of laws and regulatory oversight. The value of soft law is that soft law can be proposed easily, and you can throw it out if technological advances mean it’s no longer necessary. So it’s very agile, it’s very adaptive. Really anyone can propose the news off law mechanism. But that contributes to one of the downsides, which is you can have competing soft law, but the other downside is perhaps even more important is that you seldom have a means of enforcement if there are violations of soft law. So, on some areas you deem need enforcement, and that’s why hard law and regulatory institutions become important.

So Gary and I proposed this model. Every time we ever talked about it, people would say, “Boy, that’s a great idea. Somebody should do that.” I was going to international forums, such as going to the World Economic meetings in Davos, where I’d be asked to be a fire-starter on all kinds of subject areas by safety and food security and the law of the ocean. In a few minutes, I would quickly outline this model as a way of getting people to think much more richly about ways to manage technological development and not just immediately go to laws and regulatory bodies. All of this convinced me that this model was very valuable, but it wasn’t being taken up. All of that led to this first International Congress for the Governance of Artificial Intelligence, which will be convened in Prague on April 16 to 18. I do invite those of you listening to this podcast who are interested in the international governance of AI or really agile governance for technology more broadly to join us at that gathering.

Lucas Perry: Can you specify the extent to which you think that soft law, international norms will shape hard law policy?

Wendell Wallach: I don’t think any of this is that easy at the moment because when I started working on this project and working toward the Congress, there was almost no one in this space. Suddenly, we have a whole flock of organizations that have jumped into it. We have more than 53 lists of principles for artificial intelligence and all kinds of specifications of laws coming along like GDPR, and the EU will actually be coming out very soon with a whole other list of proposed regulations for the development of autonomous systems. So we are now in an explosion of groups, each of which in one form or another is proposing both laws and soft law mechanisms. I think that means we are even more in need of something like a governance coordinating committee. What I mean is loose coordination and cooperation, but at least putting some mechanism in place for that.

Some of the groups that have come to the floor are like the OECD, which actually represents a broad array of the nations, but not all of them. The Chinese were not party to the development of the OECD principles. The Chinese, for example, have somewhat different principles and laws that are most attractive in the west. My point is that we have an awful lot of groups, some of which would like to have a significant leadership role or are dominating role, and we’ll have to see to what extent they cooperate with each other or whether we finally have a cacophony of competing soft law recommendations. But I think even if there’s a competition at the UN perhaps with a new mechanism that we create or through each of these bodies like the OECD and IAAA individually, best practices will come to the fore over time and they will become the soft law guidelines. Now, which of those soft guidelines need to make hard law? That may vary from nation to nation.

Lucas Perry: The agility here is in part imbued by a large amount of soft laws, which will then clarify best practices?

Wendell Wallach: Well, I think like anything else, just like the development of artificial intelligence. There’s all kinds of experimentation going on, all kinds of soft law frameworks, principles which have to be developed into policy and soft law frameworks going on. It will vary from nation to nation. We’ll get an insight over time about which practices really work and which haven’t worked. Hopefully, with some degree of coordination, we can underscore the best practices, we can monitor the development of the field in a way where we can underscore where the issues that still need to be addressed. We may have forums to work out differences. There may never be a full consensus and there may not need to be a full consensus considering much of the soft law will be implemented on a national or regional view like front. Only some of it will need to be top down in the sense that it’s international.

Lucas Perry: Can you clarify the set of things or legal instruments which consist of soft law and then the side of things which make up a hard law?

Wendell Wallach: Well, hard law is always things that have become governmentally instituted. So the laws and regulatory agencies that we have in America, for example, or you have the same within Europe, but you have different approaches to hard law. The Europeans are more willing to put in pretty rigorous hard law frameworks, and they believe that if we codify what we don’t want, that will force developers to come up with new creative experimental pathways that accommodate our values and goals. In America, were reticent to codify things into hard law because we think that will squelch innovation. So those are different approaches. But below hard law, in terms of soft law, you really do have these vast array of different mechanisms. So I mentioned international standards, some of those are technical. We see a lot of technical standards come in out of the IEEE and the ISO. The IEEE, for example, has jumped into the governance of autonomous systems in a way where it wants to go beyond what can be elucidated technically to talk more about what kinds of values we’re putting in place and what the actual implementation of those values would be. So that’s soft law.

Insurance policies sometimes dictate what you can and cannot do. So that soft law. We have laboratory practices and procedures. What’s safe to do in a laboratory and what isn’t? That’s soft law. We have new approaches to implementing values within technical systems, what is sometimes referred to as value-added design. That’s kind of a form of soft law. There are innumerable frameworks that we can come up with and we can create new ones if we need to to help delineate what is acceptable and what isn’t acceptable. But again, that delineation may or may not be enforceable. Some enforcement is, if you don’t do what the insurance policy has demanded of you, you lose your insurance policy, and that’s a form of enforceability.

You can lose membership in various organizations. Soft law gets into great detail in terms of acceptable use of humans and animals in research. But at least that’s a soft law that has, within the United States and Europe and elsewhere, some ability to prosecute people who violate the rights of individuals, who harm animals in a way that is not acceptable in the course of doing the research. So what are we trying to achieve by convening a first International Congress for the Governance of Artificial Intelligence? First of all, our hope is that we will get a broad array of stakeholders present. So, far, nearly all the governance initiatives are circumspect in terms of who’s there and who is not there. We are making special efforts to ensure that we have a robust representation from the Chinese. We’re going to make sure that we have robust representation from those from underserved nations and communities who are likely to be very effected by AI, but not necessarily we’ll know a great deal about it. So having a broad array of stakeholders is the number one goal of what we are doing.

Secondly, between here and the Congress, we’re convening six experts workshops. What we intend to do with these expert workshops is bring together a dozen or more of those individuals who have already been thinking very deeply about the kinds of governance mechanisms that we need. Do understand that I’m using the word governance, not government. Government usually just entails hard law and bureaucracies. By governance, we mean bringing in many other solutions to what we call regulatory or oversight problems. So we’re hopeful that we’ll get experts not only in AI governance, but also in thinking about agile governance more broadly that we will have them come to these small expert workshops we’re putting together, and at those expert workshops, we hope to elucidate what are the most promising mechanisms for the international governance of the AI. If they can elucidate those mechanisms, they will then be brought before the Congress. At the Congress, we’ll have further discussions and a Richmond around some of those mechanisms, and then by the end of the Congress, we will have boats to see if there’s an overwhelming consensus of those present to move forward on some of these initiatives.

Perhaps, something like what I had called the governance coordinating committee might be one of those mechanisms. I happen to have also been an advisor to the UN secretary General’s higher level panel on digital cooperation, and they drew upon some of my research and combined that with others and came up with one of their recommendations, so they recommended something that is sometimes referred to a network of networks. Very similar to what I’ve been calling a governance coordinating committee. In the end, I don’t care what mechanisms we start to put in place, just that we begin to take first steps toward putting in place that will be seen as trustworthy. If we can’t do that, then why bother. At the end of the Congress, we’ll have these votes. Hopefully that will bring some momentum behind further action to move expeditiously toward putting some of these mechanisms in place.

Lucas Perry: Can you contextualize this International Congress for the Governance of AI within the broader AI governance landscape? What are the other efforts going on, and how does this fit in with all of them?

Wendell Wallach: Well, there are many different efforts underway. The EU has its efforts, the IEEE has its effort. The World Economic Forum convenes people to talk about some of these issues. You’ll have some of this come up in the Partnership in AI, you have OECD. There are conversations going on in the UN. You the higher level panels recommendations. So they have now become a vast plethora of different groups that have jumped into it. Our point is that, so far, none of these groups include all the stakeholders. So the Congress is an attempt to bring all of these groups together and ensure that other stakeholders have a place at the table. That would be the main difference.

We want to weave the groups together, but we are not trying to put in place some new authority or someone who has authority over the individual groups. We’re just trying to make sure that we’re looking at the development of AI comprehensively, that we’re talking with each other, that we have forums to talk with each other, that issues aren’t going unaddressed, and then if somebody truly has come forward with best practices and procedures, that those are made available to everyone else in the world or at least underscored for others in the world as promising pathways to go down.

Lucas Perry: Can you elaborate on how these efforts might fail to develop trust or how they might fail to bring about coordination on the issues? Is it always in the incentive of a country to share best practices around AI if that increases the capacity of other countries to catch up?

Wendell Wallach: We always have this problem of competition and cooperation. Where’s competition going to take place? How much cooperation will there actually be? It’s no mystery to anyone in the world that decisions are being made as we speak about whether or not we’re going to move towards wider cooperation within the international world or whether we have movements where we are going to be looking at a war of civilization or at least a competition between civilizations. I happen to believe there’s so many problems within emerging technologies that if we don’t have some degree of coordination, we’re all damned and that that should prevail in global climate change and in other areas, but whether we’ll actually be able to pull that off has to do with decisions going on in individual countries. So, at the moment, we’re particularly seeing that tension between China and the US. If the trade work can be diffused, then maybe we can back off from that tension a little bit, but at the moment, everything’s up for grabs.

That being said, when everything’s up for grabs, my belief is you do what you can to facilitate the values that you think need to be forwarded, and therefore I’m pushing us toward recognizing the importance of a degree of cooperation without pretending that we aren’t going to compete with each other. Competition’s not bad. Competition, as we all know, furthers innovation helps disrupt technologies that are inefficient and replace them with more efficient ways of moving forward. I’m all for competition, but I would like to see it in a broader framework where there is at least a degree of cooperation on AI ethics and international governmental cooperation.

Lucas Perry: The path forward seems to have something to do with really reifying the importance of cooperation and how that makes us all better off to some extent, not pretending like there’s going to be full 100% cooperation, but cooperation where it’s needed such that we don’t begin defecting on each other in ways that are mutually bad and incompatible.

Wendell Wallach: That claim is central to the whole FLI approach.

Lucas Perry: Yeah. So, if we talk about AI in particular, there’s this issue of lethal autonomous weapons. There’s an issue of, as you mentioned, the spread of disinformation, the way in which AI systems and machine learning can be used more and more to lie and to spread subversive or malicious information campaigns. There’s also the degree to which algorithms will or will not be contributing to discrimination. So these are all like short term things that are governance issues for us to work on today.

Wendell Wallach: I think the longer term trajectory is that AI systems are giving increasing power to those who want to manipulate human behavior either from marketing or political purposes, and they’re manipulating the behavior by studying human behavior and playing to our vulnerabilities. So humans are very much becoming machines in this AI commercial political juggernaut.

Lucas Perry: Sure. So human beings have our own psychological bugs and exploits, and massive machine learning can find those bugs and exploits and exploit them in us.

Wendell Wallach: And in real time. I mean, with the collection of sensors and facial recognition software and emotion recognition software over 5G with a large database of our past preferences and behaviors, we can be bombarded with signals to manipulate our behavior on very low levels and areas where we are known to be vulnerable.

Lucas Perry: So the question is to the extent to which and the strategies for which we can use within the context of these national and global AI governance efforts to mitigate these risks.

Wendell Wallach: To mitigate these risks, to make sure that we have meaningful public education, meaning I would say from grammar school up, digital literacy so that individuals can recognize when they’re being scammed, when they’re being lied to. I mean, we’ll never be perfect at that, but at least have ones antenna out for that and the degree to which we perhaps need to have some self recognition that if we’re going to not be just manipulable. But we’ll truly cultivate the capacity to recognize when there are internal and external pressures upon us and diffuse those pressures so we can look at new, more creative, individualized responses to the challenge at hand.

Lucas Perry: I think that that point about elementary to high school education is really interesting and important. I don’t know what it’s like today. I guess they’re about the same as what I experienced. They just seemed completely incompatible with the way the technology is going and dis-employment and other things in terms of the way that they teach and what they teach.

Wendell Wallach: Well, it’s not happening within the school systems. What I don’t fully understand is how savvy young people are within their own youth culture, whether they’re recognizing when they’re being manipulated or not, whether that’s part of that culture. I mean part of my culture, and God knows I’m getting on in years now, but it goes back to questions of phoniness and pretense and so forth. So we did have our youth culture that was very sensitive to that. But that wasn’t part of what our educational institutions were engaged in.

The difference now is that we’ll have to be both within the youth culture, but also we would need to be actually teaching digital literacy. So, for an example, I’m encountering a as scam a week, I would say right now through the telephone or through email. Some new way that somebody has figured out to try and rip off some money from me. I can’t believe how many new approaches are coming up. It just flags that this form of corruption requires remarkable degree of both sensitivity but a degree of digital knowledge so that you can recognize when you need to at least check out whether this is real or a scan before you give sensitive information to others.

Lucas Perry: The saving grace, I think for, gen Z and millennial people is that… I mean, I don’t know what the percentages are, but more than before, many of us have basically grown up on the internet.

Wendell Wallach: So they have a degree of digital literacy.

Lucas Perry: But it’s not codified by an institution like the schooling system, but changing the schooling system to the technological predictions of academics. I don’t know how much hope I have. It seems like it’s a really slow process to change anything about education. It seems like it almost has to be done outside of public education

Wendell Wallach: That may be what we mean by governance now is what can be done within the existing institutions and what has to find means of being addressed outside of the existing institutions, and is it happening or isn’t it happening? If youth culture in its evolving forms gives 90% of digital literacy to young people, fine, but what about those people who are not within the networks of getting that education, and what about the other 10%? How does that take place? I think that’s the kind of creativity and oversight we need is just monitoring what’s going on, what’s happening, what’s not happening. Some areas may lead to actual governmental needs or interventions. So let’s take the technological unemployment issue. I’ve been thinking a lot about that disruption in new ways. One question I have is whether it can be slowed down. An example for me for a slow down would be if we found ways of not rewarding corporations for introducing technologies that bring about minimal efficiencies but are more costly to the society than the efficiencies that they introduce for their own productivity gains.

So, if it’s a small efficiency, but the corporation fires 10,000 people and just 10,000 people are now on the door, I’m not sure whether we should be rewarding corporations for that. On the other hand, I’m not quite sure what kind of political economy you could put in place so you didn’t reward corporations for that. Let’s just say that you have automatic long haul trucking. In the United States, we have 1.7 million long haul truck drivers. It’s one of the top jobs in the country. First of all, long haul trucking can probably be replaced more quickly than we’ll have self driving trucks in the cities because of some of the technical issues encountered in cities and on country roads and so forth. So you could have a long haul truck that just went from on-ramp to off ramp and then have human drivers who take over the truck for the last few miles to take it to the shipping depot.

But if we’ve replaced long haul truckers in the United States over a 10 year period, that would mean putting 14,000 truck drivers out of work every month. That means you have to create 14,000 jobs a month that are appropriate for long haul truck drivers. At the same time, as you’re creating jobs for new people entering the workforce and for others whose jobs are disappearing because of automation, it’s not going to happen. Given the culture in the United States, my melodramatic example is some long haul truckers may just decide to take the semis closed down interstate highways and sit in their cap and say to the government, “Bring it on.” We are moving into that kind of social instability. So, on one hand, if getting rid of the human drivers doesn’t bring massive efficiencies, it could very easily bring social instability and large societal costs. So perhaps we don’t want to encourage that. But we need to look at it in greater depth to understand what the benefits and costs are.

We often overplay the benefits, and we under-represent the downsides and the costs. You could see a form of tax on corporations relative to how many workers they laid off and how many jobs they created. It could be a sliding tax. For corporations reducing its workforce dramatically, then it gets a higher tax on its profit than one that’s actually increasing its workforce. That would be a form of maybe how you’re funding UBI. In UBI, I would like to see something that I’ve referred to as UBI plus plus plus. I mean there’ve been various UBI pluses. But in my thought was that you’re being given that basic income for performing a service for the society. In other words, performing a service for the society is your job. There may not be anybody overseeing what service you are providing or you might be able to decide yourself what that service would be.

Maybe somebody was an aspiring actor would decide that they were going to put together an acting group and take Shakespeare into the school system, that that was their service to the society. Others may decide they don’t know how to do a service to the society, but they want to go back to school, so perhaps they’re preparing for a new job or a new contribution, and perhaps other people will really need a job and we’ll have to create high touch jobs such as those that you have in Japan for them. But the point is UBI is paying you for a job. The job you’re doing is providing a service to the society, and that service is actually improving the overall society. So, if you had thousands of creative people taking educational programs into schools, perhaps you’re improving overall education and therefore the smarts of the next generation.

Most of this is not international governance, but where it does impinge upon international considerations is if we do have massive unemployment. It’s going to be poorer nations that are going to be truly set back. I’ve been planning out in international circles that we now have the Sustainable Development Goals. Well, just technological unemployment alone could undermine the realization of the Sustainable Development Goals.

Lucas Perry: So that seems like a really big scary issue.

Wendell Wallach: It’s going to vary from country to country. I mean, the fascinating thing is how different these national governments will be. So some of the countries in Africa are leap frogging technology. They’re moving forward. They’re building smart cities. They aren’t going through our development. But other countries don’t even have functioning governments or the governments are highly autocratic. When you look at the technology available for surveillance systems now, I mean we’re very likely to see some governments in the world that look like horrible forms of dictatorship gulags, at the same time as there’ll be some countries where human rights are deeply entrenched, and the oversight of the technologies will be such that they will not be overly repressive on individual behavior.

Lucas Perry: Yeah. Hopefully all of these global governance mechanisms that are being developed will bring to light all of these issues and then effectively work on them. One issue which is related, and I’m not sure how fits in here or it fits in with your thinking, is specifically the messaging and thought around the governance related to AGI and superintelligence. Do you have any thinking here about how any of this feeds into that or your thoughts about that?

Wendell Wallach: I think that the difficulty is we’re still in a realm where when and what AGI or superintelligence will appear and what it will look like. It’s still so highly speculative. So, at this stage of the game, I don’t think that AGI is really a governmental issue beyond the question of whether government should be funding some of the research. There may also be a role for governments in monitoring when we’re crossing thresholds that open the door for AGI. But I’m not so concerned about that because I think there’s a pretty robust community that’s doing that already that’s not governmental, and perhaps we don’t need the government too involved. But the point here is, if we can put in place robust mechanisms for the international governance of AI, then potentially those mechanisms either make recommendations that perhaps slow down the adoption of technologies that could be dangerous or enhance the ethics and the sensitivity and the development of the technologies. If and when we are about to cross thresholds that open real dangers or serious benefits, that we have the mechanisms in place to help regulate the unfold into that trajectory.

But that, of course, has to be wishful thinking at this point. We’re taking baby steps at this stage of the game. Those baby steps are going to be building on the activities at FLI and OpenAI and other groups that are already engaged in. My way of approaching it is, and it’s not just with AGI, it’s also in relationship to biotech, is just a flag that are speculative dangers out there, and we are making decisions today about what pathways we, humanity as a whole, want to navigate. So, oftentimes in my presentations, I will have a slide up, and that slide is two robots kneeling over the corpse of a human. When I put that slide up, I say we may even be dealing with the melodramatic possibility that we are inventing the human species as we have known it out of existence.

So that’s my way of flagging that that’s the concern, but not trying to pretend that that’s one that governments should or can address at this point more that we are inflection point where we should and can put in place values and mechanisms to try and ensure that the trajectory of the emerging technologies is human-centered, is planet-centered, is about human flourishing.

Lucas Perry: I think that the worry of the information that is implicit to that is that if there are two AIs embodied as robots or whatever, standing over a human corpse to represent them dominating or transcending the human species. What is implicit to that is that they have more power than us because you require more power to be able to do something like that. To have more power than the human species is something governments would maybe be interested in that would be something maybe we wouldn’t want to message about.

Wendell Wallach: I mean, it’s the problem with lethal autonomous weapons. Now, I think most of the world has come to understand that lethal autonomous weapons is a bad idea, but that’s not stopping governments from pursuing them or the security establishment within government saying that it’s necessary that we go down this road. Therefore, we don’t get an international ban or treaty. The messaging with governments is complicated. I’m using the messaging only to stress what I think we should be doing in the near term.

Lucas Perry: Yeah, I think that that’s a good idea and the correct approach. So, if everything goes right in terms of this process of AI governance, then we’re able to properly manage the development of new AI technology, what is your hope here? What are optimistic visions of the future, given successful AI governance?

Wendell Wallach: I’m a little bit different than most people on this. I’m not so much caught up in visions of the future based on this technology or that technology. My focus is more that we have a conscious active decision making process in the present where people get to put in place the values and instruments they need to have a degree of control over the overall development of emerging technologies. So, yes, of course I would like to see us address global climate change. I would like us to adapt AI for all. I would like to see all kinds of things take place. But more than anything, I’m acutely aware of what a significant inflection point this is in human history, and that we’re having the pass through a very difficult and perhaps in relatively narrow doorway in order ensure human flourishing for the next couple of hundred years.

I mean, I understand that I’m a little older than most of the people involved in this process, so I’m not going to be on the stage for that much longer barring radical life extension taking place in the next 20 years. So, unlike many people who are working on positive technology visions for the future, I’m less concerned with the future and more concerned with how, in the present, we nudge technology onto our positive course. So my investment is more that we ensure that humanity not only have a chance, but a chance to truly prevail.

Lucas Perry: Beautiful. So you’re now discussing about how you’re essentially focused on what we can do immediately. There’s the extent to which AI alignment and machine ethics or whatever are trying to imbue an understanding of human preference hierarchies in machine systems and to develop ethical sensibilities and sensitivities. I wonder what the role is for, first of all, embodied compassion and loving kindness in persons as models for AI systems and then embodied loving kindness and compassion and pure altruism in machine systems as a form of alignment with idealized human preference hierarchies and ethical sensibilities.

Wendell Wallach: In addition of this work I’m doing on the governance of emerging technologies, I’m also writing a book right now. The book has a working title, which is Descartes Meets Buddha: Enlightenment for the Information Age.

Lucas Perry: I didn’t know that. So that’s great.

Wendell Wallach: So this fits in with your question very broadly. I’m both looking at if the enlightenment ethos, which has directed humanities development over the last few hundred years is imploding under the weight of its own success, then what ethos do we put in place that gives humanity a direction for flourish and over the next few hundred years? I think central to creating that new ethos is to have a new understanding of what it means to be human. But that new understanding isn’t something totally new. It needs to have some convergence with what’s been perennial wisdom to be meaningful. But the fact is when we ask these questions, how are we similar to and how do we truly differ from the artificial forms of intelligence that we’re creating? Or what will it mean to be human as we evolved through the impact of emerging technologies, whether that’s life extension or uploading or bioengineering?

There still is this fundamental question about what grounds, what it means to be human. In other words, what’s not just up for grabs or up for engineering. To that, I bring in my own reflections after having meditated for the last 50 years on my own insights shall we say and how that converges with what we’ve learned about human functioning, human decision making and human ethics through the cognitive sciences over the last decade or two. Out of that, I’ve come up with a new model that I referred to as cyber souls, meaning that as sciences illuminating the computational and biochemical mechanisms that give rise to human capabilities, we have often lost sight of the way in which evolution also forged us into integrated beings, integrated within ourselves and searching for an adapted integration to the environment and the other entities that share in that environment.

And it’s this need for integration and relationship, which is fundamental in ethics, but also in decision making. There’s the second part of this, which is this new fascination with moral psychology and the recognition that reason alone may not be enough for good decision making. And that if we have an ethics that doesn’t accommodate people’s moral psychology, then reason alone isn’t going to be persuasive for people, they have to be moved by it. So I think this leads us to perhaps a new understanding of what’s the role of psychological states in our decision making, what information is carried by different psychological states, and how does that information help direct us toward making good and bad decisions. So I call that a silent ethic. There are certain mental states, which historically have at least indicated for people that they’re in the right place at the right time, in the right way.

Oftentimes, these states, whether they’re called flow or oneness or creativity, they’re being given some spiritual overlay and people look directly at how to achieve these states. But that may be a misunderstanding of the role of mental states. Mental States are giving us information. As we factor that information into our choices and actions, those mental states fall away, and the byproduct are these so-called spiritual or transcendent states, and often they have characteristics where thought and thinking comes to a rest. So I call this the silent ethic, taking the actions, making the choices that allow our thoughts to come to rest. When our thoughts are coming to rest, we’re usually in relationships within ourself and our environments that you can think of as embodied presence or perhaps even the foundations for virtue. So my own sense is we may be moving toward a new or revived virtue ethics. Part of what I’m trying to express in this new book is what I think is foundational to the flourishing of that new virtue ethics.

Lucas Perry: That’s really interesting. I bring this up and asking because I’ve been interested in the role of idealization, ethically, morally and emotionally in people and reaching towards whatever is possible in terms of human psychological enlightenment and how that may exist as certain benchmarks or reference frames in terms of value learning.

Wendell Wallach: Well, it is a counter pose to the notion that machines are going to have this kind of embodied understanding. I’m highly skeptical that we will get machines in the next hundred years that come in close to this kind of embodied understanding. I’m not skeptical that we could have on new kind of revival movement among humans where we create a new class of moral exemplars, which seems to be the exact opposite of what we’re doing at the moment.

Lucas Perry: Yeah. If we can get the AI systems and create abundance and reduce existential risk of bunch and have a long period of reflection, perhaps there will be this space for reaching for the limits of human idealization and enlightenment.

Wendell Wallach: It’s part of what the whole question is going on, for us, philosophy types, to what extent is this all about machine superintelligence and to what extent are we using the conversation about superintelligence as an imperfect mirror to think more deeply about the ways we’re similar to in dissimilar from the AI systems we’re creating or have a potential to create.

Lucas Perry: All right. So, with that, thank you very much for your time.

 If you enjoyed this podcast, please subscribe. Give it a like or share it on your preferred social media platform. We’ll be back again soon with another episode in the AI alignment series.

End of recorded material

AI Alignment Podcast: Human Compatible: Artificial Intelligence and the Problem of Control with Stuart Russell

Stuart Russell is one of AI’s true pioneers and has been at the forefront of the field for decades. His expertise and forward thinking have culminated in his newest work, Human Compatible: Artificial Intelligence and the Problem of Control. The book is a cornerstone piece, alongside Superintelligence and Life 3.0, that articulates the civilization-scale problem we face of aligning machine intelligence with human goals and values. Not only is this a further articulation and development of the AI alignment problem, but Stuart also proposes a novel solution which bring us to a better understanding of what it will take to create beneficial machine intelligence.

 Topics discussed in this episode include:

  • Stuart’s intentions in writing the book
  • The history of intellectual thought leading up to the control problem
  • The problem of control
  • Why tool AI won’t work
  • Messages for different audiences
  • Stuart’s proposed solution to the control problem

Key points from Stuart: 

  •  “I think it was around 2013 that it really struck me that in fact we’d been thinking about AI the wrong way all together. The way we had set up the whole field was basically kind of a copy of human intelligence in that a human is intelligent, if their actions achieve their goals. And so a machine should be intelligent if its actions achieve its goals. And then of course we have to supply the goals in the form of reward functions or cost functions or logical goals statements. And that works up to a point. It works when machines are stupid. And if you provide the wrong objective, then you can reset them and fix the objective and hope that this time what the machine does is actually beneficial to you. But if machines are more intelligent than humans, then giving them the wrong objective would basically be setting up a kind of a chess match between humanity and a machine that has an objective that’s across purposes with our own. And we wouldn’t win that chess match.”
  • “So when a human gives an objective to another human, it’s perfectly clear that that’s not the sole life mission. So you ask someone to fetch the coffee, that doesn’t mean fetch the coffee at all costs. It just means on the whole, I’d rather have coffee than not, but you know, don’t kill anyone to get the coffee. Don’t empty out my bank account to get the coffee. Don’t trudge 300 miles across the desert to get the coffee. In the standard model of AI, the machine doesn’t understand any of that. It just takes the objective and that’s its sole purpose in life. The more general model would be that the machine understands that the human has internally some overall preference structure of which this particular objective fetch the coffee or take me to the airport is just a little local manifestation. And machine’s purpose should be to help the human realize in the best possible way their overall preference structure. If at the moment that happens to include getting a cup of coffee, that’s great or taking him to the airport. But it’s always in the background of this much larger preference structure that the machine knows and it doesn’t fully understand. One way of thinking about is to say that the standard model of AI assumes that the machine has perfect knowledge of the objective and the model I’m proposing assumes that the model has imperfect knowledge of the objective or partial knowledge of the objective. So it’s a strictly more general case.”
  • “The objective is to reorient the field of AI so that in future we build systems using an approach that doesn’t present the same risk as the standard model… That’s the message I think for the AI community is the first phase our existence maybe should come to an end and we need to move on to this other way of doing things. Because it’s the only way that works as machines become more intelligent. We can’t afford to stick with the standard model because as I said, systems with the wrong objective could have arbitrarily bad consequences.”

 

Important timestamps: 

0:00 Intro

2:10 Intentions and background on the book

4:30 Human intellectual tradition leading up to the problem of control

7:41 Summary of the structure of the book

8:28 The issue with the current formulation of building intelligent machine systems

10:57 Beginnings of a solution

12:54 Might tool AI be of any help here?

16:30 Core message of the book

20:36 How the book is useful for different audiences

26:30 Inferring the preferences of irrational agents

36:30 Why does this all matter?

39:50 What is really at stake?

45:10 Risks and challenges on the path to beneficial AI

54:55 We should consider laws and regulations around AI

01:03:54 How is this book differentiated from those like it?

 

Works referenced:

Human Compatible: Artificial Intelligence and the Problem of Control

Superintelligence

Life 3.0

Occam’s razor is insufficient to infer the preferences of irrational agents

Synthesizing a human’s preferences into a utility function with Stuart Armstrong

 

We hope that you will continue to join in the conversations by following us or subscribing to our podcasts on Youtube, Spotify, SoundCloud, iTunes, Google Play, StitcheriHeartRadio, or your preferred podcast site/application. You can find all the AI Alignment Podcasts here.

You can listen to the podcast above or read the transcript below. 

Lucas: Hey everyone, welcome back to the AI Alignment Podcast. I’m Lucas Perry and today we’ll be speaking with Stuart Russell about his new book, Human Compatible: Artificial Intelligence and The Problem of Control. Daniel Kahneman says “This is the most important book I have read in quite some time. It lucidly explains how the coming age of artificial super intelligence threatens human control. Crucially, it also introduces a novel solution and a reason for hope.”

Yoshua Bengio says that “This beautifully written book addresses a fundamental challenge for humanity: increasingly intelligent machines that do what we ask, but not what we really intend. Essential reading if you care about our future.”

I found that this book helped clarify both intelligence and AI to me as well as the control problem born of the pursuit of machine intelligence. And as mentioned, Stuart offers a reconceptualization of what it means to build beneficial and intelligent machine systems. That provides a crucial place of pivoting and how we ought to be building intelligent machines systems.

Many of you will already be familiar with Stuart Russell. He is a professor of computer science and holder of the Smith-Zadeh chair in engineering at the University of California, Berkeley. He has served as the vice chair of the World Economic Forum’s Council on AI and Robotics and as an advisor to the United Nations on arms control. He is an Andrew Carnegie Fellow as well as a fellow of the Association for The Advancement of Artificial Intelligence, the Association for Computing Machinery and the American Association for the Advancement of Science.

He is the author with Peter Norvig of the definitive and universally acclaimed textbook on AI, Artificial Intelligence: A Modern Approach. And so without further ado, let’s get into our conversation with Stuart Russell.

Let’s start with a little bit of context around the book. Can you expand a little bit on your intentions and background for writing this book in terms of timing and inspiration?

Stuart: I’ve been doing AI since I was in high school and for most of that time the goal has been let’s try to make AI better because I think we’ll all agree AI is mostly not very good. When we wrote the first edition of the textbook, we decided to have a section called, What If We Do Succeed? Because it seemed to me that even though everyone was working on making AI equivalent to humans or better than humans, no one was thinking about what would happen if that turned out to be successful.

So that section in the first edition in 94 was a little equivocal, let’s say, you know, we could lose control or we could have a golden age and let’s try to be optimistic. And then by the third edition, which was 2010 the idea that we could lose control was fairly widespread, at least outside the AI communities. People worrying about existential risk like Steve Omohundro, Eliezer Yudkowsky and so on.

So we included those a little bit more of that viewpoint. I think it was around 2013 that it really struck me that in fact we’d been thinking about AI the wrong way all together. The way we had set up the whole field was basically kind of a copy of human intelligence in that a human is intelligent, if their actions achieve their goals. And so a machine should be intelligent if its actions achieve its goals. And then of course we have to supply the goals in the form of reward functions or cost functions or logical goals statements. And that works up to a point. It works when machines are stupid. And if you provide the wrong objective, then you can reset them and fix the objective and hope that this time what the machine does is actually beneficial to you. But if machines are more intelligent than humans, then giving them the wrong objective would basically be setting up a kind of a chess match between humanity and a machine that has an objective that’s across purposes with our own. And we wouldn’t win that chess match.

So I started thinking about how to solve that problem. And the book is a result of the first couple of years of thinking about how to do it.

Lucas: So you’ve given us a short and concise history of the field of AI alignment and the problem of getting AI systems to do what you want. One of the things that I found so great about your book was the history of evolution and concepts and ideas as they pertain to information theory, computer science, decision theory and rationality. Chapters one through three you sort of move sequentially through many of the most essential concepts that have brought us to this problem of human control over AI systems.

Stuart: I guess what I’m trying to show is how ingrained it is in intellectual thought going back a couple of thousand years. Even in the concept of evolution, this notion of fitness, you know we think of it as an objective that creatures are trying to satisfy. So in the 20th century you had a whole lot of disciplines, economics developed around the idea of maximizing utility or welfare or profit depending on which branch you look at. Control theory is about minimizing a cost function, so the cost function described some deviation from ideal behavior and then you build systems that minimize the cost. Operations research, which is dynamic programming and Markov decision processes is all about maximizing the sum of rewards. And statistics if you set it up in general, is about minimizing an expected loss function.

So all of these disciplines have the same bug if you like. It’s a natural way to set things up, but in the long run we’ll just see it as a bad cramped way of doing engineering. And what I’m proposing in the book actually is a way of thinking about it that’s much more in a binary rather than thinking about the machine and it’s objective.

You think about this coupled system with humans or you know, it could be any entity that wants a machine to do something good for it or another system to do something good for it. And then the system itself, which is supposed to do something good for the human or whatever else it is that wants something good to happen. So this kind of coupled system, don’t really see that in the intellectual tradition. Maybe one exception that I know of, which is the idea of principle agent games in economics. So a principal might be an employer and the agent might be the employee. And then the game is how does the employer get the employee to do something that the employer actually wants them to do, given that the employee, the agent has their own utility function and would rather be sitting home drinking beers and watching football on the telly.

How do you get them to show up at work and do all kinds of things they wouldn’t normally want to do? The simplest way is you pay them. But you know, there’s all kinds of other ideas about incentive schemes and status and then various kinds of sanctions if people don’t show up and so on. So the economists study that notion, which is a coupled system where one entity wants to benefit from the behavior of another.

So that’s probably the closest example that we have. And then maybe in ecology, look at symbiotic species or something like that. But there’s not very many examples that I’m aware of. In fact, maybe I can’t think of any, where the entity that’s supposedly in control, namely us, is less intelligent than the entity that it’s supposedly controlling, namely the machine.

Lucas: So providing some framing and context here for the listener, the first part of your book, chapters one through three explores the idea of intelligence in humans and in machines. There you give this historical development of ideas and I feel that this history you give of computer science and the AI alignment problem really helps to demystify both the person and evolution as a process and the background behind this problem.

Your second part of your book, chapters four through six discusses some of the problems arising from imbuing machines with intelligence. So this is a lot of the AI alignment problem considerations. And then the third part, chapter seven through ten suggests a new way to think about AI, to ensure that machines remain beneficial to humans forever.

You’ve begun stating this problem and readers can see in chapters one through three that this problem goes back a long time, right? The problem with computer science at its inception was that definition that you gave that a machine is intelligent in so far as it is able to achieve its objectives. In reaction to this, you’ve developed cooperative inverse reinforcement learning and inverse reinforcement learning, which is sort of part of the latter stages of this book where you’re arguing for new definition that is more conducive to alignment.

Stuart: Yeah. In the standard model as I call it in the book, the humans specifies the objective and plugs it into the machine. If for example, you get in your self driving car and it says, “Where do you want to go?” And you say, “Okay, take me to the airport.” For current algorithms as we understand them, understand built on this kind of model, that objective becomes the sole life purpose of the vehicle. It doesn’t necessarily understand that in fact that’s not your sole life purpose. If you suddenly get a call from the hospital saying, oh, you know, your child has just been run over and is in the emergency room. You may well not want to go to the airport. Or if you get into a traffic jam and you’ve already missed the last flight, then again you might not want to go to the airport.

So when a human gives an objective to another human, it’s perfectly clear that that’s not the sole life mission. So you ask someone to fetch the coffee, that doesn’t mean fetch the coffee at all costs. It just means on the whole, I’d rather have coffee than not, but you know, don’t kill anyone to get the coffee. Don’t empty out my bank account to get the coffee. Don’t trudge 300 miles across the desert to get the coffee.

In the standard model of AI, the machine doesn’t understand any of that. It just takes the objective and that’s its sole purpose in life. The more general model would be that the machine understands that the human has internally some overall preference structure of which this particular objective fetch the coffee or take me to the airport is just a little local manifestation. And machine’s purpose should be to help the human realize in the best possible way their overall preference structure.

If at the moment that happens to include getting a cup of coffee, that’s great or taking him to the airport. But it’s always in the background of this much larger preference structure that the machine knows and it doesn’t fully understand. One way of thinking about is to say that the standard model of AI assumes that the machine has perfect knowledge of the objective and the model I’m proposing assumes that the model has imperfect knowledge of the objective or partial knowledge of the objective. So it’s a strictly more general case.

When the machine has partial knowledge of the objective there’s whole lot of new things that come into play that simply don’t arise when the machine thinks it knows the objective. For example, if the machine knows the objective, it would never ask permission to do an action. It would never say, you know, is it okay if I do this because it believes that it’s already extracted all there is to know about human preferences in the form of this objective. And so whatever plan it formulates to achieve the objective must be the right thing to do.

Whereas a machine that knows that it doesn’t know the full objective could say, well, given what I know, this action looks okay, but I want to check with the boss before going ahead because it might be that this plan actually violate some part of the human preference structure that it doesn’t know about. So you get machines that ask permission, you get machines that, for example, allow themselves to be switched off because the machine knows that it might do something that will make the human unhappy. And if the human wants to avoid that and switches the machine off, that’s actually a good thing. Whereas a machine that has a fixed objective would never want to be switched off because that guarantees that it won’t achieve the objective.

So in the new approach you have a strictly more general repertoire of behaviors that the machine can exhibit. The idea of inverse reinforcement learning is this is the way for the machine to actually learn more about what the human preference structure is. By observing human behavior, which could be verbal behavior, like, could you fetch me a cup of coffee? That’s a fairly clear indicator about your preference structure, but it could also be that you know, you ask a human question and the human doesn’t reply. Maybe the human’s mad at you and is unhappy about the line of questioning that you’re pursuing.

 So human behavior means everything humans do and have done in the past. So everything we’ve ever written down, every movie we’ve made, every television broadcast contains information about human behavior and therefore about human preferences. Inverse reinforcement learning really means how do we take all that behavior and learn human preferences from it?

Lucas: What can you say about how tool AI as a possible path to AI alignment fits in this schema where we reject the standard model, as you call it, in favor of this new one?

Stuart: Tool AI is a notion, oddly enough, it doesn’t really occur within the field of AI. It’s a phrase that came from people who are thinking from the outside about possible risks from AI. And what it seems to mean is the idea that rather than buildings general purpose intelligence systems. If you are building AI systems designed for some specific purpose, then that’s sort of innocuous and doesn’t present any risks. And some people argue that in fact if you just have a large collection of these innocuous application specific AI systems, then there’s nothing to worry about.

My experience of tool AI is that when you build applications specific systems, you can kind of do it in two ways. One is you kind of hack it. In other words, you figure out how you would do this task and then you write a whole bunch of very, very special purpose code. So, for example, if you were doing handwriting recognition, you might think, oh, okay, well in order to find an ‘S’ I have to look for a line that’s curvy and I follow the line and it has to have three bends, it has to be arranged this way. And you know, you write a whole bunch of tests to check each characteristic of an ad that it has all these characteristics and it doesn’t have any loops and this, that and the other. And then you see okay, that’s an S.

And that’s actually not the way that people went about the problem of handwriting recognition. The way that they did it was to develop machine learning systems that could take images of characters that were labeled and then train a recognizer that could recognize new instances of characters. And in fact, Yann LeCun at AT&T was doing a system that was designed to recognize words and figures on checks. So very, very, very application specific, very tooley and order to do that he invented convolutional neural networks. Which is what we now call deep learning.

So, out of this very, very narrow piece of tool AI came this very, very general technique. Which has solved or largely solved object recognition, speech recognition, machine translation, and some people argue will produce general purpose AI. So I don’t think there’s any safety to be found in focusing on tool AI.

The second point