FLI Podcasts Archives - Future of Life Institute

Joscha Bach and Anthony Aguirre on Digital Physics and Moving Towards Beneficial Futures

March 31, 2021/in Featured, FLI Podcasts, FLI podcasts & transcripts /by Lucas Perry

Future of Life Institute · Joscha Bach and Anthony Aguirre on Digital Physics and Moving Towards Beneficial Futures

Understanding the universe through digital physics
How human consciousness operates and is structured
The path to aligned AGI and bottlenecks to beneficial futures
Incentive structures and collective coordination

You can find FLI’s three new policy focused job postings here

1:06:53 A future with one, several, or many AGI systems? How do we maintain appropriate incentive structures?

1:19:39 Non-duality and collective coordination

1:22:53 What difficulties are there for an idealist worldview that involves computation?

1:27:20 Which features of mind and consciousness are necessarily coupled and which aren’t?

1:36:40 Joscha’s final thoughts on AGI

Lucas Perry: Welcome to the Future of Life Institute Podcast. I’m Lucas Perry. Today’s episode is with Joscha Bach and Anthony Aguirre where we explore what truth is, how the universe ultimately may be computational, what purpose and meaning are relative to life’s drive towards complexity and order, collective coordination problems, incentive structures, and how these both affect civilizational longevity, and whether having a single, multiple, or many AGI systems existing simultaneously is more likely to lead to beneficial outcomes for humanity.

If you enjoy this podcast, you can subscribe on your preferred podcasting platform by searching for the Future of Life Institute Podcast or you can help out by leaving us a review on iTunes. These reviews are a big help for getting the podcast to more people.

Before we get into the episode, the Future of Life Institute has 3 new job postings for full-time equivalent remote policy focused positions. We’re looking for a Director of European Policy, a Policy Advocate, and a Policy Researcher. These openings will mainly be focused on AI policy and governance. Additional policy areas of interest may include lethal autonomous weapons, synthetic biology, nuclear weapons policy, and the management of existential and global catastrophic risk. You can find more details about these positions at futureoflife.org/job-postings. Link in the description. The deadline for submission is April 4th, and if you have any questions about any of these positions, feel free to reach out to jobsadmin@futureoflife.org.

Joscha Bach is a cognitive scientist and AI researcher working as a Principal AI Engineer at Intel Labs. He was previously a VP of Research for AI Foundation, and a Research Scientist at both the MIT Media Lab as well as the Harvard Program for Evolutionary Dynamics. Joscha earned his Ph.D. in cognitive science and has built computational models of motivated decision making, perception, categorization, and concept-formation.

Anthony Aguirre is a physicist that studies the formation, nature, and evolution of the universe, focusing primarily on the model of eternal inflation—the idea that inflation goes on forever in some regions of the universe—and what it may mean for the ultimate beginning of the universe and time. He is the co-founder and Associate Scientific Director of the Foundational Questions Institute and is also a Co-Founder of the Future of Life Institute. He also co-founded Metaculus, which is an effort to optimally aggregate predictions about scientific discoveries, technological breakthroughs, and other interesting issues.

And with that, let’s get into our conversation with Joscha Bach and Anthony Aguirre.

All right, let’s get into it. Let’s start off with a simple question here. What is truth?

Joscha Bach: I don’t think that we can define truth outside of the context of languages. And so truth is typically formally defined in formal languages, which means in the domain of mathematics. And in philosophy, we have an informal notion of truth and a vast area of them, a number of truth theories.

And typically, I think when we talk about any kind of external reality, we cannot know truth in a sense, as we cannot know it in a formal system. What we can describe is whether a model is predictive. And so we end up looking at a space of theories, and we try to determine the shape of that space.

Lucas Perry: Do you think that there are kinds of knowledge that exist outside of that space? For example, do I require knowledge that I am aware?

Joscha Bach: I think that knowledge exists only inside of the mind, that is inside of an agent that observes things. Knowledge is not arbitrary, of course, because a lot of knowledge is formerly derived, which means it has similar structure as a fractal, for instance, the Mandelbrot fractal, where you start out with some initial conditions and some rules, and then you will derive additional information on that. But essentially, knowledge is conditional because it depends on some axiomatic presupposition that you need to introduce.

Of course, that doesn’t make it arbitrary. It means that if you make that presupposition, you get to certain types of conclusions that are somewhat inevitable. And there is a limited space of formal theories that you can make. So there’s a set of formal languages that we can explore, that is limited. So for instance, when we define numbers, the number I think is best understood as a subsequent labeling scheme. And there are a number of subsequent labeling schemes that we can use, and it turns out that they have similar properties, and we can map them onto each other.

So we have several definitions for the natural numbers, and then we can build the other types of numbers based on this initial number theory. And it turns out that other parts of mathematics are equivalent to theorems that we discover a number of theory, or many of them are. So it turns out that a lot of mathematics is looking at the same fractals in different contexts. In a way, mathematics is about studying symbol games and all languages are. I think mathematics is the set of all languages.

Anthony Aguirre: You made an interesting statement, that knowledge is something that only exists in the mind… Well, in a mind. So I suppose you’re contrasting that with information. So how would you think differently about knowledge or information? How would you draw that distinction, if you do?

Joscha Bach: I think that knowledge is applied to a particular domain. So it’s regularities inside of a domain, and this domain needs to be given. And the thing that gives it is the stance of the agent that is looking at the domain.

To have knowledge, you need to have a domain that the knowledge is about. And the domain is given by an observer that is defining the domain. And it’s not necessarily a very hard condition, in terms of the power of the mind that is involved here. So I don’t require a subject with first person consciousness, and so on. What I do require is some system that is defining the domain.

Anthony Aguirre: So by a mind, you mean something that is defining the terms or the domain, or in some way, translating some piece of information into something with more semantic meaning?

Joscha Bach: I think a mind is a system that is able to predict information, but coming up with a global model, thereby making the information explainable.

Anthony Aguirre: Okay. So predict the information, meaning there’s some information stream of input, and a mind is able to create models for that stream of input to predict further elements of that stream.

Joscha Bach: At the moment, I understand the model to be a set of variances, which are parameters that can change their value. Each of them has a range of admittable values. And then you have invariances, which are computable relationships between these values that constrain the values with respect to each other. And the model can be connected to variances that the model itself is not generating, which can be parameters that enter the system from the universe, so to speak, from whatever pattern generator is outside of that system. And the system can create a model to make the next set of values that are being set by the outside predictable. And I think this is how our own perception works.

Lucas Perry: So in terms of human beings, truth is purely mind dependent. It requires a human being, for example, a brain, which is a substrate that can encode information, which generates models?

Joscha Bach: It’s not mind dependent, because it’s going to be the same for every mind with the same properties, in terms of information processing and modeling capacity.

Lucas Perry: You mean it’s not brain dependent?

Joscha Bach: It is not subjective.

Lucas Perry: It’s not subjective.

Joscha Bach: Yes. So for instance, the knowledge that we derive while studying mathematics is not subjective knowledge. It’s a knowledge that flows out of the constraints of the domains, and our ability to recognize these constraints.

Lucas Perry: So how does this stand in relation to accounts of truth as being third person, out there, floating in the universe truths, when we bring in the dependence of models and minds?

Joscha Bach: The first person is a construct to begin with. The first person is not immediately given. It’s introduced into the system and our system as a set of priors, then you are stuck with it until we deconstruct it again. But once we have understood what we are, believe is no longer a verb. There is no relationship between the self and the set of beliefs anymore. You don’t change by changing your beliefs about a certain domain. You just recognize yourself as a system that is able to perform certain operations. And every other system that is able to perform the same operations is going to be equivalent, with respect to modeling the domain.

Anthony Aguirre: I think Lucas is maybe getting at a claim that there’s a more direct access to knowledge or information, that we as a subjective agent are directly perceiving. There are things that you infer about the world, and there are things that you directly experience about the world as an observer in it.

So I have some sensory stream. I have access to the contents of my mind at some level, and this is very direct knowledge. Well, there’s a feeling that I don’t have to do a bunch of modeling to understand what’s going on. And there’s an interesting question of whether that’s true or not. To what degree am I modeling things all the time, and that’s just happening at a subconscious level?

Lucas Perry: Yeah, so a primitive version of that would be like how in Descartes’ epistemology, he starts off with awareness. You start off with being aware of being aware. So that might include some kind of subconscious modeling. Though, in terms of direct experience, it doesn’t really seem like it requires a model. It’s kind of like a direct apprehension, which may be structured by some kind of modeling.

Joscha Bach: Of course, Descartes is telling a story when he writes all this. He is much, much further when he writes on the story. This is not the thing that he started writing as a two-year-old, and then making his path along that line. Instead, he starts out at a highly aware state, and then backtracks and tries to retrace the steps that he had possibly taken to get where he was. When we have the impression that parts of our modeling is immediately given, it means that we don’t have any knowledge about how it works. In other words, we don’t have a model that we can talk about, or that we can consciously access before we can talk about it.

And when we become aware of the way in which we construct representation, we get closer to what many Eastern philosophers would call enlightenment. That our ideas become visible in their nature as representations. That we understand that everything that we perceive as real is, in fact, a representation happening outside of our mind. And we become aware of the mechanisms that construct with representations, and that allow us to arrive at different representations if we parametrized these processes differently.

Anthony Aguirre: Yeah, so I’m interested in this statement, in comparison to an earlier statement you made about what is not subjective. When you talk about the construction of… So there are a whole bunch of things that I experience as an aware agent. The claim that you’re making, which I agree with, is that pretty much all of that is something that I have constructed as something to experience, based on some level of sensory input, but heavily processed and modeled. And it’s essentially something that I’m almost dreaming in my mind, to create this representation of the world that I’m then experiencing.

But then there’s also a supposition that we have, that there is something that is causing the feeling of objectivity. There’s something underlying our individual agents sharing similar perceptions, using similar models, and so on. So there’s, of course, some meaning that we ascribe to an external world based on that shared concepts and ideas, and things that we attribute existence to, that are underlying the model that we construct in our mind.

So I guess the question is where we draw the line between the subjective thing that we’re inventing and modeling in our internal representation, and the objective world. Is there such thing as a purely objective world? Is it all representation at some level? Is there a very objective world with lots of interesting things in it, and we more or less understand it correctly as it is? Where do you sit on that spectrum?

Joscha Bach: Everything that looks real, everything that has the property of being experientiable at some level, that can be a subject of experience, is going to be constructed inside of your mind. It’s this issue that it’s not you that is constructing it, because you are a story inside of that mind about that agent. But this I is not the agent. The I is the representation of the agent inside of the system.

And what we can say about the universe, it seems to be generating patterns. These patterns seem to be independent of what we want them to be. So it’s not that all of these patterns seem to be originating in our mind, at least this is not a model that makes very good predictions. And there seem to be other minds that seem to be able to perceive similar patterns, but in a different projection. And the projection is being done by the mind that is trying to explain the patterns.

So the universe is projecting some patterns at your systemic interface, and at your thalamus, or at your retina and body surface, or wherever you draw the line. And then you have a nervous system behind this, or some kind of other information processing system. Our best model is that it’s the nervous system that is identifying regularities in these patterns. And part of these regularities are, for instance, things. Things don’t exist independently of our decomposition of the universe into things.

This doesn’t mean that the decomposition is completely arbitrary, because our mind makes it based on what’s salient and what compresses well. But ultimately, there is just this unified state vector of the universe that is progressing, and you make it predictable by splitting it up into things, into separate entities that you model independently of each other in your mind. And they exchange information, thereby change each other objects, evolution. And this is how we get to causality, and so on. But for instance, causality doesn’t exist independently of the decomposition of the universe into things, which has to be done by a mind.

Anthony Aguirre: Well, I’m curious as to why you say what there really is, is this unified evolving state vector. I mean, how do you know?

Joscha Bach:I don’t. So this is the easiest explanation. I have difficulty to make it simpler than that, but the separation of the universe into object seems to be performed by the observing mind. Right, the alternatives don’t make sense.

Anthony Aguirre: What you said seemed to attribute a greater reality to this idea of an evolving state vector or something, than other descriptions of the same. There are lots of ways we can describe whatever it is that we call this system that we’re inhabiting. We can describe it at the level of objects and the everyday level. We can describe it at the level of fields and particles and things. We could describe it, maybe there’s a string theory description. Maybe there’s a cellular automaton cranking along. Do you feel that one of these descriptions is more true than the others? Or do you feel like these are different descriptions of the same?

Joscha Bach: One is more basic. So the description of the universe as a set of states that changes is the most basic one. And then I can say, it seems that I can take the set of states and organize it as if it was a space. There’s a certain way to project the state vector. You get to a space, to something that’s multi-dimensional, by taking your number line and folding it up, for instance, into a lattice. And you can make this with a number of operators, and then you can have information traveling in this lattice, and then you usually get to something like particles.

And if the particles don’t have a distinct location, and you just described the trend of particles going, then you have fields. But they all require additional language that you need to introduce additional assumptions. And also we find that they are not true, in a sense that there is a level of description where reality and description fall apart from each other.

So for instance, the idea of describing the universe as a Minkowski space, works relatively well at certain scales. But if you go below these scales, you get singularities. Or in certain regions of the universe, you get singularities, and you realize you’re not living in space. You can just describe the dynamics at a certain scale, usually as space. Same thing is true for particles. Particles are a way to talk about how information travels between the locations that you can discern the state vector.

No, but you should challenge me. So the thing that I’m sometimes frustrated with, with philosophers, is that they somehow think that the notions that they got in high school physics are somehow objectively given. And they know electrons, and physicists know what electrons are, and there’s a good understanding, and we touch them every day. And then we build information processing on top of the electrons. And this is just not true. It’s the other way around.

Anthony Aguirre: I mean, all of these concepts, the more you analyze them and the harder you think about them, the less you understand them, not the more. In the sense of for particles, I mean, if you ask what an electron is, that is a surprisingly hard answer to give as a physicist. An electron is an excitation of the electron field. And what is the electron field? It’s a field that has the ability to create electrons.

Joscha Bach: Yes. But when you don’t know this, it means you have never understood it. The person that came up with these notions have either pulled them out of thin air, or they understood something that you don’t. They saw something in their mind, that they tried to translate into a language that people could work with, and it allowed them to build machinery. But that did not necessarily convey the same understanding. And that to me is totally fascinating. You learn in high school how to make a radio, but you don’t get the ideas that were necessary to get somebody the idea, “Oh, this is what I should be doing to get a radio in the first place.” This is not enough information to let you invent a radio, because you have no idea why the universe would allow you to build radios.

And it’s not other people experimented and they suddenly got radios, and then they tried to come up with a fancy theory that would project electrons into the universe and magnetic fields, or electromagnetic fields to explain why that radio would work. They did have an idea on how the universe must be set up, and then they experimented and came up with the radio, and verified that their intuition must be correct at some level. And the idea that Wolfram has, for instance, that the universe is a cellular automaton, is given by the idea that the cellular automaton is a way to write down the Turing machine. It’s equivalent to a Turing machine.

Lucas Perry: Sorry. Did you say cellular automaton?

Joscha Bach: Yes.

Lucas Perry: Okay. Is a way to write down a Turing machine?

Joscha Bach: Yes. So there is a cellular automaton, or there are a number of cellular automaton which are Turing complete. And cellular automaton is a general way to describe computational systems. And since the constructivist turn, I think that mostly agree that every evolving system can be described as some automaton, as a finite state machine, or in the unbounded case as a Turing machine.

Anthony Aguirre: Well, that I think is not clear. I think it’s certainly not the way that physics, as it actually operates, works. We do all this stuff with real numbers and calculus, and all kinds of continuous quantities. And it’s always been of interest to me, if we suppose that the universe is fundamentally discrete or digital in some way, it sure disguises it quite well. In the sense that the physics that we’ve invented based on so many smooth quantities and differentials and all these things, is incredibly effective.

And even quantum mechanics, which has this fundamental discreteness to it, and is literally quantum mechanics, is all still chock-full of real numbers and smooth evolutions and ingredients, that there’s no obvious way to take the mechanics, any of the fundamental physics theories that we know, and make a nice analog version of them that turns into… A nice digital version, rather. That turns into the analog continuous version that we know and love, which is an interesting thing.

You would think that that would be simpler if the world was fundamentally discreet and quantum. So I don’t know if that’s telling us anything, it may just be historical, that these are the first tools that we developed and they worked really well. And so that became the foundation for our way of thinking about how the world works, with all sorts of smooth quantities. And certainly, discrete quantities, when you have a large enough number of them, can act smoothly.

But it’s nonetheless, I think a little bit surprising how basic and effective all of that mathematics appears to be if it’s totally wrong, if it’s essentially, fundamentally the wrong sort of mathematics to describe how reality is actually working at the base level.

Joscha Bach: It’s almost correct. You remember these stories that Pythagoras very opposed to introducing irrational numbers into the code base of mathematics. And there are even stories that he committed murder, which are probably not true, to prevent that from happening. And I think that he was really onto something. The world was not ready yet. And the issue, I think, that he had with irrational numbers is that they’re not constructive. And people didn’t have until quite recently, in the languages of mathematics, is the distinction between value and function.

In classical semantics, a value and a function are basically equivalent, because if you can compute the function, you have a value. And if the function is sufficiently defined, you can compute it. And for instance, there is no difference between pi as a function, as an algorithm that you would need to compute, and pi as a value in this classical semantics. But in practice, there is. You cannot determine arbitrary many digits of pi before your sun burns out. There’s a finite number of digits of pi that every observer in our universe will ever know.

That also means that there can be no system that we can build, or that something with similar limitations in this world can ever build. That relies on knowing the last digit of pi when it performs an operation. So if you use a code base that assumes that some things in physics have known the last digit of pi before the universe went on to go into its next state, you are contradicting this idea of constructive mathematics.

And there was some hope that this could be somehow recovered. Because mathematicians, before this constructive turn, just postulated this infinite number generator, or this infinite operation performer as a black box. You would introduce it in your axiomatic system, for instance, via the axiom of choice. And so you postulate your infinite number calculator, the thing that is able to perform infinitely many operations in a single step, or read infinitely many arguments into your function in a single step, instead of infinitely many.

And if you assume that this is possible, for instance, via the axiom of choice, you get nice properties, but some of them are very paradoxical. You probably know Hilbert’s Hotel. A hotel that has infinitely many rooms and is fully booked, and then a new person arrives. And you just ask everybody to move one room to the right, and now you have an empty room. And you can also have infinitely many buses come, and you just ask everybody to move into the room that has twice its current number. And in practice, this is not going to work because in order to calculate twice your number, you need to store it somehow. And if the number gets too large, you are unable to do that.

But it should also occur to you that if you have such a thing as Hilbert’s Hotel, that you’re not looking at a feature. You’re probably looking at a bug. You have just invented a cornucopia. You get something from nothing, or you get more from little. And that should be concerning. And so when you look into this in detail, you get to Gödel’s proof and to the halting problem, which goes two different ways of looking at the issue that when you state that you can make a proof, without actually being performing the algorithm of that proof, you’ll run into contradictions.

That means you cannot build a machine in mathematics that runs the semantics of classical mathematics without crashing. That was Gödel’s discovery. And it was a big shock to Gödel, because he strongly personally believed in the semantics of classical mathematics. And it’s confusing to people like Penrose, who thinks that, “I can do classical mathematics, computers cannot. So my mind must be able to do something that computers cannot do.” And this is obviously not true.

So what’s going on here? I think what happens is that we are looking at too many parts to count. When you look at the world of too many parts to count that are very very large numbers, you sometimes get dynamics that are convergent. That means that if you perform operation on a trillion objects, or on a billion objects in this domain, the result is somewhat similar. And the domain where you have these operators, the way these operators converge, this is geometry, by and large. And this is what we are using to describe the reality that we’re embedded in, because the reality that you and me are embedded in is necessarily composed of too many parts to count for us, because we are very, very large, with respect to our constituent parts. Because otherwise, you wouldn’t work.

Anthony Aguirre: Yeah. Again, if there are enough of something and you’re trying to model it, that real numbers will work well.

Joscha Bach: But it’s not real numbers. There are very, very large numbers. This is all.

Anthony Aguirre: Right. But the model is real numbers. I mean, that’s what we use when we do physics.

Joscha Bach: Yes. But it’s totally fine when you do classical mechanics. It’s not fine when you do foundational physics. Because at this point, you are checking out the code base from the mathematicians, that it has some comments by now. And the comments say, “This is actually not infinite.” When you want to perform operations in there, and when you want to prove the consistency of the language that you’re using, you need to make the following changes, which means that pi is now a function. And you can plug this function into your nearest sun and get numbers, but you don’t get the last digit.

So you cannot implement your physics in such a way, or your library of physics, that you rely on having known the last digit of pi. It just means that for sufficiently many elements that you are counting, this is going to convert to pi. This is all there is.

Anthony Aguirre: Right. No, I understand the motivation for thinking that way. And it’s interesting that that’s not the way people actually think, in terms of fundamental physics, in terms of quantum mechanics or quantum gravity, or anything else.

Joscha Bach: Yeah, but this is their fault. This is not our fault. We don’t have to worry about them.

Anthony Aguirre: Fair enough. So I think the claim that you’re making would be that if we do… There’s a question. If 100 years from now, the final theory of physics is discovered, and they have it available on a T-shirt near them, is that theory going to essentially contain a bunch of natural numbers and counting and discrete elements? Or is it going to look-

Joscha Bach: Yes, of course. Integers is all that is computable.

Anthony Aguirre: Right.

Joscha Bach:The universe is made from integers.

Anthony Aguirre: So this is a strong claim about what’s ultimately going to happen in physics, I think, that you’re making, which is an interesting one. And is very different from what most practicing physicists, in thinking about the foundations of physics, are taking as their approach.

Joscha Bach: Yes, but there’s a growing number of physicists which seem to agree with this. So very informally, when I was in Banff at a foundational physics conference, I made a small survey among the foundational physicists. I got something like 200 responses. And I think 6% of the accumulated physicists were amenable to the idea of digital physics, or were proponents of digital physics. I think it’s really showing that this is growing in attraction, because it’s a relatively new idea in many ways.

Lucas Perry: Okay. So if I were to try to summarize this, I guess in a few statements, correct me if I got any of this wrong, is that because you have a computationalist metaphysics, Joscha, if reality contains continuous numbers, then the states of the large vector space can’t be computed. Is that the central claim?

Joscha Bach: So the problem is that when you define a mathematical language any kind of formal language that assumes that you can perform computations with infinite accuracy, you run into contradictions. This is the conclusion of Gödel.

And that’s why I’m stuck with computation. I cannot go beyond computation, hyper computation doesn’t work because I cannot be defined constructively. I cannot build a hyper computer that runs hyper computational physics. So I will be stuck with computation. But it’s not a problem because there is nothing that I cannot do with it that I could do before. Everything that worked so far, is still working, right. Everything that I could compute in physics until now is still computable.

Lucas Perry: Okay. So to say digital is to say, what is that to say?

Joscha Bach: It means that when you look at physics and you make observations, you would necessarily have to make observations with a finite resolution. And when you describe the transitions between the observables, you will have to have functions that are computable. And now there is the question, is this a limitation that only exists on the side of the observer? Or is this also a limitation that would exist on the side of the implementation? And now the question is, how could God or whatever hypothetical circumstance, perform to do this? So the answer is if the universe is implemented any kind of language, then it’s subject to the same limitations. The only question is, could it be possible that the universe is not implemented in some kind of language? Question is, is it implemented? What does it mean to be implemented? We can no longer talk about it. It doesn’t really make sense to talk about such a thing.

We would have to talk about it in a language that is inconsistent. So when we want to talk about the universe that is continuous, we would have to define what continuity means in a way that would explain how the universe is performing continuous operations, at least at some level. And the assumption that the universe can do this, leads into contradictions. So we would have to basically deal with the fact that we have to cheat at some point. And I think what happened before constructive mathematics, was that people were cheating. They were pretending that they could read infinitely many arguments into their function, but they were never actually doing it. It was just giving them elegant notations. And these elegant notations are specifications for things that cannot be implemented. And the practice was not a problem because people just found workarounds in practice, right? They would do something slightly different than the notation.

And something that I found as a kid very irritating when I did mathematics, because sometimes the notation that I was using in mathematics, was so unlike from what I had to do in my own mind, or what I had to do on my machine to perform a certain operation. And I didn’t really understand the difference. I didn’t understand the difference between computational mathematics and why it existed. And in practice, the difference is that classical mathematics is stateless. It assumes that everything can be computed without going from state to state.

Anthony Aguirre: Yeah, but it’s beguiling because you think there’s this number pi and you feel like it exists. And whether anybody computes the ten to the 50th digit of it, it has some value. If you could compute it, it would have some value and there’s no sort of freedom in what that value is going to be. That value is defined as soon as you make the definition of pi.

Joscha Bach: But go back to this moment when you had this idea that pi is there for you with all its infinite glory. Certainly not an idea that you were born with, right? You were hypnotized into it by your teachers and they by their teachers. And so somebody at some point made this decision, and it’s not obvious.

Anthony Aguirre: No, it’s sort of a relic of Plato I guess. I think mathematicians are obviously split in this in all kinds of ways. But there are a lot of mathematicians who were platonists and really very much think that pi meaningfully exists in all its glory, in that way. And I think most physicists, well I don’t know actually. I think there are probably surveys that we could consult that I haven’t, about the degree of platonism in physicists. I suspect it varies pretty widely actually.

Joscha Bach: Now take a little step back from all these ideas of mathematics that you may or may not have gotten in school, and your teachers may or may not have gotten them in school. But look back at what you know, right. Your computer and how your computer would make sense of the universe if you could implement AI on it, right? This is something that we can see quite clearly. Your computer has a discrete set of sensory states that it could make sense out of, right? And it would be able to construct models and it can make these models in a resolution that is, if you want to higher than the resolution of our own mind, it can make observation at a higher resolution than our senses. And there is basically no limit in terms of not being able to surpass the kind of model that we are doing.

And yet everything in the mind of that system would be discrete necessarily, right? So if the system conceives of something that is continuous, if that system conceives of a language that is continuous. And we can show that this language doesn’t work, that it’s not consistent, then the system must be mistaken, right? It’s pointing indexically at some functions that perform some kind of effective geometry. And an effective geometry is like the one in the computer game, right? It looks continuous, but it certainly is not. It’s just continuous enough.

Lucas Perry: Relative to the conventional physics, what does a computational view of physics, what is that changing?

Joscha Bach: Nothing. If the conventional physicist has made a mistake, it’s exactly the same. It’s just that conventional physics invites making certain kinds of mistakes, that might lead to confusion. But in practice it doesn’t change anything. It just means that the universe is computable. And of course it should be because for it to be computable just means that you can come up with a model that describes how adjacent states in the universe are correlated.

Lucas Perry: Okay. So this leads into the next question. So we’re going to have to move along a little bit faster and more mindfully here. You’ve mentioned that it’s tempting to think of the universe as part of a fractal. Can you explain this view and what is physics according to this view?

Joscha Bach: So I think that mathematics is in some sense, something like a fractal. So the natural numbers are similar to the generator function of the Mandelbrot fractal, right? You take Peano’s axioms, and as a result you get all these dynamics if you introduce a few suitable operators. And you can define the space of natural numbers, for instance using addition, or you can do it using multiplication. In the case of multiplication, you will need to introduce the prime numbers, because you can not derive all the actual numbers just by multiplication with one and zero. You need all these elements, the basic elements of this multiplication space would be the prime numbers. And now there is the question, is there an elegant translation between the space of multiplication in the space of additions and maybe has to do with the Riemann zeta function, right? We don’t know that. But essentially we are exploring a certain fractal here. Right?

Now with respect to physics, there is the question. Why is there something rather than nothing, and why does this something look so regular? We can come up with anthropic arguments, right? If nothing, we exist, and then nobody would be asking the question. Also, if that thing was irregular, you probably couldn’t exist. The type of mind that we are and so on requires large-scale regularity. But it’s still very unsatisfying because this thing that something exists rather than nothing, should have an explanation that we want to make as simple as possible. And the easiest explanation is that maybe existence is the default. So everything that can be implemented would exist, and maybe everything that exists is the superposition of all the finite automata. And as a result you get something like a fractal, and in this fractal there is us.

This idea that the universe can be described as an evolving cellular automaton is quite old. I think it goes back to Zuse’s Rechnende Raum and it’s been formulated in different ways by for instance, Fredkin and Wolfram, and a few others. But again, it’s equivalent to saying that the universe is some kind of automaton. Some kind of machine that takes a starting value, a starting configuration, or any kind of configuration, and then moves to the next configuration based on the stable transition function.

Anthony Aguirre: But there’s a difference between saying that the universe is a cellular automaton, and the universe is all possible cellular automaton. That view is held sort of by Max Tegmark, and I think Jurgen Schmidhuber, and I think, I’m forgetting his name. But you’re talking more about that version. Aren’t you?

Joscha Bach: I’m not sure if it just means that Wolfram’s project of enumerating all the cellular automata and taking the first one that produces a universe that looks like ours, might not work. It could be that you need all of them in a certain enumeration. But it still is a cellular automaton. It’s just one that is much longer than Wolfram would hope.

Anthony Aguirre: You’re saying all possible cellular automata is the same as one cellular automata.

Joscha Bach: Yes. It would be one cellular automaton that is the result of the superposition of all of them. Which probably means that long cellular automata only act relatively rarely, so to speak. And I have no idea how to talk about this in detail and how to model this. So this is a very vague intuition.

Lucas Perry: Okay. So, could you describe the default human state of consciousness?

Joscha Bach: Consciousness I think is the control model of our attention. And human consciousness is characterized by attention to features that are taken to be currently the case that we keep stable in this phase, so we can vary other parameters, that we can make conditional variants in our perception and so on. And that we can form indexed memories for the purpose of learning.

Then we have access consciousness, which is a representation of the attentional relationship that we have to these objects. So we know that we are paying attention to them, and we know in which way we are paying attention to them. So for instance, do we single out sensory features or high-level interpretations of sensory features, or hypotheses, or memories and so on. And this is also part of the attentional representation.

And third, we have reflexive consciousness. And this is necessary because the processes in our neocortex are self-organizing to some degree. And this means that this attentional process needs to know that it is the sensory of the attentional process. So it’s going to make a perceptual confirmation of the fact that it is indeed the attentional process. And this makes consciousness reflexive. We basically check back, am I spacing out or am I still paying attention? Am I the thing that pays attention? And this loop of going back between the content and the reflection is what makes our consciousness almost always reflexive. If it stops being reflexive, drift off and often fall asleep, like literally.

Anthony Aguirre: What would you say that, that sort of self-reflexive consciousness is experienced by non-human animals? Do you consider a mouse that probably does not have that level of awareness not conscious? Or are you sort of talking about different aspects of consciousness, some of which humans have, and some of which other creatures have?

Joscha Bach: I suspect that consciousness is not that complicated. What’s complicated is perception. So setting up a system that is able to make a real time adaptive model of the world that predicts all sensory features in real time, makes a global universe model and figures out which portion of the universe model it’s looking at right now, and swaps this in and out of working memory as needed, that’s the hard part. And once you have that and you have a problem that is hard enough to model, then the system is going to model its own relationship to the universe in its own nature, right? So you get a self-model at that level. And I think that attentional learning is necessary because, just correlative learning is not working very well.

The machine learning algorithms that we are currently using largely rely on massive backpropagation over many layers. And the brain is not such a neatly layered structure, it has links all over the place. And also we know that the brain is more efficient. It needs very fewer instances of observing a certain thing before it makes connection and is able to make inferences on that connection. So you need to have a system that is able to basically single out the parts of the architecture that need to be changed. And this is what we call attention, right? When you are learning, you have a phase where you do simple associative learning. We usually have right after you’re born and before, when you form your body map and so on. And after the initial layers of formed, after the brain areas are somewhat laid out and initialized and connected to each other, you’ll do attentional learning. And this attention learning requires that you decide what to pay attention to in which context. And what to change and why. And when to reinforce it and when to undo it.

And this is something that is we are starting to do now in AI, especially with the transformer, where attention suddenly plays a role and we make models of the system that we are learning over. So in some sense, the attention agent is an agent that lives inside of a learning agent. And this learning agent lives inside of a control agent. And the control agent is directing our relationship to the universe, right? You notice that you’re not in charge of your own motivation. You notice that you’re not directly in charge of your own control. But what you can do is you can pay attention to things and the models that you generate while paying attention are informing the behavior of the overall agent. And the more we become aware of that, the more this can influence our control.

Anthony Aguirre: So Lucas was asking about the default state of consciousness. Which is, we have this apparatus which we’ve developed for a very particular reason. Survival and modeling the universe, or modeling the world well enough to predict how it’s going to work. And how we should take action to enhance our survival or whatever, set of auxiliary goals that we might have. But it’s not obvious that that manner of, that mode of consciousness or mental activity is optimal for all purposes. It’s certainly not clear that if I want to do a bunch of mathematics, the type of mind that enables me to survive well in the jungle, is the right type of mind to do really good mathematics.

It’s sort of what we have, and we shunted over to the purpose of doing mathematics. But, it’s easy to imagine that the mind that we’ve developed in sort of its default state, is pretty bad at doing a lot of things that we might like to do once we choose to do things other than surviving the jungle. So, I’m curious what you think about, how different a machine neural architecture could be if it’s trying to do very different things than what we’ve developed to do.

Joscha Bach: I suspect that given a certain entanglement, that is the ability to resolve the universe in a certain spatial and temporal way. And then having sufficient processing resources and speed, there is going to be something like an optimal model that we might be ending up with. And humans are not at this level of the optimal model. Our brain is still a compromise, I think. So we might benefit from having a slightly larger brain. I think that the differences between a human performance, are differences in innate attention. So talents are basically differences in innate attention. And you know how they say I’ll be recognized an extroverted mathematician. They look at your shoes. And what this alludes to is that good mathematicians tend to have Asperger’s. Which means that they are using parts of their brain, that normal humans use for social cognition and for worrying about what other people think about them, for thinking about abstract problems.

And often I think that the focus on analytic reasoning is a prosthesis for the absence of normal regulation. That is, if you are a nerd and you will have as a result a slightly different interface to the social environment than the normies. You as a child don’t get a manual on how to deal with this. And so you will act on your impulses, on your instincts, and unless you are surrounded by other nerds, you tend to fail in your social interactions. And you will experience guilt, shame and despair, and this will motivate you to understand what’s going on. And because as a child, you typically don’t have any tools to reverse analyze yourself, or even notice that there is something to reverse engineer. You will turn around and reverse engineer the universe.

Anthony Aguirre: Yeah. I suspect if we were able to have the experience, and maybe someday we will with some sort of mind-to-mind interface, of seeing, actually experiencing someone else’s worldview and processing reality through their apparatus. I suspect we would find that it’s much more different than we suppose, on an everyday level. We sort of assume that people are understanding and seeing the world more or less as we are, with some variations. And I would love to know, to what degree that that’s true. You read these, or I read these articles of some people who cannot imagine visual images. And there are people I compare with my wife when she imagines a visual image in her mind, it is right there in detail. She can experience it, sort of like I would experience a dream or physical reality.

When I think of something in my mind, it’s quite vague. And then there are other people who, there’s sort of nothing there. And that’s just one aspect of cognition. I think that would be a fun project to really map out, what that sort of range in qualia really is among human minds. I suspect that it’s much, much bigger than we really appreciate it. It might cause us to have a little more understanding for other people, if we understood how literally, differently, they experience the world than we do. Or, it could be that it’s very similar. That would be fascinating to the degree, I mean, we do all function fairly well in this world, so there has to be some sort of limit on how differently we can perceive and process it. But yeah, that fascinates me how big that range is.

Joscha Bach: I have a similar observation as my own wife who’s an artist. And she is very good at visualizing things, but she doesn’t notice external perception as much. And to compare with a highly perceptive person, I think I have about 20% of their sensory experience because I’m mostly looking at the world with conceptual reflection. I have Aphantasia, so outside of dreams or hypnagogic state, I cannot visualize, or don’t get conscious access to my visualizations. I can draw them, so I can make designs. But for the most part, I can not make art because I can not see these things. I can only do this in a state where I’m slightly dreaming.

So it’s also interesting to notice the difference. It’s not that I focus on this and it gets more and more plastic, and at some point I see it. It’s more like it’s being shifted in from the side. It’s similar to when you observe yourself falling asleep, and you manage to keep a part of yourself online to track this without preventing you from going to sleep. Which is sometimes hard, right? But if you pulled it off, you might notice how suddenly the images are there, and they more or less come from the side. Right? Something is shifting in and it’s taking over.

I think that there is a limit to, because Anthony brought this up, to how it makes sense, to understand the world in the way that you operate. It’s just that we often tend not to be completely aware of it during the first 400 years or so. So it takes time to reverse engineer the way in which you operate. For instance, it took me a long time to notice that the things that I care about are largely not about me as an individual. So, I mean, of course we all know this because it’s a moral prerequisite that you don’t only care about yourself, but also about the greater whole and future generations. But what does that actually mean? Right. You are basically an agent that is modeling itself as part of a larger agency. And so you get a compound agent.

And a similar thing is true for the behaviors in your own mind. They are not seeing themselves as all there is. But, at some level, model, that they are part of a greater whole of some kind of processing hierarchy, and control hierarchy in which they can temporarily take over and run the show. But then they will be taken over by other processes and they need to coordinate with them. And they are part of that coordination. And this goes down to the lowest levels of the way in which this is implemented. And ultimately you realize that information processing systems on this planet, which coordinate for instance, an organism, or that cooperation between organisms, or that cooperation within ecosystems.

And this is the purpose of our cognition in some sense. It’s the maintenance of complexity, so we can basically shift the bridgehead of order into the domain of chaos. And this means that we can harvest negentropy in regions where dumber control systems cannot do that. That’s the purpose of our cognition. And to understand our own role in it and the things that matter, is the task that we have. And there are some priors that we are born with. So ideas about the social roles that we should be having, the things that you should care about and should not. Over time, we replace these priors, these initial reflexes, with concrete models. And once we have a model in place, we turn off the reflex.

Lucas Perry: So you don’t think that the expression of proxies in evolution means that harvesting negentropy isn’t actually what life is all about. Because it becomes about other things once we gain access to proxy rewards, and rational spaces and self reflection.

Joscha Bach: I think it’s an intermediate phase where you think that the purpose of existence, is for instance, a mass knowledge, or to have insight, or to have sex, or to be loved and love, and so on. And this is just before you understand, what this is instrumental to.

Lucas Perry: Right, but like you start doing those things and they don’t contribute to negentropy.

Joscha Bach: They do. If they are useful for your performance as a living being, or for the performance of the species that your part of, or the civilization or the ecosystem, then they serve a certain role. If not, then they might be statistically useful. If not, then they might be a dead end.

Lucas Perry: Yeah. That makes sense.

Anthony Aguirre: I mean, life is quite good at extracting negentropy, or information from its hidden troves supplied by the sun. You know I don’t think we’re ever going to compete with plankton. At least not anytime soon, maybe in the distant future. So it seems like a little bit of an empty-

Joscha Bach: If you have a pool, you are competing with plankton, right? We are able to settle surfaces that simpler organisms cannot settle on. And we’re not yet interested in all the surfaces, we are just one species on this planet. Right?

Anthony Aguirre: I don’t know, I guess (crosstalk 00:54:03). If I beat the plankton at some point, I’m just not that excited about that. I mean, I think it’s true that, to sustain as life we have to be able to harvest that, transform that information from the form that it’s in to an ability to maintain our own homeostasis and fight the second law of thermodynamics essentially, and maintain a structure that is able to do. But that’s metabolism and is I think sort of the defining quality that life has.

But I… It’s hard for me to see that as more than a sort of means to an end in the sense that enables all of the activities that we do. But if you told me that someday our species will process this much entropy instead of that much entropy, that’s fine, but, I don’t know, that doesn’t seem terribly exciting to me. Just like discovering a gigantic ball of plankton and space. I wouldn’t be that impressed with it.

To me, I think there’s something more interesting in our ability to create sort of information hierarchies, where there are not just the amount of information, but information structures built on other information structures, which biological organisms do and we do mentally with all sorts of civilizational artifacts that we’ve created. The level of hierarchical complexity that we’re able to achieve seems to me more interesting than just the ability to effectively metabolize a lot.

Joscha Bach: I think that as the individual. We don’t need to know that. As an individual, we are relatively small, inconsequential. We just need to know what we need to know to fulfill our role. But if the civilization that we are part of doesn’t understand that, it’s in a long-term battle about entropy, then it is in a dire situation I think.

Anthony Aguirre: There’s so much fuel around. I just think we have to figure out how to unlock a little bit better, but we are not going to run out of negentropy.

Joscha Bach: No, life is not going to run out of negentropy. It could be that we are running out of the types of food that we require, and climates that we require, and the atmosphere that we require to survive as a species. But that is not a disaster for life itself, right? Cells are very hard to eradicate once they settled a planet. Even if you have a super volcano erupting or a meteor hitting the Earth, Earth is probably not going to turn sterile anytime soon. And so from the perspective of life, humanity is probably just an episode that exists to get the fossilized carbon back into the atmosphere.

Lucas Perry: So let’s pivot into AGI who might serve as a way of locally optimizing the increase of entropy for the Milky Way and hopefully our local galactic cluster. So, Joscha, so how do you view the path to aligned AGI?

Joscha Bach: If you build something that is smarter than you, the question is it already aligned with you or not? If it’s not aligned with you, you will probably not be able to align it. At least, not in the long run. The right alignment is in some sense about what you should be doing under the circumstances that you are in. When we talk about alignment in a human society, there are objective criteria for what you should be doing if you are part of a human society. Right? It’s given by a certain set of permittable aesthetics. An aesthetic is degrees of world that might emerge over your initial preferences. And there are a number of local optima, maybe there’s even a global optimum in the way the species can organize itself at scale.

And if you build an AI, the question is, is it able to and would it be interested? Would it be in the interest of the AI to align itself with this global optimum or with one of the existing local optima?

And what would this look like? It would probably look like different from our intuitions about what society should be like. Right? We clearly lost the plot in our society. Every generation, since the last war has fewer intuitions about how the society actually functions and what conditions need to be met to make it sustainable.

Sustainability is largely a slogan that exists for PR purposes for people in the UN and so on. It’s nothing that people seriously concerned about. We don’t necessarily think about how many people can the planet support and how can we reach that number of people? How would we make resources be consumed in closed loops? These are all questions that are not really concerning.

The idea that we can eat up the resources that we depend on, especially the ecological resources that we depend on, is relatively obvious to the people that run the machines that drive our society. So we are in some sense, in a locust mode, in a swarming mode, and this swarming mode is going to be a temporary phenomenon. And it’s something that we cannot stop. We are smart enough to understand that we are in this swarming mode, and that’s probably not going to end well. We are not obviously smart enough to be able to stop ourselves.

Lucas Perry: So it seems like you have quite a pessimistic view on what’s going to happen over the next hundred to 300 years, maybe even sooner. I think I’ve heard you say somewhere else that we’re so intelligent that we burned a hundred million years of trees to get plumbing and Tylenol.

Joscha Bach: Yes. Yeah. This is in some sense the situation that we’re in, right? It’s extremely comfortable.

Lucas Perry: Yeah, it’s pretty nice.

Joscha Bach: There has never been a time in probably the history of the entire planet where a species were so comfortable as us.

Anthony Aguirre: But why so pessimistic?

Joscha Bach: I’m not pessimistic, I think. It’s just if I look at the possible trajectories that we are in, is there a trajectory in which we can make ourselves enter a trajectory of perpetual growth? Very unlikely, right? Is there a trajectory in which we can make the thing that we are in sustainable without going through a population bottleneck first, which will be unpleasant? Unlikely. It’s more likely than the idea that we can have eternal growth, but it’s also not impossible that we can have something that is close to eternal growth. It’s just unlikely.

Anthony Aguirre: Fair. I mean, I think what I find frustrating is that it’s pretty clear that there are no serious technological impediments to having a high quality, sustainable, steadily growing civilization. The impediments are all of our own making. It’s fairly clear what we could do. There are something… problems that are quite hard. There’s some that are fairly easy. Global warming, we really could solve. If we could coordinate it, just do the things that are fairly clearly necessary to do; that’s a solvable problem.

Running out of easily accessible natural resources is harder because you have to get closer and closer to a hundred percent recycling if you want to keep growing the material economy while staying within the same resources. But it’s also true that what we’re doing is not so clear how much we have to keep growing the material economy.

When I look at my kids, they… Compared to when I grew up, they barely have any things. They’re not that interested in things. Most of the things, there are some that they really like, but if I buy… Even five years ago when I would buy my younger son a toy, he would play with the toy for a little while, and then he’d be done with the toy. And I’d be like, why did I buy that toy? They’re… Much more of their life than ours is digital. And it is consuming media. That seems to be the trajectory of humanity at some level as a whole.

And so it’s unclear. It may be that there are ways in which we can continue economic and growth in sort of quality of life, even while the actual amount of material goods that we require is not sort of exponentially increasing in that in the way that you would think. So I share your pessimism as to what’s actually going to happen, but I maybe more frustrated by how it wouldn’t be that hard to make it work if we could just get our act together.

Joscha Bach: But this ‘getting our act together’ is the actual difficulty.

Anthony Aguirre: I know.

Joscha Bach: Right, because putting your act together, as a species, means that you as an individual needs to make decisions that somehow influence the incentives of everybody else. Because everybody else is going to keep doing what they are doing, right? This argument, ‘if everybody would do as I say’, that is not a valid, ‘therefore you do it’, is not a valid argument.

Anthony Aguirre: No, no.

Joscha Bach: Right, this is not how it works. So in some sense, everybody is working based on their current incentives. And what you would need to do is to change incentives within the means that you, as an individual, have, of everybody else to bring about this difference in the global incentive structure. So it comes down to, in some sense, implementing a world government that would subdue everything under some kind of eco-fascist guideline. And the question is, would AGI be doing that? So you could, it’s very uncomfortable to think about it. It’s certainly not an argument that would be interesting to entertain for most people. And it’s definitely nothing that would get a majority. It’s just probably what needed to be done to make everything sustainable.

Anthony Aguirre: That’s true. I do think that there are within reach technological advances that are sort of genuinely better along every axis. Like once, as has happened, photovoltaic energy sources are just cheaper now than fossil fuel ones. And once you cross that threshold, there’s just, there’s no point. Maybe five or 10 years from now there simply won’t be any point at all in making fossil fuel energy sources once… So there’s sort of thresholds where the right thing actually just becomes the default and the cheapest and there isn’t any tension between doing what we should do as a civilization and doing what people want to do as an individual.

Now it may be that they’re always sort of showstoppers that are important and where that hasn’t happened or isn’t possible, but it’s not terribly obvious that there are a lot of… there seem to me to be a lot of systems where with some careful architecting, the incentives can be aligned well enough that people can do more or less what they selfishly want to do and nonetheless contribute to a more sustainable civilization. But it seems to require a level of planning and creativity/intelligence that we haven’t quite gotten together yet.

Joscha Bach: Do you think that there’s an optimal number of trees on the planet? Probably, right?

Anthony Aguirre: An optimal number of trees?

Joscha Bach: Yes. One that would be the most beneficial one for the ecosystems on earth. And everything else which happens on earth. Right? If there are too few trees, there are some negentropy which you are leaving on the table. If there are too many then something will go out of whack. And some of the trees will eventually die. And so in some sense, I think there is going to be an optimal number of trees, right? No?

Anthony Aguirre: Well, I think there are lots of different systems with preferences that will not all agree on what the optimal number of trees is. And I don’t think there’s a solved problem for how to aggregate those preferences.

Joscha Bach: Of course. No, I don’t mean that we know this number or it’s easily discernible or that it’s even stable. Right? It’s going to change over time.

Anthony Aguirre: I don’t think there is a answer to how to aggregate the preferences of a whole bunch of different agents.

Joscha Bach: Oh, it’s not about preferences, ultimately the preferences are instrumental to complexity. Basically to how much life can exist on the planet. How much negentropy can you harvest?

Ultimately there is no meaning, right? The universe doesn’t care if something lives inside of it, or not. But from the perspective of cellular life, what evolution is going for is, in some sense, an attempt to harvest the negentropy that is available.

Anthony Aguirre: Yeah. But we don’t have to listen to evolution.

Joscha Bach: Of course not, no. We don’t have to, but if we don’t do this, it probably is going to limit our ability to stay. Because we are part of it. We are not the highest possible life form. There are going to be, if apes are some kind of a success model, descendants that are not going to be that humanlike. If we indeed manage to leave the gravity well and not just send AI’s out there, it turns out that there can be descendents of us that can live in space they will probably not look like that Captain Kirk and Lieutenant Uhura.

But in far future AI, the commanders of space fleets will look very, very different from us. They will be very different organisms, right? They will not have human aesthetics. They will not solve human problems. They will live together in different ways because they interface in different ways, interact with technology in different ways. It will be a different species de facto.

And so if you think about humans as a species, we do depend on a certain way in which we interface with the environment in a healthy, sustainable way. And so I suspect that there is a number of us that should be on this planet, if we want to be stewards of the ecosystems on this planet. It’s conceivable that we can travel to the stars, but probably not in large numbers and we are not the optimal form.

Right, if you want to settle Mars, for instance, we should genetically modify our children into something that can actually live on Mars. So it would/should be something that doesn’t necessarily need to have an atmosphere around it. It should be able to hibernate. It should be super hardy. It should be very fast and be able to feed on all the proteins that are available to it which are outside of Earth, mostly humans. Right? So it should look like Alien in the Alien movies.

Anthony Aguirre: Again, I think that’s a choice. I mean, you can argue that once you have different… That that will be an inevitability.

Joscha Bach: No it’s not inevitable. You can always make a suboptimal choice. And then you can try to prevent everybody else to make a better choice than you did. And if you are successful, then you might prevail. But if you are in an environment where you can have free competition, then in principle, your choice is just an instrument of evolution.

Lucas Perry: So can we relate this global coordination problem of beneficial futures to AGI and particularly Joscha, you mentioned an eco-fascist AGI. So, a scenario in which that might be possible would be something like a singleton?

Joscha Bach: Yes.

Lucas Perry: So I’m taking this question from Anthony. So, Anthony asks, if you had to choose, knowing nothing else, a future with one, several or many AGI systems, which would you choose and why, and how would you relate that to the problem of global coordination control for beneficial futures?

Joscha Bach: There might be a benefit in having multiple AGI’s, which could be the possibility of making the individual AI model.

One of the benefits of mortality is that whatever mistakes single human makes it’s over after a certain while. So basically… For instance, the Mongols allowed their khans to have absolute power and override all the existing institutions. Right? This was possible for Genghis Khan. But they basically tried to balance this by… If this person died, they would stop everything they were doing and reconsider. Right?

And this is what stopped the Mongol invasions. Right? So it was very lucky. Oh, of course they wouldn’t have needed to do this. Right? And a similar thing happened with Stalin, for instance. Right? Stalin has burned down Russia in a way or a lot of the existing culture. And it stopped after Stalin. It didn’t stop immediately and completely, but Stalinism basically stopped with Stalin. And so, in some sense, this mortality is on a control level, adaptive.

Senescence is adaptive as well so we don’t out-compete our grandkids. But we see this issue with our institutions. They are not mortal. And as a result, they become senescent because they basically have mission creep. They are no longer under the same incentives. They become more and more post-modern. Once an institution gets too big to fail the incentives inside the institution change, especially the incentives for leadership.

So something that starts out very benevolent, like the FDA turns into today’s FDA. And if the FDA was mortal right, we would be in a better situation than the FDA being an AI that is sentient. It’s just acting on its own incentives forever. Right? The FDA is currently acting on its own incentives, which are poorly aligned, increasingly poorly aligned, with the interests that we have in terms of what we think the FDA should be doing for us. When it governs us or it governs part of us.

And that is the same issue with other AGI’s. In some sense the FDA is an AGI with humans in the loop. And if we automate all this and turn it completely into a rule-base, especially a self-optimizing rule-base, then it’s going to align itself more and more with its incentives.

So if you have multiple competing agents, we can basically prevent this thing from calcifying to some degree. And we can have the opportunity to build something like a clock into this that lets the thing die at some point, and to replace it by something else. And we can decide once the thing dies by itself, what this replacement would look like, if it would be the same thing, or if it should be something slightly different. So this would be a benefit.

But outside of this it’s not obvious, right? From the perspective of a single agent, it doesn’t make sense that other agents exist. For a single human being, the existence of other human beings seems to be obviously beneficial because we don’t get that old, our brains are very small, we cannot be everywhere at the same time. So we depend on the interaction with others.

But if you lose these constraints, if you could turn yourself into some kind of machine that can be everywhere at the same time, that can be immortal, that has no limit on its information processing, it’s not obvious why you should not just be Solaris, a planet wide intelligence that is controlling the destiny of the entire planet. And why you would get better if you had competition. Right? Ideally you want to subdue everything once you are planet wide mind or a solar system wide mind or galactic mind if that’s conceivable.

Anthony Aguirre: So there are two elements of that. One is what the AI would want. And the other is…

Joscha Bach: Yes, there are two perspectives. One is you have a mind that is infinitely scalable. This mind that is infinitely scalable would not want to have any competition. At least that’s not obvious to me. It would be surprising to me if it wanted to. If you have non-scalable minds like ours that co-exist with it, then from the perspective of these non-scalable minds, it would be probably beneficial to limit the capacity of the other minds that we are bringing into existence. If it means we would want to have multiple of them. So they can compete and then that we can turn them off without everything breaking down.

Anthony Aguirre: It’s tricky because that competition is a double-edged sword. It can compete in the sense of being able to limit each other, but creating a competition between them seems necessarily going to incentivize them to enhance themselves.

Joscha Bach: It depends on how they compete. So when you have AMD and Intel competing, we consider this to be a good thing, right? When you have Apple and Microsoft competing. We also consider this to be a good thing. If we had only one of them, they would not be incentivized to innovate anymore.

And a similar thing could happen. For instance, if we had just one FDA, we have two FDA’s. So the FDA becomes like an app and you have some central oversight that makes sure that you always get inside of the box what’s written outside of the box. But beyond that, the FDA’s could decide whether they, for instance, allow you to import a medication from Canada or not without an additional trial and you would… Basically people would subscribe to the FDA they like better and we would have constant innovation and regulation.

Anthony Aguirre: But that’s a very managed competition. So it’s less clear that the US/Soviet Union competition is something that was good to encourage in that sense.

Joscha Bach: From the perspective of central Europe, it was great. Right? The CIA helped the West German unions to make sure that the West German workers had a lot to lose. And Eastern Germany was best among all the East block countries exactly because of this competition.

Anthony Aguirre: It was great given that we managed to not have a nuclear war or a land war. The concern I think is …that I have is, yes; if you have a managed competition that tends to be good, and you can sort of set the rules, there’s an ability to forestall some sort of runaway process. When the competition is kind of at the highest level for the overall, what has sort of the overall highest level of power in the whole arena, then that’s a lot less clear to me that the competition is going to be a positive thing.

Joscha Bach: It seems to me that US governance has become dramatically worse since the end of World War Two. Since the end of the Cold War. Right? There seems to be an increased sense that it doesn’t really matter what kind of foreign policy we are driving, because it’s largely an attempt to gain favors with the press. You can win the next election. So it’s about selling points. It’s not about whether regime change in Libya is actually good. Who cares?

Anthony Aguirre: Well I think that’s a decoupling between the government actions and… We don’t have to actually be effective as a society.

Joscha Bach: Exactly because we don’t have competition anymore. So once you take competition off the table, then you lose the incentives to perform well.

Anthony Aguirre: Those incentives could be redirected in some way, I think, but we haven’t figured out a good way to do that.

Joscha Bach: Exactly.

Anthony Aguirre: The only way we figured out to create the right incentives are the competition structure. I agree with that. If we were a little bit more enlightened, we would understand that we still have to be effective.

Joscha Bach: Oh, I think we understand this. The problem is that if you are trying to run on a platform of being effective against the platform that runs on being popular. And the only thing that matters in the incentive structure is popularity. Then the popular platform is going to win. Even worse, once you manage to capture the process by which the candidates are being selected, you don’t even need to be popular. You just need to be more popular than the other guy.

Lucas Perry: Is it possible for the incentive structures to come strictly internally from the AI system? It’s unclear to me what, if it actually has to come from competition in the real world pushing it to optimize harder. Could you not create similar incentive structures internally?

Joscha Bach: The question is how can you set the top level incentive correctly? Ultimately, as long as you have some dynamic in the system, I suspect the control is going to follow the incentives. And when we look at the failure of our governments, they are largely not the result of the government being malicious or extremely incompetent, but mostly just everybody at every level, following their true incentives. And I don’t know how you can change that.

Anthony Aguirre: Well then do you feel like there’s a natural or default or unavoidable set of incentives that a high level AI system is going to feel then?

Joscha Bach: Only if it sticks around, right? It doesn’t have to stick around. There is no meaning in existing. It’s only the systems that persist tend to be set up in such a way that they act as if they want to persist.

Anthony Aguirre: Well I guess my question is to what level are the incentives that a system has… Normally the incentives that a system has are kind of set by its context and the system that it’s embedded in. If we’re talking about a sort of world spanning AI system, it’s sort of in control. So it seems like the normal model of the incentives being set by the other sort of players on the stage is going to apply. And it’s not clear to me that there’s a sort of intrinsic set of incentives that are going to apply to a system like that.

Joscha Bach: So when we act, we act based on incentives, right? This means that we have to make certain commitments. There are certain things that should rather be the case instead of not be the case. Right? And these commitments, once you have them, they define in which direction you have to go. Once you make them consistent with each other and translate them into some kind of global aesthetics, some world state/world dynamic that is compatible. These preferences. And without such a global model, you probably don’t know what you are doing and you are being outmodeled by other systems that understand in which way you are deluded.

So you basically need to find a set of identification that is sustainable, that is modeling the dynamics that you are part of. So for instance, when you decide to model yourself as part of a sustainable civilization, that only makes sense to the degree that this sustainable civilization can be instantiated by your actions and supported by your identifications, right?

So the sustainable civilization is probably something that needs to be building to and able to plan a few generation into the future and act on the models that you get. And unlike ours, where all the projections somehow stop in 2100. And this is also true for an AGI, that basically the AGI would need to be wanting to stick around. And in the same way as you and me want to stick around, maybe not necessarily as individuals, but in the sense that the actions that we are contributing to our species remain as an equal in our species and the civilization is the thing that matters that we try to support and keep around, right?

So to the degree that our actions are able to serve that goal of our civilization sticking around, or our species sticking around, or life on earth sticking around, or intelligence and consciousness sticking around, this is the degree to which we can talk about whether you are effective or not. And if an individual doesn’t subscribe to this, they are probably not going to be effective with respect to that long-term goal in the same way as those that are.

Now, how could we bring such a sentient civilization about? And sentience here means that you have an agent that understands what it is, and what its role in the environment is, what its relationship to the surrounding universe is, and what the surrounding universe is. Right? It’s something that we also struggle for when we try to understand our relationships to the underlying physics is part of us becoming sentient. When we try to reverse engineer our own mind with the tools of AI; ditto. When we try to get society to work, it’s also part of that goal, right?

We want to stick around as a thinking being, as an experiencing being, or be part of that greater thing that is experiencing things and making sense of them. And now the question would be, what role is AI going to play here?

One was obviously building AI helps us to understand ourselves. There is this big danger if we make a non-living intelligence, right? If you teach the rocks how to think are the interests of the rocks aligned with ours? Wouldn’t it be better for AI’s to sterilize the planet and just set up solar cells everywhere? I don’t know that, right? It’s a big danger. I suspect that technological civilization could turn out to be a dead end and the correct solution would have been to go all biotech. So the way to make humanity sustainable is to breed some kind of queen bee organism that we are serving. It lays a little egg into all our brains and it will look completely adorable to us and sacred. And it’s going to live for very long and it’s going to depend on us that we feed it and obey its commands, right? So it keeps us around as a species.

Lucas Perry: Joscha, we’re going to transition to being a hive mind species?

Joscha Bach: Yeah, maybe.

Lucas Perry: How would you view the role of non-duality, both metaphysically and experientially here, in terms of what you’ve just talked about? And particularly in the role of potential collective coordination, if non-duality were collectively realized. So this is like closer to a hive mind? Is that a possibility?

Joscha Bach: I think that non-duality is a state in which you are mutually recognizing that you have the same interests and the same purposes. And it falls apart in the moment when you realize that you do have different interests, right? So if you’re competing for the same partner, then your non-duality is going to break down at some level. Unless you basically share your interests again and you decide that this competition is meaningless and there is an optimal solution for the distribution of relationships, right? And you will all agree on this in a mutually beneficial way.

So it’s basically a stance. It’s not something that is fundamentally changing the way that you are relating to reality. Once you get to a certain level of awareness, it’s a stance that is available to you. So everybody who is able to understand how the self is being constructed, I think, is able to enter the state that people call non-duality. It’s just, when people do this for the first time, it typically happens in a very protected environment in which there are no conflicting interests and somebody makes sure that nothing goes wrong.

Lucas Perry: Okay. So, your view is that people who abide in, for example, stable, non-dual states, that they still have their own interests, and they’re going to act in conflict with other people?

Joscha Bach: No it depends. So while you are in that state, you will have to have the state with respect to someone in a given context. And it might fill the entirety of your experience within this context and in that moment, so you don’t experience a separation from self and other in that state.

Lucas Perry: Yeah.

Joscha Bach: And the question is, is this model correct? And it’s correct if it’s a mutual thing, right? And to the degree that it is mutual, you basically merge into the same control unit and you can experience being part of the same control.

Lucas Perry: But there’s a risk of divergence of interests, which you feel breaks the non-dual state? The shared non-dual state?

Joscha Bach: It’s not a risk. It might happen, right. The way humans are set up, they might have conflicting interests. And these conflicting interests might emerge at any point.

So what you find in practice, that people practicing non-duality, they will still have conflicts between each other that will get rid of their non-dual experience.

Lucas Perry: Okay, interesting. Is there any one of the questions about consciousness you’d like to start with, Anthony? I have some interest in starting with whether the computationalist worldview actually explains the hard problem of consciousness.

Joscha Bach: So I would like to have a different than the computationalist worldview. If I get anything else to work, I’ll be glad. It’s not that I take this worldview because I’m so enamored with it, it’s just that all the alternatives that I have read about, or that occurred to me, don’t work. So it’s basically what’s left over.

Lucas Perry:What difficulties do you see with an idealist worldview that is in part or in whole implemented through computation.

Joscha Bach: So the idea of physics is that there is a causally closed description of the universe that exists somewhere, right? It’s a causally closed lowest layer. And this hypothesis that we can describe the universe like this is very successful. And the issue that it conflicts with is all the magic that we’re experiencing. Right, if something emerges in the world that we cannot explain by low level, causal interactions, that’s magic.

For instance, if you recognize that there is a relationship between the lines of your hand and your destiny that goes beyond your ability to grasp things, that is very hard to explain using physics. And so you would be required to assume that magic exists. And the idealist explanations allow you to explain magic. And it could be that consciousness is something that cannot be explained by physics, so it might be magic. And I think this is the main appeal of idealism, that it seems to open up a window for magic.

I think that the idealist worldviews mostly come down to the idea that we live in a dream. That we you don’t live in a physical universe, but in a dream universe. And it turns out, I think that this is correct, right? We obviously live in a dream universe. And the dream is dreamed by a mind on a higher plane of existence and that is implemented in the skull of a primate. In the brain of some primate that is walking around in a physical universe. This is our best hypothesis that we have. And so we can explain all the magic that you’re experiencing by the fact that indeed we live in a dream generated in that skull.

And now the question is, how does consciousness come about? How is it possible that the physical system can be experiencing things? And the answer is no, it can’t. A physical system cannot experience anything. Experience is only possible in a dream. It’s a virtual property. Our existence as experiencing beings is entirely virtual, it’s not physical. Which means we are only experiencing things inside of the model. It’s part of the model that we experience something.

For the neurons, it doesn’t feel like anything to do this. For the brain, it doesn’t feel like anything. But it would be very useful for the brain, what it would be like to be a person. So it generates a story about that person. About the feelings of that person, the relationship that this person has with the environment, and it acts on that model. And the model gets updated, as part of that behavior.

Lucas Perry: You say, neurons are unconscious, but the model is a virtual world, which is what consciousness is. But it seems like there’s a gap there between one neuron, two neurons, n neurons, the model, and then how the model actually is grounded in being as an actual experience, rather than as a philosophical zombie model.

Joscha Bach: The gap is quite similar to the gap that you have between transistors and software, right? There is certain dynamics that happen inside of your computer and it’s basically a pattern that you project into your computer to make sense of what the transistors are doing. And the transistors are in this case, are also set up to produce exactly that pattern.

The reason why the neurons are set up to produce this kind of pattern, so we can consistently project it into, is because the utility of the pattern is determined by the regulation tasks of the brain, right? The brain needs to make certain models in order to be effective. And the neurons are set up to produce this model, so you can consistently project it into. And as a result, the model is reverse engineering its relationship to the environment and itself, and updating this. So you get written into the story, what type of agent you are.

And the more you understand what type of agent you are, the more fine-grained the model that you can make about your own modeling capabilities. The more self-aware you become as a conscious being. But the update only happens inside of the model. It’s only the story that gets updated and you can never leave this model. You’re only confined, you’re locked into the model. You can never break out into physics, because outside in physics, it doesn’t feel like anything. Nothing can feel anything out there, right? You cannot exist outside of the dream. You cannot wake up. You can only dream waking up and you will be in the next level dream.

Lucas Perry: Anthony, is there anything you’d like to ask as we come closer to the end here?

Anthony Aguirre: Well, I think a question that has been on my mind a lot in terms of AI, that I’d be curious of your views on is, to what things are sort of necessarily coupled in cognitive architectures and which ones aren’t? In the particular sense of, is it… So, we have a close connection between our sort of intelligence and effectiveness as agents who can understand the world and act on it and have a world model that is effective. And what we would call consciousness, which we also find very valuable, right? Morally valuable. We think people are sort of morally important because they have consciousness. Most of our value derives from qualia in consciousness, whether they’re positive or negative. So there’s this very tight connection in us between sort of capability and this thing that we value, even in things that aren’t that capable. So even a child, who’s not particularly capable of doing much, we still very much value the positive qualia that they experience.

One question is why should that be? Are those two inexorably connected with each other? Could we imagine a machine system that was just as effective as an agent intellectually and at making good decisions, and at getting things done, but doesn’t have the sort of qualia that we care about? A similar question to what degree does having qualia require that we have positive and negative valence to those qualia?

In other words, could we have a mind that is conscious, in sort of the same manner that ours is? It introspects, it feels like something to be that thing in the sort of, the same way that it does us. But there’s no particular preference for positive or negative things. There’s no suffering, there’s no joy, based on whether you’re pushing away or pulling toward different qualia. And so arguably, there may or may not be the sort of moral status to the preferences of that being. It might have preferences, but maybe nobody has to care about them.

And I’m curious as to how you fit these things together in your view. Because it seems like a lot of things that come as a package in our mind, may or may not be inevitably connected to each other. We can imagine different architectures that have different amounts of them and might be designed to include some of them and not the others?

Joscha Bach: First of all, to have some dimension of caring. Once we have no dimension of caring left, I think that even our perception doesn’t work anymore. So my own experience with enlightenment states, when you cycle through them, is that our degrees of disentangling yourself from your own motivations. And we have many dimensions of motivation, obviously, and when we turn them off one by one, then it turns out that we still perform mostly the same behaviors, but they are instrumental to fewer and fewer motivations, right?

So for instance, jealousy is a motivation that makes sense from an evolutionary perspective. Jealous agents might have more offspring on their own, so jealousy gets bred into a population. But for the individual, it’s not beneficial to be jealous. So if you… Because you’re not going to have more successful relationships due to jealousy. You only have bad emotions as a result. So if you can disentangle yourself from your own jealousy, you might end up statistically with slightly less offspring of your own, but individually the benefits are far surpassing that. So if you behave as if you were jealous, it’s going to only be in the context where there is some other reason that underlies this behavior that makes it meaningful by itself.

And so if you basically get rid of most of your entanglements, you end up with maybe just being motivated by love. With this willingness to connect to others and serve the sacred. And if you give up on this one, then the only thing that’s left is aesthetics. That you try to find structure in the patterns that you are seeing. And when you give up on that, then reality falls apart and becomes white noise and you fall asleep.

So there needs to be a commitment to how things ought to be at some level in your mind, I think, to keep it going. And you can become unaware of these, but something needs to keep going. So if you actually are aware of the control that you are exerting, and if you learn to modulate it, you can modulate it to the point where basically everything becomes fuzzy and you stop controlling your mental representations. But that would be the lowest level, when you stop representing the world because you no longer care about being able to track it. Then you will stop existing as any kind of sentient agent.

So in this sense, you need to have certain motivations in order to make a behavior happen. And the question is, what is the minimal motivation that you would need to put into an agent to make it sustainable if it acts out of motivation? So, if it’s self-organized and autonomous. And I think that Friston’s theory is that, the minimization of free energy is ultimately sufficient. So basically if you are able to integrate the expected reward over a long enough timeframe, as evolution in some sense is implicitly doing, then the minimization of free energy is going to give rise to all the other priors that we are born with as a shortcut to improve the convergence of the individual or the species to that global behavior.

I don’t know whether the theory is correct. It seems to be plausible to me. I don’t see many good alternatives. But I suspect that in the individual, we need these priors to make behavior happen. So, even if we want to build an AI that is learning statistics… So I would agree that you would need to put in a number of things, for instance in your language model, that is going beyond just minimizing surprises and keeping track of the structure in the data in order to make it really, really efficient.

It could be that you should need, or should add, additional things that it cares about for instance. Make an operational model of reality that allows us to do things. For instance, GPT-3 would probably get much better at arithmetic, much faster if it was also incentivized to produce certain results and get rewards for producing these results. It cares about these rewards.

Maybe one more thing. What you find in people is the range of possible states of mind that they’re currently in, right. You can be a psychopath who is not caring about qualia of others. And the type of qualia that we have are the result of picking out things with our conscious attention, so we can form indexed memories of them and have this stream of consciousness. And what we pick out is part of our perception of our mental state of the world model that we have, right?

So it’s always features in a much, much larger conceptual and perceptual graph that we highlight in their binding context. And this binding context also contains a relationship to the relevance that they have for us. So they always have some motivational color, otherwise we wouldn’t pick them out with our attention. That makes our qualia in this sense specific.

Anthony Aguirre: Yeah. We certainly do value certain qualia for their own sake, I feel. I think humans are very intrinsically motivated for particular qualia to occur.

Joscha Bach: Yes. But I suspect it’s not quite for their own sake. It’s either because they’re directly connected to (inaudible) and aversion. So for instance, to certain foods or body schemas and so on. Or because they are aesthetically relevant. Which means that they help us to find better representations or we anticipate that they might.

Lucas Perry: There’s this quote that you said that I really, really liked. You said something like, “It’s…” You said, “Art is an algorithm falling in love with the shape of the loss function itself.”

Joscha Bach: Yes. I think that an artist in some sense is confused because the artist thinks that the mental representation itself is intrinsically meaningful. It’s not instrumental to what you can do with it. It’s not instrumental to the status that you get or the food that you can get by selling it, or the impression that you can make on others, or by what you learn from applying it to your everyday life. But it’s intrinsically important.

So it’s just the observation itself that you need to capture and the ability to capture it is the important thing. And I suspect that every functional mind needs to have an art department in the same way as every society needs to have an academia that only cares about truth, regardless if it’s beneficial for the individual academics. And if you lose this, you are going to lose an important part of your regulation. But if this truth seeker or this observer is the main thing that motivates a society or a company or an individual, then they’re going to have a hard time succeeding.

For instance, if you are running a company and you are coming from academia and you are a truth seeker, you might be set up for failure, right? Because a CEO is going to task somebody with finding truth, if there is the truth deficit. Truth is not important for its own sake if you’re running a company, if you try to use your resources in the best possible way. And the same thing is going to apply to an AI that is going to allocate resources of itself to solving problems in its environment.

Lucas Perry: Joscha, I feel like I don’t know anything anymore. So I’m just going to… I’m going to end my day just being bewildered each moment. Do you have anything else you want to say just to wrap up?

Joscha Bach: I enjoyed this conversation very much with you guys. If you try to align yourself with something that is smarter than you and understands the conditions of your existence in its own better than you do, it probably means that you have to ask it what to do. And that is… The scary thing is that we might end up building something that has ultimately a different relationship to reality than us, and doesn’t really need us.

And that is one of the big dangers, I think, in AGI research. But it’s the same thing for building a technological society. You always end up building golems. And then the question is, can you stop this golem or is it going to keep walking based on the program that you gave it, until it’s too late? I don’t know how to solve that.

Lucas Perry: All right, Joscha. Maybe you gave me an ontological crisis.

Joscha Bach: I hope not. I don’t think so.

Lucas Perry: Okay.

Anthony Aguirre: I’ve been in an ontological crisis for 35 years or so. So it’s just the normal state of affairs.

Lucas Perry: Maybe mine will go on for decades. Okay. Thank you very much for coming on the podcast.

Joscha Bach: Thanks for having me.

Anthony Aguirre: Great talking to you.

Joscha Bach: Let’s hope we meet again, especially once the pandemic is over. We can do it again in person. Thanks Lucas, thanks Anthony!

Roman Yampolskiy on the Uncontrollability, Incomprehensibility, and Unexplainability of AI

March 19, 2021/in AI Alignment Podcasts, Featured, FLI Podcasts, FLI podcasts & transcripts /by Lucas Perry

Future of Life Institute · Roman Yampolskiy on the Uncontrollability, Incomprehensibility, and Unexplainability of AI

Roman’s results on the unexplainability, incomprehensibility, and uncontrollability of AI
The relationship between AI safety, control, and alignment
Virtual worlds as a proposal for solving multi-multi alignment
AI security

You can find FLI’s three new policy focused job postings here

Paper’s discussed in this episode:

On Controllability of AI

Unexplainability and Incomprehensibility of Artificial Intelligence

Unpredictability of AI

Lucas Perry: Welcome to the Future of Life Institute Podcast. I’m Lucas Perry. Today’s conversation is with Roman Yamposlkiy on uncontrollability, incomprehensibility, and unexplainability in AI systems, we also cover virtual worlds, and AI security. This episode is all about whether it’s possible to maintain the control and comprension of, as well as ability to explain systems more intelligent than us, and how that affects the possibility of beneficial outcomes from AI. Before we get into the episode, the Future of Life Institute has 3 new job postings for full-time equivalent remote policy focused positions. We’re looking for a Director of European Policy, a Policy Advocate, and a Policy Researcher. These openings will mainly be focused on AI policy and governance. Additional policy areas of interest may include lethal autonomous weapons, synthetic biology, nuclear weapons policy, and the management of existential and global catastrophic risk. You can find more details about these positions at futureoflife.org/job-postings. Link in the description. The deadline for submission is April 4th, and if you have any questions about any of these positions, feel free to reach out to jobsadmin@futureoflife.org.

Dr. Roman Yampolskiy is a Tenured Associate Professor in the department of Computer Engineering and Computer Science at the University of Louisville. He is the founding and current director of the Cyber Security Lab and an author of many books including Artificial Superintelligence: a Futuristic Approach. Roman is also a Senior member of IEEE and AGI, a member of the Kentucky Academy of Science, and a Research Advisor for the Machine Intelligence Research Institute and is an Associate of the Global Catastrophic Risk Institute.

And with that, let’s get into our conversation with Roman Yamposlkiy

All right. So right now I’m pretty curious about what has been most intellectually stimulating and captivating for you, particularly over the past few years in terms of computer science and AI, it seems like you’ve been spending essentially all your time on AI safety and alignment research.

Roman Yampolskiy: I try to spend most of my time… I still have students who I don’t want to expose to this crazy futuristic stuff. So they do kind of standard engineering work and obviously I work on that with them. But anything I’m doing full time, yeah, it’s about AI safety.

Lucas Perry: Okay, so what research directions are you most interested in?

Roman Yampolskiy: So I’m trying to understand from theoretical, not just practical, but theoretical point of view, what is possible in the field of AI safety. Can we actually create 100% safe systems? Can we understand them? Can we explain their behaviors? So all sorts of limits and impossibility results, there seems to be a lot of that type of research and mathematics and physics and other fields in standard computer science, but it feels like in AI safety, everyone just kind of said, “Well, we need to have safe machine. So let’s start working on it.” And there is 100 different proposals for what to do addressing individual little sub problems. But it’s not even obvious that the problem is solvable. So it felt like it was important to put a little bit of time to at least look at what can be done.

Lucas Perry: All right, so I’m interested in getting into that. And before we do, it’d be nice if you could unpack and explain a bit of the epistemology of and methodology of theoretical work in computer science and AI safety and alignment efforts right now, and how that is differentiated between efforts which take to apply different alignment and safety projects and explore them in systems which exist today. So what does that project of these theoretical proofs consist of? And what is it that you’re hoping to extract through theoretical proofs?

Roman Yampolskiy: So it’s similar to how this type of work is done in computer science in general, so if you look at early days of computer science with Alan Turing showing, “Okay, some problems are not even computable, some are unsolvable.” We have this theoretical Turing machine which is not really a real thing, but we can prove things about what is possible for a Turing machine given when, “Oh, okay, it can take so many steps in a given amount of time.” He was able to use contradictions to derive impossibility results such as you cannot tell if an arbitrary program has an infinite loop. So halting problem essentially.

I’m trying to do similar things, not concentrating on any particular model, of AI, I don’t know what’s going to get us to AGI, maybe it’s very big neural networks, maybe it’s something else. But I’m trying to understand, regardless of implementation, what would safety and control look like? What are the possible definitions for control we can look at? Are we interested in direct control where a system just follows orders? Are we looking for some sort of ideal advisor situation where the system knows you really well and does what’s best for you? So I try to look at all possible cases. And in each one see, well, does that actually give us what we want? Is this option safe? And are we in control in those cases? Is there an undo button, undo mechanism, where if we don’t like what’s happening, we can kind of revert back to how it was before?

Lucas Perry: Okay, so can you make more clear the applicability to practical existing systems, in particular, this infinite loop and halting impossibility result? I imagine that there would be a spectrum of how, maybe this isn’t the case, how useful in computer science and systems that exist today, certain impossibility results are. So for example, I don’t know too much about Gödel’s incompleteness theorem, is that applicable to current systems? How often do impossibility results actually inform the creation of robust and beneficial and safe computer science systems?

Roman Yampolskiy: Well, I think we greatly underestimate how common impossibility results are in science in general, I think that’s majority of cases is that you cannot do certain things. It’s very rare that you can, and we kind of do that all the time because it’s what we’re interested in, but it’s a smaller subset of all problems. Now the Turing’s result was just kind of like a first step in this direction, it was generalized. There is something called Rice’s theorem, which says, “Any non-trivial property of arbitrary programs cannot be determined.” So if we got this super intelligent code from SETI research, aliens beamed it to us, we would not be able to tell if it’s friendly or not just by looking at it. So that’s a non trivial property.

Now people correctly argue that while we’re not getting arbitrary code, we’re designing it so we don’t have to worry about Rice’s theorem, it doesn’t apply here. But first of all, in many ways, we are getting arbitrary code because most AGI projects are not doing any safety. So out of 100 projects out there, maybe 10% actually have a safety team, the rest will just kind of give us something and say, “Okay, this is what we develop.” So in a way, even that assumption is false. But even more so there is a whole set of additional capabilities you need to have to achieve control, and different fields, mathematics, computer science, economics, political theory, have their own impossibility results. And you can show that each of the sub problems is itself well known to be unsolvable. Simple things like voting, we know it’s impossible to have voting where everyone gets what they want, their true preferences, there is always some sort of a situation where somebody has to be sacrificed for a greater good or something like that. And that’s one example.

In my work, I try to kind of collect all those sub problems, which are already known to be impossible, and to show, “Okay, if we agree that this is important to be able to do, we need to agree on things.” Well, that’s impossible, we need to decide what is good long term, not just in a fixed amount of time, but for infinite amount of time. That’s impossible, and so on. And so this is kind of where it all applies to practical systems.

Lucas Perry: Okay, so we have this project of alignment and AI safety. And because it’s so new, and because it’s dealing with superintelligent systems, there’s this desire to do theoretical analysis on it in order to see whether or not safety or alignment is something that is possible under certain conditions and what that means. And so what we’re going to get into is discussing, for example, the properties of controllability, explainability and comprehensibility, particularly in relation to systems that are as smart or much smarter than us. And so there’s a desire to see what actually can be done there.

Before we get into that, to back up a little bit into something that you said earlier, why is it that if another civilization beamed us a bunch of code… And so you made the distinction here between arbitrary code and non-arbitrary code, code that we might intentionally create, why is it that we can’t tell whether or not the arbitrary code is aligned or unaligned by looking at it?

Roman Yampolskiy: So there are two options usually, you can run the code and see what happens, and you can examine the source code, but they’re essentially equivalent. The problem goes back to the halting problem, you cannot just see what it does, because it may be still running, it didn’t stop producing output, and you don’t know if it’s an infinite process, it just doesn’t stop, or you have to keep waiting longer. Now you can arbitrarily decide, “I will only look at it for 10 minutes, and after 10 minutes, I’ll decide if it’s friendly.” Well, it’s possible that after 11 minutes, it decides to kill you.

Lucas Perry: As we begin to speak about AI alignment, the concern is, for example, the risk of failures that are over a longer time periods and that are harder to see right away, right? So in computer science 101, it seems like you have people falling into infinite loops, I don’t know, within short time periods because it’s a little bit of code. But with AGI and more complex systems, the failure modes are more severe, and it seems unclear to me, for example, on what timescale these kinds of failures would emerge.

Roman Yampolskiy: Well-

Lucas Perry: And when whether they’re safe.

Roman Yampolskiy: I cannot give you a specific time exactly, because that’s the problem. You don’t know how long you have to inspect the code, or run the program for because it may take longer always, there is not a specific set of time. So it could be an infinite process.

Lucas Perry: Is there anything else here that you think it would be helpful to add about impossibility results and their importance in clarifying and bounding problems and the kinds of solutions that can come from them?

Roman Yampolskiy: I think we want to understand what we can do with those systems. And it seems that it is not possible to predict their behavior ahead of time, if they are sufficiently smarter than us. It is not possible to explain decisions they do produce, and in cases where they are capable of explaining their decisions, we’re not smart to understand them. So there seems to be a whole set of additional problems in this debugging and safety process.

Now, I see it and it’s not a general agreement in a field, alignment is just a subset of AI safety, a particular approach to doing it, and alignment itself is problematic. First of all, who are you aligning with? Is it an individual? Or is it eight billion people, if it’s eight billion people, we run into problems I described with voting mechanisms and agreement, you don’t have multi agent, value alignment, problem solved. We don’t agree on anything. Any election, you look at about half the people are miserable. Even if we had some sort of a beautiful algorithm for aligning all the humans individually into one coherent set of preferences, that set of preferences itself may not be safe. Most people are not safe, we’re just kind of act well within our local moral code. But if taken outside of it later on, this will be considered a very unsafe set of behaviors. So we need a system which is aligned to a set of values which are changing, inconsistent, and we don’t even agree on what they are.

So it seems like there is not just impossibility at one level, and everything else just falls neatly into place, but it’s a kind of fractal nature of safety problem. The more you zoom in, the more you see additional problems with no solutions.

Lucas Perry: Okay, so let’s take a look at some impossibility results or some ways that a solution might be constrained that you’ve looked at, in particular around unexplainability, incomprehensibility and uncontrollability. So let’s start off with unexplainability and incomprehensibility. What are the results that you’ve found for these two aspects of AI safety?

Roman Yampolskiy: So the way we’re doing it right now with large neural networks, you have a system which looks at, let’s say, thousands of features, and assigns weights to them, could be millions, could be billions, doesn’t matter. The decision is made based on all these features, they will impact the decision. But any type of explanation you would want to give would be a simplification of that, you cannot really just share your weights for the neural network and go, “This is why I did it.” It doesn’t really tell a human anything. So you have to reduce it.

Let’s say you are deciding on giving a loan to someone, you’re evaluating their loan applications. Maybe you look at just 100 features, you have 100 feature vector explaining why they got denied a loan, will probably be based on one or two features, “Okay, you have bad credit,” or something. So you’re not really telling them how the decision was made, you give them just so story, a simplification, kind of like what we do with kids. When a child is asking a complex question, you usually give them some simple story just to kind of make them happy, but they don’t get full comprehension out of it.

On the other hand, if you do provide a sufficiently complex explanation, something another superintelligence will be fine with, a human is not smart enough to comprehend that. And we see it with people so even within the more narrow range of abilities different humans have, not everyone gets quantum computing, you need certain level of intelligence before you can take quantum physics courses meaningfully. That’s why we test students before admitting them to programs. Otherwise, it’s just a waste of everyone’s time.

So those are complimentary results, you cannot fully explain and the lower capability agent cannot fully understand. And we see it with humans as well, if you ask a person to explain how they make certain decisions, they’re unable to do so, and then they do kind of come up with some explanation, it’s usually greatly simplified one.

Lucas Perry: Okay, so we’re focusing here then on explainability. And so what you’re saying is, by features you mean there are properties or metrics of any given situation that one can feed into some kind of decision making process in order to determine what to do, right? So whether to classify this image as a dog, or whether to disperse a loan to someone. And as systems get more and more intelligent, the more and more parameters in the decision making will be complexifying increasingly, and eventually get to a place where it will be unintelligible to humans on any timescale or on a short timescale.

Roman Yampolskiy: Well, I don’t think timescale is meaningful here, it’s more about overall complexity, you simply cannot handle that amount of information. It goes beyond standard human processing capabilities. If you say we have short-term memory of maybe seven units, and the system moves at a million variables at the same time, you simply cannot fit that type of decision making into your computational process. Now, you can do more with tools, if you can write down things, if you have narrow AIs helping you, computers, you can do better than unaided human, but still there are significant differences between what is possible and what we can do.

Lucas Perry: So what does it exactly mean then to claim that decisions made by increasingly complex AI systems are unexplainable?

Roman Yampolskiy: It means that you will not understand how that decision was made. Again, it helps to look at differences between agents we have today. Let’s say you have a young child, you can tell them about something you’re engaging with, some sort of process, but they would not understand it, and your explanation would not be sufficient for them to get it.

Lucas Perry: Well, I mean, it might be because oftentimes adults can compress decision making into a sentence that is intelligible by children, which is lossy, but is explainable.

Roman Yampolskiy: Right. You are exactly correct. It’s either a lossless compression or lossy. If it’s lossy, you providing some sort of a basic explanation, but it’s not an accurate one, you lost information about the decision making process.

Lucas Perry: Is there a difference, though, then between unexplainability and lossy explainability?

Roman Yampolskiy: If you take it to the extreme, I think they’re going to converge. It depends on how much of the intelligence gap you have between two agents. If you are nearly identical, and you just kind of skipping the last variable, maybe it’s not that important, and you can just say, “Okay, it’s a lossy explanation.” But as the gap increases, greater and greater percentage of decision will not be explained by what is provided.

Lucas Perry: Yeah, that makes sense. But in the realm of AI alignment and AI safety, isn’t the question whether or not the movement to increasingly lossy explanation allows us to have control and alignment or not? We can take a toy example like children and adults, and explanations from adults to children are always lossy, with respect to explaining why a decision was made. And so with systems that are much more intelligent than us, it seems reasonable to expect that the relationship would be similar, right? But the human adults are aligned in good cases, to the wellbeing of children. So the child can make requests of the adult and the fact that there are lossy explanations for decisions isn’t really a situation that breaks alignment necessarily.

Roman Yampolskiy: I agree with you, you can certainly use that for ideal advisor control where you do say, “Okay, the system is way smarter than me, I’m going to put it in control of my life, and it’s going to make decisions for me.” But you just surrendered control to this other agent, you are not in control. And now the question is, is it actually successfully doing what you want? I kind of mentioned that I don’t think alignment is meaningful. I don’t think there is anything it can actually align to with respect to all of us. Maybe for an individual you can have some sort of a dictatorship situation set up, but how does this resolve for billions of agents?

Lucas Perry: All right, I think it’d be interesting to talk a little bit about that later. But now just simply in terms of the impossibility results, is there anything else you’d like to add about unexplainability and how you think it feeds into informing alignment and safety?

Roman Yampolskiy: Well, I think it’s a toolbox of different results, which all are kind of useful to show a bigger picture. So each one of them maybe individually, you can say, “Okay, it is not that important that we understand, as long as the outcomes are good.” But there is a whole bunch of those. Unpredictability is another one, we cannot predict what the decisions will be for sufficiently complex systems. So when you take all of them, we essentially find ourselves completely at the mercy of those agents in every respect, we don’t understand them, we cannot predict what they’re going to do, we are not in control. I mean, if you think it’s a sustainable situation, the best an option you can have, but it’s not, I think, universally agreed on.

Lucas Perry: Okay, so let’s get into more of these results, then. Can you explain your results on comprehensibility?

Roman Yampolskiy: So it’s a complimentary result to unexplainability. And, as I said, it has to do with difference in capabilities between agents. Simply you have to be a certain level of smart to develop certain algorithms, you have to be certain level of smart to comprehend how a certain algorithm works. And if you’re not, you’ll simply won’t to get it. And depending on how big the gap is between the agents, it could be a huge problem or a small problem. But if we’re talking about superintelligent systems, which might be thousands, millions of times smarter than us, that gap is just not something we can cross, in my opinion.

Lucas Perry: So with regards to explainability and comprehensibility, in my poor understanding of alignment, it seems like there are proposals which seek to maintain the status of alignment per step of increasing intelligence, so that confidence in alignment is maintained over time, such that explanations and comprehension for systems that are smarter than human beings has a higher credence of it being accurate or true. Can you add any perspective on this? Or is this wrong in any way?

Roman Yampolskiy: Are you saying you’ll have a chain of agents with progressively slight increase of intelligence helping to control the next agent of progressively higher intelligence and so on?

Lucas Perry: Right, so I mean, one proposal that comes to mind is iterated distillation and amplification, these kind of sequential tiered processes where you’re increasing the level of the agent, while also having other agents help to maintain alignment for each step. And so as that intelligence increase happens, the system may becoming increasingly unexplainable and incomprehensible, but alignment is maintained per step of the process.

Roman Yampolskiy: It’s a great proposal, I’m happy people are working on it. My concerns are one, the complexity increases, as you have more steps, more agents, there is more possibilities for error. That’s just standard problem with software development. Additionally, I’m not sure this whole idea of alignment can work. So how do you maintain something which even doesn’t make sense initially? Again, I’m trying to-

Lucas Perry: Yeah, because of multi-multi, is what you’re talking about.

Roman Yampolskiy: Well, at all levels. So first of all, yes, we have multiple agents who don’t agree on values. Then an individual agent who does, let’s say, agree on values may have unsafe values. So it’s not obvious who is in charge of filtering out what is safe and what is not safe to have as a value. And then on top of it, you change your values, what you consider desirable right now at your current state, your level, your affordances may be completely different. And for most people, it is completely different, things I wanted when I was six years old are not the same things I want today, thank God.

So it seems to be we are trying to hit this target which doesn’t exist, keeps moving and changing. I’m not asking anyone to say, “Okay, this is an algorithm for how to do it.” But at least an explanation of what it is we’re trying to accomplish, which goes beyond just kind of saying, “Well, do what we want,” would be a good first step, and then we can talk about making this process more reliable, some sort of additional verification. Yes, absolutely.

Lucas Perry: Okay, so let’s bring in uncontrollability here. What are your arguments and proof in views around the uncontrollability of AI systems?

Roman Yampolskiy: So I tried to, as I said, look at different definitions for what we mean by control. The simplest one people usually start with is direct orders, I tell the system what to do, it follows verbatim, “Okay, this is the order, this is what I’m going to do.” This is a kind of standard Genie problem, right? You have three wishes, and you come to greatly regret everything you wished for and just hope to undo damage. This is not even taking into account malevolent users who have malevolent wishes and want to destroy and cause harm. But even if you ask for something you think is good, it is not obvious that this should be implemented, especially again, because other agents may strongly disagree with you. So I think it’s easy to see that direct control is unsafe.

Lucas Perry: Sure. So could you unpack this a bit more?

Roman Yampolskiy: Right. So in the paper on that topic, I use example of self-driving cars. The direct order would be, “Stop the car.” So you on a highway, you’re going, I don’t know, 65 miles an hour, and you give an order, “Stop the car,” and the car just slams the brakes, and you stopped in the middle of traffic on the highway, that was direct following of your orders. Maybe somebody hit you from behind, maybe something else happens, but that’s the situation. That seems unsafe. Systems should not just do whatever random human tells it to do, especially a very powerful system.

Lucas Perry: Yeah, so it seems to be a distinction here between then alignment and controllability.

Roman Yampolskiy: Absolutely. As I said, alignment is just a small subset of this bigger problem of safety and control. Direct orders do not have any alignment in them. If you’re looking for alignment, that would be indirect control where you have an ideal advisor which knows you, knows your preferences, has your model, and tries to do what you would want accomplished. So at this low level of direct control, alignment doesn’t enter into the picture.

Lucas Perry: Why is alignment a subset of control and safety, in your view?

Roman Yampolskiy: Well, the problem we’re trying to solve, in my view, is to make safe and controllable intelligent systems. Alignment is one proposal for doing that. It’s not the only proposal, people had other ideas, that’s why it’s a subset. So if you look at three laws of robotics, right, just giving very hard-coded rules, that’s a proposal for accomplishing safety and control, but we don’t think it’s a good one, but it’s not alignment based.

Lucas Perry: Okay. It’s an interesting way of breaking it up. It’s slightly unintuitive to me, because it seems like the definition and meaning of control and safety, for example, rest upon and are contingent upon human values, and the meaning with which we ascribe to what it means for something to be controllable or safe. For example, safe is a very value loaded word. And so to say that alignment is a subset rather than a superset of that seems, maybe it’s not so linear, maybe there’s a bunch of overlap, but it’s a little unintuitive to me.

Roman Yampolskiy: It’s probably unintuitive for most AI safety researchers. But I think it’s both historically accurate, we started thinking about this before alignment became a dominant idea. So if you look at science fiction, early days, they definitely worried about safety with robots and such, but alignment was not the solution proposed at the time. So the field of safety was there, solutions were proposed, but alignment was not on the table yet. You are completely correct that terms like safety and control are value dependent, that’s why I try to look at every possible interpretation of what it means to be in control, direct control, indirect control, any type of combinations of those two, there is also the hybrid approach. So yeah, if there is something missing, let me know, we can analyze that type of control as well.

Lucas Perry: Okay, can you explain the relationship between these three facets that we’ve talked about now? So between unexplainability and comprehensibility and uncontrollability, and how you view them in relationship to what AI alignment means to you?

Roman Yampolskiy: So when I talk about control, in order to be in control, to have meaningful control, you have to understand the situation. You understand what’s happening and why, and what’s likely to happen. You’re in control. If you definitely don’t know what’s going to happen, you don’t understand the things which are already happening, and you have no way of influencing future decisions, you are not giving direct orders, the system is just doing whatever it wants and thinks you would like, you are not controlling this agent, you are subject to full control by that agent. And depending on the setup, you may not have any recourse, you may not have an undo button saying, “I just want to go back to 2020.” So that’s my view of how all this results…

And there are additional results, we have an ambiguity of language for issuing orders, there are results on unverifiability of code and programs. So all of them, to me, indicate additional issues with claims of control. So you have to decide what it is you you’re trying to accomplish. Once you decide what type of relationship you want with this agent, I’m trying to look at that specific type of relationship and see, does that create any problems? Are there ways it can fail? Is it safe? And do you. have meaningful decision power over future outcomes of your existence?

Lucas Perry: Okay.

Roman Yampolskiy: Some of them rely on alignment, like ideal advisor, indirect control, some do not rely on alignment, like direct control.

Lucas Perry: Okay. So how do you see the results of these different facets, ultimately constraining what control, safety and alignment research and efforts should be like in the 21st century?

Roman Yampolskiy: Well, I think the fundamental question here is, can we succeed? So is it a question of us just having more resources, more researchers taking a little more time, and then yes, it’s possible? Or it’s just impossible, and every day we get closer to a problem we cannot solve? If our people agree with my assessment, and it is not a solvable problem, then our actions should be very different from, “Yeah, we can definitely solve it, just give us another billion dollars in research grants.”

Lucas Perry: What does it mean to be able to solve the problem? In terms of building beneficial AI systems.

Roman Yampolskiy: So the two requirements I would like to see is that humanity is in control, perhaps as a whole, not as individuals, meaning we can undo any changes which happen, and the safety requirement, meaning we are not in danger of existential crisis, we are not negatively impacted, we are not tortured or anything like that. So if those two basic requirements are met, maybe it’s a pretty good solution.

Lucas Perry: Okay. So when we claim the impossibility of something, it seems to me that we would be claiming some kind of capacity of having pure knowledge or a credence that is basically 100% kind of like a theorem that has been proved from uncontroversial axioms. What is an impossibility result for something like control, explainability, and incomprehensibility actually saying here? Is it saying that we can never arrive at 100% credence or how much control explainability and comprehensibility can we actually have? I bring this up, because I can see a situation in which impossibility results are not exactly helpful or applicable or useful in the real world, because they deal with some kind of epistemic statement, like you can’t have 100% credence over what this thing will do, but maybe we can have very, very high degrees of certainty.

Roman Yampolskiy: Right. So this is a great way of looking at it. Let’s say you definitely agree that we cannot get 100% safe decision making out of those systems, maybe it’s 999 effective. Well, if a system makes a billion decisions a minute, then you’re guaranteed to have a failure mode your first day. If what is at stake is all of humanity, that is not an acceptable safety target. When we’re talking about standard cybersecurity, yeah, we have software, yeah, it’s going to fail once in a while, somebody is going to lose a credit card number. We’ll give you a new one, let’s move on. It’s a completely different safety standard for super-intelligence safety if all of humanity can be lost as a result of such a rare event where it happens once a week, once a month, whatever.

If you look at standard software industry, how often do we have a bug or hacking attack? We cannot accept the same standards which are typically used in this industry. The standard is to make you click on the “I agree” button without reading the contract, and then whatever happens is not their problem. I don’t think that same standard can be applied here. The more the systems become capable, the greater the difference, the gap, between an average human and that system is, the greater all this problems become. If the gap is five IQ points, it’s not a big deal. Explainability, predictability, this is what we have with our humans right now. But if a gap is huge, those problems become proportionately bigger, you find yourself in a situation where you are not making decisions, you don’t understand what is being done to you or for you.

Lucas Perry: What does AI alignment really mean to you? You set up this example here where you see it as a subset of control and safety, and also you bring in the difficulty of multi-multi alignment. Right, so Andrew Critch has this taxonomy of single-single, single-multi, multi-single and multi-multi alignment, where the first word in each pair is the number of humans and the second is the number of AI agents. And you argue for an impossibility result with regards to multi-multi alignment. So if you, for example, pointed out an example in politics and governance, where we’re making trade offs between mutually incompatible values. So I’m curious here, a good future to you then looks like something where humans are in control, and it seems like because you’ve supposed that multi-multi alignment is impossible, in 2019, I believe you proposed personal universes. Would you like to kind of unpack… I guess, I don’t know, control, the deep future, alignment, and personal universes?

Roman Yampolskiy: Right. That’s a wonderful question, actually, I was about to mention that paper, and you stole my thinking on it. That’s great. So the problem I see is getting, right now, I don’t know, almost eight billion of us to agree on anything, and it seems impossible. What if we can use future technology, which is developing also very quickly, alongside with AI research, virtual reality, we’re getting very good at creating experiences for people which look real, they feel real, people enjoy spending more and more time in virtual reality. If you project it far enough into the future, you can get to the level of performance where you can’t tell if something is real, or virtual, especially if you can modify memory enough to erase the point of entering the virtual reality. You can have this perfectly, visually, texturally in every sense of a world, real experiences.

Now, if I can create an individual virtual reality for every agent, we no longer have multi agent agreement problem, I can give each agent exactly what they want in their own universe, you don’t have to agree. You still have options, you can be without agents, you can travel to different universes, but you can also have your own universe where you decide what is the proper values to align with. So I think, while it doesn’t solve the fundamental problem of controlling substrate, and which all this computation is running, and superintelligence controlling it, at least it addresses one of the biggest, in my opinion, unsolved problems of value alignment, how to get all humans into an agreement.

Lucas Perry: Right, so this is born of… Your position then is that alignment is impossible, given the divergence of and incompatibility of many human values. And so what we want is control and safety. And given the impossibility of multi-multi alignment, what we go ahead and do is create some large number of virtual or simulated worlds which allow for the expression of multiple human values to coexist in our base reality. And the key thing that you said here was, “Then we will have agreement.” Right? You’ll have agreement between humans. So it’s a proposal I think that supposes a very particular kind of metaethics, right? It’s valuing agreement of human beings and the permissibility in expression of their values over anything else, which is its own kind of met ethical commitment, right?

So it’s almost a kind of relativism, a relativism that human preferences are empirical facts, and some of them are incompatible with others. So the only way to have agreement which means humans aren’t trying to actively, I don’t know, kill each other, which saying that is worse than the expression of certain kinds of values is a particular kind of metaethical commitment. But it’s basically a vehicle for allowing the proliferation and an expression of moral relativism. Is this a fair characterization?

Roman Yampolskiy: Absolutely. And people have argued that, yes, you can have universes where there is tremendous amount of suffering because someone is a psychopath, and that’s what they in to. And there is need for lots of additional research to see if that can be addressed. If qualia is something we can control and maybe that could be an additional step where, okay, you can have all sorts of fun in your universe, but other non-playing characters in your universe are not suffering. And that seems to be solvable, technically, since they don’t have their own values we have to follow, we can design them from scratch. They’re not existing agents.

Lucas Perry: So do you think that ethical or moral statements like suffering is bad, have a truth value?

Roman Yampolskiy: In that regard? Yes, I do.

Lucas Perry: So do you think that third-person truth about it can be arrived at?

Roman Yampolskiy: About suffering being-

Lucas Perry: Bad.

Roman Yampolskiy: … the only true thing in the universe?

Lucas Perry: Of disvalue.

Roman Yampolskiy: I think so.

Lucas Perry: So if it’s possible to get truth or false statements about what value or disvalue in existence means, in being means, wouldn’t the personal universes proposal violate that kind of epistemological confidence in discovering the truth or false value of moral statements, right? Because it seems like you think that relativism is not true, because you think that, for example, suffering can be said to be bad, probably we could discover that. But the personal universes hypothesis is, it’s no longer interested in pursuing whether or not that kind of truth is discoverable, and is allowing the proliferation of, for example, suffering in some universes if the person is a psychopath.

Roman Yampolskiy: So for me, the question is, what are we comparing this outcome to? If we comparing to the base reality we have right now, suffering is already a situation on the ground, we’re not making it worse, there is a good chance that many of those universes will not have psychopaths in charge. Also, as I said, we can look exactly how non-playing characters have to or don’t have to have real experiences. They can all be philosophical zombies. So whatever you’re into, you still get that experience on your end, but it’s just game characters, they’re not actually suffering. So I think there is a solution to that problem, or at least improvement on state of the art with this universe.

Lucas Perry: Right. But it seems like the personal universes hypothesis is a way of trying to get over, for example, metaethical policing. So if I think really hard, and think that I have answers to foundational, metaethical questions, or I have some kind of, I don’t know, direct kind of knowing about the truth of moral statements, it seems like the personal universe hypothesis is trying to get around the forcible imposition of moral knowledge upon other people. And so allowing the relative expression of different preferences and values. Yet, it’s also at the same time trying to constrain what you can do, for example, in that universe by… It seems like you’re proposing the introduction of something like non-player characters, like NPCs, so characters within a universe who don’t have conscious experience. Yeah, it could be in the sadistic or masochistic value system, it would be against their values, for example, to be in a universe with NPCs.

Roman Yampolskiy: Right, so you can look at extreme cases and definitely, there are unsolved problems. I’m not claiming that that is a perfect solution to anything. I do think that for most people, again, we have nearly eight billion people, this philosophical debate about moral values is less interesting than their basic preferences in terms of what temperature I want my universe to be at. I like it hot, some like it’s cold, okay, I like free burgers. This is the type of daily reality I’m envisioning for these people. They’re not so much engaged in understanding ultimate knowledge about morality, but just having experiences they want. I don’t want to live in a communist country, I want to live in a capitalist one. Things of that nature, basic human experiences in terms of food, sex, adventures, there’ll certainly be philosophers who’d enjoy just having those podcast debates about the nature of a substrate reality and, “Is this a multiverse? Are we living in a simulation?” But this seems to be solving some real problems for people who will be impacted by artificial intelligence when it is deployed.

Lucas Perry: Yeah, sure. So I don’t want to get too lost in the future. But I mean, that would seem to be satisfactory only for a very short period of time. So I think the only thing I was trying to suggest is that if you have some inclination towards thinking that we can arrive at knowledge about moral truth claims, and particularly around metaethics, that means that the problem of multi-multi alignment rests upon our ability to come to that knowledge, right?

Roman Yampolskiy: Just because we have true knowledge doesn’t mean people are aligned with it, there is people who think the earth is flat. The fact that we have correct tensors means nothing for them agreeing with you, or being happy in your universe. Customer is always right, you have to satisfy their preferences, not true knowledge. It’s possible that the superintelligence will discover that all of us are wrong about almost everything and will be very unhappy in that other world where, “Okay, things are scientifically true, but we’re miserable.”

Lucas Perry: Right. I think that there will be maybe edge cases of that. But I think what I’m proposing is that the personal universes hypothesis is pretty uncompelling for me, to the extent to which there is a… that all people want the truth, essentially. That seems practically universal. And even people that believe things that are obviously false, they think that they’re true, and they’re acting in good faith towards that end. But if there is a truth about moral statements, then it seems like there would be an obviously… and I think there might be a necessary relationship between what is true and good. And thus, there’s a direct relationship between truth and wellbeing. And insofar as that everyone wants what is true, and what is also good for them, they want to be free of suffering, it would seem that the personal universe hypothesis would kind of throw… it would jam the wheel of progress of collective convergence upon what is true and good, by allowing people to fall into and express their, I don’t know, their disagreements.

Roman Yampolskiy: So is just a set of options, you don’t have to have your own universe, you can stay in the base reality, you can be with other and disagree and agree to compromise, that’s an option. I like offering options. Most people most of the time have no problem going with fake experiences, fake knowledge. Every time you go to the movies, every time you read a science fiction book, you willingly go into this other world, which you know is not true reality, but you’re happier there. And we do it a lot. We really don’t mind paying other people to create fake experiences for us, we just want experiences. And if it’s possible to manipulate your memory to where you really can’t tell the difference between what’s real and what’s fake, I guarantee you that industry will be very desirable. Lots of people will want to have amazing experiences, even if they’re just dreams.

Lucas Perry: Yeah, I agree with that. I think that’s true. So one thing that you’re… And we’re taking a bit of a pivot here. One thing that you’re quite well known for is the realm of AI security. So I’m curious if you could talk a bit about issues regarding AI security from current day systems all the way to AGI.

Roman Yampolskiy: So I think concern is malevolent agents. We have companies, DeepMind, OpenAI, developing very powerful systems, which will probably get a lot more powerful. Lately, they’ve been keeping the source code closed as much as they could. But if they are hacked, if the code leaks and malevolent agents get access to it, that could be another area of concern. So we’re not just trying to get the good guys to align the system to design it properly, but now we also have to worry about deliberate misuse. And that seems to be a much harder problem because you’re not just thinking how to make the code proper, but also how to control our intelligent agents and what they do. And they’re definitely misaligned with your goals of creating safe beneficial AI, the same people who are right now writing computer viruses will definitely go to the next level of technology for massive scale, social engineering attacks for blackmail for any type of crime you can automate with this more advanced AI.

Lucas Perry: Sorry, so what do you think the relative risk profile is between malevolent AI and misaligned AI? Does security not apply to both?

Roman Yampolskiy: The worst, most challenging problem is malevolent purposeful design. Because you still get all the same problems you had before, you have bugs in code, you have misalignment, but now you have additional payload, malevolent payloads specifically added to the system. So it’s strictly worse.

Lucas Perry: Yeah, but is it more likely?

Roman Yampolskiy: Is it likely that there are people who want to commit crimes using advanced technology? Yeah.

Lucas Perry: Right. But in the realm of global catastrophic and existential risk, given the level and degree of security at organizations like DeepMind and the kinds of resources that are needed to, I don’t know, deploy and train a misaligned powerful AI system, what is more risky?

Roman Yampolskiy: Well, those resources seem to be becoming more accessible, anyone can rent cloud services from Amazon, lots of governments invested in advanced weaponry, so they would be willing to hack into American companies, we’ve seen that before, it’s not unusual. Also, you have, again, insider threat, an employee at a company, we’ve seen that happen a lot, who takes the source files and makes them accessible to others for profit, or for personal reasons. You can blackmail employees, you can do all sorts of things if this as your target. So I don’t think there is any companies, or even government agencies, which have not been hacked so far.

Lucas Perry: So that’s one dimension of this. There’s also security in relation to general systems in general. So this is, I don’t know, air gapping and boxing, as I think security measures against systems which we’re trying to test for whether or not they are controllable and aligned. Can you explain a little bit more about the aspect of security in terms of constraining the causal power of systems that haven’t been deemed safe yet?

Roman Yampolskiy: Right. So the difference is external attacker, so internal attacker AI itself, you’re trying to build a prison, or you’re trying to keep people out of the prison. And the standard name I think is AI boxing, you’re trying to put AI in this limited environment where you control all the resources going in, all the information and, even more importantly, controlling output, so it is not able to engage in social engineering attacks. I think everyone who looked at that agrees it’s impossible to do it permanently as a long-term solution. But it could be a useful tool to study the systems under development for short amounts of time. So it’s good to have those tools, virtual environments, where we do have a lot of control over systems as they become more capable. But if you are observing the system, it can impact you and most likely will find ways to influence you to release it.

Lucas Perry: So I mentioned air gapping and boxing, perhaps, in some kind of virtual environment. Are there any other security apparatuses that are useful here for keeping the thing inside of the jail?

Roman Yampolskiy: So people talk about encrypting the code itself. So there is some very advanced encryption tools, which allow you to run encrypted code on encrypted data. It’s super slow, it’s not efficient, it will slow down research a million times. But in theory, it’s possible. So the system would not have access to its own source code for additional modification improvement. So that’s something probably worth looking at.

Again, this is in direct proportion to the resources you have, you can make it safer, proportionate, you can have multiple levels of virtual worlds, separated hardware, you can have separation of controls. A lot of times with boxing experiments, the classic boxing experiments, the guard has an option of releasing an AI, why would he have that option, you should never have an option of actually releasing it. That’s already kind of unrealistic example. Prison guards don’t get to release prisoners, it’s not their decision. So you need to have multiple levels of indirection for anything like that to happen. But still, I think, if you are observing the system, information is leaking out, and that’s enough long-term periods to escape.

Lucas Perry: I see. So you’re suggesting layers of redundancy in pointing out this fact that if there’s any channel of being able to observe something, that is, in and of itself, leaky with regards to the inside to the out. So is there no such things such as a one way mirror in computer science for boxing such a system?

Roman Yampolskiy: Well, if you as a human observing behavior of a system, you getting information out of the box into the world. If we’re not observing the system, then it’s useless. We’re not getting any experimental knowledge from it.

Lucas Perry: Right. But if a superintelligence is in a black box, and it has no world model, it’s only in a simulated environment and doesn’t know anything about the humans, and the window of observation for that system doesn’t give that system any more information about who’s observing it, then how would it break out of the box without knowing anything about the outside world?

Roman Yampolskiy: So this is where unpredictability is a good excuse, I cannot predict what a superintelligent system would do. My guess is you can learn a lot about designers of the world by looking at the world, how it’s made, if you have access to the source code for the system, you can see, “Okay, they’re very slow agents based on update times for the code. They really not that intelligent, given the bugs and imperfections and inefficiencies in the code.” And this type of forensic analysis can be done to figure out what the programmers are, who they are in terms of their capabilities.

Lucas Perry: I see. So forensic analysis of the AI’s own code, or the simulated world would have enough evidence and information in it to leverage in order to break out of the box if humans are observing it.

Roman Yampolskiy: At a basic level, it may be able to discover loopholes, just zero day exploits we didn’t know existed there. Second level, it can do some… depending on access to resources it has, it can perform novel experiments, maybe discovering novel physical properties. There are some attacks and computer systems based on writing to certain parts of the memory, rewriting them frequently creating side effects. You can look at how hacking is done from virtual worlds from video games, people were able to break out of Mario and get access to the operating system. And that’s pretty weird to invest your time into that, but people did that. And again, I simply cannot think what much, much smarter system would be able to do. If I could, I would be a much smarter system.

Lucas Perry: Yeah. Okay. So we’ve had a lot of failure of AI systems. Can you unpack some of your favorite historical examples of AI failures, and what that tells us about how the future of AI technology will likely also include failures?

Roman Yampolskiy: So I collect those examples. So if your listeners have any cool examples not already in my publications, I’d be very happy to get them. The pattern I found is that basically no matter what the system is designed for, so it’s designed to do X, it will fail to X. If it’s a spell checker, it will fail to correctly fix your word, give you a wrong word. If it’s a self-driving car, it will have a car accident, it doesn’t really matter. Basically, eventually systems, no matter how accurate they are, 99.99%, they will fail at it. That’s one type of problems we saw.

Another problem is people under estimating how creative other humans are in terms of what they’re going to do with the system, how are we going to use it or abuse it? Before releasing any AI service or product, it’s really worth spending some time thinking, “Well, what’s the worst thing they can do with this?” So we’ve seen examples like chatbot, which is trained by users on Twitter, I don’t think you have to be a superintelligence researcher to predict what’s going to happen there. Yet, Microsoft did that, released it, and was quite embarrassing for the company.

I like examples from evolutionary computation a lot, because they always surprise their designers. And mostly the surprise is that the system will always find a lazy way to accomplish something, you want it to do it the hard way, it will find the shortcut. And basically cheating, like cheating students, cheating businessman, it will always exploit the lowest possible way to get to the target.

Lucas Perry: Yeah okay, so the last example is reward hacking, or at least that’s what comes to my mind. Well, it’s achieving the objective in a way that is not what we actually intended, right? Or the objective is a proxy for something else and it achieves the proxy objective without achieving the actual objective.

Roman Yampolskiy: There is probably an infinite way of doing almost anything in the world, and it will find the one which takes the least resources.

Lucas Perry: And that is not always aligned with the actual objective.

Roman Yampolskiy: It is almost never aligned with the actual objectives. Because on the way on the path there, you can do a lot of damage, you can cause a lot of damage or bypass all the benefits. So you almost have to specify all the points in the path to the target to get what you want.

Lucas Perry: And so the first thing you said… It seems like, again, you are also bringing in the impossibility results, because you’re talking about this distinction between 100% certainty of how a system will behave and 99.99% certainty. And what you mentioned earlier was that if a system is running at billions, or trillions of times faster than human computation, then very small probabilities will happen. It seems like a really important distinction between when those things happen, whether or not they’re global, catastrophic, or existential.

Roman Yampolskiy: Mm-hmm (affirmative).

Lucas Perry: Right? And how often they’re actually happening. And this is I think, getting back to some of the impossibility results, is I’m trying to really understand the distinction between saying 100% certain knowledge in the verifiability of some program as being friendly or that it won’t halt or fall into an infinite loop. It seems like there’s an important distinction between what is impossible and what is so highly probable that it won’t lead to existential or global catastrophic risk in most or all universes.

Roman Yampolskiy: Right. And we can look at, for example, mathematics, right? So Gödel’s incompleteness theorem, it didn’t stop mathematics, it didn’t stop people doing mathematical proofs. It just kind of gave us this understanding of limits of what we can accomplish. In physics, you know about, okay, there are limits on maximum speeds, there are limits on certainty and observation, it didn’t stop physics, but it gives us a more accurate picture of what we are working with. I feel it’s important to have complete understanding of intelligence as a resource. I think people talk about space, time as a resource, intelligence may be another computational resource, and we need to understand what we can accomplish with it, and if something is not possible with 100% certainty, maybe we can do it with less accuracy. But at least knowing that fact can save us the resources in not following dead-end goals.

Lucas Perry: Right. But what would be a dead-end goal that comes from these impossibility results? So earlier, at the very beginning of the conversation, you mentioned that it was important that we do this work, because it hasn’t been proven whether or not the control or alignment problems have a solution. And there’s a lot of things in life that don’t clearly have solutions. Like we can’t compute the whole universe. So how does that actually affect the research efforts and alignment and control and safety paradigms as we move forward?

Roman Yampolskiy: So let’s take a hypothetical, if you knew before experimenting with nuclear weapons that it will definitely explode the whole atmosphere, definitely, would you still develop nuclear weapons?

Lucas Perry: Then there’s no point in developing it if it’s going to ignite the atmosphere.

Roman Yampolskiy: Right. So that’s the question I’m trying to answer, if we are certain that we cannot do this in a safe way, should we be doing it? If we’re certain that it’s possible to control, then obviously, the benefits are huge. If there is a probability that actually no, you can’t, then we should look at other options.

There is a lot of suggestions for maybe delaying certain technologies in favoring others. You don’t have to do everything today. What happens if we postpone it and take five years to think about it a little more?

Lucas Perry: Right. So this is like global coordination on the speed of research, which I think people have been largely pessimistic about because there’s a story that in global capitalism and between the militaristic and imperial conflict between nations and countries that it would be impossible to slow down progress on a technology like this.

Roman Yampolskiy: I don’t disagree with that additional impossibility you mentioning, I think that’s also an impossibility to stop research without completely suppressing civilization. But it doesn’t negate the other impossibility, and if control is not possible, again, if nuclear weapons were known to be definitely the end, we’d still have the same problem between Germany, Russia, China developing nuclear weapons, they still would fear not developing them. So that in itself is not an argument to say, “Oh, we have to go forward anyways.”

Lucas Perry: So when demonstrating the impossibility of control, is that not suggesting that there is still some very high credence of control that is able to be achieved?

Roman Yampolskiy: Explain why.

Lucas Perry: Well, so if you say, for example, control of such a system is impossible. Is that kind of an absolute claim that certain control is impossible? Or is it a claim that any credence of control over a superintelligent system is impossible?

Roman Yampolskiy: So I think this is where my different types of control are very useful. Which one are you referring to? So with direct control is possible, we can tell the system exactly what to do and it will follow an order. I’m saying this is unsafe, it results in clearly negative consequences. The ideal advisor indirect control situation can be set up, but you lose control, you may obtain safety, but you lose all control. If we all agree on that, we can do it. But is that a desirable future?

Lucas Perry: So can you explain a few more ways in which systems have failed? Out of your list, could you pick a few more examples that are some of your favorites? And what that tells us about the risks of AGI?

Roman Yampolskiy: So the interesting examples are all different applications, we had autonomous systems used in military response, so detecting an attack and automatically saying, “Okay, Soviets are attacking us, Russians, or Americans are attacking us, and let’s respond.” And the system was wrong, so just this type of decision and accuracy was not high enough. We have great examples where, again, with evolutionary algorithms, we’re trying to, let’s say, evolve a system for walking. But walking is hard, so instead, it learns to make tall agents and fall forward, that’s quick and easy. So they are very interesting. You would not come up with those examples if you had human common sense. But AIs usually don’t have human common sense. So they find solutions outside of our comfort box, sort of. And I would predict that that is what we’re going to see with any boxed AI, it will find escape paths which we then consider. It will not go through the door, it will find how to penetrate a wall.

Lucas Perry: Do you think that oracles are a valid approach to AI safety or alignment given expectations of commercial and industrial creation and deployment of AI systems? Because it seems like we’re really going to want these systems to be doing things in the real world. And so they already have that role, and they’ll increasingly be occupying that role. So is it at all reasonable to expect that the most cutting edge systems will always be oracles? Are there not incentives that strongly push the systems towards being more agentive?

Roman Yampolskiy: Early systems are very likely to be oracles, but I don’t think that makes them safer. You can still ask very dangerous questions, get information, which allows you… especially with malevolent actors, to either design weapons or hack some sort of a process, economic process or business process. So just the fact that they are not actively in the world and embodied or anything like that doesn’t make them a safer solution if they’re capable enough.

Lucas Perry: Do you think that the risks of oracles versus agentive systems creating global catastrophic or existential risk is about the same?

Roman Yampolskiy: I’m not sure you can separate an oracle from it being an agent to a certain degree, it still makes decisions about the resources that needs computational access, so that is not that different.

Lucas Perry: Which has causal implications in the world.

Roman Yampolskiy: Exactly.

Lucas Perry: Okay, my vague intuition seems that it’s harder to end the world as an oracle, but maybe that’s wrong.

Roman Yampolskiy: I mean, it depends on the type of advice. So let’s say you have a COVID pandemic happening and people ask you, “Can you suggest a good medicine for it?” It gives you full access to whatever you want.

Lucas Perry: I see. It gives you microchip vaccines that all detonate in one year.

Roman Yampolskiy: Well, it could be more than one year, that’s the beauty of it, you can do something four generations later. You can make it very stealth and undetectable.

Lucas Perry: So given all of your work and everything that has been going on with AI alignment, safety and security, do you have any final thoughts or core feelings you’d like to share?

Roman Yampolskiy: Right. So I understand a lot of those ideas are very controversial and there is nothing I would want more than for someone to prove me completely wrong. If someone can write something arguing formally or mathematical proof or anything, saying that, “No, in fact, control is possible. All the additional sub requirements are met.” I would be very happy and very welcoming of this development. I don’t think it will happen, but I would like to see people at least try. It is really surprising that this most important problem has not seen any attempts in that direction outside a few short blog posts where people mostly suggest, “Well, of course, it’s possible,” or, “Of course, it’s impossible.”

Lucas Perry: All right, Roman, thanks a bunch for coming on the podcast and for speaking with me.

Roman Yampolskiy: Thank you for inviting me. Good to be back.

Stuart Russell and Zachary Kallenborn on Drone Swarms and the Riskiest Aspects of Lethal Autonomous Weapons

February 25, 2021/in Featured, FLI Podcasts, FLI podcasts & transcripts /by Lucas Perry

Future of Life Institute · Stuart Russell and Zachary Kallenborn on Drone Swarms and the Riskiest Aspects of Autonomous Weapons

The current state of the deployment and development of lethal autonomous weapons and swarm technologies
Drone swarms as a potential weapon of mass destruction
The risks of escalation, unpredictability, and proliferation with regards to autonomous weapons
The difficulty of attribution, verification, and accountability with autonomous weapons
Autonomous weapons governance as norm setting for global AI issues

You can check out the new lethal autonomous weapons website here

Lucas Perry: Welcome to the Future of Life Institute Podcast. I’m Lucas Perry. Today’s conversation is with Stuart Rusell and Zachary Kallenborn on lethal autonomous weapons. Particularly, this conversation is focused on the highest risk aspects of such weapons, such as the potential for generating large numbers of autonomous weapons through their industrial scalability, and their miniaturization, which both together could lead to swarms of lethal autonomous weapons which could then lead to their classification as weapons of mass destruction. Stuart Russell is back on the podcast for his 3rd time. He is a Professor of Computer Science and holder of the Smith-Zadeh chair in engineering at the University of California, Berkeley. He has served as the vice chair of the World Economic Forum’s Council on AI and Robotics and as an advisor to the United Nations on arms control. He is the author with Peter Norvig of the definitive and universally acclaimed textbook on AI, Artificial Intelligence: A Modern Approach. Zachary Kallenborn is a Policy Fellow at the Schar School of Policy and Government, a Research Affiliate with the Unconventional Weapons and Technology Division of the National Consortium for the Study of Terrorism and Responses to Terrorism (START), and a Senior Consultant at ABS Group. His work has been published in outlets like Foreign Policy, Slate, War on the Rocks, and the Nonproliferation Review.

The release of this episode coincides with the launch of the brand new autonomousweapons.org website. This website was an effort on the part of the Future of Life Institute and contains information about the technology of autonomous weapons and the ongoing debate about governance issues pertaining to such weapons. So, if you’re interested in learning more in addition to this podcast, head on over to autonomousweapons.org. To introduce this podcast, and a key author behind the autonomousweapons.org website, please allow me to introduce Emilia Javorsky:

Emilia Javorsky: Hi, I’m Emilia Javorsky and at my role at the Future of Life Institute I lead our advocacy efforts on lethal autonomous weapons. And a question that comes up a lot is, “Why is this issue so important?” Most people, not all, acknowledge that the words “lethal,” “autonomous,” and “weapons” in sequence sound like a very bad idea, but today we are bombarded with so many important and underserved causes to care about. Why does this issue in particular warrant taking up bandwidth in our thinking? And I’d argue, anyone who cares about a positive outcome for our collective future with AI should care about this issue. AI is and increasingly will be the most powerful technology we’ve ever developed. It has the power to completely transform society for the better, but to realize these benefits we need to develop and deploy the technology with wisdom. And this means drawing clear lines between acceptable and unacceptable uses of the technology. And these early decisions that we make will have a tremendous impact on steering the trajectory of AI towards being used in a way that is positive, it’s safe, it builds public trust, and it advances society. Or one that is destabilizing and triggers backlash and has a net negative impact on our world. Much of our celebration of biotechnology today for curing disease and generating vaccines is arguably due to the early decisions to stigmatize its weaponization in the form of the bioweapons ban. By contrast, the fallout of nuclear weapons has served to curtail the potential of peaceful uses of nuclear power for decades and the question at the heart of the lethal autonomous weapons conversation is, “Do we as a global community want to cede the decision to kill humans to algorithms, to AI?” I strongly believe that if we fail to develop a governance mechanism to gain agreement on that AI shouldn’t be allowed to kill people based on sensor data, we are completely toast with regards to our pursuit of a positive future. If we can’t set this precedent, we’re unlikely and in so many ways have morally abdicated our authority to be able to make many of the wise decisions that lie ahead that are needed to steer the deployment of AI and society to realizing its positive outcome. We have to get this question right and we have to set the precedent that humans must retain control over the use of lethal force.

And beyond this key ethical dimension of drawing a red line on uses of AI, these weapons systems would also be highly destabilizing to society. These are weapons that have been called “unpredictable by design,” meaning that it’s highly unlikely the typical training data would be sufficient to predict the behavior of these weapons in real world settings in rapidly evolving circumstances and in the context of encountering other lethal autonomous weapons systems in an adversarial context. By removing human control and judgement, there’s a strong risk of accidental or intentional escalation of conflict and the inherent scalability of technology that relies on software means that these weapons systems would pose substantial proliferation risks, not only among states, but also to non-state actors. All of these destabilizing risks are happening in the context of a global AI arms race where the incentives are seeking to rush these systems into the battlefield as soon as possible, and we’re not talking on the order of decades. The age of lethal autonomous weapons is here. There are systems that are around today that are capable of selecting and targeting humans without human control. So, the window for us to be able to draw that line is rapidly closing, and we really need to take action before it’s too late. This is why I’m really excited about the conversation that lies ahead today between Zak and Stuart who are two individuals whose work has really served to highlight both of these categories of risks both on the ethical dimensions of lethal autonomous weapons and the risks that that poses, but also the security and stability risks of these systems. So, I’m really looking forward to the conversation ahead.

Lucas Perry: All right, let’s start off here with a bit of background about this issue. And so could you guys explain simply what a lethal autonomous weapon is and what are the various forms that they currently are taking today?

Zachary Kallenborn: Yes, I suppose I’ll go ahead. There’s certainly quite a bit of discussion about what exactly do we mean by autonomous when we’re talking about these system. Primarily concern is what happens if we let machines make decisions about who to kill and who not. In terms of the specific development of that, there’s been quite a lot of developments in recent years, but it’s still a relatively nascent technology. When we’re talking about lethal autonomous weapons, I think for the most part we’re talking about formal more sophisticated weapons, like the SGR-A1 Gun Turret, developed by South Korea and deployed, I believe, at the South Korean border between South Korea and North Korea that has the capability of selecting and engaging personnel target.

That said, I think when we talk about autonomous weapons, I think it actually goes well beyond what we’re seeing at the moment, because if you think about like landmines, in a sense, landmines are a very simplistic form of autonomy. Military very explicitly exclude landmines from their discussions of lethal autonomous weapons. But certainly, in a sense it is a system that ingests information from outside the world, it makes a decision based on physical properties and physical stimuli about who to explode and who not.

Stuart Russell: I would add, just for the record so to speak, there is the beginnings of an agreed upon definition, that the United Nations, and I think to some extent, the United States think makes sense, which is a weapon that can locate, select and engage human targets without human supervision. We can define any category of weapons that we want, and you can expand or contract the definition to rule in or rule out different kinds of weapons systems.

There is no right definition, but whenever you write a definition, there are weapons that fall in and out, and you want to know, have I left out some weapons systems that I should be worried about and that do we need to be subject to regulation? Or have I included some weapon systems that are actually, as far as weapons systems ever are, innocuous and that they don’t raise any particular concerns? So engineering, these definitions is a way of actually putting off the real issue, which is what are we going to do about it?

Zachary Kallenborn: To an extent but I think there is sort of value in getting that precise definition, at least in so far as like you need to have some sort of common definition, in order to sort of engage in the global conversations about it. It’s like if I use lethal autonomous weapons to refer to not just human targets, potentially, including like vehicles or other targets, that potentially changes our understanding of where those risks are.

And as well as prohibits, potentially, movement towards something more robust to deal with policy level.

Stuart Russell: Can we talk about technologies that are being created now and advertised and sold? I think that many people still regard the possibility of lethal autonomous weapons as science fiction. And for example, quite recently, we’ve heard the Russian Ambassador to the talks in Geneva, say that these things are 20 or 30 years in the future, and it’s a complete waste of time to be talking about them now.

Zachary Kallenborn: I think that’s definitely accurate. I think it also goes beyond just lethal autonomous weapons. Only now are countries beginning to recognize just the threat of drone systems just by themselves. And that’s not even an emerging technology, we’ve had remote controlled airplanes since ’68, I think was the first remote controlled airplane. But only now are states realizing like, hey, this is a huge issue. You know, much less talking about cooperative drones working together in the form of swarms or other types of systems.

Lucas Perry: All right, so can we get a bit more specific about the kinds of systems that are being deployed today and which already exist and be a bit more specific about what we’re talking about? So, the short definition is that these are systems which can autonomously select and engage targets through the use of force. And so for, I don’t know about a decade or so, I’m not sure how many years we’ve had, for example, turrets on aircraft carriers that could autonomously engage incoming missiles. In Israel, they have the Iron Dome, there’s a defense system. And now there’s the expansion of the autonomy towards engaging and selecting human targets. It’s the beginning of the integration of machine learning and neural nets in the selection, acquisition and engagement of human targets, but also other kinds of military targets, like tanks and other kinds of vehicles.

Stuart Russell: I would argue that actually it doesn’t matter how the AI system does it. I think, to a large extent, although not completely, it doesn’t matter whether it’s created using machine learning or neural nets or Bayes nets or rule-based systems or whatever. What matters is the autonomy and the capability. So the examples, I think probably one of the best known examples that you can actually buy today and have been able to buy for several years, is the Israeli Harpy missile also called the Harop. And so this is what’s called a loitering weapon, which means a weapon that can wander around. So, it’s a small aircraft about 11 feet long, I think, and it can wander around a designated geographical region for up to 6 hours. And it’s given a targeting criterion, which has a radar signature that resembles an anti-aircraft system, or it could be looks like a tank, because it has onboard camera systems and can use image recognition algorithms as well.

Whenever it finds something that meets its targeting criterion, it can dive bomb it and it carries a 50 pound explosive payload, which is very destructive and can blow up whatever it is. So that seems to me to meet all the criteria for being a lethal autonomous weapon. A version was used as a by Azerbaijan and actually blew up a school bus, which was claimed to be carrying soldiers although I think there’s still controversy about exactly why it was targeted. Certainly didn’t look like a tank. So that’s Israel Aircraft Industries, they are advertising something now called the Mini Harpy, which is a much smaller fixed wing aircraft about maybe a 18 inch or 2 foot wing span. And the advertising materials clearly emphasizing the autonomous attack capabilities. If you go to Turkey, there’s a company called STM, which is advertising something called the Kargu drone, and they explicitly say autonomous hit capabilities, human face recognition, human tracking, human targeting, all their advertising materials, basically telling you this is a lethal autonomous weapon that you can buy in as larger quantity as you can afford and use to kill your enemies.

Zachary Kallenborn: Yeah. I want to jump back to Stuart made the point earlier about like, it doesn’t really matter what technology you talk about, whether it’s like neural nets or machine learning. I think that’s really right. Especially from a risk perspective or at the end of the day, when we’re talking about risk, what really matters is like the threat to human life and the outcome of how these weapons are used in that particular risk of accident. And to an extent that’s going to be while there may be differences and between different types of techniques that you’re using at the, in the end, like that’s really what matters is sort of safeguarding folks. And I would also add to the sort of question of like broader spread of technology. There’s definitely very clear interest across many different militaries, as you saw at the National Security Commission on Artificial Intelligence recently mentioned that the United States is interested in potentially pursuing autonomous weapons, provided that they’re world lastly, tested and have appropriate ethical controls, but they see value in it, particularly through the advantages of speed and decision-making, and because in the military conflict, that’s really fast moving, that can be extremely advantageous.

And I think that’s likely to grow even further when we talk about sort of the spread of drones and general autonomous systems that may even if they don’t necessarily have human control, or even if they have human control over targeting. I think as we have broader spread of drones and robotics, there is very clear incentives towards much higher degrees of autonomy, particularly to mitigate concerns over electronic warfare and jamming where you stop where a military stops the signals from coming into a particular robotic system, if you have an autonomous system that you don’t need to worry about that. So there may be military reasons sort of move in that direction. And I think certainly we’ve seen increasing interest in drones generally across many states, especially after the recent conflict between Armenia and Azerbaijan, where Azeri drones were quite deaf proved quite devastating to Armenian tanks and a wide variety of capabilities.

Lucas Perry: So are there any key properties or features of autonomous weapons that would be wise to highlight here that are a part of why they pose such a high risk in the evolution of conflict and global governance?

Stuart Russell: So I think historically, and this discussion has been going on for about a decade. The first set of concerns had to do with what we might call collateral damage that because we might program them poorly, or we might not understand what they’re going to do once they’re deployed that they might end up killing a large numbers of civilians by accident related to that, the idea of accidental escalation that they might misinterpret activities by a neighboring country and initiate a hostile response, which will then be interpreted by their autonomous weapons correctly as a hostile attack, leading to a very rapid escalation, possibly on a timescale that humans could not even contain it before it became too serious.

So those are, I think are still real concerns. The first one, the ability of the AI system to correctly distinguish between legitimate and non-legitimate targets and so on, is obviously a moving target in the sense that as the technology improves, we would expect the instances of collateral damage to reduce. And many people argue that in fact, we have a moral obligation to create such weapons because they can reduce collateral damage to civilians. But I think the real concern, and this has been, I think at the forefront of the AI community’s concern for the last five or six years, is that as a logical consequence of autonomy, you also get scalability. And scalability is a word that computer scientists use a lot and it basically means, can you do a thousand times as much stuff if you have a thousand times as much hardware and you know, we’ll use this know Google onsets, billions and billions of queries every day, not by having billions of billions of people, but by having lots of computers.

So they have a scalable service that they can make it twice as big, but as buying twice as many computers. And the problem is when you have scalable death, you have a weapon of mass destruction and it’s precisely because you don’t need the human to supervise and intervene in each individual attack, that you can create these scalable weapons where you just, instead of buying a handful Kargu drones, each of which is about the size of a football, you buy a couple of hundred thousand of them. And then you launch a very large-scale attack that wipes out an entire city or an entire ethnic group. And so creating a low cost, easily proliferated weapon of mass destruction, that’s probably going to be used against civilians. So even though you could make AI systems that are very careful, you could also make AI systems that aren’t very careful to be used against civilians. So why we would want to create and manufacture and distribute such a technology. I don’t know. I can’t think of a good reason to do it.

Zachary Kallenborn: Yeah. So I’ll expand about that. So, cause I feel like this is probably going to end up being a fairly long answer. I think there is particular issues with scalability and I think it goes even beyond the issues that Stuart rightly raised. And I also personally have some concerns about, even if you have improved targeting with autonomous weapons, that actually could be a concern in and of itself or other reasons, particularly related to chemical and biological warfare. So first to expand the scalability issue, I definitely share that concern around having numerous drones. I mean, we know already just from open source reporting that numerous states are exploring this technology and exploring it very quickly. So just in the past few months, there’s been announcements over the U.S Naval Postgraduate School is doing some research. What if you potentially had a million drones all operating together where some are boats, some are flying somewhere under the sea, how would you deal with that from both an offensive and defensive perspective?

Now that was like more, very early on modeling and I’m sure they’re very, very far away from actually doing something of that scale, but they’re very clearly thinking about it. There’s also a recent announcement from India. They showed off in their recent army parade, a group of 75 drones that they claimed were autonomous. That sounds like there’s some pretty clear question of whether they were and to what extent they actually collaborated as true drones swarms, but there was very clearly at least interest in that. Now I think where the concerns are, is that goes even beyond scalability, because I think the error risks around targeting becomes multi-fold more complex. So even if we assume that we have an advanced military, like the United States or someone who is very sensitive to these ethical concerns and they do a whole bunch of work to make these systems really full-proof.

Then so they, their error rate becomes really, really small. If you have like 10,000 drones, even a really small error rate may end up resulting in all sorts of error. And I think you have potential errors coming through multiple directions. So when we’re talking about drone swarms, much of the research focuses on rather than just having a whole bunch of drones. You particularly have drones that are talking to one another and able to communicate and share information from an error perspective that creates extra levels of problem. Because if you have one of those drones makes a mistake and then communicates it to all 10,000, you now have 10,000 that have made a mistake. So you have potential propagation of error all throughout the entire system. And there’s also concerns about emergent behavior where, one of the interesting properties of swarming and why many militaries are interested in it is through like the emergent properties that come from a small number of things, working with simple rules that sort of create complex behavior.

The same way, if we talk about like starlings or can’t remember what the particular bird is, where you see like these really fancy flocking formations that birds have based on really simple rules of like how they work together. So how would that work if you have from like an error perspective? Like what if your error isn’t actually in a particular autonomous system where it miss identifies like a civilian as a soldier or vice versa, but rather small bits of information that may all be correctly read, ultimately lead to problematic conclusions when you move at scale. And I think from a scaling perspective, that’s also a particularly huge issue because there’s been a number of research that look, let’s look at, how do we manage that level of complexity? And the simple reality is that humans can’t manage huge amounts of drones. And in fact, the key limiter has been drone autonomy as in terms of the scaling, because we have imagined like a security guard sitting at a desk, they can fairly easily watch a monitor, a video like 50 different video feeds because nothing is happening.

And it’s just a bunch of empty hallways. But if you’re talking about like in a military conflict, you can’t do nearly as that because your drone is moving about, you have things moving, entering the battlespace leaving the battlespace folks, very deliberately trying to camouflage or disguise their activities. So they creates huge incentives to move towards much higher levels of autonomy. And I think that’s where you get the true weapon of mass destruction type idea, because that’s the two fundamental properties we’ve seen with traditional weapons of mass destruction, namely chemical, biological, and nuclear weapons, where it’s not just that you have a lot of harm that you can inflict, but you very much can’t control that harm. Where if you have a rant, wind blows in the wrong direction, you potentially spray, move your chemical agent into, instead of hitting a soldier, it hits a bunch of civilians.

And that sort of leads me to the second aspect where I think control is itself may create its own types of risks, particularly with these weapons that are already uncontrollable like chemical and biological agents. If you can increase your level of control, that could potentially be a real problem from a chemical and biological norms perspective and particularly proliferation there, we’re part of the reason militaries have given up chemical and biological weapons is they concluded that these weapons aren’t really useful for us because we are so concerned that it’s just going to spray us right back in the face, or create some harm to larger civilian groups and other types of groups that like this isn’t advantageous to what we’re trying to achieve. But if you have autonomous weapons that can improve those capabilities in a substantial way, that targeting capabilities substantial ways, then you now have a much stronger as, or you remove the disincentive against having those weapons in general.

And that’s particularly concerning given the current issues surrounding chemical and biological weapons, where some of these norms seem to be fraying a little bit, some of the assassinations using Novichok agents in the United Kingdom. And there’s some allegations from Bellingcat about Russia’s current chemical weapons program that’s actually much more developed than is publicly known. I haven’t dived in the report myself, but like from some of my own research at least seems plausible. So if you could find lethal autonomous weapons with, these already horrible agents, it could be a potentially really major concern from a global security perspective.

Stuart Russell: Yeah. So I think this is an example of the kind of the blow back problem. And we hear over and over again from the U.S and I heard this, directly in the White House that although these things are possible, the U.S would never develop or use weapons in such a way. Now. To believe that you have to gloss over the history of the U.S’ own biological and chemical weapons programs. But even if you take that at face value, that seems to be, I would say actually quite naive because once that technology is created and manufactured on a mass scale, just as has happened with automatic rifles, they spread around the world in vast numbers.

There are 75 million Kalashnikovs’ in private hands, for example. So the idea that we can manufacture millions of swarm devices, but they only remain in the hands of highly trained, highly civilized Western militaries is completely naive. And one of the reasons behind the U.S abandonment of biological weapons, in addition to the military difficulties of using them, was the argument made first by biologists. And then by Henry Kissinger to Nixon, that such a weapon would reduce the security of the United States, because it would quickly proliferate. It was very cheap and will be used by our enemies against U.S Civilian populations. And I think all of those arguments apply to lethal drone swarms as well.

Zachary Kallenborn: I think that’s right. I think we have to add a little bit of nuance to it, because I think there’s necessarily going to be some limiters in terms of how much it spreads and what types of weapons, cause if we think about drone swarms, it’s not like a bunch of off the Shelf quadcopters interacting together, you potentially are looking at like a large unmanned ground vehicles, like with some sort of artillery or tank that’s interacting with other types of similar advanced systems, stuff like that is going to at least be harder to proliferate as opposed to like the computing or like the algorithms and other processing technology that goes into quadcopters. And we can see this if we look at particularly non-state actors, which just where my previous experiences before jumping into a drone swarm issues. So a number of non-state actors have also been quite interested in chemical and biological weapons, but even fairly robust and well-resourced organizations have had considerable failure when it comes to some of these more advanced ones. Now there’s all sorts of incidents of people like getting ricin and trying to poison people and often failing and realize that it’s not a very good inhalant, but my favorite example is Aum Shinrikyo. They were the folks who carry it out and it was the 1995, sarin gas attacks on the Tokyo subway. Before, that they attempted to pursue a fairly robust biological weapons program. And I don’t really like using the term, but I think it’s appropriate in many ways. It was laughable. You know, even though we’re talking about like attempt to kill lots of people, I think in many ways, at one point they were trying to make botulinum toxin, extremely deadly toxin where that kills at the scale of like a handful of like micrograms or milligrams, like extremely, extremely small.

One of their folks actually fell into a VAT of what they thought was there were producing botulinum toxin, and the fellow emerged completely unharmed. And given how little it takes to kill that suggest like they really did not a very good job of actually cultivating and developing this. And these were fairly well well-resourced folks, I think estimates, but they had somewhere like neighbor of like a billion dollars worth of resources. And they were also working with like former Russian military. And they were like mining for uranium in Australia and doing all sorts of odd things.

But the point being is like, I think when we talk about these things, they are very simple things, like getting large amounts of quadcopters, getting those to work together. And I think it is a real concern, proliferation perspective. When we’re talking about something like these truly massive, like million scale working across multiple domains. And I think there’s proliferation to like some of the higher, more advanced states like China, Russia, United States, maybe even some like Turkey and some of the like middling powers. But I think, maybe not necessarily to like an Armenia or Azerbaijan, or something like that.

Stuart Russell: Yeah. That’s why I specifically talked about proliferation of swarm weapons, which I mean quadcopters of fixed wing, but large numbers of small devices can spread in the same way that Kalashnikovs have spread. And on the subject of biological and chemical weapons, I think it’s important to point out that one of the reasons it’s been difficult for non-state actors is because they are not companies in the chemical and biological weapons, business manufacturing those weapons and selling them. Whereas they are already companies doing that with lethal autonomous weapons. And unless we do something, there will be a lot more and there’ll be manufactured in much larger quantities. And inevitably as with small arms, those will filter out into the wider world.

Lucas Perry:All right. So there’s sort of a range of places that we can notice these systems beginning to emerge. There are various versions of autonomous weapons systems increasingly in the military, so you talked about vehicles, or tanks, or particular kinds of drones, and these are their own form of autonomous weapon. In terms of swarms, there are both collective drone swarms, which can range from wingspans of… Stuart was saying something like 13 feet to more like commercial quadcopters, to smaller. And so Stuart is suggesting that the increasing miniaturization of these quadcopter-like things are what potentially might look something like the Kalashnikov of tomorrow, is it that those kinds of systems could proliferate because they’re sufficiently cheap, and there was this scalability aspect of the ability to murder people or to kill people.

So the WMD aspect is due to the scalability, like how Google’s computation is scalable based on its hardware. The scalability is a part of both the hardware and the software, the autonomy, which takes out human decision-making from the system. So I think before we get into more of the particular risks, I’m curious if both of you could paint a little bit more of a concrete picture of the current state of production. So you’ve mentioned that companies that are investing in creating these systems unlike with chemical and biological weapons. So, what is the state of industry in the creation of autonomous weapons? And is there any more clarity that you can add in relation to the extent to which global superpowers are pursuing these weapons?

Stuart Russell: I think this is a great pair of questions because the technology itself, from the point of view of AI, is entirely feasible. When the Russian ambassador made the remark that these things are 20 or 30 years off in the future, I responded that, with three good grad students and possibly the help of a couple of my robotics colleagues, it will be a term project to build a weapon that could come into the United Nations building and find the Russian ambassador and deliver a package to him.

Lucas Perry: So you think that would take you eight months to do?

Stuart Russell: Less, a term project.

Lucas Perry:Oh, a single term, I thought you said two terms.

Stuart Russell: Six to eight weeks. All the pieces, we have demonstrated quadcopter ability to fly into a building, explore the building while building a map of that building as it goes, face recognition, body tracking. You can buy a Skydio drone, which you basically key to your face and body, and then it follows you around making a movie of you as you surf in the ocean or hang glide or whatever it is you want to do. So in some sense, I almost wonder why it is that at least the publicly known technology is not further advanced than it is because I think we are seeing, I mentioned the Harpy, the Kargu, and there are a few others, there’s a Chinese weapon called the Blowfish, which is a small helicopter with a machine gun mounted on it. So these are real physical things that you can buy, but I’m not aware that they’re able to function as a cohesive tactical unit in large numbers.

Yeah. As a swarm of 10,000. I don’t think that we’ve seen demonstrations of that capability. And, we’ve seen demonstrations of 50, 100, I think 250 in one of the recent US demonstrations. But relatively simple tactical and strategic decision-making, really just showing the capability to deploy them and have them function in formations for example. But when you look at all the tactical and strategic decision-making side, when you look at the progress in AI, in video games, such as Dota, and StarCraft, and others, they are already beating professional human gamers at managing and deploying fleets of hundreds of thousands of units in long drawn out struggles. And so you put those two technologies together, the physical platforms and the tactical and strategic decision-making and communication among the units.

Stuart Russell: It seems to me that if there were a Manhattan style project where you invested the resources, but also you brought in the scientific and technical talent required. I think in certainly in less than two years, you could be deploying exactly the kind of mass swarm weapons that we’re so concerned about. And those kinds of projects could start, or they may already have started, but they could start at any moment and lead to these kinds of really problematic weapon systems very quickly.

Zachary Kallenborn: I think that’s right. I have a few data points to add on. So the most robust that I’ve seen, I think is the Perdix swarm, they launched 103 drones out of an F/A-18 back in like 2015, 2016. I think that’s the one you were alluding to where they formed up, traveled across battlefield and then did some other formation. What always struck me as really interesting about that is that the system was actually designed by MIT engineering students. Those are probably some of the most brilliant engineering students in the entire world, but they are when it comes down to it. Sorry, what?

Stuart Russell: Second most brilliant.

Zachary Kallenborn: Okay. Yeah. Fair, fair, fair. But certainly up at the top anyway, I think we could agree to that, but they were students, this isn’t some huge skunkworks project that got thrown $100,000,000 with a bunch of high level military people who’ve been making weapons for 30 years, these are students. I think that’s really important even though, yes, I think that we haven’t really seen too many examples of that. It’s very clear a number of states have expressed very real interest and are moving towards this. Just in like, as I mentioned previously, like there was the Indian swarm of 75 drones. I’m not convinced it’s actually a swarm or even all that autonomous, but it does show that they were very clearly interested in it. And recently Spain also announced the creation of not an arms swarm, but working towards, like for intelligence purposes. I know Turkey has, quite a few programs about this. The US military I know has announced like several swarming related programs looking at, for example, having Marine infantry units that can control some really small group of drones, coming out there.

Yeah. I do find that interesting as well, the fact that we haven’t seen it yet. In fact, I have a little Google search alert for like any drone swarm related news, and I’ve had a few Reddit threads pop up where it was like current master’s students, I don’t even know if there’s master’s, it may even be at bachelor’s level. Or, ‘I’m working on a swarm project as my term project.’ Hopefully not to kill someone, but they’re working at it at a very low level and already figuring out how these things work.

What’d say? I was going to go in a direction and I forgot what it was.

Yeah. So I think from like proliferation perspective. Absolutely. And I think that also creates all sorts of challenges, particularly because most of this is coming from the actual, like the algorithms and the computer control systems that allow you to get all of these drones to collaborate, which makes it much easier to spread about.

Oh yeah, that’s what it was. So what strikes me, is I think part of the reason why it’s been somewhat slow is that often adversarial groups are fairly slow to adopt new technology, and they do so largely in response to the conditions of security forces. So, as I said, a drone has been, first early remote controlled planes have been around since like the fifties or sixties, but the earliest example I could find of any terrorist group actually interested in was Aum Shinrikyo, in 1995, so 20, 30 years later or so. And you never really saw major interest until most recently, ISIS in 2013, I believe during the battle of Mosul where they used 300 and some odd drones at once in various missions during a single month.

And so I think that what’s different now, is that you’re starting to get much greater awareness of it and in part it’s a response to some of the security apparatuses that have developed after 9/11 and some of the responses to terrorism, where now states have started to recognize that we need to worry about ground-based attacks where vehicle born explosives and stuff like that. And so our security measures have been designed around them. So I live in DC and you can very easily tell what a government building is because they always have like bollards, big concrete things that are all around them. So that if an IED tries to blow up the entranceway, they can’t get to it.

Now when we talk about drones and some of these quadcopters and stuff like that, the advantage is that you can then just fly over many of these ground-based defenses, they become irrelevant. And therefore these terrorist actors who would previously have been using vehicle born explosives and stuff like that can now shift to a new posture and even though there’re some costs in terms of, these drones have smaller explosives, so it’s not as useful as truck bombs that may have like 50 or 100 pounds of explosives, but they can get access to targets that they hadn’t previously from things that are high up in the air, like radar installations or potentially hitting other types of sensitive targets that may be of concern.

Stuart Russell: I think the other thing that’s changed is that the early remotely controlled planes that the military worked with were large and very expensive, and that’s true of the Predator Drones and Global Hawks and so on. These are big things that are not easy for a non-state actor to acquire clandestinely and to operate because you need actually quite a lot of technical expertise. I’ve heard up to 15 people per drone required for your launch recovery, health, maintenance, monitoring, targeting, navigation, et cetera, et cetera. But when you’ve got small drones, the size of a pizza box, whether it’s fixed wing or, or quadcopter, then it’s much more within the range of capability of a small organization to field and they’ve been pretty effective, I think, in this kind of asymmetric warfare, because infantry tactics just haven’t adjusted to the idea that at any moment, something can arrive out of the sky and blow your head off. And so that’s going to change very rapidly, and we might see battlefields that are almost devoid of human beings because there’s just no way for a human being to survive in this kind of environment.

So one possible approach to this risk of proliferation in the use by non-state actors, would be a selective ban to say that the tanks and the submarines, the major powers are going to do whatever they want to do there, but the small anti-personnel weapons, those are the ones that we need to control because those are the ones that will proliferate, and be used for genocide and you name it. And there’s an interesting precedent for this in a 19th century treaty called the St Petersburg agreement, which actually banned explosive ordinance below, I believe it was 400 grams of explosive. So you could have large artillery shells that blow up buildings, but you can’t have small explosive bullets that are mainly designed to blow holes and people.

And that was for partly humanitarian reasons. ‘If I’m going to shoot you then you’re kind of out of combat and I don’t need to then blow a hole in you as well.’ So for whatever reason, there’s a precedent for the idea that we ban weapons below a certain size because of potential negative consequences from their use.

And that seems to be a version. I mean, it’s sort of real politic, right? If you believe that we can’t achieve a blanket global ban on lethal autonomous weapons because of the opposition of major powers like the United States and Russia, perhaps in their own self-interest, and I keep coming back to this idea, it’s in the self-interest of countries, not to want to be facing these weapons, either on the battlefield or in their own cities, to then just put a ban on weapons below a certain size. It might be a five kilogram or 20 kilogram payload or whatever it might be. And if that’s what we have to do in order to get agreement from the major powers to move forward, then perhaps that’s a tactic that might in the long run be more effective than insisting on a total ban.

Zachary Kallenborn: I think that’s right, and I would go a bit farther than that because I think the risks go beyond just proliferation when it comes to the broader military powers. So a lot of the more military oriented literature, looking at the significance of swarming or drone swarming specifically, as opposed to swarming as a tactic tends to focus on what the threat poses to existing weapons systems and large platforms. So the United States militaries have been heavily invested in, for I don’t know how long, but certainly decades at least, these really big exquisite expensive platforms, like the F-35 and they F-22. These things that cost like hundreds of millions of dollars to make. But if you can get a whole bunch of cheap weapons that can potentially cause considerable harm or even just disable that, that creates a huge problem for the US military as well as militaries in general because certainly others have done that as well. So I think there’s even a potentially a military argument to make there that these are presenting pretty serious concerns.

There was actually a really interesting study over at the Naval postgraduate school. It was done by some students that it was, or I think it was like master’s students, they’re all like current military folks. They did some mathematical modeling. So if you had, I think it was eight drones attacking a currently equipped US destroyer, at the time, I think it was a few years ago, but using only about eight drones, I think four of them would typically get through the current defenses that they had. And they were looking at, I think it was four of the Israeli Harops, and then I think four of the commercial off the shelf type drones.

And you can think about all sorts of situations where that would be pretty terrifying from a military perspective, even if you’re talking about some of these really cheap drones, because the consideration of ‘what happens if a bunch of drones fly into, say an engine of a really expensive fighter jet or something of that sort?’. And even the concern about that could be really impactful because that potentially restricts access to that airfield. If you have 500 drones or some cheap things flying around that are like $5 quadcopters or not $5, but a cheap quadcopter you then lose access to potentially fly planes out of that.

And I would also add to the point about sort of restricting the spread. I think there’s also other precedents as well when it talks both in the US and internationally, when talks about spreading weapons to the non-state actors. But I think the big one internationally is UN resolution 1540. I’m not a huge expert on it, but I understand the gist of it is that basically mandate states pass laws to restrict folks within their country from getting access to chemical, biological, radiological, and nuclear weapons. And I think that same sort of logic could apply very well there. Now that’s partially why I think the framing of weapons of mass destruction is useful to sort of start connecting drone swarms with some of these existing weapons that we think about in this way, because I think the solution sets are going to be very similar. So I think modifying something like the UN resolution 1540 to incorporate these weapons systems is very possible and it could even happen at like the domestic level.

So I understand US law around weapons of mass destruction, terrorism takes a much more expansive view than how generally militaries think about weapons of mass destruction, where the FBI and other folks who focus on these look at consider explosives to be falling under the scope of weapons of mass destruction. Because in part, when we’re talking about non-state actors, even relatively limited capabilities could cause pretty significant harm if they were used to assassinate a head of state or blow up a chemical facility or an office building or something like that. So I think some of these mechanisms absolutely exists that can be leveraged to great success.

Stuart Russell: I know that you’ve argued in some of your papers that drone swarms should be characterized as weapons of mass destruction. Is there pushback? I’ve come across a view that weapon of mass destruction equals CBRN, chemical, biological, radiological, nuclear, and possibly because of the various sort of legal consequences of expanding that definition I’ve come across a lot of resistance to characterizing drone swarms that way.

Zachary Kallenborn: I think so, to an extent. But I think it’s partially for a handful of reasons. So weapons of mass destruction is a term, I think has been occasionally abused for political reasons. And so both within bureaucracy, as well as higher level politics, certainly we can talk about opinions over the Iraq War, and very clearly, the call to engage with weapons of mass destruction was a big part of that. So it starts fitting in with all of the politics that went into that conflict there, which in general, I have a lot of issues and a lot of views on that issue. Entirely won’t go into them too much, but I think there’s sort of that that’s coming along with it. And yeah, I think typically in more recent years that weapons of mass destruction has been associated with chemical, biological, radiological and nuclear weapons.

But historically, look at where weapons of mass destruction as a term came from, that was never the case. So it originally came from, I think it was 1954, I believe. I can’t remember the exact year, but it was around then coming from some discussions at, I believe, the United Nations, where they establish what researchers who have very seriously and exhaustively looked at weapons of mass destruction considered to be the most authoritative definition, where they consider weapons of mass destruction to be weapons of capability like chemical, biological, and nuclear weapons, but not necessarily limited to them. In fact, many of their discussions, they argue that we have no reason to believe that another weapons system will never emerge. That is just the same and deserve the same sort of considerations. At the moment, it’s just these weapons. And so I think that’s part of it, there’s some inertia going on of like, we’ve never really thought about expanding this, but I think among the folks that I know who more seriously do think about the term and what that means, there seems to be some interest.

So recently the National Defense University, their Center for Weapons of Mass Destruction Studies (Center for the Study of Weapons of Mass Destruction)? I can’t remember, I apologize to those folks if they’re listening for not getting their name right. They just published a really lengthy study looking at the future of countering weapons of mass destruction. And they included a pretty big section ‘unmanned aerial systems’ and included my ideas about ‘this could be a weapon of mass destruction’ is a pretty major focus of that. I don’t believe they explicitly said ‘we agree with the conclusion,’ but just the fact that they believe it’s at least plausible enough to engage, I think is a persuasive argument that at least there is some interest. And I think they’re sort of going beyond that. I think some of that, there’s also sort of bureaucratic questions about what exactly this means because, or not bureaucratic, but broader multi-political issues about what we focus on.

Some of the pushback I’ve gotten is argued no, that weapons of mass destruction are really just nuclear weapons, and drone swarms don’t reach that scale. I don’t really dispute that claim. I think at extreme scales we’re talking about a million drones. A drone swarm could meet Nagasaki, Hiroshima, type nuclear weapon, or I should say Fat Man, Little Boy, because that’s the weapon, level of harm. But I find it hard to believe you would start seeing like Tsar Bomba, which was the 60 megaton nuclear weapon Russia detonated in like the 60s, 70s, something like that. I can’t imagine it reaching that capability. And like from a broader risk perspective, it makes sense that nuclear weapons still should be more important than drone swarms. But I think nonetheless, they are important because of all these risks that we’ve decided about.

So I think the idea is nascent, as far as I’ve seen. I haven’t been very convinced by any of the pushback that I’ve gotten. I know that like there was that discussion you and Paul Scharre had in IEEE, which was interesting, but personally I didn’t find any of the counter-arguments all that convincing.

Stuart Russell:Yeah. And I think there are some characteristics of drone swarms that although I agree that the physical impact of Tsar Bomba is not likely to be matched, but the geographical scale certainly could be, and I think the naive vision of what war is about, which is to kill as many people as possible, soldiers will tell you, no, that’s not the purpose of war at all. The purpose of war is to get another entity to do what you want. And if you can do that without killing any of them even better.

Zachary Kallenborn: I they’ll immediately Clausewitz to you on war is politics by other means, but yeah, go ahead.

Stuart Russell: Yeah. So a large drone swarm could certainly control and threaten an entire metropolitan area of the size of New York without necessarily killing everybody but, killing some small fraction of selected targets as a demonstration of power, and then simply sort of being in residence as a way of controlling the remainder of the population. So it can have the same scale of impact. And I think the fact that it doesn’t destroy all the buildings is an advantage. There’s not much value to a large smoking radioactive creator, to anybody, but an intact city and an intact industrial infrastructure that is now under your physical control is in fact one of the ideal outcomes of warfare.

Lucas Perry: All right. So I’d like to pivot here into moving through particular risky aspects of autonomous weapons and moving through them sequentially probably spending about five to eight minutes on each one. So we’ll have to be a little bit more brief with each of these sections to make it through all of them. Let’s start off here with the inherent unpredictability of lethal autonomous weapons systems, both individually and also collectively. So if you have swarms interacting with swarms, or autonomous weapons systems operating in relation to other countries autonomous weapons systems, which they’re not necessarily familiar with or whose behavior may lead to catastrophic behavior on either side. Can you guys speak a little bit to this inherent unpredictability of these systems, especially in multiagent scenarios so where we have multiple autonomous agents of one side, perhaps interacting with multiple autonomous agents from another side?

Stuart Russell:So I think this comes back to a caveat that I made earlier, that, that sometimes it matters that the system is built and is using machine learning in particular, because a system that follows a fixed algorithmic decision-making approach is vulnerable to countermeasures that the other side is adapting to that fixed strategy and finds a way to overcome it. So to give you an example, if you’re playing rock paper scissors, and you just play rock every time, then the other side figures out that you’re playing rock every time and plays paper every time and you lose. So in these kinds of adversarial settings, the ability to adapt to devise sort of defeating strategies against the opponent is crucial to survival. If you don’t do that, then you’ll be subject to that. And so that creates as you say, this sort of escalating unpredictability, because you don’t know what the adversarial strategy is going to evolve into, and therefore you don’t know what your own strategy is going to evolve into to counter that. And so it’s actually extremely hard to model what could happen when two large heterogeneous groups of agents are engaged in the struggle with each other. And we’ve seen this, of course, in human warfare, in the past, what was happening by the end of the war often didn’t look very much like what was anticipated at the beginning of the war.

Zachary Kallenborn: I think that’s right. And I think that also creates, or I think we already, to an extent, like even know that’s a problem, at least for swarms, so there was some work done by the Naval Postgraduate School, Tim Chung, who now works at DARPA and he did a 50 versus 50 swarm fight and where he actually set up, like I think it was all homogenous drone, like just quadcopter type things. So not as much complexity basically to see, like what are the factors that influence outcomes in these sort of behaviors instead of very controlled, I recall it like the key issue was speed and how quickly, not only like the physical speed of how, like the velocity that they’re actually moving it, but the decision-making speed where you have 50 different drones. Like they need to be able to quickly recognize and respond to all of these different targets.

And that ends up being like speedily, like algorithmic sense of like how quickly can they process and detail information. And that obviously creates a problem when we can talk about error and risk, because the faster you’re making decisions necessarily, as we know as just being humans, that tends to be where errors come in, if we’re making rapid decision making about. And there’s probably a pretty thoughtful reference that we could talk about like condiments in system one and system two type thinking, or I can’t remember the exact details, but at the folks get the point, if you make fast decisions, they tend to create much errors and as things go larger and they interact together, those risks amplify.

Lucas Perry:Yeah. Right, so, I mean, this is making me think of, I think OODA cycles from the Center for Applied Rationality, they take it from jet pilots. I think the way that they frame dogfighting is whoever can make the most updates per second for movement decision-making in the dog fight is the one who’s going to win. So with drones, it’s both this aspect of increasing the cycle at which you’re like making decisions as a swarm to outmaneuver the enemy swarm. And then also the factor of that increasing the likelihood of error.

Zachary Kallenborn:Yes. As well as expanding that over multiple different agents. Because when we’re talking about drone swarms, we’re talking about potentially 50-100 thousand agents. So it becomes even more complicated because you now have a hundred or whatever thousand points of where that decision process could get messed up.

Lucas Perry: All right. So let’s get into escalation risks then, because this seems pretty related is we have these systems are inherently unpredictable and which that unpredictability risk rises as you have multiple agents interacting with multiple agents and this race towards faster and faster decision-making over time in order to gain a strategic edge. So it seems like these factors of unpredictability and escalation risks are increasing rapidly in order to gain a strategic advantage in say drone conflicts. So what kind of perspective can you guys offer on the escalation risks at borders between countries or between autonomous weapon systems that lead to both accidental and intentional risky outcomes, which then, interdependently with the increasing power of technology in the 21st century will lead to potentially worse and worse outcomes as the proliferation of more powerful technologies to do more damage increases along with this increasing capacity for the escalation of risk due to the proliferation of drone technology.

Zachary Kallenborn: Okay. Yeah, I think when it comes to escalation risks, I think that’s an issue cause like, the challenge is that, okay, what happens if this swarm gets out of control or even just an element of that swarm gets out of control or any type of lethal autonomous weapon system and that potentially kills an actor that wasn’t intended and it happens to be particularly important. When I think about escalation, I often think about like, just jump back to World War One where the assassination of, my head is always really bad for history. I can never remember terms and dates.

Lucas Perry: Archduke Ferdinand.

Zachary Kallenborn: Yes. Thank you. There we go, yeah. Where the assassination ended up setting off a chain of events that ended up getting a much broader conflict. So depending on the context of one that’s used and how it’s used, you potentially have pretty major problems. Like what happens if depending on the context, if there’s some like high-level military person who’s coming or… political leader, who’s coming through an area and the drone happens to go out of control and attacks their convoy or something like that. That could be a huge, I mean, hopefully they have the intelligence and wherewithal to know, like let’s not drive by an active conflict zone, but nonetheless, like people make mistakes and, drones go out of control. So, I think there are very real risks that can come there. And I think there’s also escalation risks sort of in other ways. So one of the things I found really interesting when I was looking at the conflict between Armenia and Azerbaijan is that the Azeri drones were reportedly and I don’t know, I’m not a hundred percent confident this is true, but at least it seems possible

We’re actually run by Turkish military folks who were flying the drones. So what that means is that if the control system for the drones was attacked and blown up, then Turkey is now potentially involved with this conflict in a more direct way. And so if you apply that strategy in other situations where a state or two states are fighting each other, one’s using drones and they essentially attack some of those ground control stations to better managing that larger complex swarm. You have all sorts of concerns of cascading effects, where you’re potentially bringing in other military powers that also has bigger problems of instability. And even if the US and other countries don’t necessarily care about have stakes in that actual conflict, there may be some level of obligation to like get involved to resolve issues. And that in and of itself can be an expenditure of resources, soft power, and in a variety of other potential costs that come along with it beyond the obvious cost of we don’t want more people to die.

Stuart Russell: I think there’s a potential for escalation risks even before a conflict begins. And we’ve obviously seen this in the Cold War with false alarms about nuclear attacks. And I imagine they happened on both sides, but we’re aware of several times when Soviet nuclear missiles were almost launched in response to what they imagined to be US attacks, but because there was at least some human in the loop, those were averted. And the difficulty, once you have autonomous weapon capabilities, war can move so much faster that it might not be prudent to have humans in the loop coordinating defensive responses, or even deciding that a defensive response is needed because by the time the human has been woken up and dragged out of bed and informed about what’s going on, it’s too late, you’ve already lost the war, 75% of your installations have been wiped out.

And so there’s always a temptation to say, because of this, the OODA loop, the speed of that accelerating, we have to have an autonomous response capability against attacks. And the problem is, yes, you respond autonomously to attacks, but you will also respond autonomously to things that are perceived to be attacks even when they aren’t. And so any error in the detection and classification of potential attacks could lead to an autonomous response, which would be a real response, which would then lead to an accelerated response from the other side. And this is something the US military have explicitly expressed concern about.

And in recent discussions between the US and China that were reported in an article, a joint article by Fu Ying and John Allen, they talked about this as one area where everyone agrees, we would like to put into place protocols that will prevent this kind of escalation. And I think, in this area of international negotiations over autonomous weapons, anything that we can agree on and make progress would be, I think, beneficial because it sort of, it sets the precedent. It creates those channels of communication. It builds trust between the technologists working within each of the military hierarchies and could potentially expand to greater levels of agreement.

Zachary Kallenborn:I’ll add just very briefly that I think there’s also how those incentives play out, could create all sort of interesting and potentially harmful ways. It’s like if you remove, like you keep humans at some level in the loop over ethical or whatever concerns, or have just some minor role, they’ve potential also become a target for other militaries. Because if you kill the humans, then you potentially cause much greater harm. So was the folks that have been, trained for a long time and that’s at the end of it. And often what the politics people resonate with is, oh that’s my son who… or my daughter that was killed.

So you have more of a strategic effect, but that creates an interesting incentives where like, okay, that if we need to target the humans, then we need to put them in defensive and very carefully guarded places, which may be like in the homeland somewhere. Okay, then we need to guard them, which then creates incentive to like, okay, we’ll let’s strike the humans where they are. So let’s potentially, instead of like having a bunch of drones battle in the middle of the ocean or something like that, let’s just strike directly at wherever that homeland is. So that could be particularly I think problematic depending on how that, how those incentives evolve.

Stuart Russell: Yeah. I think it’s worth mentioning that many people ask, well wouldn’t it be great if the drones fight it out and nobody has to die.

Zachary Kallenborn: Yeah.

Stuart Russell: And I think this is just a misunderstanding because, you could also ask, well, why don’t we just play baseball to decide who wins the war? Yeah, we could do that too, but we don’t do that because just because I am losing a game of baseball, doesn’t mean I’m going to pay you eternal tribute and let you enslave my population. So I’m only going to let you enslave my population if I have no physical choice, because you have destroyed my defenses and began killing my population and my government hierarchy and my military hierarchy. So it’s only when on the sustainable costs are imposed, that one side is going to surrender. So there is no circumstance that I can really imagine in which there could be a bloodless war.

Lucas Perry: That’s a really good point. I appreciate that. This is a good point then Stuart, about there not ever being a bloodless war. So there’s the risk of these weapons lowering the barrier to conflict because of an increased willingness to engage in conflict because autonomous weapons systems destroying each other is not the story of sons and daughters coming home and in body bags and caskets. So there’s the extent to which stigma operates at sort of conditioning and controlling behavior. And so, the existence of machines are just on the line at the edges of conflict. It seems like there’s this risk then of lowering the barrier to conflict because machines are on the line rather than people you have to still leads to an escalation of real conflict. Do you guys have any perspective on this?

Zachary Kallenborn: Yeah, I’ve heard those arguments personally. I’ve never found them overly persuasive. It was more of an intuition and not necessarily anything, all that concrete, but I did see there was a study that came out recently that folks that or states that have used drones in general have been tended to be more careful when they’re using it and sort of rationalizing by own intuitive sense. Like, what makes sense to me is that like military generally are going to be very cognizant of escalation issues, deliberate escalation issues, I should say.

The folks that I follow who are like think tank policy wonks, they’re constantly talking about like it’s controlling the ladder of escalation and when a conflict emerges, and I think that’s something that like states are already cognizant of. So like to me, outside of like error, which I think would be an issue where you no longer sort of lose that control of the escalation ladder, I’m not all that convinced that lethal autonomous weapons would have a meaningful effect. Like there were probably need to be like some discussions of the Pentagon, like, okay, how are we going to use this? Okay. Here’s where it’s going to be useful, here’s not, here are some of the care we want to have at least from an escalation perspective.

Lucas Perry: Sorry. So, so there might be a distinction here then perhaps the risk of escalation is increased if it’s an accidental escalation, but remains the same for intentional escalation.

Zachary Kallenborn: I think so yeah.

Stuart Russell: Yeah, I think it also depends on the political environment. For example, right now the US is operating drones in regions of Pakistan. There’s no way we would be deploying US troops on the ground in Pakistan. If we didn’t have drones that we would not be involved in military conflict in Pakistan right now, but it hasn’t led to major escalation because the Pakistan government is partly, it’s a very complicated situation, but they are unofficially tolerating it. And they don’t view it as an attack on their own government and regime. And so it hasn’t escalated even after several years of operations.

Zachary Kallenborn: Right.

Stuart Russell: And, but you could imagine in another circumstance that the same level of military activity using drones would lead to a direct response against the homeland of the United States. Then things would probably get out of hand very quickly.

Lucas Perry:So they function also as increasing the willingness for superpowers to engage in conflict abroad, where it would be too politically or socially expensive to actually have boots on the ground. So you can use autonomous weapons systems as a means of sort of lowering the cost of entering into conflict.

Stuart Russell: I think it lowers the cost of entry, but it can also be maintained at a low temperature. Whereas if you put troops there and then troops start getting killed, that tends to raise the temperature and raise the stakes. And no government wants to be seen retreating from a battle because a few of their soldiers have died and so on and so forth, whereas you have with your autonomous weapons, you can expand or contract your level of activity without serious political consequences at home.

So I think there’s a lot of complicated effects going on here. And I think I agree with Zach it’s too simplistic just to say it lowers the threshold for war because war itself is not a Boolean thing. It’s not either or it’s actually a whole continuum of activities.

Lucas Perry:All right. So we touched on this earlier when we were talking about the miniaturization of the hardware and then also the proliferation of the software also Stuart mentioned how it would only take him and a few grad students a term to make something that could take people out at the UN. So there’s also this democratization and proliferation of the knowledge with which to make these systems. For example, I think that when CRISPR was discovered, it only took something like a year to go from being at the vanguard of science, to being something that’s able to be taught in university and classrooms.

So I assume that we can expect some similar evolution of this with autonomous weapons, as both the hardware and algorithms, which are required to make really serious forms of them become cheaper and more widespread. So what are your perspectives on the proliferation risks of these very high risk swarm versions of autonomous weapons systems? So like not just the kinds of systems that militaries might be increasingly adopting, but these small, cheaper forms. And the associated risks of the diffusion of human knowledge and human talent in order to build these systems.

Stuart Russell: I think I want to separate two things. And many people will actually sort of mix them up and say, okay, if the knowledge becomes publicly available, then the weapons become publicly available and therefore trying to ban them or control them is completely pointless. And you only have to look at chemical weapons to see that that’s not the case, the reactions you need to make chemical weapons are things that can be done in a high school chemical chemistry classrooms. But because we have a treaty that includes engagement of industry in monitoring the production and sale of precursor chemicals, and we have a stigma and we have very strict national laws. We have not seen wide-scale proliferation of chemical weapons and same with biological weapons, even though that lacks any verification component and that’s being fixed right now, I think because of things like CRISPR people will see a more urgent need to get oversight of industry and precursor devices.

Lucas Perry: Can I can ask you questions about that.

Stuart Russell: Sure, yeah. So I wanted to just say that we could imagine a situation where, we might wish that these things were restricted to the US military or, other quote, “serious countries.” But if there is large scale manufacturing, there will be large scale manufacturing for sale to anybody who has the money to buy them. And so the risk there is not that people would, find the software on the web and then go out and buy a bunch of toy drones and fix them in their garage. The risk is that they will just go to the arms markets, the same place as they buy the Kalashnikovs by the boatload and buy lethal autonomous weapons by the boatload. So you’re changing the entire system from one of widespread large-scale manufacturing availability to one where there are serious legal restrictions and practical restrictions. That would mean that it will be much, much more difficult for large scale events to take place.

Lucas Perry: Sorry, Zach, I’ll bring you in just a second. I just want to ask Stuart a quick question about that. So, in terms of chemical weapons, you’re saying that they require a high school level chemical reactions to manufacture, but that the reason why we don’t see, for example, even more non-state actors like terrorist organizations, effectively deploying them in the United States is because of the strict regulation and law in the federal government, towards industry for regulating the precursor chemicals, which will lead to the manufacturing of chemical weapons.

So I feel confused. Maybe you can help explain it to me, why it’s different with autonomous weapons. You said that it would seem to me that there’s like the proliferation of the hardware seems to just be a fact and you couldn’t necessarily regulate that. And the kinds of algorithmic components which go into building the thing in your garage, as you said, there’s like facial recognition and there’s like 3D environment mapping and there was moving through environments and then perhaps getting some kind of payload onto it would be the most difficult part, but I don’t see why it seems to me like it would be easier to make a lethal autonomous weapon in my garage if I had an undergraduate degree in computer science and mechanical engineering than it would to be, to figure out how to make a chemical weapon.

Stuart Russell: Okay, well, so first of all, I think I would actually disagree with that as a practical matter, but we’re not talking about making one. We’re talking about making a 100,000.

Lucas Perry: A swarm of them, yeah.

Stuart Russell: And the same issue with chemical weapons. You can get small amounts of pool chemicals, which include chlorine and so on and make a crude chemical weapon. But… and that has happened and some of the things that the Syrian government did involved chlorine, which is relatively easy to get hold of, but even there, the Syrian government had to go through, they created multiple shell companies and all kinds of subterfuge to obtain the materials that they needed to scale up their chemical weapons program. So I don’t think you can prevent people making sort of on the scale of tens of weapons-

Lucas Perry:I see.

Stuart Russell: and using them for terrorist attacks and I think we will see as a result. So if you said the typical terrorist attack is killing, it’s a small truck bomb kills 50 to a 100 people in a market in Baghdad or whatever. I think you will see the scale of that go up by a factor of 10. Because it’ll be-

Lucas Perry: With autonomous weapons.

Stuart Russell: it’ll be an attack with 20 or 30 things with shrapnel bombs that explode above a soccer stadium or something. So this would be obviously terrible, but the attack that wipes out an entire city or an entire ethnic group could only happen if you allow large-scale manufacturing. And so the industry controls as with chemical weapons, there are things on the list and anyone who wants to buy things on the list has to be vetted and it has to be accounting for the quantities and the purpose of the use and non diversion and so on and so forth. And so you could do the same thing with physical platforms. I don’t think there’s much you can do about the software. I think people will write that software it’ll get there, but I think you look at the physical platforms as the place where you can start to impose some control.

Zachary Kallenborn: Yeah. So I want to jump in on that. So that shares many of my thoughts like that regulation is probably going to be a big part of it. I think it goes a bit beyond that to broader just awareness, training within law enforcement. Because it strikes me that, well in some sense the problems aren’t really that different than existing ones in the sense of like, okay, if I’m a terrorist and I want to kill someone, I don’t even need to get a bunch of chlorine, I can just go buy a knife from Walmart for like 50 bucks or however much it costs. And then just stab someone if I really wanted to or need to do. And of course we saw ISIS do exactly that, or just simply drive a car through a crowded area, which is horrible, but it’s a pretty cheap way that you can cause pretty significant harm.

And responding to that, I think in some sense, it’s sort of the same issue. In that its law enforcement awareness, collecting intelligence about these organizations, understanding how they operate, where are the folks who we need to worry about from an ideological perspective. And when I say ideological, I don’t necessarily mean in terms of whether they’re religious or right or left or cults or something like that. But I particularly mean that are interested in violence and that tend to have really maximalist views. That is like, if I’m an organization that wants to overthrow the existing world order and establish something new, in order to do that, I must necessarily have to take pretty significant action to do so. So identifying those types of organizations, collecting intelligence about them, and monitoring what’s going on as well as some of these sales, I think could be really effective because at the end of the day, it’s dealing with drone-based terrorism and swarm-based terrorism and laws-based terrorism is really just a subset of all the terrorist attacks.

That if you don’t have a terrorist organization to support it, then necessarily you can’t, or you won’t be able to support those tactics. And it’s often, I think, important to recognize that many terrorist organizations are often organizations of people and they run into the same types of challenges that people run into. Recall there was an Al-Qaeda member who turned state witness and testified against Al-Qaeda about a bombing in Kenya, because he was really pissed off that Al-Qaeda didn’t pay for his wife’s C-section. It’s ridiculous, but at the end, they are human beings they run into the same challenges. Which means if you break apart, intercept and engage with these organizations and prevent them and the rest of folks who are responsible, even if the technology is available, you make it much more difficult for them to realize that potential. Because any organization still needs even if it’s relatively simple needs the resources, the people, and simply the people who are excited and willing to take that action and do so.

Lucas Perry: All right. So we’ve got about 15 minutes here. Let’s go into a bit of a rapid fire portion now and keep answers pretty short. So Stuart, if I understand what you’re saying correctly, so when we’re talking about swarms, what really unlocks that is industry level production capability. And so that’s why there’s going to be in an importance of regulation and monitoring at the industrial level. What to me seems relevant about the proliferation of the hardware, and knowledge and talent for smaller scale versions would be risks of assassination. So with swarms, you have this weapon of mass destruction difficulty, but then with the proliferation of smaller, cheaper weapons, there is an increase in the risks of assassination, which are also destabilizing and high-risk.

So do either of you guys have a few sentences on the increased risk of political assassination, or you wouldn’t even … So for example, there was some hacking at the Florida water treatment plant recently, you don’t just only need the weapons to, for example, directly target people, but they could deliver other kinds of things to water treatment facilities or other locations that could cause mass harm. And then you wouldn’t really need a swarm, but maybe the high school chemistry plus the dozens of autonomous weapons. So do either of you guys have perspectives on this?

Stuart Russell: I think the assassination risk is very real, and I know that the US government is very concerned. They have at least since Kennedy if not before, gone to great lengths to protect political figures from assassination by sweeping buildings, by checking all the lines of sight, et cetera, et cetera, for any public appearance. But of course, with an autonomous weapon, you could be five miles away and launching the weapon. And so they’re very concerned what kinds of defenses could you come up with. But certainly I would much rather be defending against a few amateur homemade weapons than against a large number of high-tech professionally designed and engineered systems.

Zachary Kallenborn: 100% agreed on that. The challenge with drones or drone swarms and particularly is fundamentally that issue of mass where you just keep throwing drone upon drone upon drone. And we’re talking about like it’s an assassination type event, even if you have 20 drones and the security forces are successful and shoot down 19 of them, but even a single one goes through with a small charge and it explodes your head of state that’s a complete failure and the ability to protect what they’re trying to do. And I think that’s why you’ve seen increasing focus among all sorts of law enforcement folks really across the globe have been increasingly focused an interest in this. I think for much of that reason, because there are some pretty significant harms that you can do not only in water treatment plants, but the one that really concerns me is chemical facilities. Particularly what happens if you go explode some like that of some dangerous chemical that then releases and then spreads to some sort of local populace that would create much wider harms using only potentially a relatively small charge. Yeah.

Lucas Perry: All right. So again, continuing with the rapid fire, let’s talk a little bit here about attribution and accountability issues. So this is the question of who is responsible, for example, for attacks of all sorts, whether it be military. You guys talked about recently … How do you pronounce that? Azerbaijan conflict?

Zachary Kallenborn: Azerbaijan. Yeah.

Lucas Perry:That Turkey is piloting some of the drones. So there’s increasing difficulty of understanding, especially for large-scale systems or swarm attacks. Who should be held responsible for the system? Who is piloting it? If it malfunctions, is it the operator’s fault? Is it the industry or the manufacturer’s fault? These kinds of things.

Zachary Kallenborn: So I’m going to focus on the answer primarily through like more security perspective rather than like … Because I don’t really know how regulatory issues come if there’s a mistake in the system. But what are the key challenges I see is … Or I think the way to deal with that is through intelligence, law enforcement, folks on the ground as well as post attack, digital forensics where … One of the things about swarms, if you have large amounts of drones that are attacking you necessarily, you’re going to have quite a few that fall. From an attribution perspective, creates some really great opportunities because if you have this fallen drone, you potentially examine all sorts of things about how it works, where it came from and stuff like that. And I think from an attribution perspective, one of the key challenges for responders that they need to look beyond the actual drone itself.

So if I was a terrorist attacker trying to use a drone, I would launch the drone from the back of a pickup truck. So that basically as soon as I launch the drone and especially if it’s autonomous, it has some sort of waypoint navigation. I could just send it towards wherever I want to blow it up and then get in my pickup truck and then drive away. And then I’m already gone before the attack even happens in the first place. And that means that law enforcement folks needs to look beyond just the immediate threat to look at. Okay, the areas that are surrounding places that could be attacked. To understand what’s going on, who’s around and be prepared for whatever comes.

Lucas Perry: Nothing to add Stuart?

Stuart Russell: No, I think it’s forensic identification and accountability tracing is a really hard problem. I mean, it’s also a hard problem for other kinds of non-autonomous weapon systems. You can launch mortar weapons that you buy on the open market. And just because the mortar bomb was made in Vietnam doesn’t mean that the attack was launched by the Vietnamese. Same basic issue that you have with autonomous weapons.

Lucas Perry: All right. So technology is becoming increasingly powerful throughout the 21st century and particularly artificial intelligence. This represents a preliminary global issue in the governance of AI and cooperation and coordination on it as an emerging and evolving technology that will have tremendous power in the 21st century and into the future. So what is your guys’ perspective on lethal autonomous weapons as an opportunity for norm setting and as a precedent for trust building for the global governance of AI?

Stuart Russell: I think there’s increasing awareness that viewing AI as a virtual arms race. So here, I’m not talking about physical weapons, but AI as a technology that would confer global dominance on whoever develops the most capable AI. I think there’s an increasing understanding that that view is in many ways outdated and counterproductive. And one way of thinking about that is that if we do develop what some people call human level AI or artificial general intelligence, that technology would be essentially a source of unlimited wealth for the human race and for one entity to try to control that would be as pointless as trying to control all the copies of a digital newspaper. Because you can just make more copies and if someone else has a copy, it doesn’t mean you don’t have a copy. It’s not a zero sum game. And I think it would change the whole nature and feeling of how nations interact with each other.

We’ve had this zero sum mindset for thousands of years, that whatever we have, you don’t have and vice versa. And that is increasingly invalid for all kinds of reasons, climate change among others. But on AI, this is a technology where we would all benefit to a much greater extent if it was shared. So if we can start by planting a flag, drawing a line in the sand and saying, “Okay, let’s not get into an arms race on the physical technology.” I think that would help. And I’m seeing encouraging signs from several major powers that there’s beginnings of an understanding of this at the highest levels.

Zachary Kallenborn: Yeah. So Stuart looked to the future. So I suppose I’ll look to the past. So if we think about humanity has been building weapons pretty much as long as humanity has existed. You know we started with sharpening rocks and strapping it to sticks. We made bows and arrows and swords of bronze and iron. And we made guns, tanks, fighter jets, and eventually nuclear weapons. And I think pretty much throughout that history, the fundamental rationale has been largely the same. We build weapons to keep ourselves safe as well as those we love in our broader country. But over the past century or so, we’ve come to realize that at some point, weapons stop making us safer when they start creating all of those risks to global stability and to potentially the very folks that are using them as we discussed around like chemical and biological weapons, to a lesser extent, land mines, nuclear weapons and all sorts of other examples.

So I think to move towards those international norms, there needs to be an understanding that these weapons are somehow risky and rather than keeping us safe, they create more problems than they actually have. And so in order to do that, I think there needs to be that focus on those particularly high-risk LAWS, whether particularly drone swarms, as well as the intersection between LAWS and chemical or biological, radiological, nuclear weapons. Because we’re talking about autonomous weapon that’s controlling a nuclear weapon. The risk of error suddenly become not only like a handful of folks, but potentially the survival of humanity, if that starts a nuclear war.

Lucas Perry: All right. And so this last risk facet for us to touch on as we’re wrapping up here is verification. So some of the proposals and discussion around the governance and regulation of autonomous weapons systems has been the need for keeping a human in the loop in such systems. So that the decision of target acquisition, selection and execution is not solely in the hands of the system, but that there’s a human being in the loop in the decision making of the system. So that there is responsibility and accountability and human judgment involved in the decision to take human life. How do you guys view the problem of verification with regards to fully autonomous and semi-autonomous weapon systems and the difficulty of regulation due to verification yet the importance of norm setting to achieve stability in the absence of easy verification with … Keeping in mind the perspective of how the bio weapons ban has no verification regime, but has held through international stigma.

Stuart Russell: So I think it’s quite a difficult question. I would point out that the absence of verification in the biological weapons convention meant that the Soviet Union actually expanded its biological weapons program after signing the treaty. And so I completely accept what I take to be the US position that a treaty with no verification and enforcement may not be in the best interest of the United States if other parties are not going to comply with it. So I think the idea that you could verify the software, although it’s technically possible, I think that’s a non-starter because it’s so easy to change software so that what was a human controlled, human in the loop system becomes autonomous just by changing the software. So I would not be happy with a treaty that allowed people to build weapons that could become autonomous by changing the software.

So one obvious solution there is to separate the firing circuits from any on-board computation whatsoever, right? So you can have onboard image processing, navigation, doo-da-doo-da-doo-da, but whatever is on-board, doing all that computing has no physical connection to the firing circuits that has to be done by a remote control. There are ways of bypassing that by putting autonomy back at the home base, right? So taking the human controller who’s sitting in his trailer in Nevada and replacing him with an AI system as well, but that certainly makes autonomous control more difficult, slower, easier to jam. But also you want to actually to have a sort of one-to-one or one to a fixed number correspondence between the number of weapons that you are building and a number of human control stations that you have. So if someone makes a couple of 100 million autonomous weapons, but only has five human control stations, then you know that that’s not going to be used with human in the loop. So I think those two things would be absolute minimum. I think there are probably other requirements as well, but I will say that’s a good starting place.

Zachary Kallenborn:I want to totally agree with that, but I’m a bit more optimistic. But at least when it comes to swarms, because I think there are certain aspects of swarming that actually make it even much more conducive to verification measures, particularly as I think I discussed before. The advantage you get from swarming is throwing large amounts of autonomous systems against defenses and you don’t mind so much if a bunch get knocked down. But what that means though, is that if you have a whole bunch that are knocked down or disabled, those are all then sources of post attack verification around what exactly has happened. And because you can look at some of those systems and get a sense of, okay, there may be indicators and potentially and hopefully very clear ones about like, “Okay, here’s how the system’s operating.” Likewise, I think there’s a lot of potential around because of how swarms … The difficulty of managing swarms, because it’s not feasible for a human to control like 1,000, 10,000 drones all flying together.

If you see that happening, you must infer that there must be extremely high levels of autonomy, likely including lack of control over that human targeting when it’s being used. In both cases, I think there was a third that I can’t remember. But that basically allows you to do some level of post attack verification response, which is I think often key to many of these norms. So if we look at the use of sarin in Syria, much of the response was not at investigating the chemical facilities that Syria was building, rather the response came after the United States that is recognized that Syria is using the agents based on the secondary effects. And then afterwards there were military strikes. I don’t remember if there were sanctions, but there was all sorts of things that came after.

And to an extent that’s sort of what you need is that level of punishment that happens after. Now, that probably won’t work at a situation where you have ongoing war between two major powers. Okay, yeah, we’re alreadyin a state of conflict, this probably doesn’t matter much. But I think in some of these smaller cases, it could be quite successful in sort of at the very least, naming and shaming who was responsible for this.

Lucas Perry:All right. So ending here now, starting with you, Stuart, do you have any final words then about autonomous weapons or anything you haven’t said that you feel unresolved about that you’d like to just end on?

Stuart Russell: I feel quite concerned right now that the mindset of the major powers has gone from, “Okay, this is interesting. Let’s think about what we’re doing to, I’m concerned that my enemies are going to do this, so I have to do this now.” And that mindset is how arms races take place. And once that starts to accelerate and the brakes are off, then I think it’s very, very hard to put it back in the box. So I think we have a very short time if we’re going to make some progress on regulation.

Zachary Kallenborn: Yeah. And the challenge I would mention is that if we look historically at arms control bans, they’ve typically come after the weapon systems have largely been used. I think that’s one of the fundamental challenges we see with LAWS that we really haven’t seen those on a large scale. In fact, I very briefly checked my email while we were chatting here. And I saw a note from arms control expert that I know who was commenting, “Oh, well, we haven’t even used drone swarms yet, so is this really much of a concern here?” And I think the challenge is how you get global, public as well as governments to recognize that. And I think the solution to that is fear. We often talk about fear is very negative emotion because we only ever experienced it in terrible situations. You’re not afraid when you’re sitting on your couch, holding hands with your partner or watching TV.

You feel fear when that angry looking dog growls at you or you’re sitting alone and there’s that weird noise downstairs. Like, “Was that really a robber or was it that I’m just being paranoid.” That fear motivates us to pay closer attention to a threat, to figure out what’s actually happening here and then take positive action to do something about it. And I think in order to get a broader ban on LAWS prior to it being used, there has to be that strong element of fear.

So I think work like Stuart’s around Slaughterbots and bringing this to global attention are extremely valuable for that. To recognize that these types of weapons while they may be useful in some cases, if we’re talking about non-autonomous systems really create very serious and real risks that global governments need to address now. And it’s better to do it now before things spread throughout the globe.

Stuart Russell: Yeah. And there is precedent for preemptive regulation and I would say biological weapons being one of them. Although, there were scattered uses of biological weapons in warfare. The kinds of weapons that people were developing were intended as global pandemics and the nation would vaccinate its own population and then wipe out the rest of the world and that was viewed as a legitimate form of warfare. So fortunately we didn’t wait until that was done in practice before regulating it.

Lucas Perry:All right. So I’m taking away the need for fear and the need for thinking about expected value. The expected value and engaging in another arms race in the 21st century might not be the best idea. But it’s also making me think of COVID and how poor humans are at preparing for high impact, low probability events and I feel slightly pessimistic about, like from what Zak said, it seems like we need to see the carnage of the thing before we actually take measures to avoid that risk in the future.

So thanks so much for joining me. Stuart and Zak it’s been a pleasure.

Stuart Russell: Okay. Thank you.

Zachary Kallenborn: Thanks for having me.

Beatrice Fihn on the Total Elimination of Nuclear Weapons

January 21, 2021/in Featured, FLI Podcasts, FLI podcasts & transcripts /by Lucas Perry

Future of Life Institute · Beatrice Fihn on the Total Elimination of Nuclear Weapons

The current nuclear weapons geopolitical situation
The risks and mechanics of accidental and intentional nuclear war
Policy proposals for reducing the risks of nuclear war
Deterrence theory
The Treaty on the Prohibition of Nuclear Weapons
Working towards the total elimination of nuclear weapons

4:28 Overview of the current nuclear weapons situation

6:47 The 9 nuclear weapons states, and accidental and intentional nuclear war

9:27 Accidental nuclear war and human systems

12:08 The risks of nuclear war in 2021 and nuclear stability

17:49 Toxic personalities and the human component of nuclear weapons

23:23 Policy proposals for reducing the risk of nuclear war

23:55 New START Treaty

25:42 What does it mean to maintain credible deterrence

26:45 ICAN and working on the Treaty on the Prohibition of Nuclear Weapons

28:00 Deterrence theoretic arguments for nuclear weapons

32:36 Reduction of nuclear weapons, no first use, removing ground based missile systems, removing hair-trigger alert, removing presidential authority to use nuclear weapons

39:13 Arguments for and against nuclear risk reduction policy proposals

46:02 Moving all of the United State’s nuclear weapons to bombers and nuclear submarines

48:27 Working towards and the theory of the total elimination of nuclear weapons

1:11:40 The value of the Treaty on the Prohibition of Nuclear Weapons

1:14:26 Elevating activism around nuclear weapons and messaging more skillfully

1:15:40 What the public needs to understand about nuclear weapons

1:16:35 World leaders’ views of the treaty

1:17:15 How to get involved

Max Tegmark and the FLI Team on 2020 and Existential Risk Reduction in the New Year

January 8, 2021/in Featured, FLI Podcasts, FLI podcasts & transcripts /by Lucas Perry

Future of Life Institute · Max Tegmark and the FLI Team on 2020 and Existential Risk Reduction in the New Year

FLI’s perspectives on 2020 and hopes for 2021
What our favorite projects from 2020 were
The biggest lessons we’ve learned from 2020
What we see as crucial and needed in 2021 to ensure and make improvements towards existential safety

54:35 Emilia Javorksy on the importance of returning to multilateralism and global dialogue

56:00 Jared Brown on the need for robust government engagement

57:30 Lucas Perry on the need for creating institutions for existential risk mitigation and global cooperation

1:00:10 Outro

Future of Life Award 2020: Saving 200,000,000 Lives by Eradicating Smallpox

December 11, 2020/in Featured, FLI Podcasts, FLI podcasts & transcripts /by Lucas Perry

Future of Life Institute · Future of Life Award 2020: Saving 200,000,000 Lives by Eradicating Smallpox

William Foege’s and Victor Zhdanov’s efforts to eradicate smallpox
Personal stories from Foege’s and Zhdanov’s lives
The history of smallpox
Biological issues of the 21st century

18:51 Implementing surveillance and containment throughout the world after success in West Africa

23:55 Wrapping up with eradication and dealing with the remnants of smallpox

25:35 Lab escape of smallpox in Birmingham England and the final natural case

27:20 Part 2: Introducing Michael Burkinsky as well as Victor and Katia Zhdanov

29:45 Introducing Victor Zhdanov Sr. and Alissa Zhdanov

31:05 Michael Burkinsky’s memories of Victor Zhdanov Sr.

39:26 Victor Zhdanov Jr.’s memories of Victor Zhdanov Sr.

46:15 Mushrooms with meat

47:56 Stealing the family car

49:27 Victor Zhdanov Sr.’s efforts at the WHO for smallpox eradication

58:27 Exploring Alissa’s book on Victor Zhdanov Sr.’s life

1:06:09 Michael’s view that Victor Zhdanov Sr. is unsung, especially in Russia

1:07:18 Part 3: William Foege on the history of smallpox and biology in the 21st century

1:07:32 The origin and history of smallpox

1:10:34 The origin and history of variolation and the vaccine

1:20:15 West African “healers” who would create smallpox outbreaks

1:22:25 The safety of the smallpox vaccine vs. modern vaccines

1:29:40 A favorite story of William Foege’s

1:35:50 Larry Brilliant and people central to the eradication efforts

1:37:33 Foege’s perspective on modern pandemics and human bias

1:47:56 What should we do after COVID-19 ends

1:49:30 Bio-terrorism, existential risk, and synthetic pandemics

1:53:20 Foege’s final thoughts on the importance of global health experts in politics

Sean Carroll on Consciousness, Physicalism, and the History of Intellectual Progress

December 1, 2020/in Featured, FLI Podcasts, FLI podcasts & transcripts /by Lucas Perry

Future of Life Institute · Sean Carroll on Consciousness, Physicalism, and the History of Intellectual Progress

Important intellectual movements and their merits
The evolution of metaphysical and epistemological views over human history
Consciousness, free will, and philosophical blunders
Lessons for the 21st century

Mohamed Abdalla on Big Tech, Ethics-washing, and the Threat on Academic Integrity

November 17, 2020/in Featured, FLI Podcasts, FLI podcasts & transcripts /by Lucas Perry

Future of Life Institute · Mohamed Abdalla on Big Tech, Ethics-washing, and the Threat on Academic Integrity

Topics discussed in this episode include:

How Big Tobacco used it’s wealth to obfuscate the harm of tobacco and appear socially responsible
The tactics shared by Big Tech and Big Tobacco to preform ethics-washing and avoid regulation
How Big Tech and Big Tobacco work to influence universities, scientists, researchers, and policy makers
How to combat the problem of ethics-washing in Big Tech

Timestamps:

0:00 Intro

1:55 How Big Tech actively distorts the academic landscape and what counts as big tech

6:00 How Big Tobacco has shaped industry research

12:17 The four tactics of Big Tobacco and Big Tech

13:34 Big Tech and Big Tobacco working to appear socially responsible

22:15 Big Tech and Big Tobacco working to influence the decisions made by funded universities

32:25 Big Tech and Big Tobacco working to influence research questions and the plans of individual scientists

51:53 Big Tech and Big Tobacco finding skeptics and critics of them and funding them to give the impression of social responsibility

1:00:24 Big Tech and being authentically socially responsible

1:11:41 Transformative AI, social responsibility, and the race to powerful AI systems

1:16:56 Ethics-washing as systemic

1:17:30 Action items for solving Ethics-washing

1:19:42 Has Mohamed received criticism for this paper?

1:20:07 Final thoughts from Mohamed

Citations:

Where to find Mohamed’s work

The Future of Life Institute AI policy page

We hope that you will continue to join in the conversations by following us or subscribing to our podcasts on Youtube, Spotify, SoundCloud, iTunes, Google Play, Stitcher, iHeartRadio, or your preferred podcast site/application. You can find all the AI Alignment Podcasts here.

You can listen to the podcast above or read the transcript below.

Lucas Perry: Welcome to the Future of Life Institute Podcast. I’m Lucas Perry. Today’s conversation is with Mohamed Abdalla on his paper The Grey Hoodie Project: Big Tobacco, Big Tech, and the threat on academic integrity. We explore how big tobacco has used and still uses its wealth and influence to obfuscate the harm of tobacco by funding certain kinds of research, conferences, and organizations, as well as influencing scientists, all to shape public opinion in order to avoid regulation and maximize profits. Mohammed explores in his paper and in this podcast how big technology companies engage in many of the same behaviors and tactics of big tobacco in order to protect their bottom line and appear to be socially responsible.

Some of the opinions presented in the podcast may be controversial or inflammatory to some of our normal audience. The Future of Life Institute support’s hearing a wide range of perspectives without taking a formal stance on it as an institution. If you’re interested to know more about FLI’s work in AI policy, you can head over to our policy page on our website at futureoflife.org/ai-policy, link in the description.

Mohamed Abdalla is a PhD student in the Natural Language Processing Group in the Department of Computer Science at the University of Toronto and a Vanier scholar, advised by Professor Frank Rudzicz and Professor Graeme Hirst. He holds affiliations with the Vector Institute for Artificial Intelligence, the Centre for Ethics, and ICES, formerly known as the Institute for Clinical and Evaluative Sciences.

And with that, let’s get into our conversation with Mohamed Abdalla

So we’re here today to discuss a recent paper of yours titled The Grey Hoodie Project: Big Tobacco, Big Tech, and the threat on academic integrity. To start things off here, I’m curious if you could paint in broad brush strokes how you view big tech as actively distorting the academic landscape to suit its needs, and how these efforts are modeled and similar to what big tobacco has done. And if you could also expand on what you mean by big tech, I think that would also be helpful for setting up the conversation.

Mohamed Abdalla: Yeah. So let’s define what big tech is. I think that’s the easiest of what we’re going to tackle. Although in itself, it’s actually not a very easy label to pin down. It’s unclear what makes a company big and what makes a company tech. So for example, is Yahoo still a big company or would it count as big tech? Is Disney a tech company? Because they clearly have a lot of technical capabilities, but I think most people would not consider them to be big tech. So what we did was we basically had a lot of conversation with a lot of researchers in our department, and we asked for a list of companies that they viewed as big tech. And we ended up with a list of 14 companies. Most of them we believe will be agreeable. Google, Facebook, Microsoft, Apple, Amazon, Nvidia, Intel, IBM Huawei, Samsung, Uber, Alibaba, Element AI, and OpenAI. This is a very restrictive set on what we believe the big tech companies are. Like for example, a clear missing one here is Oracle. There’s a lot of other big companies that are missing, but we’re not assigning this as a prescriptive or a definitive list of what big tech companies are.

But adding more companies to this list would only help us strengthen the conclusions we’ve drawn in our paper because they will show how much more influence these companies have. So by limiting it to a small group, we’re actually taking a pessimistic view on the maximum amount of influence that they have. That’s what we define as big tech.

Then the question comes, what do we mean by they have an outsized influence or how they go about influencing policy? And we will get into specific examples here, but I think the best way of demonstrating why there should be cause for concern is through a very simple analogy.

So imagine if there was a health policy conference, which had tens of thousands of researchers. And among the topics they discussed was how do you deal with the negative effects of increased tobacco usage. And its largest funding bodies were all big tobacco companies. Would this be socially acceptable? Would the field of health policy accept this? No. In fact, there are guidelines such as the Article 5.3 on the World Health Organization’s Framework for Convention on Tobacco Control. Which states that if you are developing public health policies with respect to tobacco control, you are required to act to protect these policies from commercial and other vested interests of the tobacco industry. So they are aware of the fact that industrial funding has a very large negative effect on the types of research, the types of conclusions, how strong the conclusions are that can be drawn from research.

But if we flip that around. So instead of health policy, replace machine learning policy or AI policy. Instead of big tobacco, you replace it with big tech. And instead of the negative effects of increased tobacco usage, the ethical concerns of increased AI deployment. Would this be accepted? And this is not even a hypothetical, because all of the big machine learning conferences among their top funding bodies are all these big tech companies. If you look at NeurIPS or you look at FAccT, the Fairness, Accountability, and Transparency Conference, their platinum sponsors or gold sponsors, whatever their highest level is depending on the conference, is all of these companies. Even if one wants to say it’s okay, because these companies are not the same as big tobacco, this should be justified. There is no justification for why we allow big tech to have such influence. I haven’t proven that this influence exists yet in my speaking so far. But there is precedence to believe that industrial funding warps research. And there’s been no critical thought of whether or not big tech, as computer science industrial funding warps research. And I argue in the paper that it does.

Lucas Perry: All right. So with regards to understanding how industry involvement in research and development of a field or area can have a lot of influence, what you take as a historical example to learn from the strategies and the playbook of a historical industry to see if big tech might be doing the same things as big tobacco. So can you explain in the broadest brush strokes, how big tobacco became involved in shaping industry research on the health effects of tobacco, and how big tech is using all of the same or most of the same moves to shape the research landscape to make big tech themselves have a public image of trust and accountability when that may not be the case?

Mohamed Abdalla: The history is that shortly after World War II in the mid 1950s, there was a pronounced decrease in demand for their product. What they believe caused or at least in part caused this drop in demand was a Reader’s Digest article that was published called Cancer by the Carton. And it discussed the scientific links between smoking and lung cancer.

It was later revealed after litigation that big tobacco actually knew about these links, but they also admitted it would result in increased legislation, decreased profits. So they didn’t want to publicly agree with the conclusions of the paper, despite having internal research which showed that this indeed was the case.

After this article was published, it was read by a lot of people and people were getting scared about the health effects of smoking. They employed a PR firm. And their first strategy was to publish a full-page ad in the New York Times that was seen by I think approximately 43 million people or so, which is a large percentage of the population. And they would go into state and I quote, “They accept an interest in people’s health as basic responsibility, paramount to every other consideration in our business.” So despite having internal research that showed that these links were conclusive in their full-page ad, not only did they state that they believed these links were not conclusive, they also lied and said that the health of their people was paramount to every other consideration, including profit. So clear, blatant lies, but dressed up really nicely. And it reads really well.

Another action that they were instructed to do by the PR firm was to fund academic research that would not only draw questions on the conclusiveness of these links, but also to sort of add noise, cause controversy, slowed down legislation. And I’ll go into the points specifically. But the idea to fund academia was actually the PR firms’ idea. And it was under the instruction of the PR firm that they funded academia. And despite their publicly stated goal of funding independent research because they wanted the truth and they care about health, it was revealed again after litigation that internal documents existed that showed that their true purpose was to sow doubt into the research that showed conclusive links between smoking and cancer.

Lucas Perry: Okay. And why would an industry want to do this? Why would they want to lie about the health implications of their products?

Mohamed Abdalla: Well, because there’s a profit motive behind every company’s actions. And that is basically unchanged to this day. Where while they may say a lot of nice sounding stuff, it’s just a simple fact of life that the strongest driving factor between any company’s decision is the profit motive. Especially if they’re publicly traded, they have a legal obligation to their shareholders to maximize profit. It’s not like they’re evil per se. They’re just working within the system that they’re in.

We see sort of the exact same thing with big tech. People can argue about when the decline of opinion regarding these big tech companies started. Especially if you’re American centric. Since I’m in Canada, you’re in the States. I think that the Cambridge Analytica scandal with Facebook can be seen as sort of a highlight, although Google has its own thing with Project Maven or Project Dragonfly.

And the Pew Research Firm shows that the amount of people that view big tech as having a net positive social change in the world started decreasing around the mid 2010s. And I’m only going to quote Facebook here, well Mark Zuckerberg specifically in his testimony to Congress. But in Congress, he would state, “It’s clear that we didn’t do enough and we didn’t focus enough on preventing abuse.” And he stressed that he admitted fault and they would double down on preventing abuse. And this was simply that they didn’t think about how people could do harm. And again, this statement did not mention the leaked internal emails, which stated that they were aware of companies breaking their policies. And they explicitly knew that Cambridge Analytica was breaking their scraping policies and chose to do nothing about it.

Even recently, there have been leaks. So Buzzfeed News leaked Sophie Zhang’s resignation letter, which basically stated that unless they thought that they were going to catch flack in terms of PR, they would not act to moderate a lot of the negative events that was happening on their platforms.

So this is a clear profit incentive thing, and there’s no reason to think that these companies are different. So then the question is how much of their funding of AI ethics or AI ethicists is driven by benevolent desire to see goodness in the world? And I’m sure there are people that work that have this sort of desire. But this is the four criticisms that you can get for reasons that you can fund academia. How much of it is used to reinvent yourself as socially responsible, influence the events and decisions made by funded universities, influence the research questions of individual scientists, and discover receptive academics.

So while we can assume that some people there may actually have the social good in mind and may want to improve society, we need to also consider these the reasons that big tobacco funded academia, and we need to check does big tech also fund academia? And is the effects of their funding academia the exact same as the effects of big tobacco funding academia? And if so, we as academics, or the public body, or government, or whoever needs to take steps to minimize the undue influence.

Lucas Perry: So I do want to spend the main body of this conversation on the big tobacco and big tech engage in these four points that you just illustrated, in order to reinvent themselves in the public image and be seen as socially responsible, even when they may not be actually trying to be socially responsible. It’s about the image and the perception. And then two, that they are trying to influence the events and decisions made by funded universities. And that three, they’ll be influencing the research questions and plans of individual scientists. This helps funnel research into areas that will make you look benevolent or socially responsible. Or you can funnel people away from topics that will lead to regulation.

And then the last one is to discover receptive academics who can be leveraged. If you’re in the oil industry and you can find a few scientists who have some degree of reputability and are willing to cast doubt on the science of climate change, then you’ve found some pretty good allies in your fight for your industry.

Mohamed Abdalla: Yep, exactly.

Lucas Perry: So before we jump into that, do you want to go through each of these points and say what big tobacco did in each of them and what big tech did in each of them? Or do you want to just start by saying everything that big tobacco did?

Mohamed Abdalla: Since there’s four points and there’s a lot of evidence for each of these points, I think it’s probably better to do for the first point, here’s what big tobacco did. And then here’s what big tech did.

Lucas Perry: Let’s go ahead and start then with this first point. From the perspective of big tobacco and big tech, what have they done to reinvent themselves in the public image as socially responsible? And again, you can just briefly touch on why it’s their incentive to do that, and not actually be socially responsible.

Mohamed Abdalla: So the benefit of framing yourself as socially responsible without having to actually take any actions to become socially responsible or to earn that label is basically increased consumer confidence, increased consumer counts, a decreased chance of legislation if the general public, and thereby the general politician believes that you are socially responsible and that you do care about the public good, you are less likely to be regulated as an industry. So that’s a big driving factor for trying to appear socially responsible. And we actually see this in both industries. I’ll cover it later, but a lot of stuff that’s leaked basically shows that spoiler, a lot of the AI research being done, especially in AI ethics is seen as a way to either delay or prevent the legislation of AI. Because they’re afraid that it will eat into their profit, which is against their profit motive, which is why they do a lot of the stuff that they do.

So first, I’ll go over what big tobacco did. And then we’ll try to draw parallels to what big tech did. To appear socially responsible, they were suggested by their PR firm Hill+Knowlton Strategies to fund academics and to create research centers. The biggest one that they created was CTR, which is the Council for Tobacco Research. And when they created CTR, they created it in a very academically appealing way. What I mean by that is that CTR was advised by distinguished scientists who serve on its scientific advisory boards. They went out of their way to recruit these scientists so that the research center gains academic respectability and is trusted by if not only the lay person, by academics in general. That they’re a respectable organization despite being funded by big tobacco.

And then what they would do is fund research questions. They’d act essentially as a pseudo granting body and provide grants to researchers who were working on specific questions that was decided by this council. At surface level, it seems okay. I’m not 100% sure how it works in the States, but at least in Canada we have research funding bodies. So we have NSERC or CIHR Natural Sciences and Engineering Research Council, which decides who gets the grants from the government’s research money. And we have it for all the different fields. And in theory, the research questions should be given in terms of validity of research, potential impact. A lot of the academically relevant considerations.

But what we ended up showing after litigation again, was that at least in the case of big tobacco, there were more lawyers than scientists involved in the distribution of money. And the lawyers were aware of what would and would not likely hurt the bottom line of these companies. So quoting previous work, their internal documents showed that they would simply refuse to fund any proposal that acknowledged that nicotine was addictive or that smoking was dangerous.

They basically went out of their way to fund research that was sort of unrelated to tobacco use so that they get good PR while minimizing the risk that said research would harm their profit motive. And during any sort of litigation, for example during a cigarette product liability trial, the lawyers presented a list of all the universities and medical schools supported by this Council for Tobacco Research as proof that they care about social responsibility and they care about the wellbeing. And they use this money as proof.

Basically at first glance, all of their external facing actions did seem that they cared about the well-being of people. But it was later revealed through internal documents that this was not the case. And this was basically a very calculated move to prevent legislation, beat litigation, and other self-serving goals in order to maximize profit.

In big tech, we see similar things happening. In 2016, the Partnership on AI to Benefit People and Society was established to, “Study and formulate best practices on AI technologies and to study AI and its influences on people and society.” Again, a seemingly very virtuous goal. And a lot of people signed up for this. A lot of non-profit organizations, academic bodies, and a lot of industries signed up for this. But it was later leaked that despite sounding rosy, the reality on the ground was a little bit darker. So reports from those involved, and this was a piece published at The Intercept. Demonstrated how neither prestigious academic institutions such as MIT, nor civil liberty organizations like the ACLU had much power in the direction of the partnership. So they ended up serving as a legitimate function for big tech’s goals. Basically railroading other institutions while having their brand on your work helps appear socially responsible. But if you don’t actually give them power, it’s only the appearance of social responsibility that you’re getting. You’re not actually being forced to be socially responsible.

There’s other examples of going to litigation specifically. During his testimony to Congress, Mark Zuckerberg states that in order to tackle this problems, they’ll work with independent academics. And these independent academics would be given oversight over their company. It’s unclear how an academic that is chosen by Facebook, theoretically compensated by Facebook, and could be fired by Facebook would be independent of Facebook after being chosen, receiving compensation, and knowing that they can lose that compensation if they do something to anger Facebook.

Another example almost word for word from big tobacco’s showing off to jurors is that Google boasts that it releases more than X research papers on topics in responsible AI in a year to demonstrate social responsibility. This is despite arm’s length involvement with the military minded startups. So if you build on that, Alphabet Google faced a lot of internal backlash with Project Maven, which is basically they’re working on image recognition algorithms for drones. They faced a lot of backlash. So publicly, they appeared to have stopped. They promised to stop working with the military. However, internally, Gradient Ventures, which is basically the venture capital arm of Alphabet still funds, provides researchers, and provides data to military startups. So despite their promise not to work in military, despite their research in responsible AI, they still work in areas that don’t necessarily fit the label of being socially responsible.

Lucas Perry: It seems there’s also this dynamic here where in both tobacco and in tech, it’s cheaper to pretend to be socially responsible than to actually be socially responsible. In the case of big tobacco, that would’ve actually meant dismantling the entire industry and maybe bringing e-cigarettes on 10 years before that actually happened. Yet in the case of big tech, it would seem to be more like hampering short term profit margins and putting a halt to recommender algorithms and systems that are already deployed that are having a dubious effect on American democracy and the wellbeing of the tens of millions of human brains that are getting fed garbage content by these algorithms.

So this first point seems pretty clear to me. I’m not sure if you have anything else that you’d like to add here. Public perception is just important. And if you can get policymakers and government to also think that you’re playing socially responsible, they’ll hold off on regulating you in the ways that you don’t want to be regulated.

Mohamed Abdalla: Yeah, that’s exactly it. Yeah.

Lucas Perry: Are these moves that are made by big tobacco and big tech also reflected in other contentious industries like in the oil industry or other greenhouse gas emitting energy industries? Are they making generally the same type of moves?

Mohamed Abdalla: Yeah, 100%. So this is simply industry’s reaction to any sort of possible legislation. Whether it’s big tobacco and smoking legislation, big tech and some sort of legislation on AI, oil companies and legislation on greenhouse gas emissions, clean energy, so on and so forth. Even a lot of food industry. I’m not sure what the proper term for it is, but a lot of the nutritional science research is heavily corrupted by funding from whether it’s Kellogg’s, or the meat industry, or the dairy industry. So that’s what industry does. They have a profit motive, and this is a profitable action to take. So it’s everywhere.

Lucas Perry: Yeah. So I mean, when the truth isn’t in your favor, and your incentive is profit, then obfuscating the truth is your goal.

Mohamed Abdalla: Exactly.

Lucas Perry: All right. So moving on to the second point then, how is it that big tobacco and big tech in your words, work to influence the events and decisions made by funded universities? And why does influencing the decisions made by funded universities even matter for large industries like big tobacco and big tech?

Mohamed Abdalla: So there’s multiple reasons to influence events. What it means to influence events is also a variety of actions. You could either hold events, you could stop holding events, or you can change how events that are being held operate. So events here, at least in the academic sense, I’m going to talk about conferences. And although they’re not always necessarily funded by universities, they are academic events. So why would you want to do this? Let’s talk about big tobacco first and show by example why they gained from doing this.

First, I’ll just go over some examples. So at my home university, the University of Toronto, Imperial Tobacco, which is one of the companies that belongs in big tobacco, withheld its funding from U of T’s Faculty of Law conference as retribution for the fact that U of T law students were influential in having criminal charges be laid against Shoppers Drug Mart for selling tobacco to a minor. As one of their spokespersons said, they were biting the hand that feeds them. If universities events such as this annual U of T law conference relies on funding from industry in general, then they have an oversized say of what you as an institution will do, or what people working for you can and cannot do. Because you’ll be scared of losing that consistent money.

Lucas Perry: I see. So you feed as many people as you can. Knowing that if they ever bite you, the retraction of your money or what you’re feeding them is an incentive for them to not hurt you?

Mohamed Abdalla: Exactly. But it’s not even the retraction, it’s the threat of retraction. If in the back of their mind 50% of their funding comes from whatever industry, can you afford to live without 50% of your funding? Most people would say no, and that causes worry and will call you to self self-censor. And that’s not a good thing in academia.

Lucas Perry: And replacing that funding is not always easy?

Mohamed Abdalla: It’s very difficult.

Lucas Perry: So not only are you getting the public image of being socially responsible by investing in some institutions or conferences, which appear to be socially responsible or which contain socially responsible workshops and portions. But then you also have in the back of the mind of the board that organizes these conferences, the worry and the knowledge that, “We can’t take a position on this because we’d lose our funding. Or this is too spicy, so we know we can’t take a position on this.” So you’re getting both of these benefits. The constraining of what they may do within the context of what may be deemed ethically and socially responsible, which may be to go against your industry in some strong way. But you also gain the appearance of just flat out being socially responsible while suppressing what would be socially responsible free discourse.

Mohamed Abdalla: 100%. And to build on that a little bit. Since we brought in boards, people that decide what happens. An easier way of influencing what happens is to actually plant or recruit friendly actors within academia. There is a history at least by my home university again, where the former president and dean of law of U of T was the director of a big tobacco company. And someone on the board of Women’s College Hospital, which is a teaching hospital affiliated with my university was the president and chief spokesperson for the Canadian Tobacco Manufacturer’s Council. So although there is no proof that they necessarily went out of their way to change the events held by the university, if a large percentage of your net worth is held in tobacco stocks, even if you’re a good human being, just because you’re a human being, you will have some sort of incentive to not hurt your own wellbeing. And that can influence the events that your university holds, the type of speakers that you invite, the types of stances that you allow your university to take.

Lucas Perry: So you talked about how big tobacco was doing this at your home university. How do you believe that big tech is engaging in this?

Mohamed Abdalla: The first thing that we’ll cover is the funding of large machine learning AI conferences. In addition to academic innovation or whatever reason they may say that they’re funding these conferences. And I believe that a large portion of this is because of academic innovation. You can see that the amount of funding that they provide also helps give them a say. Or at least in the back of the organizer’s mind. NeurIPS, which is the biggest machine learning AI conference has had always at least two big tech sponsors at the highest tier of funding since 2015. And in recent years, the number of big tech companies has exceeded five. This also carries over to workshops where over the past five years, only a single ethics related workshop did not have at least one organizer belonging to big tech. And that was 2018s robust AI in financial services workshop, which instead featured the foreheads of AI branches at big banks, which is not necessarily better. It’s not to say that those working in these companies should not have any say. But to have no venue that doesn’t rely on big tech in some sort of way or is not influenced in big tech in some sort of way is worrying.

Lucas Perry: Or whose existence is tied up in the incentives of big tech. Because whatever big tech’s incentives are, that’s generating the profit which is funding you. So you’re protecting that whole system when you accept money from it. And then your incentives become aligned with the incentives of the company that is suppressing socially responsible work.

Mohamed Abdalla: Yeah. Fully agree. In the next section where we talk about the individual researchers, I’ll go more into this. There’s a very reasonable framing of this issue where big tech isn’t purposely doing this. And industry is not purposely being influenced, but the influence is taking place. But basically exactly as you said. Even if these companies are not making any explicit demands or requests from the conference organizers, it’s only human nature to assume that these organizers would be worried or uncomfortable doing anything that would hurt their sponsors. And this type of self-censorship is worrying.

The example that I just showed was for NeurIPS, which is largely a technical conference. So Google does have incentive to fund technical research because a really good optimization algorithm will help their industry or their work, their products. But even when it comes to conferences that are not technical in their goal, so for example the FAccts conference, the Fairness Accountability, and Transparency Conference has never had a year without big tech funding at the highest level. Google’s three out of three years. Microsoft is two out of three years. And Facebook is two out of three years.

FAccT has a statement regarding sponsorship and financial support where they say that you have to disclose this funding. But it’s unclear how disclosure alone helps combat direct and indirect industrial pressures. A reaction that I often get is basically that those who are involved are very careful to disclose the potential conflicts of interest. But that is not a critical understanding of how the conflict of interest actually works. Disclosing a conflict of interest is not a solution. It’s just simply highlighting the fact that a problem exists.

In the public health sphere, researchers push that resources should be devoted to the problems associated with sequestration, which is elimination of relationships between commercial industry and professionals in all cases where it’s remotely feasible. So that means how this policy realizes that simply disclosing is not actually debiasing yourself.

Lucas Perry: That’s right. So you said that succinctly, disclosing is not debiasing. Another way of saying that is it’s basically just saying, “Hey, my incentives are misaligned here.” Full-stop.

Mohamed Abdalla: Exactly.

Lucas Perry: And then like okay, everyone knows that now, and that’s better than not. But your incentives are still misaligned towards the general good.

Mohamed Abdalla: Yeah, exactly. And it’s unclear why we think that AI ethicists are different from other forms of ethicists in their incorruptibility. Or maybe it’s an unfounded in terms of research view that thinks or believes that we would be able to post-hoc adjust for the biases of researchers, but that’s simply unfounded by research. So yeah, one of the ways that they influence that events is simply by funding the events and funding the people organizing these events.

But there’s also examples where some companies in big tech knowingly are manipulating the events. And I’ll quote here from my paper. “As part of a campaign by Google executives to shift the antitrust conversation, Google sponsored and planned a conference to influence policy makers going so far as to invite a token Google critic capable of giving some semblance of balance.” So it’s clear that these executives know what they’re doing, and they know that by influencing events, they will influence policy, which will influence legislation, and in turn litigation. So it’s clear from the leaks that are happening that this is not simply needless worrying, but this is an active goal of industry in order to maximize their profit.

There is some work for big tobacco that has not been done in big tech. I don’t think it can be done in big tech, and I’ll speak about why. But basically, when it comes to influencing events, there is research that shows that events held that were sponsored by big tobacco, such as symposiums or workshops about secondhand smoking are not only skewed, but also are poor quality compared to events not sponsored by big tobacco. So when it comes to big tobacco research, if the event is sponsored by big tobacco, the causation here is not clear whether or not it’s subconscious, whether or not it’s conscious. The causation might not be perfectly clear, but the results are. And it shows that if you’re funded by big tobacco, you’re more skewed and poorer quality in terms of research about the effects of secondhand smoking or the effects of smoking.

We can’t do this sort of research in big tech because there isn’t an event that isn’t sponsored by big tech. So we can’t even do this sort of research. And that should be worrying. If we know in other fields that sponsorship leads to lower quality of work, why are we not trying to have them divest from funding events directly anyway?

Lucas Perry: All right. Yeah. So this very clearly links up with the first example you gave at the beginning of our conversation about imagine having a health care conference and all of the main investors are big cigarette companies. Wouldn’t we have a problem with that?

So these large industries, which are having detrimental effects to society and civilization first have the incentive to portray a public image of social responsibility without actually being socially responsible. And then they also have an incentive to influence events and decisions made by funded universities as to one, make the incentives of those universities and events aligned with their own because their funding is dependent on these industries. And then also, to therefore constrain what can and may be said at these conferences in order to protect that funding. So the next one here is how is it that big tobacco and big tech influence the research questions and plans of individual scientists?

Mohamed Abdalla: So in the case of big tobacco, we know especially from leaked documents that they actively sought to fund research that placed the blame of lung cancer on anything other than smoking. So there’s the classic example on owning a bird is more likely to increase your chance of getting lung cancer. And that’s the reason why you got lung cancer instead of smoking.

Lucas Perry: Yeah, I thought that was hilarious. They were like, “Maybe it’s pets. I think it’s pets that are creating cancer.”

Mohamed Abdalla: Yeah exactly. And when they choose to fund this research question, not only do they get the positive PR. But they also get the ability to say that this is not conclusive. “Because look, here are these academics in your universities that think that it might not be smoking causing the cancer. So let’s hold off on litigation until this type of research is done.” So this steering of funds instead of exploring the effects of tobacco on lung cancer, they would study just the basic science of cancer instead. And this would limit the amount of negative PR that they get. So that’s one reason for doing it. But number two is it allows them sow doubt and say that there’s confusion, or that we haven’t arrived at some sort of consensus.

So that’s one of the ways that they did it, finding researchers who they termed critics or skeptics. And they would fund them and amplify their voices. And they had a specific set of money for set people, especially if they were smokers. So they tried to actively seek for people that were smokers because they felt that they’d be more sympathetic to these companies. They would purposely steer funds towards these people and they would change the research sphere.

There’s also very egregious actions that they took. So for example, Professor Stanton Glantz. He’s in UCSF I think, University of California, San Francisco. They would take out ads against him in the newspapers where they would put lies to point out flaws in his studies. And these flaws aren’t really flaws. It’s just a twisting of the truth. It’s basically if you go against us, we’re going to attack you. We’re going to make it very hard for you to get further funding. You’re going to have a lot of bad PR. It’s just sort of dis-incentivizing anyone else from doing critical issues against them.

They would work with elected politicians as well to block funding of scientists of opposing viewpoints. So it’s not like they didn’t have their fingers in government as well. During litigation, an email covered where HHS, which is the U.S. Department of Health and Human Services appropriations continuing resolution will include language to prohibit funding for Glantz who is Stanton Glantz, the same scientist. So it’s clear that through intimidation, but also through acting as a funding body, they’re able to change what researchers have work on.

Big tech works in essentially the same way. The first thing to note here is that when it comes to ethical AI or AI ethics, big tech in general has a very specific conception of what it means for an algorithm to be ethical. Whether it’s inspired by their insular culture where it’s sort of a very echo-y place where everyone sort of agrees with each other, there’s sort of an agreed upon culture. There is previous work owning ethics that discusses how there’s three main factors in defining AI ethics. And there’s three values of Silicon Valley. Meritocracy, trust in the market. And I forgot the last one. And basically, their definition is simply different from that, which the rest of the world, or the rest of the country generally has.

So Silicon Valley has a very specific view of what AI ethics is or should be. And that is not necessarily shared by everyone outside of Silicon Valley. That is not necessarily in itself a bad thing. But when they act as a pseudo granting body that is, they provide grants or money for researchers, it becomes an issue. Because if for example, you are a professor. And as a professor, one of your biggest roles is simply to bring in money to do research. And if your research question does not agree with the underlying logics of Silicon Valley’s funding bodies, whoever makes these decisions.

Lucas Perry: Like if you question the assumption about trust in the market?

Mohamed Abdalla: Yeah, exactly. Or you question meritocracy, and maybe that’s not how we should be basing our societal values. Even if the people granting the money are not lawyers like they were in big tobacco, even if they were research scientists, the fact that they’re at a high enough level to be choosing who gets money likely means that they’ve been there for awhile. And there’s an increased chance that they personally believe or agree with the views that their companies hold. Not necessarily always the case, but the probability is a lot higher.

So if you believe in what your company believes in, and there is a researcher who is working from a totally different set of foundations whose assumptions do not match your assumption. As human nature, you’re less likely to believe in this research. You’re less likely to believe that it’s successful or on the true path. So you’re less likely to fund it. And that requires no malicious intent on your side. It’s just that you are part of an industry that has a specific view. And if a researcher does not share this view, you’re not going to give them that money.

And you switch it over to the researcher side. If I want to get tenure, I’m not a professor. So I can’t even get tenure. But if I want to get hired, I have to show that I can bring in money. If I’m a professor and I want to get tenure, I have to show that I can bring in even more money. And if I see that these companies are giving away vast sums of money, it is in my best interest to ask a research question that will be funded by them. Because what good am I if I get no money and I don’t get hired? Or what good am I if I get no money and I don’t get tenure?

So what will end up happening there is that it’s a cyclical thing where researchers view that these companies fund specific types of research or researchers that are based in fundamental assumptions that they may not necessarily agree with. And in order to maximize their opportunities to get this money, they will change their research question. Whether it’s complete, or slight adjustment, or changing the assumptions to match what will get them the money. And the cycle will just keep happening until there’s no difference.

Lucas Perry: So there’s less opportunity for researchers and institutions that fundamentally disagree with some axiom that these industries hold over ethics and accountability, or whatever else?

Mohamed Abdalla: Exactly. 100%. And the important thing to note here is that for this to happen, no one needs to be acting maliciously. The people in big tech probably believe in what they’re pushing for. At least I like to make this assumption. And I think it makes it for the easiest sell, especially for those within the computer science department because there’s a lot of pushback to this type of thought. Where even if the people deciding who gets the money, imagine they’re completely disinterested researchers who have very agreeable goals, and they love society, and they want the general good. The fact that they are in a position to be deciding who gets the money means that they’re likely higher up in these companies. You don’t get to be higher up and stay for these companies long enough, unless you agree with the viewpoint.

Lucas Perry: And that viewpoint though, in a market has to be in some sense, aligned with the impersonal global corporate objective of maximizing the bottom line. There’s this values filtration process internally in a company where maybe you’ll have all the people who are against Project Maven, but none of them are high enough. Right?

Mohamed Abdalla: Exactly.

Lucas Perry: You need to sift those people out for the higher positions because the higher positions are the ones which have to be aligned with the bottom line, with maximizing profits for shareholders. Those people could authentically think that maximizing the profit of some big industrial company is a good thing, because you really trust in the market and how it serves the market.

Mohamed Abdalla: I think there are people that actually believe this. So I know you say it kind of disbelievingly, but I think that people actually believe this.

Lucas Perry: Yeah, people really do believe this. I don’t actually think about this stuff a lot. But yeah. I mean, it makes sense to me. We buy all your stuff. So you’re serving me, I’m making a transaction with you. But this fact about this values sifting towards the top to be aligned with the profit maximization, those are the values that will remain for then deciding the funding of researchers at institutions. So no one has to be evil in the process. You just have to be following the impersonal incentives of a global capitalist industry.

Mohamed Abdalla: Yeah. I do not aim to shame anybody involved from either side. Certain executives I shame and certain attorneys I shame. But I work under the assumption that all computer science, AI, ethicists, researchers, whatever you want to call them are well-intentioned. And the way the system is set up is that even well-intentioned researchers can have negative impact on the research being done, and can have a limiting impact on the types of questions being considered.

And I hope by now you agree that at least theoretically, that by acting as a pseudo granting body, there’s a chance for this influence to occur. But then in my work, what I did was I actually counted how many people were actually looking to big tech as a pseudo granting body. So I looked at the CVs of all computer science faculty at four schools. University of Toronto, Massachusetts Institute of Technology, Stanford, and Berkeley. Two private schools, two public schools. Two eastern coast universities, two western coast. And for each CV that I could find, I looked to answer a certain number of questions. So whether or not a specific faculty works on AI, whether or not they work on the ethics of AI. I very loosely defined that as being having at least one paper defined about any sort of societal impact of AI. Whether or not they have ever received faculty funding from big tech. So that is grants or awards from companies. Whether they have received graduate funding from big tech. So was any portion of this faculty’s graduate education funded by big tech? And whether or not they are or were employed by big tech. So at any time that they have any sort of previous or current financial relationship with big tech?

What the research shows is that of all computer science faculty, 52% of them. So at least half view big tech as a funding body. So that means as a professor, they have received a grant or an award to do research from big tech. And universities technically are here to not maximize profit for these companies, but to do science. And in theory, public good kind of things. At least half of the researchers are looking to these companies as granting bodies.

If you narrow that down to computer science faculty that work in AI, that percentage goes up to 58%. If you limit it to computer science faculty who work in the ethics of AI or who are AI ethicists, it remains at 58%. Which means that 58% of the people looking to answer the really hard questions about AI and society, whether it’s short-term or long-term, view these companies as a funding body. Which in turn, as we discussed, opens them up to influence whether it’s subconscious or conscious.

Lucas Perry: So then if you’re Mohamed Abdalla and you come out with a paper like you came out with, is the thought here that it’s very much less likely than in the future you will receive grants from big tech?

Mohamed Abdalla: So it’s unclear. There’s a meta game to play here as well. A classic example here is Michael Moore. The filmmaker, political activist. I’m not sure the title you want to give him. But a lot of his films are funded by Fox or some subsidiary of Fox.

Lucas Perry: Yeah. But they’re all leftist views.

Mohamed Abdalla: Exactly. So as in the example that I gave previously where Google would invite a token critic to their conferences to give it some semblance of balance, simply disagreeing with them will not reject you from their funding. It’s just that they will likely limit the amount of people who are publicly disagreeing with them by choosing. Again, it seems too self-serving to say, “I’m a martyr. I’ve sacrificed myself.” I don’t view that as the case, although I did get some feedback saying, “Maybe you shouldn’t push this now until you get a job,” kind of thing. But I’m pushing that it shouldn’t be researchers deciding who they get money from. This is a higher level issue.

If you go into a pure hypothetical. And for the listeners, this is not what I actually believe. But let us consider big tech to be evil, right? And publicly minded researchers who refuse to take money from big tech as good. If every good researcher. Again, good here is not being used in the prescriptive sense, but just in our hypothetical. If all of the good researchers refuse to take money from these evil corporations, then what you’re going to be ending up with is these researchers will not get jobs, will not get promoted. Their viewpoints will die out. But also, the people who are not good will have no problem taking this money. And they will be less likely to challenge these evil corporations. So in a game theoretic perspective, if you go from a pure utility perspective, it makes sense for you as a good researchers to take this bad money.

So that’s why I state in the paper whatever our fix to this is, can’t be done at the individual researcher level. You have to assume that all researchers are good, but you have to come up with a system level solution. Whether that’s legislation from governments, whether that’s a funding body solution, or a collection of institutions that come up with an institutional policy that applies to all of these top schools or all computer science departments all over the world. Whoever we can get to agree together. So that’s basically what I’m pushing for. But there’s also ways that you can influence research questions without directly funding them. And the way that you do this is by repeated exposure to your ideas of ethics or your ideas of what is fair, what is not fair. However you want to phrase it.

I got a lot of puzzled looks when I tell people that I also looked at whether or not a professor was funded during graduate school by these companies. And there is some rightful questioning there. Because I saying, or am I assuming that the fact that they got a scholarship from let’s say Microsoft during their PhD, is that going to impact their research question 20 years down the line when they’re a professor? And I do not think that’s actually how this works. But the reason for asking this was to show how often or how much exposure these faculty members were given to big tech’s values or Silicon Valley values, however you want to say it.

Even if they’re not actively going out of their way to give money to researchers to affect their research questions. If every single person who becomes a faculty member at all of these prestigious schools has at one point done some term. Whether that’s a four month internship, or a one-year stint, or a multiple year stint in big tech in Silicon Valley. It’s only human to worry that repeated exposure to such views will impact whatever views you end up developing yourself. Especially if you’re not going into this environment trying to critically examine their views. You’re just likely to adopt it internally, subconsciously before you have to think about it.

And what we show here is that 84% of all computer science faculty have had some sort of financial connection with big tech. Whether that’s receiving funding as graduate, or a faculty member, or been previously employed.

Lucas Perry: We know what the incentives of these industries are all about. So why would they even be interested in funding the graduate work of someone, if it wasn’t going to groom them in some sense? Are there tax reasons?

Mohamed Abdalla: I’m not 100% sure how it works in the United States. It exists in Canada, but I’m not sure if it does in the U.S.

Lucas Perry: Okay.

Mohamed Abdalla: So there’s multiple reasons for doing this. There is of course as usual, the PR aspects of it. We are helping students pay off their student loans in the states I guess. There’s also if you fund someone’s graduate funding, you’re building connections, making them easier to hire possibly.

Lucas Perry: Oh yeah. You win the talent war.

Mohamed Abdalla: Yeah, exactly. Yeah. If you want a Microsoft Fellowship, I think you also win an internship at Microsoft, which makes you more likely to work for Microsoft. So it’s also a semi-hiring thing. There’s a lot of reasons for them to do this. And I can’t say that to influence is the only reason. But the fact that if you limit it to CS faculty who work in AI ethics, 97% of them have had some sort of financial connection to big tech. 97% of them have had exposure to the dominant views of ethics by Silicon Valley. What percentage of this 97 is going to subconsciously accept these views, or adopt these views because they haven’t been presented with another view, or they haven’t been presented with the opportunity to consider another critical view that disagrees with their fundamental assumptions? It’s not to say that it’s impossible, it’s just to say should they be having such a large influence?

Lucas Perry: So we’ve spent a lot of time here then on this third point on influencing the research questions and plans of individual scientists. It seems largely again that by giving money, you can help align their incentives with your own. You can help direct what kind of research questions you care about. You can help give the impression of social responsibility. When actually, you’re constraining and funneling research interest and research activity into places which are beneficial to you. You’re also, I think you’re arguing here exposing researchers to your values and your community.

Mohamed Abdalla: Yeah. Not everyone’s view of ethics is fully formed when they get a lot of exposure to big tech. And this is worrying because if you’re a blank slate, you’re much more easily drawn upon. So they’re more likely to impart their views on you. And if 97% of the people are exposed, it’s only safe to assume that some percentage will draw this. And then that will artificially inflate the amount of people that agree with big tech’s viewpoints. And therefore further push academia or the academic conversation into alignments with something they find favorable.

Lucas Perry: All right. So the last point here then is how big tobacco and big tech discover receptive academics who can be leveraged. So this is the point about finding someone who may be a skeptic or critic in a community of some widely held scientific view, and then funding and propping them up so that they introduce some level of fake or constructed, and artificial doubt and skepticism and obfuscation of the issue. So would you like to unpack how big tobacco and big tech have done this?

Mohamed Abdalla: When it comes to the big tobacco, we did cover this a tiny bit before. For example, when we talked about how they would fund research that questions whether it’s actually keeping birds as pets that caused lung cancer. And so long as this research is still being funded and has not been published in the journal, they can honestly speaking say that it is not yet conclusive. There is research being done, and there are other possible causes being studied.

Despite having internal research to show that this is not true. If you go from a pure logic standpoint where is it conclusive being defined as there exists no oppositional research, they’ve satisfied the conditions such that it is not conclusive and there is fake doubt.

Lucas Perry: Yeah. You’re making logically accurate statements, but they’re epistemically dishonest.

Mohamed Abdalla: Yeah. And that’s basically what they do when they leverage these academics to sow doubt. But they knew that especially in Europe a little bit after this, that there was a lot of concern being drawn regarding the funding of academics by big tobacco. So they would purposefully search for European scientists who had no previous connection to them who they could leverage to testify. And this was part of a larger project that they called the White Coat Project, which resulted in infiltrations in governing bodies, heads of academia, and editorial boards to help with litigation legislation. And that’s actually why I named my paper The Grey Hoodie Project. It’s an homage to the White Coat Project. But since computer scientists don’t actually wear white coats, we’re more likely to wear gray hoodies. That’s where the name of the paper comes from. So that’s how big tobacco did it.

When it comes to big tech, we have clear evidence that they have done the same. Although it’s not clear the scope of which they have done this because there haven’t been enough leaks yet. This is not something that’s usually publicly facing. But Eric Schmidt, previously the CEO of Google was, and I quote from an Intercept article, “Advised on which academic AI ethicists his private foundation should fund.” I think Eric Schmidt has very special views regarding the place of big tech and its impact on society that likely would not be agreed with by the majority of AI ethicists. However, if they find an ethicist that they agree with and they amplify him and give him hundreds of thousands of dollars a year, he is basically pushing his viewpoint on the rest of the community by way of funding.

Another example where Eric again, Eric Schmidt asked that he should fund a certain professor, and this certain professor later served as an expert consultant to the Pentagon’s innovation board. And Eric Schmidt is now on some military advisement role in the U.S. government. And that’s a clear example of how those from big tech are looking to leverage receptive academics. We don’t have a lot of examples from other companies. But the fact that it is happening and this one got leaked, do we have to wait until other ones get leaked to worry about this?

An interesting example that I personally view as quite weak. I don’t like this example, but the irony will show in a little while. There is a professor at George Mason University who had written academic research that was funded indirectly by Google. And his research criticized the antitrust scrutiny of Google shortly before joining the FTC, the Federal Trade Commission. And after he joined the FTC, they dropped their antitrust suits. They’ve picked it up now again. But this claim, which basically draws into question whether or not Google funded him one, because of his criticism of antitrust against Google. So that is a possible reason they chose to fund him. There’s another unstated question here in this example with did he choose to criticize antitrust scrutiny of Google because they fund him? So which direction does this flow? It’s possible that neither direction flow. But when he joined the FTC, did they drop their case because essentially they hired a compromised academic?

I do not believe and I have no proof of any of this. But Google’s response to this hinting question was that this expose was pushed by the Campaign for Accountability. And Google said that this evidence should not be acceptable because this nonprofit Campaign for Accountability is largely funded by Oracle, which is another tech company.

So if you sort of abstract this away, what Google is saying is that claims made regarding societal impacts, or legislation, or anything to do with AI ethics. If that researcher is funded by a big tech company, we should be worried about what they’re saying, or we should be very skeptical about what they’re saying. Because they’re essentially saying that you should not trust this because it’s funded by Oracle. It’s largely backed by Oracle. You know, you abstract that away, it’s largely backed by big tech. Does that not apply to everything that Google does or everything that big tech in general does? So it is clear that they themselves know that industry money has a corrupting influence on the type of research being done. And that sort of just pushes my entire piece.

Lucas Perry: Yeah. I mean at some sense, none of this is mysterious. They couldn’t not be doing this. We know what industry wants and does, and they’re full of smart people. So, I mean, if someone from industry who is participating in this or listening to this conversation, they would be like, “You’ve woken up to the obvious. Good job.” And that’s not to downplay the insight of your work yet. It also makes me think of lobbying.

Mohamed Abdalla: 100%.

Lucas Perry: We could figure out all of the machinations of lobbying and it would be like, “Well yeah, they couldn’t not be doing this, given their incentives.”

Mohamed Abdalla: So I fully agree. If you come into this knowing all of the incentives, what they’re doing is the logical move. I fully agree that this is obvious, right?

Lucas Perry: I don’t think it’s obvious. I think it naturally follows from first principles, but I feel like I learned a lot from your paper. Not everyone knows this. I would say not even many people know this.

Mohamed Abdalla I guess obviously it wasn’t the correct word. But I was going to say that the points that I raise show that there’s a clear concern here. And I think that once people hear the points, they’re more likely to believe this. But there are people in academia. A common criticism I get is that people know who pay them. So they say that it’s unfair to assume that someone funded by a company cannot be critical of that company or big tech in general. And several researchers who work at these companies are critical of their employer’s technology. So the point of my work is to lay this out flat to show that it doesn’t matter if people know who pay them, the academic literature shows that this has a negative effect. And therefore, disclosure isn’t enough. I don’t want to name the person who said this criticism, but they’re pretty high up. The idea that conflict of interest is okay simply because it’s disclosed seems to be a uniquely computer science phenomenon.

Lucas Perry: Yeah. It’s a weird claim to be able to say, “I’m so smart and powerful, and I have a PhD, and giving me dirty money or money that carries with it certain incentives, I’m just free of that.”

Mohamed Abdalla: Yeah. Or whether or not it’s incorrectly perceived ability to self-correct for these biases. That’s sort of the current that I’m trying to fight against. Because the mainstream current in academia is sort of like, “Yeah, but we know who pays us, so we’re going to adjust for it.” And although the conclusions I draw are intuitive I think, the intuition that people have regarding big tech is basically, big tobacco, everyone has an intuitively negative gut feeling. So it’s very easy for them to agree. It’s a little bit more difficult to convince them that even if you believe that big tech is a force for good, you should still be worried.

Lucas Perry: I also think that the word here that is better than obvious is it’s self-evident once it’s been explained. It’s not obvious. Because if it were obvious, then you wouldn’t have needed to write this paper. And I already would’ve known about this, and everyone would have. So if you were just to wrap up and summarize in a few bullet points here this last point on discovering receptive academics and leveraging them, how would you do that?

Mohamed Abdalla: I kind of summarize this to policymakers. When policymakers to try to make policy, they tend to converse with three main parties. They will converse with industry, they converse with the academics, and they converse with the public. And they believe that getting this wide viewpoint will help them arrive at the best compromise to help society move in the way that it should. However, the very mindful way that big tech is trying to leverage academics, a policymaker will talk to industry. He’ll talk to the very specific researchers who are handpicked by industry. And therefore, are basically in agreement with the industry and they will talk to the public. So two thirds of the voices they hear are industry aligned voices, as opposed to previously one third. And that’s something that I cover in the paper.

And that’s the reason why you want to leverage receptive academics, because it gives you the ability so that the majority of whatever a policymaker hears, and they’re really busy people, and they don’t have the time to do the research themselves. If two out of every three people is pushing policy or pushing views that are in alignment with whatever’s good for big tech’s profit motive, then you’re more likely to believe that viewpoint. As opposed to having an independent academia where if the right decision is to agree with big tech then you assume they would. If the right decision is to disagree, then you assume they would. But if they leverage the academics, this is less likely to happen. Therefore, academia is not playing its proper role when it comes to policy-making.

Lucas Perry: All right. So I think this pretty clearly lays out then how industries in general, whether it be big tobacco, big tech, oil companies, greenhouse gas emitting energy companies, you even brought up the food industry. I mean, just anyone really who have the bottom line as their incentive. These strategies are just naturally born of the impersonal incentive structure of a corporation or industry.

This next question is maybe a bit more optimistic. All these organizations are made up of people, and these people are all I guess more or less good or more or less altruistic. And you expect that if we don’t go extinct, these industries always get caught, right? Big tobacco got caught, oil industries are in the midst of getting caught. And next we have big tech. And I mean, the dynamics are also a little bit different because cigarettes and oil can be booted. But we’re kind of married to the technology of big tech forever. Literally.

Mohamed Abdalla: I would agree with that.

Lucas Perry: Yeah. So the strategy for those two seems to be obfuscate the issue for as long as possible so your industry exists as long as possible, and then you will die. There is no socially responsible version of your industry. That’s not going to happen with big tech. I mean, technology is here to stay. So does big tech have any actual incentives for genuine social responsibility, or are they just playing the optimal game from their end where you obfuscate for as long as possible, and you bias all of the events and the researchers as much as possible? Eventually, there’ll be enough podcasts like this and minds changed that they can’t do that any longer without incurring a large social cost in opinion, and perhaps market. So is it always simply the case that promoting the facade of being socially responsible is cheaper and better than the incentive of actually becoming socially responsible?

Mohamed Abdalla: So a thing that I have to say, because the people that I worked with that still work in health policy regarding tobacco would be hurt if I didn’t say it. Big tobacco is still heavily investing in academia, and they’re still heavily pushing research and certain viewpoints. And although the general perception has shifted regarding big tobacco, they’re not done yet. So although I do agree with your conclusion that it is a matter of time when they’re done, to think that the fight is over is simply not true yet. There’s still a lot of health policy folks that are pushing as hard as they can to completely get rid of them. Even within the United States and Europe, they create new institutions that do other research. They’ve become maybe a little bit more subtle about it. But to declare victory I think is the correct path on what will happen, but it has not yet happened. So there’s still work to be done.

Regarding whether or not big tech has an actual incentive to do good, I like to assume the best of people. I assume that Mark Zuckerberg actually founded Facebook because he actually cared about connecting people. I believe that in his heart of hearts, he does have at least generally speaking, a positive goal for society. He doesn’t want to necessarily do bad or be wrecking democracies across the world. So I don’t think that’s his goal, right?

So I think that starting from that viewpoint is helpful because one, it will make you heard. But also, it shows how this is a largely systemic issue. Because despite his well-intentioned goals that we’re assuming exist. And I actually do believe at some level, it’s true. The incentives in the system in which he plays adds a caveat to everything he says, that we aren’t putting there

So for example, when Facebook says they care about social responsibility or that they will take steps to minimize the amount of fake news, whatever that means. All of the statements made by any industry in any company, because of the fact that we’re in a capitalist system, is prior on the statements given it does not hamper profits, right? So when Facebook wants to deal with fake news, they will turn to automated AI algorithms. And they say we’re doing this because it’s impossible to moderate the amount of stories that we get.

From a strictly numeric perspective, this is true. But what they’re not saying is that it is not possible for us to use humans to moderate all of these stories while staying profitable. So that is to say the starting point of their action may be positive. But the fact that it has to be warped to fit the profit motive ends up largely negating, if not completely negating the effects of the actions they take.

So for example, you can take Facebook’s content moderation in the continent of Africa. They used to have none. And until recently, they got only one content center moderation in the entire continent of Africa. The amount of languages spoken in that continent alone, how many people do you have to hire in that one continent’s moderation? How many people per language are you hiring? Sophie Zhang’s resignation letter basically showed that despite being aware of all of these issues and having employees especially at the lower levels who were passionate about the social good. So it’s clear that they are trying to do a social good. The fact that everything is prior to whether or not it will result in money, hurts the end result of their action. So I believe and I agree with you that this industry is different. And I do believe that they have an incentive for the social good. But unless this incentive is forced upon everyone else, they are hurting themselves if they refuse to take profit that they could take, if that makes sense.

But if you choose to not do something because it is socially good but it will hurt your profits, some other company is going to do that thing. And they will take the profits and they will beat your market share until you can find a way to account for it in the stock price.

Lucas Perry: People value you more now that you are being good.

Mohamed Abdalla: Yeah. But I don’t think we’re at a stage where that’s possible or that’s even well-defined what it means. So I agree that even if this research is well-intentioned, the road to hell is paved with good intentions.

Lucas Perry: Yeah. Good intentions lead to bad incentives.

Mohamed Abdalla: Or the good incentives are required to be forced through the lens of bad incentive. It has to be aligned with the bad incentive for them to actually manifest. Otherwise, they will always get blocked.

Lucas Perry: Yeah. By that you mean the things which are good for society must be aligned with the bad incentives of maximizing profit share, or they will not manifest.

Mohamed Abdalla: Exactly. And that’s the issue when it comes to funding academia. Because it is possible to change society’s viewpoint on one, what is possible. But two, what is preferable to match the profit incentives of these companies. So you could find a way of what is ethical AI, what does it cover? What sort of legislation is feasible? What sort of legislation is desirable? In what context does it apply, does it not apply? What jurisdiction, so on and so forth. These are all still open questions. And it is the incentive of these companies to help mold these answers such that they have to change as little as possible.

Lucas Perry: So when we have benefits that are not accruing from industry, or where we have negative externalities or negative effects from the incentives of industry leading to detrimental outcomes for society, the thing that we have for remedying that is regulation. And I imagine that more than the general population, I would guess are libertarian attitudes at big tech companies. Which in this sense I would summarize as socially liberal or left leaning and then against regulation. So valuing the free market. So there’s this value resistance. We talked about how the people at the top are going to be sifted through. You’re not going to have people at the top of big tech companies who really love regulation, or think that regulation is really good for making a beautiful world. Because regulation is just always hampering the bottom line.

Yet, it’s the tool that we have for trying to mitigate negative externalities and negative outcomes from industry maximizing their bottom line. So what do you suggest that we do? Is it just that we need good regulation? We need to find some meaningful regulatory system and effective policy? Because otherwise, nothing will happen. They’ll just keep following their incentives, and they have so much power. And they’ll just do what they do to keep doing the same thing. And the only way to break that is regulation.

Mohamed Abdalla: So I agree. The solution is basically regulation. The question is, how do we go about getting there? Or what specific rules do we want to use or laws do we want to create? And I don’t actually answer any of this in my work. I answer a question that comes before the legislation or the regulation. Which is basically, I propose that AI ethics should be a different department from computer science. So that in the same way that bioethics is no longer in the same department as biology or medicine, AI ethics should be its own separate department. And in that way, anyone working in this department is not allowed to have any sort of relationship with these companies.

Lucas Perry: You call that sequestration.

Mohamed Abdalla: It’s not my own term. But yeah, that’s what it’s called.

Lucas Perry: Yeah. Okay. So this is where you’re just removing all of the incentives. Whether you’re declaring conflict of interest or not, you’re just removing the conflict of interest.

Mohamed Abdalla: Yes. Putting myself on the spot here, it’s very difficult to assume that I myself have not been corrupted by repeated exposure. As much as I try to view myself as a critical thinker, the research shows repeated exposure will influence what you think and what you believe. I’ve interned at Google for example, and they have a very large amount of internal propaganda pointed at their employees.

So I can’t barge in here saying that I am a clean slate or, “I’m a clean person. You should listen to my policies.” But I think that academia should try to create an environment where it is possible. Or dare I say, encouraged to be a clean person where clean means no financial involvement with these companies.

That said, there’s a lot of steps that can be done when it comes to regulation. Slightly unrelated, but kind of not unrelated is fixing the tax code in the U.S. and Canada and around the world. A large reason of why a lot of computer science faculty and computer scientists in general look to industry for funding is because governments have been cutting or at least not increasing with the rates of research being done in fields the amount of money available for research funding. And why do they not have as much money? This is probably in part because these companies are not paying their fair share when it comes to paying their taxes, which is how a lot of researchers get their funding. That’s one way of doing it. If you want to go into specifics, it’s more difficult and much harder to sell for specific policies. I don’t think regulation of specific technologies would be effective because all the technologies changed very fast.

I think creating a governmental body whose role it is to sue these companies when they violate stuff that we don’t believe match our social norms is probably the way to go about it. But I don’t know. It’s hard for me to say. It’s a difficult question that I don’t have an answer for. We don’t even know who to ask for legislation because every computer scientist is sort of corrupted. And they’re like, “Okay, do we not use computer scientists at all? Are we relying only on economists and moral philosophers to do this sort of legislation possibly?” I don’t know.

Lucas Perry: So I want to talk a little bit about transformative AI, and the role that this transition plays in that. There is a sense, and this is a meme that I think needs to be combated. The race between China and America on AI with the end goal of that being AI systems that are increasingly powerful.

So some sense that any kind of regulation used to try to fix any of these negative externalities from these incentives is just shooting ourself in the knee. And the evil other is racing to beat us.

Mohamed Abdalla: That’s the Eric Schmidt argument.

Lucas Perry: So we can’t be implementing these kinds of regulations in the face of the geopolitical and international problem of racing to ever more powerful AI systems. So you already said this is the Eric Schmidt argument. What is your reaction to this kind of argument?

Mohamed Abdalla: There’s multiple possible reactions. And I don’t like to state which one I believe in personally, but I’d like to walk through them. Because I think first off, let us assume that the U.S. and China are racing for an automated general intelligence, AGI. Would you not then increase government funding and nationalize this research such that it belongs to the government and not to a multinational corporation? In the same way that if for example, Google, Facebook, Microsoft, Alibaba, Huawei were in the race to develop nukes. Would you say, leave these companies alone so they can develop nuclear weapons? And once they develop a nuke, we’ll be able to take it. Or would you not nationalize these companies? Or not nationalize them, but basically it has to become only for the U.S. They cannot have any incentives in any other country. That is a form of legislation or regulation.

Governments would have to have a much bigger say in the type of research being done, who’s doing it, what can be done. For example, in the aerospace industry, you can’t employ non U.S. citizens. Is this what you’re pushing for in artificial intelligence research? Because if not, then you’re conceding that it’s not likely to happen. But if you do believe that this is likely to happen, then you would be pushing for some sort of regulation. You could argue what the regulation, but I don’t find the viewpoint that we want these companies to compete with the Chinese companies, because they’re going to create this thing that we need to beat the Chinese at. If you believe that this is going to happen, you’d be still in support of regulation. It’d just be different regulation.

Lucas Perry: I mean obviously I can’t speak for Eric Schmidt. But the kinds of regulation that stops the Chinese from stealing the AGI secrets is good regulation. And then anything else that slows the power of our technology is bad regulation.

Mohamed Abdalla: Yes. But for example, when Donald Trump banned the H1B visa. Or not banned, he put a limit or a pause. I’m not sure the exact thing that happened.

Lucas Perry: Yes. He’s made it harder for international students to be here and to do work here.

Mohamed Abdalla: Yes, exactly. That is the type of regulation that you would have if you believed AI was a threat, like we are racing the Chinese. If you believed that, you would be for that sort of regulation because you don’t want these companies training foreign nationals and the development of this technology. Yet this is not what these companies are going for. They are not agreeing with the legislation or the regulation that limits the amount of foreign workers they can bring in.

Lucas Perry: Yeah. Because they just want all the talent.

Mohamed Abdalla: Exactly. But if they believe that this was a matter of national security, would they not support this? You can’t make the national security arguments when they say, “Don’t regulate us, because we need to develop as much as we can, as fast as we can.” While also pushing against regulation that if this was truly dangerous, if we did truly need to leave you unregulated internally, we should limit who can work for you in the same way that we do it for rocketry. Who can work on rockets, who can work at NASA? They have to be U.S. citizens.

Lucas Perry: Why is that contradictory?

Mohamed Abdalla: Because they’re saying, “Don’t regulate us in terms of what we can work on,” but they’re saying also, “Do not regulate us in terms of who can work for us.” If what you’re working on is a matter of national security and you care about national security, then by definition, you want to limit who can work on it. If you want anyone, or you say there should be no limit on who can work for us, then you are basically admitting that this is not a matter of national security, or profits over everything else. Google, Facebook, Microsoft, when possible legislation comes up, the Eric Schmidt argument gets played. And it’s like, “If you legislate us, if you regulate us, you are slowing down our progress towards this technology.”

But if any sort of regulation against the development of tech will slow down the arrival of AGI, which we assume that the Department of Defense cares is important. Then what you’re saying is that these companies are essentially striving towards that should they not be protected from foreign workers infiltrating. So this is where the companies hold two opposing viewpoints. Where depending on who they’re talking to, no don’t regulate us because we’re working towards AGI and you don’t want to stop us. But at the same time, don’t regulate immigration because we need these workers. But if what you were working on is sensitive, then you shouldn’t even be able to take these workers.

Lucas Perry: Because it would be a national security risk.

Mohamed Abdalla: Exactly. When a lot of your researchers come from another country and they’re likely to go back to that country or at least have friends, have conversations with other countries.

Lucas Perry: Or just be an agent.

Mohamed Abdalla: Yeah, exactly. So if this is actually your worry that this regulation will slow down the development of AGI, how can you at the same time be trying to hire foreign nationals?

Lucas Perry: All right. So let’s do some really rapid fire here.

Mohamed Abdalla: Okay.

Lucas Perry: Is there anything else that you wanted to add to this argument about incentives and companies actually just being good? And we are walking through this Eric Schmidt argument.

Mohamed Abdalla: Yeah. So the thing I want to highlight is that this is a system level problem. So it’s not a problem with any specific company, despite some being in the news more than others. It’s also not a problem with any specific researchers or institutions. This is a systemic issue. And since it’s a high level problem, the solution needs to be at a high level as well. Whether it’s at the institutional level or national level, some sort of legislation, it’s not something that research individually can solve.

Lucas Perry: Okay. So let’s just blast through the action items here then for solving this problem. You argue that everyone should post their funding information online in historic informations. This increases transparency on conflicts of interest. But as we discussed earlier, they actually just need to be removed. You also argue that universities should publish documents highlighting their position on big tech funding for researchers.

Mohamed Abdalla: Yeah. Basically I want them to critically consider the risks associated with accepting such funding. I don’t think that it’s a consideration that most people are taking seriously. And if they are forced to publicly establish a position, they’ll have to defend it. And that will I believe lead to better results.

Lucas Perry: Okay. And then you argue that more discussion on the future of AI ethics and the role of industry in this space is needed. Can’t argue with that. That computer science should explore how to actively court antagonistic thinkers.

Mohamed Abdalla: Yeah. I think there’s a lot of stuff that people don’t say because it’s either not in the zeitgeist, or it’s weird, or seems an attack on a lot of researchers.

Lucas Perry: Stigmatized.

Mohamed Abdalla: Yeah, exactly. So instead of trying to find people who simply list on their CV that they care about AI ethics or AI fairness, you should find who will to disagree with you. If you’re able to beat points to disagree with, it doesn’t matter if you don’t agree with their viewpoint.

Lucas Perry: Yeah. I mean usually, the people that are saying the most disruptive things are the most avant garde and are sometimes bringing in the revolution that we need. You also encourage academia to consider the splintering of AI ethics into different department from computer science. This would be analogous to how bioethics is separated from medicine and biology. We talked about this already as sequestration. Are there any other ways that you think that the field of bioethics can help inform the development of AI ethics on academic integrity?

Mohamed Abdalla: If I’m being honest, I’m not an expert on bioethics or the history of the field of bioethics. I only know it in relation to how it has dealt with the tobacco industry. But I think largely, more historical knowledge needs to be used by people deciding what we as computer scientists do. There’s a lot of lessons learned by other disciplines that we’re not using. And they’ve basically been in a mirror situation. So we should be using this knowledge. So I don’t have an answer, but I think that there’s more to learn.

Lucas Perry: Have you received any criticism from academics in response to your research, following the publication that you want to discuss or address?

Mohamed Abdalla: In this specific publication, no. But it may be because of the COVID pandemic. I have raised these points previously. And I have received some pushback, but not for this specific piece. Although this piece was covered in WIRED and there are some criticisms of the piece in the WIRED article, but I kind of raised them up in this talk.

Lucas Perry: All right. So as we wrap up here, do you have anything else that you’d like to just wrap up on? Any final thoughts for listeners?

Mohamed Abdalla: I just want to stress if they’ve made it this way without hating me, this work is not meant to call into question the integrity of researchers, whether they’re in academia or in industry. And that I think these are critical conversations to be had now. It may be too late for the initial round of AI legislation. But for the future, good. And for longer-term problems, I think it’s more important.

Lucas Perry: Yeah, there’s some meme I think going around, like one of the major problems in the world is good people who are running the software of bad ideas on their brain. And I think similar to that is all of the good people who are caught up in bad incentives. So this is just sort of amplifying your non-critical or judgmental role, but that the universality of the human condition is that we all get caught up in these systemic negative incentive structures that lead to behavior that is harmful for the whole.

So thank you so much for coming on. I really learned a lot in this conversation. I really appreciate that you wrote this article. I think it’s important, and I’m glad that we have this thinking early and we can hopefully try and do something to make the transformation of big tech into something more positive happen faster and more rapidly than it has historically with other industries. So if people want to follow you, or look into more of your work, or get in contact with you, where the best places to do that?

Mohamed Abdalla: I’m not on any social media. So email is the best way to contact me. It’s on my website. If you search my name and add the University of Toronto and the end of it, I should be near the top. It’s cs.toronto.edu/msa. And that’s where all my work is also posted.

Lucas Perry: All right. Thanks so much, Mohamed.

Mohamed Abdalla: Thank you so much.

Maria Arpa on the Power of Nonviolent Communication

November 2, 2020/in Featured, FLI Podcasts, FLI podcasts & transcripts /by Lucas Perry

Future of Life Institute · Maria Arpa on the Power of Nonviolent Communication

Topics discussed in this episode include:

What nonviolent communication (NVC) consists of
How NVC is different from normal discourse
How NVC is composed of observations, feelings, needs, and requests
NVC for systemic change
Foundational assumptions in NVC
An NVC exercise

Timestamps:

0:00 Intro

2:50 What is nonviolent communication?

4:05 How is NVC different from normal discourse?

18:40 NVC’s four components: observations, feelings, needs, and requests

34:50 NVC for systemic change

54:20 The foundational assumptions of NVC

58:00 An exercise in NVC

Citation:

The Center for Nonviolent Communication’s website

We hope that you will continue to join in the conversations by following us or subscribing to our podcasts on Youtube, Spotify, SoundCloud, iTunes, Google Play, Stitcher, iHeartRadio, or your preferred podcast site/application. You can find all the AI Alignment Podcasts here.

You can listen to the podcast above or read the transcript below.

Lucas Perry: Welcome to the Future of Life Institute Podcast. I’m Lucas Perry. Today’s conversation is with Maria Arpa on nonviolent communication, which will be referred to as NVC for short throughout the episode. This podcast continues to explore the theme of wisdom in relation to the growing power of our technology and our efforts to mitigate existential risk, which was covered in our last episode with Stephen Batchelor. Maria and I discuss what nonviolent communication consists of, its four components of observations, feelings, needs, and requests, we discuss the efficacy of NVC, its core assumptions, Maria’s experience using NVC in the British prison system, and we also do an on the spot NVC exercise towards the end of the episode.

I find nonviolent communication to be a powerful upgrade in relating, resolving conflict, and addressing the needs and grievances we find in ourselves and others. It cuts through many of the bugs of normal human discourse around conflict and makes communication far more wholesome and collaborative than it otherwise might be. I honestly view it as a quite a powerful and essential skill or way of being that has had quite a transformative impact on my own life, and may very well do the same for others. It’s a paradigm shift in communication and human relating that I would argue is an essential part of the project of waking up and growing up.

Maria joined the Center of Nonviolent Communication as the Executive Director in November 2019. She was introduced to NVC by UK Trainer Daren de Witt, who invited her to see Marshall Rosenberg, the founder of NVC, speak in London, a moment that changed her life. She was inspired to attend one of the Special Sessions on Social Change in Switzerland in 2005 and then invited Marshall to Malta where she organised a conference between concentration camp survivors and the multi-national corporation that had bought the site formerly used as a place of torture. Since then she has worked with marginalised, hard to reach individuals and communities and taken her work into prisons, schools, neighbourhoods and workplaces.

And with that, let’s get into our conversation with Maria Arpa.

I really appreciate you coming on, and I’m excited to learn more about NVC. A lot of people in my community, myself included, find it to be quite a powerful paradigm, so I feel excited and grateful that you’re here. So I think we can kick things off here with just a pretty simple question. What is nonviolent communication?

Maria Arpa: Thank you. Yes, and it’s really good to be here. So nonviolent communication is a way or an approach or a system for putting words to living a nonviolent life. So if you’ve chosen nonviolence as a philosophy or as a way of life, then what Marshall Rosenberg proposed in creating nonviolent communication in the 1960s is a way of communicating interpersonally based on the idea that we need to connect as human beings first.

So NVC, which is short for nonviolent communication, is both a spiritual practice and a set of concrete skills. And it’s based on the idea of listening deeply to ourself and to others, in order to establish what are the real needs that we’re trying to meet? And from the understanding of the needs that come through empathic listening, then we can begin to build strategies.

Lucas Perry: Alright, so can you juxtapose what NVC does, compared to what usually happens in normal discourse and conflict resolution between people?

Maria Arpa: Yeah, lovely. Thank you. I like that question. In society, we have been brought up, and I would go so far as to say indoctrinated, with taking adversarial positions. And we do that because we think about things like the legal profession, academia, science, the military, and all of those disciplines, which are highly prized in society, but they all use a debate model of discourse.

And that’s wonderful if you want to prove a theory or expand the knowledge between people, but if we’re actually trying to just build a relationship in order that we can coexist, do things, envision, create something new in society, debate just doesn’t work. So at the very micro level, in the family system, what I experienced with couples and families is a sort of table tennis match. And while you’re speaking, I am preparing my counter argument. I would say that we’ve been indoctrinated with a debate model of conversation that even extends into our entertainment, because in my understanding, if you go to scriptwriting school, they will tell you that to make a Hollywood blockbuster, you need to leave a conflict in every scene.

Maria Arpa: So that has become the way in which we communicate with each other, which is played out in our legal systems, in the justice system, it’s played out in education. When I position that against nonviolent communication, nonviolent communication says we need to build a relationship first. What is the relationship between us? So if we even take this podcast, Lucas, you and I had built a relationship. We didn’t just make an appointment, come on, and do this, we got to know each other a bit. Is that a helpful answer?

Lucas Perry: That’s a good question. Yeah, I think that is a helpful answer. I’m also curious if you could expand a bit more on what the actual communicative features of this debate style of conversation focus on. So where NVC focuses on talking about needs and feelings, feelings being evidence of needs, and also making observations, observations as being opposed to judgments, what is it that the normal debate style or adversarial kind of conversations that we’re having, what is the structure and content of that kind of conversation that is not needs, feelings, and observations?

To me, it often seems like it deals with much more constructed and synthetic concepts where, NVC you boil down concepts to things which are very simple and basic and core, whereas the adversarial relationship deals with more complex constructive concepts like respect, abandonment, and is less simple in a sense. So do you have anything else you’d add here on to what actually constitutes the adversarial kind of conversation?

Maria Arpa: Yeah, definitely. So in an adversarial conversation now we have two levels. And the first level, because I’m really thinking about how we’ve been programmed, in the first level, we’re battling backwards and forwards and what we’re out to do is win the argument. So what I want is for my argument to prevail over yours. And in families, it’s generally who’s the worst off person, who’s the most tired, who’s the most unresourced, who does most of the work, who’s earning most of the money, that level. It’s a competition. It’s a competitive conversation.

At its worst ends of the spectrum, and when we think about school and education, it’s a debate, which also includes the prospects of enforcement, which is punitive. So we could simply be competing to win an argument or we could be actually out to provide evidence of the other person’s wrongness in order to punish them, maybe not even physically punish them, but punish them with the withdrawal of our love.

Lucas Perry: Right, so if we’re not sufficiently mindful, there is this kind of default way that we have been conditioned into communicating, where I noticed a sense of strong ego identification. It’s a bit selfish. It’s not collaborative in any sense. As you said, it’s kind of like if I can make and give points which are sufficiently strong, then the person will see that I’m the most tired, I’m the most overworked, I have contributed the most, and then my needs will be satisfied. And then my needs being satisfied is more implicit, and it’s never made explicit. And so NVC is more about making the needs and feelings explicit and the core and cutting out the argument.

Maria Arpa: Yes, I really agree with that. The problem with that debate model, that adversarial model, is that I might get my needs met, I might be able to bulldoze or bully my way, or I might be able to play the victim and get my needs met, but usually I have induced the other person to respond to me out of fear, guilt, or shame, not out of love. Someone, somewhere will pay for that. It’s that whole thing, you may have won the battle, but you haven’t won the war. While at a very micro level, day by day, I can scrape by and score points and win this and get my need met, in the long term, I’m actually still feeding the insecurity that nothing can come without struggle.

Lucas Perry: That’s a really good point.

Maria Arpa: And when you say it cuts out the argument, the argument bit which is not necessary, I’m saying that we will go into the dialogue with the idea that once we’ve established the needs, then we can actually build agreements. Now, that’s not to say that I’m going to get everything I want, and you’re going to get everything I want. But there’s a beauty in the negotiation that comes from the desire to contribute.

Lucas Perry: Yeah, so I found what you said to be quite beautiful and powerful around, you may gain short term benefits from participating in what one might call violent communication, or this kind of adversarial debate format, you might get your needs met but there’s a sense in which you’re conditioning in yourself and others this toxic communicative framework, where first, you’re more deeply instantiating this duality between you and other people, which you’re reifying by participating in this kind of conversation. And that’s leading to, I think, a belief in an unwanted world, a world that you don’t want to live in, a world where my needs will only be met, if I’ll take on this unhappy, defensive, self-centered stance in communication. And so that’s a contradicted belief. If I don’t do the debate format, then I will not be safe, or I will not get my needs met or I will be unhappy. You don’t want that to be true, and so it doesn’t have to be true if you pivot into NVC.

Maria Arpa: Yes, and got two things really to say about that. One is that we are often counting the cost of something and one of the things that we very rarely factor in is the emotional cost to getting our needs met, the emotional cost to taking on the task. And sometimes when I work with people, we find that the price is too high. I may be getting my fame and fortune or whatever it is I’m after, but actually, the emotional price to my soul is just too high.

And most of us have been taught not to imagine that as being of any importance. In fact, most of us have been taught, rather like scientific experiments, that when I’m looking at a situation, even when I’m not taking a self-centered point of view, even when I’m looking at it and wanting to be generous and benevolent, I look at a situation and I fracture myself into that situation, as if I don’t matter, or I don’t count, and the truth is everybody matters, and everybody counts.

So that’s one thing, and then when I heard you talking about this idea of adopting the adversarial approach to life and how that will feed the system, the best example I have of that is in the prison that I’ve been working in for the last four years. Prisons are pretty mean places, and so most people come into prison in the belief that they need to develop a very strong set of armor in order to defend themselves, and sometimes attack is the best form of defense, in order to survive this new world. And what I’ve been able to demonstrate with the guys that I’ve been training is that actually, if you throw away the armor, and if you actually come to be able to work out what your needs are, and be able to help other people find their needs, actually, it becomes a different place.

And the proof of that has been 25 men that I have trained to do what I do within the prison, working with other prisoners across the prison, and turning the prison into one of the safest jails in the UK.

Lucas Perry: Wow.

Maria Arpa: So the first thing that happens is I deliver training, and it’s intensive, it’s grueling, there’s a lot of work to do, and what happens to the guys as they’re doing the training. Because see, I believe that the people that cause the problems are the ones who have the answers to the problems. We should go to the people that cause the problems and say, “How do we not end up here again?” So as they do this training and they realize the potential to make their prison sentence go better, what a nice thing to be able to do, and then they realize that actually, I can’t do this for anyone else until I’ve done it for myself. So that’s part of the transformative bit where they go, “Well hang on, I’ve got so much conflict, or I’ve got so much inner conflict or dislike of myself, or whatever chaos going on inside, I can’t possibly do this for anyone else. So, right, now we can begin because now we have to begin as a team.”

And so what’s been remarkable for me is 25 men who, on the outside of prison would never ordinarily come into contact with each other, from completely different walks of life, different areas, different ages, different crimes, we’ve got everything from what in America you call homicide, through sexual offenses, to fraud, and that type of thing.

So these 25 men have been through a process to be able to work together as a team and to be able to understand each other, and they are completely blown away by the idea that they have a process so when they feel that the system that they’re using, because they get overrun with casework, and that what I’ve taught them to do is you have to put yourselves first. So when that happens, you actually need to stop everything and come back as a team, and work out what are the petty conflicts that are arising, and use this nonviolent communication process to come back to center. So for me, very much, there’s a huge difference in being able to live this way, and no other greater place have I proved it than in a place called prison.

Lucas Perry: Yeah, exactly. I bet that’s quite liberating for them, and I bet that there’s a deeper sense of security that is unavailable when you’re taking the adversarial, aggressive stance.

Maria Arpa: There’s a deeper sense of security. There’s a huge amount of gratification for somebody who has literally been thrown away by society and told that they’re worthless and have no place, and I don’t want to get into the crime or whether they did it or didn’t do it, but they’ve been thrown away by society, to actually find that they can be in service of others, and they can actually start to love themselves. So there’s a really huge gratification in being able to do that and not do it in a self sacrificing way, to be able to do it in a way that enriches the other person and enriches themselves. That for me is monumental.

And the second thing is that, in places like prisons and family, I often compare schools to prisons, you can be overwhelmed by the power of enforcement and the misuse of authority. So often, one person in a position of power may dish out the rules one way today in a different way tomorrow, may treat one person differently to another. And so what we’ve been able to establish is that if the guys sit in circle and invite officers to those circles, they can clear up some of the things that just create unnecessary conflict.

Now, obviously, in prison, some topics are non-negotiable, and we don’t go there, right? And if you don’t think you should be imprisoned, then you need to take the appropriate steps through your legal advisors. But if it’s a case of the laundry’s messed up every week and there are fights starting over it, if it’s a case of exactly who’s collecting the slips for the lunches, and that’s creating, or there’s an argument when people are queuing up for their food, these things can be sat down and had out and people could talk about their different experiences, and we can clear up those gray areas by the guys coming up with a policy. And they do this using needs.

Lucas Perry: All right, it’s encouraging to see it as effective and liberating in the case of the prison system. I’m interested in talking with you more about how this might apply to, for example the effective altruism movement, or altruism in general, and also working on really big problems in the world that involve both industry and government, who at times function and act rather impersonally. Where, for example, the collective incentives of some corporation are to maximize some bottom line, and so it’s unclear how one could NVC with something where the aggregate of all the behavior is something rather impersonal.

So right before we get to that, I just want to more concretely lay out the four components of NVC practice, just so listeners have a better sense of what it actually consists of. So could you take us through how NVC consists of observations, feelings, needs, and requests?

Maria Arpa: Yeah, I’d love to, yes, thank you. So usually, in our heads, if we’re indoctrinated in that adversarial and we don’t even know it, whatever, usually what’s happening is when we see something, we’re busy judging it, evaluating it, deciding whether we like it or don’t like it, imposing our diagnosis, and generally having an opinion about it, good or bad. And then, of course we live in a world now where people can take to social media and destroy other people if they choose. So we can actually just act out of what we think we’re seeing.

In nonviolent communication, what we do is we try to get to what we call the observation without the evaluation. So what is it that’s actually happening? And that is really trying to separate the reality from the perception. A really good example of that was I saw a demonstration once, a woman came down from the audience, and she wanted to talk about how angry she was with her flatmate. And she gave out this whole story and the trainer would say, “Well, we need to get to the observation, we need to get to the observation.” And out of this whole mass, the only two observations that she could come up with were that her flatmate occasionally leaves a dirty plate and a dirty mug and dirty cutlery in the sink without washing it up. And on occasion, her flatmate when she leaves the flat or the apartment, allows the door to slam and make a very loud noise behind her. And those are the only two observations that she could come up with that were actually really happening. Everything else was what she made up around it.

Lucas Perry: Yeah. So there’s this sense in which we’re telling ourselves stories on top of what may be just kind of simple, brute facts about the world. And then she’s just suffering so much over the stories that she’s telling herself.

Maria Arpa: Yeah, so the story attached to someone leaving the dirty plates in the sink is, she’s doing it on purpose, doing it to get at me. Those are the sorts of things we might tell ourselves. Or we might be the opposite and say, “She’s just so selfish.” And I really love what you just said. Of course, the person that I’m causing the most grief and pain to is myself. I’m cutting myself off from my own channel of love.

So that’s how we get to the observation, and it’s a really important part of the NVC process, because it helps us to identify that which is what we are telling ourselves and that which is actually happening in front of us. And the way that I could tell if it’s an observation or an evaluation, is I could record it on a video camera and show it to you and you would see the same thing.

Lucas Perry: Yeah, that makes sense. You’re trying to describe things as more factual without the story elements. This happened, then this happened, rather than, “My asshole roommate decided to leave her dirty shit everywhere because she just doesn’t care and she sucks.”

Maria Arpa: Yeah, so that’s the first step. And then what we’re looking to do is check in with ourselves on how do we feel. This is a really important step, because in nonviolent communication, what we propose is that our feelings are the red warning light on the dashboard of the car that tells you to pull over and look under the hood, you would say, we would say bonnet, and to check what else is going on. So feelings are a gateway. They’re our doorway. So they are our barometer.

So it’s really important to develop a really good vocabulary around feelings. And it’s really important to get to the feeling itself, whether it’s sadness, or anger, or upset, or despair or grief, or joy, or happiness, it’s really important to develop a vocabulary of feelings because if I ask someone how they feel, and I get, “I feel like,” no feeling is coming after the word “like.” I feel like jumping off a cliff. I feel like just going to bed and never getting up again. I feel like running away. That’s not a feeling. People can use those kinds of metaphors for us to try and guess the feelings, but actually what I want is the feeling.

Lucas Perry: Right, those are overly constructed. They need to be deconstructed into core feelings, which is a skill in itself that one learns. So you could say for example, “I feel abandoned,” but saying, “I feel abandoned,” needs to be deconstructed. Being abandoned is being afraid and feeling lonely.

Maria Arpa: So if we say, “I feel abandoned,” and I’m particularly referring to the ED at the end, or, “I feel disappointed,” rather than disappointment, then what I’m doing is I’m actually, by the backdoor, I’m accusing somebody of doing it to me.

Lucas Perry: Yeah, that’s right. It’s a belief about the world actually, that other people have abandoned you or are capable of abandoning you.

Maria Arpa: Yeah, exactly. So there’s a skill in the language. However, if you go to the cnvc.org website, we have a free feelings list and a needs list that’s downloadable for anybody that wants to go and get it. And that helps you to really get closer to the language.

Lucas Perry: Okay, so I do have a objection here that I don’t want to spend too much time on, but I’m curious what your reaction is. So I think that what you would call the “abandoned” word is a faux feeling, is that the right word you guys use? There’s a sense in which it needs to be further deconstructed, which you mentioned, because it’s a belief about the world, yet is there not also some reality that we need to respect and engage with, where abusers or toxic people may actually be doing the kind of thing which seems like a belief in the world.

Maria Arpa: That’s where I would get back to the observation. Because those things do happen. I work a lot in domestic violence. I understand this. And there are two things, and one that we’ll go on to later. There’s getting back to the observation, because if I heard you say, “I feel abandoned,” what I would want to do is go back to figure out what’s the observation that brings you to that sense. Because actually, if you’re telling yourself you’ve been abandoned, or if somebody has abandoned you, and we can see that in an observation, then I’m guessing you’re feeling a huge amount of misery, grief, and despair, or loneliness, those would be the feelings.

And then later on, when we get to it, I’ll talk about the use of protective force. Because it isn’t all happy, dippy, and let’s all get on our hippie barge and have a great life. Without putting too fine a point in it, shit has happened, shit is happening, and shit’s always going to happen. And that’s the way of the world. What I’m talking about is how we respond to it.

Lucas Perry: Yeah, that’s right. You can try and NVC Hitler, and when you realize it’s not going to work, that’s when you mobilize your armies.

Maria Arpa: That’s a very interesting thing, you could try to work with Hitler, because actually, I don’t know if you’ve seen, I have a copy of it somewhere, Gandhi actually wrote a letter to Hitler.

Lucas Perry: Yeah, it didn’t work.

Maria Arpa: It didn’t work. And actually, if you look at the letter, it’s a shame because there was nothing in there for me that I recognized as NVC that may have generated at least a response.

Lucas Perry: Alright, so we have feelings, and we want to be sure to deconstruct them into simple feelings, which is a skill that one develops. And the thing here that you said that the feelings are like the warning to check the engine of the car, which is a metaphor to say feelings are a signal giving you information about needs being unmet, or at least even the impression or ignorance or delusion that you think your needs are not being met. Whether it’s actually your needs not being met, or a kind of ignorance to your needs being met, either way, they are a signal of that kind of perception.

Maria Arpa: Yes, absolutely. They’re a signal for something. And so when we talk about feelings, what I’m trying to do is capture the real emotion here and name it.

Lucas Perry: And so then there’s a sense that when you communicate needs to other people, they cannot be argued with and they’re also universally shared. So you can recognize the unmet needs of another person as a reflection or a copy and paste of your own needs.

Maria Arpa: So, this is a really interesting part of the conversation when we get to needs because that sits in something called needs-based theory. And Marshall Rosenberg does not have the monopoly on needs-based theory. I mean, most people will have heard of Maslow’s hierarchy of needs. There’s a Chilean economist called Manfred Max Neef, who boiled all the needs down to just nine and said that everything else is just ways or satisfiers, to try and meet those needs.

For me, needs-based theory is an art, not a science. And so again, you could go on the cnvc.org website, and you can pull off a list of needs, and you’ll recognize them. Now, when I say it’s an art not a science, on there could be, say, the need for order and structure. Okay, so let’s say I have a need for order and structure.

Lucas Perry: That seems like it needs to be deconstructed.

Maria Arpa: Yes. So I would then say, “Maybe that is a strategy to get to a deeper need of inner peace, but at the moment, that seems to be the very present need for me. I come downstairs, my desk looks like a bomb’s hit it, I’ve got calls to get on, and I just don’t feel like I can get my day started until I’ve created a sense of order around myself.”

It is a simple need in that moment, but the idea is that when we look at the fundamental needs like air, and movement, and shelter, and nutrition, and water, those are universal. I mean, I don’t think anyone could disagree with that. And then we get into more spiritual needs and social needs. Things like discovery and creativity and respect, what a big word, respect is. And the way I like to look at it is, you see, all the arguments we ever have can never be over needs, because I can recognize that need in myself, as well as in others, but they’re over the strategies that we’re trying to use to meet the need that may be at a cost to someone else’s needs, or to my own deeper needs.

So a really good example is if you take our need for air. There’s only one strategy to meet our need for air and that’s to breathe. How many arguments do people have over the need for air and the strategy of breathing?

Lucas Perry: Zero.

Maria Arpa: Right. Now, let’s take a really big word that gets bandied about everywhere, respect. How many arguments do we have over the strategies we’re using to meet a need for respect?

Lucas Perry: A million. And another million.

Maria Arpa: Exactly, exactly. And so the arguments are only ever about strategy. And once you’ve understood it, and practiced it, and embodied that, and you can see the world through that lens, everything changes. And that’s why I can do what I do.

Lucas Perry: Yeah, well, so let’s stick with air. So some people have a strategy for meeting their needs by polluting the air with things. So there’s some strategy to meet needs where the air gets worse, and everyone has this more basic need to breathe clean air, and so the government has to step in and make regulations so that some more basic need is protected. But then so there’s this sense that strategies may harm other people’s needs, and there’s a sense in which sometimes the strategies are incompatible. But there’s this assumption that I think is brought in that the world is sufficiently abundant to meet everyone’s needs and that’s a way, I think, of subverting or getting around this contradictory strategy problem where it would suggest that, okay, oil companies, we can meet your needs some other way, as long as you change your strategy and we’ll help you do that so that we have clean air. Does this make sense?

Maria Arpa: It makes total sense, and I’ve got two sort of parallel answers, maybe even three. So the first one, we’ve got where we have where there are people in the world who don’t mind, or maybe they do mind secretly, doesn’t matter, where there are people in the world who will pollute the air for profit. And we’ve reached that point because we have been using an adversarial system with each other that means that as long as I can turn someone into the enemy, I can justify doing whatever I want. So we create bad people and good people.

So in this adversarial system, one of the things we can do is justify what we’re doing by holding up other people as being in the way. So we’ve created that system and actually what we’re finding, is that the system is failing. I don’t know, I don’t want to predict things, I’m not an economist or a politician, but it seems to me that the system is failing rapidly, more and more. More harm is being visited on the planet than is necessary and lots of people are waking up to that.

So now we’re hitting some kind of tipping point where in giving people things like the internet and all this stuff to self soothe them, actually, a lot of people got educated and started to ask better quality questions about the world they’re living in. And I think there’s a bit of an age difference between us, the wrong way for my end, but people of your generation are definitely asking better quality questions, and they’re less willing to be fobbed off.

So now we’ve got to figure out, how do we change things? And while I understand that from time to time, we need to go out and we need to actually put our foot down and make a protest and make a stand and say, “We’re not putting up with this,” and use protective force, and nonviolent resistance, and civil disobedience, while we need to do those things, we will never change things if we’re only operating at the incident level. If you try to do everything and fix it at the incident level without somebody working long-term on the system… People need to organize, and work out how people like you could get into positions of power.

I mean, I did a lovely piece of work with a Somalian community many years ago, and they’d arrived in the UK as refugees, and when they first arrived, they thought they were only going to be around for a few years and that the war would sort itself out and they’d all go back home. So they kept to themselves and they were very excluded and left out of society, and some of the sons were getting into trouble with the police because they hadn’t really worked out how to live in this society, and after they realized that actually they weren’t going back, “We’re here, this is our home,” what they’ve realized is they needed to start organizing. They needed to become teachers and doctors and lawyers and actually start to help their own community in that way. And I found that very moving and very empowering, and I loved doing the work with them. And the work we were doing was literally around the mothers and the sons. So that’s changing things at a system level.

Lucas Perry: Okay, so the final point here is about making requests. And I think this is a good way to pivot into talking about, you can’t make requests to make systemic change, because the power structures are sufficiently embedded in the incentives are structured such that, “Hey, excuse me, please stop having all that power and money, my needs are not being met,” isn’t going to work.

So let’s talk about the efficacy of NVC and how it’s effectively used. So I think it’s quite obvious how NVC is excellent for personal relationships, where there’s enough emotional investment in one another, and authentic care and community where everyone’s clearly invested in actually trying to NVC if they can. Then the question becomes, for bigger problems in the world like existential risk, if NVC can be effective in social movements, or with policy makers, or with politicians, or with industry or other powerful actors whose incentives aren’t aligned with our needs and who function impersonally. What is your reaction to NVC as applied to systemic problems and impersonal, large, powerful actors who have historically never cared about individual needs?

Maria Arpa: That’s a really interesting question because in my experience of the world, nothing happens without some kind of relationship between people. I mean, you can talk about powerful actors that don’t care, but bring me a powerful actor that doesn’t care and let me have a conversation with them. So for me, I agree that there’s a place for NVC in a group of people who care. There’s also a place for NVC in making the conversation irresistible, finding that place in somebody, because if we work on the basis that there are human beings in the world that have no self-love, or no love at all, if we work on the basis that there are human beings that walk the planet that are just all selfish and dangerous and nothing else, then of course, we’re doomed.

But I don’t believe that, you see, I believe that we are all selfish, greedy, kind, and considerate. And I know this from doing this work in prisons, that often what’s happened is the kind and considerate has just gone to sleep, or it’s paralyzed, or it’s frozen, but it is there to be woken up. And that’s the power of this work, when the person has sufficiently embodied it, has practiced this, and really understands that this involves seeing the world through a different lens. That actually, my role in the work I do, and I work in the front line of some of the worst things that go on in society, my role is to wake up the part of a person that is kind and considerate, and nurture it and bring it to life and grow it and work with it. And that doesn’t happen in one conversation. I don’t do that because I want something, I do that because I generally care about how that person is destroying themselves.

I can give you an example of somebody I met in prison who had been imprisoned for being part of a very, very violent gang, been in violent gangs for most of his life, done a lot of time in prison, and the judge called him evil, and greedy, or whatever. And he came on one of my trainings in around 2013. And he kept coming to talk to me in the breaks, it was like he really wanted some kind of connection or some affirmation or something, and he said, “I did a restorative justice training last month, and I really have to think about the harm I’ve done to my victims.” And I said, “You also have to think about the harm you’ve done to yourself.” And that was the first moment of engagement. And actually, now this man will be out of prison in I think 2022. He has put himself through a therapeutic prison for six years. I’ve never seen a life change to such an extent or such a degree. We’re thinking about employing him when he comes out of prison.

And that’s the thing is, it’s how do you engage a person to look at themselves and to look at how they may be destroying themselves in the pursuit of whatever it is they think they need. So, bring me somebody who is a powerful actor, who doesn’t care about anyone else, and we’ll open the conversation. That’s how I see it. The reason that I can do this and I can have these conversations is I don’t have an agenda for another human being. I simply want to understand what is going on, what the motivations are, what the needs are, and work out with that person, is that strategy actually working for you? And if you’re meeting your need for power or growth or structure or whatever, is it costing you in some other needs that’s actually killing you slowly?

Lucas Perry: Yeah, I mean, I think that, for example, at risk of becoming too esoteric, non-dual wisdom traditions, I think would see this kind of violent satisfaction of one own’s needs is also a form of, first of all self harm, because your needs extend beyond what the conceptual egoistic mind would expect to be your needs. I’m thinking of someone who owns a cigarette company, and who’s selling them and knows that he’s basically helping to lie about the science of it, and also promoting that kind of harm. There’s a sense in which it’s spiritually corrupting, and leads to the violation of other needs that you have when you engage in the satisfaction of your needs through these toxic methodologies.

Maria Arpa: Absolutely, and it’s a kind of addiction, it’s a kind of habit, or obsession. One of the things that I’m really interested in is, at the end of the day, when we get to the request part of NVC, the real request is change. Whatever it is I’m asking for, whether I’m making a request of myself or the person in front of me, I’m requesting change. And change isn’t easy for most people. People need to go through a change process. And so it’s not just about the use of NVC as an isolated tool that is going to change the world, it is about contextualizing the use of NVC within other structures and systems like change processes, understanding group dynamics, understanding psychology, and all of those things, and then it has its place.

Lucas Perry: Yeah, it has a place amongst many other tools in which it becomes quite effective, I imagine. I suppose my only reaction here then is, you have this perspective, like, “Bring me someone in one of these positions of power, or who has sufficient leverage on something that looks to be extremely impersonal, and let’s have a conversation,” those conversations don’t happen. And no one could bring you that person really, and the person probably wouldn’t want to even talk to you, or anyone really who they know is coming at them from some kind of paradigm like this.

Maria Arpa: Oh, I don’t know about that. I mean, in the work I do, it’s a very small world, I’m not trying to affect global change. I would love to, but I’m not. But in the prison work I was doing, we managed to get the prison’s minister to come and see the work, and I managed to then have a meeting with him. And I managed to convince him on one or two things that had an effect at the time. So I don’t know that these things don’t happen. I think it’s about the courage and determination of the people to get those meetings, not coming from having an agenda for that person, but coming from really wanting to understand what the thinking is.

Again, in my experience, having been around the block a few times, the people making policies would be absolutely horrified if they saw how those policies are being delivered on the ground. There’s a huge gap between people sitting somewhere making a policy, and then how it gets translated down hierarchical systems, and then how it gets delivered. I like to think that policy makers aren’t sitting around the table going, “How can we make life worse for everybody, because we hate everybody.” Policy makers are sometimes very misguided and detached and unable to connect, but I don’t think policy makers are sitting there going, “We hate everybody, let’s just make life difficult.” They really genuinely believe they’re solving problems. But the issue with solving problems is that we’re addicted to strategy before understanding the needs.

Lucas Perry: We’re addicted to strategy before understanding needs.

Maria Arpa: Yeah. Our whole mentality is, “Problem? Fix it.”

Lucas Perry: So I mean, the idea here then, is that the recognition of needs, as well as bringing in some other assumptions that we can talk about shortly, and relaxing this adversarial communicative paradigm into a needs-based one where you take people on good faith and you recognize the universality of human needs, and there’s this authentic care and empathy which is born of, not something which you’re fabricating, but something which participating in actually serves some kind of need that you actually already have to have authentic human connection, or maybe that boils down to love. And so NVC can be an expression of love in which NVC becomes something spiritual. And then that this kind of process is what leads to a reexamination of strategy.

Maria Arpa: Yeah, so the idea is that because we have a problem, fix it mentality, we are skipping over the main part which is to sit with the pain of not knowing. So what we do is we jump to strategy, whether that’s in our daily lives, “I feel bad, I’ll go and get a haircut or buy myself a new wardrobe, or I’ve got a problem, and it’s going to create a big PR problem, so I’m going to do this,” and what we’re missing is the richness of understanding that when you do that, you’re acting out of fear, you’re jumping, because you’ve got triggered or stimulated in some way, and you’re acting out of fear to prevent yourself from the feelings that you don’t believe you’re going to be able to cope with.

And what I’m saying is that we understand, we get to the observation, we identify, is this an issue? Is it not an issue? And then we go within, in a group, and we sit with the pain, the mourning, of the mistakes we’ve made, or the problem we haven’t solved, or the world we’ve created, whatever it is, and it’s in sitting together with that, and being willing to say, “I don’t know what the answer is right now, or today. Maybe I just need to breathe,” in being able to do that, we reach our creativity.

So we’re coming out of a place of absolute creativity and love, not jumping out of fear. And there’s a tremendous difference in operating in the world in this way. But it requires us to be willing to be vulnerable, and I think that’s what I think you’re talking about when you talk about people being detached. They’re so far away from their vulnerability, and when people are so far away from their vulnerability, they can do terrible things to other people or themselves.

Lucas Perry: Yeah, I mean, this is a sense of vulnerability in which, it’s a vulnerability of the recognition and sensitivity of your needs, but there’s a kind of stable foundation and security in that vulnerability. It’s a kind of vulnerability of self-knowing, it seems.

Maria Arpa: It’s vulnerability, plus trust.

Lucas Perry: It seems to me then, NVC’s place in the repertoire of an effective altruist, or someone interested in systemic change or existential risk, is that it becomes a tool in your toolkit for having a kind of discourse with people that may be surprising. I definitely believe in the capacity for enlightened or awakened people to exist as an example of what might be possible. And so if you come at someone with NVC who’s never experienced NVC before, I agree with you that that is where, “Oh, just have the conversation with the person,” might lead to some kind of transformative change. Because if you exist as a transformative example of what is possible, then there is the capacity for other people to recognize the goodness in you that is something that they would want and that leads to peace and freedom. NVC is obviously not the perfect solution to conversation, or the perfect solution to the problem of strategy, for example, and I guess, broadly, strategy can also be understood as game theory, where you’re going to have lots of different actors with different risk tolerances and incentives, but it is a much, much better way of communicating, full stop.

Maria Arpa: I notice I feel a slight discomfort when you call NVC a tool, because I don’t see it as a tool, I see it as a way of life.

Lucas Perry: Yeah, I hear that.

Maria Arpa: When I’m in that frame, because I look at the person I was 20 years ago, and I look at the person I am now and I see the transformation, but it’s because of the embodiment of something. It’s because it’s really helped me to look at all aspects of my life. It’s helped me to understand things that I wasn’t understanding, it helped me to wake up and become functional, and mindful, and all of those things, but that’s who I am now. I mean, I’m not saying that I’m some perfect person, and of course, occasionally, the shadows always there, but I’ve learned not to act on my shadow. I’ve learned to play with it. But when I am that embodiment or that person, then I’m bringing a new perspective into any conversation I have. And sometimes people find that disarming in an engaging way.

Lucas Perry: Yeah, that’s right. It can be disarming and engaging. I like that you use the word waking up. We just had a podcast with a former Buddhist monk and we talked a lot about awakening. And I agree with you that calling it a tool is an instrumentalisation of it, which lends itself to the problem-solving mindset, which is kind of adversarial with relation to the object which is being problem solved, which in this case, could be a person. So if it becomes a kind of non-conceptual embodied knowing or being, then there is the spiritual transformation and growth that is born of adopting this upgraded way of being. If you download the software of NVC, things will run much better.

Maria Arpa: So then I wanted to comment on the strategy. NVC is a way of unlocking something, okay. Now once I’ve unlocked it, and once I’ve got to the part where we’re now looking at strategies that will satisfy needs, now, we might need a different way of conversing. Now it might be very robust, it might be from the point of negotiation, and that negotiation may be very gentle and sensitive, but it can also be very boundaried. And so yeah, NVC for me is the way to unlock something, to bring people into a consciousness that what we’re going to do is, what’s the point of making strategies if we don’t understand the needs we’re trying to meet, and then using those needs as the measurement for whether the strategy is going to satisfy or not.

Lucas Perry: Okay. And I think I do also want to put a flag here, and you can tell me if this is wrong, that even those negotiations can fail. And that comes back to this kind of assumption that the world has sufficient abundance, that everyone’s needs can be met. So I mean, my perspective is I think that the negotiations can fail, and that when they fail, then something else is needed.

Maria Arpa: So if the negotiation has failed, in my experience, it’s because somebody wanted something, even if it was just speed, that wasn’t available. And so a really big deal for me is understanding where we want to get to, having that shared vision that we’re all trying to get to this place, and working towards it at the speed and tolerances of the whole group, and yet not allowing it to go at the slowest person’s pace. And that’s an art. There’s a real skill to, “We’re not going at the slowest person’s pace, but we’re also not going to take people out of their tolerances.”

Lucas Perry: But it seems like often with so many different people, tolerances are all over the place and can be strictly incompatible.

Maria Arpa: So that means we didn’t do enough work, and our shared vision isn’t right, and maybe we need to go back and look at the deeper needs. One of the things I talk about in this work is you’re never going to undo 30 years of poor communication in one conversation. It’s a process, and what I’m looking for is progress. And sometimes progress is literally just the agreement that the person will have another conversation with me, rather than slam the door in my face.

I’ve done neighbor disputes where I have knocked on someone’s door, they haven’t responded to the letters or the phone calls, and I have knocked on someone’s door, and I’ve got 30 seconds before they slam the door in my face and tell me, in no uncertain terms, tell me to whatever. And so for me at that moment, just then giving me two minutes, and then just getting to the agreement, I’m not going to try and do any business with you right now, just to get to the agreement that you will have another conversation with me, is progress.

And so it’s really about expectations and how quickly we think we can undo things or change things. And change processes are complex. How many times did you wake up and say, “I want to get fit or eat healthier food or lose weight or stop smoking or drink less,” or whatever it was, and then did you execute it straight away? No, you fluctuated. You probably relapsed, and relapse is a really important part of change. But then do we give up? Do we say, “Well that’s it, it’s over, we can’t negotiate,” or do we say, “Well, okay, that didn’t work. What else could we try?”

So in my world, and what I’ve understood, is the art, or the trick to life is not constantly searching to get your needs met. The trick to life is understanding that I have many needs, and on any day, week, month, year, some go met, and some go unmet, and I’m okay with that. It’s just looking on balance. Because if the aim of the game is to go, “Yeah, my need for this, this, this and this are all not being met, so therefore, I’m going to just make it my mission to get my needs met,” you’re still in the adversarial paradigm.

So I have lots of needs that go unmet, and you know what, it’s fine. It doesn’t mean I can’t express gratitude for what I do have. It doesn’t mean I don’t love everybody and everything in the way it is. It’s fine. I have no expectation that all my needs will get met.

Lucas Perry: Yeah, so you’re talking here about some of your experience, which I think boils down to some axioms or assumptions that one makes in NVC that I think are quite wholesome and beneficial. And I’ll just read off a bunch of them here and if you have any reactions or want to highlight any of them, then you can.

So the first is that all human beings have capacity for compassion and empathy. I believe in that. Behavior stems from trying to meet needs, so everything that we do is an expression of just trying to meet needs. You said earlier there are no bad or good people, they are just people trying to meet needs. Needs are universal, shared, and never in conflict. I think that one’s maybe 99.9% true. I don’t know how psychopaths fit in there, like Jeffrey Dahmer, fits in.

Maria Arpa: Well, I mean, I’ve worked with people in prisons who have been labeled as psychopaths, and I have on that very clear basis that people are selfish, greedy, kind and considerate, but the kind and considerate is either not on show, not available, has been put in a box, paralyzed, not today, I have woken up the kind and considerate.

Lucas Perry: You don’t think that there are people that are sufficiently neurodivergent and neuroatypical, that they don’t fit into these frameworks? It seems clearly physically possible.

Maria Arpa: It only runs out when I run out of patience, love, and tolerance to try. It only ends at that point, when I run out of patience, love and tolerance to try, and there might be many reasons why I would say, “I’m no longer going to try,” don’t get me wrong. We’re not asking everybody to just carry on regardless. But yeah, when I say I’ve had enough, and I don’t want to do this anymore, that’s when it, the trouble is we do that I think far too quickly with most people.

Lucas Perry: Yeah. All right. And so kind of moving a bit along here. The world has enough resources for meeting everyone’s basic needs, we mentioned this.

Maria Arpa: I do want to just comment on the world is abundant, and it has enough abundance to meet everybody’s needs.

Lucas Perry: Yeah.

Maria Arpa: The issue is, if it hasn’t, what’s the conversation we’re going to have? Or do we just want to inflict more and more unnecessary human suffering on each other? If, as is predicted, there’s going to be climate change on a scale that renders parts of the planet uninhabitable and there’s going to be mass migration, what are we going to do? Are we going to just keep killing them? Are we going to have a race to the bottom?

Lucas Perry: Are we going to leave it to power dynamics?

Maria Arpa: Yeah, or are we going to say, “Actually, things are getting tighter now, so we need to figure out how to collaborate so that we don’t kill each other.”

Lucas Perry: And then, so I was saying, if feelings point to needs being met or unmet, I would argue this is more like feelings point to needs being met or unmet, or the perception that they are being met or unmet.

Maria Arpa: So I just say, being able to identify the feeling and name the emotion is an alarm system. It’s our body’s natural alarm system, and we can use that to our advantage.

Lucas Perry: And I’ll just finish with the last one here that I think is related to the spiritual path, which says that the most direct path to peace is through self connection. So through self connection to needs, one becomes, I think, increasingly empathetic and compassionate, which leads to a deepening and an expression of NVC, which leads to more peace.

Maria Arpa: Yeah, the first marriage is this one. The first marriage is the one between I and I, and if that one ain’t working, nothing else is going to work.

Lucas Perry: All right. So can we engage in an NVC exercise now as an example of the format, so moving from observations to feelings to needs to request?

Maria Arpa: Okay, so I think it would be really useful if you could tell me about a situation that’s on your mind right now. And Marshall Rosenberg would say, “Can you tell me in 40 words or less what’s alive in you right now?”

Lucas Perry: So let’s NVC about this morning, then? I was late, and I kept you waiting for me, and I also showed up to the Zoom call late because… I guess the because doesn’t matter. Unless that’s part of the chronology of what happened. But yeah, I showed up to the Zoom call late and then my microphone wasn’t working, so I had to get another microphone. And I feel… how do I feel? I feel bad that I mean, bad isn’t the right word, though, right?

Maria Arpa: Mm-hmm (affirmative). But you just say it, just say it, and then we’ll go over it together.

Lucas Perry: Yeah, so I guess I regret and I feel bad that I wasn’t fully prepared, and then it took me 15, 20 minutes to get things started. And this probably relates to some need I have the podcasts go well, but also that I not damage my relationship with guests, which relates to some kind of need about… I mean, this probably goes all the way down to something like a need for love or self-love, very quickly, it would seem. And yeah, it’s unclear that we’ll have another conversation like this anytime soon, so it’s unclear to me what the kinds of requests are, though maybe there’s some requests for understanding and compassion for my failure to completely show up and arrive for the interview perfectly. How’s that for a start?

Maria Arpa: Yeah, it’s really very sweet, and so I’d love to just tell you some of what I’ve heard, first. So I heard you say that you’re feeling bad because you turned up late and unprepared for our interview, and that feeling bad or regretful is linked to some kind of need, at the end of the day you went there, and you said, “It’s probably a need for self-love.” And it’s hard to know what a request would be like, but I guess what I heard you say is you’d like to request some understanding.

Lucas Perry: Yeah, that’s right.

Maria Arpa: Before I respond to you, I would really love to just break down how you did observation, feeling, need, request, and then work with you a little on each of those. Would that be okay?

Lucas Perry: Yeah, that sounds good.

Maria Arpa: So the biggest judgment in your observation was the word late. It takes a bit of understanding, but who defines late?

Lucas Perry: Yeah, it started 15 minutes after the agreed upon time, is more of a concrete observation.

Maria Arpa: Well, a concrete observation actually, is that we got online at the time we agreed, and we didn’t start the interview until 15 minutes later.

Lucas Perry: Well, I was five minutes late to getting to the Zoom call.

Maria Arpa: Okay. Well, yeah, that word late again.

Lucas Perry: Sorry, I arrived five minutes…

Maria Arpa: After the agreed time.

Lucas Perry: Thank you.

Maria Arpa: Because late can be a huge weapon for self punishment. So the observation is, you came on the call five minutes after the agreed time, and we didn’t begin the interview until 15 minutes after the agreed time. So unprepared, late, and all of those things, they’re what you’re telling yourself. It’s part of the story, because from my perspective, since the example you gave includes me in that narrative, from my perspective, we didn’t have an agreement about what to expect at 10:00 AM your time, 3:00 PM my time. So how do I know you were unprepared? I’ve never done this before with you.

Lucas Perry: Okay.

Maria Arpa: Does that make sense? Can you see that you put a huge amount of judgment into what you thought was your observation, when in actual fact, who knows?

Lucas Perry: Yeah, so introducing other observations now here, is the observation that, I believe I pushed this back twice.

Maria Arpa: Once.

Lucas Perry: I pushed this back once?

Maria Arpa: Mm-hmm (affirmative).

Lucas Perry: Okay, so I pushed this back once. And then there was also this long period where I did not get back to you about scheduling after we had an initial conversation. So the feelings are things that need to be deconstructed. They’re around messiness, or disorganization, or not doing a good job, which are super synthetic in an evaluative, and would need to be super deconstructed, but that’s what I have right now.

Maria Arpa: So on some level, you didn’t meet your own standards.

Lucas Perry: No, I did not meet my own standards.

Maria Arpa: Right. So on some level, you didn’t meet your own standards, and that’s giving rise to a number of superficial feelings, like you’re feeling bad and guilty, and all of those things. And I can only guess, I’ve told myself, but perhaps you’re feeling regret, possibly some shame, and I don’t know why the word loneliness comes up for me. Isolation or loneliness, disconnection, something around, that you’ve screwed up, and now you have to sit in there and now there’s some shame and some regret and some embarrassment. Embarrassment, that’s it. I’m telling myself the feeling is embarrassment.

Lucas Perry: I’m trying to focus and see how accurate these are.

Maria Arpa: Yeah, and I could be completely wrong. I can only guess.

Lucas Perry: Yeah, I mean, you’re trying to guide me towards explaining my own feelings.

Maria Arpa: Yeah, so does anything resonate with embarrassment, or shame, or regret, or mourning.

Lucas Perry: Mostly regret. This kind of loneliness thing is interesting, because I think it’s related to the feeling of… If there was sufficient understanding and compassion and self-love, then these feelings wouldn’t arise because there would be understanding. And so the loneliness is born out of the self-rejection of the events as they transpired. And so there’s this need for wholeness, which is a kind of self-love and understanding and compassion and knowledge. It’s just a aligned state of being, and I’ve become unaligned by creating this evaluative story that’s full of judgment. Because all this could happen and I could feel totally fine, right? That roughly captures the feelings.

Maria Arpa: Okay. And then it really resonated for me and I heard you say that this need for wholeness, and definitely for understanding and love, and a deep need for mutuality.

Lucas Perry: Yeah. There’s a sense that I can’t fully meet you when I haven’t fully accepted myself for what I have done. Or if there’s a kind of self consciousness around the events and how it has impacted you.

Maria Arpa: Yeah, so I’m telling myself that we made an agreement, and actually, part of it is a story that you’re telling yourself, and part of it has some reality in it, that you didn’t meet the terms of our agreement.

Lucas Perry: Yeah.

Maria Arpa: And then what that’s doing to you is, when you didn’t meet the terms of the agreement, I’m telling myself that now what happens to you is you worry, I think that’s the word. Ah, that’s it, maybe the feeling’s worry, or anxiety, that then any connection that we might have made is disconnecting or breaking, or we’re losing mutuality, because I may be now looking at you differently for not having met the terms of the agreement.

Lucas Perry: Yeah, and there’s also a level of self-rejection. And someone who is self-rejecting is in contradiction and cannot fully connect with someone else. If you find that yourself is unlovable or that you do not love yourself, then it’s like impossible to love someone else. So I think there’s a sense also, then of, if you’re creating a bad narrative and that’s leading to a sense of self-rejection, then there’s an inability to fully be with the other person. So then I think that’s why you were pointing out to the sense of loneliness, because this kind of self-rejection leads to isolation and inability to fully meet the person as they are.

Maria Arpa: Yeah, so we went over the observation, we got to some of the feelings, we got to some of the needs, now do you have a request? And I think I heard you say you have a request for understanding.

Lucas Perry: Yeah, understanding and compassion, probably mostly from myself, but also from you.

Maria Arpa: Yeah, so I’m wondering if you’d like me to respond to that request now. Would that be helpful for you?

Lucas Perry: Yeah, sure.

Maria Arpa: So I guess when I hear your request for understanding and compassion, and that you’re also recognizing you need to give it to yourself, and that’s a relief for me that you know you need to give it to yourself, and yet on some level, we do have a situation over an agreement that was broken. I would love for you to be able to hear where I am. And I’m just wondering, would you be willing to hear where I am in that, to support you in your request?

Lucas Perry: Yeah, and probably some need for love in the sense of, I mean, there are different kinds of love. So whatever kind of love exists between you and I, as human beings that is coworker love, or colleague love, or whatever kind of relationship we have, I don’t know how to explain our relationship, but whatever kind of love is appropriate in that context.

Maria Arpa: So I guess where I’m coming from, is, I feel deeply privileged and honored to be asked to do this podcast. I’ve heard some of your other podcasts, and I think they’re masterpieces. So to be invited to do this at all and for us to have met and for you to have actually said, “Yeah, let’s go ahead and do this,” went a long way for me to believe in myself as well.

So you may be having your own moment of self punishment, and so did I. “What happens if he doesn’t like me at the end of our interview, and doesn’t want to do it?” And in terms of our agreement, as far as I’m concerned, you got online roughly at the time we said, and I have no idea, we didn’t have a tacit agreement that the interview then starts at, and so in terms of being alongside you while you make preparations and whatever, actually, it helped me to see you as human. So it actually increased my love for you.

Lucas Perry: Oh, nice.

Maria Arpa: Because I saw in that first meeting, you were kind of interviewing me and seeing if there was suitability for a podcast. And of course, I know I know my stuff, right. That’s not the issue. But there was a sense of me wanting to be on best behavior. But now I come to this call, and there you were just being human and expressing it, and I was able to say a few things to you. And I felt that 15 minutes was very connecting, it was very connecting for me. And so I just wonder when you hear that, does it change anything for you?

Lucas Perry: Yeah, I feel much better. I feel more capacity to connect with you and I appreciate the honesty and transition of how you felt with regards to the first time we talked where, because I couldn’t find any of your content online, so I didn’t really know anything about you, so we had this first conversation where you felt almost as if there was kind of evaluative relationship going on, which was then I guess, dissolved by having this conversation in particular, but also the beginning of the podcast where I was being human and my microphone wasn’t working, and my electricity was out this morning and things weren’t working out. So yeah, I appreciate that. I feel much better and warm and more capacity for self-love and connection. So thanks.

Maria Arpa: Yeah.

Lucas Perry: I think that means the NVC was successful.

Maria Arpa: Yeah. And then just to add one thing, during the interview, I heard you say something like, “I don’t know what my requests would be because the opportunity for us to connect again like this,” or whatever, you said something about that we probably wouldn’t speak to each other again in a hurry. I actually felt really sad when I heard that. I felt such sadness, “Oh, no, I’ve connected with Lucas now.” So I hope that there’ll be other opportunities to just chat or stay in touch or whatever, because there’s something about you that I feel really resonates. And I love where you’re coming from, and I love what you’re trying to do. It’s really important.

Lucas Perry: Well, thank you, that’s really sweet. I appreciate it.

Maria Arpa: Thank you. And look at that, we’re bang on time.

Lucas Perry: So that means you escaped question eight, which is my criticisms of NVC.

Maria Arpa: We could come back and add that on another time, but I can’t do it now. If you want to do another bit, we can do another bit. I’m really happy to do that.

Lucas Perry: Yeah. Great. So as we wrap up, if people want to follow you or to learn more about NVC, or the Center for Nonviolent Communication, where are the best places to follow you or get more information or check out more of Marshall’s work?

Maria Arpa: So obviously, CNVC has a website which is CNVC, Center for Nonviolent Communication, .org. That’s your first port of call. Marshall’s books are published by Puddle Dancer Press. I know Meiji very well. I know him reasonably well, and he’s a really wonderful guy, so buy some books from Puddle Dancer Press, because Marshall’s books are amazing. There are 700 NVC trainers across the world, and you can find those on the website if you go to the right bit and search. So if you want to find someone local in your area, and they all work differently and specialize in different things. If you put NVC into Facebook, you will find countless NVC pages. And if you’re looking for me, Google my name, Maria Arpa, and I will come up. Thank you.

Lucas Perry: All right. Thanks, Maria.

Stephen Batchelor on Awakening, Embracing Existential Risk, and Secular Buddhism

October 15, 2020/in Featured, FLI Podcasts, FLI podcasts & transcripts /by Lucas Perry

Future of Life Institute · Stephen Batchelor on Awakening, Embracing Existential Risk, and Secular Buddhism

Topics discussed in this episode include:

The projects of awakening and growing the wisdom with which to manage technologies
What might be possible of embarking on the project of waking up
Facets of human nature that contribute to existential risk
The dangers of the problem solving mindset
Improving the effective altruism and existential risk communities

Timestamps:

0:00 Intro

3:40 Albert Einstein and the quest for awakening

8:45 Non-self, emptiness, and non-duality

25:48 Stephen’s conception of awakening, and making the wise more powerful vs the powerful more wise

33:32 The importance of insight

49:45 The present moment, creativity, and suffering/pain/dukkha

58:44 Stephen’s article, Embracing Extinction

1:04:48 The dangers of the problem solving mindset

1:26:12 Improving the effective altruism and existential risk communities

1:37:30 Where to find and follow Stephen

Citations:

Stephen’s website

Stephen’s teachings and courses

We hope that you will continue to join in the conversations by following us or subscribing to our podcasts on Youtube, Spotify, SoundCloud, iTunes, Google Play, Stitcher, iHeartRadio, or your preferred podcast site/application. You can find all the AI Alignment Podcasts here.

You can listen to the podcast above or read the transcript below.

Lucas Perry: Welcome to the Future of Life Institute Podcast. I’m Lucas Perry. Today, we have a special episode for you with Stephen Batchelor. Stephen is a secular and skeptical Buddhist teacher and practitioner with many years under his belt in a variety of different Buddhist traditions. You’ve probably heard often on this podcast about the dynamics of the race between the power of our technology and the wisdom with which we manage it. This podcast is primarily centered around the wisdom portion of this dynamic and how we might cultivate wisdom, and how that relates to the growing power of our technology. Stephen and I get into discussing the cultivation of wisdom, what awakening might entail or look like. And also his views on embracing existential risk and existential threats. As for a little bit more background, we can think of ourselves as contextualized in a world of existential threats that are primarily created due to the kinds of minds that people have and how we behave. Particularly how we decide to use industry and technology and science and the kinds of incentives and dynamics that are born of that. And so cultivating wisdom here in this conversation is seeking to try to and understand how we might better gain insight into and grow beyond the worst parts of human nature. Things like hate, greed, and delusion, which motivate and help to cultivate the manifestation of existential risks. The flipside of understanding the ways in which hate, greed, and delusion motivate and lead to the manifestation of existential risk is also uncovering and being interested in the project of human awakening and developing into our full potential. So, this just means that whatever idealized kind of version you think you might want to be or that you might strive to be there is a path to getting there and this podcast is primarily interested in that path and how that path relates to living in a world of existential threat and how we might relate to existential risk and its mitigation. This podcast contains a bit of Buddhist jargon in it. I do my best in this podcast to define the words to the best of my ability. I’m not an expert but I think that these definitions will help to bring a bit of context and understanding to some of the conversation.

Stephen Batchelor is a contemporary Buddhist teacher and writer, best known for his secular or agnostic approach to Buddhism. Stephen considers Buddhism to be a constantly evolving culture of awakening rather than a religious system based on immutable dogmas and beliefs. Through his writings, translations and teaching, Stephen engages in a critical exploration of Buddhism’s role in the modern world, which has earned him both condemnation as a heretic and praise as a reformer. And with that, let’s get into our conversation with Stephen Batchelor.

Thanks again so much for coming on. I’ve been really excited and looking forward to this conversation. I just wanted to start it off here with a quote by Albert Einstein that I thought would set the mood and the context. “A human being is a part of the whole called by us universe, a part limited in time and space. He experiences himself, his thoughts and feelings as something separated from the rest, a kind of optical delusion of his consciousness. This delusion is a kind of prison for us, restricting us to our personal desires and to affection for a few persons nearest to us. Our task must be to free ourselves from this prison by widening our circle of compassion to embrace all living creatures, and the whole of nature and its beauty. Nobody is able to achieve this completely. But the striving for such achievement is in itself a part of the liberation and a foundation of inner security.”

This quote to me is compelling, because, one, it comes from someone who is celebrated as one of the greatest scientists that have ever lived. In that sense, it’s a calling for the spiritual journey, it seems, from someone who, for people who are skeptical of something like the project of awakening or whatever a secular Dharma might be or look like. I think it sets up well the project. I mean, he talks about here how this idea of separation is a kind of optical delusion of his consciousness. He sets it up as the problem of trying to arrive at experiential truth and this project of self-improvement. It’s in the spirit of this, I think, seeking to become and live an engaged and fulfilled life that I am interested and motivated in having this conversation with you.

With that in mind, the problem, it seems, that we have currently in the 21st century is what Max Tegmark and others have called the race between the power of our technology and the wisdom with which we manage it. I’m basically interested in discussing and exploring how to grow wisdom and about how to grow into and develop full human potential so that we can manage powerful things like technology.

Stephen Batchelor: I love the quote. I think I’ve heard it before. I’ve come across a number of similar statements that Einstein has made over the years of his life. I’ve always been impressed by that. As you say, this is a man who’s not regarded remotely as a religious or a spiritual figure. Yet, obviously, a highly sensitive man, a man who has plumbed the depths of physics in a way that has transformed our world. Clearly, someone with enormous insight and understanding of the kind of universe we live in. Yet, at the same time, in these sorts of passages, we realize that he’s not just the stereotyped, detached scientist separated out from the world looking at things clinically and trying to completely subtract his own subjectivity.

This, I think, is often the problem with scientific approaches. The idea is that you have to get yourself out of the way in order to somehow see things as they really are. Einstein breaks that stereotype very well and recognizes the need that if we are to evolve as human beings and not just as scientists who get increasingly clear and maybe very deep understandings into the workings of the universe, som