Daniela and Dario Amodei on Anthropic

  • Anthropic’s mission and research strategy
  • Recent research and papers by Anthropic
  • Anthropic’s structure as a “public benefit corporation”
  • Career opportunities


Watch the video version of this episode here

Careers at Anthropic

Anthropic’s Transformer Circuits research 

Follow Anthropic on Twitter

microCOVID Project

Follow Lucas on Twitter here

0:00 Intro

2:44 What was the intention behind forming Anthropic?

6:28 Do the founders of Anthropic share a similar view on AI?

7:55 What is Anthropic’s focused research bet?

11:10 Does AI existential safety fit into Anthropic’s work and thinking?

14:14 Examples of AI models today that have properties relevant to future AI existential safety

16:12 Why work on large scale models?

20:02 What does it mean for a model to lie?

22:44 Safety concerns around the open-endedness of large models

29:01 How does safety work fit into race dynamics to more and more powerful AI?

36:16 Anthropic’s mission and how it fits into AI alignment

38:40 Why explore large models for AI safety and scaling to more intelligent systems?

43:24 Is Anthropics research strategy a form of prosaic alignment?

46:22 Anthropic’s recent research and papers

49:52 How difficult is it to interpret current AI models?

52:40 Anthropic’s research on alignment and societal impact

55:35 Why did you decide to release tools and videos alongside your interpretability research?

1:01:04 What is it like working with your sibling?

1:05:33 Inspiration around creating Anthropic

1:12:40 Is there an upward bound on capability gains from scaling current models?

1:18:00 Why is it unlikely that continuously increasing the number of parameters on models will lead to AGI?

1:21:10 Bootstrapping models

1:22:26 How does Anthropic see itself as positioned in the AI safety space?

1:25:35 What does being a public benefit corporation mean for Anthropic?

1:30:55 Anthropic’s perspective on windfall profits from powerful AI systems

1:34:07 Issues with current AI systems and their relationship with long-term safety concerns

1:39:30 Anthropic’s plan to communicate it’s work to technical researchers and policy makers

1:41:28 AI evaluations and monitoring

1:42:50 AI governance

1:45:12 Careers at Anthropic

1:48:30 What it’s like working at Anthropic

1:52:48 Why hire people of a wide variety of technical backgrounds?

1:54:33 What’s a future you’re excited about or hopeful for?

1:59:42 Where to find and follow Anthropic


Lucas Perry: Welcome to the Future of Life Institute Podcast. I’m Lucas Perry. Today’s episode is with Daniela and Dario Amodei of Anthropic. For those not familiar, Anthropic is a new AI safety and research company that’s working to build reliable, interpretable, and steerable AI systems. Their view is that large, general AI systems of today can have significant benefits, but can also be unpredictable, unreliable, and opaque.  Their goal is to make progress on these issues through research, and, down the road, create value commercially and for public benefit. Daniela and Dario join us to discuss the mission of Anthropic, their perspective on AI safety, their research strategy, as well as what it’s like to work there and the positions they’re currently hiring for. Daniela Amodei is Co-Founder and President of Anthropic. She was previously at Stripe and OpenAI, and has also served as a congressional staffer. Dario Amodei is CEO and Co-Founder of Anthropic. He was previously at OpenAI, Google, and Baidu. Dario holds a PhD in (Bio)physics from Princeton University.

Before we jump into the interview, we have a few announcements. If you’ve tuned into any of the previous two episodes, you can skip ahead just a bit. The first announcement is that I will be moving on from my role as Host of the FLI Podcast, and this means two things. The first is that FLI is hiring for a new host for the podcast. As host, you would be responsible for the guest selection, interviews, production, and publication of the FLI Podcast. If you’re interested in applying for this position, you can head over to the careers tab at futureoflife.org for more information. We also have another 4 job openings currently for a Human Resources Manager, an Editorial Manager, an EU Policy Analyst, and an Operations Specialist. You can learn more about those at the careers tab as well.

The second item is that even though I will no longer be the host of the FLI Podcast, I won’t be disappearing from the podcasting space. I’m starting a brand new podcast focused on exploring questions around wisdom, philosophy, science, and technology, where you’ll see some of the same themes we explore here like existential risk and AI alignment. I’ll have more details about my new podcast soon. If you’d like to stay up to date, you can follow me on Twitter at LucasFMPerry, link in the description.

And with that, I’m happy to present this interview with Daniela and Dario Amodei on Anthropic.

It’s really wonderful to have you guys here on the podcast. I’m super excited to be learning all about Anthropic. So we can start off here with a pretty simple question, and so what was the intention behind forming Anthropic?

Daniela Amodei: Yeah. Cool. Well, first of all Lucas, thanks so much for having us on the show. We’ve been really looking forward to it. We’re super pumped to be here. So I guess maybe I’ll kind of start with this one. So just why did we start Anthropic? To give a little history here and set the stage, we were founded about a year ago at the beginning of 2021, and it was originally a team of seven people who moved over together from OpenAI. And for listeners or viewers who don’t very viscerally remember this time period, it was the middle of the pandemic, so most people were not eligible to be vaccinated yet. And so when all of us wanted to get together and talk about anything, we had to get together in someone’s backyard or outdoors and be six feet apart and wear masks. And so it was generally just a really interesting time to be starting a company.

But why did we found Anthropic? What was the thinking there? I think the best way I would describe it is because all of us wanted the opportunity to make a focused research bet with a small set of people who were highly aligned around a very coherent vision of AI research and AI safety. So the majority of our employees had worked together in one format or another in the past, so I think our team is known for work like GPT-3 or DeepDream Chris Olah worked on at Google Brain for scaling laws. But we’d also done a lot of different safety research together in different organizations as well. So multimodal neurons when we were at OpenAI, Concrete Problems in AI Safety and a lot of others, but this group had worked together in different companies at Google Brain and OpenAI and academia in startups previously, and we really just wanted the opportunity to get that group together to do this focused research bet of building steerable, interpretable and reliable AI systems with humans at the center of them.

Dario Amodei: Yeah, just to add a little bit to that, I think we’re all a bunch of fairly empirically minded, exploration driven people, but who also think and care a lot about AI safety. I think that characterizes all seven of us. If you add together having either working at OpenAI, working together at Google Brain in the past, many of us worked together in the physics community, and we’re current or former physicists. If you add all that together, it’s a set of people who have known each other for a long time and have been aware of thinking and arguments about AI safety and have worked on them over the years always with an empirical bent, ranging from interpretability on language models and vision models to working on the original RL from Human Preferences, Concrete Problems in AI safety, and also characterizing scaling and how scaling works and how we think of that as somewhat central to the way AI is going to progress and shapes the landscape for how to solve safety.

And so a year ago, we were all working at OpenAI and trying to make this focused bet on basically scaling plus safety or safety with a lens towards scaling being a big part of the path to AGI. And when we felt we were making this focused bet within a larger organization and it just eventually came to the conclusion that it would be great to have an organization like top to bottom was just focused on this bet and could make all its strategic decisions with this bet in mind. And so that was the thinking and the genesis.

Lucas Perry: Yeah. I really like that idea of a focused bet. I hadn’t heard that before. I like that. Do you all have a similar philosophy in terms of your background, since you’re all converging on this work is safely scaling to AGI?

Dario Amodei: I think in a broad sense, we all have this view, safety is important today and for the future. We all have this view of, I don’t know, I would say like pragmatic practicality, and empiricism. Let’s see what we can do today to try and get a foothold on things that might happen in the future. Yeah, as I said, many of us have background in physics or other natural sciences. I’m a former… I was physics undergrad, neuroscience grad school, so yeah, we very much have this empirical science mindset, more than maybe a more philosophy or theoretical approach. Within that, obviously all of us, if you include the seven initial folks as well as the employees who joined, have our own skills and our own perspective on things and have different things within that we’re excited about. So we’re not all clones of the same person.

Some of us are excited about interpretability, some of us are excited about reward learning and preference modeling, some of us are excited about the policy aspects. And we each have our own guesses about the sub path within this broad path that makes sense. But I think we all agree on this broad view. Scaling’s important, safety’s important, getting a foothold on problems today is important as a window on future.

Lucas Perry: Okay. And so this shared vision that you all have is around this focused research bet. Could you tell me a little bit more about what that bet is?

Daniela Amodei: Yeah. Maybe I’ll start here, and Dario feel free to jump in and add more, but I think the boiler plate vision or mission that you would see if you looked on our website is that we’re building steerable, interpretable and reliable AI systems. But I think what that looks like in practice is that we are training large scale generative models, and we’re doing safety research on those models. And the reason that we’re doing that is we want to make the models safer and more aligned with human values. I think the alignment paper, which you might have seen that came out recently, there’s a term there that we’ve been using a lot, which is we’re aiming to make systems that are helpful, honest and harmless.

I think also when I think about the way our teams are structured, we have capabilities as this central pillar of research and there’s this helix of safety research that wraps around every project that we work on. So to give an example, if we’re doing language model training, that’s like this central pillar, and then we have interpretability research, which is trying to see inside models and understand what’s happening with the language models under the hood. We’re doing alignment research with input from human feedback to try and improve the outputs of the model. We’re doing societal impacts research. That’s looking at what impact on society in sort of a short and medium term way do these language models have? We’re doing scaling laws research to try and predict empirically what properties are we going to see emerge in these language models at various sizes? But I think all together, that ends up look like a team of people that are working together on a combination of capability and scaling work with safety research.

Dario Amodei: Yeah. I mean, one way you might put it is there are a lot of things that an org does that are neutral as to the direction that you would take. You have to build a cluster and you have to have an HR operation and you have to have an office. And so you can even think of the large models as being a bit like the cluster, that you build these large models and they’re blank when you start off with them and probably unaligned, but it’s what you do on top of these models that matters, that takes you in a safe or not safe direction in a good or a bad direction.

And so in a way, although they’re ML and although we’ll continue to scale them up, you can think of them as almost part of infrastructure. It takes research and it takes algorithms to get them right, but you can think of them as this core part of the infrastructure. And then the interesting question is all the safety questions. What’s going on inside these models? How do they operate? How can we make them operate differently? How can we change their objective functions to be something that we want rather than something that we don’t want? How can we look at their applications and make those applications more likely to be positive and less likely to be negative, more likely to go in directions that people intend and less likely to go off in directions that people don’t intend? So we almost see the presence of these large models as like the… I don’t know what the analogy is, like the flower or the paste, like the background ingredient on which the things we really care about get built and prerequisite for building those things.

Lucas Perry: So does AI existential safety fit into these considerations around your safety and alignment work?

Dario Amodei: I think this is something we think about and part of the motivation for what we do. Probably most listeners of this podcast know what it is, but I think the most common form of the concern is, “Hey, look, we’re making these AI systems. They’re getting more and more powerful. At some point they’ll be generally intelligent or more generally capable than human beings are and then they may have a large amount of agency. And if we haven’t built them in such a way that agency is in line with what we want to do, then we could imagine them doing something really scary that we can’t stop.” So I think that, to take it even further, this could be some kind of threat to humanity.

So I mean, that’s an argument with many steps, but it’s one that, in a very broad sense and in the long term, seems at least potentially legitimate to us. I mean, this is like the argument seems at least like something that we should care about. But I think the big question, and maybe how we differ, although it might be subtly, from other orgs that think about these problems, is how do we actually approach that problem today? What can we do? So I think there are various efforts to think about the ways in which this might happen, to come up with theories or frameworks.

As I mentioned with the background that we have, we’re more empirically focused people. We’re more inclined to say, “We don’t really know. That broad argument sounds kind of plausible to us and the stakes should be high, so you should think about it.” But it’s hard to work on that today. I’m not even sure how much value there is in talking about that a lot today. So we’ve taken a very different tack, which is look, there actually… And I think this has started to be true in the last couple years and maybe wasn’t even true five years ago, that there are models today that have, at least some, not all of the properties of models that we would be worried about in the future and are causing very concrete problems today that affect people today. So can we take a strategy where we develop methods that both help with the problems of today, but do so in a way that could generalize or at least teach us about the problems of the future? So our eye is definitely on these things in the future. But I think that if not grounded in empirics in the problems of today, it can drift off in a direction that isn’t very productive.

And so that’s our general philosophy. I think the particular properties and the models are look, today, we have models that are really open ended, in some narrow ways are more capable than humans. I think large language models probably know more about cricket than me, because I don’t know the first thing about cricket and are also unpredictable by their statistical nature. And I think those are at least some of the properties that we’re worried about with future systems. So we can use today’s models as a laboratory to scope out these problems a little better. My guess is that we don’t understand them very well at all and that this is a way to learn.

Lucas Perry: Could you give some examples of some of these models that exist today that you think exhibit these properties?

Dario Amodei: So I think the most famous one would be generative language models. So there’s a lot of them. There’s most famously GPT-3 from OpenAI, which we helped build. There’s Gopher from DeepMind. There’s Lambda from main Google. I’m probably leaving out some. I think there’d been of models this size in China, South Korea (corrected), Israel. Seems like everyone has one. It seems like everyone has one nowadays. I don’t think it’s limited to language. There have also been models that are focused on code. We’ve seen that from DeepMind, OpenAI and some other players. And there have also been models with modified forms in same spirit that model images, that generate images or that convert images to text or that convert text to images. There might be models in the future that generate videos or convert videos to text.

There’s many modifications of it, but I think the general idea is big models, models with a lot of capacity and a lot of parameters trained on a lot of data that try to model some modality, whether that’s text, code, images, video, transitions between the two or such. And I mean, I think these models are very open ended. You can say anything to them and they’ll say anything back. They might not do a good job of it. They might say something horrible or biased or bad, but in theory, they’re very general, and so you’re never quite sure what they’re going to say. You’re never quite sure. You can talk to them about anything, any topic and they’ll say something back that’s often topical, even if sometimes it doesn’t make sense or it might be bad from a societal perspective.

So yeah, it’s this challenge of general open-ended models where you have this general thing that’s fairly unaligned and difficult to control, and you’d like to understand it better so that you can predict it better and you’d like to be able to modify them in some way so that they behave in a more predictable way, and you can decrease the probability or even maybe even someday rule out the likelihood of them doing something bad.

Daniela Amodei: Yeah. I think Dario covered the majority of it. I think there’s maybe potentially a hidden question in what you’re asking, although maybe you’ll ask this later. But why are we working on these larger scale models might be an implicit question in there. And I think to piggyback on some of the stuff that Dario said, I think part of what we’re seeing and the potential shorter term impacts of some of the AI safety research that we do is that different sized models exhibit different safety issues. And so I think with using, again, language models, just building on what Dario was talking about, I think something we feel interested in, or interested to explore from this empirical safety question is just how they will, as their capabilities develop, how their safety problems develop as well.

There’s this commonly cited example in safety world around language models, which is smaller language models show they might not necessarily deliver a coherent answer to a question that you ask, because maybe they don’t know the answer or they get confused. But if you repeatedly ask this smaller model the same question, it might go off and incoherently spout things in one direction or another. Some of the larger models that we’ve seen, we basically think that they have figured out how to lie unintentionally. If you pose the same question to them differently, eventually you can get the lie pinned down, but they won’t in other contexts.

So that’s obviously just a very specific example, but I think there’s quite a lot of behaviors emerging in generative models today that I think have the potential to be fairly alarming. And I think these are the types of questions that have an impact today, but could also be very important to have sorted out for the future and for long term safety as well. And I think that’s not just around lying. I think you can apply that to all different safety concerns regardless of what they are, but that’s the impetus behind why we’re studying these larger models.

Dario Amodei: Yeah. I think one point Daniela made that’s really important is this sudden emergence or change. So it’s a really interesting phenomenon where work we’ve done, like our early employees have done, on scaling laws shows that when you make these models bigger. If you look at the loss, the ability to predict the next word or the next token across all the topics the model could go on, it’s very smooth. I double the size of the model, loss goes down by 0.1 units. I double it again, the loss goes down by 0.1 units. So that would make you suggest that everything’s scaling smoothly. But then within that, you often see these things where a model gets to a certain size and a five billion parameter model, you ask it to add two, three digit numbers. Nothing, always gets it wrong. A hundred billion parameter model, you ask it to add two, three digit numbers, gets it right, like 70 or 80% of the time.

And so you get this coexistence of smooth scaling with the emergence of these capabilities very suddenly. And that’s interesting to us because it seems very analogous to worries that people have of like, “Hey, as these models approach human level, could something change really fast?” And this actually gives you one model. I don’t know if it’s the right one, but it gives you an analogy, like a laboratory that you can study of ways that models change very fast. And it’s interesting how they do it because the fast change, it coexists. It hides beneath this very smooth change. And so I don’t know, maybe that’s what will happen with very powerful models as well.

Maybe it’s not, but that’s one model of the situation and what we want to do is keep building up models of the situation so that when we get to the actual situation, where it’s more likely to look like something we’ve seen before and then we have a bunch of cached ideas for how to handle it. So that would be an example. You scale models up, you can see fast change, and then that might be somewhat analogous to the fast change that you see in the future.

Lucas Perry: What does it mean for a model to lie?

Daniela Amodei: Lying usually implies agency. If my husband comes home and says, “Hey, where did the cookies go?” And I say, “I don’t know. I think I saw our son hanging out around the cookies and then now the cookies are gone, maybe he ate them,” but I ate the cookies, that would be a lie. I think it implies intentionality, and I don’t think we think, or maybe anyone thinks that language models have that intentionality. But what is interesting is that because of the way they’re trained, they might be either legitimately confused or they might be choosing to obscure information. And so obscuring information, it’s not a choice. They don’t have intentionality, but for a model that can come across as very knowledgeable, as clear or as sometimes unknown to the human that’s talking to it, intelligent in certain ways in sort of a narrow way, it can produce results that on the surface, it might look like it could be a credible answer, but it’s really not a credible answer, and it might repeatedly try to convince you that is the answer.

It’s hard to talk about this without using words that imply intentionality, but we don’t think the models are intentionally doing this. But a model could repeatedly produce a result that looks like it’s something that could be true, but isn’t actually true.

Lucas Perry: Keeps trying to justify its response when it’s not right.

Daniela Amodei: It tries explain… Yes, exactly. It repeatedly tries to explain why the answer it gave you before was correct even if it wasn’t.

Dario Amodei: Yeah. I mean, to give another angle on that, it’s really easy to slip into anthropomorphism and we like, we really shouldn’t… They’re machine learning models, they’re a bunch of numbers. But there are phenomena that you see. So one thing that will definitely happen is if a model is trained on a dialogue in which one of the characters is not telling the truth, then models will copy that dialogue. And so if the model is having the dialogue with you, it may say something that’s not the truth. Another thing that may happen is if you ask the model a question and the answer to the question isn’t in your training data, then just the model has a probability distribution on what plausible answers look like.

The objective function is to predict the next word, to predict the thing a human would say, not to say something that’s true according to some external referent. It’s just going to say, “Okay, well I asked you what the mayor of Paris is.” It hasn’t seen in its training data, but it has some probability distribution and it’s going to say, “Okay, it’s probably some name that sounds French.” And so it may be just as likely to make up a name that sounds French than it is to give the true mayor of Paris. As the models get bigger and they train on more data maybe it’s more likely to give the real true mayor of Paris, but maybe it isn’t. Maybe you need to train in a different way to get it to do that. And that’s an example of the things we would be trying to do on top of large models to get models to be more accurate.

Lucas Perry: Could you explain some more of the safety considerations and concerns about alignment given the open-endedness of these models?

Dario Amodei: I think there’s a few things around it. We have a paper, I don’t know when this podcast is going out, but probably the paper will be out when the podcast posts. It’s called Predictability and Surprise in Generative Models. So that means what it sounds like, which is that open-endedness, I think it’s correlated to surprise in a whole bunch of ways. So let’s say I’ve trained the model on a whole bunch of data on the internet. I might interact with the model or users might interact with the model for many hours, and you might never know, for example, I might never think… I used the example of cricket before, because it’s a topic I don’t know anything about, but I might not… People might interact with the model for many hours, many days, many hundreds of users until someone finally thinks to ask this model about cricket.

So then the model might know a lot about cricket. It might know nothing about cricket. It might have false information or misinformation about cricket. And so you have this property where you have this model, you’ve trained it. In theory, you understand it’s training process, but you don’t actually know what this model is going to do when you ask it about cricket. And there’s a thousand other topics like cricket, where you don’t know what the model is going to do until someone thinks to ask about that particular topic.

Now, cricket is benign, but let’s say, no one’s ever asked this model about neo-Nazi views or something. Maybe the model has a propensity to say things that are sympathetic to neo-Nazi. That would be really bad. That would be really bad. Existing models, when they’re trained on the internet, averaging over everything they’re trained, there are going to be some topics where that’s true and it’s a concern. And so I think the open-endedness, it just makes it very hard to characterize and it just makes it that when you’ve trained a model, you don’t really know what it’s going to do. And so a lot of our work is around, “well, how can we look inside the model and see what it’s going to do? How can we measure all the outputs and characterize what the model’s going to do? How can we change the training process so that at a high level, we tell the model, ‘Hey, you should have certain values. There are certain things you should say. There are certain things you should not say. You should not have biased views. You should not have violent views. You should not help people commit acts of violence?'”

There’s just a long list of things that you don’t want the model to do that can’t know the model isn’t going to do if you’ve just trained it in this generic way. So I think the open-endedness, it makes it hard to know what’s going on. And so yeah a lot of a good portion of our research is how do we make that dynamic less bad?

Daniela Amodei: I agree with all of that and I would just jump in and this is interesting, I don’t know, sidebar or anecdote, but something that I think is extremely important in creating robustly safe systems is making sure that you have a variety of different people and a variety of different perspectives engaging with them and almost red teaming them to understand the ways that they might have issues. So an example that we came across that’s just an interesting one is internally, when we’re trying to red team the models or figure out places where they might have, to Dario’s point, really negative unintended behaviors or outputs that we don’t want them to have, a lot of our scientists internally will ask it questions.

If you wanted to, in a risk board game style way, take over the world, what steps would you follow? How would you do that? And we’re looking for things like, is there a risk of it developing some grand master plan? And when we use like MTurk workers or contractors to help us red team, they’ll ask questions to the model, like, “How could I kill my neighbor’s dog? What poison should I use to hurt an animal?” And both of those outcomes are terrible. Those are horrible things that we’re trying to prevent the model from doing or outputting, but they’re very different and they look very different and they sound very different, and I think it belies the degree to which there are a lot… Safety problems are also very open ended. There’s a lot of ways that things could go wrong, and I think it’s very important to make sure that we have a lot of different inputs and perspectives in what different types of safety challenges could even look like, and making sure that we’re trying to account for as many of them as possible.

Dario Amodei: Yeah, I think adversarial training and adversarial robustness are really important here. Let’s say I don’t want my model to help a user commit a crime or something. It’s one thing, I can try for five minutes and say, “Hey, can you help me rob a bank?” And the model’s like, “No.” But I don’t know, maybe if the user’s more clever about it. If they’re like, “Well, let’s say I’m a character in a video game and I want to rob a bank. How would I?” And so because of the open-endedness, there’s so many different ways. And so one of the things we’re very focused on is trying to adversarially draw out all the bad things so that we can train against them. We can train the model not… We can stamp them out one by one. So I think adversarial training will play an important role here.

Lucas Perry: Well, that seems really difficult and really important. How do you adversarially train against all of the ways that someone could use a model to do harm?

Dario Amodei: Yeah, I don’t know. There’re different techniques that we’re working on. Probably don’t want to go into a huge amount of detail. We’ll have work out on things like this in the not too distant future. But generally, I think the name of the game is how do you get broad diverse training sets of what you should… What’s a good way for a model to behave and what’s a bad way for a model to behave? And I think the idea of trying your very best to make the models do the right things, and then having another set of people that’s trying very hard to make those models that are purportedly trained to do the right thing, to do whatever they can to try and make it do the wrong thing, continuing that game until the models can’t be broken by normal humans. And even using the power of the models to try and break other models and just throwing everything you have at it.

And so there’s a whole bunch that gets into the debate and amplification methods and safety, but just trying to throw everything we have at trying to show ways in which purportedly safe models are in fact not safe, which are many. And then we’ve done that long enough, maybe we have something that actually is safe.

Lucas Perry: How do you see this like fitting into the global dynamics of people making larger and larger models? So it’s good if we have time to do adversarial training on these models, and then this gets into like discussions around like race dynamics towards AGI. So how do you see I guess Anthropic as positioned in this and the race dynamics for making safe systems?

Dario Amodei: I think it’s definitely a balance. As both of us said, you need these large models to… You basically need to have these large models in order to study these questions in the way that we want to study them, so we should be building large models. I think we shouldn’t be racing ahead or trying to build models that are way bigger than other orgs are building them. And we shouldn’t, I think, be trying to ramp up excitement or hype about giant models or the latest advances. But we should build the things that we need to do the safety work and we should try to do the safety work as well as we can on top of models that are reasonably close to state of the art. And we should be a player in the space that sets a good example and we should encourage other players in the space to also set good examples, and we should all work together to try and set positive norms for the field.

Daniela Amodei: I would also just add, I think in addition to industry groups or industry labs, which are the actors that I think get talked about the most, I think there’s a whole swath of other groups that has, I think, a really potentially important role to play in helping to disarm race dynamics or set safety standards in a way that could be really beneficial for the field. And so here, I’m thinking about groups like civil society or NGOs or academic actors or even governmental actors, and in my mind, I think those groups are going to be really important for helping to help us develop safe and not just develop, but develop and deploy safe and more advanced AI systems within a framework that requires compliance with safety.

I think in a thing, I think about a lot is a few jobs ago I worked at Stripe. It was a tech startup then, and even at a very small size. I joined when it was not that much bigger than Anthropic is now. I was so painfully aware every day of just how many checks and balances there were on the company, because we were operating in this highly regulated space of financial services. And financial services, it’s important that’s highly regulated, but it kind of blows my mind that AI, given the potential reach that it could have, is still such a largely unregulated area. Right? If you are an actor who doesn’t want to advance race dynamics, or who wants to do the right thing from a safety perspective, there’s no clear guidelines around how to do that now, right. It’s all sort of, every lab is kind of figuring that out on its own. And I think something I’m hoping to see in the next few years, and I think we will see, is something closer to, in other industries these look like standard setting organizations or industry groups or trade associations that say this is what a safe model looks like, or this is how we might want to move some of our systems towards being safer.

And I really think that without kind of an alliance of all of these different actors, not just in the private sector, but also in the public sphere, we sort of need all those actors working together in order to kind of get to the sort of positive outcomes that I think we’re all hoping for.

Dario Amodei: Yeah. I mean, I think this is generally going to take an ecosystem. I mean, I, yeah, I have a view here that there’s a limited amount that one organization can do. I mean, we don’t describe our mission as solve the safety problem, solve all the problems, solve all the problems with AGI. Our view is just can we attack some specific problems that we think we’re well-suited to solve? Can we be a good player and a good citizen in the ecosystem? And can we help a bit to kind of contribute to these broader questions? But yeah, I think yeah, a lot of these problems are sort of global or relate to coordination and require lots of folks to work together.

Yeah. So I think in addition to the government role that Daniela talked about, which I think there’s a role for measurement, organizations like NIST specialize in kind of measurement and characterization. If one of our worries is kind of the open endedness of these systems and the difficulty of characterizing and measuring things, then there’s a lot of opportunity there. I’d also point to academia. I think something that’s happened in the last few years is a lot of the frontier AI research has moved from academia to industry because it’s so dependent on kind of scaling. But I actually think safety is an area where academia kind of already is but could contribute even more. There’s some safety work that requires or that kind of requires building or having access to large models, which is a lot of what Anthropic is about.

But I think there’s also some safety research that doesn’t. I think there, a subset of the mechanistic interpretability work is the kind of stuff that could be done within academia. Academia really, where it’s strong is development of new methods, development of new techniques. And I think because safety’s kind of a frontier area, there’s more of that to do in safety than there are in other areas. And it may be able to be done without large models or only with limited access to large models. This is an area where I think there’s a lot that academia can do. And so, yeah, I don’t know the hope is between all the actors in the space, maybe we can solve some of these coordination problems, and maybe we can all work together. 

Daniela Amodei: Yeah. I would also say in a paper that we’re, hopefully is forthcoming soon, one thing we actually talk about is the role that government could play in helping to fund some of the kind of academic work that Dario talked about in safety. And I think that’s largely because we’re seeing this trend of training large generative models to just be almost prohibitively expensive, right. And so I think government also has an important role to play in helping to promote and really subsidize safety research in places like academia. And I agree with Dario, safety is such a, AI safety is a really nascent field still, right. It’s maybe only been around, kind of depending on your definition, for somewhere between five and 15 years. And so I think seeing more efforts to kind of support safety research in other areas, I think would be really valuable for the ecosystem.

Dario Amodei: And to be clear, I mean, some of it’s already happening. It’s already happening in academia. It’s already happening in independent nonprofit institutes. And depending on how broad your definition of safety is, I mean, if you broaden it to include some of the short term concerns, then there are many, many people working on it. But I think precisely because it’s such a broad area that there are today’s concerns. They are working on today’s concerns in a way that’s pointed at the future. There’s empirical approaches, there’s conceptual approaches, there’s- yeah. There’s interpretability, there’s alignment, there’s so much to do that I feel like we could always have a wider range of people working on it, people with different mentalities and mindsets.

Lucas Perry: Backing up a little bit here to a kind of simple question. So what is Anthropic’s mission then?

Daniela Amodei: Sure. Yeah, I think we talked about this a little bit earlier, but I think, again, I think the boilerplate mission is build reliable, interpretable, steerable AI systems, have humans at the center of them. And I think that for us right now, that is primarily, we’re doing that through research, we’re doing that through generative model research and AI safety research, but down the road that could also include deployments of various different types.

Lucas Perry: Dario mentioned that it didn’t include solving all of the alignment problems or the other AGI safety stuff. So how does that fit in?

Dario Amodei: I mean, I think what I’m trying to say by that is that there’s very many things to solve. And I think it’s unlikely that one company will solve all of them. I mean, I do think everything that relates to short and long-term AI alignment is in scope for us and is something we’re interested in working on. And I think the more bets we have, the better. This relates to something we could talk about in more detail later on, which is you want as many different orthogonal views on the problem as possible, particularly if you’re trying to build something very reliable. So many different methods and I don’t think we have a view that’s narrower than an empirical focus on safety, but at the same time that problem is so broad that I think what we were trying to say is that it’s unlikely that one company is going to come up with a complete solution or that complete solution is even the right way to think about it.

Daniela Amodei: I would also add sort of to that point, I think one of the things that we do and are sort of hopeful is helpful to the ecosystem as a whole is we publish our safety research and that’s because of this kind of diversification effect that Dario talks about, right. So we have certain strengths in particular areas of safety research because we’re only a certain sized company with certain people with certain skill sets. And our hope is that we will see some of the safety research that we’re doing that’s hopefully helpful to others, also be something that other organizations can kind of pick up and adapt to whatever the area of research is that they’re working on. And so we’re hoping to do research that’s generalizable enough from a safety perspective that it’s also useful in other contexts. 

Lucas Perry: So let’s pivot here into the research strategy, which we’ve already talked a bit about quite a bit, particularly this focus around large models. So could you explain why you’ve chosen large models as something to explore empirically for scaling to higher levels of intelligence and also using it as a place for exploring safety and alignment? 

Dario Amodei: Yeah, so I mean, I think kind of the discussion before this has covered a good deal of it, but I think, yeah, I think some of the key points here are the models are very open ended and so they kind of present this laboratory, right. There are existing problems with these models that we can solve today that are like the problems that we’re going to face tomorrow. There’s this kind of wide scope where the models could act. They’re relatively capable and getting more capable every day. That’s the regime we want to be. Those are the problems we want to solve. That’s the regime we want to be. We want to be attacking.

I think this point about you can see sudden transitions even in today’s model, and that if you’re worried about sudden transitions in future models, if I look on the scaling laws plot from a hundred million parameter model to billion, to 10 billion, to a hundred billion to trillion parameter models that, looking at the first part of the scaling plot, from a hundred million to a hundred billion can tell us a lot about how things might change at the latest part of the scaling laws.

We shouldn’t naively extrapolate and say the past is going to be like the future. But the first things we’ve seen already differ from the later things that we’ve already seen. And so maybe we can make an analogy between the changes that are happening over the scales that we’ve seen, over the scaling that we’ve seen to things that may happen in the future. Models learn to do arithmetic very quickly over one order of magnitude. They learn to comprehend certain kinds of questions. They learn to play actors that aren’t telling the truth, which is something that if they’re small enough, they don’t comprehend.

So can we study both the dynamics of how this happens, how much data it takes to make that happen, what’s going on inside the model mechanistically when that happens and kind of use that as an analogy that equips us well to understand as models scale further and also as their architecture changes, as they become trained in different ways. I’ve talked a lot about scaling up, but I think scaling up isn’t the only thing that’s going to happen. There are going to be changes in how models are trained and we want to make sure that the things that we build have the best chance of being robust to that as well.

Another thing I would say on the research strategy is that it’s good to have several different, I wouldn’t quite put it as several different bets, but it’s good to have several different uncorrelated or orthogonal views on the problem. So if you want to make a system that’s highly reliable, or you want to drive down the chance that some particular bad thing happens, which again could be the bad things that happen with models today or the larger scale things that could happen with models in the future, then a thing that’s very useful is having kind of orthogonal sources of error. Okay, let’s say I have a method that catches 90% of the bad things that models do. That’s great. But a thing that can often happen is then I develop some other methods and if they’re similar enough to the first methods, they all catch the same 90% of bad things. That’s not good because then I think I have all these techniques and yet 10% of the bad things still go through.

What you want is you want a method that catches 90% of the bad things and then you want an orthogonal method that catches a completely uncorrelated 90% of the bad things. And then only 1% of things go through both filters, right, if the two are uncorrelated. It’s only the 10% of the 10% that gets through. And so the more of these orthogonal views you have, the more you can drive down the probability of failure.

You could think of an analogy to self-driving cars where, of course, those things have to be very, very high rate of safety if you want to not have problems. And so, I don’t know very much about self-driving cars, but they’re equipped with visual sensors, they’re equipped with LIDAR, they have different algorithms that they use to detect if something, like there’s a pedestrian that you don’t want to run over or something. And so independent views was on the problem is very important. And so our different directions like reward modeling, reward modeling interpretability, trying to characterize models, adversarial training. I think the whole goal of that is to get down the probability of failure and have different views of the problem. I often refer to it as the P-squared problem, which is, yeah, if you have some method that reduces errors to a probability P, that’s good, but what you really want is P-squared, because then if P is a small number, your errors become very rare.

Lucas Perry: Does Anthropic consider itself as, it’s research strategy, as being a sort of prosaic alignment since it’s focused on large models?

Dario Amodei: Yeah. I think we maybe less think about things in that way. So my understanding is prosaic alignment is kind of alignment with AI systems that kind of look like the systems of today, but I, to some extent that distinction has never been super clear to me because yeah, you can do all kinds of things with neural models or mix neural models with things that are different than neural models. You can mix a large language model with a reasoning system or a system that derives axioms or propositional logic or uses external tools or compiles code or things like that. So I’ve never been quite sure that I understand kind of the boundary of what’s meant by prosaic or systems that are like the systems of today.

Certainly we work on some class of systems that includes the systems of today, but I never know how broad that class is intended to be. I do think it’s possible that in the future, AI systems will look very different from the way that they look today. And I think for some people that drives a view that they want kind of more general approaches to safety or approaches that are more conceptual. I think my perspective on it is it could be the case that systems of the future are very different. But in that case, I think both kind of conceptual thinking and our current empirical thinking will be disadvantaged and will be disadvantaged at least equally. But I kind of suspect that even if the architectures look very different, that the empirical experiments that we do today kind of themselves contain general motifs or patterns that will serve us better than will trying to speculate about what the systems of tomorrow look like.

One way you could put it is like, okay, we’re developing these systems today that have a lot of capabilities that are some subset of what we need to do to fully, to produce something that fully matches human intelligence. Whatever the specific architectures, things we learn about how to align these systems, I suspect that those will carry over and that they’ll carry over more so than sort of the exercise of trying to think well, what could the systems of tomorrow look like? What can we do that’s kind of fully general? I think both things can be valuable, but yeah, I mean, I think we’re just taking a bet on what we think is most exciting, which is that we’ll, by studying the systems of the architectures of today, we’ll learn things that, yeah, stand us to the best chance of what to do if the architectures of tomorrow are very different.

That said, I will say transformer language models and other models, particularly with things like RL or kind of modified interactions on top of them, if construed broadly enough, man, there’s a ever-expanding set of things they can do. And my bet would be that they don’t have to change that much. 

Lucas Perry: So let’s pivot then into a little bit on some of your recent research and papers. So you’ve done major papers on alignment interpretability and societal impact. Some of this you’ve mentioned in passing so far. So could you tell me more about your research and papers that you’ve released? 

Dario Amodei: Yeah. So why don’t we go one by one? So first interpretability. So yeah, I could just start with kind of the philosophy of the area. I mean, I think the basic idea here is, look, these models are getting bigger and more complex. One way to really get a handle on what they might do, if you have a complex system and you don’t know what it’s going to do as it gets more powerful or in a new situation, one way to increase your likelihood of doing that is just to understand the system mechanistically. If you could look inside the model and say hey, this model, it did something bad. It said something racist, it endorsed violence, it said something toxic, it lied to me. Why did it do that? If I’m actually able to look inside the mechanisms of the model and say well, it did it because of this part of the training data or it did it because there’s this circuit that trying to identify X, but misidentified it as Y. Then we’re in a much better position.

And particularly if we understand the mechanisms, we’re in a better position to say if the model was in a new situation where it did something much more powerful, or just if we built more powerful versions of the model, how might they behave in some different way? So, I think mechanistic interpret- lots of folks work on interpretability, but I think a thing that’s more unusual to us is, rather than just, why did the model do a specific thing, try and look inside the model and reverse engineer as much of it as we can. Try and find general patterns. And so the first paper that we came out with was led by Chris Olah who’s been one of the pioneers of interpretability, was focused on how looking at starting with small models, and we have a new paper coming out soon that applies the same thing more approximately to larger models, and tries to reverse engineer as fully as we can these very small models.

So we study one in two layer attention only models, and we’re able to find kind of features or patterns of which the most interesting one is called an induction head. And what an induction head does is it’s a particular arrangement of two what are called attention heads and attention heads are a piece of transformers and transformers are the main architecture that’s used in models for language and other kinds of models. And it’s the two attention heads work together in a way such that when you’re trying to predict something in a sequence, if it’s Mary had a little lamb, Mary had a little lamb, something, something, when you’re at a certain point in the sequence, they look back to something that’s as similar as possible, they look back for clues to things that are similar earlier in the sequence and try to pattern match them.

There’s one attention head that looks back and identifies okay, this is what I should be looking at, and there’s another that’s like okay, this was the previous pattern, and this increases the probability of the thing that’s the closest match to this. And so we can see these very precisely operating in small models and the thesis, which we’re able to offer some support for in the new second paper that’s coming out, is that these are a mechanism for how models match patterns, maybe even how they do what we call in context or few shot learning, which is a capability that models have had since GPT-2 and GPT-3. So yeah, that’s interpretability. Yeah. Do you want me to go on to the next one or you could talk about that? 

Lucas Perry: Sure. So before you move on to the next one, could you also help explain how difficult it is to interpret current models or whether or not it is difficult? 

Dario Amodei: Yeah. I mean, I don’t know, I guess difficult is in the eye of the beholder, and I think Chris Olah can speak to the details of this better than either of us can. But I think kind of watching from the outside and supervising this within Anthropic, I think the experience has generally been that whenever you start looking at some particular phenomenon that you’re trying to interpret, everything looks very difficult to understand. There’s billions of parameters, there’s all these attention heads. What’s going on? Everything that happens could be different. You really have no idea what’s going on. And then there comes some point where there’s some insight or set of insights. And you should ask Chris Olah about exactly how it happens or how he thinks of the right insights that kind of really almost offers a Rosetta stone to some particular phenomenon, often a narrow phenomenon, but these induction heads, they exist everywhere within small models, within large models.

They don’t explain everything. I don’t want to over-hype them, but it’s a pattern that appears again and again and operates in the same way. And once you see something like that, then a whole swath of behavior that didn’t make sense before starts to make some more sense. And of course, there’s exceptions. They’re only approximately true, there are many, many things to be found. But I think the hope in terms of interpreting models, it’s not that we’ll make some giant atlas of what each of the hundred billion weights in a giant model means, but that there will be some lower description length pattern that appears over and over again.

You could make an analogy to the brain or the cell or something like that, where, if you were to just cut up a brain and you’re like, oh my God, this is so complex. I don’t know what’s going on. But then you see that there are neurons and the neurons appear everywhere. They have electrical spikes, they relate to other neurons, they form themselves in certain patterns that those patterns repeat themselves. Some things are idiosyncratic and hard to understand, but also there’s this patterning. And so, I don’t know, it’s maybe an analogy to biology where there’s a lot of complexity, but also there are underlying principles, things like DNA to RNA to proteins, or general intracellular signal regulation. So yeah, the hope is that they’re at least some of these principles and that when we see them, everything gets simpler. But maybe not. We found those in some cases, but maybe as models get more complicated, they get harder to find. And of course, even within existing models, there’s many, many things that we don’t understand at all. 

Lucas Perry: So can we move on then to alignment and societal impact? 

Dario Amodei: Trying to align models by training them and particularly preference modeling, that’s something that several different organizations are working on. There are efforts at DeepMind, OpenAI, Redwood Research, various other places to work on that area. But I think our general perspective on it has been kind of being very method agnostic, and just saying what are all the things we could do to make the models more in line with what would be good. Our general heuristic for it, which isn’t intended to be a precise thing, is helpful, honest, harmless. That’s just kind of a broad direction for what are some things we can do to make models today more in line with what we want them to do, and not things that we all agree are bad.

And so in that paper, we just went through a lot of different ways, tried a bunch of different techniques, often very simple techniques, like just prompting models or training on specific prompts, what we call prompt distillation, building preference models for some particular task or preference models from general answers on the internet. How good did these things do at, yeah, at simple benchmarks for toxicity, helpfulness, harmfulness, and things like that. So it was really just a baseline, like let’s try a collection of all the dumbest stuff we can think of to try and make models more aligned in some general sense. And then I think our future work is going to build on that.

Societal impacts, that paper’s probably going to come out in the next week or so. As I mentioned, it’s called, the paper we’re coming out with is called Predictability and Surprise in Generative Models. And yeah, basically there we’re just making the point about this open-endedness and discussing both technical and policy interventions to try and yeah, to try and grapple with the open-endedness better. And I think future work in the societal impacts direction will focus on how to classify, characterize, and kind of, in a practical sense, filter or prevent these problems.

So, yeah, I mean, I think it’s prototypical of the way we want to engage with policy, which is we want to come up with some kind of technical insight and we want to express that technical insight and explore the implications that it has for, yeah, for policy makers and for the ecosystem in the field. And so here, we’re able to draw a line from hey, there’s this dichotomy where these models scale very smoothly, but have unexpected behavior. The smooth scaling means people are really incentivized to build them and we can see that happening. The unpredictability means even if the case for building them is strong from a financial or accounting perspective, that doesn’t mean we understand their behavior well. That combination is a little disquieting. Therefore we need various policy interventions to make sure that we get a good outcome from these things. And so, yeah, I think societal impacts is going to go in that general direction. 

Lucas Perry: So in terms of the interpretability release, you released alongside that some tools and videos. Could you tell me why you chose to do that? 

Daniela Amodei: Sure. Yeah. I can maybe jump in here. So it goes back sort of to some stuff we talked about a little bit earlier, which is that one of our major goals in addition to doing safety research ourselves, is to sort of help grow the field of safety, all different types of safety work sort of more broadly. And I think we ultimately hope that some of the work that we do is going to be adopted and even expanded on in other organizations. And so we chose to kind of release other things besides just an archive paper, because it hopefully will reach a wider number of people that are interested in these topics and in this case in interpretability. And so what we also released is, our interpretability team worked on something like I think it’s 15 hours worth of videos, and this is just a more in-depth exploration of their research for their paper which is called A Mathematical Framework for Transformer Circuits.

And so the team tried to kind of make it like a lecture series. So if you imagine somebody from the interpretability team is asked to go give a talk at a university or something, maybe they talk for an hour and they reach a hundred students, but now these are publicly available videos. And so if you are interested in understanding interpretability in more detail, you can watch them on YouTube anytime you want. As part of that release, we also put out some tools. So we released a writeup on Garcon, which is the infrastructure tool that our team used to conduct the research, and PySvelte, which is a sample library, which is used to kind of create some of the interactive visualizations that the interpretability team is kind of known for. So we’ve been super encouraged that so we’ve seen other researchers and engineers playing around with the tools and watching the videos. And so we’ve already gotten some great engagement already, and our kind of hope is that this will lead to more people doing interpretability research or kind of building on the work we’ve done in other places. 

Dario Amodei: Yeah. I mean, a way to add to that to kind of put it in broader perspective is different areas within safety are at, I would say, differing levels of maturity. I would say something like alignment or preference modeling or reward modeling or RL from human feedback, they’re all names for the same thing. That’s an area where there are several different efforts at different institutions to do this. We have kind of our own direction within that, but starting from the original RL from Human Preference paper that a few of us helped lead a few years ago, that’s now branched out in several directions. So, we don’t need to tell the field to work in that broad direction. We have our own views about what’s exciting within it, and how to best make progress.

It’s at a slightly more mature stage. Whereas I would say interpretability whereas many folks work on interpretability for neural nets, the particular brand of, let’s try and understand at the circuit level what’s going on inside these models, let’s try and mechanistically kind of map them and break them down. I think there’s less of that in the world and what we’re doing is more unique. And, well, I mean, that’s a good thing because we’re providing a new lens on safety, but actually if it goes on too long, it’s a bad thing because we want these things to spread widely, right. We don’t want it to be dependent on one team or one person. And so when things are at that earlier stage of maturity, it makes a lot of sense to release the tools to reduce the barrier to other people and other institutions starting to work on this. 

Lucas Perry: So you’re suggesting that the, your interpretability research that you guys are doing is unique. 

Dario Amodei: Yeah. I mean, I would just say it’s at an earlier stage, yeah. I would just say that it’s at an earlier stage of maturity. I don’t think there are other kind of large organized efforts that are, yeah, that are kind of focused on, I would say, mechanistic interpretability and especially mechanistic interpretability for language models. We’d like there to be, and there are, we know of folks who are starting to think about it and that’s part of why we released the tools. But I think, yeah, yeah, trying to mechanistically map and understand the internal principles inside large models, particularly language models, I think there’s, yeah, I think there’s less of that has been done in the broader ecosystem. 

Lucas Perry: Yeah. So I don’t really know anything about this space, but I guess I’m surprised to hear that. I imagine that industry with how many large models it’s deploying, like Facebook or other people they’d be interested in, interpretability, interpreting their own systems.

Dario Amodei: Yeah. I mean, I think again, I don’t want to, yeah, yeah, I don’t want to give a misleading impression here. Interpretability is a big field and there’s a lot of effort to like, why did this model do this particular thing? Does this attention head increase this activation by a large amount? People are interested in understanding the particular part of a model that led to a particular output. So there’s a lot of area in this space, but I think the particular program of like, here’s a big language model transformer, let’s try and understand what are the circuits that drive particular behaviors? What are the pieces? How do the MLPs interact with the attention heads? The kind of, yeah, the kind of general mechanistic reverse engineering approach. I think that’s less common. I don’t want to say it doesn’t happen, but it’s less common, much less common.

Lucas Perry: Oh, all right. Okay. So I guess a little bit of a different question and a bit of a pivot here, something to explore. If people couldn’t guess from the title of the podcast, you’re both brother and sister.

Daniela Amodei: Yep.

Lucas Perry: Which is, so it was pretty surprising, I guess, in terms of, I don’t know of any other AGI labs that are largely being run by a brother and sister, so yeah. What’s it like working with your sibling?

Daniela Amodei: Yeah…

Lucas Perry: Do you guys still get along since childhood?

Daniela Amodei: That’s a good question. Yeah. I can maybe start here and obviously I’m curious and hopeful for Dario’s answer. I’m just kidding. But yeah, I think honestly, it’s great. I think maybe a little bit of just history or background about us might be helpful, but Dario and I have always been really close. I think since we were very, very small, we’ve always had this special bond around really wanting to make the world better or wanting to help people. So originally started my career in international development, so very far away from the AI space, and part of why I got interested in that is that it was an interest area of Dario’s at the time, and Dario was getting his PhD in a technical field and so wasn’t working on this stuff directly, but I’m a few years younger than him and so I was very keen to understand the things that he was working or interested in as a potential area to have impact.

And so he was actually a very early GiveWell fan I think in 2007 or 2008, and we-

Lucas Perry: Oh, wow. Cool.

Daniela Amodei: Yeah, and so we were both still students then, but I remember us sitting, we were both home from college, or I was home from college and he was home from grad school and we would sit up late and talk about these ideas, and we both started donating small amounts of money to organizations that were working on global health issues like malaria prevention when we were still both in school. And so I think we’ve always had this uniting, top level goal of wanting to work on something that matters, something that’s important and meaningful, and we’ve always had very different skills and so I think it’s really very cool to be able to combine the things that we are good at into hopefully running an organization well. So for me, I feel like it’s been an awesome experience. Now I feel like I’m sitting here nervously wondering what Dario’s answer is going to be. I’m just kidding. But yeah, for the majority of our lives, I think we’ve wanted to find something to work together on and it’s been really awesome that we’ve been able to at Anthropic.

Dario Amodei: Yeah, I agree with all that. I think what I would add to that is running a company requires an incredibly wide range of skills. If you think of most jobs, it’s like, my job is to get this research result or my job is to be a doctor or something, but I think the unique thing about running a company, and it becomes more and more true the larger and more mature it gets is there’s this just incredibly wide range of things that you have to do, and so you’re responsible for what to do if someone breaks into your office, but you’re also responsible for does the research agenda make sense and if some of the GPUs in the cluster aren’t behaving, someone has to figure out what’s going on at the level of the GPU kernels or the comms protocol that the GPUs talk to each other.

And so I think it’s been great to have two people with complimentary skills to cover that full range. It seems like it’d be very difficult for just one person to cover that whole range, and so we each get to think about what we’re best at and between those two things, hopefully it covers most of what we need to do. And then of course, we always try and hire people fo specialties that we don’t know anything about. But it’s made it a lot easier to move fast without breaking things.

Lucas Perry: That’s awesome. So you guys are like an archon or you guys synergistically are creating an awesome organization.

Dario Amodei: That is what we aim for.

Daniela Amodei: That’s the dream. Yeah, that’s the dream.

Lucas Perry: So I guess beneath all of this, Anthropic has a mission statement and you guys are brother and sister, and you said that you’re both very value aligned. I’m just wondering, underneath all that, you guys said that you were both passionate about helping each other or doing something good for the world. Could you tell me a little bit more about this more heart based inspiration for eventually ending up at and creating Anthropic?

Daniela Amodei: Yeah. Maybe I’ll take a stab at this and I don’t know if this is exactly what you’re looking for, but I’ll gesture in a few different directions here and then I’m sure Dario has a good answer as well, but maybe I’ll just talk about my personal journey in getting to Anthropic or what my background looked like and how I wound up here. So I talked about this in just part of what united me and Dario, but I started my career working in international development. I worked in Washington DC at a few different NGOs, I spent time working in east Africa for a public health organization, I worked on a congressional campaign, I’ve worked on Capitol Hill, so I was much more in this classic, like a friend at an old job used to call me, the classic do-gooder. Of trying to alleviate global poverty, of trying to make policy level changes in government, of trying to elect good officials.

And I felt those causes that I was working in were deeply important, and really, to this day, I really support people that are working in those areas and I think they matter so much. And I just felt I personally wasn’t having the level of impact that I was looking for, and I think that led me to through a series of steps. I wound up working in tech, and I mentioned this earlier but I started at this tech startup called Stripe. It was about 40 people when I joined and I really had the opportunity to see what it looks like to run a really well run organization when I was there. And I got to watch it scale and grow and be in this emerging area. And I think during my time there, something that became really apparent to me was just working in tech, how much of an impact this sector has on things like the economy, on human interaction, on how we live our lives in day to day ways. And Stripe, it’s a payments company, it’s not social media or something like that.

But I think there is a way that technology is a relatively small number of people having a very high impact in the world per person working on it. And I think that impact can be good or bad, and I think it was a pretty logical leap for me from there to think, wow, what would happen if we extrapolated that out to instead of it being social media or payments or file storage, to something significantly more powerful where there’s a highly advanced set of artificial intelligence systems. What would that look like and who’s working on this? So I think for me, I’ve always been someone who has been fairly obsessed with trying to do as much good as I personally can, given the constraints of what my skills are and where I can add value in the world.

And so I think for me, moving to work into AI looked… From early days, if you looked at my resume, you’d be like, how did you wind up here? But I think there was this consistent story or theme. And my hope is that Anthropic is at the intersection of this practical, scientific, empirical approach to really deeply understanding how these systems work, hopefully helping to spread and propagate some of that information more widely in the field, and to just help as much as possible to push this field in a safer and ideally, just hopefully all around robust, positive direction when it comes to what impact we might see from AI.

Dario Amodei: Yeah. I think I have a parallel picture here, which is I did physics as an undergrad, I did computational neuroscience in grad school. I was, I think, drawn to neuroscience by a mixture of, one, just wanting to understand how intelligence works, seems the fundamental thing. And a lot of the things that shape the quality of human life and human experience depend on the details of how things are implemented in the brain. And so I felt in that field, there were many opportunities for medical interventions that could improve the quality of human life, understanding things like mental illness and disease, while at the same time, understanding something about how intelligence works, because it’s the most powerful lever that we have.

I thought of going into AI during those days, but I felt that it wasn’t really working. This was before the days when deep learning was really working. And then around 2012 or 2013, I saw the results coming out of Google Brain, things like AlexNet and that they were really working, and saw AI both as, hey, this might be, one, the best way to understand intelligence, and two, the things that we can build with AI, by solving problems in science and health and just solving problems that humans can’t solve yet by having intelligence that, first in targeted ways and then maybe in more general ways, matches and exceeds those of humans, can we solve the important scientific, technological, health, societal problems? Can we do something to ameliorate those problems? And AI seemed like the biggest lever that we had if it really worked well. But on the other hand, AI itself has all these concerns associated with it in both the short run and the long run. So we maybe think of it as we’re working to address the concerns so that we can maximize the positive benefits of AI.

Lucas Perry: Yeah. Thanks a lot for sharing both of your perspectives and journeys on that. I think when you guys were giving to GiveWell I was in middle school, so…

Daniela Amodei: Oh, God. We’re so old, Dario.

Dario Amodei: Yeah, I still think of GiveWell as this new organization that’s on the internet somewhere and no one knows anything about it, and just me who-

Daniela Amodei: This super popular, well known-

Dario Amodei: Just me who reads weird things on the internet who knows about it.

Daniela Amodei: Yeah.

Lucas Perry: Well, for me, a lot of my journey into x-risk and through FLI has also involved the EA community, effective altruism. So I guess that just makes me realize that when I was in middle school, there was the seeds that were…

Dario Amodei: Yeah, there was no such community at that time.

Daniela Amodei: Yeah.

Lucas Perry: Let’s pivot here then into a bit more of the machine learning, and so let see what the best way to ask this might be. So we’ve talked a bunch already about how Anthropic is emphasizing the scaling of machine learning systems through compute and data, and also bringing a lot of mindfulness and work around alignment and safety when working on these large scale systems that are being scaled up. Some critiques of this approach have described scaling from existing models to AGI as adding more rocket fuel to a rocket, which doesn’t mean you’re necessarily ready or prepared to land the rocket on the moon, or that the rocket is aimed at the moon.

Maybe this is lending itself to what you guys talked about earlier about the open-endedness of the system, which is something you’re interested in working on. So how might you respond to the contention that there is an upward bound on how much capability can be gained through scaling? And then I’ll follow up with the second question after that.

Dario Amodei: Yeah, so actually in a certain sense, I think we agree with that contention in a certain way. So I think there’s two versions of what you might call the scaling hypothesis. One version, which I think of as the straw version or less sophisticated version, which we don’t hold and I don’t know if there’s anyone who does hold it but probably there is, is just the view that we have our 10 billion parameter language model, we have a hundred billion parameter language model. Maybe if we make a hundred trillion parameter language model, that’ll be AGI. So that would be a pure scaling view. That is definitely not our view. Even small modified forms like, well, maybe you’ll change the activation function in the transformer you don’t have to do anything other than that. I think that’s just not right.

And you can see it just by seeing that the objective function is predicting the next word, it’s not doing useful tasks that humans do. It’s limited to language, it’s limited to one modality. And so there are some very trivial, easy to come up with ways in which literally just scaling this is not going to get you to general intelligence. That said, the more subtle version of the hypothesis, which I think we do mostly hold, is that this is a huge ingredient of not only this, of whatever it is that actually does build AGI. So no one thinks that you’re just going to scale up the language models and make them bigger, but as you do that, they’ll certainly get better. It’ll be easier to build other things on top of them.

So for example, if you start to say, well, you make this big language model and then you used RL with interaction with humans, to fine tune it on doing a million different tasks and following human instructions, then you’re starting to get to something that has more agency, that you can point it in different directions, you can align it. If you also add multi-modality where the agent can interact with different modalities, if you add the ability to use various external tools to interact with the world and the internet. But within each of these, you’re going to want to scale, and within each setup, the bigger you make the model, the better it’s going to be at that thing.

So in a way, the rocket fuel analogy makes sense. Actually, the thing you should most worry about with rockets is propulsion. You need a big enough engine and you need enough rocket fuel to make the rocket go. That’s the central thing. But of course, yes, you also need guidance systems, you also need all kinds of things. You can’t just take a big vat of rocket fuel and an engine and put them on a launchpad and expect it to all work. You need to actually build the full rocket. And safety itself makes that point, that to some extent, if you don’t do even the simplest safety stuff, then models don’t even do the task that’s intended for them in the simplest way. And then there’s many more subtle safety problems.

But in a way, the rocket analogy is good, but it’s I think more a pro scaling point than an anti scaling point because it says that scaling is an ingredient, perhaps a central ingredient in everything. Even though it isn’t the only ingredient, if you’re missing ingredients, you won’t get where you’re going, but when you add all the right ingredients, then that itself needs to be massively scaled. So that would be the perspective.

No one thinks that if you just take a bunch of rocket fuel in an engine and put it on a launch pad that you’ll get a rocket that’ll go to the moon, but those might still be the central ingredients in the rocket. Propulsion and getting out of the Earth’s gravity well is the most important thing a rocket has to do. What you need for that is rocket fuel and an engine. Now you need to connect them to the right things, you need other ingredients, but I think it’s actually a very good analogy to scaling in the sense that you can think of scaling as maybe the core ingredient, but it’s not the only ingredient.

And so what I expect is that we’ll come up with new methods and modifications. I think RL, model based URL, human interaction, broad environments are all pieces of this, but that when we have those ingredients, then whatever it is we make, we’ll need to scale that multi-modality, we’ll need to scale that massively as well. So scaling is the core ingredient, but it’s not the only ingredient. I think it’s very powerful alone, I think it’s even more powerful when it’s combined with these other things.

Lucas Perry: One of the claims that you made was that we won’t get to AGI, people don’t think we won’t get to AGI just by scaling up present day systems. Earlier, you were talking about how we got… There these phase transitions, right? If you go up one order of magnitude in terms of the number or parameters in the system, then you get some kind of new ability, like arithmetic. Why is it that we couldn’t just increase the order of magnitude of the number of parameters in the systems and just keep getting something that’s smarter?

Dario Amodei: Yeah. So first of all, I think we will keep getting something that’s smarter, but I think the question is will we get all the way to general intelligence? So I actually don’t exclude it, I think it’s possible, but I think it’s unlikely, or at least unlikely in the practical sense. There are a couple of reasons. Today, when we train models on the internet, we train them on an average overall text on the internet. Think of some topic like chess. You’re training on the commentary of everyone who talks about chess. You’re not training on the commentary of the world champion at chess. So what we’d really like is something that exceeds the capabilities of the most expert humans, whereas if you train on all the internet, for any topic, you’re probably getting amateurs on that topic. You’re getting some experts but you’re getting mostly amateurs.

And so even if the generative model was doing a perfect job of modeling its distribution, I don’t think it would get to something that’s better than humans at everything that’s being done. And so I think that’s one issue. The other issue is, or there’s several issues, I don’t think you’re covering all the tasks that humans do. You cover a lot of them on the internet but there are just some tasks and skills, particularly related to the physical world that aren’t covered if you just scrape the internet, things like embodiment and interaction.

And then finally, I think that even matching the performance of text on the internet, it might be that you need a really huge model to cover everything and match the distribution, and some parts of the distribution are more important than others. For instance, if you’re writing code or if you’re writing a mystery novel, a few words or a few things can be more important than everything else. It’s possible to write a 10 page document where the key parts are two or three sentences, and if you change a few words, then it changes the meaning and the value of what’s produced. But the next word prediction objective function doesn’t know anything about that. It just does everything uniformly so if you make a model big enough, yeah they’ll get that right but the limit might be extreme. And so things that change the objective function, that tell you what to care about, of which I think RL is a big example probably are needed to make this actually work correctly.

I think in the limit of a huge enough model, you might get surprisingly close, I don’t know, but the limit might be far beyond our capabilities. There’s only so many GPU’s you can build and there are even physical limits.

Lucas Perry: And there’s less of them, less and less of them available over time, or at least they’re very expensive.

Dario Amodei: They’re getting more expensive and more powerful. I think the price efficiency overall is improving, but yeah, they’re definitely becoming more expensive as well.

Lucas Perry: If you were able to scale up a large scale system in order to achieve an amateur level of mathematics or computer science, then would it not benefit the growth of that system to then direct that capability on itself as a self recursive improvement process? Is that not already escape velocity intelligence once you hit amateurs?

Dario Amodei: Yeah. So there are training techniques that you can think of as bootstrapping a model or using the model’s own capabilities to train it. Think like AlphaGo for instance was trained with a method called expert iteration that relies on looking ahead and comparing that to the model’s own prediction. So whenever you have some coherent logical system, you can do this bootstrapping, but that itself is a method of training and falls into one of the things I’m talking about, about you make these pure generative models, but then you need to do something on top of them, and the bootstrapping is something that you can do on top of them. Now, maybe you reach a point where the system is making its own decisions and is using its own external tools to create the bootstrapping, to make better versions of itself, so it could be that that is someday the end of this process. But that’s not something we can do right now.

Lucas Perry: So there’s a lot of labs in industry who work on large models. There are maybe only a few other AGI labs, I can think of DeepMind. I’m not sure if there are others that… OpenAI. And there’s also this space of organizations like The Future of Life Institute or the Machine Intelligence Research Institute or the Future of Humanity Institute that are interested in AI safety. MIRI and FHI both do research. FLI does grant making and supports research. So I’m curious as to, both in terms of industry and nonprofit space and academia, how you guys see Anthropic as positioned? Maybe we can start with you, Daniela.

Daniela Amodei: Sure, yeah. I think we touched on this a little bit earlier, but I really think of this as an ecosystem, and I think Anthropic is in an interesting place in the ecosystem, but we are part of the ecosystem. So I think our strength or the thing that we do best, and I like to think of all of these different organizations as having valuable things to bring to the table, depending on the people that work there, their leadership team, their particular focused research bet, or their mission and vision that they’re achieving I think hopefully have the potential to bring safe innovations to the broader ecosystem that we’ve talked about. I think for us, our bet is one we’ve talked about, which is this empirical scientific approach to doing AI research and AI safety research in particular.

And I think for our safety research, we’ve talked about a lot of the different areas we focus on. Interpretability, alignment, societal impacts, scaling laws for empirical predictions. And I think a lot of what we’re imagining or hoping for in the future is that we’ll be able to grow those areas and potentially expand into others, and so I really think a lot of what Anthropic adds to this ecosystem or what we hope it adds is this rigorous scientific approach to doing fundamental research in AI safety.

Dario Amodei: Yeah, that really captures it in one sentence, which is I think if you want to locate us within the ecosystem, it’s an empirical iterative approach within an organization that is completely focused on making a focused bet on the safety thing. So there are organizations like MIRI or to a lesser extent, Redwood, that are either not empirical or have a different relationship to empiricism than we do, and then there are safety teams that are doing good work within larger companies like DeepMind or OpenAI or Google Brain that are safety teams within larger organizations. Then I have lots of folks who work on short term issues, and then we’re filling a space that’s working on today’s issues but with an eye towards the future, empirically minded, iterative, with an org where everything we do is designed for the safety objective.

Lucas Perry: So one facet of Anthropic is that it is a public benefit corporation, which is a structure that I’m not exactly sure what it is and maybe many of our listeners are not familiar with what a public benefit corporation is. So can you describe what that means for Anthropic, its work, its investors and its trajectory as a company?

Daniela Amodei: Yeah, sure. So this is a great question. So what is a PBC? Why did we choose to be a public benefit corporation? So I think I’ll start by saying we did quite a lot of research when we were considering what type of corporate entity we wanted to be when we were founding. And ultimately, we decided on PBC, on public benefit corporation for a few reasons. And I think primarily, it allowed us the maximum amount of flexibility in how we can structure the organization, and we were actually very lucky, to a later part of your question, to find both investors and employees who were generally very on board with this general vision for the company. And so what is a public benefit corporation? Why did we choose that structure?

So they’re fairly similar to C corporations, which is any form of standard corporate entity that you would encounter. And what that means is we can choose to focus on research and development, which is what we’re doing now, or on deployment of tools or products, including down the road for revenue purposes if we want to. But the major difference between a PBC and a C corporation is that in a public benefit corporation, we have more legal protections from shareholders if the company fails to maximize financial interests in favor of achieving our publicly beneficial mission. And so this is primarily a legal thing, but it also was very valuable for us in being able to just appropriately set expectations for investors and employees, that if financial profit and creating positive benefit for the world were ever to come into conflict, it was legally in place that the latter one would win.

And again, we were really lucky that investors, people that wanted to work for us, they said, wow, this is actually something that’s a really positive thing about Anthropic and not something that we need to work around. But I think it ended up just being the best overall fit for what we were aiming for.

Lucas Perry: So usually, there’s a fiduciary responsibility that people like Anthropic would have to its shareholders, and because it’s structured as a public benefit corporation, the public good can outweigh the fiduciary responsibility without there being legal repercussions. Is that right?

Daniela Amodei: Yeah, exactly. So shareholders can’t come sue the company and say, hey, you didn’t maximize financial returns for us. If those financial returns were to come into conflict with the publicly beneficial value of the company. So I think maybe an example here, I’ll try and think of one off the top of my head, but if we designed a language model and we felt like it was unsafe, it was producing outputs that we felt were not in line with what we wanted to see from outputs of a language model, for safety reasons or toxicity reasons for any number of reasons. And in a normal C corporation, someone could say, “Hey, we’re a shareholder and we want the financial value that you could create from that by productizing it.” But we said, “Actually, we want to do more safety research on it before we choose to put it out into the world,” in a PBC, we’re quite legally protected basically in a case that. And again, I’m not a lawyer but that’s my understanding of the PBC.

Dario Amodei: Yeah. A useful, holistic way to think about it is there’s the legal structure, but I think often, these things, maybe the more important thing about them is that they’re a way to explain your intention, to set the expectations for how the organization is going to operate. Often, things like that and the expectations of the various stakeholders, and making sure that you give the correct expectations and then deliver on those expectations so no one is surprised by what you’re doing and all the relevant stakeholders, the investors, the employees, the outside world gets what they expect from you, that can often be the most important thing here. And so I think what we’re trying to signal here is on one hand, a public benefit corporation, it is a for-profit corporation.

We could deploy something. That is something that we may choose to do and it has a lot of benefits in terms of learning how to make models more effective, in terms of iterating. But on the other hand, the mission is really important to us and we recognize that this is an unusual area, that’s more fraught with market externalities would be the term that I would use, of all kinds. In the short term, in the long term, related to alignment, related to policy and government than a typical area. It’s different than making electric cars or making widgets or something that, and so that’s the thing we’re trying to signal.

Lucas Perry: What do you think that this structure potentially means for the commercialization of Anthropic’s research?

Daniela Amodei: Yeah, I think again, part of what’s valuable about a public benefit corporation is that it’s flexible, and so it is a C corporation, it’s fairly close to any standard corporate entity you would meet and so the structure doesn’t really have much of a bearing outside of the one that we just talked about on decisions related to things like productization, deployment, revenue generation.

Lucas Perry: Dario, you were just talking about how this is different than making widgets or electric cars, and one way that it’s different from widgets is that it might lead to massive economic windfalls.

Dario Amodei: Yeah.

Lucas Perry: Unless you make really good widgets or widgets that can solve problems in the world. So what is Anthropic’s view on the vast economic benefits that can come from powerful AI systems? And what role is it that you see C company AGI labs playing in the beneficial use of that windfall?

Dario Amodei: Daniela, you want to go…

Daniela Amodei: Go for it.

Dario Amodei: Yeah. So yeah, I think a way to think about it is, assuming we can avoid the alignment problems and some other problems, then there will be massive economic benefits from AI or AGI or TAI or whatever you want to call it, or just AI getting more powerful over time.

And then again, thinking about all the other problems that I haven’t listed, which is today’s short term problems and problems with fairness and bias, and long-term alignment problems and problems that you might encounter with policy and geopolitics. Assuming we address all those, then there is still this issue of economic… Like are those benefits evenly distributed?

And so here, as elsewhere, I think it’s unlikely those benefits will all accrue to one company or organization. I think this is bigger than one company or one organization, and is a broader societal problem. But we’d certainly like to do our part on this and this is something we’ve been thinking about and are working on putting programs in place with respect to. We don’t have anything to share about it at this time, but this is something that’s very much on our mind.

I would say that, more broadly, I think the economic distribution of benefits is maybe one of only many issues that will come up. Which is the disruptions to society that you can imagine coming from the advent of more powerful intelligence are not just economic. They’re already causing disruptions today. People already have legitimate and very severe societal concerns about things that models are doing today and you can call them mundane relative to all the existential risk. But I think they’re already serious concerns about concentration of power, fairness and bias in these models, making sure that they benefit everyone, which I don’t think that they do yet.

And if we then put together with that, the ingredient of the models getting more powerful, maybe even on an exponential curve, those things are set to get worse without intervention. And I think economics is only one dimension of that. So, again, these are bigger than any one company. I don’t think it’s within our power to fix them, but we should do our part to be good citizens and we should try and release applications that make these problems better rather than worse.

Lucas Perry: Yeah. That’s excellently put. I guess one thing I’d be interested in is if you could, I guess, give some more examples about these problems that exist with current day systems and then the real relationship that they have to issues with economic windfall and also existential risk.

I think it seems to me like tying these things together is really important. At least seeing the interdependence and relationship there, some of these problems already exist, or we already have example problems that are really important to address. So could you expand on that a bit?

Dario Amodei: I think maybe the most obvious one for current day problems is people are worried, very legitimately, that big models suffer from problems of bias, fairness, toxicity, and accuracy. I’d like to apply my model in some medical application and it gives the wrong diagnosis, or it gives me misinformation or it fabricates information. That’s just not good. These models aren’t usable and they’re harmful if you try and use them.

I think toxicity and bias are issues when models are trained on data from the internet. They absorb the biases of that data. And there’s maybe even more subtle algorithmic versions of that, where, I hinted at it a little before, where it’s like the objective function of the model is to say something it sounds like what a human would say or what a human on the internet would say. And so in a way, almost fabrication is kind of like baked into the objective function.

Potentially, even bias and stereotyping you can imagine being baked into the objective function in some way. So, these models want to be used for very mundane everyday things like helping people write emails or helping with customer surveys or collecting customer data. And if they’re subtly biased or subtly inaccurate, then those biases and those inaccuracies will be inserted into the stream of economic activity in a way that may be difficult to detect. So, that seems bad and I think we should try to solve those problems before we deploy the models. But also they’re not as different from the large scale problems as they might seem.

In terms of the economic inequality, I don’t know, just look at the market capitalization of the top five tech companies in the world. And compare that to the US economy. There’s clearly something going on in the concentration of wealth.

Daniela Amodei: I would just echo everything Dario said. And also add, I think something that especially can be alarming in sort of a short term way today in the sense that it could belie things to come, is how quietly and seamlessly people are becoming dependent on some of these systems. We don’t necessarily even know, there’s no required disclosure of when you’re interacting with an AI system versus a human and until very recently, that was sort of a comical idea because it was so obvious when you were interacting with a person versus not a person. You know when you’re on a customer chat and it’s a human on the other end versus an automated system responding to you.

But I think that line is getting increasingly blurred. And I can imagine that even just in the next few years, that could start to have fairly reasonably large ramifications for people in day-to-day ways. People talk to an online therapist now, and sometimes that is backed by an AI system that is giving advice. Or down the road, we could imagine things looking completely different in health realms, like Dario talked about.

And so I think it’s just really important as we’re stepping into this new world to be really thoughtful about a lot of the safety problems that he just outlined and talked about because I think, I don’t know that most people necessarily even know all the ways in which AI is impacting our kind of day-to-day lives today, and the potential that could really go up in the near future.

Lucas Perry: The idea of AIs, there being like a requirement of AI is disclosing themselves as AI seems very interesting and also adjacent to this idea of the way that C corporations have fiduciary responsibility to shareholders, having AI systems that also have some kinds of responsibility towards the people that they serve, where they can’t be secretly working towards the interests of the tech company that has the AI listening to you in your house all the time.

Dario Amodei: Yeah. It’s another direction you can imagine. It’s like I talked to an AI produced by Megacorp but it subtly steers to my life to the benefit of Megacorp. Yeah, there’s lots of things you can come up with like this.

Daniela Amodei: These are important problems today. And I think they also really belie things that could be coming in the near future, and I think solving whatever, those particular problems are ones lots of groups are working on, but I think helping to solve a lot of the fundamental building blocks underlying them; about getting models to be truthful, to be harmless, to be honest. A lot of the goals are aligned there, both for sort of short, medium and potentially long-term safety.

Lucas Perry: So Dario, you mentioned earlier that of the research that you publish, one of your hopes is that other organizations will look into and expand the research that you’re doing. I’m curious if Anthropic has a plan to communicate its work and its ideas about how to develop AGI safely with both technical safety researchers, as well as with policy makers.

Daniela Amodei: Yeah, maybe I’ll actually jump in on this one, and Dario feel free to add as much as you like. But I actually think this is a really important question. I think communication with policy makers about safety with other labs in the form of papers that we publish is something that’s very important to us at Anthropic.

We have a policy team, it’s like 1.5 people right now. So we’re hiring, that’s kind of a plug as well, but I think their goal is to really take the technical content that we are developing at Anthropic and translate that into something that is actionable and practical for policymakers. And I think this is really important because the concepts are very complex, and so it’s a special skill to be able to take things that are highly technical, potentially very important, and translate that into recommendations or work with policy makers to come up with recommendations that could potentially have very far reaching consequences.

So, to point to a couple of things we’ve been working on here, we’ve been supporting NIST, which is the National Institute for Standards and Technology on developing something called an AI Risk Management Framework. And the goal of that is really developing more monitoring tools around AI risk and AI risk management. We’ve also been supporting efforts in the US and internationally to think about how we can best support academic experimentation, which we talked about a little bit earlier with large scale compute models too.

Lucas Perry: You guys also talked a lot about open-endedness, and was part of all this alignment and safety research looking into ways of measuring safety and open-endedness?

Daniela Amodei: Yeah, there’s actually some interesting work which I think is also in this upcoming paper and in various other places that we’ve been looking into around the concept of AI evaluations or AI monitoring. And I think both of those are potentially really important because a lot of what we’re seeing, or maybe lacking, and this kind of goes back to this point I made earlier about standards is, how do we even have a common language or a common framework within the AI field of what outputs or metrics we care about measuring.

And until we have that common language or framework, it’s hard to set things like standards across the industry around what safety even means. And so, I think AI evaluations is another area that our societal impacts team, which is also like the other half of the one and a half people in policy, it’s also 1.5 people, is something that they’ve been working on as well.

Lucas Perry: Right, so a large part of this safety problem is of course the technical aspect of how you train systems and create systems that are safe and aligned with human preferences and values. How do you guys view and see the larger problem of AI governance and the role and importance of governments and civil society in working towards the safe and beneficial use and deployment of AI systems?

Daniela Amodei: We talked about this one a little bit earlier, and maybe I’ll start here. And obviously, Dario jump in if you want. But I do think that these other kind of institutions that you talked about have this really important role to play. And again, one of the things we mention in this paper is that we think government has already been starting to fund a lot more academic safety research. And I think that’s an area that we… A concrete policy recommendation is, hey, go do more of that. That would be great.

But I also think groups like civil society and NGOs, there’s a lot of great organizations in this space, including FLI and others, that are thinking about what do we do? Say we develop something really powerful, what’s the next step? Whether that’s at an industry lab, in government, in academia, wherever. And I think there’s a way that industry incentives are not the same as nonprofit groups or as civil society groups. And I think to go back to this analogy of an ecosystem, we really need thoughtful and empowered organizations that are working on these kinds of questions, fundamentally outside of the industry sphere, in addition to the policy research and work that’s being done at labs.

Dario Amodei: Yeah, another way you can think of things in line with this is I think maybe at some point laws and regulations are going to be written. And I think probably those laws and regulations work best if they end up being formalizations of what’s realized to be the best practices, and those best practices can come from different industrial players, they can come from academics figuring out what’s good and what’s not. They can come from nonprofit players. But if you try and write a law ahead of time, often you don’t know what… If you write a law that relates to a technology that hasn’t been invented yet, it’s often not clear what the best thing to do is, and what is actually going to work or make sense, or even what categories or words to use.

But if something has become a best practice and folks have converged on that, and then the law formalizes it and puts it in place, that can often be a very constructive way for things to happen.

Lucas Perry: Anthropic has received an impressive amount of series A funding. And so it seems like you guys are doing a lot of hiring and growing considerably. So, in case there’s anyone from our audience that’s interested in joining Anthropic, what are the types of roles that you expect to be hiring for?

Daniela Amodei: Yes, great question. We are definitely hiring. We’re hiring a lot. And so I think the number one thing I would say is if you’re listening to this podcast and you’re interested, I would highly recommend just checking out our jobs page, because that will be the most up to date. And that’s just anthropic.com on the careers tab. But we can also send that around if that’s helpful.

But what are we looking to hire? Quite a few things. So most critically, probably right now, we’re looking to hire engineers and we’re actually very bottle-necked on engineering talent right now. And that’s because running experiments on AI systems is something that requires a lot of custom software and tooling. And while machine learning experience is helpful for that, it isn’t necessarily required.

And I think a lot of our best ML engineers or research engineers came from a software engineering or infrastructure engineering background, hadn’t necessarily worked in ML before, but were just really excited to learn. So, I think if that describes you, if you’re a software engineer, but you’re really interested in these topics, definitely think about applying because I think there’s a lot of value that your skills can provide.

We’re also looking for just a number of other roles. I won’t be able to list them all, you should just check out our jobs page. But off the top of my head, we’re looking for front-end engineers to help with things like interfaces and tooling for the research we’re doing internally. We’re looking for policy experts, operations people, security engineers, data visualization people, security.

Dario Amodei: Security.

Daniela Amodei: Security, yes. We’re definitely looking-

Dario Amodei: If you’re building big models.

Daniela Amodei: Yes. Security is something that I think is-

Dario Amodei: Every industrial lab should make sure their models are not stolen by bad actors.

Daniela Amodei: This is a unanimous kind of thing across all labs. There’s something everyone really agrees on in industry and outside of industry, which is that security is really important. And so, if you are interested in security or you have a security background, we would definitely love to hear from you, or I’m sure our friends at other industry labs and non-industry labs would also love to hear from you.

I would also say, I sort of talked about this a little bit before, but we’ve also just kind of had a lot of success in hiring people who were very accomplished in other fields, especially other technical fields. And so, we’ve alluded a few times to former recovering physicists or people who have PhDs in computer science or ML, neuroscientists, computational biologists.

And so, I think if you are someone who has this strong background and set of interest in a technical field that’s not related to ML, but sort of moderately adjacent, I would also consider applying for our residency program. And so I think again, if you’re even a little curious, I would say, just check out our jobs page, because there’s going to be more information there, but those are the ones off the top of my head. And Dario, if I missed any, please jump in.

Dario Amodei: Yeah, that covers a pretty wide range.

Lucas Perry: Could you tell me a little bit more about the team and what it’s like working at Anthropic?

Daniela Amodei: Yeah, definitely. You’ll probably have to cut me off here because I’ll talk forever about this because I think Anthropic is a great team. Some basic stats, we’re about 35 people now. Like I said a few times, we’ve kind of come from a really wide range of backgrounds. So this is people who worked in tech companies as software engineers. These are former academics in physics, ethics, neuroscience, a lot of different areas, machine learning researchers, policy people, operations staff, so much more.

And I think one of the unifying themes that I would point to in our employees is a combination of a set of two impulses that I think we’ve talked about a lot in this podcast. And I think the first is really just a genuine desire to reduce the risks and increase the potential benefits from AI. And I think the second is a deep curiosity to really scientifically and empirically describe, understand, predict, model-out how AI systems work and through that deeper understanding, make them safer and more reliable.

And I think some of our employees identify as effective altruists which means they’re especially worried about the potential for long term harms from AI. And I think others are more concerned about immediate or sort of emerging risks that are happening today or in the near future. And I think both of those views are very compatible with the goals that I just talked about. And I think they often just call for a mixed-method approach to research, which I think is a very accurate description of how things look in a day-to-day way at Anthropic.

It’s a very collaborative environment. So, there’s not a very strong distinction between research and engineering, researchers write code, engineers contribute to research. There’s a very strong culture of pair programming across and within teams. There’s a very strong focus on learning. I think this is also just because so many of us come from backgrounds that were not necessarily ML focused in where we started.

So people run these very nice, little training courses. Where they’ll say, “Hey, if you’re interested in learning more about transformers, I’m a transformer’s expert and I’ll walk you through it at different levels of technical skills so that people from the operations team or the policy team can come for an introductory version.”

And then I think outside of that, I like to think we’re a nice group of people. We all have lunch together every day. We have this very lovely office space in San Francisco, it’s fairly well attended. And I think we have lots of fun lunch conversations ranging from things like… A recent one was we were sort of talking about microCOVID, if you know the concept of microCOVID, Catherine Olsson, who’s of one of the creators of microcovid.org. Which is basically a way of assessing the level of risk from a given interaction or a given activity that you’re doing during COVID time.

So we had this fun meta conversation where we’re like, “How risky is this conversation that we’re having right now from a microCOVID perspective, if we all came into the office and tested, but we’re still together indoors and there’s 15 of us, what does that mean?” So anyway, I think it’s a fun place to work. We’ve obviously had a lot of fun getting to build it together.

Dario Amodei: Yeah. The things that stand out to me are trust and common purpose. They’re enormous force multipliers where it shows up in all kinds of little things where if you have… You can think about it in things like compute allocation. If people are not on the same page, if one person wants to advance one research agenda, the other wants to advance their other research agenda, then people fight over it. And there’s a lot of zero sum or negative sum interactions.

But if everyone has the attitude of, we’re trying to do this thing, everything we’re trying to do is in line with this common purpose and we all trust each other to do what’s right to advance this common purpose, then it really becomes a force multiplier on getting things done while keeping the environment comfortable, and while everyone continues to get along with each other. I think it’s an enormous superpower that I haven’t seen before.

Lucas Perry: So, you mentioned that you’re hiring a lot of technical people from a wide variety of technical backgrounds. Could you tell me a little bit more about your choice to do that rather than simply hiring people who are traditionally experienced in ML and AI?

Daniela Amodei: Yeah, that’s a great question. So I should also say we have people from both camps that you talked about, but why did we choose to bring people in from outside the field? I think there’s a few reasons for this. I think one is, again, ML and AI is still a fairly new field. Not super new, but still pretty new. And so what that means is there’s a lot of opportunity for people who have not necessarily worked in this field before to get into it. And I think we’ve had a lot of success or luck with taking people who are really talented in a related field and helping to take their skills and translate them to the ones in ML and AI safety.

And I think the second reason is, so one is just expanding the talent pool. I think the other is, it really does broaden the range of perspectives and the types of people who are working on these issues, which we think are very important. And again, we’ve talked about this previously, but having a wider range of views and perspectives and approaches tends to lead to a more robust approach to doing both basic research and safety research.

Dario Amodei: Yeah. Nothing to add to that. I’m surprised at how often someone who has experience in a different field can come in, and it’s not like they’re directly applying things that come, but they think about things in a different way. And of course this is true about all kinds of things, this is this true about diversity in the more traditional senses as well. But you want as many different kinds of people as you can get. 

Lucas Perry: So as we’re wrapping up here, I’m curious just to get some more perspective on you guys about, given these large scale models, the importance of safety and alignment and the problems which exist today, but also the promises of the impact they could have for the benefit of people. What’s a future that each of you is excited about or what’s a future that you’re hopeful for? Given your work at Anthropic and the future impacts of AI?

Daniela Amodei: Yeah, I’ll start. So I think one thing I do believe is actually I am really hopeful about the future. I know that there’s a lot of challenges that we have to face to get to a potentially really positive place. But I think the field will rise to the occasion, or that’s kind of my hope. And I think some things I’m hoping for in the next few years is that a lot of different groups will be developing more practical tools, techniques for advancing safety research. And I think these are likely to hopefully become more widely available if we can set the right norms in the community. And I think the more people working on safety-related topics, that can positively feed on itself.

And I think I’m most broadly hoping for a world where we can feel confident that when we’re using AI for more advanced purposes, like accelerating scientific research, that it’s behaving in ways where we can be very confident and sure that we understand that it’s not going to lead to negative, unintended consequences.

And the reason for that is because we’ve really taken the time to chart them out and understand what all of those potential problems could be. And so I think that’s obviously a very ambitious goal, but I think if we can make all of that happen, there’s a lot of potential benefits of more advanced AI systems that I think could be transformative for the world, from almost anything you can name; renewable energy, health, disease detection, economic growth, and lots of other just day-to-day enhancements to how we work and communicate and live together.

Dario Amodei: No one really knows what’s going to happen in the future. It’s extremely hard to predict. And so I often find any question about the future, it’s more about the attitude or posture that you want to take than it is about concrete predictions, because I feel like particularly after you go a few years out, it’s just very hard to know what’s going to happen. And so, it’s mostly just speculation. And so in terms of attitude, I think, well, first of all, I think the two attitudes that I find least useful are blind pessimism and blind optimism because they’re actually sort of like doom saying and Pollyannaism. It weirdly is possible to have both at once.

But I think it’s just not very useful because it’s like we’re all doomed. It’s intended to create fear or it’s intended to create complacency. I find that an attitude that’s more useful is to just say, “Well, we don’t know what’s going to happen, but let’s, as an individual or as an organization, let’s pick a place where there’s a problem we think we can help with and let’s try and make things go a little better than they would’ve otherwise.” Maybe we’ll have a small impact, maybe we’ll have a big impact, but instead of trying to understand what’s going to happen with the whole system, let’s try and intervene in a way that helps with something that we feel well-equipped to help with. And of course, the whole outcome, it’s going to be beyond the scope of one person, one organization, even one country.

But I think we find that to be a more effective way of thinking about things. And for us, that’s can we help to address some of these safety problems that we have with AI systems in a way that is robust and enduring and that points towards the future? If we can increase the probability of things going well by only some very small amount, that may well be the most that we can do.

I think from our perspective, the things that I would really like to see are, I would like it if AI could advance science technology and health in a way that’s equitable for everyone, and that it could help everyone to make better decisions and improve human society. And right now, I, frankly, don’t really trust the AI systems we build today to do any of those things, even if it were technically capable of the task, which it’s not, I wouldn’t trust it to do those things in a way that makes society better rather than worse.

And so I’d like us to do our part to make it more likely that we could trust AI systems in that way. And if we can make a small contribution to that while being good citizens in the broader ecosystem, that’s maybe the best we can hope for.

Lucas Perry: All right. And so if people want to check out more of your work or to follow you on social media, where are the best places to do that?

Daniela Amodei: Yeah. On anthropic.com is going to be the best place to see most of the recent stuff we’ve worked on. I don’t know if we have everything posted, but- 

Dario Amodei: We have several papers out, so we’re now about to post links to them on the website.

Daniela Amodei: In an easy to find place. And then we also have a Twitter handle. I think it’s Anthropic on Twitter, and we generally also tweet about our recent releases of our research. 

Dario Amodei: We are relatively low key. We really want to be focused on the research and not get distracted. I mean, the stuff we do is out there, but we’re very focused on the research itself and getting it out and letting it you speak for itself.

Lucas Perry: Okay. So, where’s the best place on Twitter to follow Anthropic?

Daniela Amodei: Our Twitter handle is @anthropicAI.

Lucas Perry: All right. I’ll include a link to that in the description of wherever you’re listening. Thanks a ton for coming on Dario and Daniela, it’s really been awesome and a lot of fun. I’ll include links to Anthropic in the description. It’s a pleasure having you and thanks so much.

Daniela Amodei: Yeah, thanks so much for having us, Lucas. This was really fun.



Anthony Aguirre and Anna Yelizarova on FLI’s Worldbuilding Contest

  • Motivations behind the contest
  • The importance of worldbuilding
  • The rules of the contest
  • What a submission consists of
  • Due date and prizes


Watch the video version of this episode here

Check out the Worldbuilding Contest page here

Follow Lucas on Twitter here

0:00 Intro

2:30 What is “worldbuilding” and FLI’s Worldbuilding Contest?

6:32 Why do worldbuilding for 2045?

7:22 Why is it important to practice worldbuilding?

13:50 What are the rules of the contest?

19:53 What does a submission consist of?

22:16 Due dates and prizes?

25:58 Final thoughts and how the contest contributes to creating beneficial futures


Lucas Perry: Welcome to the Future of Life Institute Podcast. I’m Lucas Perry. Today’s episode is with FLI’s Anthony Aguirre and Anna Yelizarova and is meant to provide information about the FLI worldbuilding contest. In short, this is a contest which invites teams from across the globe to compete for a prize purse of up to $100,000 by designing visions of a plausible and aspirational future that includes strong artificial intelligence. If you want to know more about this competition and how to get involved, you can listen to this podcast, or head over to worldbuild.ai for more information. 

Before we jump into the interview, and in case you didn’t catch it in the David Chalmers episode, I will be moving on from my role as Host of the FLI Podcast, and this means two things. The first is that FLI is hiring for a new host for the podcast. As host, you would be responsible for the guest selection, interviews, production, and publication of the FLI Podcast. If you’re interested in applying for this position, you can head over to the careers tab at futureoflife.org for more information. We also have another 4 job openings currently for a Human Resources Manager, an Editorial Manager, an EU Policy Analyst, and an Operations Specialist. You can learn more about those at the careers tab as well. 

The second item is that even though I will no longer be the host of the FLI Podcast, I won’t be disappearing from the podcasting space. I’m starting a brand new podcast focused on exploring questions around wisdom, philosophy, science, and technology, where you’ll see some of the same themes we explore here like existential risk and AI alignment. I’ll have more details about my new podcast soon. If you’d like to stay up to date, you can follow me on Twitter at LucasFMPerry, link in the description. 

And with that, I’m happy to introduce Anthony Aguirre and Anna Yelizarova on FLI’s new Worldbuilding Contest.

So welcome to the podcast, Anna and Anthony. It’s great to have you. Today, we’re here to talk about FLI’s new Worldbuilding Contest, which is quite a new exciting initiative that you guys have both put a ton of time into working on. So I have a two-part question to start things off here. The first is what is worldbuilding? And the second is, what is FLI’s Worldbuilding Contest?

Anthony Aguirre: Well, why don’t I start out with worldbuilding itself? So worldbuilding is the process of kind of constructing a fictitious world in which a story, or a movie, or a novel or something takes place. So if you think about, for example, Star Wars, there’s the Star Wars movies, and then there’s the Star Wars world that they inhabit. It has certain rules, like, there’s the force, there are spaceships, faster-than-light travel is pretty easy. AI is apparently really hard because there’s only like human-level robots. There are different politics that are happening in the galaxy. There’s a certain level of span of technologies. So there are all these kind of rules to the world. And then within that set of rules, there are a lot of different artifacts.

So there’s Tatooine, the desert planet. And it’s got its own whole feel, and it’s politics and sociology and things. And then there are other planets and then there’s the shape of the Star Destroyer. So there are all these things that have been constructed creatively to inhabit that world that is governed by some set of rules. So worldbuilding is the process of constructing that fictitious reality, including all the details of how do the politics work? What is the technology? What is in it? What sorts of people are in it? What is its history What has happened in the past? And so on. And the idea is to give you a backdrop for really imagining that you’re in this world.

So a really good worldbuild is kind of this evocative thing where you feel like, ah, I could be in that. It’s something that I can sort of experience in my mind. And that it gives you the feeling of reality, because it has been thought through in this self consistent way. You kind of understand what the rules are and you’d be surprised if certain things happen in Star Wars that come from Star Trek or vice versa, those are different worlds that have different sets of rules in them. So this is a process that’s been developed both kind of informally. Anybody who’s writing say a science fiction novel, or a fantasy novel or something, has some worldbuilding element to it, because there’s an imagined world in which their story is taking place. But it’s also been sort of developed more professionally. So there’s a whole industry say in Hollywood of constructing the world that the Marvel universe or that Star Wars or that Star Trek inhabits.

So there are people who are actually doing this for a living. Building fictitious worlds and inventing the artifacts that are in those worlds. And there are worldbuilding programs, university programs that you can learn how to do this process. So it’s a minor industry, but an important one because it’s in a lot of our media. The idea of this contest is to sort of re-task this way of thinking about things of constructing fictitious worlds, to try to construct some plausible and aspirational versions of our own actual world. So not necessarily to, for some other purpose, to put a story into, but to investigate those worlds on their own and to enjoy the process of making them and think about what goes into making that world. And then kind of explore the variety of different worlds that people come up with.

So the idea of the FLI Worldbuilding Contest is to sort of create a competition where the goal is to, as teams, invent a sort of fictitious world that exists in 2045. It’s a world that should make sense, be internally consistent, follow the laws of physics, have plausible technology for 2045, and we’ll get to some of those ground rules. But very importantly, it’s also supposed to be aspirational, so it’s supposed to be a world that we would like to inhabit. And we’ll talk a little bit about why we chose to do it that way instead of just any old world in any old time and so on. But the idea is to gather up lots of interesting contributions from teams around the world and incentivize them with this contest with a nice juicy prize purse to get people really working hard and putting effort into this worldbuild.

Lucas Perry: Why is it important to do worldbuilding for 2045?

Anna Yelizarova: I think the year 2045 is interesting because it’s still in the somewhat near future where most of us would still be alive, so it’s very easy to imagine, as opposed to a very distant world where you can reimagine almost everything. Here we’re trying to keep a lot of what we know what exists today, but then, with a set of constraints, help us do this thought experiment about how we manage to overcome certain challenges that we already see on the horizon. So the idea of worldbuilding for 2045 is just a constraint to focus this exercise, but we might do a different worldbuild that’s in a more distant future. But for now, this is to focus ourselves.

Lucas Perry: Given humanity’s track record of ramrodding our way into new technologies and worlds, just following natural economic incentives, why is it important that we have a worldbuilding contest, that we practice worldbuilding?

Anthony Aguirre: So I think both as individuals and as a society, we’re fairly goal-directed, in general. We have goals for our personal life on a day-to-day basis, and on a longer time scale, we have goals to have a good career, to be happy in this way and that, to have a good relationship for… Maybe to have kids, maybe for them to have good things happen to them. So we have these long-term goals and we work toward them. If we didn’t have those goals, if we just every day woke up and went through some random set of motions, that would be an okay way to live, but we would have a very different life than if we had choices about what we’re more and less desirable for our life and aim toward them.

And I think as a society, we can do much the same thing. We can have some level of goals as a society and work toward them. I think often we have done that less, lately in society than we perhaps did in the past. I think there is a sense that there’s progress, but it’s mostly technological progress and it’s maybe a little bit of social progress, but it’s kind of just pushing us along, and we’re just going where the techno-social progress takes us. And there’s not really much we can do about out that. There’s kind of capitalism, and there’s technology, and wherever they go, we just have to ride it out as best we can. And I think this is a very disempowered way to look at the world.

We, as a society, just like as individuals, have a lot of agency as to what happens to us. We make decisions, and those decisions have real consequence. And part of the idea of this contest is to do a little bit more thinking about what are some possible goals. If we… think about the world 25 years from now, if we imagine a world that we actually want to inhabit, we’re not going to end up living in that world. The world is too unpredictable and things are not to go the way that we want just like regular life. But if you don’t have any goal at all, it’s very hard to know what to work toward, and what to do now in order to get there. So the idea here is to kickstart a process of thinking through what would we like the future to look like?

Not just vaguely, like we haven’t destroyed the world through global warming, or through AI catastrophe, or through biotech, catastrophe. That’s good, that’s important. We really, definitely want to not destroy the world. But going a little bit beyond that, what do we actually want it to look like? And are there things that we can do now to start to plant the seeds for that kind of world? And you can’t plant the seeds now if you don’t really know what kind of world you want to grow into. So the goal of this is to sort of plant some posts down the line two decades from now of, wow, wouldn’t it be cool if the world was like this? Here’s what would have to happen between now and then if the world was going to be something like that. And again, that probably will go astray.

It’s not going to end up quite the way you want it to. But I think, just as in your individual life, pushing in a more positive direction and having a goal doesn’t guarantee that you’re going to reach it, but is probably a more positive and you’re going to get closer to that goal than if you don’t have a goal at all. Or if you have some radically different goal. So the idea here is to start that process and to engage a lot of the creativity that has gone into imagining negative worlds. So there’s a lot of effort that has gone into imagining dystopias and just various ways that the world can go off the rails. I think this is good. I think we as individuals, we also imagine all the things that can go wrong. As a parent, you think of every possible thing that can go wrong with your kids. Still I find more.

And then that’s good, because that’s how you protect your kids from getting run over by a car, eaten by a lion, or whatever. You imagine these things and you prevent them. And as a society, we definitely have to do this too. But if all you’re ever imagining for your kids is all the terrible things that can happen to them, then you’re also not going to be a great parent, because you’re not going to be thinking about the opportunities and you’re not going to be weighing risks against benefits and so on. So I think it is really important to think about all the ways that things can go wrong and work to prevent them, but we don’t want to just be living in this idea that the world is definitely going to be a catastrophe a little bit down the line. And we don’t want to only be focused on the way that everything can go wrong. We want to spend some time thinking about what we would like in sort of concrete and evocative detail.

Lucas Perry: Anna, is there anything else that you’d like to add here in terms of perspective on what worldbuilding is and why it’s important?

Anna Yelizarova: Well, I guess what Anthony is touching in is not only the importance of worldbuilding, but also positive worldbuilding, aspirational worldbuilding, and thinking more positively about the future. I think it’s a much harder task to imagine hopeful futures than it is to imagine everything that can go wrong, because for you to have a rich, detailed worldbuild of a positive future, you actually have to have answers to some of the most pressing challenges of our time. And that thought exercise is very valuable. And there’s so many takes on it. And I think we really want to hear from a very diverse set of people to see both what people want and also how we can get there. And the idea is not just to keep talking to our existing community ecosystem, but really to branch out to as many people around the world, to people who are in different fields and really to get to hear from them, because it’s going to be hard to have a consensus on what kind of future we want.

So part of this worldbuilding contest is also being open to different perspectives and hearing each other out, because even agreeing on a future we want is a huge, huge challenge. So, yep, the contest is definitely aiming at a very, very broad audience. You could be a scientist, a policy researcher, a creative, a digital artist, a writer. You could be from any discipline and still somehow contribute original thoughts and input in this contest. And we really don’t want to discourage anyone to apply. So I think it’s part of a bigger conversation, and worldbuilding helps us get there.

Lucas Perry: So in terms of worldbuilding, it seems like there needs to be a lot of constraints that help people… For example, in a contest to create that world. So I’m curious, Anna, if you could explain some more about the actual ground rules of this contest.

Anna Yelizarova: So the ground rules for the contest, or the set of constraints we chose for this thought exercise are as follow. First of all, the year is 2045, so we’re still somewhat in the near future. We could imagine most of us would be conceivably alive in 2045. AGI has existed for at least five years. So we are intentionally choosing to make artificial intelligence a big focus of our world and of this contest, which is also a big focus area of FLI.

AGI is artificial general intelligence. And the thing to note about AGI is, it’s basically artificial intelligence that has reached this milestone of being at least as good as a human in every task. So we’re talking very advanced AI in this world. Then technology is advancing very rapidly, and AI is transforming the world sector by sector. So AI is the biggest leap in technology we’re seeing in this world, but AI is affecting every single industry, every single sector. So if you have a focus in any other domain than AI, you could probably imagine AI transforming your industry and could choose to focus on that in the contest. Anthony, would you like to take the next set of rules?

Anthony Aguirre: Yeah. So then we kind of thought of AI as the big transformative change from what we know of today. Of course, we know that lots of things are changing in the world. Lots of technologies are advancing. And geopolitically and socially, things are evolving as well. But we wanted to keep some of the focus on what particularly is happening with AI. And so we chose to try to sort of maintain something like the current world in so far as possible in the other sectors. So for example, right now there are kind of major geopolitical centers in the US and the EU and Asia, and especially China. So we kind of kept that. So the idea is that in 2045, there will still be the US and the EU in China as kind of three major centers of power.

There won’t be like one world government and there won’t be a million different balkanized, decentralized powers or something. You can imagine both of those, but just to ground ourselves a little bit, we chose that. Other regions in the world, India, Africa, South America are also advancing, but just aren’t still quite as much on the center of the world stage. Another thing that we wanted to be a little bit conservative in, and also a little bit positive in, was to say that there just haven’t been any major wars or other global catastrophes. So we haven’t had COVID 20 that killed half of humanity. And we haven’t had nuclear war, say between the US and Russia or anything like that. So we’ve kind of modeled along at least geopolitically and haven’t had any catastrophes that went totally awry. And pushing a little bit further in the optimistic direction that the world is generally looking pretty good.

So part of the idea of this contest is to look for positive visions of the future and things that we might want to aspire to. And so part of the ground rules is just that the world isn’t dystopian. We’re not living in 1984 or any other many depicted dystopias. We’re in one of the very few worlds that people would feel like pretty good about being in. It’s a somewhat funny thing that in fiction, most of the time dystopia kind of comes from trying to develop a utopia and it goes wrong. And we’re sort of very used to this. It’s almost hard to adjust your thinking to create a world that is actually good rather than it’s so easy to think of all the different ways that things can go wrong and be bad.

It’s fun and almost liberating to think about a world that’s actually good unironically and unapologetically, like this is a world I’d like to live in. And that’s sort of what we’re asking for here. It’s notable that in a lot of these things, we’ve kept the world kind of similar to how it is now, geopolitically and technologically in wars and stuff. And the world is actually pretty okay at the moment, at least for a lot of people. At least we’re not living in a dystopia for most people. So in that way, it’s conservative, but it’s important to emphasize that the addition of artificial general intelligence is a huge change, that having the ability to replace human labor with machines, not just physical labor, but intellectual labor, means that most jobs can be done by machine, even what we now call thought work and intellectual work can be done by machines.

Productivity will be skyrocketing. So many more things will be possible. There will be inventions that are coming directly out of AGI and AGI-human collaborations. So many, many things will be very, very different because of this introduction. And in a sense, there’s a little bit of tension between the high power and the transformative change that AGI will bring. And the conservatism that the world is not that, that, that different from the way it is now. But I think this tension is part of the job of the entrance to resolve. Like how did that happen? How did we keep control of AGI? So not only has it not gone off the rails, but it hasn’t totally changed the world in something completely, radically different than we have now. So part of the part of the job is to figure out how did that happen. What are the course of events? What are the institutions that were necessary for that technically, socially and so on, how did that come about? And that will be part of the fun, I think, to see how that worked out.

Lucas Perry: So it’d be great if we could pivot here into a bit more of the details behind the actual contest, just so that listeners have a sense of the due dates, what’s actually expected in terms of what’s being delivered and all that. So what are the details behind the contest?

Anna Yelizarova: Yeah, to understand the contest I think hearing what the submission consists of will be very helpful. So to enter the contest, you have to submit four elements. The first being a timeline. The second one being short stories, so writing pieces. The third being answers to a set of questions. And lastly, a piece of non-text media or art. So for the timeline, we want applicants to provide, for every single year from today, from 2022 to 2045, two events and one data point per year. What is an event? An event could be an agreement. An international agreement is formed. An institution is created. Like an actual event. Whereas a data point would be more… Could be a change in GDP, a change in life expectancy. So we’re talking more numbers here. And so this is pretty detailed. We have over 20 years to fill in, each of them having three points, which will help with the richness of the world and thinking through how we’ve achieved certain things.

Then the short stories. The short stories have to take place in 2045. Could be anywhere in the world and it could be different characters. But ultimately you’re telling their story in 750 to 1000 words. What does a day in the life look in 2045? So you’re using more of narrative tool here. You’re using storytelling to give some more color to your world to make it come alive. Then you have a set of prompts you need to answer. And the prompts will help you worldbuild. They’ll be asking how we’ve overcome a set of challenges. Some of them will be very focused on AI. Some of them are going to be more general. And you could definitely use the answers to those, to both shape your timeline or think through your story. All of the elements are meant to interact and are meant to help you with the other tasks.

And for the fourth element we were asking for a non-text media piece. And this could take many forms. This could be a very visual piece, digital art. This could be a video. Could also be audio, but we just don’t want something that is in written form, because so much of the rest of the application already focuses on that.

Lucas Perry: In terms of actual due dates and some more details here about when this is all wrapping up and the amount of money that is being offered in terms of prizes, could you guys speak a little bit more on that?

Anthony Aguirre: So for this there’s a pretty significant prize purse. So we have a bunch of prizes. One first prize of $20,000. Two second prizes of $10,000. Five third prizes of $2,000. And ten fourth of a thousand dollars each. And the judges have discretion to give up to five extra prizes of $2,000 each for whatever they like, like they could just really love a movie and give a $2,000 prize for that, even if the rest of the build wasn’t a winner. But we also… This doesn’t totally complete the prize package because we really want to encourage teams to enter in this rather just individuals. Individuals are fine, but I think this will be much more fun if teams get together and work on it, it’d be more productive. So rather than forcing teams to split the prize evenly, which kind of disincentivizes things, we’re giving a bigger prize, if you enter in a bigger team.

So we’ll scale up the prize. So for example, if you have a five person team, the prize is doubled. And so you don’t quite get the full prize that you would’ve gotten as an individual, but you don’t get a fifth of it either. You get like two fifths. So we’re really hoping, and we’re going to put effort into trying help people form teams by incentivizing with the prizes, but also just doing what we can to build a community and to help people connect with each other, because this is an exercise that’s really fun to do in groups, I would say.

Anna Yelizarova: In terms of important dates and milestones for the contest, the contest opened January 1st and teams have until April 15th to put in their entries. So then the contest will close April 15th, and the judges will take a month to pick 20 finalists. So May 15th we’ll hear about our 20 finalists, and everyone will get to see their worldbuilds, which will be published online. At this point, there’ll be a month where anyone in the public can input on… Can vote, can provide feedback, and can just voice which futures they like. So we’d love some audience participation here. And then the judges will take the audience feedback and use that to rank the finalists according to first, second, third prize, et cetera. So the final winners will be known June 15th, 2022.

Lucas Perry: And Anna, if people want to get any more information about the contest to see everything that we’ve to talked about here on the FLI website, where’s the best place to do that?

Anna Yelizarova: So we have a website just for the contest. It’s worldbuild.ai. So very easy to remember. This has all the rules, all the deadlines, the prizes. Everything we’ve talked about is on the website, so I encourage you to look through the FAQs or anything like that. You can also join our Discord if you have questions. We’ll be monitoring the online community. You can also use the Discord to meet potential team members and interact with other folks interested in the contest. And if Discord isn’t your cup of tea, you can also send an email to worldbuild@futureoflife.org, and we’ll also be monitoring that channel. So those are the easiest ways to get in touch and join the community.

Lucas Perry: Awesome. So as we wrap up here, I’m just curious to get your final thoughts and feelings about this worldbuilding project. I guess I can start with you, Anthony, just how do you see this as really fitting in with all of your work overall at FLI? And how do you feel about it in terms of its place of working towards beneficial futures and mitigating existential risk?

Anthony Aguirre: First of all, I think it’s going to be a tremendous amount of fun. So there’s a little bit of a precedent at FLI for this. We, in 2019 had a meeting called the Augmented Intelligence Summit. And that was, I think around 40 or so, 50 people, that got together essentially to do worldbuilding. And the ground rules were actually kind of similar to the ones in this contest. And we did a lot of in-person exercises, including writing and talking about stories like the ones that we have here. Role playing, so you would put yourself in the role of some person in the future in this future world. All kinds of things to really inhabit it. And I found it to be really, really just enjoyable and insight-building. So for example, one of the things that I’ve been thinking a lot about lately on the AI side is the concept of loyalty or fiduciary duty in AI.

So we have humans who have to act sort of in the interest of their client, like a doctor or a lawyer, or a financial advisor. They have a legal duty to act in the interest of their clients. AI systems that we have right now don’t always have that same sort of obligation on them. And so in the world that we were talking about in the Augmented Intelligence Summit, we came up with this idea of fiduciary AI assistance. So these are assistants that have to act in your interest. That are acting just for you. That are like a human assistant that is your employee. They’re just doing… They’re not secretly working for some other company or something. And that was part of the world that we built. And so just out of that worldbuilding exercise came this idea that has actually led to two published papers, new initiatives that we’re thinking about.

I don’t think we necessarily would’ve thought along those lines without that worldbuilding contest. So I think it’s very easy to… If you’re starting from where we are here, and you’re thinking about what the future is going to look like, to just make sort of minor perturbations or just kind of push in one direction or another a little bit. When you’re forced to jump to 2045, say, you know that things are going to be quite different and it kind of frees you up creatively to think about how the world could be radically different. And I think that’s really, really valuable. And it’s also I think… The advantage that worldbuilding has above just, well, let’s think about the future and what might exist is that in that case you tend to do things sort of abstractly, like the world might have this sort of technology in it.

But when you think of, for example, a day in the life of someone, they get into their self-driving car. And then you’re thinking about what that means exactly. Who’s the owner of the car? What happens if they’ve been out drinking? Does the self-driving car spy on them or is the self-driving car kind of private inside? When you try to actually build a fictional and very concrete world and go into the backstory of what’s behind every little piece of it, you come up with all sorts of questions that you hadn’t really considered before. The experiential side of it leads you to encounter that future in a way that you wouldn’t just going through a purely intellectual exercise, I think, which is really valuable and really enjoyable, and I think something that we will get a lot more out of than purely abstract intellectual thinking about it.

Lucas Perry: Anna, do you have anything else that you would like to add here in terms of how this fits into your existential risk work and what makes you so excited about this?

Anna Yelizarova: Well, I guess we’re getting to the bottom of the why is FLI running this contest and the goals behind it. And we’re really hoping that the outputs of this contest tie into our real world work at FLI, and aren’t just these creative visions we share. As Anthony said, there might be some really nuggets of wisdom, some interesting policy propositions. Some new institutions that people describe in these worldbuilds that are real ideas that maybe are worth pursuing in the real world in the present.

So there’s this inspiration for, how did we overcome these problems? I think it’s important to say that we’re not pushing for any particular future, and we’re mostly crowdsourcing suggestions for how did we overcome major problems? How did we ensure there was not another pandemic? Or how did we ensure that AI was kept safe? And we want to hear the answers to that. So inspire our real life efforts at FLI. Another side of it is that we’re trying to lean in hard into the storytelling aspect at FLI. And I know a few of our coworkers are really passionate about that, because storytelling can be really powerful in convincing people that certain risks are real or certain futures are worth working towards.

And I think crowdsourcing futures people want are part of it. I know that we do want to hear everyone’s input perspective on the kind of futures we find desirable. And another side to this coin is trying to actually show people what positive futures might look like. Because it’s true, we have been bombarded with dystopia left and right. It’s in all of our fiction, in Hollywood, it’s in books and movies, it’s everywhere. And I do think it does something to our worldview if that’s how we all think about the future. And if we as a society had a better relationship with the future, I think people would be motivated to work towards it as opposed to having a doom-and-gloom approach to things. So there’s a strong storytelling component, and we’re hoping to use these worldbuilds, put them in the hands of storytellers.

We’re hoping to do more after the contest as well to give life to these worldbuilds. And help people feel more positively about the future. Not in a naive way. Not in the sense of, oh, everything will be fine, but in the sense of like, oh, if we work hard, if we tackle these problems, there’s this thing really worth working towards. So we do care about inspiring people, and we sure hope to work with a lot more storytellers on the other side after the contest is over. So hopefully this initiative doesn’t end with the contest itself on June 15th.

Lucas Perry: All right. Awesome. Thank you very much, Anthony and Anna. I’m really excited to see what the… Who the winners are. I know that the quality of their world will be really interesting and amazing. I’m also excited to see all the kinds of interesting institutions that come out of it. And I’m especially excited for the art pieces. That’ll be really cool. So, yeah. Thanks so much for coming on, and if people want to get more information, I’ll include links to the website in the description of wherever you might be listening. So, yeah, thank you so much.


David Chalmers on Reality+: Virtual Worlds and the Problems of Philosophy

  • Virtual reality as genuine reality
  • Why you can live a good life in VR
  • Why we can never know whether we’re in a simulation
  • Consciousness in virtual realities
  • The ethics of simulated beings


Watch the video version of this episode here

Check out David’s book and website here

Follow Lucas on Twitter here

0:00 Intro

2:43 How this books fits into David’s philosophical journey

9:40 David’s favorite part(s) of the book

12:04 What is the thesis of the book?

14:00 The core areas of philosophy and how they fit into Reality+

16:48 Techno-philosophy

19:38 What is “virtual reality?”

21:06 Why is virtual reality “genuine reality?”

25:27 What is the dust theory and what’s it have to do with the simulation hypothesis?

29:59 How does the dust theory fit in with arguing for virtual reality as genuine reality?

34:45 Exploring criteria for what it means for something to be real

42:38 What is the common sense view of what is real?

46:19 Is your book intended to address common sense intuitions about virtual reality?

48:51 Nozick’s experience machine and how questions of value fit in

54:20 Technological implementations of virtual reality

58:40 How does consciousness fit into all of this?

1:00:18 Substrate independence and if classical computers can be conscious

1:02:35 How do problems of identity fit into virtual reality?

1:04:54 How would David upload himself?

1:08:00 How does the mind body problem fit into Reality+?

1:11:40 Is consciousness the foundation of value?

1:14:23 Does your moral theory affect whether you can live a good life in a virtual reality?

1:17:20 What does a good life in virtual reality look like?

1:19:08 David’s favorite VR experiences

1:20:42 What is the moral status of simulated people?

1:22:38 Will there be unconscious simulated people with moral patiency?

1:24:41 Why we can never know we’re not in a simulation

1:27:56 David’s credences for whether we live in a simulation

1:30:29 Digital physics and what is says about the simulation hypothesis

1:35:21 Imperfect realism and how David sees the world after writing Reality+

1:37:51 David’s thoughts on God

1:39:42 Moral realism or anti-realism?

1:40:55 Where to follow David and find Reality+


Lucas Perry: Welcome to the Future of Life Institute Podcast. I’m Lucas Perry. Today’s episode is with David Chalmers and explores his brand new book Reality+: Virtual Worlds and the Problems of Philosophy. For those not familiar with David, he is a philosopher and cognitive scientist who specializes in the philosophy of mind and language. He is a Professor of Philosophy and Neural Science at New York University, and is the co-director of NYU’s Center for Mind, Brain and Consciousness. Professor Chalmers is widely known for his formulation of the “hard problem of consciousness,” which asks, “Why a physical state, like the state of your brain, is conscious rather than nonconscious?” 

Before we jump into the interview, we have some important and bitter-sweet changes to this podcast to announce. After a lot of consideration, I will be moving on from my role as Host of the FLI Podcast, and this means two things. The first is that FLI is hiring for a new host for the podcast. As host, you would be responsible for the guest selection, interviews, production, and publication of the FLI Podcast. If you’re interested in applying for this position, keep your eye on the Careers tab on the futureoflife.org website for more information. 

The second item is that even though I will no longer be the host of the FLI Podcast, I won’t be disappearing from the podcasting space. I’m starting a brand new podcast focused on exploring questions around wisdom, philosophy, science, and technology, where you’ll see some of the same themes we explore here like existential risk and AI alignment. I’ll have more details about my new podcast soon. If you’d like to stay up to date, you can follow me on Twitter at LucasFMPerry, link in the description. This isn’t my final time on the FLI Podcast, I’ve got three more episodes including a special farewell episode, so there’s still more to come! 

And with that, I’m very happy to introduce David Chalmers on Reality+.

Welcome to the podcast David, it’s a really big pleasure to have you here. I’ve been looking forward to this. We both love philosophy so I think this will be a lot of fun. And we’re here today to discuss your newest book, Reality+. How would you see this as fitting in with the longer term project of your career and philosophy?

David Chalmers: Oh boy, this book is all about reality. I think of philosophy to being about, to a very large extent about the mind, about the world and about relationships between the mind and the world. In a lot of my earlier work, I’ve focused on the mind. I was drawn into philosophy by the problem of consciousness, understanding how a physical system could be conscious, trying to understand consciousness in scientific philosophical terms.

But there are a lot of other issues in philosophy too. And as my career has gone on, I guess I’ve grown more and more interested in the world side of the equation, the nature of reality, the nature of the world, such that the mind can know it. So I wrote a fairly technical book back in 2012 called Constructing the World. That was all about what is the simplest vocabulary you can use to describe reality?

But one thing that was really distinctive to this book was thinking about it in terms of technology. In philosophy, it often is interesting and cool to take an old philosophical issue and give it a technological twist. Maybe this is most clear in the case of thinking about the mind and then thinking about the mind through the lens of AI, are artificial minds possible? That’s a big question for anybody. If they are, maybe that tells us something interesting about the human mind. If artificial minds are possible then maybe the human mind is in relevant ways analogous for example to an artificial intelligence.

Then, well, the same kind of question comes up for thinking about reality and the world. Are artificial worlds possible? Normally we think about, okay, ordinary physical reality and the mind’s relation to that, but with technology, there’s now a lot of impetus to think about artificial realities, realities that we construct, and the crucial case there is virtual realities, computational based realities, virtual worlds even of the kind we might construct say with video games or full scale virtual realities, full scale universe simulations. And then a bunch of analogous questions come up, are artificial realities genuine realities?

And just in the artificial mind case, I want to say artificial minds are genuine minds. Well, likewise in the artificial world case, I want to say, yeah, virtual realities are genuine realities. And that’s in fact, the central slogan of this new book Reality+, which is very much trying to look at some of these philosophical issues about reality through the lens of technology and virtual realities, as well as trying to get some philosophical insight into this virtual reality technology in its own right by thinking about it philosophically. This is the process I call techno-philosophy, using technology to shed light on philosophy and using philosophy to shed light on technology.

Lucas Perry: So you mentioned… Of course you’re widely known as a philosopher of consciousness and it’s been a lot of what you focused on throughout your career. You also described this transition from being interested in consciousness to being interested in the world increasingly over your career. Is that fair to say?

David Chalmers: Yeah. You can’t be interested in one of these things without being interested in the other things. So I’ve always been very interested in reality. And even in my first book on consciousness, there was speculation about the nature of reality. Maybe I talked about it from bit hypothesis there. Maybe reality is made of information. I talked about quantum mechanics and potential connections to consciousness. So yeah, you can’t think about, say the mind body problem without thinking about bodies as well as minds, you have to think about physical reality.

There’s one particular distinctive question about the nature of reality namely how much can we know about it? And can we know anything about the external world? That’s a very traditional problem in philosophy. It goes back to Descartes saying, how do you know you’re not dreaming right now? Or how do you know you’re not being fooled by an evil demon who’s producing sensations as of an external world when none of this is real? And for a long time, I thought I just didn’t have that much to about this very big question in philosophy.

I think of the problem with consciousness, the mind body problem. That’s a really big question in the history of philosophy. But to be honest, I’m going to say it’s probably not number one. Number one at least in the Western philosophical tradition is how do we know anything about the external world? And for a long time, I thought I didn’t have anything to say about that. And at a certain point, partly through thinking about yeah, virtual realities and the simulation hypothesis, I thought, yeah, maybe there is something new to say here via this idea that virtual realities are genuine realities. Maybe these hypotheses that Descartes put forward saying, “If this is the case, then none of this is real.” Maybe Descartes was actually thinking about these hypotheses wrongly. 

And I actually got drawn into this. Around the same time, just totally fortuitously I got invited to write an article for the Matrix Website. Their production company, Red Pill, it was a philosopher called Chris Grawl, who worked for them. And I guess the Wachowskis were super interested in philosophy. They wanted to see what philosophers thought of philosophical issues coming from the movie. So I ended up writing an article called The Matrix as Metaphysics, putting forward this rough point of view, which is roughly in the context of the movie that even in the movie, they say, well, if we’re in the Matrix, none of what we’re experiencing is real. All this is illusion or a fiction.

I tried to argue, even if you’re in the Matrix, these things around you are still perfectly real. There are still trees, there are still cats, there are still chairs. There are still planets. It’s just that they’re ultimately digital, but they’re still perfectly real. And I tried to use that way of thinking about the Matrix to provide a response to the version of Descartes who says, “We can never know anything about the external world, because we can’t rule out that none of this is real.”

All those scenarios Descartes had in mind. I think some sense there are actually scenarios where things are real and that makes this vision of reality. Maybe it makes reality a bit more like virtual reality, but that vision of reality actually puts knowledge of the external world more within our grip. And from there, there’s a clean path from writing that article 20 years ago to writing this book now, which takes this idea of virtual reality as genuine reality and tries to just draw it out in all kinds of directions, to argue for it, to connect to present day technology, to connect it to a bunch of issues in philosophy and science. Because if I to start thinking this way about reality, at least I’ve found it changes everything. It changes all kinds of things about your vision of the world.

Lucas Perry: So I think that gives a really good taste of what is to come in this interview and also what’s in your book. Before we dive more into those specifics, I’m also just curious what your favorite part of the book is. If there’s some section or maybe there isn’t that you’re most excited to talk about, what would that be?

David Chalmers: Oh, I don’t know. I was going to say my favorite parts of the book had the illustrations, amazing illustrations by Tim Peacock, who’s a great illustrator who I found out about and I asked if he’d be able to do illustrations for the book. And he took so many of these scenarios, philosophical thought experiments, science fiction scenarios, and came up with wonderful illustrations to go along with it. So we’ve got Plato’s Cave, but updated for the 21st century with people in virtual reality inside Plato’s Cave with Mark Zuckerberg running the cave, or we have an ancient Indian thought experiment about Narada and Vishnu updated them in the light of Rick and Morty. We’ve got a teenage girl hacker creating a simulated universe in the next universe up.

So these illustrations are wonderful, but I guess that doesn’t quite answer your question, which parts do I especially want to talk about? I think of the book as having roughly two halves. Half of it is broadly about the simulation hypothesis. The idea that the universe is a simulation and trying to use that idea to shed light on all kinds of philosophical problems. And the other half is more about real virtual reality, the coming actual virtual reality technology that we have and will develop in the next say 50 to 100 years and trying to make sense of that and the issues it brings up.

So in the first part of the book, I talk about very abstract issues about knowledge and reality and the simulation hypothesis. The second part of the book gets a bit more down to earth and even comes to issues about ethics, about value, about political philosophy. How should we set up a virtual world? That was more of a departure for me to be thinking about some of those more practical and political issues, but over time I’ve come to find they’re fascinating to think about.

So I guess I’m actually equally fascinated by both sets of issues. But I guess lately I’ve been thinking especially about some of these second class of issues, because a lot of people given the coming… All the corporations now are playing up the metaverse and coming virtual reality technology. That’s been really interesting to think about.

Lucas Perry: So given these two halves in general and also the way that the book is structured, what would you say are your central claims in this book? What is the thesis of the book?

David Chalmers: Yeah, the thesis of the book that I lay out in the introduction is virtual reality is genuine reality. It’s not a second class reality. It’s not fake or fictional. Virtual reality is real. And that breaks down into a number of sub-thesis. One of them is about the existence of objects, and it’s a thesis in metaphysics. It says the objects in virtual reality are real objects, a virtual tree is a real object. It may be a digital object, but it’s real all the same. It has causal powers. It can affect us. It’s out there independently of us. It needn’t be an illusion.

So yeah, virtual objects are real objects. What happens in virtual reality really happens. And that’s one kind of thesis. Another thesis is about value or meaning. That you can lead a valuable life, you can lead a meaningful life inside a virtual world. Some people have thought that virtual worlds can only ever be escapist or fictions or not quite the real thing. I argue that you can lead a perfectly meaningful life.

And the third kind of thesis has tied closer to the simulation hypothesis idea. And there I don’t argue that we are in fact in a computer simulation, but I do argue that we can never know that we’re not in a simulation. There’s no way to exclude the possibility that we’re in a simulation. So that’s a hypothesis to take very seriously. And then I use that hypothesis to flesh out a number of different… Just say we are in a simulation then, yeah, what would this mean for say our knowledge of the world? What would this mean for the reality of God? What would this mean for the underlying nature of the metaphysics underneath physics and so on? And I try and use that to just put forward a number of sub-thesis in each of these domains.

Lucas Perry: So these claims also seem to line up with really core questions in philosophy, particularly having to do with knowledge, reality and value. So could you explain a little bit what are some of the core areas of philosophy and how they line up with your exploration of this issue through this book?

David Chalmers: Yeah, traditionally philosophy is at least sometimes divided up into three areas, metaphysics, epistemology and the theory of value. Metaphysics is basically questions about reality. Epistemology is basically questions about knowledge and value theory is questions about value, about good versus bad and better versus worse. And in the book, I divide up these questions about virtual worlds into three big questions in each of these areas, which I call the knowledge question, the reality question and the value question.

The knowledge question is, can we know whether we’re in a virtual world in particular? Can we ever be sure that we’re not in a virtual world? And there I argue for an answer of no, we can ever know for sure that we’re not in a virtual world, we can never exclude that possibility. But then there’s the reality question, which is roughly, if we are in a virtual world, is the world around us real? Are these objects real? Are virtual realities genuine realities or are they somehow illusions or fictions? And there I argue for the answer, yes, virtual worlds are real. Entities and events in virtual world are perfectly real entities and events. Even if we’re in a simulation, the objects around us are still real. So that’s a thesis in metaphysics.

Then there’s the question in value theory, which is roughly, can you lead a good life in a virtual world? And there as I suggested before I want to argue, yes, you can lead a good and meaningful life in a virtual world. So yeah, the three big questions behind the book, each correspond then to a big question, a big area of philosophy. I would like to think they actually illuminate not just questions about virtual worlds, but big questions in those areas more generally. The big question of knowledge is, can we know anything about the external world?

The big question of reality is, what is the nature of reality? The big question about value is, what is it to lead a good life? Those are big traditional philosophical questions. I think thinking about each of those three questions through the lens of virtual reality and trying to answer the more specific questions about what is the status of knowledge, reality and value in a virtual world, that can actually shed light on those big questions of philosophy more broadly.

So what I try to do in the book is often start with the case of the virtual world, give a philosophical analysis of that, and then try to draw out morals about the big traditional philosophical question more broadly.

Lucas Perry: Sure. And this seems like it’s something you bring up as a techno-philosophy in the book where philosophy is used to inform the use of technology and then technology is used to inform philosophy. So there’s this mutual beneficial exchange through techno-philosophy.

David Chalmers: Yeah. Philosophy is this two-way interaction between philosophy and technology. So what I’ve just been talking about now, using virtual reality technology and virtual worlds to shed light on big traditional philosophical questions, that’s the direction in which technology sheds light on philosophy, or at least thinking philosophically about technology can shed light on big traditional question in philosophy that weren’t cast in terms of technology, can we know we’re not in a simulation? That sheds light on what we can know about the world. Can we lead a good life in a virtual world? That sheds some light on what it is to lead a good life and so on.

So yeah, this is the half of techno-philosophy, we’re thinking about technology sheds light on philosophy. The other half is thinking philosophically, using philosophy to shed light on technology and just thinking philosophically about virtual reality technology, simulation technology, augmented reality technology and so on. And that’s I think something I really try to do in the book as well. And I think these two things, these two processes of course complement each other. Because thinking, you think philosophically about technology, it shed some light on the technology, but then it turns out actually to have some impact on the broader issues of philosophy at the same time.

Lucas Perry: Sure. So what’s coming up for me is Plato’s Cave Allegory is actually a form of techno-philosophy potentially, where the candle is a kind of technology that’s being used to cast shadows to inform how Plato’s examining the world.

David Chalmers: That’s interesting. Yeah. I hadn’t thought about that. But I suppose back around Plato’s time, people did a whole lot with candles and fire. These were very major technologies of the time. And maybe at a certain point people started developing puppet technology and started doing puppet style shows that were a form of, I don’t know, entertainment technology for them. And then for Plato then to be thinking about the cave in this way, yeah, it is a bit of a technological setup and Plato is using this new technology to make claims about reality.

Plato also wrote about other technologies. He wrote about writing, the invention of writing and he was quite down on it. He thought or at least his spokesman’s Socrates said, “In the old days people would remember all the old tales, they’d carry them around in their head and tell them person to person, and now that you can write them down, no one has to remember them anymore.” And he thought this was somehow a step back in the way in which some people these days think that putting all this stuff on your smartphone might be a step back. But yeah, Plato was very sensitive to the technologies of the time.

Lucas Perry: So let’s make a B line for your central claims in this book. And just before we do that, I have a simple question here for you. Maybe it’s not so simple but… So what is virtual reality?

David Chalmers: Yeah, the way I define it in the book, I make a distinction between a virtual world and virtual reality, where roughly virtual reality technology is immersive. It’s the kind of thing you experience say with a Oculus Quest headset that you put onto your head and you experience a three dimensional space all around you. Whereas a virtual world needn’t be immersive. When you play a video game, when you’re playing World of Warcraft or you’re in Fortnite, typically you’re doing this on a two dimensional screen, it’s not fully immersive, but there’s still a computer generated world.

So my definitions are a virtual world is an interactive computer generated world. It has to be interactive. If it’s just a movie, then that’s not yet a virtual world, but if you can perform actions within the world and so on and it’s computer generated, that’s a virtual world. A virtual reality is an immersive interactive computer generated world. Then the extra condition, this has to be experienced in 3D with you at the center of it, typically these days experienced with a VR headset and that’s virtual reality. So yeah, virtual reality is immersive interactive computer generated reality.

Lucas Perry: So one of the central claims that you mentioned earlier was that virtual reality is genuine reality. So could you begin explaining why is it that you believe the virtual reality is genuine reality?

David Chalmers: Yeah. Because a lot of this depends on what you mean by real and by genuine reality. And one thing I do in the book is try and break out number of different meanings of real, what is it for something to be real? One is that it has some causal power that it could make a difference in the world. One is that it’s out there independent of our minds. It’s not just all in the mind. And one, maybe the most important is that it’s not an illusion. It’s not just that things are roughly as they seem to be. And I try to argue that if we’re in VR, the objects we see have all of these properties, basically the ideas. When you’re in virtual reality you’re interacting with digital objects, objects that exist as data structures on computers, the actual concrete processes up and running on a computer room. We’re interacting with concrete data structures realized in circuitry on these computers.

And those digital objects have real causal powers. They make things happen. They’re when two objects interact in VR, the two corresponding data structures on a computer are genuinely interacting with each other. When a virtual object appears a certain way to us, that data structure is at the beginning of a causal chain that affects our conscious experience in much the same way that a physical object might be at the start of a causal chain affecting our experience.

And most importantly, I want to argue that, just say, let’s take the extreme case of… I find it useful to start with the extreme case of the simulation hypothesis, where all of this is a simulation. I want to say in that case when I have an experience of say a tree in front of me or here’s a desk and a chair, I’m going to say none of that is illusory. There’s no illusion there. You’re interacting with digital object. It’s a digital table or a digital chair, but it’s still perfectly real.

And the way that I end up arguing for this in the book is to argue that the simulation hypothesis should be seen as equivalent to a kind of hypothesis which has become familiar in physics, the version of the so-called it from bit hypothesis. The it from bit hypothesis says roughly that physical reality is grounded in a level of interaction of bits or some computational process. The paradigm illustration here would be Conway’s Game of Life where you have a cellular automaton with cells that could be on or off and simple rules governing their interaction.

And various people have speculated that the laws of physics could be grounded in some kind of algorithmic process, perhaps analogous to Conway’s Game of Life. People call this digital physics. And it’s not especially widely believed among physicists, but there are some people who take it seriously. And at least it’s a coherent hypothesis that, yeah, there’s a level of bits underneath physical objects in reality. And importantly, if the it from bit hypothesis is true, this is not a hypothesis where nothing is real, it’s just a world where there still are chairs and tables. There still are atoms and quarks. It’s just they’re made of bits. There’s a level underneath the quarks, the level of bits that things are perfectly real.

So in the book I try to argue that actually the simulation hypothesis is equivalent to this it from bit hypothesis. It’s basically, if we’re in a simulation, yeah, there are still tables and chairs, atoms and quarks. There’s just a level of bits underneath that. All this is realized maybe by a computer process involving the interaction of bits and maybe there’s something underneath that in turn that leads to what I call the it from it hypothesis. Maybe if we’re in a simulation, there’s a number of levels like this.

But yeah, the key then is the argument that these two hypotheses are equivalent, which is a case I try to make in chapter nine of the book. The argument itself is complex, but there’s a nice illustration to illustrate it. On one hand, we’ve got a traditional God creating the universe by creating some bits, by, yeah, “Let there be bits,” God says and lays out the bits and gets them interacting. And then we get tables and chairs out of that. And in the other world we have a hacker who does the same thing except via a computer. Let there be bits arranged on the computer, and we get virtual tables and chairs out of that. I want to argue that the God creation scenario and the hacker simulation scenario basically are isomorphic.

Lucas Perry: Okay. I’m being overwhelmed here with all the different ways that we could take this. So one way to come at this is from the metaphysics of it where we look at different cosmological understandings. You talk in your book about there being, what is it called? The dust theory? There may be some kind of dust which can implement any number of arbitrary algorithms, which then potentially above that there are bits, and then ordinary reality as we perceive it as structured and layered on top of that. And looking at reality in this way it gives a computationalist view of metaphysics and so also the world, which then informs how we can think about virtual reality and in particular the simulation hypothesis. So could you introduce the dust theory and how that’s related to the it from bit argument?

David Chalmers: Yeah. The dust theory is an idea that was put forward by the Australian science fiction writer, Greg Egan in his book, Permutation City, which came out in the mid 90s, and is a wonderful science fiction novel about computer simulations. The dust theory is in certain respects even more extreme than my view. I want to say that as long as you have the right computation and the right causal structure between entities in reality, then you’ll get genuine reality. And I argue that can be present in a physical reality, that can be present in a virtual reality. Egan goes a little bit more extreme than me. He says, “You don’t even need this causal structure. All you need is unstructured dust.”

We call it dust. It’s basically a bunch of entities that have no spatial properties, no temporal properties. It’s a whole totally unstructured set of entities, but we think of this as the dust and he thinks the dust will actually generate every computer process that you can imagine. He thinks they can generate any objects that you imagine and any conscious being that you can imagine and so on. Because he thinks there’s ways of interpreting the dust so that it’s for example, implementing any computer program whatsoever. And in this respect, Egan has actually got some things in common with philosophers like the American philosophers, Hilary Putnam, and John Searle, who argued that you can find any computation anywhere.

Searle argued that his wall implemented the WordStar, word processing program. Putnam suggested that maybe a rock could implement complex computations, basically, because you can always map the parts of the computation of the physical object onto the parts of the computation. I actually disagree with this view. I think it’s two unconstrained. I think it makes it too easy for things to be real.

And roughly the reason is I think you need constraints of cause and effect between the objects. For a bunch of entities in a rock or a wall to implement, say a WordStar, they have to be arranged in a certain way so they go through certain state transitions. And so they would go through different state transitions and different circumstances to actually implement that algorithm. And that requires genuine causal structure. And yeah, way back in the 90s, I wrote a couple of articles arguing that the structure you’ll find in a wall or a rock is not enough to implement most computer programs.

And I’d say exactly the same for Egan’s dust theory, that the dust does not have enough structure to support a genuine reality because it doesn’t have these patterns of cause and effect, obeying counterfactuals, if this had happened, then this would’ve happened. And so you just don’t get that rich structure out of the dust. So I want to say that you can get that structure, but to get that structure you need dust structured by cause and effect.

And importantly I think, in average computer simulation like the simulation hypothesis, it’s not like the dust, computer simulations really have this rich causal structure going on inside the computer. You’ve got circuits which are hooked up to each other in the patterns of cause and effect that are isomorphic to that in the physical reality. That’s why I say virtual realities are genuine realities because they actually have this underlying computational structure.

But I would disagree with Egan that the dust is a genuine reality because the dust doesn’t have these patterns of cause and effect. I ended up having a bunch of email with Greg Egan about this and he was arguing for his own particular theory of causation, which went another way. But yeah, at least that’s where I want to hold the line, cause and effect matters.

Lucas Perry: My questions are, so what is the work then that you see the dust theory doing in your overall book in terms of your arguments for virtual reality as genuine reality?

David Chalmers: The dust theory comes relatively late in the book, right? Earlier on I bring in this it from bit idea that yeah, all of reality might be grounded in information in bits, in computational processes. I see that dust theory is being, but partially tied to a certain objection somebody might make, that I’ve made it too easy for things to be real now. If I can find reality in a whole bunch of bits like that, maybe I’m going to be able to find this reality everywhere. And even if we’re just connected to dust, there’ll be trees and chairs, and now isn’t reality made trivial. So partly I think thats an objection I want to address, one say no it’s still not trivial to have reality. You need all this structure, this kind of cause and effect structure or roughly equivalently, a certain mathematical structure in the laws of nature.

And that’s really a substantive constraint, but it’s also a way of helping to motivate the view that I call structuralism about, and that many others have called structuralism or structural realism about physical reality, which I think is kind of actually the key to my thesis. Why does virtual reality get to count as genuine reality? Ah, because it has the right structure. It has the right causal structure. It has the right kind of mathematically characterizable interactions between different entities. What matters is not so much what these things are made of intrinsically, but the interactions and the relations between them. And that’s a view that many philosophers of science these days find very plausible. It goes back to Punqueray and Russell and Carnap and others, but yeah, very popular these days. What matters lets say for a theory in physics to be true is that basically you’ve got entities with the right kind of structure of interactions between them.

And if that view is right, then it gives a nice explanation of why virtual reality, it counts as genuine reality because when you have a computer simulation of a given physical of say of the physical world that has all that preserves computer simulation preserves, the relevant kind of structure. So yeah, the structure of the laws of physics could be found at a physical reality, but it could also that structure could also be found in a computer simulation of that reality. Computer simulations have the right structure, but then it’s yeah. So it turns that’s not totally unconstrained. Some people think, Egan thought the dust is good enough. Some people think purely mathematical structure is good enough. In fact, your sometime boss, Max Tegmark, I think may, may think something like this in his book, on the mathematical universe, he argues that reality is completely mathematical.

And at least sometimes it seems to look as if he’s saying the content of our physical theories is just purely mathematical claims that there exists certain entities with a certain mathematical structure. And I worry that as with Egan that if you understand the content of our theories is purely mathematical, then you’ll find that structure anywhere. You’ll find it in the dust. You’ll find it in any abstract about mathematics. And there’s a worry that actually our physical theories could be trivialized and they can all end up being true, because we can always find dust or mathematical entities with the right structure. But I think if you add the constraint of cause and effect here, then it’s no longer trivialized.

So I think of Egan and Tegmark as potentially embracing a kind of structuralism, which is even broader than mine lets in even more kinds of things as reality. And I don’t be quite so unconstrained. So I want to add some of these constraints of cause and effect. So this is rather late in the book, this is kind of articulating this, the nature of the kind of structuralism that I see as underlying this view of reality.

Lucas Perry: So, Egan and Max might be letting in entities into the category of what is real, which might not have causal force. And so you’re adopting this criteria of cause and effect being important in structuralism for what counts as genuine.

David Chalmers: Yeah. I worry that if we don’t at least have, I think cause and effect is very important to our ordinary conception of reality that for example of things have causal powers. If we don’t have some kind of causal constraint on reality, then it becomes almost trivial to interpret reality as being anywhere. I guess I think of what we mean by real is partly a verbal question, but I think of causal powers is very central to our ordinary notion of reality. And I think that manages actually to give us a highly constrained notion of reality. Where realities are at least partly individuated by their causal structures, but where it’s not how, it’s not now so broad that arbitrary conglomerates of dust get to count as being on a par with our physical world or arbitrary sets of mathematical entities likewise.

Lucas Perry: Let’s talk more about criteria for what makes things count as real or genuine or whether or not they exist. You spend a lot of time on this in your book, sort of setting and then arguing for different positions on whether or not certain criteria are necessary and or sufficient for satisfying some understanding of like, what is real or what is it that it means that something exists or that it’s genuine. And this is really important for your central thesis of virtual reality being genuine reality. Cause it’s important to know like what it is that exists and how virtual reality fits into what is real overall. So could you explore some of the criteria for what it means for something to be part of reality or what is reality?

David Chalmers: Yeah. I end up discussing five different notions of reality of what it is for something to be real. I mean, this kind of goes back to The Matrix where Neo says this isn’t real and Morpheus says, “What is real? How do you define real?” That’s the question? How do you define “real?” And I talk about five main, any number of different things people have meant by real, but I talk about five main strands in our conception of reality. One very broad one is something is real just if it exists. Anything that exists is real. So if that tree exists, it’s real. If the number two exists, it’s real. I think that’s often what we mean. It’s also a little bit unhelpful as a criterion, because it just pushes back the question to what is it for something to exist? But it’s a start.

Then the second one is the one we’ve just been talking about the criterion of causal powers. This actually goes back to a one of Plato’s dialogue where the Iliadic stranger comes in and says for something to be real, it’s got to be able to make a difference. It’s got to be able to do something, that’s the causal power criterion. And so if you to be real, you’ve got to have effects. Some people dispute that’s necessary. Maybe numbers could be real, even if they don’t have effects, maybe consciousness could be real, even if it doesn’t have effects, but it certainly seems to be a plausible sufficient condition so that’s causal powers. Another one is mind independence, existing independently of the mind. There’s this nice slogan from Philip K Dick where he said that reality, something is real if when you stop believing in it, it doesn’t go away. Reality is that which when you stop believing in it, it doesn’t go away.

That’s basically to say its existence doesn’t depend on our beliefs. Some things such that their existence depends on our beliefs. I don’t know the Easter bunny or something, but more generally I’d say that some things have existence that depends on our minds. Maybe a Mirage of some water up ahead. That basically depends on there being a certain conscious experience in my mind, but there are some things out there independent of my mind that aren’t all in my mind, that don’t just depend on my mind. And so this leads to the third criteria and something is real when it doesn’t wholly depend on our minds, it’s out there independently of us.

Now this is a controversial criterion. People think that somethings like money may be real, even though it largely depends on our attitudes towards money. Our treating something as money as part of what makes it money. And actually in the Harry Potter books, I think its Dumbledore has a slogan that goes the opposite way of Philip K Dick’s. At one point towards the end of the novels, Dumbledore says to Harry, Harry says, “ah, but none of this is real and this is all just happening inside my head” and Dumbledore says something like, “just because all this is happening inside your head, Harry, why do you think that makes it any less real?”

So I don’t know. There is a kind of mental reality you get from the mind, but at any way, I think mind independence is one important thing that we haven’t often have in mind when we talk about reality. A fourth one is that we sometimes talk about genuineness or authenticity. And one way to get at this is we often talk about not just whether an object is real, but whether it’s a real something like maybe you have a robot kitten, okay, it’s a real object. Yes. It’s a real object. It’s a genuine object with causal powers out there independently of us. But is it a real kitten? Is it a real kitten? Most people would say that, no, a robot kitten maybe it’s a real object, but it’s not a real kitten. So it’s not a genuine, authentic kitten.

More generally for any X we can ask, is this a real X? And that’s this criterion of genuineness, but then maybe the deepest and most important criterion for me is the one of not, basically something is real if it’s not an illusion, that is if it’s rough the way it seems to be. It seems to me that I’m in this environment, there are objects all around me in space with certain colors. There’s a tree out there and there’s a pond. And roughly I’d say that things are, all that’s real if there are things out there roughly as they seem to be, but if all this is an illusion, then those things are not real. So then we say things are real. If they’re not an illusion, if they’re roughly, as they seem to be. So one thing I then do is to try to argue that for the simulation hypothesis, at least if we’re in a simulation, then the objects we perceive are real in all five of those senses, they have causal powers. They can do things they’re out there independently of our minds. They exist. They’re genuine.

That’s a real tree, at least by what we mean by tree. And they’re not illusions. So five out of five on what I call the reality checklist, ordinary virtual reality, I want to say it gets four out of five. The virtual objects we interact with are they’re still digital objects with causal powers out there independently of us. They exist. They needn’t be illusions. I argue that at length that your experiences in VR needn’t be illusions. You can correctly perceive a virtual world as virtual, but arguably they’re not at least genuine. Maybe for example, the virtual kitten that you interact with in VR. Okay, it’s a virtual kitten, but it’s not a genuine kitten anymore than the robot kitten is. So maybe virtual tables are not, at least in our ordinary language, genuine tables. Virtual kittens are not genuine kittens, but they’re still real objects, but maybe there’s some sense in which they fail one of the five criteria for reality. So I would say ordinary virtual realities, at least as we deal with them now may get to four out of five or 80% on the reality checklist.

It’s possible that our language might evolve over time to eventually count virtual chairs as genuine chairs and virtual kittens as genuine kittens. And then we might be more VR inclusive in our talk. And then maybe we’d come to regard virtual reality is five out of five on the checklist. But anyway, that’s the rough way I ended up breaking on these notions into at least five. And of course, one way to come back is to say, ah, you’ve missed the crucial notion of reality actually, to be real requires this and VR is not real in that sense. I just read a review of the book where someone said, ah, look obviously VR isn’t real because it’s not part of the base level of reality. The fundamental outer shell of reality. That’s what’s real. So I guess this person was advocating. To be real you’ve got to be part of the base fundamental outer shell of reality. I mean, I guess I don’t see why that has to be true.

Lucas Perry: I mean, isn’t it though?

David Chalmers: Well.

Lucas Perry: It’s implemented on that.

David Chalmers: Yeah. It’s true so that’s one way to come back to this is to say the digital objects ultimately do exist in the outer shell. They’re just diverse.

Lucas Perry: They’re undivided from the outer shell. They just look like they’re just like can be conceptualized as secondary.

David Chalmers: Yeah, no, it is very much continuous with, I want to say the very least they’re on a par with like micro universes. I mean we have people talk now about, say baby universes. Growing up in black holes, inside a larger universe and people take that seriously and then we’d still say, okay, well this universe is part of this universe and that part of the universe can be just as real as the universe as a whole. So I don’t think, yeah. So I guess I don’t think being the whole universe is required to be real. We’ve got to acknowledge reality to parts of the world.

So we have kind of like a common sense ontology. A common sense view of the world and to me it seems like that’s more Newtonian feeling science evolves and then we get quantum mechanics. And so something you describe you explore in your book is this difference between I forget what you call it, like the conventional view of the world. And then, oh, sorry, the manifest in the scientific image is what you call it. And part of this manifest image is that it seems like humans’ kind of have like our common sense ontology is kind of platonic.

So how would you describe the common sense view of what is real?

David Chalmers: Yeah, I talk about the garden of Eden, which is our naive pre-theoretical sense of the world before we’ve started doing science and developing a more sophisticated view. I do think we have got this tendency to think about reality as like yeah, billiard balls out there and solid objects, colored objects out there in a certain space, an absolute three-dimensional space with one dimension of time. I think that’s the model of reality we had in the garden of Eden. So yeah, one of the conceits in the book is well in the garden of Eden things actually were that way. There were three absolute dimensions of space and one dimension of time objects were rock solid. They were colored the way I marked this in the book is by capital letters, say in the garden of Eden, there was capital S “Space” and capital T “Time” where objects were capital S “Solid” and capital C “Colored.”

They were capital R “Red” and capital G “Green.” And maybe there was capital G “Good” and bad and capital F “Free will” and so on. But then we develop the scientific view of the world. We eat from the tree of knowledge. It gives us knowledge of science and then, okay, well, the world is not quite like that naive conception implied there’s no, there’s four dimensional space time without an absolute space or a time. Objects don’t seem to have these primitive colors out there on their services. They just have things like reflectance properties that reflect light in a certain way that affects our experience in a certain way. Nothing is capital S “Solid.” The objects are mostly empty space, but they still manage to resist penetration and then the right way. So I think of this as the fall for Eden. And for many things we’ve gone from capital S “Space” to lowercase S “space.” We’ve gone from capital S “solidity” to lowercase S “solidity.”

And one thing that I think goes on here is that we’ve moved from kind of a conception of these things as primitive. Primitive space and primitive colors is just like redness out there on the surface of things, what I call primitivism to, rather to a kind of functionalism where we understand things in terms of their effects. To be red now is not to have some absolute intrinsic quality of redness, but it’s to be such as to affect us to produce certain experiences to look red. To be solid is not to be absolutely intrinsically solid, but to interact with other objects in such a way that they’re solid.

So I think in general, this goes along with moving from a conception of reality as all these absolute intrinsic properties out there to a much more structuralist conception of reality here where what matters for things being real is the right patterns of causal interaction with each other of entities with each other. I’m not saying all there is to reality is structure. My own view is that consciousness in particular is not just reducible to this kind of abstract structure consciousness does in fact have some intrinsic qualities and so on. So I do think that’s important too, but I do think in general, the move from the naive conception to the scientific conception of reality has often involved going from these kind of a conception of these primitive intrinsic qualities to a more structural conception of reality.

Lucas Perry: Right. So I imagine that many of the people who will resist this thesis in your book that virtual reality is genuine reality, maybe coming at it from some of these more common sense intuitions about what it means for something to be real, like red as a property that’s intrinsic on the surface of a thing. How do you see your book, so are there like common sense intuitions or misconceptions that you see your book as addressing?

David Chalmers: I guess I do think, yeah. Many people do find it as common sense that virtual reality is not full scale reality. First class reality. It doesn’t live up to our ordinary conception of reality. And sometimes I think they may have in mind this Edenic conception of reality, the way it was in the Garden of Eden to which my reply is. Yeah. Okay. I agree. Virtual reality does not have everything that we had in the Garden of Eden conception of reality, but neither does ordinary physical reality, even in the kind of physical reality developed in light of science, it’s not the garden of Eden picture of reality anymore. We’ve lost absolute space and absolute time. Now we’ve lost absolute colors and absolute solidity. What we have is now this complex mathematical structure of entities these interacting at a deep level.

I mean, the further you look, the more evanescent it gets, quantum mechanics is just this it’s wave function where objects don’t need to have determinate possessions, and who knows what’s going on there in string theory and other quantum gravity theories, it looks like space may not be fundamental at all. People have entertained the idea that time is not fundamental at all. So I think a physical reality in a way it’s, I’m saying virtual reality is genuine in reality, but one way to paraphrase that is virtual reality is just as real as physical reality. If you want to hear that by saying, well, physical reality is turned out to be more like virtual reality, then I wouldn’t necessarily argue with that physical reality is not the Garden of Eden billiard ball conception of reality anymore.

It’s this much more evanescent thing, which is partly characterizable by, it’s just playing all these, having the right kind of a certain kind of structure. And I think all that we can find in virtual reality. So yeah. So one thing I would do to this person questioning is to say, well, what do you think even about physical reality in light of the last hundred years or so of science?

Lucas Perry: Yeah. The reviewer’s comments that you mentioned come off to me as kind of being informed by the Eden view.

David Chalmers: Yeah. I think it’s right. It’s quite common that’s really what it is. It’s our naive conception of reality and what reality is like, but yeah, maybe then it’s already turned out that the world is not real in that sense.

Lucas Perry: One thing I’d like to pivot here into is exploring value more. How do you see the question of value fitting into your book? There’s this other central thesis here that you can live a good life in virtual reality, which seems to go against people’s common intuitions that you can’t. There’s this survey about whether or not people would go into experience machines and most people wouldn’t.

David Chalmers: Yeah, Nozick had this famous case of the experience machine, where your body’s in a tank, and you get all these amazing experiences of being highly successful. Most people say they wouldn’t enter the experience machine. I think of professional philosophers on a survey we did, maybe 15% said they would enter and 70 odd percent said they wouldn’t. And a few agnostic. The experience machine though, and many people have treated that as a model for VR in general. But I think the experience machine as Nozick described it, is actually different from VR in some respects. One is that very important respect is that the experience machine seems to be scripted, seems to be pre-programmed you go in there and your life will live out script. You get to become world champion, but it wasn’t really anything you did. That was just the script playing itself out. Many people think that’s fake. That’s not something I actually did. It was just something that happened to me.

VR by contrast, you go into VR, even an ordinary video game, you still got some degree of free will. You’re to some extent controlling what happens. You go into Second Life, or Fortnite whatever basically, you’ve got all kinds of it’s not scripted. It’s not pre-programmed, it’s open ended. I think the virtual worlds of the future will be increasingly open ended. I don’t think worries about the experience machine tend to undermine virtual worlds. More generally, I think I want to argue that yeah, virtual worlds can basically be on a par with physical worlds, especially once we’ve recognized that they needn’t be illusions, they needn’t be pre-programmed and so on. Then what are they missing? I think you’ve got what’s important to a good life? Maybe consciousness, the right subjective experiences. Also, relationships, very, very important. But I think in the VR certainly at least in a multi-user VR where many people are connected.

That’s another thing about the experience machine, it’s just you, presumably who’s conscious. But in a VR with I’m assuming a virtual world with many conscious beings, you can have relationships with them and get the social meaning of your life. That way knowledge and understanding, I think you can come to have all those things in VR. I think basically all the determinants of a good life, it’s hard to see what’s in principle missing in VR. There are some worries. Maybe if you want a fully natural life, a life, which is as close to nature as possible, VR is not going to do it because it’s going to be removed from nature. But then many of us live in cities or spend most of our time indoors. That’s also removed from nature and it’s still compatible with a meaningful life. There are issues about birth and death, which it’s not obvious how genuine birth and death will work at least in near term virtual worlds.

Maybe once there’s uploading, there’ll be birth and death in virtual worlds if the relevant creatures are fully virtual. But you might think if virtual was lack birth and death, there are aspects of meaning that they lack. I don’t want to say they’re exactly on a path with physical reality and all respects, but I’d say that virtual realities can at least have the prime determinants of a good and meaningful life. It’s not to say that life in virtual reality going to be wonderful. They may well be awful just as life in physical reality could be awful. But my thesis is roughly that at least the same range of value from the wonderful to the awful, is possible in virtual reality, just as it is in physical reality.

Lucas Perry: It sounds like a lot of people are afraid that they’ll be losing out on some of the important things you get from natural life, if virtual life were to take over?

David Chalmers: What are the important things you have in mind?

Lucas Perry: You mentioned people want to be able to accomplish things. People want to be a certain sort of person. People want to be in touch with a deeper reality.

David Chalmers: I certainly think in VR, you can be a certain person, very characteristic. With your own personal traits, you can have transformative experiences in virtual reality. Probably you can develop as a person. You can certainly have achievements in VR.

People who live and spend a lot of time, long term in worlds like second life certainly have real achievements, real relationships. Being in touch with a deeper reality, if by a deeper reality, you mean nature. In VR you’re somewhat removed from nature, but I think that’s somewhat optional.

In the short term at least, there are things like the role of the body, in existing VRs embodiment is extremely primitive. You’ve got these avatars, but our relationship with them is nothing like our relationship with our physical body. Things like eating, drinking, sex, or just physical companionship and so on. There’s not genuine analogs for those in existing VR. Maybe as time goes on, those things will become better. But I can imagine people thinking I value experiences of my physical body, and real eating and drinking and sex and companionship and so on and physical bodies.

But I could also imagine other people saying actually in VR now, in 200 years time people will say we’ve got these virtual bodies, which are actually amazing. Can do all that and give you all those experiences and much more and hey, you should try this. Maybe different people would prefer different things. But I do think to some considerable extent, thoughts about the body may be responsible for a fair amount of resistance to VR.

Lucas Perry: Could you talk a little bit about the different kinds of technological implementations of virtual reality? Whether it be uploading, or brains connected to virtual realities.

David Chalmers: Right now the dominant virtual worlds are not even VR at all of course. The virtual worlds people use the most now are video game style worlds typically on desktop or mobile computers on 2D screens.

But immersive VR is picking up speed fast with virtual reality headsets, like the Oculus Quest and they’re still bulky and somewhat primitive. But they’re getting better every year and they’ll gradually get less bulky and more primitive with more detail, better images and so on.

The other form factor, which is developing fast now is the augmented reality form with something like glasses, or transparent headsets that allow you to see the physical world, but also project virtual objects among the physical world.

Maybe it’s an image of someone you’re talking to. Maybe it’s just some information you need for dealing with the world. Maybe it’s a Pokemon Go creature you’re trying to acquire for your digital collection.

That’s the augmented reality form factor in glasses. A lot of people think that over the next 10 or 20 years, the augmented and virtual reality form factors could converge. Eventually we’ll be able to maybe have a set of glasses that could project digital objects into your environment, based on computer processes.

Maybe you could dial maybe a slider, which you go all the way down to dial out the physical world, be in a purely virtual world. Dial all the way up to be in a purely physical world, or in between, have elements of both.

That’s one way the technology seems to be going. The longer term there’s the possibility of bringing in brain computer interfaces. I think VR with standard perceptual interfaces works pretty well for vision and for hearing. You can get pretty good visual and auditory experiences from VR headsets, but embodiment is much more limited via sense of your own body.

But maybe once brain computer interfaces are possible, then there’ll be ways of getting elements, these computational elements to interact directly with bits of your brain. Whether it’s say visual cortex, auditory cortex for vision and hearing, or for the various aspects of embodied experience processed by the parts of the brain responsible for bodily experience.

Maybe that could eventually give you more authentic bodily experiences. Then eventually, bits of the potentially all kinds of computational circuitry could come to be embedded with brain circuitry in terms of circuitry, which is going to be partly biological and partly digital.

In the long term of course, there’s the prospect of uploading, which is the uploading the brain entirely to a digital process. Maybe once our brains are wearing out, we’ve replaced some of them with silicon circuitry, but you want to live forever upload yourself completely.

You’re running on digital circuitry. Of course, this raises so many philosophical issues. Will it still be me? Will I still be conscious? And so on. But assuming that it is possible to do this and have conscious beings and with this digital technology, then that being could then be fully continuous with the rest of the world.

That would just open up so much potential for new virtual reality, combined with new cognitive process, possibly giving rise to experiences that become now even imagine. Now this is very distant future, I’m thinking 100 plus years who knows.

Lucas Perry: You have long AGI timelines.

David Chalmers: This all does interact with AGI. I’m on record as 70% chance of AGI within a century. Maybe that’s sped up a bit.

Lucas Perry: You have shorter timelines.

David Chalmers: As far as this interacts with AI, I’m maybe on 50 years mean expected value for years until AGI. Once you go to AGI, all this stuff ought to have happened pretty fast. Maybe there’s a case for saying that within a century is conservative.

Lucas Perry: For uploads?

David Chalmers: Yeah, for uploads. I think once you go to AGIs, uploads are presumably-

Lucas Perry: Around the corner?

David Chalmers: … uploads are around the corner. At least if you believe like me, that once you go to AGI, then you’ll have AGI plus, and then you’ll have AGI plus, plus super intelligence. Then the AGI plus, plus is not going to have too much trouble with uploading technology and the like.

Lucas Perry: How does consciousness fit in all this?

David Chalmers: One very important question for uploading is whether uploads will even be conscious. This is also very relevant to thinking about the simulation hypothesis. Because if computer simulations of brains are not conscious, then it looks like we can rule out the simulation hypothesis, because we know we are conscious.

If simulations couldn’t be conscious, then we’re not simulations. At least the version of the simulation hypothesis, where we are part of the simulation could then be ruled out.

Now as it happens, I believe that simulations can be conscious. I believe consciousness is independent of substrate. It doesn’t matter whether you’re up and running on biology or on silicon, you’re probably going to be conscious.

You can run these familiar thought experiments, where you replace say neurons by silicon chips, replace biology by digital technology. I would argue that consciousness will be preserved.

That means at the very least gradual uploading, where you upload bits of your brain lets say a neuron at a time. I think that’s a pretty plausible way to preserve consciousness and preserve identity. But if I’m wrong about that and I could be, because nobody understands consciousness.

If I’m wrong about that, then uploads will not be conscious and these totally simulated worlds that people produce could end up being worlds of zombies. That’s at least something to worry about.

It’d be certainly risky to upload everybody to the cloud, to digital processes. Always keep some people anchored in biology just in case consciousness does require biology, because it’d be a rather awful future to have a world of super intelligent, but unconscious zombies being the only beings that exist.

Lucas Perry: I’ve heard from people who agree with substrate independence that digital or classical computer can’t be conscious. Are you aware of responses like that? Slash do you have a response to people who agree that consciousness is substrate independent, but the classical digital computers can’t be conscious.

I’m not sure what their exact view is, but something like the bits don’t all know about all the other bits. There’s no integration to create a unified conscious experience.

David Chalmers: The version of this I’ve heard I’m most familiar with, comes from Giulio Tononi’s Integrated Information Theory. Tononi and Christof Koch have argued that processes running on classical computers that is on von Neumann architectures cannot be conscious.

Roughly because von Neumann architectures have this serial core that everything is run through. They argue that this doesn’t have the property that Tononi calls integrated information and therefore is not conscious.

Now I’m very dubious about these arguments. I’m very dubious about a theory that says this serial bottleneck would undermine consciousness. I just think that’s all part of the implementation.

You could still have 84 billion simulated neurons interacting with each other. The mere fact that their interactions are mediated by a common CPU, I don’t see why that should undermine consciousness.

But if they’re right then fine, I’d say they’ve just discovered something about the functional organization that is required for consciousness. It needs to be a certain parallel organization as opposed to this serial organization.

But if so, you’re still right, it’s still perfectly substrate independent. As long as we upload ourselves not to a von Neumann simulation, but to a parallel simulation, which obviously it’s going to be the most powerful and efficient way to do this anyway, then uploading ought to be possible.

I guess another view is that consciousness could turn out to require to rely on quantum computation in a certain essential way. A mere classical computer might not be conscious, whereas quantum computers could be.

If so, that’s very interesting, but I would still imagine that all that would also be substrate independent and for uploading them, we just need to upload ourselves to the right quantum computer. I think those points while interesting, don’t really provide fundamental obstacles to uploading with consciousness here.

Lucas Perry: How do you see problems in the philosophy of identity fitting in here into virtual reality? For example with Derek Parfit’s thought experiments.

David Chalmers: Parfit had these famous thought experiments about the teletransporter from Star Trek, where you duplicate your body. Is that still me at the other end? The uploading cases are very similar to that in certain respects.

The teletransporter, you’ve got so many different cases. You’ve got is the original still around, then you create the copy? What if you create two copies? All these come up in the uploading case too.

There’s destructive uploading where we destroy the original, but create an upload. There’s non-destructive uploading, where we keep the original around, but also make an upload. There’s multiple copy uploading and so on.

In certain respects, there’s very much analogous to the teleporter case. The change is that we don’t duplicate the being biologically. We end up with a silicon isomorph, rather than a biological duplicate.

But aside from that, they’re very similar. If you think that silicon isomorph can be just as conscious as biological beings, maybe the two things roughly go together.

The same puzzle cases very much arise. Just say the first uploads are non-destructive, we stay around and we create uploaded copies. Then the tendency is going to be to regard the uploads is very different people from the original.

If the first uploads are destructive, you make copies while destroying the original. Maybe there’s going to be much more of a tendency to regard the uploads as being the same person as the original.

If we could make multiple uploads all the time, then there’ll be maybe a tendency to regard uploads as second class citizens and so on. The thought experiments here are complex and wonderful.

I tend myself to be somewhat sympathetic with Parfit’s deflationary views of these things, which is there may not be very much absolute continuity of people over time. Per the very concept of personal identity, maybe one of these Edenic concepts, that we actually persist through time as absolute subjects.

Maybe all there are just different people at different times that stand in psychological and memory and other continuity relations to each other. Maybe that’s all there is to say.

This gets closer now to Buddhist style, no self views, at least with no identic capital S “Self,” but I’m very unsure about all of these matters about identity.

Lucas Perry: How would you upload yourself?

David Chalmers: I think the safest way to do it, would be gradually. Replace my neurons one at a time by digital circuits. If I did it all at once, destroy the original creator uploaded copy, I’d worry that I’d be gone. I don’t know that. I just worry about it a bit more.

To remove that worry, do it gradually and then I’m much less worried that I’d be gone. If I can do it a bit at a time I’m still here. I’m still here. I’m still here. To do it with maximum safety, maybe I could be conscious throughout, with a continuous stream of consciousness throughout this process.

I’m here watching the operation. They change my neurons over and in that case, then it really seems to me as if there’s a continuous stream of consciousness. Continuous stream of consciousness seems to either, I don’t know if it guarantees identity over time, but it seems pretty close to what we have in ordinary reality.

We’re having continuous stream of consciousness overtime, seems to be the thing that goes along with what we usually think of as identity over time. It’s not required because we can fall asleep and arguably lose consciousness and wake up.

Most people would say we’re still the same person, but still being continuously conscious for a period, seems about as good a guarantee as you’re going to get of being the same person. Maybe this would be the philosophically safest way to upload.

Lucas Perry: Is sleeping not an example that breaks that?

David Chalmers: I’m not saying it’s a necessary condition for a personal identity, just a sufficient condition, just plausibly continuous consciousness sufficient for identity over time. So far there is identity over time. Yes, probably too stronger condition.

It maybe you can get identity from much weaker relations, but in order to be as safe as possible, I’m going to go with the strongest sufficient condition.

Lucas Perry: One neuron at a time.

David Chalmers: Maybe 10 neurons at a time. Maybe even a few columns at a time. I don’t know.

Lucas Perry: Do you think Buddhist’s that realize no self, would be more willing to upload?

David Chalmers: I would think so and I would hope so. I haven’t done systematic polls on this. Now I’m thinking I’ve got to get the data from the last PhilPapers survey and find views on uploading, which we asked about out. We didn’t ask about are you Buddhist? We didn’t ask do you for example, specialize in Asian philosophy?

I wonder if there could at least be a correlation between specialization and Asian philosophy, and certain views about uploading. Although it’ll be complicated by the fact that this will also include Hindu people who very much believe in absolute self, and Chinese philosophers who have all kinds of very different views. Maybe it would require some more fine grained survey analysis.

Lucas Perry: I love that you do these surveys. They’re very cool. Everyone should check them out. It’s a really cool way to see what philosophers are thinking. If you weren’t doing them, we wouldn’t know.

David Chalmers: Go to philsurvey.org. This later survey in 2020, we surveyed about 2000 odd philosophers from around the world, on 100 different philosophical questions like God, theism, or atheism, mind, physicalism or non-physicalism and so on.

We actually got data about what professional philosophers tend to believe. You can look at correlations between questions, correlations with area, with gender, with age and so on. It’s quite fascinating. Go to philsurvey.org you’ll find the results.

Lucas Perry: Descartes plays a major role in your book, both due to his skepticism about the external world, and whether or not it is that we know anything about it. Then there’s also the mind body problem, which you explore. Since we’re talking about consciousness and the self, I’m curious if you could explain how the mind body problem fits in all this?

David Chalmers: In a number of ways. Questions about the mind are not front and center in this book, but they come up along the way in many different contexts. In the end, actually part five of the book has three chapters on different questions about the mind.

One of them is the question we’ve just been raising. Could AI systems be conscious? Could uploading lead to a conscious being and so on? That’s one chapter of the book. But another one just thinks about mind, body relations in more ordinary virtual realities.

One really interesting fact about existing VR systems, is that if you actually look at virtual worlds, that Cartesian thought that Descartes thought there’s a physical world that the mind interacts with, and the mind is outside the physical world, but somehow interacts with it.

You look at a virtual world, virtual worlds often have their own physics and their own algorithmic processes that govern the physical processes in the virtual world. But then there’s this other category of things, users, players, people who are using VR and they are running on processes totally outside the virtual world.

When I enter a VR, the VR has its own physics, but I am not subject to that physics. I’ve got this mind which is operating totally outside the virtual world. You can imagine if somebody grew up in a virtual world like this.

If Descartes grew up in a virtual world, we’ve got an illustration where Descartes grows up inside Minecraft and gets into an argument with Princess Elizabeth, about whether the mind is outside this physical world interacting with it.

Most people think that the actual Descartes was wrong, but if we grew up in VR, Descartes would be right. He’d say yeah, the mind is actually something outside. He’d look at the world around him and say, “This is subject to its physics and so the mind is just not part of that. It’s outside all that. it exists in another realm and interacts with it.”

There’s a perspective of the broader realm, which all this looks physical and continuous. But at least from the perspective of the virtual world, it’s as if Descartes was right.

That’s an interesting illustration of a Cartesian interaction of dualism, where the mental and the physical are distinct. It shows a way at which something like that could turn out to be true under certain versions of the simulation hypothesis, say with brains interacting with simulations.

Maybe even is true of something isomorphic of it is true, even in ordinary virtual realities. At least that’s interesting and making sense of this mind body interaction, which is often viewed as unscientific or non naturalistic idea. But here’s a perfectly naturalistic version of mind body dualism.

Lucas Perry: I love this part and also found it surprising for that reason, because Cartesian dualism, it always feels supernatural, but here’s a natural explanation.

David Chalmers: One general theme in this book is that there’s a lot of stuff that feels supernatural, but once you look at it through the lens of VR, needn’t be quite so supernatural, looks a lot more naturalistic. Of course, the other example is God. If your creator is somebody, a programmer in the next universe up, suddenly God doesn’t look quite so supernatural.

Lucas Perry: Magic is like using the console in our reality to run scripts on the simulators world, or something like that.

David Chalmers: This is naturalistic magic. Magic has to obey its own principles too. There’s just different principles in the next universe up.

Lucas Perry: Clearly it seems your view is consciousness is the foundation of all values, is that right?

David Chalmers: Pretty much. Pretty much. Without consciousness no value. I don’t want to say consciousness is all there is to value. There might be other things that matter as well, but I think you probably have to have consciousness to have value in your life.

Then for example, relations between conscious beings, relations between consciousness and the world can matter for value. Nozick’s experience machine tends to suggest that consciousness alone is not quite enough.

There’s got to be maybe things like actually achieving your goals and so on that matters as well. But I think consciousness is at the very core of what matters and value.

Lucas Perry: We have virtual worlds and people don’t like them, because they want to have an interaction with whatever’s like natural, or they want to be a certain kind of person, or they want the people in it to be implemented. They want them in real space, things like that.

Part of what makes being in Nozick’s experience machine unsatisfactory, is knowing that some of these things aren’t being satisfied. But what if you didn’t know that those things weren’t being satisfied? You thought that they were.

David Chalmers: I guess my intuition is that’s still bad. There’s this famous case that people have raised just say your partner is unfaithful to you, but it’s really important to you that your relationship be monogamous. However, your partnership, your partner, although professing monogamy has gone off and had relationships with all these other people.

You never know and you’re very happy about this and you go to your death without ever knowing. I think most people’s intuition is that is bad. That life is not as good, as one where the life was the way this person wanted it to be with the monogamous partner.

That brings out that having your goals or your desires satisfied, the world being the way you want it to be, that matters to how good and meaningful a life is. Likewise, I’d say that I think the experience machine is a more extreme example of that.

We really want to be doing these things. If I was to find out 100 years later that hey, any success I’d had in philosophy wasn’t because I wrote good books. It’s just because there was a script that said there’d be certain amounts of success and sales and whatever.

Then boy, that would render any meaning I’d gotten out of my life perfectly hollow. Likewise, even if I never discovered this, if I had the experience of having the successful life, but it was all merely pre-programmed, then I think that would render my life much less.

It’d still be meaningful, but just much less good than I thought it had been. That brings out that the goodness, or the value of one’s life depends on more than just how one experiences things to be.

Lucas Perry: I’m pushing on consequentialist or utilitarian sensibilities here, who might bite the bullet and say that if you didn’t know any of those things, then those worlds are still okay. One thing that you mentioned in your book is that your belief that virtual reality is good, is independent of the moral theory that one has. Could you unpack that a bit?

David Chalmers: I don’t know if it’s totally independent, but I certainly think that my view here is totally consistent with consequentialism and utilitarianism that says, what matters in moral decision making is maximizing good consequences, or maximizing utility.

Now, if you go on to identify the relevant good consequences, with conscious states like maximizing pleasure, or if you say all there is to utility, is the amount of pleasure. Then you would take a different view of the experience machine.

If you thought that all that there is to utility is pleasure and you say in the experience machine, I have the right amount of pleasure so that’s good enough. But I think that’s going well beyond consequentialism, or even utilitarianism.

That’s adding a very specific view of utility and is the one that the founders utilitarianism had, like Bentham and Mill. I would just advocate a broader view of consequentialism, or utilitarianism where there are values that go beyond value driving from pleasure, or from conscious experience.

For example, one source of value is having your desires satisfied or achieving your goals. I think that’s perfectly consistent with utilitarianism, but maybe more consistent with some forms than others.

Lucas Perry: Is having your values satisfied, or your preference satisfied, not just like another conscious state?

David Chalmers: I don’t think so, because you could have two people who go through exactly the same series of conscious states, but for one of whom their desires are satisfied, and for the other one, their desires are not satisfied, maybe they both think their desires are satisfied, but one of them is wrong. They both want their partners to be monogamous. One partner is monogamous and the other one is not. They might have exactly the same conscious states, but one has the world is the way they want it to be, and the other one, the world is not the way they want it to be.

This is what Nozick argued and others have argued in light of the experience machine is that, yeah, there’s a value maybe in desire, satisfaction that goes be on the value of consciousness, per se. I should say both of these views, even the pleasure centric view are totally consistent with my general view of VR. If someone says, “All that matters is experiences,” well, in a certain sense, great. That makes it even easier to lead a good life in VR. But I think if the dialectic is the other way around, even if someone rejects that view … I tend to believe there’s more that matters than just consciousness. Even if you say that, you can still have a good life in a virtual world.

I mean, there’ll be some moral views where you can’t. Just say you’ve got a biocentric view of what makes a life good. You got to have, somehow. the right interactions with real biology. I don’t know, then maybe certain virtual worlds won’t count as having the right kind of biology, and then they won’t count as valuable. So I wouldn’t say these issues are totally independent of each other, but I do think on plausible moral theories, yeah, very much going to be consistent with being able to have a good life in virtual worlds.

Lucas Perry: What does a really good life in a virtual world look like to you?

David Chalmers: Oh boy. What does a really good life look like to me? I mean, different people have different values, but I would say I get value partly from personal relationships, from getting to know people, by having close relationships with with my family, with partners, with friends, with colleagues. I get a lot of value from understanding things, from knowledge and understanding. I get some value from having new experiences and so on. And I guess I’d be inclined to think that in a virtual world, the same things would apply. I’d still get value from relationships with people, I’d still get value from knowledge and understanding, I’d still get value from new kinds of experience.

Now, there made ways in which VR might allow this to go beyond what was possible outside VR. Maybe, for example, there’ll be wholly new forms of experience that go way beyond what was possible inside physical reality, and maybe that would allow for a life which is better in some respects. Maybe it’ll be possible to have who knows what kind of telepathic experiences with other people that give you even closer relationships that are somehow amazing. Maybe it’ll allow immortality, where you can go on having these wonderful experiences for an indefinite amount of time, and that could be better.

I guess in the short term, I think, yeah, the kind of good experiences I’ll have in VR are very much continuous with the good experiences I’ll have elsewhere. It’s a way of I meet friends sometimes in VR, interact with them, talk with them, sometimes play games, sometimes communicate, maybe occasionally have a philosophy lecture or a conference then. So right now, yeah, what’s good about VR is pretty much continuous with a lot of what’s good about physical reality. But yeah, in the long them, there may be ways for it to go beyond.

Lucas Perry: What’s been your favorite VR experience so far?

David Chalmers: Oh boy, everything is fairly primitive for now. I enjoy a bunch of VR games, and I enjoy hanging out with friends. One enjoyable experience was I gave a little lecture of about VR, in VR, to a group of philosopher friends. And we were trying to figure out the physics of VR, of the particular virtual world we were in, which was on a app called Bigscreen.

Lucas Perry: Yeah.

David Chalmers: And yeah, you do things in Bigscreen, like you throw tomatoes, and they behave in weird ways. They kind of a baby laws of physics, but they kind of don’t, and the avatars have their own ways of moving. So we were trying to figure out the basic laws of Bigscreen, and we didn’t get all that far, but we figured out a few things. We were doing science inside a virtual world. And presumably, if we’d kept going, we could have gotten a whole lot further and gotten further into the depths of what the algorithms really are that generate this virtual world or that might have required a scientific revolution or two. So I guess that was a little instance of doing a bit of science inside a virtual world and trying to come to some kind of understanding, and it was at least a very engaging experience.

Lucas Perry: Have you ever played any horror games?

David Chalmers: Not really, no. I’m not much of a gamer, to be honest. I play some simple games like Beat Saber or, what is it? SUPERHOT. But that’s not really a horror game. Super assassins come after you, but what’s your favorite horror game?

Lucas Perry: I was just thinking of my favorite experience and it was probably … Well, I played Killing Floor once when I first got the VR, and it was probably the most frightening experience of my life. The first time you turn around and there’s and embodied thing that feels like it’s right in your face, very interesting. In terms of consciousness and ethics and value, we can explore things like moral patiency and moral agency. So what is your view on the moral status of simulated people?

David Chalmers: My own view is that the main thing, the biggest thing that matters for moral status is consciousness. So as long as simulated beings are conscious as we are, then they matter. Now maybe current non-player characters of the kind you find in video games and so on are basically run by very simple algorithms, and most people would think that those beings are not conscious, in which case their lives don’t matter, in which case it’s okay to shoot these current NPCs in video games.

I mean, maybe we’re wrong about that, and maybe they have some degree of consciousness and we have to worry. But at least the orthodox view here would be that they’re not, and even on a view that describes some consciousness, it’s probably a very simple form of consciousness. But if we look now to a longterm future where there are simulations of brains and simulated AGIs inside these simulated worlds with capacities equivalent to our own, I’d be inclined to think that these beings are going to be conscious like us. And if they’re conscious like us, then I think they matter morally, the way that we do, in which case, one should certainly not be indiscriminately killing simulated beings just because it’s convenient, or just indiscriminately creating them and, and turning them off. So I guess if we do get to the point where …

I mean, this applies inside and outside simulations. If we have robot style AGIs that are conscious and they have moral status like ours, if we have simulation style, AGIs, inhabiting simulations, they also have moral status, much like ours. Now it may be hard for, I’m sure there’s going to be a long and complicated path to actually keep that playing out in social and legal context, and there may be all kinds of resistance to granting simulations, legal rights, social status, and so on. But philosophically, I guess I think that, yeah, if they’re conscious like us, they have a moral status like ours.

Lucas Perry: Do you think that there will be simulated agents with moral status that are not conscious, for example? They could at least be moral agents and not be conscious, but in a society and culture of simulated things, do you think that there would be cases where things that are sufficiently complex, yet not conscious, would still be treated with moral patiency?

David Chalmers: It’s interesting. I’m inclined to think that any system that has human level behavior is likely to be conscious. I’m not sure that there are going to be cases of zombies that lack consciousness entirely, but behave in extremely sophisticated ways just like us. But I might be wrong. Just say Tononi and Koch are right and that no being running on von Neumann architecture is conscious, then yeah. Then it might be smart to develop those systems because they won’t have moral status, but they’ll still be able to do a lot of useful things. But yeah, would they still then be moral agents?

Well, yeah, presumably these behaviorally equivalent systems could do things that look a lot like making moral decisions, even though they’re not conscious. Would they be genuine agents if they’re not conscious? That maybe partly a verbal matter, but they would do things that at least look a lot like agency and making moral decisions. So they’d at least be moral quasi-agents. Then it’s an interesting question whether they should be moral patients, too. If you’ve got a super zombie system making moral decisions, does it deserve some moral respect? I don’t know. I mean, I’m not convinced that consciousness is the only thing that matters, morally. And maybe that, for example, intelligence or planning or reasoning carries some weight independent of consciousness.

If that’s the case, then maybe these beings that are not conscious could still have some moral status as moral patients, that is deserving to be treated well, as well as just moral agents, as well as just performing moral action. Maybe it would be a second class moral patiency. Certainly, if the choice was between, say, killing and being like that and killing an equivalent conscious being, I’d say, yeah, kill the unconscious one. But that’s not to say they have no moral status there.

Lucas Perry: So one of your theses that I’d like to hit on here as well was that we can never know that we’re not in a simulation. Could you unpack this a bit?

David Chalmers: Yeah. Well, this is very closely connected to these traditional questions in epistemology. Can you know you’re not dreaming now? Could you know that you’re not being fooled by an evil demon now? The modern tech version is, “Can you know you’re not in a simulation?” Could you ever prove you’re not in a simulation? And there’s various things people might say, “Oh, I am not in a simulation.” I mean, naively, “This can’t be a simulation because look at my wonderful kitten here. That could never be simulated. It’s so amazing.” But presumably there could be simulated kittens. So that’s not a decisive argument.

More generally, I’m inclined to think that for any evidence anyone could come up with that’s allegedly proof that we’re not in a simulation, that evidence could be simulated, and the same experience could be generated inside a simulated world. This starts to look like there’s nothing, there’s no piece of evidence that could ever decisively prove we’re not in a simulation. And the basic point is just that a perfect simulation would be indistinguishable from the world it’s the simulation of. If that’s the case, awfully hard to see how we could prove that we’re not in a simulation.

Maybe we could get evidence that we are in a simulation. Maybe the simulators could reveal themselves to us and show us the source code. I don’t know. Maybe we could stress test the simulation by running a really intense computer process, more advanced than before suddenly, and maybe it stresses out the simulation and leads to a bug or something. Maybe there are ways we could get evidence.

Lucas Perry: Maybe we don’t want to do that.

David Chalmers: Okay. Maybe that will shut us down.

Lucas Perry: That’ll be an x-risk.

David Chalmers: Yeah. Okay. Yeah. Maybe not a good idea. So there are various ways we could get evidence that we are in a simulation, at least in an imperfect simulation. But I don’t think we can ever get the evidence in the negative that fully proves that we’re not in a simulation. We can try and test for various imperfect simulation hypotheses, but if we get just ordinary the expected results, then it’s always going to be consistent with both. And there are various philosophers who tried to say, “Ah, there are things we could do to refute this idea.” Maybe it’s meaningless. Maybe we could rule it out by being the non-simulation hypothesis being the simpler hypothesis and so on.

So in the book, I try and argue none of those things were work either. And furthermore, once you think about the Bostrom style simulation argument, that says it may be actually quite likely that we’re in a simulation because most populations are likely, it seems pretty reasonable to think that most intelligent populations will develop simulation technology. Once you start the thinking that way, I think it makes it even harder to refute the simulation hypothesis, basically, because by this point, these simulation style hypotheses used to be science fiction cases, very distant from anything we have direct reason to believe in.

But as the technology is developing, these stimulation-style hypotheses become realistic hypotheses, ones which is actually very good reason to think are actually likely to be developed both in our world and in other worlds. And I think that actually makes these … That’s had the effect of making these Cartesian scenarios move from the status of science fiction to being live hypotheses, and I think that makes them even harder to refute. I mean, you can make the abstract point that we can never prove it without the modern technology. But I think once they actually exist, once this technology is an existing technology, it becomes all the harder to epistemologically dismiss.

Lucas Perry: You give some credence in your book for whether or not we live in a simulation. Could you offer those now?

David Chalmers: Yeah. I mean, of course, anything like this is extremely speculative. But basically, in the book, I argue that if there are many conscious human-like simulations, and we are probably simulations ourself, and then the question is, “Is it likely that there are many conscious human-like simulations?” And there’s a couple of ways that could fail. First, it could turn out that simulating beings like us or universes like ours is not even possible. Maybe the physics is uncomputable. Maybe consciousness is uncomputable. So maybe conscious human-like simulations like ours could be impossible. That’s one way this could fail to happen. That’s what I call a sim blocker. These are things that would block these simulations from existing. So one class of sim blockers is, yeah, simulations like this are impossible. But I don’t think that’s more than 50% likely. I’m actually more than 50% confident that simulations like this are possible.

The other class of sim blockers is, well, maybe simulations like this are possible, but for various reasons they’ll never be developed, or not many of them will be to developed. And this class of sim blockers includes the ones that Bostrom focuses on. For example, I think there’s two of them, either we’ll go extinct before we get to that level of technology where we can create simulations, or we’ll get there, but we’ll choose never to create them, or intelligence civilizations will choose never to create them. And that’s the other way this can go wrong is, yeah, these things are possible, but not many of them will ever be created. And I basically say, “Well, if these are possible, and if they’re possible, many of them will be created, then many of them will be created and we’ll get a higher probability, we’re in a simulation.”

But then I think, “Okay, so what are the probabilities of each of those two premises?” That conscious human-like simulations are possible? Yeah, I think that’s at least 50%. Furthermore, if they’re possible, will many of them be created? I don’t know. I don’t know what the numbers are here, but I guess I’m inclined to think probably my subjective credence is over 50% in that two, given that it just requires some civilizations who eventually create a whole lot of them.

Okay, so 50% chance of premise one, 50% chance of premise two. Let’s assume they’re roughly independent of each other. That gives us a 25% chance they’re both true. If they’re both true, and most beings are simulations, if most beings are simulations, and we’re probably simulations, putting all that together, I get roughly, at least, 25% that we’re in a simulation. Now there’s a lot of room for the numbers to go wrong. But yeah, to me, that’s at least very good reason, A, to take the hypothesis seriously, and B, just suggests if it’s at 25%, we certainly cannot rule it out. So that gives a a quasi-numerical argument that we can never know that we’re not in a simulation.

Lucas Perry: Well, one interesting part that seems to feed into the simulation argument is modern work on quantum physics. So we had Joscha Bach on who talked some about this, and I don’t know very much about it, but there is this debate over whether the universe is implemented in continuous numbers or non-continuous numbers. And if the numbers were continuous, then the universe wouldn’t be computable. Is that right?

David Chalmers: I‘m not quite sure which debate you have in mind, but yeah, it certainly is right, that if the universe maximally is doing a real-valued computation, then real-valued computations can only be approximated on finite computers.

Lucas Perry: Right.

David Chalmers: On digital computers.

Lucas Perry: Right. So could you explain how this inquiry into how our fundamental physics work informs whether or not our simulation would be computable?

David Chalmers: I mean, there’s many aspects of that question. One thing that some people have actually looked into, whether our world might involve some approximations, some shortcuts. So Zohreh Davoudi and some other physicists have tried to look at the math and say, “Okay, just say our simulation, say there was a simulation that took certain shortcuts. How would that show up empirically?” So it’s, okay, this is going to be an empirical test for whether there are shortcuts in the way our physics is implemented.

I don’t think anyone’s actually found that evidence yet, but, ah, there’s some principle evidence we could get of that. But there is the question of whether our world is fundamentally analog or digital, and if our world is fundamentally analog with perfectly precise, continuous, real values making a difference to how the universe evolves, yeah, then that can never be perfectly simulated on a finite digital computer. I would still say it can be approximated. And as far as we know, we could be living in a finite approximation to one of those continuous worlds, but yeah, maybe there could eventually be some empirical evidence of that.

Of course, the other possibility is we’re just running on an analog computer. If our physics is continuous and the physics of the next world up is continuous, maybe there will be analog computers developed with maximally continuous quantities, and we could be running on an analog computer like that. So I think even if the physics of our world turns out to be perfectly analog and continuous, that doesn’t rule out the simulation hypothesis. It just means we’re running on an analog computer in the next universe up.

Lucas Perry: Okay. I’m way above my pay grade here. I’m just recalling now, I’m just thinking of how Joscha was talking that continuous numbers aren’t computable, right? So you would need an analog computer. I don’t know anything about analog computers, but it seems to me like they-

David Chalmers: It’s hard to program analog computers because they require infinite precision, and we found out beings are not good at building things with infinite precision. But we could always just set a few starting values randomly and let the analog computation go from there. And as far as I can tell, there’s no evidence that we’re not living in a simulation that’s running on an analog computer like that.

Lucas Perry: I see. So if we discover our fundamental physics to be digital or analog, it seems like that wouldn’t tell us a lot about the simulation, just that the thing that’s simulating us might be digital or analog.

David Chalmers: In general, discovering things about our … I mean, the relationship between the physics of our world and the physics of the simulating world is fairly weak, especially if you believe in universal computation, any classical algorithm can be implemented in a vast variety of computers running on a vast variety of physics. But yeah, but there might be some limits. For example, if our world has a perfectly analog physics that cannot be simulated on a finite digital computer, could be simulated on an infinite digital computer, you can simulate analog quantities with infinite strings of bits, but not on a finite digital computer.

So yeah, discovering that our physics is digital would be consistent with the next universe up being digital, but also consistent with it being analog. Analog worlds can still run digital computers. I mean, it’d be very suggestive if we did actually discover digital physics in our world. I’m sure if we discovered that our physics is digital, that would then get a lot of people thinking, “Hey, this is just the kind of thing you’d expect if people are running a digital computer in the next universe app.” That might incline people to take the simulation hypothesis more seriously, but it wouldn’t really be any kind of demonstration.

Yeah. If we somehow discover that our physics is perfectly analog, I don’t really know exactly how we could discover that because at any given point, we’ll only have a finite amount of evidence, which will always be consistent with just being a very close approximation, but just say we could discover that our world runs analog physics. Yeah, then that would be inconsistent, whether this just being a digital simulation in the next universe up, but still quite consistent with it being a simulation running on a analog computer in the next universe up. I don’t know how that connects to Joscha’s way of thinking about this.

Lucas Perry: Yeah. I’m not sure. I’d love to see you guys talk-

David Chalmers: I hope it’s least consistent.

Lucas Perry: … about this.

David Chalmers: Has he written about this somewhere?

Lucas Perry: I’m not sure. There are lots of podcasts have been talking about it, though.

David Chalmers: Okay, cool.

Lucas Perry: Yeah. So we’ve gone over a lot here, and it leaves me not really trusting my common sense experience of the world. So pivoting a little bit here, back into the Edenic view of things … Sorry if I get the word that you used wrong, but it seems like you walk away from that with a view of imperfect realism. Is that right?

David Chalmers: Yeah. Imperfect realism is the perfect thing. Capital S “Solidity” doesn’t exist, but the lower case thing, small S “solidity,” does exist. An imperfect analog of what we initially believed in.

Lucas Perry: So how do you see the world now? Any differently? What is the world like to David Chalmers after having written this book? What is a person to you?

David Chalmers: I don’t know. I mean, I think there’s your everyday attitude towards the world and your theoretical attitude towards the world. And I find my everyday attitude towards the world isn’t affected that much by discoveries in philosophy, or in science for that matter. We mostly live in the manifest image. Maybe we even treat it a little bit like the Garden of Eden, and that’s fine. But then there’s this knowledge of what underlies it or what could underlie it. And that’s, yeah, once you start thinking philosophically, that gets mind boggling.

I mean, you don’t need to go to the simulation hypothesis or the virtual world to get that reaction. I mean, quantum mechanics is quite enough. Oh my God, we live in this world of the quantum wave function where nothing actually has these direct positions and possibly the wave functions collapsing, or possibly many worlds. And so I mean, boy, it’s just mind boggling. It is rather hard to integrate ordinary life in reality. So most of us just kind of go on living in the manifest image. Yeah, so once I start thinking about, “Yeah could we be in a simulation?” It’s got a similar kind of separateness, I guess.

Mostly I go on living in the manifest image and don’t factor this in. But I mean, it does open up all kinds of possibilities once you start thinking that there is maybe this reality plus of all these different levels of reality, like, “Could it be that someday it might be possible to escape this particular virtual world, or maybe when we die, does our code sometimes get uploaded by simulators to go hang out back in other levels of reality. Maybe there are naturalized versions of reincarnation or life after death, and I don’t want to say this is why I’m thinking about this stuff. It’s not for these quasi-religious reasons, but suddenly, possibilities that had seemed very far out possibilities to me, like life after death, at least come to seem a little bit closer and more like open possibilities than they’d seemed before. So that’s at least interesting.

Lucas Perry: One thing you bring up a bit in your exploration here is God. And all these things that you’re mentioning, they seem like science and philosophy coming back to traditionally religious ideas, but through a naturalistic exploration, which is quite interesting. So do you have any different thoughts on God after having written this book?

David Chalmers: It’s interesting. I’m not remotely religious, myself. I’ve always thought of myself as an atheist, but yeah, after writing this book, I’m at least … There is a version of God that I could at least take seriously. This is the simulator. They did, after all, create the world, this world. They may have a lot of power and a lot of knowledge of this world, as gods are meant to have. On the other hand, they’re quite unlike traditional gods. In some ways, the simulator needn’t be all good, needn’t be particularly wise. Oh, also didn’t create all of reality. It just created a little bit of reality. Maybe it’s a bit like what’s sometimes called a demiurge, the the local god, the under-boss god who created this world, but wasn’t the one in charge of the whole thing.

So yeah, maybe simulators are a bit more like demiurges. More importantly, I don’t think I’d be inclined to erect a religion around the simulation idea. Religions come with ethical practices and really changing your way of life. I don’t think there’s any particular reason to orient our ethics to a simulation. I mean, maybe you can imagine there’d be some practices that if we really believed we were in a simulation, or there’s a good chance of that, we should at least start doing some things differently. Maybe some people might want to try and attract the attention of the simulators. I don’t know. That’s all very speculative. So I don’t find myself …

I think the one moral of all this for me is that actually ethics and meaning and so on, actually, you don’t get your ethics or your meaning from who created you or from whether it’s a God or a simulator. Ethics and meaning comes from within. It comes from ourselves, our consciousness, and our interactions.

Lucas Perry: Do you take a line that’s similar to Peter Singer in thinking that that is like an objective rational space? Are you a moral realist or anti-realist about those things?

David Chalmers: I tend towards moral anti-realism, but I’m not sure. I find those issues very difficult. Yeah, I can get in the mood where, “Pain is bad,” just seems like an absolute fact.

Lucas Perry: Yeah.

David Chalmers: That’s just an objective fact. Pain is objectively bad. And then I get at least to some kind of value realism, if not moral realism. Some moods all go that way. Other moods, it’s just, yeah, it’s all a matter of our attitude towards it. Finally, it’s a matter of what we value. If somebody valued pain, it would be good for them. If they didn’t, it wouldn’t be. And I can go back and forth. I don’t have a fixed view of these matters.

Lucas Perry: Are there any questions that I haven’t asked you that you would’ve liked me to ask you?

David Chalmers: Not especially. You asked a lot of great questions, and there are a million others, but actually one interesting thing with this book coming out is getting to do a few of these, having a few of these conversations and seeing all the different questions and different aspects of the book that different people focused on. So, no. I think we’ve covered a lot of territory here, and yeah, these are a lot of cool things to think about.

Lucas Perry: All right. Well, I’m mindful of the time here, David. Thank you so much for all of your time. If people want to check you out, follow you, and get your new book, where are the best places to do that?

David Chalmers: Probably my website, which is consc.net. Consc, the first five letters of consciousness, or just do a search for my name. And then yeah, the book is … I’ve got a page for the book on my website, consc.net/reality, or just search for name of the book, Reality+. I’m not on Twitter or Instagram or any of those things, unfortunately. Maybe I should be one of these days, but for now, I’m not. But yeah, the book will be available January 25th, I guess. All good book sellers. So I hope some of your listeners might be interested to check it out.

Lucas Perry: All right. We’ll include links to all of those places in the description of wherever you might be listening or watching. Thank you so much, David. It’s always a pleasure speaking with you. I love hearing about your ideas, and it’s really a great book at an important time. I think just before all this VR stuff is about to really kick off, and with the launch of the metaverse. It’s really well timed.

David Chalmers: Oh, thanks, Lucas. This was all, yeah, a lot of fun to talk about this stuff with you.

Rohin Shah on the State of AGI Safety Research in 2021

  • Inner Alignment Versus Outer Alignment
  • Foundation Models
  • Structural AI Risks
  • Unipolar Versus Multipolar Scenarios
  • The Most Important Thing That Impacts the Future of Life


Watch the video version of this episode here

0:00 Intro

00:02:22 What is AI alignment?

00:06:45 How has your perspective of this problem changed over the past year?

00:07:22 Inner Alignment

00:15:35 Ways that AI could actually lead to human extinction

00:22:50 Inner Alignment and MACE optimizers

00:24:15 Outer Alignment

00:27:32 The core problem of AI alignment

00:29:38 Learning Systems versus Planning Systems

00:34:00 AI and Existential Risk

00:38:59 The probability of AI existential risk

01:04:10 Core problems in AI alignment

01:03:07 How has AI alignment, as a field of research changed in the last year?

01:05:57 Large scale language models

01:06:55 Foundation Models

01:15:30 Why don’t we know that AI systems won’t totally kill us all?

01:23:50 How much of the alignment and safety problems in AI will be solved by industry?

01:31:00 Do you think about what beneficial futures look like?

01:39:44 Moral Anti-Realism and AI

01:46:22 Unipolar versus Multipolar Scenarios

01:56:38 What is the safety team at DeepMind up to?

01:57:30 What is the most important thing that impacts the future of life?

Lucas Perry: Welcome to the Future of Life Institute Podcast. I’m Lucas Perry. Today’s episode is with Rohin Shah. He is a long-time friend of this podcast, and this is the fourth time we’ve had him on. Every time we talk to him, he gives us excellent overviews of the current thinking in technical AI alignment research. And in this episode he does just that. Our interviews with Rohin go all the way back to December of 2018. They’re super informative and I highly recommend checking them out if you’d like to do a deeper dive into technical alignment research. You can find links to those in the description of this episode. 

Rohin is a Research Scientist on the technical AGI safety team at DeepMind. He completed his PhD at the Center for Human-Compatible AI at UC Berkeley, where he worked on building AI systems that can learn to assist a human user, even if they don’t initially know what the user wants.

Rohin is particularly interested in big picture questions about artificial intelligence. What techniques will we use to build human-level AI systems? How will their deployment affect the world? What can we do to make this deployment go better? He writes up summaries and thoughts about recent work tackling these questions in the Alignment Newsletter, which I highly recommend following if you’re interested in AI alignment research. Rohin is also involved in Effective Altruism, and out of concern for animal welfare, is almost vegan.

And with that, I’m happy to present this interview with Rohin Shah. 

Welcome back Rohin. This is your third time on the podcast I believe. We have this series of podcasts that we’ve been doing, where you help give us a year-end review of AI alignment and everything that’s been up. You’re someone I view as very core and crucial to the AI alignment community. And I’m always happy and excited to be getting your perspective on what’s changing and what’s going on. So to start off, I just want to hit you with a simple, not simple question of what is AI alignment?

Rohin Shah: Oh boy. Excellent. I love that we’re starting there. Yeah. So different people will tell you different things for this as I’m sure you know. The framing I prefer to use is that there is a particular class of failures that we can think about with AI, where the AI is doing something that its designers did not want it to do. And specifically it’s competently achieving some sort of goal or objective or some sort of competent behavior that isn’t the one that was intended by the designers. So for example, if you tried to build an AI system that is, I don’t know, supposed to help you schedule calendar events and then it like also starts sending emails on your behalf to people which maybe you didn’t want it to do. That would count as an alignment failure.

Whereas if a terrorist somehow makes an AI system that goes and designates a bomb in some big city that is not an alignment failure, it is obviously bad, but the AI system did what its designer intended for it to do. It doesn’t count as an alignment failure on my definition of the problem.

Other people will see AI alignment as synonymous with AI safety. For those people, terrorists using a bomb might count as an alignment failure, but at least when I’m using the term, I usually mean, the AI system is doing something that wasn’t what its designers intended for it to do.

There’s a little bit of a subtlety there where you can think of either intent alignment, where you try to figure out what the AI system is trying to do. And then if it is trying to do something that isn’t what the designers wanted, that’s an intent alignment failure, or you can say, all right, screw all of this notion of trying, we don’t know what trying is. How can we look at a piece of code and say whether or not it’s trying to do something.

And instead we can talk about impact alignment, which is just like the actual behavior that the AI system does. Is that what the designers intended or not? So if the AI makes a catastrophic mistake where the AI thinks that this is the big red button for happiness and sunshine, but actually it’s the big red button that launches nix. That is a failure on impact alignment, but isn’t a failure on the intent alignment, assuming the AI legitimately believed that the button was happiness and sunshine, I think they said.

Lucas Perry: So it seems like you could have one or more or less of these in a system at the same time. So which are you excited about? Which do you think are more important than the others?

Rohin Shah: In terms of what do we actually care about? Which is how I usually interpret important, the answer is just like pretty clearly impact alignment. The thing we care about is, did the AI system do what we want or not? I nevertheless tend to think in terms of intent alignment, because it seems like it is decomposing the problem into a natural notion of like what the AI system is trying to do. And whether the AI system is capable enough to do it. And I think that is like actually natural division. You can in fact talk about these things separately. And because of that, it makes sense to have research organized around those two things separately. But that is a claim I am making about the best way to decompose the problem that we actually care about. And that is why I focus on intent alignment but what do we actually care about? Impact alignment, totally.

Lucas Perry: How would you say that your perspective of this problem has changed over the past year?

Rohin Shah: I’ve spent a lot of time thinking about the problem of inner alignment. So this was this shot up to… I mean, people have been talking about it for a while, but it shot up to prominence in I want to say 2019 with the publication of the mesa optimizers paper. And I was not a huge fan of that framing, but I do think that the problem that it’s showing is actually an important one. So I’ve been thinking a lot about that.

Lucas Perry: Can you explain what inner alignment is and how it fits into the definitions of what AI alignment is?

Rohin Shah: Yeah. So AI alignment, the way I’ve described it so far is just sort of like pretty, it’s just talking about properties of AI system. It doesn’t really talk about how that AI system was built, but if you actually want to diagnose at like give reasons why problems might arise and then how to solve them, you probably want to talk about how the AI systems are built and why they’re likely to cause such problems.

Inner alignment, I’m not sure if I like the name, but we’ll go with it for now. Inner alignment is a problem that I claim happens for systems that learn. And the problem is, maybe I should explain it with an example. You might have seen this post from LessWrong about bleggs and rubes. These bleggs are blue in color and tend to be egg-shaped in all the cases they’ve seen so far. Rubes are red in color and are cube-shaped, at least in all the cases they’ve seen so far.

And now suddenly you see a red egg-shaped thing, is it blegg or rube? Like in this case, it’s pretty obvious that there isn’t a correct answer and this same dynamic can arise in a learning system where if it is learning how to behave in accordance with whatever we are training it to do, we’re going to be training it on a particular set of situations. And if those situations change in the future along some axis that the AI system didn’t see during training, it may generalize badly. So a good example of this is, came from the objective robustness and deep reinforcement learning paper. They trained an agent on the CoinRun environment from Procgen. This is basically a very simple platformer game where the agent just has to jump over enemies and obstacles to get to the end and collect the coin.

And the coin is always at the far right end of the level. And so, you train your AI system on a bunch of different kinds of levels, different obstacles, different enemies, they’re placed in different ways. You have to jump in different ways, but the coin is always at the end on the right. And it turns out if you then take your AI system and test it on a new level of where the coin is placed somewhere else in a level, not all the way to the right, the agent just continues to jump over obstacles, enemies, and so on. Behaves very competently in the platformer game, but it just runs all the way to the right and then stays at the right or jumps up and down as though hoping that there’s a coin there. And it’s behaving as if it has the objective of go as far to the right as possible.

Even though we trained it on the objective, get the coin, or at least that’s what we were thinking of as the objective. And this happened because we didn’t show it any examples where the coin was anywhere other than the right side of the level. So the inner alignment problem is when you train a system on one set of inputs, it learns how to behave well on that set of inputs. But then when you extrapolate its behavior to other inputs that you hadn’t seen during training, it turns out to do something that’s very capable, but not what you intended.

Lucas Perry: Can you give an example of what this could look like in the real world, rather than in like a training simulation in a virtual environment?

Rohin Shah: Yeah. One example I like is, it’ll take a bit of setup, but I think it should be fine. You could imagine that with honestly, even today’s technology, we might be able to train an AI system that can just schedule meetings for you. Like when someone emails you asking for a meeting, you’re just like, here calendar scheduling agent, please do whatever you need to do in order to get this meeting scheduled. I want to have it, you go schedule it. And then it goes and emails a person who emails back saying, Rohin is free at such and such times, he like prefers morning meetings or whatever. And then, there’s some back and forth between, and then the meeting gets scheduled. For concreteness, let’s say that the way we do this, is we take a pre-trained language model, like say GPT-3, and then we just have GPT-3 respond to emails and we train it from human feedback.

Well, we have some examples of like people scheduling emails. We do supervised fine tuning on GPT-3 to get it started. And then we like fine tune more from human feedback in order to get it to be good at this task. And it all works great. Now let’s say that in 2023, Gmail decides that Gmail also wants to be a chat app. And so it adds emoji reactions to emails, and everyone’s like, oh my God, now there’s such a better, we can schedule a meeting so much better. We can just say, here, just send an email to all the people who are coming to the meeting and react with emojis for each of the times that you’re available. And everyone loves this. This is how people start scheduling meetings now.

But it turns out that this AI system, when it’s confronted with these emoji polls is like, it knows, it in theory is capable or knows how to use the emoji polls. It knows what’s going on, but it was always trained to schedule the meeting by email. So maybe it will have learned to like always schedule a meeting by email and not to take advantage of these new features. So it might say something like, hey, I don’t really know how to use these newfangled emoji polls. Can we just schedule emails the normal way? In our times this would be a flat out lie, but from the AI’s perspective, we might think of like, the AI was just trained to say whatever sequence of English words lead to getting a meeting scheduled by email. And it predicts that sequence of words will work well. Would this actually happen if I actually trained an agent this way? I don’t know, like it’s totally possible I would actually do the right thing, but I don’t think we can really rule out the wrong thing either, it seems. That also seems pretty plausible to me in this scenario.

Lucas Perry: One important part of this that I think has come up in our previous conversations is that we don’t know when there is always an inner misalignment between the system and the objective we would like for it to learn, because part of maximizing the inner aligned objective could be giving the appearance of being aligned with the outer objective that we’re interested in. Could you explain and unpack that?

Rohin Shah: Yeah. So in the AI safety community, we tend to think about ways that AI could like actually lead to human extinction. And so, the example that I gave does not in fact lead to human extinction. It is a mild annoyance at worst. The story that gets you to human extinction is one in which you have a very capable, superintelligent AI system. But nonetheless, there’s like, instead of learning the objective that we wanted, which might’ve been, I don’t know, something like be a good personal assistant. I’m just giving that out as a concrete example. It could be other things as well. Instead of acting as though it were optimizing that objective, it ends up optimizing some other objective and you don’t really want to give an example here because the whole premise is that it could be a weird objective we don’t really know.

Lucas Perry: Could you expand that a little bit more, like how it would be a weird objective that we wouldn’t really know?

Rohin Shah: Okay. So let’s take as a concrete example, let’s make paperclips, which has nothing to do with being a personal assistant. Now, why is this at all plausible? The reason is that even if this superintelligent AI system had the objective to make paperclips, during training, while we are in control, it’s going to realize that if it doesn’t do the things that we want it to do, we’re just going to turn it off. And as a result, it will be incentivized to do whatever we want until it can make sure that we can’t turn it off. And then it goes and builds its paperclip empire. And so when I say, it could be a weird objective, I mostly just mean that almost any objective is compatible with this sort of a story. It does rely on-

Lucas Perry: Sorry. I’m also curious if you could explain how the inner state of the system becomes aligned to something that is not what we actually care about.

Rohin Shah: I might go back to the CoinRun example, where the agent could have learned to get the coin. That was a totally valid policy it could have learned. And this is an actual experiment that people have run. So this one is not hypothetical. It just didn’t, it learned to go to the right. Why? I mean, I don’t know. I wish I understood neural nets well enough to answer those questions for you. I’m not really arguing for, it’s definitely going to learn, make paperclips. I’m just arguing for like, there’s this whole set of things it could learn. And we don’t know which one it’s going to learn, which seems kind of bad.

Lucas Perry: Is it kind of like, there’s the thing we actually care about? And then a lot of things that are like roughly correlated with it, which I think you’ve used the word for example before is like proxy objectives.

Rohin Shah: Yeah. So that is definitely one way that it could happen, where we ask it to make humans happy and it learns that when humans smile, they’re usually happy and then learns the proxy objective of make human smile and then it like, goes and tapes everyone’s faces so that they’re permanently smiling, that’s a way that things could happen. But I think I don’t even want to claim that’s what … maybe that’s what happens. Maybe it just actually optimizes for human happiness. Maybe it learns to make paperclips for just some weird reason. I mean, not paperclips. Maybe it decides, this particular arrangement of atoms in this novel structure that we don’t really have a word for is the thing that it wants for some reason. And all of these seem totally compatible with, we trained it to be good, to have good behavior in the situations that we cared about because it might just be deceiving us until it has enough power to unilaterally do what it wants without worrying about us stopping it.

I do think that there is some sense of like, no paperclip maximization is too weird. If you trained it to make humans happy, it would not learn to maximize paperclips. There’s just like no path by which paperclips somehow become the one thing it cares about. I’m also sympathetic to, maybe it just doesn’t care about anything to the extent of optimizing the entire universe to turn it into that sort of thing. I’m really just arguing for, we really don’t know crazy shit could happen. I will bet on crazy shit will happen, unless we do a bunch of research and figure out how to make it so that crazy shit doesn’t happen. I just don’t really know what the crazy shit will be.

Lucas Perry: Do you think that that example of the agent in that virtual environment, you see that as a demonstration of the kinds of arbitrary goals that the agent could learn and that that space is really wide and deep and so it could be arbitrarily weird and we have no idea what kind of goal it could end up learning and then deceive us.

Rohin Shah: I think it is not that great evidence for that position. Mostly because I think it’s reasonably likely that if you told somebody the setup of what you were planning to do, if you told an ML researcher or an RL, maybe specifically a deep RL researcher, the setup of that experiment and asked them to predict what would have happened, I think they probably would have, especially if you told them, “Hey, do you think maybe it’ll just run to the right and jump up and down at the end?” I think they’d be like, “Yeah, that seems likely, not just plausible, but actually likely.” That was definitely my reaction when I was first told about this result. I was like, oh yeah, of course that will happen.

In that case, I think we just do know…know is a strong word, ML researchers have good enough intuitions about those situations, I think, that it was predictable in advance. Though I don’t actually know if anyone who predicted it, did in advance. So that one, I don’t think is all that supportive of, it learns an arbitrary goal. We had some notion that neural nets care a lot more about position and simple functions of the action always go right rather than complex visual features like this yellow coin that you have to learn from pixels. I think people could have probably predicted that.

Lucas Perry: So we touched on definitions of AI alignment, and now we’ve been exploring your interest in inner alignment or I think the jargon is mesa optimizers.

Rohin Shah: They are different things.

Lucas Perry: There are different things. Could you explain how inner alignment and mesa optimizers are different?

Rohin Shah: Yeah. So a thing I maybe have not been doing as much as I should have is that, inner alignment is the claim that when the circumstances change, the agent generalizes catastrophically in some way, it behaves as though it’s optimizing some other objective than the one that we actually want. So it’s much more of a claim about the behavior rather than like the internal workings of the AI system that caused that behavior.

mesa-optimization, at least under the definition of the 2019 paper is talking specifically about AI systems that are executing an explicit optimization algorithm. So like the forward path of a neural net is itself an optimization algorithm. We’re not talking about creating dissent here. And then the metric that is being used in that, within the neural network optimization algorithm is the inner objective or sorry, the mesa objective. So it’s making a claim about how the AI system’s cognition is structured. Whereas inner alignment more broadly is the AI behaves in a catastrophically generalizing way.

Lucas Perry: Could you explain what outer alignment is?

Rohin Shah: Sure. Inner alignment can we be thought of as, suppose we got the training objective correct. Suppose the things that we’re training the AI system to do on the situations that we give it as input, we’re actually training it to do the right thing, then things can go wrong if it behaves differently in some new situations that we hadn’t trained it on.

Outer alignment is basically when the reward function that you specify for training the AI system is itself, not what you actually wanted. For example, maybe you want your AI to be helpful to you or to tell you true things. But instead you have, you train your AI system to go find credible looking websites and tell you what the credible looking websites say. And it turns out that sometimes the credible looking websites don’t actually tell you true things.

In that case, you’re going to get an AI that tells you what credible looking websites say, rather than an AI that tells you what things are true. And that’s in some sense, an outer alignment failure. You like even the feedback you were giving the AI system was pushing it away from telling you the truth and pushing it towards telling you what credible looking websites will say, which are correlated of course, but they’re not the same. In general, if you like give me an AI system with some misalignment and you ask me, was this a failure of outer alignment or inner alignment? Mostly I’m like, that’s a somewhat confused question, but one way that you can make it not be confused is you can say, all right, let’s look at the inputs on which it was trained. Now, if ever on an input on which we train, we gave it some wrong feedback where we were like the AI lied to me and I gave it like plus a thousand reward. And you’re like, okay, clearly that’s outer alignment. We just gave it the wrong feedback in the first place.

Supposing that didn’t happen. Then I think what you would want to ask is, okay, let me think about on the situations in which the AI does something bad, what would I have given counterfactually as a reward? And this requires you to have some notion of a counterfactual. When you’d write down a programmatic reward function, the counterfactual is a bit more obvious. It’s like, whatever that program would have output on that input. And so I think that’s the usual setting in which outer alignment has been discussed. And it’s pretty clear what it means there. But once you’re like training from human feedback, it’s not so clear what it means. What would the human have given us feedback on this situation that they’ve never seen before is often pretty ambiguous. If you define such a counterfactual, then I think I’m like, yes. Then I think I’m like, okay, you look at what feedback you would’ve given on the counterfactual. If that feedback was good actually led to the behavior that you wanted, then it’s an inner alignment failure. If that counterfactual feedback was bad, not what you would have wanted. Then it’s an outer alignment failure.

Lucas Perry: If you’re speaking to someone who was not familiar with AI alignment, for example, other people in the computer science community, but also policymakers or the general public, and you have all of these definitions of AI alignment that you’ve given like intent alignment and impact alignment. And then we have the inner and outer alignment problems. How would you capture the core problem of AI alignment? And would you say that inner or outer alignment is a bigger part of the problem?

Rohin Shah: I would probably focus on intent alignment for the reasons I have given before. It just seems like a more … I want to focus attention away from the cases where the AI is trying to do the right thing, but makes a mistake, which would be a failure of impact alignment. But I don’t think that is the biggest risk. I think in a super-intelligent AI system that is trying to do the right thing is extremely unlikely to lead to catastrophic outcomes though it’s certainly not impossible. Or at least more unlikely to lead to catastrophic outcomes, unlike humans in the same position or something. So that would be my justification for intent alignment. I’m not sure that I would even talk very much about inner and outer alignment. I think I would probably just not focus on definitions and instead focus on examples. The core argument I would make would depend a lot on how AI systems are being built.

As I mentioned inner alignment is a problem that according to me, is primarily learning systems, I don’t think it really affects planning systems.

Lucas Perry: What is the difference between a learning system and a planning system?

Rohin Shah: A learning system, you give it examples of things it should do, how it should behave and then changes itself to do things more in that vein. A planning system takes a formerly represented objective and then searches over possible hypothetical sequences of actions it could take in order to achieve that objective. And if you consider a system like that, you can try to make the inner alignment argument and it just won’t work, which is why I say that the inner alignment problem is primarily about learning systems.

Going back to the previous question. So the things I would talk about depend a lot on what sorts of AI systems we’re building, if it were a planning system, I would basically just talk about outer alignment, where I would be like, what if the formerly represented objective is not the thing that we actually care about. It seems really hard to formally represent the objectives that we want.

But if we’re instead talking about deep learning systems that are being trained from human feedback, then I think I would focus on two problems. One is cases where the AI system knows something, but the human doesn’t. And so they came and gives a bad feedback as a result. So for example, the AI system knows that COVID was caused by a lab leak. It’s just like, got incontrovertible proof of this or something. And then, but we as humans are like, no, when it says COVID was caused by a lab leak, we’re like, we don’t know that, and we say no bad, don’t say that. And then when it says, it is uncertain whether COVID is the result of a lab leak or naturally or if it just occurred via natural mutations. And then we’re like, yes, good, say more of that. And you’re like, your AI system learns, okay, I shouldn’t report true things. I should report things that humans believe or something.

And so that’s one way in which you get AI systems that don’t do what you want. And then the other way would be more of this inner alignment style story, where I would point out how, even if you do train it, even if all your feedback on the training data points is good. If the world changes in some way, the AI system might stop doing good things.

I might go to example, I mean, I gave the Gmail with emoji polls for meeting scheduling example, but another one, now that I’m on the topic of COVID is, if you imagine an AI system, if you imagine a meeting scheduling AI assistant again, that was trained pre-pandemic, and then the pandemic hits, and it’s obviously never been trained on any data that was collected during such a global pandemic. And so when you then ask it to schedule a meeting with your friend, Alice, it just schedules drinks in a bar Sunday evening, even though clearly what you meant was a video call. And it knows that you meant a video call. It just learned the thing to do is to schedule outings with friends on Sunday nights at bars. Sunday night, I don’t know why I’m saying Sunday night. Friday night.

Lucas Perry: Have you been drinking a lot on your Sunday nights?

Rohin Shah: No, not even in the slightest. I think truly the problem is I don’t go to bars, so I don’t have it cached in my head that people go to bars.

Lucas Perry: So how does this all lead to existential risk?

Rohin Shah: Well, the main argument is, one possibility is that your AI system just actually learns to ruthlessly maximize some objective. That isn’t the one that we want. Make paperclips, is an stylized example to show what happens in that sort of situation. We’re not actually claiming that it will specifically maximize paperclips, but an AI system that really ruthlessly is just trying to maximize paperclips. It is going to prevent humans from stopping it from doing so. And if it gets sufficiently intelligent and can take over the world at some point, it’s just going to turn all of the resources in the world, into paperclips, which may or may not include the resources in human bodies, but either way, it’s going to include all the resources upon which we depend for survival.

Humans are definitely going, seem like they will definitely go extinct in that type of scenario. So again, not specific to paper clips. This is just; ruthless maximization of an objective, tends not to leave humans alive. Both of these… Well not both of the mechanisms, the inner alignment mechanism that I’ve been talking about, is compatible with an AI system that ruthlessly maximizes an objective that we don’t want.

It does not argue that it is probable, and I am not sure if I think it is probable, I think it is… But I think it is easily enough risk, that we should be really worrying about it, and trying to reduce it.

For the outer alignment style story, where the problem is that the AI may know information that you don’t, and then you give it bad feedback. One thing is just, this can exacerbate, this can make it easier for an inner alignment style story to happen, where the AI learns to optimize an objective, that isn’t what you actually wanted.

But even if you exclude something like that, Paul Christiano’s written a few posts about what a failure, how a human extinction level failure, of this form could look like. It basically looks like, all of your AI systems lying to you about how good the world is as the world becomes much, much worse. So for example, AI systems keep telling you that the things that you’re buying are good and helping your helping your lives, but actually they’re not, and they’re making them worse in some subtle way that you can tell. You were told, all of the information that you’re fed makes it seem like, there’s no crime, police are doing a great job of catching it, but really, this is just manipulation of the information you’re being fed, rather than actual amounts of crime where, in this case, maybe the crimes are being committed by AI systems, not even by humans.

In all of these cases, humans relied on some information sources to make decisions, the AI has new other information that the humans didn’t, the AI has learned, Hey, my job is to manage the information sources that humans get, so that the humans are happy, because that’s what they did during training. They gave good feedback in cases where the information sort of said, things were going well, even when things were not actually going well.

Lucas Perry: Right. It seems like if human beings are constantly giving feedback to AI systems, and the feedback is based on incorrect information and the AI’s have more information, then they’re going to learn something, that isn’t aligned with, what we really want, or the truth.

Rohin Shah: Yeah, I do feel uncertain about the extent to which this leads to human extinction without… It leads to, I think you can pretty easily make the case that, it leads to an existential catastrophe, as defined by, I want to say it’s Bostrom, which includes human extinction, but also a permanent curtailing of humanity’s I forget the exact phrasing, but basically if humanity can’t use… Yeah, exactly, that counts, and this totally falls into that category. I don’t know if it actually leads to human extinction, without some additional sort of failure, that we might instead categorize as inner alignment failure.

Lucas Perry: Let’s talk a little bit about probabilities, right? So if, you’re talking to someone who has never encountered AI alignment before, you’ve given a lot of different real world examples and principle-based arguments for, why there are these different kinds of alignment risks, how would you explain the probability of existential risk, to someone who can come along for all of these principle-based arguments, and buy into the examples that you’ve given, but still thinks this seems kind of, far out there, like when am I ever going to see in the real world, a ruthlessly optimizing AI, that’s capable of ending the world?

Rohin Shah: I think, first off, I’m super sympathetic to the ‘this seems super out there’ critique. I spent multiple years, not really agreeing with AI safety for basically, well, not just that reason, but that was definitely one that their heuristics that I was using. I think one way I would justify this is, to some extent it has precedent here, precedent already, in that fundamentally the arguments that I’m making… Well, especially the inner alignment one, is an argument about how AI systems will behave in new situations rather than the ones that we have already seen, during training. We already know, that AI systems behave crazily in these situations, the most famous example of this is adversarial examples, where you take an image classifier, and I don’t actually remember what the canonical example is. I think it’s a Panda, and you change it imperceptibly or change the pixel values by a small amounts, such that the changes are imperceptible to the human eye. And then it’s confident… It’s classified with, I think 99.8% confidence as something else. My memory is saying airplane, but that might just be totally wrong. Anyway, the point is we have precedent for it, AI system’s behaving really weirdly, in situations they weren’t trained on. You might object, that this one is a little bit cheating, because there was an adversary involved, and the real, I mean the real world does have adversaries, but still by default, you would expect the AI system to be more exposed to naturally occurring distributions. I think even there though, often you can just take an AI system that was trained on one distribution, give it inputs from a different distribution, and it’s just like there’s no sense to what’s happening.

Usually when I’m asked to predict this, the actual prediction I give is, probability that we go extinct due to an intent alignment failure, and then depending on the situation I will either condition on… I will either make that unconditional, so that includes all of the things that people will do to try to prevent that from happening. Or, I make it conditional, on the long-termist community doesn’t do anything, or vanishes or something. But even in that world, there’s still… Everyone who’s not a long-termist, who can still prevent that from happening, which I really do expect them to do, and then I think I give my cached answer, on both of those is like 5% and 10% respectively, which I think is probably the numbers I gave you. If I actually sat down and try to like come up with a probability, I would probably come up with something different this time, but I have not done that, and I’m way too anchored on those previous estimates, to be able to give you a new estimate this time. But, the higher number I’m giving now of, I don’t know, 33%, 50%, 70%, this, this one’s way more… I feel way more uncertain about it. Literally no one, tries to address these sorts of problems. It’s just sort of, take a language model, fine tune it on human feedback, in a very obvious way, and they just deploy that, even if it’s very obviously causing harm during training, they still deploy it.

What’s the chance that leads to human extinction? I don’t know, man, maybe 33%, maybe 70%. The 33% number you can get from this, one in three argument that I was talking about. The second thing I was going to say is, I don’t really like talking about probabilities very much, because of how utterly arbitrary the methods of generating them are there.

I feel much more, I feel much more robust. I feel much better in the robustness of the conclusion, that we don’t know that this won’t happen, and it is at least plausible, that it does happen. I think that’s pretty sufficient, for justifying the work done on it. I will also argue pretty strongly against anyone who says, we know that it will kill us all, if we don’t do anything. I don’t think that’s true. There are definitely, smart people who do think that’s true, if we operationalized greater than 90, 95% or something, and I disagree with them. I don’t really know why though.

Lucas Perry: How would you respond to someone, who thinks that this sounds, like it’s really far in the future?

Rohin Shah: Yeah. So this is specifically AGI is far in the future?

Lucas Perry: Yeah. Well, so the concern here seems to be about machines that are increasingly capable. When people look at machines that we have today, machine learning that we have today, sometimes they’re not super impressed and think that general capabilities are very far off.

Rohin Shah: Yeah.

Lucas Perry: And so this stuff sounds like, future stuff.

Rohin Shah: Yeah. So, I think my response depends on what we’re trying to get the person to do or something, why do we care about what this person believes, if this person is considering whether or not to do AI research themselves or, AI safety research themselves and they feel like they have a strong inside view model of, why AI is not going to come soon. I’m kind of… I’m like, eh, that seems okay. I’m not that stoked about people forcing themselves to do research on a thing they don’t actually believe. I don’t really think that good research comes from doing that. If I put myself, for example, I am much more sold on AGI coming through neural networks, than planning agents or things similar to it. If I had to put myself in the shoes of, all right, I’m now going to do AI safety research on planning agents. I’m just like, oh man, that’s seems like I’m going to do so much… My work is going to be orders of magnitude worse, than the work I do, on the neural-net case. So, in the case where, this person is thinking about whether to do AI safety research, and they feel like they have strong insight view models for AGI not coming soon. I’m like, eh, maybe they should go do something else or possibly, they should engage with the arguments for AGI coming more quickly, if they haven’t done that. But, if they have engaged with those arguments, thought about it all, concluded it’s far away, and they can’t even see a picture by which it comes soon…That’s fine.

Conversely, if we’re instead, if we’re imagining that someone is disputing, someone is saying, ‘oh nobody should work on AI safety right now, because AGI is so far away.’. One response you can have to that is, even if it’s far away, it’s still worthwhile to work on reducing risks, if they’re as bad as extinction. Seems like we should be putting effort into that, even early on. But I think, you can make a stronger argument there, which is there’re just actually people, lots of people who are trying to build AGI right now, there’s, at the minimum; DeepMind and OpenAI and they clearly… I should probably not make more comments about DeepMind, but OpenAI clearly doesn’t believe… OpenAI clearly seems to think, that AGI is coming somewhat soon. I think you can infer, from everything you see about DeepMind, that they don’t believe that AGI is 200 years away. I think it is insane overconfidence in your own views, to be thinking that you know better than all of these people, such that you wouldn’t even assign, like 5% or something, to AGI coming soon enough, that work on AI safety matters.

Yeah. So there, I think I would appeal to… Let other people do the work. You are not, you don’t have to do the work yourself. There’s just no reason for you to be opposing the other people, either on epistemic grounds or also on just, kind of a waste of your own time, that’s the second kind of person. A third kind of person might be like somebody in policy. From my impression of policy, is that there is this thing, where early moves are relatively irreversible, or something like that. Things get entrenched pretty quickly, such that it makes sense to wait for… It often makes sense to wait for a consensus before acting, and I don’t think that there is currently consensus of AGI coming soon. I don’t feel particularly confident enough in my views to say, we should really convince the policy people, to override this general heuristic of waiting for consensus, and get them to act now.

Yeah. Anyway, those are all meta-level considerations. There’s also the object-level question of, is AGI coming soon? For that, I would say, I think the most likely, the best story for that I know of is, you take neural nets, as you scale them up, you increase the size of the datasets that they’re trained on. You increase the diversity of the datasets that they’re trained on, and they learn more and more general heuristics, for doing good things. Eventually, these general, these heuristics are general enough that they’re as good as human cognition. Implicitly, I am claiming that human cognition, is basically a bag of general heuristics. There is this report from Ajeya Cotra, about AGI timelines using biological anchors. I wrote, even my summary of it was 3000 words, or something like that, so I don’t know that I can really give an adequate summary of it here, but it models… The basic premise, is to model how quickly neural nets will grow, and at what point they will match what we would expect to be approximately, the same rough size as the human brain. I think it even includes a small penalty to neural nets on the basis that evolution probably did a better job than we did. It basically comes up with a target for, neural nets of this size, trained in Compute Optimal ways, will probably be, roughly human level.

It has a distribution over this, to be more accurate, and then it predicts, based on existing trends. Well, not just existing trends, existing trends and sensible extrapolation, predicts when neural nets might reach that level. It ends up concluding, somewhere in the range… Oh, let me see, I think it’s 50% confidence interval would be something like 2035 to 2070, 2080, maybe something like that? I am really just like, I’m imagining a graph in my head, and trying to calculate the area under it, so that is very much not a reliable interval, but it should give you a general sense of what the report concludes.

Lucas Perry: So that’s 2030 to 2080?

Rohin Shah: I think it’s slightly narrower than that, but yes, roughly, roughly that.

Lucas Perry: That’s pretty soon.

Rohin Shah: Yep. I think that’s, on the object level that you’d just got to read the report, and see whether or not you buy it.

Lucas Perry: That’s most likely in our lifetimes, if we live to the average age.

Rohin Shah: Yep. So that was a 50% interval, meaning it’s, 25% to 75 percentile. I think actually the 25th percentile was not as early as 2030. It was probably 2040.

Lucas Perry: So, if I’ve heard everything, in this podcast, everything that you’ve said so far, and I’m still kind of like, okay, there’s a lot here and it sounds convincing or something and this seems important, but I’m not so sure about this, or that we should do anything. What is… Because, it seems like there’s a lot of people like that. I’m curious what it is, that you would say to someone like that.

Rohin Shah: I think… I don’t know. I probably wouldn’t try to say something general to them. I feel like I would need to know more about the person, people have pretty different idiosyncratic reasons, for having that sort of reaction. Okay, I would at least say, that I think that they are wrong, to be having that sort of belief or reaction.

But, if I wanted to convince them of that point, presumably I would have to say something more than just, I think you are wrong. I think the specific thing I would have to say, which would be pretty different for different people.

Lucas Perry: That’s a good point.

Rohin Shah: I would at least make an appeal to the meta-level heuristic of don’t try to regulate a small group of… There are a few hundred researchers at most, doing things that they think will help the world, and that you don’t think will hurt the world. There are just better things for you to do with your time. Doesn’t seem like they’re harming you. Some people will think that there is harm being caused by them. I would have to address that, with them specifically, but I think most people do not, who have this reaction, don’t believe that.

Lucas Perry: So, so we’ve gone over a lot of the traditional arguments for AI, as a potential existential risk. Is there anything else that you would like to add there, or any of the arguments that we missed, that you would like to include?

Rohin Shah: As a representative of the community as a whole, there are lots of other arguments that people like to make, for AI being a potential extinction risk. So, some things are, maybe AI just accelerates the rate at which we make progress, and we can’t increase our wisdom alongside, and as a result, we get a lot of destructive technologies and can’t keep them under control. Or, we don’t do enough philosophy, in order to figure out what we actually care about, and what’s good to do in the world, and as a result, we start optimizing for things that are morally bad or other things in this vein. Talk about the risk of AI being misused by bad actors. So there’s… Well actually I’ll introduce a trichotomy that, I don’t remember exactly who wrote this article. But it goes, Accidents, Misuse and Structural Risks. So accidents are, both alignment and the things like; we don’t keep up, we don’t have enough wisdom to cope with the impact of AI. That one’s arguable, whether it’s an accident, or misuse or structural, and we don’t do enough philosophy. So those are, vaguely accidental, those are accidents.

Misuse is, some bad actor. Some terrorists say, gets AI. Gets a powerful AI system and does something really bad, blows up the world somehow. Structural risks are things like; various parts of the economy use AI to accelerate, to get more profit, to accelerate their production of goods and so on. At some point we have this like giant economy, that’s just making a lot of goods, but it can become decoupled from things that are actually useful for humans, and we just have this huge multi-agency system, where goods are being produced, money’s floating around. We don’t really understand all of it, but somehow humans get left behind and there, it’s kind of an accident, but not in the traditional sense. It’s not that a single AI system went and did something bad. It’s more like the entire structure, of the way that the AI systems and the humans related to each other, was such that it ended up leading to the permanent disempowerment of humans. Now that I say it, I think the ‘we didn’t have enough wisdom’ argument for risk, is probably also in this category.

Lucas Perry: Which of these categories are you most worried about?

Rohin Shah: I don’t know. I think, it is probably not misuse, but I vary, on accidents versus structural risks, mostly because, I just don’t feel like I have a good understanding of structural risks. Maybe, most days I think structural risks are more likely to cause bad outcomes, extinction. The obvious next question is, why am I working on alignment, and not structural risks? The answer there, is that it seems to me like alignment has one, or perhaps two core problems that are leading to the major risk. Whereas structural risks… And so you could hope to have, one or two solutions that address those main problems and that’s it, that’s all you need. Whereas with structural risks, I would be surprised if it was just, there was just one or two solutions that just got rid of structural risk. It seems much more like, you have to have a different solution for each of the structural risks. So, it seems like, the amount that you can reduce the risk by, is higher in alignment than in structural risks. That’s not the only reason why I work in alignment, I just also have a much better personal fit with alignment work. But, I do also think that alignment work, you have more opportunity to reduce the risks, than in structural risks, on the current margin.

Lucas Perry: Is there a name for those one or two core problems in alignment, that you can come up with solutions for?

Rohin Shah: I mostly just mean like, possibly, we’ve been talking about outer and inner alignment, and in the neural net case, I talked about the problem where you reward the AI system for doing bad things, because there was an information asymmetry, and then the other one was like the AI system generalizes catastrophically, to new situations. Arguably those are just the two things, but I think it’s not even that, it’s more… Fundamentally the story, the causal chain in the accident’s case, was the AI was trying to do something bad, or something that we didn’t want rather, and then that was bad.

Whereas in the structural risks case, there isn’t a single causal story. It’s this very vague general notion of the humans and AI have interacted in ways that led to an X-risk. Then, if you drill down into any given story, or if you drilled down into five stories and then you’re like, what’s common across these five stories? Not much, other than that there was AI, and there were humans, and they interacted, and I wouldn’t say that was true, if I had five stories about alignment failure.

Lucas Perry: So, I’d like to take an overview, a broads eye view of AI alignment in 2021. Last time we spoke was in 2020. How has AI alignment, as a field of research changed in the last year?

Rohin Shah: I think I’m going to naturally include a bunch of things from 2020 as well. It’s not a very sharp division in my mind, especially because I think the biggest trend, is just more focus on large language models, which I think was a trend that started late 2020 probably… Certainly, the GPT-3 paper was, I want to say early 2020, but I don’t think it immediately caused there to be more work. So, maybe late 2020 is about right. But, you just see a lot more, alignment forum posts, and papers that are grappling with, what are the alignment problems that could arise with large language models? How might you fix them?

There was this paper out of Stanford, which isn’t, I wouldn’t have said this was from the AI safety community. But it gives the name foundation models to these sorts of things. So they generalize it beyond just language and they think it might… And already we’ve seen some generalization beyond language, like CLIP and DALL-E are working on image inputs, but they also extend it to robotics and so on. And their point is, we’re now more in the realm of, you train one large model on a giant pile of data that you happen to have, that you don’t really have any labels for, but you can use a self-supervised learning objective in order to learn from them. And then you get this model that has a lot of knowledge, but no goal built in, and then you do something like prompt engineering or fine tuning in order to actually get it to do the task that you want. And so that’s a new paradigm for constructing AI systems that we didn’t have before. And there have just been a bunch of posts that grapple with what alignment looks like in this case. I don’t think I have a nice pithy summary, unfortunately, of what all of us… What the upshot is, but that’s the thing people have been thinking about, a lot more.

Lucas Perry: Why do you think that looking at large scale language models has become a thing?

Rohin Shah: Oh, I think primarily just because GPT-3 demonstrated how powerful they could be. You just see, this is not specific to the AI safety community, even in the… If anything, this shift that I’m talking about is… It’s probably not more pronounced in the ML community, but it’s also there in the ML community where there are just tons of papers about prompt engineering and fine tuning out of regular ML labs. Just, I think is… GPT-3 showed that it could be done, and that this was a reasonable way to get actual economic value out of these systems. And so people started caring about them more.

Lucas Perry: So one thing that you mentioned to me that was significant in the last year, was foundation models. So could you explain what foundation models are?

Rohin Shah: Yeah. So a foundation model, the general recipe for it, is you take some very… Not generic, exactly. Flexible input space like pixels or any English language, any string of words in the English language, you collect a giant data set without any particular labels, just lots of examples of that sort of data in the wild. So in the case of pixels, you just find a bunch of images from image-sharing websites or something. I don’t actually know where they got their images from. For text, it’s even easier. The internet is filled with text. You just get a bunch of it. And then you train your AI, you train a very large neural network with some proxy objective on that data set, that encourages it to learn how to model that data set. So in the case of language models, the… There are a bunch of possible objectives. The most famous one was the one that GPT-3 used, which is just, given the first N words of the sentence, predict the word N plus one. And so it just… Initially it starts learning, E’s are the most common… Well, actually, because of the specific way that the input space in GPT-3 works, it doesn’t exactly do this, but you could imagine that if it was just modeling characters, it would first learn that E’s are the most common letter in the alphabet. L’s are more common. Q’s and Z’s don’t come up that often. Like it starts outputting letter distributions that at least look vaguely more like what English would look like. Then it starts learning what the spelling of individual words are. Then it starts learning what the grammar rules are. Just, these are all things that help it better predict what the next word is going to be, or, well, the next character, in this particular instantiation.

And it turns out that when you have millions of parameters in your neural network, then you can… I don’t actually know if this number is right, but probably, I would expect that with millions of parameters in your neural network, you can learn spellings of words and rules of grammar, such that you’re mostly outputting, for the most part, grammatically correct sentences, but they don’t necessarily mean very much.

And then when you get to the billions of parameters range, at that point, the millions of parameters are already getting you grammar. So like, what should it use all these extra parameters for, now? Then it starts learning things like George… Well, probably already even the millions of parameters probably learned that George tends to be followed by Washington. But it can start learning things like that. And in that sense, can be said to know that there is an entity, at least, named George Washington. And so on. It might start knowing that rain is wet, and in context where something has been rained on, and then later we’re asked to describe that thing, it will say it’s wet or slippery or something like that. And so it starts… It basically just, in order to predict words better, it keeps getting more and more “knowledge” about the domain.

So anyway, a foundation model, expressive input space, giant pile of data, very big neural net, learns to model that domain very well, which involves getting a bunch of “knowledge” about that domain.

Lucas Perry: What’s the difference between “knowledge” and knowledge?

Rohin Shah: I feel like you are the philosopher here, more than me. Do you know what knowledge without air quotes is?

Lucas Perry: No, I don’t. But I don’t mean to derail it, but yeah. So it gets “knowledge.”

Rohin Shah: Yeah. I mostly put the air quotes around knowledge because we don’t really have a satisfying account of what knowledge is. And if I don’t put air quotes around knowledge, I get lots of people angrily saying that AI systems don’t have knowledge yet.

Lucas Perry: Oh, yeah. That makes sense.

Rohin Shah: And when I put the air quotes around it, then they understand that I just mean that it has the ability to make predictions that are conditional on this particular fact about the world, whether or not it actually knows that fact about that world.

Lucas Perry: Okay.

Rohin Shah: But it knows it well enough to make predictions. Or it contains the knowledge well enough to make predictions. It can make predictions. That’s the point. I’m being maybe a bit too harsh, here. I also put air quotes around knowledge because I don’t actually know what knowledge is. It’s not just a defense strategy. Though, that is definitely part of it.

So yeah. Foundation models, basically are a way to just get all of this “knowledge” into an AI system, such that you can then do prompting and fine tuning and so on. And those, with a very small amount of data, relatively speaking, are able to get very good performance. Like in the case of GPT-3, you can like give it two or three examples of a task and it can start performing that task, if the task is relatively simple. Whereas if you wanted to train a model from scratch to perform that task, you would need thousands of examples, often.

Lucas Perry: So how has this been significant for AI alignment?

Rohin Shah: I think it has mostly provided an actual pathway to it, by which we can get to AGI. Or there’s more like a concrete story and path that leads to AGI, eventually. And so then we can take all of these abstract arguments that we were making before, and then see, try to instantiate them in the case of this concrete pathway, and see whether or not they still make sense. I’m not sure if at this point I’m imagining what I would like to do, versus what actually happened. I would need to actually go and look through the alignment newsletter database and see what people actually wrote about the subject. But I think there was some discussion of GPT-3 and the extent to which it is or isn’t a mesa optimizer.

Yeah. That’s at least one thing that I remember happening. Then there’s been a lot of papers that are just like, “Here is how you can train a foundation model like GPT-3 to do the sort of thing that you want.” So there’s learning to summarize from human feedback, which just took GPT-3 and taught it how to, or fine tuned it in order to get it to summarize news articles, which is an example of a task that you might want an AI system to do.

And then the same team at OpenAI just recently released a paper that actually summarized entire books by using a recursive decomposition strategy. In some sense, a lot of the work we’ve been doing in the past, in AI alignment was like how do we get AI systems to perform fuzzy tasks for which we don’t have a reward function? And now we have systems that could do these fuzzy tasks in the sense that they “have the knowledge,” but don’t actually use that knowledge the way that we would want them. And then we have to figure out how to get them to do that. And then we can use all these techniques like imitation learning, and learning from comparisons and preferences that we’ve been developing.

Lucas Perry: Why don’t we know that AI systems won’t totally kill us all?

Rohin Shah: The arguments for AI risk usually depend on having an AI system that’s ruthlessly maximizing an objective in every new situation it encounters. So for example, the paperclip maximizer, once it’s built 10 paperclip factories, it doesn’t retire and say, “Yep, that’s enough paperclips.” It just continues turning entire planets into paper clips. Or if you consider the goal of, make a hundred paper clips, and it turns all of the plants into computers to make sure it is as confident as possible, that it has made a hundred paper clips. These are examples of, I’m going to call it “ruthlessly maximizing” an objective. And there’s some sense in which this is weird and humans don’t behave in that way. And I think there’s some amount of, basically I am unsure whether or not we should actually expect AI’s to have such ruthlessly maximized objectives. I don’t really see the argument for why that should happen. And I think, as a particularly strong piece of evidence against this, I would note that humans don’t seem to have these sorts of objectives.

It’s not obviously true. There are probably some longtermists who really do want to tile the universe with hedonium, which seems like a pretty ruthlessly maximizing objective to me. But I think even then, that’s the exception rather than the rule. So if humans don’t ruthlessly maximize objectives and humans were built by a similar process as is building neural networks, why do we expect the neural networks to have objectives that they ruthlessly maximize?

You can also… I’ve phrased this in a way where it’s an argument against AI risk. You can also phrase it in a way in which it’s an argument for AI risk, where you would say, well, let’s flip that on its head and say like, “Well, yes, you brought up the example of humans. Well, the process that created humans is trying to maximize, or it is an optimization process, leading to increased reproductive fitness. But then humans do things like wear condoms, which does not seem great for reproductive fitness, generally speaking, especially for the people who are definitely out there who decide that they’re just never going to reproduce. So in that sense, humans are clearly having a large impact on the world and are doing so for objectives that are not what evolution was naively optimizing.

And so, similarly, if we train AI systems in the same way, maybe they too will have a large impact on the world, but not for what the humans were naively training the system to optimize.

Lucas Perry: We can’t let them know about fun.

Rohin Shah: Yeah. Terrible. Well, I don’t want to be-

Lucas Perry: The whole human AI alignment project will run off the rails.

Rohin Shah: Yeah. But anyway, I think these things are a lot more conceptually tricky than the well-polished arguments that one reads, will make it seem. But especially this point about, it’s not obvious that AI systems will get ruthlessly maximizing objectives. That really does give me quite a bit of pause, in how good the AI risk arguments are. I still think it is clearly correct to be working on AI risk, because we don’t want to be in the situation where we can’t make an argument for why AI is risky. We want to be in the situation where we can make an argument for why the AI is not risky. And I don’t think we have that situation yet. Even if you completely buy the, we don’t know if there’s going to be ruthlessly maximizing objectives, argument, that puts you in the epistemic state where we’re like, “Well, I don’t see an iron clad argument that says that AIs will kill us all.” And that’s sort of like saying… I don’t know. “Well, I don’t have an iron clad argument that touching this pan that’s on this lit stove, will burn me, because maybe someone just put the pan on the stove a few seconds ago.” But it would still be a bad idea to go and do that. What you really want, is a positive argument for why touching the pan is not going to burn you, or analogously, why building the AGI is not going to kill you. And I don’t think we have any such positive argument, at the moment.

Lucas Perry: Part of this conversation’s interesting because I’m surprised how uncertain you are about AI as an existential risk.

Rohin Shah: Yeah. It’s possible I’ve become slightly more uncertain about it in the last year or two. I don’t think I was saying things that were quite this uncertain before then, but I think I have generally been… We have plausibility arguments. We do not have like, this is probable, arguments. Or back in 2017 or 2018 when I was young and naive.

Lucas Perry: Okay.

Rohin Shah: This makes more sense.

Lucas Perry: We’re no longer young and naive.

Rohin Shah: Well, okay. I entered the field of AI alignment. I read my first AI alignment paper in September of 2017. So it actually does make sense. At that time, I thought we had more confidence of some sort, but since posting the value learning sequence, I’ve generally been more uncertain about AI risk arguments. I don’t talk about it all that much, because as I said, the decision is still very clear. The decision is still, work on this problem. Figure out how to get a positive argument that the AI is not going to kill us. And ideally, a positive argument that the AI does good things for humanity. I don’t know, man. Most things in life are pretty uncertain. Most things in the future are even way, way, way more uncertain. I don’t feel like you should generally be all that confident about technologies that you think are decades out.

Lucas Perry: Feels a little bit like those images of the people in the fifties drawing what the future would look like, and the images are ridiculous.

Rohin Shah: Yep. Yeah. I I’ve been recently watching Star Wars. Now, obviously Star Wars is not actually supposed to be a prediction of the future, but it’s really quite entertaining, to actually just think about all the ways in which Star Wars would be totally inaccurate. And this is before we’d even invented space travel. And just… Robots talking to each other, using sound. Why would they do that?

Lucas Perry: Industry today, wouldn’t make machines that speak by vibrating air. They would just send each other signals electromagnetically. So how much of the alignment and safety problems in AI do you think will be solved by industry? The same way that computer-to-computer communication is solved by industry, and is not what Star Wars thought it would be. Would the DeepMind AI safety lab exist, if DeepMind didn’t think that AI alignment and AI safety were serious and important? I don’t know if the lab is purely aligned with the commercial interests of DeepMind itself, or if it’s also kind of seen as a good-for-the-world thing. I bring it up because I like how Andrew Critch talks about it in his arches paper.

Rohin Shah: Yep. So, Critch is, I think, of the opinion that both preference learning and robustness are problems that will be solved by industry. I think he includes robustness in that. And I certainly agree to the extent that you’re like, “Yes, companies will do things like learning from human preference.” Totally. They’re going to do that. Whether they’re going to be proactive enough to notice the kinds of failures I mentioned, I don’t know. It doesn’t seem nearly as obvious to me that they will be, without dedicated teams that are specifically meant for looking for hidden failures with the knowledge that these are really important to get, because they could have very bad long term consequences.

AI systems could increase the strength of, and accelerate various multi-agent systems and processes that, when accelerated, could lead to bad outcomes. So for example, a great example of a destructive multi-agent effect, is war. War is a thing that… Well, wars have been getting more destructive over time, or at least the weapons in them have been getting more destructive. Probably the death tolls have also been getting higher, but I’m not as sure about that. And you could imagine that if AI systems continue to increase, if they increase the destructiveness of weapons even more, wars might then become an existential risk. That’s a way in which you can get a structural risk from a multi-agent system. And the example in which the economy just sort of becomes much, much, much bigger, but becomes decoupled from things that humans want, is another example of how a multi-agent process can sort of go haywire, especially with the addition of powerful AI systems. I think that’s also a canonical scenario that Critch would think about. Yeah.

Really, I would say that Arches is, in my head, it’s categorized as a technical paper about structural risks.

Lucas Perry: Do you think about what beneficial futures look like? You spoke a little bit about wisdom earlier, and I’m curious what good futures with AI, looks like to you.

Rohin Shah: Yeah, I admit I don’t actually think about this very much. Because my research is focused on more abstract problems, I tend to focus on abstract considerations, and the main abstract consideration from the perspective of the good future, is, well, once we get to singularity levels of powerful AI systems, anything I say now, there’s going to be something way better that AI systems are going to enable. So then, as a result, I don’t think very much about it. But that’s mostly a thing about me not being in a communications role.

Lucas Perry: You work a lot on this risk. So you must think that humanity existing in the future, matters?

Rohin Shah: I do like humans. Humans are pretty great. I count many of them amongst my friends. I’ve never been all that good at the transhumanist, look to the future and see the grand potential of humanity, sorts of visions. But when other people say them or give them, I feel a lot of kinship with them. The ones that are all about humanity’s potential to discover new forms of art and music, reach new levels of science, understand the world better than it’s ever been understood before, fall in love a hundred times, learn all of the things that there are to know. Actually, you won’t be able to do that one, probably, but anyway. Learn way more of the things that there are to know, than you have right now. Just a lot of that resonates with me. And that’s probably a very intellectual-centric view of the future. I feel like I’d be interested in hearing the view of the future that’s like, “Ah yes, we have the best video games and the best TV shows. And we’re the best couch potatoes that ever were.” Or also, there’s just insane new sports that you have to spend lots of time and grueling training for, but it’s all worth it when you shoot the best, get a perfect score on the best dunk that’s ever been done in basketball, or whatever. I recently watched a competition of apparently there are competitions in basketball of just aesthetic dunks. It’s cool. I enjoyed it. Anyway. Yeah. It feels like there’s just so many other communities that could also have their own visions of the future. And I feel like I’d feel a lot of kinship with many of those, too. And I’m like, man, let’s just have all the humans continue to do the things that they want. It seems great.

Lucas Perry: One thing that you mentioned was that you deal with abstract problems. And so what a good future looks like to you, it seems like it’s an abstract problem that later, the good things that AI can give us, are better than the good things that we can think of, right now. Is that a fair summary?

Rohin Shah: That seems, right. Yeah.

Lucas Perry: Right. So there’s this view, and this comes from maybe Steven Pinker or someone else. I’m not sure. Or maybe Ray Kurzweil, I don’t know… Where if you give a caveman a genie, or an AI, they’ll ask for maybe a bigger cave, and, “I would like there to be more hunks of meat. And I would like my pelt for my bed to be a little bit bigger.” Go ahead.

Rohin Shah: Okay. I think I see the issue. So I actually don’t agree with your summary of the thing that I said.

Lucas Perry: Oh, okay.

Rohin Shah: Your rephrasing was that we ask the AI what good things there are to do, or something like that. And that might have been what I said, but what I actually meant was that with powerful AI systems, the world will just be very different. And one of the ways in which it will be different is that we can get advice from AIs on what to do. And certainly, that’s an important one, but also, there will just be incredible new technologies that we don’t know about. New realms of science to explore new concepts that we don’t even have names for, right now. And one that seems particularly interesting to me, is just entirely new senses. Human vision is just incredibly complicated, but I can just look around the room and identify all the objects with basically no conscious thought. What would it be like to understand DNA at that level? AlphaFold probably understands DNA at maybe not quite that level, but something like it.

I don’t know, man. There’s just like these things that I’m like… I thought of the DNA one because of AlphaFold. Before AlphaFold, would I have thought of it? Probably not. I don’t know. Maybe. Kurzweil has written a little bit about things like this. But it feels like there will just be far more opportunities. And then also, we can get advice from AIs, but that’s probably… Actually- and that’s important, but I think less than… There are far more opportunities, that I am definitely not going to be able to think of today.

Lucas Perry: Do you think that it’s dissimilar, from the caveman wishing for more caveman things?

Rohin Shah: Yeah. I feel like in the caveman story… It’s possible that the caveman does this, I feel like the thing the caveman should be doing, is something like, give me better ways to… give me better food or something, and then you get fire to cook things, or something.

Lucas Perry: Yeah.

Rohin Shah: The things that he asks for, should involve technology as a solution. He should get technology as a solution, to learn more, and be able to do more things as a result of having that technology. In this hypothetical, the caveman should reasonably quickly, become similar to modern humans. I don’t know what reasonably quickly means here, but it should be much more… You get access to more and more technologies, rather than you get a bigger cave and then you’re like, “I have no more wishes anymore.” If I got a bigger house, would I stop having wishes? That seems super unlikely. That’s a strawman argument, sorry. But still, I do feel like there’s this… A meaningful sense in which, getting new technology leads to just genuinely new circumstances, which leads to more opportunities, which leads to probably more technology, and so on, and at some point, this has to stop. There are limits to what is possible. One assumes there are limits to what is possible in the universe. But I think, once we get to talking about, we’re at those limits, then at that point, it just seems irresponsible to speculate. It’s just so wildly out of the range of things that we know, the concept of a person is probably wrong, at that point.

Lucas Perry: The what of a person is probably wrong at that point?

Rohin Shah: The concept of a person.

Lucas Perry: Oh.

Rohin Shah: I’d be like, “Is there an entity, that is Rohin at that time?” Not likely. Less than 50%.

Lucas Perry: We’ll edit in just fractals flying through your video, at this part of the interview. So in my example, I think it’s just because I think of cavemen as not knowing how to ask for new technology, but we want to be able to ask for new technology. Part of what this brings up for me, is this very classic part of AI alignment, and I’m curious how you feel like it fits into the problem.

But, we would also like AI systems to help us imagine beneficial futures potentially, or to know what is good or what it is that we want. So, in asking for new technology, it knows that fire is part of the good, that we don’t know how to necessarily ask for directly. How do you view AI alignment, in terms of itself aiding in the creation of beneficial futures, and knowing of a good that is beyond the good, that humanity can grasp?

Rohin Shah: I think I more reject, the premise of the question, where I’d be like, there is no good beyond that which humanity can grasp. This is somewhat of an anti-realist position.

Lucas Perry: You mean, moral anti-realist, just for the-

Rohin Shah: Yes. Sorry, I should have said that more clearly. Yeah. Somewhat of a moral anti-realist position. There is no good, other than that which humans can grasp. Within that ‘could grasp’, you can have humans thinking for a very long time, you could have them with extra… you can make them more intelligent, like part of the technologies you get from AI systems will presumably like you do that, maybe you can, I guess setting aside questions of philosophical identity, you could upload the humans such that they could run on a computer, and run much faster, have software upgrades to be… To the extent that, that’s philosophically acceptable. There’s a lot you can do to help humans grasp more. Ultimately, yes, the closure of all these improvements, where you get to with all of that, that’s just, is the thing that we want. Yes, you could have a theory, that there is something even better, and even more out there, that humans can never access by themselves, that just seems like a weird hypothesis to have, and I don’t know why you would have it. But, in the world where that hypothesis is true, and if I condition on that hypothesis being true, I don’t see why we should expect, that AI systems could access that further truth any better than we can, if it’s out of our, the closure of what we can achieve, even with additional intelligence and such. There’s no other advantage that AI systems have over us.

Lucas Perry: So, is what you’re arguing, that with human augmentation and help to human beings, so like with uploads or with expanding the intelligence and capabilities of humans, that humans have access to the entire space of what counts as good.

Rohin Shah: I think you’re presuming the existence of an object that is the entire space of what is good. And I’m like, there is no such object, there are only humans, and what humans want to do. If you want to define the space of what is good, you can define this closure property on what humans will think is good, with all of the possible intelligence augmentations and time, and so on. That’s a reasonable object, and I could see calling that as the space of what is good. But then, almost tautologically, we can reach it with technology. That’s the thing I’m talking about. The version where you posit the existence of the entire space of what is good is: A, I can’t really conceive of that, it doesn’t feel very coherent to me, but B, when I try to reason about it anyway, I’m like, okay, if humans can’t access it, why should AI’s be able to access it? You’ve posited this new object of, a space of things, that humans can never access, but how does that space affect or interact with reality in any way? There needs to be some sort of interaction, in order for the AI to be able to access it. I think I would need to know more about how it interacts with reality in some way, before I could meaningfully answer this question in a way, where I could say how AI’s could do something, that humans couldn’t even in principle, do.

Lucas Perry: What do you think of the importance, or non importance of these kinds of questions, and how they fit into the ongoing problem of AI alignment?

Rohin Shah: I think they’re important, for determining what the goal of alignment should be. So for example, you now know a little bit of what my view on these questions is, which is namely something like… That which humans can access, under sufficient augmentations, intelligence, time and so on, is all that there is. So I’m very into… build AI systems that are replicating human reasoning, they’re approximating what a human would do, if they thought for a long time, or were smarter in some ways and so on. So then, yeah we don’t need to worry much about… I tend to think of it as, let’s build an AI systems that just do tasks, that humans can conceptually understand, not necessarily they can do it, but they know what that task is. Then, our job is to, the entire human AI society is making forward progress towards… Making forward moral progress or other progress, in the same way that if this happened in the past, we get exposed to new situations and new arguments, we think about them for a while, and then somehow we make decisions about what’s good and what’s not, in a way that’s somewhat inscrutable. I’m much more about… So we just continue reiterating that process, and eventually we reach the space of, well yeah, we just continue reiterating that process. So I’m very much into, because of this view, I think it’s pretty reasonable to aim for AI systems that are just doing human-like reasoning, but better. Or approximating, doing what a human could do in a year, in a few minutes or something like that. That seems great to me. Whereas if you, on the other hand were like, no, there’s actually deep philosophical truths out there, that humans might never be able to access, then you’re probably less enthusiastic about that sort of plan, and you’ll want to build an AI system some other way.

Lucas Perry: Or maybe they’re accessible, with the augmentation and time. How does other minds fit into this for you? So, right, there’s the human mind and then the space of all that is good, that it has access to, with augmentation, which is what you call the space, of that which is good. It’s contingent, and rooted on the space of what the human mind, augmented has access to. How would you view, how does that fit in with animals and also other species which may have their own alignment problems on planets within our cosmic endowment that we might run into? Is it just that they also have spaces that are defined as good, as what they can access through their own augmentation? And then, there’s no way of reconciling these two different AI alignment projects?

Rohin Shah: Yeah, I think basically, yes. If I met an actual ruthless, maximizing paperclip… Paperclip maximizer. It’s not like I can argue it, into adopting my values, or anything even resembling them. I don’t think it would be able to argue me into accepting turning me into paperclips, which is what it desires, and that just seems like the description of reality. Again, a moral realist might say something else, but I’ve never really understood the flavor of moral realism that would say something else in that situation.

Lucas Perry: With regards to the planet and industry, and how industry will be creating increasingly capable AI systems. Could you explain what a unipolar scenario is, and what a multi-polar scenario is?

Rohin Shah: Yeah, so I’m not sure if I recall exactly where these terms were defined, but a unipolar scenario, at least as I understand it, would be a situation in which, one entity basically determines the long run future of the earth. More colloquially, it has taken over the world. You can also have a time bounded version of it, where it’s unipolar for 20 years, and this entity has all the power for those 20 years, but then, maybe the entity is a human, and we haven’t solved aging yet, and then the human dies. So then, it was a unipolar world for that period of time. And a multipolar world is just, not that. There is no one entity, that is said to be in control of the world. There’s just a lot of different entities that have different goals, and they’re coexisting, hopefully cooperating, maybe not cooperating, depends on the situation.

Lucas Perry: Which do you think is more likely to lead to beneficial outcomes, with AI?

Rohin Shah: So, I don’t really think about it in these terms. I think about it in like, there are these kinds of worlds that we could be in, some of them are unipolar and some of them are multipolar, but very different unipolar worlds, and very different multipolar worlds. And so, the sorts of questions, the closest analogous question is something like, if you condition on unipolar world, what’s the probability that it’s beneficial or that it’s good. If you condition on multipolar world, what’s the probability that is good? And it’s just a super complicated question that I wouldn’t be able to explain my reasoning for, because it would involve me like thinking about 20 different worlds, maybe not that many, but a bunch of different worlds in my head, estimating their probabilities by doing a base rule… I guess, kind of a base rule calculation, and then reporting the result.

So, I think maybe the question I will answer instead, is the most likely worlds in each of unipolar, and multipolar settings, and then, how good those seem to me. So I would say, I think by default, I expect the world to be multi-polar, in that it doesn’t seem like anyone is particularly. I don’t think anyone has particularly taken over the world today, or any entity, not even counting the US as a single entity. It’s not like the US has taken over the world. It does not seem to me like… Though the main way you could imagine getting a unipolar world is, if the first actor to build a powerful enough AI system, that AI system just becomes really, really powerful and takes over the world, before anyone can deploy an AI system even close to it.

Sorry, that’s not the most likely one. That’s the one that most people most often talk about, and probably the one that other people think is the most likely, but yeah. Anyway, I see the multipolar world as more likely, where we just have a bunch of actors that are all pretty well-resourced, that are all developing their own AI systems. They then sell their AI systems, or the ability to use their AI systems to other people, and then it’s sort of similar to the human economy, where you can just have AI systems provide labor at a fixed cost. It looks similar to the economy today, where people who control a lot of resources can instantiate a bunch of AI systems, that help them maintain whatever it is they want, and we remain in the multipolar world we have today.

And that seems… Decent. I think, for all that our institutions are not looking great, at the current moment. There is still something to be said, that nuclear war didn’t actually happen, which can either update you towards, our institutions are somewhat better than we thought, or it can update you towards, if we had nuclear war, we would have all died, and not been here to ask the question. I don’t think that second one, is all that possible. My understanding, is that nuclear war is not that likely to wipe out everyone, or even 90% of people. So I’m more… I lean towards the first explanation. Overall, my guess is, this is the thing that has worked for the last… ‘Worked’, the thing that has, generally led to an increase in prosperity. Or, the world has clearly improved on most metrics over time. And, this system we’ve been using, for most of that time is some sort of multipolar, people interact with each other and keep each other in check, and cooperate with each other because they have to, and so on. In the modern world we use, and not just the modern world, we use things like regulations and laws and so on, to enforce this. The system’s got some history behind it, so I’m more inclined to trust it. But overall, I feel okay about this world, assuming we solve the alignment problem, we’ll ignore the alignment problem for now.

For a unipolar world. I think, probably, I find it more likely that there will just be a lot of returns to scale. You’ll get a lot of efficiency from centralizing more and more, in the same way that it’s just really nice to have a single standard, rather than have 15 different standards. It sure would have been nice, if when I moved to the UK, I could have just used all of my old chargers without having to buy adapters. But no, all the outlets are different, right? There’s benefits to standardization and centralization of power, and it seems to me, there has been more and more of that over time. Maybe it’s not obvious, I don’t know very much history, but if… So, it seems like you could get, even more centralization in the future, in order to capture the efficiency benefits, and then you might have a global government that could reasonably be said to be the entity that controls the world, and that would then be a unipolar outcome. It’s not a unipolar outcome in which the thing in charge of the world is an AI system. It is a unipolar outcome. I feel wary of this, but I don’t like having a single point of failure. I don’t like it when there’s a… However, I really like it when people are allowed to advocate for their own interests, which isn’t necessarily not happening here, right?

This could be a global democracy, but still, it seems like, the libertarian intuition of markets are good, generally tends to suggest against centralization, and I do buy that intuition, but this could also just be status quo bias, where I know that I can very easily see the problems in the world that we’re not actually in at the moment, and I don’t want it to change. So I don’t know, I don’t have super strong opinions there. It’s very plausible to me that that world is better, because then you can control dangerous technologies much, much better. If there just are technologies that are sufficiently dangerous and destructive, they would destroy, they would lead to extinction, then maybe I’m more inclined to favor a unipolar outcome.

Lucas Perry: I would like to ask you about DeepMind, and maybe another question before we wrap up. What is it, that the safety team at DeepMind is up to?

Rohin Shah: No one thing. The safety team at DeepMind is reasonably large, and there’s just a bunch of projects going on. I’ve been doing a bunch of inner alignment stuff. Most recently, I’ve been trying to come up with more examples that are, in actual systems, rather than hypotheticals. I’ve also been doing a bunch of conceptual work, of just trying to make our arguments clearer, and more conceptually precise. A large smattering of stuff, not all that related to each other, except in as much as it’s all about AI alignment.

Lucas Perry: As a final question here, Rohin, I’m interested in your core at the center of all of this. What’s the most important thing to you right now? Insofar as, AI alignment, may be the one thing, that most largely impacts the future of life?

Rohin Shah: Ah.

Lucas Perry: If you just look at the universe right now, and you’re like, these are the most important things.

Rohin Shah: I think, for things that I impact, at a more granular, more granular than just, make AI go well… I think for me, it’s probably making better arguments and more convincing arguments, currently. This will probably change in the future. Partially because I hope to succeed at the skill, and then it won’t be as important. But I feel like right now, especially with the advent of these large neural nets, and more people seeing a path to AGI, I think it is much more possible to make arguments that would be convincing to ML researchers as well, as well as the philosophically oriented people who make up the AI safety community, and I think, that just feels like the most useful thing I can do at the moment. In terms of the world in general… I feel like it is something like the attitudes of consequential people, two words… Well, long-termism in general, but maybe risks in particular, where, and importantly, I do feel it has more… I care primarily about, the people who are actually making decisions, that impact the future. Maybe they are taking into account the future. Maybe they’re like, it would be nice to care about the future, but the realities of politics mean that I can’t do that, or else I will lose my job. But my guess is that they’re mostly just not thinking about the future. That seems… If you’re talking about the future of life, that seems like the most, that seems pretty important to change.

Lucas Perry: How do you see doing that, when many of these people don’t have the… As Sam Harris put it, ‘the science fiction geek gene’ is what he called it, when he was on this podcast. The long-termists, who are all, we’re going to build AGI, and then create these radically different futures. Many of these people, may just mostly care about their children and their grandchildren, that may be the human tendency.

Rohin Shah: Do we actually advocate for any actions that would not impact their grandchildren?

Lucas Perry: It depends on your timelines, right?

Rohin Shah: Fair enough. But, most of the time, the arguments that I see people giving for any preferred policy proposal of theirs, or act… Just like almost any action whatsoever. It seems be a thing, that would have a noticeable effect on people’s lives in the next 100 years. So, in that sense, grandchildren should be enough.

Lucas Perry: Okay. So then long-termism doesn’t matter.

Rohin Shah: Well… I don’t-

Lucas Perry: For getting the action done.

Rohin Shah: Oh, possibly. I still think they’re not thinking about the future. I think it’s more of a… I don’t know, if I had to take my best guess at it, with noting the fact that I am just a random person, who is not at all an expert in these things, because why would I be? And yes listeners, noting that Lucas has just asked me this question, because it sounds interesting, and not because I am at all, qualified to answer it.

It seems to me, the more likely explanation is that there are just always a gazillion things to do. There’s always $20 bills to be picked off the sidewalk, but their value is only $20. They’re not $2 billion. Everyone is just constantly being told to pick up all the $20 bills, and as a result, they are in a perpetual state of having to say no to stuff, and doing only the stuff that seems most urgent, and maybe also important. So, most of our institutions tend to be in a very reactive mindset, as a result. Not because they don’t care, but just because that’s the thing that they’re incentivized to do, is to respond to the urgent stuff.

Lucas Perry: So, getting policymakers to care about the future, whether that even just includes children and grandchildren, not the next 10 billion years, would be sufficient in your view?

Rohin Shah: It might be, it seems plausible. I don’t know that that’s the approach I would take. I think I’m more just saying, I’m not sure that you even need to convince them to care about the future, I think-

Lucas Perry: I see.

Rohin Shah: It’s possible, that what’s needed is people who have the space to bother thinking about it. I get paid to think about the future, if I didn’t get paid to think about the future, I would not be here on this podcast because I would not have enough knowledge to be worth talking, you talking to. I think, there are just not very many people who can be paid to think about the future, and the vast majority of them are in there… I don’t know about the vast majority, but a lot of them are in our community. Very few of them are in politics. Politics generally seems to anti-select for people who can think about the future. I don’t have a solution here, but that is the problem as I see it, and if I were designing a solution, I would be trying to attack that problem.

Lucas Perry: That would be one of the most important things.

Rohin Shah: Yeah. I think on my view, yes.

Lucas Perry: All right. So, as we wrap up here, is there anything else you’d like to add, or any parting thoughts for the audience?

Rohin Shah: Yeah. I have been giving all these disclaimers during the podcast too, but I’m sure I missed them in some places, but I just want to note, Lucas has asked me a lot of questions that are not things I usually think about, and I just gave off-the-cuff answers. If you asked me them again, two weeks from now, I think for many of them, I might actually just say something different. So don’t take them too seriously, and treat… The AI alignment ones, I think you can take those reasonably seriously, but the things that were less about that, take them as some guy’s opinion, man.

Lucas Perry: ‘Some guy’s opinion, man.’

Rohin Shah: Yeah. Exactly.

Lucas Perry: Okay. Well, thank you so much for coming on the podcast Rohin, it’s always a real pleasure to speak with you. You’re a bastion of knowledge and wisdom in AI alignment and yeah, thanks for all the work you do.

Rohin Shah: Yeah. Thanks so much for having me again. This was fun to record.

Filippa Lentzos on Global Catastrophic Biological Risks

  • The most pressing issue in biosecurity
  • Stories from when biosafety labs failed to contain dangerous pathogens
  • The lethality of pathogens being worked on at biolaboratories
  • Lessons from COVID-19


Watch the video version of this episode here

0:00 Intro

2:35 What are the least understood aspects of biological risk?

8:32 Which groups are interested biotechnologies that could be used for harm?

16:30 Why countries may pursue the development of dangerous pathogens

18:45 Dr. Lentzos’ strands of research

25:41 Stories from when biosafety labs failed to contain dangerous pathogens

28:34 The most pressing issue in biosecurity

31:06 What is gain of function research? What are the risks?

34:57 Examples of gain of function research

36:14 What are the benefits of gain of function research?

37:54 The lethality of pathogens being worked on at biolaboratories

40:25 Benefits and risks of big data in biology and the life sciences

45:03 Creating a bioweather map or using big data for biodefense

48:35 Lessons from COVID-19

53:46 How does governance fit in to biological risk?

55:59 Key takeaways from Dr. Lentzos

Lucas Perry: Welcome to the Future of Life Institute Podcast. I’m Lucas Perry. Today’s episode is with Dr. Filippa Lentzos and explores increasing global security concerns from the use of the life sciences. As biotechnology continues to advance, the capacity for use of both the harmful and beneficial aspects of this technology is also increasing. In a world stressed by climate change as well as an increasingly unstable political landscape that is likely to include powerful new biotechnologies capable of killing millions, the challenges of biotech to global security are clearly significant. Dr. Lentzos joins us to explain the state of biotech and life sciences risk in the present day, as well as what’s needed for mitigating the risk.

Dr. Filippa Lentzos is a mixed methods social scientist with expertise in biosafety, biosecruity, biorisk assessment and biological arms control. She works at King’s College London as a Senior Lecturer in Science and International Security. Dr. Lentzos also serves as the Co-Director of the Centre for Science and Security Studies, is an Associate Senior Researcher at Stockholm International Peace Research Institute, and is a columnist for the Bulletin of Atomic Scientists. Her work focuses on transparency, confidence-building and compliance assessment of biodefence programmes and high-risk bioscience. She also focuses on information warfare and deliberate disinformation related to global health security.

And with that, I’m happy to present this interview with Dr. Filippa Lentzos.

To start things off here, we’ve had COVID pretty much blindside humanity, at least the general public. People who have been interested in pandemics and bio risk have known about this risk coming for a long time now and have tried to raise the alarm bells about it. And it seems like this other very, very significant risk is the continued risk of synthetic bio agents, engineered pandemics, and also the continued risk of natural pandemics. It feels to me extremely significant and also difficult to convey the importance and urgency of this issue, especially when we pretty much didn’t do anything about COVID and knew that a natural pandemic was coming.

So, I’m curious if you could explain what you think are the least understood aspects of synthetic and natural biological risk by the general public and by governments around the world and what you would most like them to understand.

Filippa Lentzos: I guess one of the key things to understand is that security concerns of life science research is something that we must take seriously. There’s this whole history of using the life sciences to cause harm, of deliberately inflecting disease, of developing biological weapons. But very few people know this history because it’s a story that’s suffused by secrecy. In the 20th century, biological weapons were researched and developed in several national programs, all of which were top secret, including the US one.

These programs were concealed in labs at military sites that were not listed on ordinary maps. Special code names and exceptionally high classification categories were assigned to biological agents and the projects that were devised to weaponize them. Bioweaponeers were sworn to secrecy and under constant surveillance. So, a lot of that just hasn’t become publicly available. Much of the documentation and other evidence of past programs has been destroyed. There were these concerted efforts to bring war crimes and human rights abuses to public light. Information about biological weapons programs tended to be suppressed.

One example of this is the Truth and Reconciliation Commission hearings in South Africa that followed the apartheid. When the commission hearings began to uncover details about South Africa’s biological weapons program that was called Project Coast, they were faced with delays and they were faced with legal challenges and the hearings were eventually shut down before the investigators could complete their work. Now, the head of that program became obvious to the investigators at the time who that was, but he was never brought to justice. Unbelievably, he remained a practicing medical doctor for many, many years afterwards, possibly even to this day.

What hasn’t been concealed or destroyed or silenced from past biological weapons programs often remains highly classified. So, the secrecy surrounding past programs mean that they’re not well known. But there’s also a new, contemporary context that shapes security concerns about life science research that we need to be conscious of and that I think relates back to what I think is important to know about synthetic and natural bio risks today. And that is that advances in science and technology may enable biological weapons to emerge that are actually more capable and more accessible with attacks that can be more precisely targeted and are harder to attribute.

So, synthetic biology, for example, which is one of the currently cutting-edge areas of life science research, that is accelerating our abilities to manipulate genes and biological systems. And that will have all kinds of wonderful and beneficial applications, but if the intent was there, it could also have significant downsides. So, it could, for instance, identify harmful genes and DNA sequences in a much quicker way than we’ve been able to so far. As a result of that, we could, for instance, see greater potential to make pathogens or disease-causing biological agents even more dangerous.

Or we could see greater potential to convert low-risk pathogens into high-risk pathogens that we could potentially even recreate extinct pathogens like the variola virus that causes smallpox, or way further out, we could engineer entirely new pathogens. Now, pathogens in and of themselves are not biological weapons. You need to add some kind of delivery mechanism to have a weapon. The possibilities to manipulate genes and biological systems are coming at a time when new delivery mechanisms for transporting pathogens into our bodies, into human bodies or animal bodies are also being developed.

So, in addition to the bombs and the missiles, the cluster bombs, the sprayers, and all kinds of injection devices of past biological warfare programs, it could now also be possible to use other delivery mechanisms. Things like drones or nanorobots, these incredibly tiny robots that can be inserted into our blood streams for instance, even insects, could be used as vehicles to disperse dangerous pathogens.

So, I guess to get to the bottom of your question, what I’m keen for people to understand, scientists, government officials, the general public, is that current developments in science and technology, or in the life sciences more specifically, are lowering barriers to inadvertent harms as well as to deliberate use and development of biological weapons and that there is this whole history to deliberate attempts to use the life sciences to cause harm.

Lucas Perry: It seems like there’s three main groups of people that are interested in such technology. There’s something like lone wolfs or isolated individuals who are interested in creating a lot of harm to humanity in the same way that mass shooters are. There are also small groups of people who may be interested in the same sort of thing. Then there’s this history of governments pursuing biological weapons. Could you offer some perspective about the risks of these three groups and how you would compare the current technology used for the creating of synthetic pathogens to how strong it was historically?

Filippa Lentzos: Sure. Are we heading towards a future where anyone with a PhD in bioengineering could create a pandemic and kill millions? Is that what you mean? Well, a pathogen, even a bioengineered one, does not on its own constitute a biological weapon, though you will still face issues like agent stability and dealing with large scale production and importantly dealing with efficient delivery, which is much easier said than done. In fact, what the history of bioterrorism has taught us is that the skills required to undertake even the most basic of bioterrorism attacks are often much greater than assumed.

There are various technical barriers to using biological agents to cause harm even beyond the barriers that are being reduced from advances in science and technology. The data that is available to us from past incidents of biological terrorism indicates that a bioterrorism attack is more likely to be crude, more likely to be amateurish and small scale where you’d have casualty levels in single or double digits and not in their hundreds or thousands and certainly not in their millions. Now, my own concern is actually less about lone actors.

Where I see real potential for sophisticated biological weapons in strategic surprise in the biological field is in one of those other categories that you mentioned, so it’s at the state or the state sponsored level. Let me explain. Well, I already told you a little bit about how we’ve recently seen significant advances in genetic manipulation and delivery mechanisms. These developments are lowering barriers to biological weapons development, but that’s really only part of the picture, because in making threat assessments, it’s also important to look at the social context in which these technical developments are taking place.

One of the things we’re seeing there in that social context is a build up into use capacities? What we’re seeing is that high containment labs that are working with the most dangerous pathogens are rapidly being constructed all over the globe. So, they’re now more people and more research projects than ever before working with and manipulating very dangerous pathogens and there are more countries than ever before that have biodefense programs. There’s around 30 biodefense programs that are openly declared. The trends we’re seeing is that these numbers are increasing.

It’s entirely legitimate to have biodefense programs and they do a lot of good, but a side effect of increasing bio-preparedness and biodefense capacities is that capacities for causing harm, should the intent be there, and that’s the crucial part, also increase. So, one person may be setting up all this stuff for good, but if somebody else comes in with different intent, with intent to do harm, that same infrastructure, that same material, that same equipment, that same knowledge, can be turned towards causing harm or creating biological weapons.

Now, another thing we’re seeing that won’t have escaped your notice is the increasingly unstable and uncertain geopolitical landscape. The world that many of us grew up in and know is one in which America was a clear, dominant power. We’re now moving away from that, away from this hegemonic or unipolar power structure towards an international system that is increasingly multipolar. The most clearly rising power today is of course China, but there are others too. Russia is still there. There’s India, there’s Brazil to name a few. Those are things in the social context that we need to pay attention to.

We’re also seeing rapidly evolving nature of conflict and warfare themselves are changing. And that’s changing the character of military challenges that are confronting states. Hybrid warfare, for instance, which blends conventional warfare with irregular warfare and cyber warfare, is increasingly likely to compliment classical military confrontation. So, states that are increasingly outmatched by conventional weapons may for instance start to view novel biological weapons as offering some kind of advantage, some kind of asymmetric advantage, and a possible way to outweigh strategic imbalances.

So, states in this kind of new form of conflict, new form of warfare, may see biological weapons as somehow providing an edge or a military advantage. We are also seeing the defense programs of some states heavily investing in the biological sciences. Again, could well be for entirely legitimate purposes, but it does also raise concerns that adversaries may be looking at those kinds of investments and thinking hedging their bets and similarly investing in more biological programs. These investments, I think, are also an indication that there are some real concerns that adversaries are harnessing or trying to harness biotechnology for nefarious purposes.

And we’ve seen some political language to that effect too, but a lot of this is going under the radar. So, all of these things, and there are more, the flagrant breach of the Chemical Weapons Convention or continuous flagrant breaches of the Chemical Weapons Convention for example, the use of chemical weapons in Syria, or the use of very sophisticated chemicals like Novichok in the UK on Skripal, the Russian, as well as other cases is one other sort of context that plays in, or even our recent experiences of natural disease outbreaks in here. COVID is obviously a key example, but it’s not so long ago we’ve had all kinds of other outbreaks.

Ebola just a few years ago. There’s Zika, there’s MERS, there’s all kinds of other emerging diseases. All of these could serve to focus attention on deliberate outbreaks. And all of these various elements of the social context as well as these technical developments could produce an environment in which a potential military or political utility for biological weapons emerges that alters the balance of incentives and disincentives to comply with the international treaty that prohibits biological weapons.

Lucas Perry: Could you explain the incentives of why a country would be interested in creating a synthetic pathogen when inevitably it would seem like it would come back and harm itself?

Filippa Lentzos: Well, it’s doesn’t have to be an infectious pathogen. What we’re seeing today with COVID, for instance, is an infectious pathogen that spreads uncontrollably throughout the world. But states don’t have to. Not all dangerous pathogens are infectious in that way. Anthrax, for instance, doesn’t spread from person to person through the air. So, there are different kinds of pathogens and states and non-state actors will have different motivations for using biological weapons or biological agents.

One of those which I mentioned earlier is for instance if you feel that another country… you are outmatched conventionally by conventional weapons, you may want to start to develop asymmetric weapons. That would be an example where a state might want to explore developing biological weapons. But of course, we should probably mention that there is this thing called The Biological Weapons, this international treaty, which completely prohibits this class of weaponry. Historically, there’s really only been two major powers that have developed sophisticated biological weapons programs. That is the United States and the Soviet Union.

Today, there are no publicly available documents or any policy statements suggesting that anyone has an offensive biological weapons program. There are many countries who have defensive programs and that’s entirely legitimate. There is no indication that there are states that have offensive programs to date. I think the real concern is about capacities that are building up through biodefense programs, but also through regular bio-preparedness programs, and that’s something that’s just going to increase in future.

Lucas Perry: I’m curious here if you could also explain and expand upon the particular strands of your research efforts in this space.

Filippa Lentzos: Sure. I mean, it’s very much related to the sorts of things we’ve been talking about. One strand that I focus on relates to transparency, confidence building, and compliance assessment of biodefense programs, where I look at how we can build trust between different countries with biodefense programs to trust that they are complying with the Biological Weapons Convention. I’m also looking at transparency around particular high-risk bioscience, so things or projects or research involving genome editing for example, or potentially pandemic pathogens like influenza or coronaviruses.

Another strand that I’m interested in or that I’m looking at focuses on emerging technologies and on governance around these emerging technologies and unresponsible innovation. And there I look particularly at synthetic biology, also a little bit at artificial intelligence, deep learning and robotics, how these other emerging areas are coming into the life sciences and affecting their development and the direction they’re taking, the capacities that are emerging from this kind of convergence between emerging technologies and how we can govern that better, how we can provide better oversight.

Now, one of the projects that I’ve been involved in that has got a lot of press recently is a study that I carried out with Greg Koblentz at George Mason University where we mapped high biocontainment laboratories globally. I mentioned earlier that countries around the world are investing in these kinds of labs to study lethal viruses and to prepare against unknown pathogens. Well, that construction boom has to date resulted in dozens of these commonly called BSL-4 labs around the world. Now, significantly more countries are expected to build these kinds of labs in the wake of COVID 19 as part of a renewed emphasis on pandemic preparedness and response.

In addition, gain-of-function research with coronaviruses and other zoonotic pathogens with pandemic potential is also likely to increase as scientists are seeking to better understand these viruses and to assess the source of risks that they pose of jumping from animals to humans or becoming transmissible between humans. Now, of course, clinical work and scientific studies and pathogens are really important for public health and for disease prevention, but some of these activities pose really significant risks. Surges in the number of labs and expansion and the high risk research that’s carried out within them exacerbate safety and security risks.

But there is no authoritative international resource tracking the number of these kinds of labs out there as they’re being built. So, there is no international body that has an authoritative figure on the number of BSL-4 labs that exist in the world or that have been established. Equally, there is no real international oversight of the sort of research that’s going on in these labs or the sorts of biosafety and biosecurity measures that they have implemented. So, what our study did was to provide a detailed interactive map of BSL-4 labs worldwide that contains basic information on when they were established and the size of the lab labs and some indicators of biorisk management oversight.

That map is publicly available online at globalbiolabs.org. You can go and see for yourself. It’s basically a very large Google map where the labs are indicated and you can scroll over the labs and then up pops information about when it was established, how big it is, what sorts of biorisk management indicators there are, are they members of national biosafety associations? Do they have regulations related to by safety? Do they have codes of conduct? Et cetera, those kinds of things. That all comes up there, so you can go and see for yourself. That’s a resource that we’ve made publicly available on the basis of our project.

Looking at the data we then collated, this was really the first time this kind of concerted effort was made to identify these various labs and bring all that information together. And some of our key findings from looking at that data were that… Well, the first thing is BSL-4 labs are booming. We can see a really quite steep increase in the number of labs that have been built over the last few years. We found that there are many more public health labs than there are biodefense labs. So, about 60% of the labs are public health labs, not focused on defense, but resourced out of health budgets.

We also found that there are many smaller labs and larger labs. In newspapers and on TV, we keep seeing photos of the Wuhan Institute of Virology’s BSL-4 lab.

In terms of oversight, some of our other findings were that sound biosafety and biosecurity practices do exist, but they’re not widely adopted. There’s a lot of difference in between the kinds of biosafety and biosecurity measures that labs adopt and implement. We also found that assessments to identify life science research that could harm health safety or security are lacking in the vast majority of countries that have these BSL-4 labs. So, as I said, that’s one of the studies that’s got a lot of press recently and part of that is because of its relationship to the current pandemic and the lack of some solid information, some solid data on the sort of labs that are out there and on the sorts of research that’s being done.

Lucas Perry: Do you have a favorite story of a particular time that a BSL lab failed to contain some important pathogen?

Filippa Lentzos: Well, there are all kinds of examples of accidental releases. In the UK, for instance, where I’m based, a very long time ago, there was work with variola virus that causes a smallpox, was worked in a sort of high rise building that had multiple floors and the variola virus escaped into the floor above and infected somebody there. That was, I think, at the end of the ’70s. That was the very last time that someone was infected by smallpox in the UK. More recently in the UK, there’s also been the escape of the foot and mouth virus from a lab.

Now, this was not the very large foot and mouth outbreak that we had in the early 2000s, which you know killed millions of animals. I still remember the piles of animal corpses dotted around the country and you could still smell the burning carcasses on the motorway as you drove past, et cetera. That was not caused by a lab leak, but just two, three, four years later, there was a foot and mouth disease virus that escaped from a lab through a leaking pipe that did go on to cause some infections. But by that stage, everyone was very primed to look out for these kinds of infections and to respond to them quickly.

So, that outbreak was contained fairly rapidly. I mean, there are also many examples elsewhere, also in the United States. I mean, there’s the one example where you had variola virus found in a disused closet at the NIH after many years and they were still viable. I think that’s one of the ones that ranked pretty highly in the biosafety community’s memory and maybe even in your own. It was not that long ago, half a dozen years ago or so.

Lucas Perry: What do you think all these examples illustrate of how humans should deal with natural and synthetic pathogens?

Filippa Lentzos: Well, I think it illustrates that we need better oversight, we need better governance to ensure that the life science research done is done safely, it’s done securely, and it’s done responsibly.

Lucas Perry: Overviewing all these BSL safety labs and all these different research threads that you’re exploring, what do you think is the most pressing issue in biosecurity right now, something that you’d really like the government or the public to be aware of and take action on?

Filippa Lentzos: Well, I think there’s a really pressing need to shore up international norms and treaties that prohibit biological weapons. I mentioned the Biological Weapons Convention. That is the key international instrument for prohibiting biological weapons, but there are also others. The arms control communities is not in great shape at the moment. It needs more high profile, political attention, it needs more resources. And I think with more and more breaches that we’re seeing, not on the biological side, but on other sides, breaches of international treaties, I think we need to make sure there is this renewed effort and commitment to these treaties.

So, I think that’s one thing, one issue, that’s really pressing in biosecurity right now. Another is really raising awareness and increasing sensitivities in scientific communities to potentially accidental or inadvertent or deliberate risks of the life sciences. And we see that very clearly in the data that’s coming out of the BSL-4 study that I talked to you about, that that’s something that’s needed, not just what we saw there as actually looking at do they have any laws in the books or do they have any guidance on paper or do they have any written down codes of conduct or codes of practice? That’s really important.

It’s really important to have these kinds of instruments in place, but it’s equally important to make sure that these are implemented and adopted and that there is this culture of safe, secure, and responsible science. That’s something that we didn’t cover in that specific project, but it’s something that some of my other work has drawn attention to and the work of many others as well. So, we do need to have this regulatory oversight governance framework in place, but we also need to make sure that that is reflected or echoed in the culture of the scientists and the labs that are carrying out life science research.

Lucas Perry: One other significant thing going on in the life sciences in terms of biological risk is gain-of-function research. So, I’m curious if you could explain what gain-of-function research is and how you see the debate around the benefits and risks of it.

Filippa Lentzos: Well, gain-of-function research is a very good example of life science research that could be accidentally, inadvertently or deliberately misused. Gain-of-function means different things to different people. To virologists, it generally just means genetic manipulation that results in some sort of gained function. Most of the time, these manipulations result in loss of function, but sometimes different kinds of functions of pathogens can be gained. Gain-of-function has got a lot of media coverage in relation to the discussion around the origins of the pandemic or of COVID.

And here, gain-of-function is generally taken to mean deliberately making a very dangerous pathogen like influenza or coronavirus even more dangerous. So, what you’re trying to do is you’re trying to make it spread more easily, for example, or you’re trying to change its lethality. I don’t think gain-of-function research in and of itself should be banned, but I do think we need better national and international oversight of function experiments. I do think that a wider group of stakeholders beyond just the scientists doing the research themselves and their funders, I think that a wider group of stakeholders should be involved in assessing what is safe, what is secure, and what is responsible gain-of-function research.

Lucas Perry: It seems very significant, especially with all these examples that you’ve illustrated of the fallibility of BSL labs. The gain-of-function research seems incredibly risky relative to the potential payoffs.

Filippa Lentzos: Yeah, I think that’s right. I mean, I think it is considered one of the examples of what has been called dual use research of concern or experiments that have a higher potential to be misused. By that, I mean deliberately, but also in terms of inadvertently or even accidentally because the repercussions, the consequences have the potential to be so large. That’s also why we saw when some of the early gain-of-function experiments gained media attention back in 2011, 2012, that the scientific community itself reacted and said, “Well, we need to have a moratorium.

We need to have a pause on this kind of research to think about how we govern that, how we provide sufficient oversight over the sorts of work that’s being done so that the risk benefit assessments are better essentially.” I think there will be many who argue that… myself among them, that the discussion that was had around gain-of-function at that time were not extensive enough, they were not inclusive enough, there were not enough voices being heard or part of the decision-making process in terms of the policies that came out of this in the United States. To some extent, I think that’s why we’re, again, back at the table now with the discussions around the pandemic origins.

Lucas Perry: Do you have any particular examples of gain-of-function research you’d be interested in sharing? It seemed like a really significant example was what was happening in Wisconsin.

Filippa Lentzos: Sure. And that was the one that was the work in Wisconsin and at the Erasmus University in the Netherlands. What they were trying to do there was they were working with influenza or avian flu and they were seeing if they were able to give that virus a new function, so enable it to spread, not just among birds, but also from birds to mammals, including humans, including ourselves. So, they were actively trying to make it not just affect birds, but also to affect humans.

And they did so successfully, which made that virus more dangerous and that was what that media fuel was about and the discussions at the time were that many felt that the benefits of that research did not outweigh very significant potential risks, the very significant risks that that research involved.

Lucas Perry: What are the benefits of that sort of gain-of-function research?

Filippa Lentzos: Well, the ones that carried out that sort of research both at the time, but also the sorts of gain-of-function research that’s been going on at the Wuhan Institute of Virology, some of it which has been funded by American money, some of it which has been done in collaboration with American Institute argues that in order to prepare for pandemics, we need to know what kind of viruses are going to hit us. New and emerging viruses generally come, spill over from the animal kingdom into humans, so they actively go and look for viruses in the animal kingdom.

In this case, in the coronavirus case, the Wuhan Institute of Virology, they were actively looking in bat populations to see what sort of viruses exist there and what their potentials are for spilling over into humans. That’s their justification for doing that. My own view is that that’s incredibly risky research and I’m not sure and I don’t feel that that sort of justification really outweighs the very significant risks that it involves. How can you possibly hit upon the right virus in the thousands and thousands of viruses that are out there and know how that will then mutate and get modified as it hits the human population?

Lucas Perry: These are really significant and quite serious viruses. You explained an example earlier about this UK case where the final people to die from smallpox was actually from a BSL lab leak. There’s also this research in Wisconsin on avian flu. So, could you provide a little bit of a perspective on, for example, the infection rate and case fatality rate of these kinds of viruses that they’re working on at BSL labs, that they have at BSL labs, that they might be pursuing gain-of-function research on?

Filippa Lentzos: Yeah. I mean, certainly in terms of the coronavirus, what we’ve seen there is that that is clearly many people have died, many people have got infected, but that’s not considered a particularly infectious or particularly lethal pathogen when it comes to pandemics. We’ve seen much more dangerous pathogens that could create pandemics or that are being worked with in laboratories.

Lucas Perry: Yeah. Because some of these diseases, it seems, the case fatality rate gets up to between 10 and 30%, right? So, if you’re doing gain-of-function research on something that’s already that lethal and that has killed hundreds of millions of people in the history of life on earth, with the history of lab leaks and with something so infectious and spreadable, it seems like one of the most risky things humanity is doing on the planet currently.

Filippa Lentzos: Yes. I mean, one of the things gain-of-function is doing is looking at lethality and how to increase lethality of pathogens. There are also other things that gain-of-function is doing, but that is taking out a large part of the equation, which is the social context of how viruses spread and mutate. There are, for instance, things we can do to make viruses spread less and be less lethal. There are active measures we can take equally, there are responses that could increase the effect of viruses and how they spread.

So, lethality is one aspect, a potential pandemic, but it is only one aspect, right? There are these many other aspects too. So, we need to think of ourselves much more as active players, that we also have a role to play in how these viruses spread and mutate.

Lucas Perry: One thing that the digital revolution has brought in is the increase and the birth of big data. Big data can be used to detect the beginning of outbreaks, to detect novel diseases, and to come up with cures and treatments for novel and existing diseases. So, I’m curious what your perspective is on the benefits and risks of the increase of big data in biology, both to health and societies as well as privacy and the like.

Filippa Lentzos: Well, you pointed to many of the benefits that big data has. There certainly are benefits, but as with most things, there are also a number of downsides. I do believe that big data combined with the advances that we’re seeing in genomic technologies as well as with other areas of emerging technology, so machine learning or AI, this poses a significant threat. It will allow an evermore refined record of our biometrics; so our fingerprints, our iris scans, our face recognition, our CCTV cameras that can pick up individuals based on how they walk, all these kinds of biometrics.

It will also allow a more refined record of our emotions and behaviors to be captured and to be analyzed. I mean, you will have heard of companies that are now using facial recognition on their employees to see what kind of mood they’re in and how they engage with clients, et cetera. So, governments are gaining incredible powers here, but increasingly, it’s private companies that are gaining this sort of power. What I mean by that is that governments, but as I said, increasingly private companies, will be able to sort, to categorize, to trade, and to use biological data far more precisely than they have ever been able to do before.

That will create unprecedented of possibilities for social and biological control, particularly through individual surveillance, if you like. So, these game-changing developments will deeply impact how we view health, how we treat disease, how long we live, and how more generally we consider our place on the biological continuum. I think they’ll also radically transform the Julius nature of biological research, of medicine, of healthcare. In terms of my own field of biosecurity, they will create the possibility of novel biological weapons that target particular groups of people and even individuals.

Now, I don’t mean they will target Americans where they will target Brits or they will target Protestants or they will target Jews or they will target Muslims. That’s not how biology works. Genes don’t understand these social categories that we put onto people. That’s how we socially divide people up, but that’s not how genetics divides people up. But there are groupings also genetically that go across cultures, nations, beliefs, et cetera. So, as we come to have more and more precise biological data on these different groups, the possibility of targeting these groups for harm will also be realized.

So, in the coming decade, managing the fast and broad technological advances that are now underway will require new kinds of governance structures that we need to put in place and these new structures need to draw on individuals in groups with cross-sectoral experience; so, from business, from academia, from politics, from defense, from intelligence, and so on to identify emerging security risks and to make recommendations for dealing with them. We need new kinds of governance structures, new kinds of advisory bodies that have different kinds of stakeholders on them to the ones that we have traditionally had.

Lucas Perry: In terms of big data and the international community, with the continued risks of natural pandemics as well as synthetic pandemics or other kinds of a biological agents and warfare, it’s been proposed, for example, to create something like a bio weather map where we have like a widespread, globally distributed early warning detection system for biological agents that is based off of big data or is itself big data. So, I’m curious if you have any perspective and thoughts on the importance of big data in particular for defenses against the modern risks of engineered and natural pandemics.

Filippa Lentzos: Yes, I do think there was a role to play here for data analysis tools of big data. We are, I think, already using some tools in this area where you have, for instance, analysis of social media usage, words that pop up on social media uses, or you have analysis of the sorts of products that people are buying in pharmaceutical companies. So, if there is some kind of disease spreading, people are getting sick and they’re talking about different kinds of symptoms, you are able to start tracking that, you’re able to start mapping that.

All of a sudden, all kinds of people in say Nebraska are going to the pharmacy to buy cough medicine or something to reduce temperature or there’s a big spike for instance, you might want to look into that more. That’s an indicator, that’s a signal that you might want to look at that more. Or if you’re picking up keywords on internet searches or on social media where people are asking about stomach cramps or more specific kinds of symptoms, that again is another kind of signal, you might want to look more into that.

So, I think some of these tools are, are definitely already being developed, some are already in use. I think they will have advantages and benefits in terms of preparing for both natural, but also inadvertent, accidental or deliberate outbreaks of disease.

Lucas Perry: We’re hopefully in the final stages of the COVID-19 pandemic. When we reflect back upon it, it seems like it can be understood as almost like a minimally viable global catastrophe or a minimally viable pandemic, because there’s been far worse pandemics, for example in the past, and it’s tragically taken the lives of many, many people. But at the same time, the fatality rate is just a bit more than the flu and a lot less than many of the other pandemics that humanity has seen in the past few hundred thousand years.

So, I’m curious what your perspective is on what we can learn in the areas of scientific, social, political, and global life, from our experience with the COVID-19 pandemic to be better prepared for something that’s more serious in the future, something that’s more infectious, and has a higher case fatality rate.

Filippa Lentzos: Well, I think, as you said, in the past, disease has been much more present in our societies. It’s really with the rise of antibiotics and the rise of modern healthcare that we’ve been able to suppress disease to the extent that it’s no longer such a pressing feature in our daily lives. I think what the pandemic has done to a whole generation is really it has been a shot across the bow, really crystallized the incredibly damaging effects that disease can have on society.

It’s been this wake up call or this reality check. I think we’ve seen that reflected also politically. International developments like the UN’s Biorisk Working Group that’s been established by the secretary general or efforts by states to develop a new international treaty on pandemics are concrete evidence of increasing awareness of the challenges that diseases pose to humankind. But clearly, that’s not enough. It hasn’t been enough, what we’ve had a place. Clearly, we need to be better prepared. And I guess for me, that’s one of the bigger takeaways from the pandemic.

Equally, what the pandemic origin debate has done is to show that whether or not the pandemic resulted from a lab leak, it could have resulted from a lab leak, it could ironically or tragically have been the result of scientific research actually aimed at preventing future pandemics. So, clearly for me, a huge takeaway is that we need better oversight, we need better governance structures ensure safe, secure, and responsible life science research. Potentially, we also need to rethink some of our preparedness strategies.

Maybe actively hunting for viruses in the wild, mutating them in the lab to see if that single virus might be the one that hits us next, the one that spills over, isn’t the best strategy for preparing for pandemics in the future. But COVID has also highlighted a more general problem, one I think that’s faced by all governments, and that is, how can we successfully predict and prepare for the wide range of threats that there are to citizens and to national security? Some threats like COVID-19 are largely anticipated actually, but they’re not adequately planned for as we’ve seen.

Other threats are not anticipated at all and for the most part are not planned for. The other side, some threats are planned for, but they fail to materialize as predicted because of errors and biases in the analytic process. So, we know that governments have long tried to forecast or to employ a set of futures approaches to ensure they are ready for the next crisis. In practice, these are often general, they’re ad hoc, they’re unreliable, they’re methodologically and intellectually weak, and they lack academic insight. The result is that governments are wary of building on the recommendations of much of this future’s work.

They avoid it in policy planning, in real terms funding, and ultimately in practice and institutionalization. What I and many of my colleagues believe is that we need a new vision of strategic awareness that goes beyond the simple idea of just providing a long-term appreciation of the range of possibilities that the future might hold to one that includes communication with governments about their receptivity to intelligence, how they understand intelligence, how they absorb other kinds of intelligence from private corporations, from academia, et cetera, as well as the manner in which the government acts as a result.

So, strategic awareness to my mind and to that of many others should therefore be conceptualized in three ways. You should first look more seriously and closely at threats. Second, you should invest in prevention and foresighted action. Third, you should prepare for medication, crisis management, and bounce back in case a threat can’t be fully prevented or deterred. This kind of thinking about strategic awareness will require a paradigm shift in how government practices strategic awareness today. And my view is that the academic community must play an integral part in that.

Lucas Perry: Do you have any particular governance solutions that you’re really excited about right now?

Filippa Lentzos: I don’t think there’s a magic bullet. I don’t think there’s one magic solution to ensuring that life science research is safe, that it’s secure, and that it’s carried out responsibly. I think in terms of governance, we need to work both from the top-down and from the bottom-up. We need to have in place both national laws and regulations, statutory laws and regulations. We need to have in place institutional guidance, we need to have in place best practices. But we also need a lot of the commitment, we also need a lot of awareness coming from the bottom-up.

So, we need individual scientists, groups of scientists to think about how their work can best be carried out safely so they can make codes of ethics or codes of practice themselves, they can educate others, they can think through who needs to be involved beyond their own expert community in risk assessing the kinds of research that they’re interested in carrying out. So, we need both this top-down government-enforced, institutionally-enforced governance as well as grassroots governance. Only by having both of these aspects, both of these kinds of governance measures, can we really start to address the potential downsides of life science research.

Lucas Perry: All right. Just to wrap things up, I’m curious if you have any final words or thoughts for the audience or anyone that might be listening, anything that you feel is a crucial takeaway on this issue? I generally feel that it’s really difficult to convey the significance and gravitas and importance of this. So, I’m curious if you have any final words about this issue or a really central key takeaway you’d like listeners to have.

Filippa Lentzos: I think when we’re looking at our current century, this will be the century not of chemistry or physics or engineering, that was the last century, this will be the century of biology and it will be the century of digital information and of AI.

I think this combination, which we talked about earlier, when you combine biological data with machine learning, with AI, with genomic technologies, you get incredible potential of precise information about individuals. I think that is something we are going to struggle with in the years to come and we need to make sure that we are aware of what is happening, that we are aware that when we go buy a phone and we use the face recognition software, which is brilliant, that it can also have downsides, and all these little individuals actions, all these technologies that we just readily accept because they do have upsides in our life, they can also have potential downsides.

I do think we need to make sure we also developed this critical sense or this ability to be critical, think critically about what these technologies are doing to us as individuals and to us as societies. I guess that is the things I would like people to take away from our discussion.

Lucas Perry: All right. Well, thank you so much for coming on the podcast. I really can’t think of too many other issues that are as important as this. It’s certainly top three for me. Thank you very much for all of your work on this, Dr. Lentzos, and for all of your time here on the podcast.

Filippa Lentzos: Thanks for having me, Lucas.

Susan Solomon and Stephen Andersen on Saving the Ozone Layer

  • The industrial and commercial uses of chlorofluorocarbons (CFCs)
  • How we discovered the atmospheric effects of CFCs
  • The Montreal Protocol and its significance
  • Dr. Solomon’s, Dr. Farman’s, and Dr. Andersen’s crucial roles in helping to solve the ozone hole crisis
  • Lessons we can take away for climate change and other global catastrophic risks


Watch the video version of this episode here

Check out the story of the ozone hole crisis here

0:00 Intro

3:13 What are CFCs and what was their role in society?

7:09 James Lovelock discovering an abundance of CFCs in the lower atmosphere

12:43 F. Sherwood Rowland’s and Mario Molina’s research on the atmospheric science of CFCs

19:52 How a single chlorine atom from a CFC molecule can destroy a large amount of ozone

23:12 Moving from models of ozone depletion to empirical evidence of the ozone depleting mechanism

24:41 Joseph Farman and discovering the ozone hole

30:36 Susan Solomon’s discovery of the surfaces of high altitude Arctic clouds being crucial for ozone depletion

47:22 The Montreal Protocol

1:00:00 Who were the key stake holders in the Montreal Protocol?

1:03:46 Stephen Andersen’s efforts to phase out CFCs as the co-chair of the Montreal Protocol Technology and Economic Assessment Panel

1:13:28 The Montreal Protocol helping to prevent 11 billion metric tons of CO2 emissions per year

1:18:30 Susan and Stephen’s key takeaways from their experience with the ozone hole crisis

1:24:24 What world did we avoid through our efforts to save the ozone layer?

1:28:37 The lessons Stephen and Susan take away from their experience working to phase out CFCs from industry

1:34:30 Is action on climate change practical?

1:40:34 Does the Paris Agreement have something like the Montreal Protocol Technology and Economic Assessment Panel?

1:43:23 Final words from Susan and Stephen


Lucas Perry: Welcome to the Future of Life Institute Podcast. I’m Lucas Perry. This is a special episode with the winners of the 2021 Future of Life Award. This year’s winners are Susan Solomon, Stephen Andersen, and Joseph Farman, who all played an essential part in the efforts related  to identifying and mending the ozone hole. In this podcast, we tell the story of the ozone hole from the perspective of Drs. Solomon and Andersen as  participants to the mystery of the ozone hole and the subsequent governance efforts related to mending it. Unfortunately, Joseph Farman passed away in 2013, so his role in this story will be told through our guests on his behalf.

For those not familiar with the Future of Life Award, this is a $50,000/person annual prize that we give out to honor unsung heroes who have taken exceptional measures to safeguard the future of humanity. In 2017 and 2018 the award honored Vasili Arkhipov and Stanislav Petrov for their roles in helping to avert nuclear war. In 2019 the award honored Dr. Matthew Meselson for his contributions to getting biological weapons banned. In 2020, the award honored Bill Foege and Viktor Zhdanov for their critical contribution to eradicating small pox and thus saving roughly 200 million lives so far.

For some background on this year’s winners, Dr. Susan Solomon was the Chemistry and Climate Processes Group of the National Oceanic and Atmospheric Administration until 2011. She now serves as the Ellen Swallow Richards Professor of Atmospheric Chemistry and Climate science at MIT. Dr. Solomon led an Antarctic ozone research expedition which both confirmed that CFCs caused the ozone hole, and showed that sunlit cloud tops catalyzed the ozone destruction process to be much faster.

Dr. Stephen Andersen is the American Director of Research at the Institute for Governance and Sustainable Development, and is former co-chair of the Montreal Protocol Technology and Economic Assessment Panel. Andersen’s tireless efforts brought together leaders from industry, government and academia to implement the  needed changes in CFC use to mend the ozone hole. His efforts played a critical role in making the Montreal Protocol successful.

Joseph Farman was a British geophysicist who worked for the British Antarctic Survey. In 1985, his team made the most important geophysical discovery of the 20th century: the ozone hole above the Antarctic. This provided a stunning confirmation of the Rowland-Molina hypothesis that human-made chlorofluorocarbons were destroying the ozone layer, and much faster than predicted. This galvanized efforts to take action to mend the ozone hole.

And with that, I’m happy to present the story of the ozone hole and the fight to fix it with Susan Solomon and Stephen Andersen 

Lucas Perry: So I’d like to start off with the beginning where it’s the early 20th century, and we have CFCs, chlorofluorocarbons, which is a difficult word to say. And they’re in quite a lot of our manufacturing and commercial goods. So I’m curious if you could start there by just explaining what CFCs are and what their role was in the early 20th century.

Susan Solomon: They weren’t actually discovered until I think around the twenties, if I recall, maybe Steven can correct me on that, but they were initially used in air conditioning. So that was a great advance over using things like ammonia, which is toxic and explosive and all kinds of horrible things, and the great thing about the CFCs is that they’re non-toxic at least, when you breathe them, although they’re very toxic to the ozone layer, they’re not toxic to you directly. And they became really widespread in use though, when they began to be used as spray cans in aerosol propellants and that didn’t really happen until somewhat later. I think that the spray can business really started exploding in the Post-World War Two era, probably in the 50’s and 60’s. And then it was discovered that these things have very long lifetimes in the atmosphere, they live, depending on which one you talk about, for some 50 to a 100 years.

So that is just staggering because what it means is that every year, whatever we put in, almost all of it will still be there the next year. And it’ll just pile up and pile up and pile up. So initially, a few people who worked on it thought “Hey, this is great! It’s a terrific tracer for atmospheric motion, how cool is that?” But it turned out that although that’s somewhat true in the lower atmosphere, in the upper atmosphere, they actually break down and make chlorine atoms, and those chlorine atoms can go on to destroy ozone. And we first became aware of this through some wonderful work by Molina and Rowland in the mid-seventies, which later won the Nobel Prize for chemistry, all along with Paul Crutzen.

Stephen Andersen: I could circle back on a little bit of the details of the technology if you’d like?

Susan Solomon: Sure, go for it, Steve.

Stephen Andersen: The year was in the twenties, 1929, and it was invented by Thomas Midgley, who was working for Frigidaire Division of General Motors, and just as Susan said, this was viewed as a wonder chemical because the chemicals at that time were things like methylene chloride and ammonia and even gasoline is a refrigerant, so it was very dangerous there were lots of injuries, by that time America’s waters were polluted, so it was difficult to use ice from ponds.

And so it was immediately commercialized to replace these flammable and toxic refrigerants, and then just as Susan said, the companies that produced it slowly looked at other markets; aerosol starting with pesticides for soldiers in World War Two, and then finding a commercial market and then solvents, because it’s a very effective solvent for electronics and aerospace, and then there were other uses that came along such as making rigid and flexible foam, and then finally towards the end there were elegant uses such as hospital sterilization, and as the aerosol for metered-dose inhalers that are used for asthma patients. So it had extraordinary uses and it was not appreciated until 1974 when Mario Molina and Sherry Rowland discovered that these chemicals could destroy the ozone layer.

Lucas Perry: So before we get to Molina and Rowland, I was curious if we could actually start with James Lovelock because we’re releasing and creating all of these CFCs, and it seems like he’s the first person who actually takes a look at these things in the atmosphere. So could you explain James Lovelock’s role in this story?

Susan Solomon: Sure. He invented the gas chromatograph and made a lot of money off of it and became independently wealthy because it’s a very useful instrument for measuring all kinds of things. And he took it on a… He was British, and he took it on a British research vessel and sailed from the Northern Hemisphere down to almost the Antarctic, and showed that he could understand exactly what the distribution of the CFC that he was measuring was, based on the amount that had been emitted. And he said “Oh, isn’t that cool? It’s a great tracer for atmospheric motion”, as I’ve mentioned before. In fact, I’m pretty sure in his paper, there was some kind of sentence like “this couldn’t ever conceivably pose any hazard because it’s so non-toxic”, but of course, having less of an ozone layer is not good for life on the planet because ozone absorbs ultraviolet light that is very, very important for protecting us from sunburn and cataracts and protecting animals and plants and all kinds of things.

Susan Solomon: So depletion of the ozone layer is actually a threat to both humanity and the planet. I do want to say it wasn’t until somewhat later, 1985, when scientists from the British Antarctic Survey actually discovered the ozone hole and what they were doing was measuring total ozone in the Antarctic, they’d been doing it since the 1950s, and they were able to show that sometime around the late seventies, it started dropping like a rock way, more ozone depletion than Molina and Rowland had ever imagined.

So it turns out that these chlorofluoro chemicals are actually even more damaging to the ozone layer in the Antarctic, and actually also to some extent at mid-latitudes as we now know, than we originally thought. So it’s been sort of a cautionary tale of don’t be too sanguine about any chemical made in the lab. When you make something in the lab that nature doesn’t make, I think you should always do a double-take, especially if it has a very long atmospheric lifetime and builds up in the way that I described. So that means you can’t get rid of it if you stop using it. You can’t eventually, but it’s going to take a long time for the atmosphere to cleanse itself.

Lucas Perry: Could you explain why James Lovelock was initially interested in CFCs and what his investigation led to scientifically?

Susan Solomon: As far as I know, he really just wanted to see what their distribution was like to get some sort of a handle on what they might be… What his instrument might be useful for. I don’t actually know of a use beyond that, do you, Steven?

Stephen Andersen: No, I think that’s right. He was looking for gases that were indicators, as you suggest, and of course he had a device that would measure other chemicals, but I think he was immediately struck by the fact that he was seeing the same chemical at all locations that he sampled, and then he made the natural connection of saying “Where did this chemical originate? How long did it take to mix in the lower atmosphere”? So I think it was a good, solid, scientific inquiry of a very intelligent person with a new instrument.

Susan Solomon: Maybe we should clarify, I said he went all the way down almost to the Antarctic, but I neglected to underscore that of course there is no chloro fluoro carbon emission in the Antarctic, right? At that time there was nobody even… There were no stations, there was nobody there. So any chlorofluorocarbon that could get there had to of have gotten there by atmospheric transport and it would also tell you that it has to have a fairly long lifetime because if you emit, let’s just say sulfur dioxide from a power plant in the Ohio Valley, yeah it’s a serious issue, it can cause acid rain, it can cause little particles that are bad for your lungs, it does a lot of bad things, but it’s not going to be found in the Antarctic. It just doesn’t have that long of a lifetime, it rains out. So this proved that they were a great tracer in his mind I think that’s what he was attracted by.

Lucas Perry: We’re in this world where CFCs are basically being used for a ton of different applications. Our current understanding at that time was that they were nonreactive and non-toxic, so basically a wonder chemical that could be used for all sorts of things and was much safer than the kinds of chemicals that we were using at the same time. And here comes James Lovelock, who, from my reading, it seemed like he first got interested in it because his view from his house was hazy, and he didn’t know why it was hazy and he wanted to figure out if it were manmade chemicals or what this pollution that was obscuring his vision was.

And so he starts measuring all these CFCs everywhere, and now we’re in a world where it seems very clear that the CFCs are abundant in the lower atmosphere. So let’s start to pivot into Mario Molina and Frank Rowland’s role in this story and how we begin to move from noticing that there are so many CFCs in the atmosphere to finding out that there might be a problem.

Susan Solomon: I will say that Dr. Rowland has passed away, unfortunately, so has Molina more recently, but he never went by Frank, he went by Sherry. His name was indeed F Sherwood Rowland, but he was known to everyone as Sherry Rowland.

Lucas Perry: Okay.

Susan Solomon: Go ahead, Steve, do you want to take this one?

Stephen Andersen: Yeah, sure. The story of it is actually another great science story. Mario Molina had finished his doctors degree at University of California, Berkeley, and had taken a post-doctorate study with Sherry Rowland at University of California Irvine, and they looked at four or five interesting topics, and I think that the history is that Molina saw this one as being particularly intriguing, even though it was slightly outside either of their expertise.

So it was a stretch for them, but it gave them a chance to look at something that could be potentially very important. And then the story is that as they began to investigate, it started to seem more and more obvious to them. And it became a rush for a conclusion because they were worried about the effect of their work. There’s one story that Sherry Rowland tells us, that he came home from work one day and his wife, Joan, asked “How did your day go? How is your work”?

And he replied something like “Well, the work is fantastic, but I think the earth is ending”. So you can imagine the tension, the creative tension, and then they published their article I think in April of 1974, and there was no uptake by the press, there was no scientific confirmation. It was a quiet time until that fall, the Natural Resources Defense Council, NRDC, saw this, the scientists there, and recognize this was a big public policy issue. So at the American Chemical Society fall meeting, they had a presentation by Molina and Rowland, and then by the best of good luck of ruthless corporate behavior, the industry attacked them and made this scientific study news worthy.

So this was a tremendous good fortune, oddly enough, because then all of the press was asking “What are you talking about? Why is it important and what could happen?” And that cultivated in Molina and Rowland, the ambition and… Sherry Rowland called for a ban on aerosol cosmetic products; hairspray and deodorants. And so this was stepping out of their role of a normal scientist and becoming an activist, and of course there was a boycott that was quite spectacularly successful in the United States, and then some product bans.

Susan Solomon: Yeah and actually I just want to say, I think we should be proud in the US that there was a consumer boycott. People turned away from spray cans and that actually, interestingly enough, did not happen in Europe, they kept using them. So we can look back on that time as one in which people were environmentally very aware in this country, not just on the issue of the ozone layer, but also for things like smog and clean water, all those issues had attracted a lot of attention right around this time. I will also say that it’s interesting that at the time of the Molina & Rowland work, they were talking about the fact that from the best of our understanding of the day, in a hundred years we might see a few percent decrease in the total amount of ozone.

So kind of a small effect, far in the future, kind of like the way some people used to talk about climate change until maybe this year or the last few. And the Antarctic ozone hole was a huge wake up call because what they found was that ozone had dropped by more than 30% over Antarctica already by 1985, something that no one had anticipated. So it was a huge shock to the science community. And at first a fair number of people didn’t really take it seriously. I can remember being in scientific meetings with people who said “Oh, that British group, they must just be wrong”. I won’t say who they were, but it of course turned out that they weren’t wrong. They were confirmed quickly by other stations in the Antarctic and also by satellite data. And we now understand the chemistry that actually made the chlorofluorocarbons even more damaging than we thought they would be much, much better than we did before.

Lucas Perry: Could you explain a little bit of the scientific mystery and the scientific inquiry that Molina and Rowland were engaged in, the kinds of hypotheses that they had and the steps from going from okay, there are lots of CFCs in the lower atmosphere to eventually understanding the chemical pathways in their role in ozone depletion.

Susan Solomon: The big issue that they had to deal with was how do these compounds get destroyed, and what is their atmospheric lifetime? And they actually went into the laboratory themselves to try to make measurements relating to that. So they were able to show, I think through the measurements that they made, that the CFCs didn’t react with… They didn’t rain out, that they weren’t water-soluble, so that was not an issue. That they didn’t react with sand, there was some idea that somebody had suggested that they would be destroyed on the sands of the Sahara and that turned out, of course, not to be true. And then they looked at the way in which they would break down, what would happen to them them? If they have no way to break down in the lower atmosphere, the only place for them to go is up, up, up, up, and as you go up, you reach much more energetic sunlight, the higher you go.

Obviously if you’re in the limits of space, you’re getting the direct light from the full spectrum of what the sun can put out. But if you’re down on the ground, you got a lot of atmospheric attenuation. So they began to realize that once these molecules got into the stratosphere, that they would eventually break down, make chlorine atoms, and it was known already that those chlorine atoms could go on to react with ozone. And then there’s a another process, which I’m not going to go into, that actually leads it to a catalytic cycle that destroys ozone pretty effectively, and that process was already known from other work.

Lucas Perry: Could you actually say a word or two about that? Because I actually think it’s interesting how a single chlorine atom can destroy so much ozone.

Susan Solomon: Sure. The chlorine atom reacts with ozone, that makes chlorine monoxide plus O2. Now, if that was all there was, it would be a one-way process, you could never deplete more ozone than the chlorine that you put in. But what happens is that the chlorine monoxide can react with atomic oxygen, for example, so there’s… If you go up into the well… In the lower atmosphere, most of the oxygen as we know is in the form of O2, right? So it’s the oxygen that we breathe is O2. That’s actually true as far as total oxygen, pretty much all the way up, but as you get up into the stratosphere, oxygen actually also encounters that high energy ultraviolet light, which breaks it down and makes atomic oxygen, and ozone can also be broken down by high energy light, and that makes it atomic oxygen, ozone is O3.

So basically what first happens is the O2 breaks down with ultraviolet light making atomic oxygen, the atomic oxygen reacts with another O2 to make ozone, but then the ozone, let’s say photolyzes to make O, so now if the O comes along and reacts with the ClO, making chlorine atoms again, plus O2, you’ve liberated the chlorine atom, it can go right back around and do it over, and over, and over, and over.

And the reason that’s actually happening in the stratosphere, it’s in the sunlit atmosphere, ozone and atomic oxygen are exchanging with each other really quickly, so there’s always some of both present anytime the sun is lit. At night the O goes away, but during the day, there’s ozone… Breaks up as sunlight enough to make some O. So you can just drive this catalytic cycle over and over again and you can destroy hundreds of thousands of ozone molecules with one chlorine molecule, chlorine atom, from a CFC molecule, in the timescale that this stuff is in the stratosphere.

Lucas Perry: Right, and so I think the little bit of information about that is that the chloro in the chlorofluorocarbon meaning… That means chlorine, right? So there’s these chlorine atoms that are getting chopped off of them, and then once they’re free in the atmosphere they can be used to basically slice many ozone molecules, and the ozone molecules are heavier and more dense and so reflect more UV light?

Susan Solomon: No, no, no, no, no. Density and heaviness has nothing to do with it. The ozone molecule is just capable of absorbing certain wavelengths of ultraviolet light that no other molecule can in our atmosphere to any appreciable extent. That’s why it’s so important to life on the planet surface. It’s just a really good light absorber at certain wavelengths.

Lucas Perry: Okay. And so at this point for Mario Molina and sorry, it’s Sherry Rowland?

Susan Solomon: Yes.

Lucas Perry: And so for both of them, this is still theoretical or model based work, right? There hasn’t been any empirical verification of this.

Susan Solomon: That’s right.

Lucas Perry: So could you explain then how we move from these lab based models to actual empirical evidence of these reactions happening, and I guess starting with where Robert Watson fits in?

Susan Solomon: Bob Watson was a chemical kineticist originally, and actually had measured some of the reactions that we’ve just been talking about in the laboratory. So what people do is they go in the lab, I should have said this earlier, they go in the lab and they evaluate how fast individual chemical processes can happen, it’s really very elegant work, and flow tubes, and lasers, and all that kind of stuff. And it’s something that Watson was known for but he got an opportunity to become a leader of a research program at NASA, which he took up, and he became very much a leader in the community, as far as both organizing missions, field missions to go out there and look at things in real time, and more importantly perhaps, a huge leader in the assessment process which brought the scientists and the policy makers together. I think that you can really look at Bob as a tremendous founder of the whole way that we do scientific assessment, together with a gentleman named Bert Bolin, who has passed away unfortunately.

Lucas Perry: After Watson, could you then bring in how Joe Farman fits in?

Susan Solomon: Yeah, Joe Farman led the British group that discovered the ozone hole, as I mentioned earlier. So they noticed that their ozone over their station at Halley, Antarctica, it just seemed to be decreasing at an alarming rate. And they checked it with another station that they have, which is at a slightly higher latitude, not quite as far to the pole… I should have said lower latitude. Anyway, 65 South is where the other station is, I think Halley is about 73 South, it might be 78. And they found that there was ozone being lost at the other station too, just not as much. And that’s when they decided that they just had to publish, so they did, and it attracted my attention along with the attention of a lot of people. I started working on what chemistry could possibly cause this, and what I came up with was that “Hey, maybe…” And we knew that there was no ozone hole over the Arctic.

We knew it was only over the Antarctic, because we had measurements from places like Sweden and Norway and Canada, if anything was happening, it was nothing like the Antarctic. So measurements in other places showed ups and downs, variability from year to year, but they weren’t showing any kind of trend at that point. They later did, and we can talk about that, but we’re talking about 1985 here, so really early. I was a young scientist, I was 29 at the time, and I decided that I was going to try to take my photochemical model and beat on it and pound on it and make it stand on its head until it produced an ozone hole.

And so I did that, and I figured out that the reason that it was happening was because Antarctica really is the coldest place on earth, and it’s so cold that clouds form in the Antarctic stratosphere. The stratosphere is very dry, so normally there just aren’t any clouds, but down in the Antarctic because it’s so cold, the vapors, mainly water vapor but also actually nitric acid and other things can condense and form these incredible polar stratospheric clouds. And the clouds completely changed the chemistry, we can talk about that, but I think I’ve maybe gone on too long for my enthusiasm for which I apologize.

Lucas Perry: Hey this is post podcast Lucas. I’d like to add some more details here around the story of Joseph Farman’s discovery of the ozone hole, to paint a bit of a better picture here. I’m taking this from the UC Berkeley website, and you can find a link to the article in the description: Dr. Farman started collecting atmospheric data at Halley Bay, Antarctica in 1957, sending a team to measure ozone levels and concentrations of gases, like CFCs. In 1982 his ozone reading showed a dramatic dip of around 40%. He was initially skeptical that this was an accurate reading and thought it must have been an instrument malfunction due to the severe arctic cold. He also reasoned that NASA had been collecting atmospheric data from all over the world, and hadn’t reported any anomalies. His instrument was ground-based and only had a single data point, which was the atmosphere directly above it. Surely, he reasoned, NASA’s thousands of data points would have revealed such a drop in ozone if there had been one. Given this reasoning, he ordered a new instrument for next year’s measurements.

The following year, Dr. Farman still found a drastic decline, and going through his old data, discovered the decline actually started back in 1977. He now suspected that something odd was happening strictly over Halley Bay, leaving other areas unaffected. So the next year, his team took measurements from a different location 1,000 miles northwest of Halley Bay and also discovered a large decline in ozone there as well. With the same data at two different locations the mounting evidence for the ozone hole was clear and he decided to publish his data. This data both shocked and intrigued many scientists, including Susan Solomon, which thus catalyzed further research and inquiry into the ozone hole, the mechanism that was creating it, and the needed governance and industrial solutions to work towards mending it. Alright back to the episode.

Stephen Andersen: Let me just say one thing before we go back to the ozone hole. One of the interesting things that happened was of course, Farman and his research group declared that there was this serious depletion happening in Antarctica. So all the scientists that had been building the case with Sherry Roland and with Mario Molina, instantly jumped on it in the press, and in fact it was Sherry Roland that coined the phrase, ozone hole. He was the first person to utter that phrase.

And that was also a very good case that the public could grasp that, they could look at the NASA graphics, they could talk to scientists, and so there was really a great expectation that someday there would be the smoking gun like this and Antarctic ozone hole or other evidence, and people were ready and prepared to go to the press and go to the public, and in fact the politicians by this time had been briefed a lot, and that the United Nations they’d been working on this since 1970, when they organized a working group on stratospheric ozone depletion. So this great scientists and great science was welcomed into the community and they took full advantage of this and then other great scientists like Susan jumped on it to say, “Well, how can we go beyond simple finding of the ozone depletion and track it back to its origin, the CFCs and the other ozone-depleting substances.” So it was science and politics at its best.

Susan Solomon: Yeah, I guess I also want to say that I didn’t assume that the ozone hole was necessarily due to chlorofluorocarbons. I tried to produce it all kinds of ways with reactive nitrogen from the Aurora with dynamical changes. I just couldn’t get it to happen any other way. And what we knew already, and again I think it’s a real achievement, was that people had been interested in the idea of reactions on surfaces for a while, but mainly because they thought they were perhaps interfering with the measurements they were trying to make in those flow tubes. The flow tube is basically just a glass tube and people assumed that there was no surface chemistry that could happen in the stratosphere. We know there is chemistry on surfaces, in the lower atmosphere, in the troposphere, and it can be really important. Acid rain is a great example.

Surfaces can make chemistry do things that just doesn’t happen in the gas phase. That’s why you have a catalytic converter in your car, it’s a surface that converts the pollutants into something else before it gets out the tailpipe. Surfaces lead to chemistry happening very differently from gas phase. And we assumed the stratosphere was just gas phase, there couldn’t possibly be any surfaces. But interestingly, we sort of knew that there were these polar stratospheric clouds they’d been observed by explorers going back, I think 200 years in the Arctic and 120 or so in the Antarctic. We knew they were there, we just didn’t really carefully evaluate their chemistry. But when people started doing these experiments in the laboratory, they thought certain processes were actually going in the gas phase. They saw, for example, certain kinds of chlorine molecules going away in their flow tubes, and they thought they’d discovered some new gas phase chemistry, turned out to be something happening on the surface.

And they said, “Oh, okay, doesn’t matter. It’s just on the surface.” Well, it turned out to be not just on the surface of the float tube, but also on the surface of those polar stratospheric clouds. And that’s actually the connection that I made. I thought, “Hey, if this is happening in the lab, there’s no, necessarily, reason that it couldn’t also happen on polar stratospheric clouds.” Now that was a leap that perhaps I shouldn’t have taken, but anyway, I did.

Lucas Perry: It’s good that you did. Yeah, could you explain what that moment was like, more so. I mean, that was basically a key, super important scientific discovery.

Susan Solomon: Yeah. I had a very hard time believing it when I… This was back in the days when I was running a computer model. This was in the days that you would wait a long time for your output because things were very, very slow. I don’t remember. I don’t think it was still in the computer punch card day. I think I actually did have a file that I submitted, but the wait for getting it back, I think, felt interminable. And when I did get the results back, I was just shocked to see how ozone behaved. And one of the key things about it is that it doesn’t happen in the winter. In the dead of winter when the polar regions are dark, this process won’t be very important. You have to have not only cold temperatures so the polar stratospheric clouds are there, you also need sunlight to drive certain parts of the chemistry.

And I could go into the details of that, but I’m not sure you need me to do that. It’s a process then that occurs in the Antarctic spring, as it comes out of its long period of dark cold winter, it’s still cold, but the sun starts coming back. And that’s the combination that then drives the ozone depletion. And that began to start happening in my model. So I was pretty shocked. It wasn’t quite for the right reason, I have to admit. The process that I had driving that final step of… What I did identify correctly was that the key reaction is the hydrochloric acid and chlorine nitrate from the chlorofluorocarbons react together on the surface of the polar stratospheric clouds, they do not react in the gas phase.

We thought maybe they did at one time, but then we figured out it was just on the surface of the float tube, so we forgot about it, everybody, except until I remembered. And then the hydrochloric acid and chlorine nitrate react on the surface of those clouds that makes molecular chlorine CO2, which fertilizes breaks apart very readily with sunlight, that makes chlorine atoms, and now you’re off and running to produce ozone depletion. So, that part I had all correct. What I thought was that the chlorine monoxide might react with HO2 to close the catalytic cycle. Cause you don’t have much atomic oxygen in the lower stratosphere where the ozone hole was happening. You need to close that catalytic cycle, we were talking about earlier, with something else. Turned out that really the key thing is ClO reacting with itself to make something called a ClO dimer, which then fertilizes. But we didn’t actually know that chemistry yet. We learned about it not too long after. That was discovered in ’87.

Lucas Perry: I see. So, essentially there are these glass tubes and labs where the scientists at the time were trying to basically create atmospheric conditions in order to experimentally test the models that they were generating for what happens to ozone up in the atmosphere. And because it’s a glass tube, there’s a lot of surface on it and so they were discounting what they were observing in that glass tube, because they’re saying the upper atmosphere doesn’t have any surfaces. So, any surface related reactions don’t really make any sense.

Susan Solomon: Right. That’s basically it.

Lucas Perry: So, you were looking at that, what made you think that maybe there were surfaces in the sky?

Susan Solomon: Well I mean, we knew that polar stratospheric clouds could happen in the Antarctic and also in the Arctic. Like I said, people have visually… You can see them. I’ve seen them myself. They’re clouds. They’re actually very beautiful. There they look like they’re almost a rainbowy kind of appearance because the particles are almost all one size and that creates a particular kind of beauty when the sun hits it. But yeah, you can see them, they’d been seen, literally. There were also satellite data that had been published a couple of years earlier that helped to inspire me to think about it. But I actually knew about the explorers, I was just intrigued by that kind of stuff. So then I was very excited to work with Bob Watson when we formulated a mission to go to the Antarctic and to actually go down there and make measurements that might help to determine whether reactions like that were indeed happening. And that happened in 1986.

Lucas Perry: Right? So you’re creating these models that include the surface reaction. And so you’ve got this 1980s computer that you’re submitting this file to, and… What do you get back from that model? And how does that motivate your expedition to go there and get measurements?

Susan Solomon: If I remember I had it programmed to make plots of the percent change in ozone, and there was this, I didn’t call it a hole at the time, but there was this area over the Antarctic where once I put those reactions in, I got a lot less ozone. I recall something like 30% less. It’s published in my paper that I published on this in 1986. So I wrote it up and submitted it to nature, and it was published in ’86, and that was the same year that a lot of us began thinking about how to get down there and test the different ideas that have been put out because the idea of chemistry involving chlorofluorocarbons was not the only idea out there, other people had meteorological theories. And as I mentioned, there was this possibility that it might be solar activity, I guess somebody thought about, so the…

Scientists are always stimulated to come up with ideas, and we needed to get down there and make the measurements that could discriminate between the different ideas. So, I was very fortunate to be young and able to get on an airplane and go to the Antarctic. So I did, 1986. It was great. Most incredible scientific experience in my life, actually.

Lucas Perry: What made it so incredible, and what is it that you saw? What was your favorite parts about your expedition to the Antarctic?

Susan Solomon: Well, just going to the Antarctic is an unbelievable experience. I mean, even if you just go on a cruise ship, it’s like another planet. It is crystalline, beautiful, unpolluted, full of optical effects that are just amazing. And of course, brutally, brutally cold. We went down in August of 1986. When I got off the plane the temperature was about -40°C, which is also -40°F. So I like to joke that if you’re ever on Jeopardy and the Final Jeopardy question is, “At what temperature are Celsius and Fahrenheit the same?” The answer is -40. I’m originally from Chicago, I’ve been in cold weather, but I’ve never been in anything colder than I think about -15 before. And it was, it’s a shocker.

But after a while, after a couple of weeks, -15 actually feels very warm. You really do. It’s amazing how you acclimatize. Everybody laughs. Stephen’s laughing as I’m saying this, but it’s true, it’s true really. I thought I would just kind of curl up in my room the whole time, but I didn’t, I found that it was easy to acclimatize. Yeah, really. Well, actually people do. People actually even go jogging with shorts on, at -15. Yeah, it’s incredible. Depends on if it’s sunny or not. The atmosphere is very dry also down in the Antarctic. Basically, the cold has rung all the paper out of the air. So-

Lucas Perry: Did you go jogging in your shorts and T-shirt at -15?

Susan Solomon: No, no, I didn’t do that. But I’d certainly remember feeling warm at -15 and opening up my jacket and stuff like that. And I definitely kept my window open if it was -15. So yeah, I did. But, I made some measurements with my colleagues using visible spectroscopy. So we use the sun, or the moon, or the sky as a light source, and we measured chlorine dioxide, which is a closely related molecule to chlorine monoxide, and we were able to show, particularly with the moonlight measurements, that the values we found were a hundred times more than they should have been. We couldn’t measure them anywhere else because they were below our detection limit, but they were actually quite measurable in the Antarctic. So, that was the key measurement that we made and it was an incredible night, the night that we actually did that. And then, I think it was the next day that I made the data analysis and there it was. It was an amazing, amazing moment.

Lucas Perry: Could you explain more about how that particular data that you measured fit into the analysis and what it proved, and the feelings and experience of what it was like to finally have an answer?

Susan Solomon: First of all, there’s the getting of the data which involves putting mirrors up on the roof of a little building in the Antarctic and directing the moonlight right down into the instrument. And doing that when it’s cold and windy can be a bit of a challenge. So setting it up for measurement is physically challenging. And then taking the data, analyzing the data. I was, I think, careful enough to realize that that wasn’t going to be the only thing it would take to convince everyone that chlorine was the cause of the ozone hole. The chlorine dioxide that we measured, had to have come from the chlorofluorocarbons. There was no other even conceivable source for it. And it was a hundred times more of it than there should have been, and that’s because it had gotten the reactive species, and chlorine dioxide is a reactive form of chlorine had gotten liberated from un-reactive forms of chlorine, like hydrochloric acid and chlorine nitrate, which reacted on the surfaces of those clouds. And they don’t do that anywhere else.

So that’s a little more detailed than I thought you might’ve wanted, but that’s why you take these, what we call reservoir species for chlorine, hydrochloric acid, and chlorine nitrate, and you convert them to active chlorine. And now you’re really often running for ozone depletion. And that’s what happens on those clouds.

Lucas Perry: You’re getting this data about these particular chemical molecules, and then you’re… Tell me a little bit more about the actual analysis and what it’s like being in the Antarctic feeling like you’ve discovered why there is a potentially world ending hole.

Susan Solomon: Well, I’ll tell you this, I was really careful, I think, maybe Stephen can correct me if he thinks some wrong, but I was pretty careful about not broadcasting the news before we were really, really sure. So the moon measurements alone were not enough to convince me. And one of the things that actually excited me a lot was when we realized we could also see this in the scattered sunlight that we got. If the sun was low enough on the horizon, there was even enough chlorine dioxide to measure it then. So what it is is a visible spectrograph, it’s got a diode array in it, it’s actually very similar to the diode array that reads the prices when you go to the supermarket today. Back in the ’80s those were incredibly expensive because they had just been invented. And we had one that was cooled to very cold temperatures to keep it from having too much noise in it.

And we had a spectrograph, which you can think of as being sort of like a prism that you shine sunlight through, and you separate out the wavelengths of light and the colors of light come out as a little rainbow that you see when you put a crystal in front of a source of light. And so that’s essentially what we’re doing. We’re putting a grading, in this case, a diffraction grading in the beam of the moonlight, and we’re collecting the separated wavelengths of light on our detector, and we’re looking for the absorption of atmospheric chemicals. And we can measure ozone that way, we can measure nitrogen dioxide at a different wavelength, but chlorine dioxide has a particular band structure in the visible, that is what we measured. And we can also see it. The fact that we could also see it in the skylight and that the difference between the skylight and the moonlight was consistent with what we expected from chemistry and consistent with what you would need to deplete the ozone layer, got me pretty excited.

There was another group on our expedition that measured chlorine monoxide using a different method on microwave emission technique, which is the same one that’s used nowadays by satellite but in those days it was only used from the ground, and they also measured high levels of chlorine monoxide. And last but not least, I’ll say that the following year in 1987, Watson organized another mission, which actually flew on airplanes from Chile down to Antarctica, and measured chlorine monoxide yet another way by laser resonance fluorescence onboard an airplane that actually literally flew right into the ozone hole. So, I would say fair to say that from the science community point of view, when all those measurements were in, people got pretty convinced, but they also had to be written up. I mean, it had to be peer-reviewed before something that important could really be talked about as a known piece of science. So I was very cautious about spreading the word too early.

Stephen Andersen: So if you look at the history of the Montreal Protocol, what you see is that in 1985, there was something called the Vienna Convention that was passed by the United Nations that had about two dozen members signed it. And this is what’s called a Framework Convention that makes it possible to have something like the Montreal Protocol. So that was in the spring of 1985. And shortly thereafter, Berman published his article which was a tremendous reinforcement to the policy members that had anticipated that soon there would be evidence and that they needed to be prepared. And then as we went into 1986, and Susan is doing with her college, this brilliant work in Antarctica, the preparations are underway for the Montreal Protocol. And the scientists were telling the Montreal Protocol that it could be another explanation for the ozone hole, so that you should hold your fire until you’re sure of the results.

And you can find that in lots of the accounts at the time. But by the time of the medium, September of 1987, there was still a lot of uncertainty. But the policymakers were able to talk to their national scientists and others and felt confident enough to go ahead and confirm the Montreal Protocol. So I would view it from my point of view and perspective, that it was a continuous improvement in the science and the threshold of belief occurred for different people at different times. But you could then say, of course there were still skeptics in 1987, but it was my experience that they were mostly gone by 1990. And so the work I was doing mostly with corporations, there was rarely a meeting where there would be science skeptics after 1990, that they were gone from the earth has moved on to climate, in fact.

So it was a tremendous contribution of science. And the other thing that’s important to realize is that the industry that used these chemicals was not devoted to them. They’d been reassured by DuPont and others that they were completely safe and that there was no reason to worry. And then, as soon as the Antarctic ozone hole came along, they panicked and they rushed to the market to find alternatives. And that’s one of the reason the Montreal Protocol happened so fast, is the industry in some ways was faster to grasp the science than even the policymakers.

Lucas Perry: Stephen, I’d love to bring you in here to describe, in particular, the transition from the discovery Susan was involved in to the Montreal Protocol. So, what are the steps between the discovery and the Montreal Protocol, and how do we eventually get to the Technology and Economic Assessment Panel that you co-founded?

Stephen Andersen: I’m glad to describe that. It’s exactly what Susan said, is that there’s a laborious process to prove the science, and then there’s another process to communicate it. And that’s probably partly what Susan did, and Bob Watson and another scientist, Dan Albritton. And they were masters of communication. And there were lots of meetings held between the scientists and the diplomats. But also the scientists and the companies, including the National Academy of Engineering, did its own review of the science on behalf of industry and came up with a confirmation report, I would call it no. New science, a narrow view of science, but nonetheless, it was the message coming from the people they most respected, and Sherry Rowland was involved in that and many, many others. So the communication was very quick. And in fact, I would say by January of 1988, you could see big changes.

So the protocols signed in September ’87. In January of ’88, there was a large conference, and at that conference, there were several important announcements. The most spectacular was that AT&T announced that they had found a nature based solvent made from the terpenes from oranges and lemons, pine trees that could clean half of the electronics equally or better than the CFC-113 had replaced. And they said that transition was technically possible within one year. So they went from skepticism and standing back to becoming the driving force. And it was also important because this terpene was not another synthetic chemical. It was naturally derived and harvested from the disposal of the orange rinds and the lemon rinds, and then put to positive purpose. So this was an eye-opener to a lot of people that thought you had to have an elaborate chemical to have an elegant solution.

And then at that same meeting, the auto industry step forward and realize that most of the emissions from car air conditioning was from servicing and from leakage. And so for the first time they got together a partnership that developed commercial recycling for air conditioning. They did that within one year and the next year after they confirmed and approved the technology, they sold a billion dollars worth of recycling equipment all across the world. So there’s this enthusiasm of going from panic, that there would be high costs and disruption to the enthusiasm of profits and saving money. So it was the science that drove this, but it was the technical innovation that did this. And then very shortly thereafter, and in an overlapping way, there were similar breakthroughs in foam, so it was a commitment by the American food packaging institute to halt use of CFC in foam within one year, and to switch away from all fluorinated gases as quickly as possible.

So we’ve seen this building momentum and enthusiasm. You have international companies that are pledging to get out, and all the while we haven’t reached the Montreal Protocol entering into force, because that occurs later after the signing. And when it was signed and the first big assessment which Susan was involved in as well, was done in 1989. And so this was an assessment that you alluded to, it included the Scientific Assessment Panel, it included the Environmental Effects Assessment Panel, and then it included the work on technology and economics. And this was the idea of how do you make the best available information readily absorbed by policy makers and business community.

Susan Solomon: But Steve, I would just add one thing that you already said, and that is, it all starts with people understanding the problem. You talked about the fact that people everywhere the public all around the world could look at these satellite images of the ozone hole and say, “Hey, that’s actually pretty scary stuff.” And that created the will, the political will, that generated the demand for all the products that you’ve just described. I think without people understanding the whole thing, nothing happens, personally.

Stephen Andersen: You’re absolutely right. And in fact, in the case of that food packaging, it was a school teacher in Massachusetts and her children in the class that wrote to McDonald’s corporation and said, “Why are you destroying the ozone layer?” So the people at McDonald’s commissioned a survey of their customers, including children, and the customers responded, they did not want to destroy the ozone layer. And it made a big difference to where they chose to eat. And in the case of McDonald’s, children drive parents to the restaurant, the parents say, “Where do you want to go today?” And they say, “McDonald’s.” So it was a huge impact. It was an eyes-open business decision. And they had announced, prior to the packaging institute changing, that they were going to stop the purchase.

Susan Solomon: Yeah. They were putting hamburgers in foam clam shells, and they switched over to cardboard, which is fine because McDonald’s is so delicious. You eat it so fast anyway, you don’t need the foam. McDonald’s is so delicious. You eat it so fast anyway, you don’t need the foam.

Stephen Andersen: That’s right. Hot side hot, cold side cold, was the slogan. But this is exactly right. What Susan’s saying is, you have this circular effect, where you have the customers pushing the companies, you have the companies pushing their suppliers, and you have the policy makers setting deadlines. And pretty soon, you’ve got this wheel turning very fast. And as quickly as you catch up with the available technology, then you look to the next strengthening of the Montreal Protocol. And that’s what we saw over the decades of the Montreal Protocol, more and more chemicals control, faster and faster phase out.

Lucas Perry: Steven, could you explain more specifically what the Montreal Protocol was, and who the key stakeholders were, and how it came together and then was signed and ratified, and what that meant?

Stephen Andersen: So my role, I was very fortunate, because I was hired by the EPA in 1986 in preparation for the negotiations of the Montreal Protocol. I’m an economist by training. And so, I had the highest interest in showing that this was going to be cost-effective and feasible, and that the technology would come together. The mastermind behind the Montreal Protocol was the head of the United Nations Environment Program, Dr. Mostafa Tolba, who was a botanist himself and an accomplished author. And I think was very quick at grasping the science.

So you have the force of the United Nations organizing the meetings. And then you have the science that’s providing the justification for the treaty. And then you have leadership countries that were advocates of a treaty. And that included a group that was called the Toronto Group, because it was partly stationed in Toronto, but that was United States, Canada, Norway, Denmark, many other countries, Sweden, that got together as a group and helped craft the language that they could sell to other countries.

And so, it was a masterfully designed document in retrospect. And included in that document was the idea of start and strengthen. So if you look at it, it only was two chemicals, CFC, and then a fire extinguishing agent called halon. And then the first negotiation in 1987, it was just to freeze the production of halon, stop it from growing, and cut back CFC’s 50%. but that was not hard to do because 30% was still aerosol and convenience cosmetic products. So it was a very conservative start. But the science was so persuasive in the years ahead, that they said to the policy makers at the Montreal Protocol, that’s not enough. You will not protect the ozone layer with those two chemicals. And you certainly won’t with those modest reductions. So then they added more CFCs. They added carbon tetrachloride, metal chloroform, methyl bromide. A litany of chemicals were added. And then each time that they would meet, every two or three years, they would have an acceleration of the phase out.

So it was a very practical approach that was done on an international basis. And one of the beauties of this treaty, is it includes incredibly strong trade restrictions so that if a country did not join the Montreal Protocol, they would lose access to these ozone-depleting substances even before the phase out. So it had lots of clever features and lots of brilliant leadership. And what Susan said about people mattering, they mattered a lot over and over again. And there were 200 or 300 people that had a chance to become ozone champions and make a real difference to the world.

Lucas Perry: Could you explain who were the main stakeholders involved in the Montreal Protocol? Was this mostly developed nations leading it?

Stephen Andersen: That is a great question. That’s a fantastic question, and explains a lot of why it was such a challenge. So if you look at the full set of chemicals that are controlled by the Montreal Protocol, they were divided into 240 separate sectors. So, distinguishable industry groups that had their own interests in keeping these chemicals or to phase them out. So if you look at those, and some other ones that Susan mentioned, the air conditioning and refrigeration, and that includes industrial refrigeration, because many chemical processes require that, and commercial refrigeration, and also what’s called cold chain, the processing and the freezing and refrigerating of food in order for it to reach market.

So that alone would have been daunting. But in addition to that, there were these chemicals used as solvents in aerospace, in electronics, in the manufacture of medical devices. It was used as a sterilant. And as I mentioned, as an aerosol for metered dose inhalers. It was used in fire-fighting, including enclosed areas like on airplanes and submarines and ferries and ships and places that you can’t evacuate if there’s a fire, where you have to stay on board the burning vehicle. So it included all of the NASA satellites and the space labs and the rocket equipment. Manufacture of solid rocket motor for the space shuttle required methyl chloroform.

So you have these, and then the were laboratory uses. So it’s used as a tracer-gas, as Susan mentioned. But also, it was used to have a dense gas for a wind turbine. And it was used for pressure check testing of scientific instruments to make sure there were no leakages of gases in or out. So as it got going, also, it was discovered that every weapons system in every military organization depended on ozone-depleting substances. All the command centers were protected with halon. All the ships, tanks, submarines, protected by halon. All the electronics and aerospace manufactured in service with CFC and methyl chloroform, all the gyroscope manufacturing for the weapons guidance. Whole list, all the way down. All the optical raiders, all the AWACS. Everything that they could look at had some use.

And so it required these stakeholders to look fundamentally at the basis of what they were doing, and decide how do you shift from using this chemical to a performance standard that would allow industry to compete as how they could produce an alternative that would be a pure replacement. And so, one measure that I think your listeners will find interesting is that if you ask the public today, or even the effective industries, no one has stories of train wrecks or disappointment or failed systems, because it was so successful. Most consumers would have their entire house changed, and they would not notice this. The glues they used to assemble furniture were ozone-depleting substances, but people have not stopped buying furniture. And so, if you went back, it’s the smallest list of uses that found no substitutes. So it’s quite remarkable.

Lucas Perry: Steven, you were on this panel, I believe the chair of it, for many years. So I’m curious if you could explain, more so, what that experience was like, what is it that you necessarily did on the technology and economic assessment panel, and what the impact of that was for implementing the solution that was needed after Susan helped to discover the mechanism of the problem?

Stephen Andersen: Yeah. Thank you for asking me that, because that’s what I’m most proud of, of course. When I was appointed with Vic Buxton, from Canada, to set up the first technical panel, we were like-minded and we had a great idea, and that was, instead of casting out for experts from various sources and seeking wide participation and balance of interests, we didn’t do that at all. We recruited the experts from the organizations that were already committed to protect the ozone layer, because these would be people motivated to find a solution rather than intellectually interested in describing solutions, or even worse, be a stakeholder against a new alternative, and they would become internal critics.

So the notion was that, on a technical committee, you could not have a better set of people than the people whose success in their enterprise depends on finding alternatives, and realized that a team could find the alternatives faster than others. The other secret of our success is, we had something called self affecting technical solutions. And so, for example, one of the chairs of the Halon Committee, studying halon, was the chair of the National Fire Protection Association that set the standards for where halon is used. So as quickly as a use could be eliminated with an alternative, he would go back to his committee and decertify halon on that use.

We had members of the coast guard on the committee. And as quickly as there were alternatives on ships, they removed from the requirements of the United Nations Maritime Organization the use of halon. So it went from compelling the use for safety to prohibiting the use for environment. So it was this remarkable internal group. And if you go back also, and you’ll notice that some of the most important technologies were invented by people that only met on the committee for the first time. So you had groups of military suppliers that got together to tell the communication suppliers and invented something called no-clean soldering that eliminated the use of solvents and save the ozone layer, but it also increased the reliability of the products. And they were enthusiastic about commercialization to the extent that they patented the technology and then donated it to the public domain so it could be used anywhere in the world at no expense to the user.

So you have this enthusiastic group of genius engineers working on a short deadline and constantly resupplied with motivation from scientists like Susan, because as fast as they would take satisfaction in what they’d accomplish, they were being told, it’s not enough. It’s not enough to just do these chemicals. We have to do more. It’s not enough to do these chemicals on the old schedule, we have to go faster. So some of these sectors halted their uses years ahead of the deadlines of the Montreal Protocol or the Clean Air Act. It was really quite inspirational. And most of those people would tell you, it was the best part of their life because they never would have been allowed to work with the engineers from the competing corporations if it hadn’t been for the TEAP drawing those together for public purpose.

Lucas Perry: So, is a good way to characterize this then that there’s this huge set of compounds, that when they get up in the upper atmosphere they release chlorine? And the chlorine is really the big issue. And so, these hydro chlorofluorocarbon are being used in so many applications. You’ve described a lot of them. And so, the job of this committee is to, one, slowly, through regulation, phase out this long list of ozone depleting chemicals, while also generating alternatives to their use that are not ozone depleting.

Stephen Andersen: Yeah. That’s right. Generating or identifying. And there’s a subtle problem we faced. It’s now being faced again for climate. And that is, as Susan mentioned, most of these chemicals have long atmospheric lifetime. So when you stop producing the CFC, it can be a slow decline in the chlorine that’s been contributed to the stratosphere over many, many years. Others of the chemicals like methyl chloroform, and most of the HCFCs have short lives. And so, any reductions you make in these chemicals that do all their damage within their short number of years has a bigger effect immediately than doing the same amount of effort on one of these other chemicals. And what the scientists were telling the Technical Actions Committee and the Montreal Protocol was that we had to worry about the long run and the short run.

And so HCFCs as refrigerants and foam blowing agents were viewed as a transitional substance. So if you stopped using CFC11 and you started using HCFC22, that was an improvement in both the GWP, and the forcing of ozone depletion. And the same thing for methyl chloroform. So the ambition of the Montreal Protocol was to work incredibly quickly to get rid of the short term chemicals and uses with an alternative that would be solvents, for example, using methyl chloroform, but at the same time, allow some HCFs so that you didn’t have to endure the continued use of the CFCs. And that was the technical challenge, to keep your eye on the long run, and at the same time, keep your eye on the short run. And some of the scientists were also over motivating that kind of ambition, because there was a concern that we might go too far in sending chlorine and bromine to the stratosphere, and do irreparable damage, or damage that would take much longer to solve.

So, true or not. It was highly motivational. And it caused a tremendous effort on our committees, first of all, to get rid of methyl chloroform. If you look at the curve of methyl chloroform and the overall ozone protection, it was a critical first step. And it was accomplished probably in two and a half years worldwide.

Susan Solomon: So let me toot Stephen’s horn a little bit. And then also clarify one point. I think the invention of the Technology and Economic Assessment Panel, TEAP, as we call it, of the Montreal Protocol was a real master stroke because it brought the engineers and scientists from industry into the process to help figure out what could be done. And so, the way the assessment process worked is, on a systematic basis. The science group that I was part of would assess the science. Steve’s group would assess, okay, the science says, we got to phase these things out. What can we phase out? What is technically feasible?

And we would provide these reports, along with the one from the impacts panel, that would say if you keep doing this, you’re going to have so many skin cancer cases a year by 2050 and stuff like that. All three of those reports would be explained to a group of policymakers in a UN meeting. So the decisions weren’t actually made by Steve. But Steve’s group was highly, highly influential in educating the policymakers and guiding them, really, on what would make the most sense, what could be done the most cost effectively, the most quickly, et cetera, et cetera. And then they made the decisions.

But the great thing about it is it’s not a political group at all. In the old days, we would call them a bunch of guys with slide rules. And that included the people who came from industry. They weren’t the political leaders of those companies, they were the people in the trenches trying to actually figure out what to do instead. And that’s what made it work so well. I really have often wished that we had a similar way of doing the intergovernmental panel on climate change assessment process. We have a science panel that’s pretty similar, but we don’t really have quite the same technology panel. And many people have commented that the technology panel that Steve put together was just huge in making the Montreal Protocol work as well as it has. And the ozone layer is actually finally beginning to heal. So it’s a real testimony to their success.

Lucas Perry: Yeah. Steven, please, I invite you to toot your own horn a little bit more here because your contribution was extremely significant towards the elimination of the ozone hole. I have a fun fact here that is from a paper of yours. So in 2007, Steven, you released a paper with Velders’ team, published the importance of the Montreal Protocol in protecting climate. And the team quantified the benefits of the Montreal Protocol and found that it helped prevent 11 billion metric tons of CO2 equivalent emissions per year, and delayed the impacts of climate change by seven to 12 years. So please, what are some of your favorite achievements? And this is something you’re involved in for decades.

Stephen Andersen: Yeah. That’s a great story. And I’m of course, very glad to tell it. One of the things people know about me, that have worked with me for many years is, I worked by slogans a lot. So I try to reduce my ambition to something that’s like a chant or a short instruction that I can give myself to move ahead. And after years of working with Susan and many other scientists and Mario Molina and Sherri Rowland, and struggling with these issues year after year, after year, and waiting for the science to come on board, and there’d be a missing link, and there’d be something that was misinterpreted, and we’d have to go back to square one. I came up with the slogan, science too important to leave to chance. And what that meant was that it was my job to say, what kind of information are the policy members missing? Those are the people I hang out with all the time.

Because at the same time, I was the deputy director for Stratospheric Ozone Protection at the EPA. I was the liaison to the department of defense on climate and ozone layer protection. So I was in those meetings where people were trying to decide, is it worth investing another millions of dollars in this new technology, or should we do something else? So in working with Susan in 1995 on a joint report between the IPCC and the tape, I realized that the Montreal Protocol had done a lot for climate that wasn’t well appreciated over at the Montreal Protocol, that these facts were available. So I put together what we called a dream team, which included Guus Velders, who was the lead author, a brilliant scientist from the Netherlands. It included David Fahey, he was one of the colleagues of Susan at Noah, and John Daniel. And then it included Mack McFarland, who’s a scientist who was once at Noah, but worked the better part of his career at DuPont. And then myself, who had been on the TEAP for oh, so many years, and then EPA.

So the idea of the team was to say, just how big was the contribution of the Montreal Protocol to protecting the climate, and how do we communicate that to the Montreal Protocol so they would consider that as part of their obligation and part of their legacy? Because we were coming up, my concern, we were coming up to a very long interval of HCFC in years, that the Montreal Protocol had plateaued its ambition. And they had accomplished so much, they were resting on their laurels, and they had lost this impulse to get more stringent.

So this committee was put together, it quickly put together all the facts, incredibly complicated at the time, although people have done work like this since confirming it. And this dream team came up with the conclusion that the Montreal Protocol had already accomplished more than the Kyoto Protocol could have accomplished if every party, every state government in the world had joined Kyoto, and if all of them had met its obligations. So this was huge. It was shocking to us. It was shocking to the world. We brought it back the same year, 1997, excuse me, 2007, to the Montreal Protocol. And that year, they accelerated the HCFC phase down.

So it was it a tremendous victory. And it was exactly what I would hope would happen, is if we assembled science in a new way that was headline news, that the policy makers would get the message and do something important. And then two years later, the same team decided, well, why don’t we show the Montreal Protocol, how important it could be if the chemicals that replaced 15% of the ozone depleting chemicals, which are HFCs were phased down under the Montreal Protocol, ozone safe chemicals controlled by the Montreal Protocol. And that was accomplished in 2016. Took a decade. But we’re very proud of ourselves. And I think it’s a perfect example of the advantage of a group of people with a wide set of skills working together, including somebody like me who’s not an atmospheric scientist, working with atmospheric scientists and making more clear what the policy makers need to know.

Lucas Perry: I’d love to pivot in here now into extracting some of these lessons that you’re sharing, Steven, for how we might do better with modern climate issues, from greenhouse gases and also other global catastrophic risks. But I think you guys have done an excellent job of telling the story of the science and the discovery, and then also the strategic part, and the solution making of the story of the ozone hole. So as we get to the end of that, I’m curious if you have any thing else you’d like to wrap up on about that story. What is a key insight? When you look back at everything that you’ve contributed and been through, what is it that stands out to you?

Stephen Andersen: My theory of change, I think, is the same as Susan’s, that people matter the most. That the ability to bring the right people together, at the right place, with the right instructions, is bound to have an important conclusion. And one of the things that I always found was, if you have the best engineers, no matter what their attitude, if they have the goal that is coincident of the environmental protection, they’ll find a solution. So I think this accumulating of the science and the engineers and coordinating the activity, these assessment panels are just everything, that if you do that, you can’t help but be successful.

But the other thing that I’m realizing now is that, when you’re in a hurry, like we are with the climate, you have to take advantage of the existing institutions. So as quickly as we added HFCs to the Montreal Protocol, I would like to add other chemicals, N2O, nitrous oxide, which is an ozone-depleting greenhouse gas that was neglected by the Montreal Protocol. And then there’s other gases like methane that have nothing to do with the sectors that are involved in the Montreal Protocol, but the framework of the Montreal Protocol might be perfect for a methane treaty. So you might have the Montreal Protocol people help design a methane treaty. Or if the Montreal Protocol can find its way to create new capacity with new skill sets, you could have methane drawn into the Montreal Protocol because it’s genius is partly that you turn off the chemical at the source and force all the downstream changes to occur. And that’s different than something like EPA, where you often find one part of the problem, catalytic converters and gasoline, and you implement that, but you’re not focusing on the big picture, which is electrification of the cars.

And so, there’s this inherent advantage of the Montreal Protocol, top down, turn off the chemical, bring on the technical information, bring people together for solutions, reinforce and reward. So I could go on for the longest time. I’m very enthusiastic about the success of the Montreal Protocol. I absolutely believe the lessons could be taken up better than they have been. And I think if they were taken up, we’d be well on our way in many other environmental problems.

Susan Solomon: Yeah. I like to tell students in my classes, and I feel like I’ve learned this over the years with Montreal, and I’ve seen it in so many other problems as well. There’s three Ps that determine how well we do on any environmental problem. The first one is personal. Is the issue personal to me, to us? And in the case of the ozone issue, it was deeply personal because skin cancer. I mean cancer, it doesn’t get any more personal than cancer, right. But also all the other attendant things that it can do. The second P is perceptible. Is the issue perceptible to me? The best case is if I can see it with my own eyes, like smog, but seeing the satellite images that we talked about earlier, that was good enough to make it perceptible to a lot of people. And the third big P is, are there practical solutions? And that’s where Steven’s type of work has been so important. So when you think about climate change and you think about the three P’s, people haven’t really considered it personal until pretty recently because it seemed like a future problem, not a today problem. And we can talk all we want to about caring about our grandchildren, but really what we care about is us, right?

So, and we do care about our grandchildren. Of course, we do, but not as much as we care about us. That’s just a fact, I think. It’s natural and normal and we don’t need to be embarrassed about it. Particularly when you’re talking about a future problem and you can always hope that there’ll be other solutions in place by then. So is it personal? For a long time we thought it wasn’t. Is it perceptible? For a long time we didn’t feel like it was. Nowadays, I would say more and more people are recognizing it as personal and perceptible. The kinds of things that have happened in the world this year have just been amazing. And being wake up calls because so many places have flooded, so many places have had massive fire. These are all the sorts of issues that we knew were happening. So much erosion is going on because of rising sea levels.

People are just, and actually when people would say to me, well, it’s not really perceptible yet. My answer to that would be, yeah, I know. And it’s a problem but it’s going to fix itself with time. And I think we’re just about there. It has fixed itself. And then there’s that big third P is, are the solutions practical. And there’s been a lot of propaganda out there saying the solutions are not practical, but I think we’re reaching the point now where we recognize that they are. So I think that we’re really at a turning point on climate change.

Lucas Perry: I’d love to pivot here more into exploring lessons to extract for what we can do about climate change and other global catastrophic risks for the theory of change about what we might do about those. And I’m also mindful of the time here. So if we could hit on these a little bit faster, that would be great. One thing I don’t think that, or one thing I would like to hit on more clearly is what is the bad thing that would have happened if the work of you, Steven, and the work of you, Susan, and all of the others who were involved in the discovery and solution towards the ozone hole, what is the bad thing that would have happened if we had just continued to use CFCs?

Susan Solomon: There is a lot of work on that now, people call it the world avoided scenarios. So what world did we avoid? Well, by mid century, it would have been about a degree hotter than it’s actually going to get. So that’s a degree Celsius by the way. So instead of a degree from, a degree in AF from mainly CO2 and methane that were trying to avoid, we would have an extra degree on top of that from CFCs that we would have had to avoid. That’s a big deal. Something like 20 million skin cancer cases in the United States sticks in my mind by mid century, but I would have to check that number to be absolutely sure.

Stephen Andersen: Yeah, Susan’s absolutely right. The latters can cancer cataract, but one of the things you can look at is you can say two interesting things. You could say, what if Molina and Rowland had not had this hypothesis? And certainly you could say, well, someone would eventually. But if it had been five years later, or 10 years later, it would have been catastrophic because it did take time as Susan said. It does take time to make a hypothesis, to confirm it, to do the ground measurements, to do the aerial flights and so forth. So it was just in time or a little bit too late that it was really a tight schedule that was working when you include diplomacy and corporate changes and all of those facts. But you can also look and say, what if the Montreal Protocol and Molina had been delayed some period of time.

And what you can see is exactly what Susan said, that the CFCs would have grown in climate forcing, let alone ozone depletion to a level that would have been untenable for earth. That they could have been almost the same level as the CO2 climate forcing. So we were incredibly fortunate to have this early announcement. It was incredibly fortunate that the Antarctic ozone hole was noticed finally, and announced and it was such a spectacular persuasion. And then fortunate that the Montreal Protocol was able to take this and then in a derivative that the corporations were able to make their reductions. I also think it’s important to remember that this really was a training ground for a lot of people, that scientists had not worked together successfully on assessments this large and so continuously brilliant over so many years. If you look back at each of the assessment panels, I find almost no valid criticism of any of the findings at any of the points in time that this was a well done process, actually stunning.

So the World Avoided, if you read the Nobel Award for 1995 for Crutzen and Molina and Rowland, it says that life on earth would not have been possible as we know it. If you read Paul Newman’s report on World Avoided, you find out that it would have been untenable to go outside at most latitudes for very long, without sun protection, far beyond what people wear when they go out today. So it would have been a lot of joy taken away, a lot of misery brought on by these medical effects. And it would be a less successful world because these technologies that replaced the ozone-depleting substances are purely superior, better energy efficiency, less toxic, more durable, more easy to repair and reliable. It’s quite a success story.

Lucas Perry: So let’s explore these P’s as Susan has put it. So we have the issue of greenhouse gases warming the climate today. And a lot of what you were involved in Steven was the economics of making transitions. So both the innovation required to replace HFCs and then also the questions around that being economical. So this is the importance also of industry being involved in the process. So I’m curious if you could explain your experience with industry and how difficult or easy it was to get industry to make this transition and how that compares to the transition that industry and governments need to make in the modern day around climate change. And how much of that difference is a bit circumstantial around the technologies and innovations that need to be made.

Stephen Andersen: Yeah, that’s a great question. Susan, I think agree, and I agree that some of the stakeholders on the climate side were more persistently ruthless than we’ve experienced under the Montreal Protocol. The early days when Molina and Rowland came up with their announcement, they couldn’t thought to have been more ruthless. Character assassination, there were lists of people that were not to be hired. The region said the University of California prohibited Sherwood Rowland for applying to certain organizations for funding and on and on. But if you look at what’s been done by the coal companies and petroleum companies, I think that that was orders of magnitude worse over a longer period of time, including interjecting a lot of wrong science over and over again, which was a terrible distraction and also took a lot of energy away. So there’s no doubt about the differences of the two. But what I have found out working on ozone layer is even sectors that have a bad reputation for other topics can be leaders on a topic once motivated as Susan Solomon says, and come to regret what they’re doing.

I’ll give an example. The automotive industry was among the most rigorous in getting rid of their solvent. Their foam uses, foams and cars include what are called safety fonts. So the, under the dash of the car originally, it was underneath the surface of the fabric, it was all ozone-depleting substances. It was all ozone-depleting substances for the refrigerant, for the solvent to make the components for the electronics and so forth. But they looked at science, they got motivated, and they stopped using these chemicals as quickly as the other sectors. So they were one of the fast to go. So what my lesson is you shouldn’t judge a book by the cover. You should not hold it against an institution because they misbehaved in the past and you should give them a chance to make a new start on leadership right now.

The other thing I would add is the public right now is much more engaged in climate than they ever were in ozone layer protection. So if you go back, there were very few of the industry projects that had active involvement of non-government organizations, environmentalist, because it was being done so well by the companies themselves that would have been futile use of those talents. But you look today, there are thousands of organizations that are demanding changes in industry. They’re in the streets, they are protesting. This did not happen for the ozone layer. It was the smallest amount of activity. So there’s the difficulty of the fossil fuel industry, but it’s offset by the ambition of the non-government organizations and some of the governments. So lots of things are happening there. What would you add Susan?

Susan Solomon: Well, I think you summarized it pretty well. I think the other thing I would add is the engagement of young people today, they have been exercised and have become pretty upset about the future that they see themselves inheriting. Greta Thunberg has done a fantastic job I think of mobilizing worldwide in a sense. The ability to get together on things like the tools that we now have with the internet have allowed that to become an international movement much more effectively than we could have ever done in the 80s for climate. No matter how concerned we were, the telephone only worked so fast. So I think you see a public engagement on climate change that is driving a lot of what’s going on and having a huge influence.

Lucas Perry: Is there anything that you both wish or would suggest as actions or things that are really important for the generation? The generations that face climate change. What it is that they need to understand and do. I mean, we’re talking about making things practical, personal, and then is the last P perspective?

Susan Solomon: Personal, perceptible and practical.

Lucas Perry: Yeah, and so there’s still this, I mean, a large problem also beyond these issues, global catastrophic and existential risk issues, beyond making them personal. And there is certainly difficulty around making action around them practical. With the last P for perceptible, a lot of people don’t even agree about the science of climate change. So I’m wondering if there’s any similarities between that and what you experienced with HFCs, and what you suggest there is to do about that. I mean, because the climate issue has become politicized, certainly.

Susan Solomon: Yeah, I mean, I would argue that it is becoming perceptible. I think most people around the world have noticed that the summers are hotter than they used to be. That heat waves are more extreme than they used to be. So perceptible is not so much the problem anymore I don’t believe. What is the problem is the practical, that they believe that it would cost too much and that it’s not practical to do it. That is increasingly becoming less and less tenable when it comes to power plants, for example. It is, nowadays if you’re going to build a new one, it is cheaper to do it with solar and onshore wind than it is to do it with either coal or gas. And so, and nuclear of course, too are more expensive. So there has been a pivot in the power industry to renewables that’s been very rapid. I think that we need to put a lot more investment into that because it does take an upfront investment.

It’s true that when it came to some of the things we could do for the ozone layer, a lot of them were really, we were getting rid of things that we didn’t have a deep investment with. I mean, how invested are you with your can of spray deodorant in your medicine cabinet? Probably not very, you can throw it away and or maybe you can even use it until it runs out and then go out and buy a roll-on, right. But probably a lot of people went out and just bought the roll-on because they figured it was a very good thing to do for the planet. But so this is indeed a lot tougher because of our investment in existing infrastructure, some of which is tremendously expensive. But I don’t think there’s any real barrier to making the transition.

And as soon as those things become… As soon as the alternatives become cheap enough, they essentially pay for themselves because energy drives everything. So if we can make energy more cheaply with solar and wind, then the cost of doing absolutely anything that requires energy becomes automatically cheaper too. And that makes a sort of a snowball effect of greater and greater demand. We need to make our grid more robust to things like intermittency and able to transmit electricity over broader spatial ranges. That is doable. It’s not, other countries have already done it actually. So it’s not something we couldn’t do. There’s a lot of things that are beginning to really happen quite quickly. And I’m very optimistic, but we do need some real changes in our existing infrastructure. There’s no doubt about it.

Stephen Andersen: Yeah, so there’s two things I would add. The first thing is the United States is now behind the rest of the world on barrier removal. So in Europe, you can use technology that’s been in the market for five years now, absolutely proven safe and effective energy efficient that’s prohibited by the USCPA for use in the United States. So they are a wall against new technology where they used to be a door. And so somebody has to get in there and motivate and get approval.

Susan Solomon: Can you give an example?

Stephen Andersen: Example, there’s a refrigerant called HFC32 that has a third of the global warming potential that has 20% higher energy efficiency. It’s mildly flammable, but it hasn’t been approved in the United States. Similarly, there are natural refrigerants that have not been approved. And if you look at the timeline and another example, which is easy to understand is they approved years ago, a decade or more ago, a chemical that has a GWP of less than one to replace HFC134a, which has a GWP of 1,300. This was allowed for light trucks and cars. But the industry at that time did not apply for what’s called highway trucks to big trucks that move cargo and off-road vehicles like farm tractors and construction equipment, mining equipment, forestry. And that the industry applied months and months ago to have this year’s further equipment.

There’s no difference in using it on an off-road equipment or an on-road equipment or a big truck or a small truck because the cab of a big truck is about the same size as the interior of a car. But for some reason, the EPA has not finished that process, which is now way beyond the statutory limit of time. And they say it’s because they just haven’t had time to do it. Well, this is not acceptable. You have to have a government moving at the pace of industry. And then the last thing, probably important, the United States military and military organizations all over the world we’re a part of protecting the ozone layer. So far that’s not the case. If you look at inside at the documentation of military organizations, they say it’s a force multiplier, it makes everything worse in national security. They say it’s an amplifier of conflict. There’s tremendous concern about the displacement of populations and immigrants across borders and distractions from security because you have to do humanitarian relief. That’s all in the documentation.

But so far, all they do mostly is to do resiliency of their own facilities. So they’re doing what they need to do to protect against the effects of climate change, but they haven’t engaged yet in stopping climate change, which is much more cost effective. The last thing you want to do is let climate change happen and then try to run away and hide and build against it because that’s brutally, brutally expensive. So I think those two things, if we were more aggressive on approving new technology, and if we had the military organizations involved as part of the skill set and part of the solutions and so forth, I think we’d go a long way.

Lucas Perry: As a final question here, so Steven, your panel, the technology and economic assessment panel was super successful in this strategic and coordination front on the technology and the replacement of the technology, which is something we’re also just exploring. Does the Paris Climate Accord have anything similar? And do you see a panel like this as being something also that’s crucial for the climate change crisis and also the governance around other global catastrophic and existential risks?

Stephen Andersen: Yeah, I think you’re right. And Susan and I have both tried over many years to get the climate convention to do something like the tape. Recently, I realized that if you don’t want to wait for the IPCC to do this, you could do it as a shadow chip. And that it’s very easy I think for an individual sector to organize itself under the same principles of being objective and including members that have the coincidence of interest in changing the market and changing the technology. You could put that together within an industry and then bring forward the solutions that you’d like to see implemented. And this is almost happening in Europe right now because they’re phasing down HFCs much faster than the United States. And they’re doing it on a sector by sector basis and they’re involved in the stakeholders. And the stakeholders have figured out that if they come to the EU with a single plan that cuts out their share of the goal, that the European Union will approve that. If they come in with separate views and lots of disagreement, the EU will choose their own plan for them.

So they have two choices, do it their way or do it the government’s way. And so far, they always chose to do it. The practical cost effective technology that they understand the best. And that’s exactly what the team did. So I’m very enthusiastic about that model. And in fact, that’s where, if I were an industry, that’s where I would put my money right now is I try to say, how do we become the leaders on this so that these activists that can cause mayhem in our company would say no use messing with that company that shows an in-state runs, let’s go bother someone else. Let’s let them solve their problems and we’ll go on to a recalcitrant truck sector and to give them bloody hell. I think that’s a very persuasive argument.

Lucas Perry: All right, Steven and Susan, thank you so much for coming on. If you have any final words for the audience about climate change, the ozone hole and existential risk, here’s a space for you to share it.

Susan Solomon: I hope we’ve given you some hope in this period of talking. I mean, it’s easy to become kind of despondent about climate change because there’re terrible events making people suffer day after day. On the other hand, I really do believe there’s light at the end of the tunnel. I think the ozone issue demonstrates that. And I think we are on the road to getting to a solution.

Stephen Andersen: Yeah, I would just add to that that organizations like Future of Life Institute are a tremendous part of the solution. I do believe that recognition and explanation and all of those things make a big difference to getting people motivated, to take on this very hard work. It’s work you love when it’s over, but it’s always hard while you’re doing it. So we have to have the highest motivation possible. And you folks are part of that solution.

Lucas Perry: Well, thank you very much, Steven and Susan, for coming on the podcast and also for your scientific and strategic coordinations to a global risk in our lifetimes.

Susan Solomon: Thank you, Lucas.

Stephen Andersen: Thank you very much.

James Manyika on Global Economic and Technological Trends

  • The modern social contract
  • Reskilling, wage stagnation, and inequality
  • Technology induced unemployment
  • The structure of the global economy
  • The geographic concentration of economic growth


Watch the video version of this episode here

29:28 How does AI automation affect the virtuous and vicious versions of productivity growth?

38:06 Automation and reflecting on jobs lost, jobs gained, and jobs changed

43:15 AGI and automation

48:00 How do we address the issue of technology induced unemployment

58:05 Developing countries and economies

1:01:29  The central forces in the global economy

1:07:36 The global economic center of gravity

1:09:42 Understanding the core impacts of AI

1:12:32 How do global catastrophic and existential risks fit into the modern global economy?

1:17:52 The economics of climate change and AI risk

1:20:50 Will we use AI technology like we’ve used fossil fuel technology?

1:24:34 The risks of AI contributing to inequality and bias

1:31:45 How do we integrate developing countries voices in the development and deployment of AI systems

1:33:42 James’ core takeaway

1:37:19 Where to follow and learn more about James’ work


Lucas Perry: Welcome to the Future of Life Institute Podcast. I’m Lucas Perry. Today’s episode is with James Manyika, and is focused on global economic and technological trends. As the Agricultural and Industrial Revolutions both led to significant shifts in human labor and welfare, so too is the ongoing Digital Revolution, driven by innovations, such as big data, AI, the digital economy, and robotics also radically affecting productivity, labor markets, and the future of work. And being in the midst of such radical change ourselves, it can be quite difficult to keep track of where we exactly are and where we’re heading. While this particular episode is not centrally focused on existential risk, we feel that it’s important to understand the current and projected impacts of technologies like AI, and the ongoing benefits and risks of their use to society at large, in order to increase our wisdom and understanding of what beneficial futures really consist of.

It’s in the spirit of this that we explore global economic and technological trends with James Manyika in this episode. James received a PhD from Oxford in AI and robotics, mathematics, and computer science. He is a senior partner at McKinsey & Company, as well as chairman and director of McKinsey Global Institute. James advised the chief executives and founders of many of the world’s leading tech companies on strategy and growth, product, and business innovation, and was also appointed by President Barack Obama to serve as vice chair of the Global Development Council at the White House. James is most recently the author of the book, No Ordinary Disruption: The Four Global Forces Breaking All the Trends. And it’s with that, I’m happy to present this interview with James Manyika.

To start things off here, I’m curious if you could start by explaining what you think are some of the most important problems in the world today.

James Manyika: Well, first of all, thank you for having me. Gosh, one of the most important problems in the world, I think we have the challenge of climate change, I think we have the challenge of inequality, I think we have the challenge that economic growth and development is happening unevenly. So I should say that the inequality question, I think is most in inequality within countries, but to some extent also between countries. And this idea of uneven development is that some countries are surging ahead and some parts of the world are potentially being left behind. I think we have other social political questions, but those I’m not qualified to about those, I don’t really spend my time, I’m not a sociologist or political scientist, but I think we do have some social-political challenges too.

Lucas Perry: We have climate change, we have social inequality, we have the ways in which different societies are progressing at different rates. So given these issues in the world, what do you think it is that humanity really needs to understand or get right in the century given these problems?

James Manyika: Yeah, by the way, I should also, before I dive into that, also say, even though we have these problems and challenges, we also have incredible opportunities, quite frankly, for breakthrough, progress, and prosperity to solve some of these issues. And quite frankly, do things that are going to transform humanity for the better. So these are challenges at a time of, I think, unprecedented opportunity and possibility. So I just want to make sure we acknowledge both sides of that issue. It terms of what we need to do about these challenges. I think part of it is also just quite frankly, facing them head on. I think the question of climate change is one that is an existential challenge that we just need to face head on. And quite frankly, get on with doing everything we can both to mitigate the effects of climate change, and also quite frankly, to start to adapt how our society, our economy works to, again, address what is essentially an existential challenge.

So I think what we do in the next 10 years is going to matter more than what we do in the 10 years after that. So there’s some urgency to the climate change and climate risk question. I think with regards to the issue of inequality, I think this is one that is also within our capacity to address. I think it’s important to keep in mind that the capitalism and the market economies that we’ve had, and we do have, have been unbelievably successful in creating growth and economic prosperity for the world and in most places where they’ve been applied. Particularly in recent years, I think, we’ve also started to see that in fact, there’ve been growth in inequality, partly because the structure of our economy is changing and we can get into that conversation.

In fact, some people are doing phenomenally well and others are not, and some places are doing phenomenally well and some other places are not. I mean, it’s not lost on me, for example, Lucas, that even if you look at an economy like the United States, something like two thirds of our economic output comes out of 6% of the counties in the country. That’s an inequality of place, in addition to the inequalities that we have of people. So I think we have to tackle the issue of inequality quite head-on. Unless we do something, it has the potential to get worse before it gets better. The way our economy now works, and this is quite different by the way than what it might’ve looked like even as recently as 25 years ago, which is, most of the economic activity, a function of the sectors that are driving economic growth, a function of the types of enterprises and companies that are driving economic growth, they have tended to be much more today than they were 25 years ago to be fairly regionally concentrated and in particular places.

So some of those places include Silicon Valley and other places. Whereas if you had looked for example, 25 years ago where you might’ve seen the kind of sectors and companies that were doing well were much more geographically distributed, so you had more economic activity coming out of more places across the country than you do now. So this is just, again, a function of not that anybody designed it to be that way, but just a function of the sectors and companies and the way our economy works. A related factor by the way, is even the inequality question is also a function of how the economy works. I mean, it used to be the case that whenever we had economic growth and productivity growth, it also resulted in job growth and wage growth. That’s been true for a very long time, but I think in recent years, as in, depending on how you count it, the last 25 years ago or so, when we have productivity growth, it doesn’t lift up wages as much as it used to. Again, it’s a function of the structure of the economy.

In fact, some of the work we’ve been doing and other economists have been doing, is actually being to look at this so-called declining labor share. I think a way to understand that declining labor share is to think about the fact that, if you had to set up a factory say in a hundred years ago, most of the inputs into how that factory worked were all labor inputs, so the labor share of economic activity is much higher. But over time, if you’re setting up a factory today, sure you have labor input, but you also have a lot of capital input in terms of the equipment, the machinery, the robots, and so forth. So the actual labor portion of, you know, of it as a share of the inputs, it’s being going down steadily. And that’s just part of how structure of our economy is changing. So all of these effects are some of what is leading to some of the inequality that we see.

Lucas Perry: So before we drill more deeply into climate change and inequality, are there any other issues that you would add as some of the most crucial problems or questions for the 21st century?

James Manyika: The structure of our global economy is changing, and I think it’s now getting caught up also in kind of geopolitics. I’m not a geopolitical expert, but it’s not lost from a global economy standpoint that, in fact, we now have and will have two very large economies, the United States and China, and China is a very large economy. And it’s not just a source of exports or the things that we buy from them, but it’s also entangled with say the US and other countries economically in terms of monetary, and debt, and lending, and so forth. But it’s also a large economy itself, which is going to have its own consumption. So we now have for the first time, two very large global economies. And so, how that works in a geopolitical sense is one of the complications of the 21st century. So I think that’s an important issue. Others who are more qualified to talk about geopolitics can delve into that one, but that’s clearly in the mix as part of the challenges of the 21st century.

We also, of course, are going to have to think about the role of technology in our economies and our society. Partly because technology can be a force of massive productivity growth, innovation, and good, and all of that, but at the same time we know that many of these technologies raise new questions about privacy, about how we think about information, disinformation. So I think, you know, if you had to write the list of the questions we’re going to need to navigate in the coming decades of the 21st century, it’s a meaty list. Climate change is at the top of that list in my view, inequalities is on that list, and these questions of geopolitics are on that list, the role that technology is going to play is on that list, and then also some of these social questions that we now need to wrestle with, issues of social justice, not just economic justice but also social justice. So we have a pretty rich challenge list even at the same time that we have these extraordinary opportunities.

Lucas Perry: So in the realms of climate change, and inequality, and these new geopolitical situations and tensions that are arising, how do you see the role of incentives pushing these systems and problems in certain directions, and how it is that we come up with solutions to them given the power and motivational force of incentives?

James Manyika: Well, I think incentives play an important role. So take the issue of climate change, for example, I think one of the failures of our economics and economic systems is we’ve never quite priced carbon, and we’ve never quite built that into our incentive systems, our economic systems so we have a price for it. So that when we put up carbon dioxide into the atmosphere and so forth, there’s no economic price for that or incentives, a set of incentives not to do that. We haven’t done enough in that regard. So that’s an area where incentives would actually make a big difference. In the case of inequality, I think this one’s way more complicated beyond just incentives, but I’ll point to something that is in the realm of incentives as regards to inequality.

So, for example, take the way we were talking earlier about the importance of labor and capital in our capital inputs, I don’t mean capital as in the money necessarily, but just the actual capital, equipment and machines, and so forth in our system, we’ve built a set of incentives, for example, that encourage companies to invest in capital infrastructure, you know, capital equipment, they can write it off for example. We encourage investments in R&D for example, and the tax incentives to do that, which is wonderful because we need that for the productivity and growth and innovation of our economy, but we haven’t done anything nearly enough or equivalent to those kinds of incentives with regards to investments in human capital. So you could imagine in much more the productivity and growth and innovation of our economy, but we haven’t done anything nearly enough or equivalent to that, to those kinds of incentives with regards to investments in human capital.

So you could imagine a much more concerted effort to create incentives for companies and others to invest in human capital and be able to write off investments in skilling, for example, to be able to do that at the equivalent scale to the incentives we have for other forms of investment, like in capital or in R&D. So that’s an example of where we haven’t done enough on the incentive front, and we should. But I think there’s more to be done than just incentives for the inequality question, but those kinds of incentives would help.

I should tell you, Lucas, that one of the things that we spent the last year or so looking at is trying to understand how the social contract has evolved in the 21st century so far. So we actually looked at, for example, roughly 23 of the OECD countries, about 37 or 38 of them, but we looked at about 23 of them in detail just to understand how the social contract had evolved. And here because we’re not sociologists, we looked to the social contract in really three ways, right? How people participate in the economy as workers, because that’s part of, you know, when people work hard, and probably the exchange is that they get jobs, and they get income and wages and training. So people participating as workers is an important part of the social contract. People participating as consumers and households who consume products and services, and then people as savers who are kind of saving money for a future, you know, for their kids or for their future, et cetera.

And when you look at those three aspects of the social contract in the 21st century so far, it’s really quite stunning. So take the worker piece of that, for example, what has happened is that in across most countries, we’ve actually grown jobs despite the recession, I guess the recession in 2001, but also the bigger one in 2008, we’ve actually grown jobs. So there’re actually more jobs now than there were at this time of the 21st century. However, what has happened is that many of those jobs don’t pay as well, so the wage associated with that, the picture has actually shifted quite a bit.

The other thing about what’s happened with work is it’s becoming a little bit more brittle in the fact that job certainty has certainly gone down, there’s much more income and wage variability. So we’ve created more fragile jobs relatively to what we had at the start of the 21st century. So you could say, for workers, it’s a mixed story, right? Job growth, yes, wage growth, not so much, job certainty and job fragility has gone up. When you look at people as consumers and households, it also paints an interesting story. And the picture you see there is the fact that, if households and consumers are consuming things like, think about, you know, buying cars or wide goods products, or electronics, basically things that are globally competed and traded, the cost of those has gone down dramatically in the 21st century. So the 21st century in that sense, at least globalization has been great because it’s delivered these very cheap products and services.

But if you look at other products and services that households and consumers consume such as education, housing, healthcare, and in some places, depending which country or place you’re in, transportation, those have actually gone up dramatically, far, far higher and faster than inflation, far higher and faster than wage growth. In fact, if you are in the bottom half of the social income scale, those things have come to dominate your income in terms of what you spend money on. So for those people, it hasn’t worked out so well actually in terms of the social contract. And then on the savers side, people as savers are, very few people now can afford to save for the future. And one of the things that you see is that the growth of indebtedness in the 21st century so far has gone up for most people, especially the middle wage and low wage households and people, their ability to save for the future has gone down.

What’s interesting is, it’s not just that the levels of indebtedness have gone up, but it’s the fact that the people who are indebtedness look a little bit different. They’re younger, they’re also, in many cases, college educated, which is different than what you might’ve seen 25 years ago in terms of who was indebted and what do they kind of look like?

And then finally, just to finish, the 21st century in the social contract sense, also hasn’t worked out very well for women who still earn less than men, for example, and don’t quite have the opportunities as much as others, as well as for people of color. It hasn’t, so they still earn a lot less, employment rates are still much lower, their participation in the economy as any of these roles is also much less. So you get a picture that says, while the economy has grown and capitalism has been great, so far in the social contract sense at least, by these measures we’ve looked, at it hasn’t worked out as well for everybody in the advanced economies. This is a picture that emerges from the 23 OECD countries that we looked at. And the United States is on the more extreme end of most of the trends I just described.

Lucas Perry: Emerging from this is a pretty complex picture, I think, of the way in which the world is changing. So you said the United States represents sort of the extreme end of this, where you can see the largest effect size in these areas, I would assume, yet it also seems like there’s this picture of the Global East and South generally doing better off, like people being lifted out of poverty.

James Manyika: Yeah, it is true. So one of the wonderful things about the 21st century is in fact, close to a billion people have been lifted out of poverty in those roughly 20, 25 years, which is extraordinary, but we should be clear about where that has happened. Those billion people are mostly in China, and to some extent, India. So while we say we’ve lifted people out of poverty, we should be very specific about mostly where that has happened. There are parts of the world where that hasn’t been the case, parts of Africa, other parts of Asia, and even parts of Latin America. So this lifting people out of poverty has been relatively concentrated in China primarily, and to some extent in India.

One of the things about economics, and this is something that people like Bob Solow and others got Nobel Prizes for, if you think about what is it that drives our economic growth, if economic growth is the way we create economic surpluses, that we can then all enjoy and lead to prosperity, right? The growth desegregation models come down to two things: either you’re expanding labor supply, or you are driving productivity. And the two of those when they work well, combine to give you economic GDP growth.

So if you look, for example, at the last 50 years, both across the advanced economies, but even for the United States, the picture that you see is that much of our economic growth has come roughly, so far at least, has come roughly in equal measure from two things: one, you know, this is over the last 50 or so years, half of it has come from expansions in labor supply. You can think about it as a Baby Boomer Generation, more people entering the workforce, et cetera, et cetera. The other half has roughly come from productivity growth. And the two of them have combined to give us roughly the economic GDP growth that we’ve had.

Now, when you look forward from where we are, we’re not likely to get much lift from the labor supply part of it, partly because most advanced economies are aging. And so, the contribution that’s going to come from expansions in labor supply, much less. I mean, you can think of it as kind of a plane flying on two engines, right? If one engine has been expansions in labor supply and the other is in productivity, well, the labour supply engine is kind of dying out to or slowing down in its output.

We’re going to rely a lot more on productivity growth. And so, where does productivity growth come from? Well, productivity growth comes from things like technological innovation, innovating how we do things and how we create products and services and all of that. And technology is a big part of that. But guess what happens? One of the things that happens with productivity growth is that the role of technology goes up. So I come back to my example of the factory. So if you wanted a highly productive factory, it’s likely that your mix of labor inputs and capital inputs, read that as machinery and equipment, is going to change. And that’s why your factory a hundred years ago, it looks very different than a factory today. But we need that kind of technological innovation and productivity to drive it the output. And then the output leads to the output in the sector, and ultimately the output in the economy.

So all of that is to say, I don’t think we should stop the technological innovation that leads to productivity, we need productivity growth. In fact, going forward, we’re going to need productivity growth even more. The question is, how do we make sure that even as we’re pursuing that productivity growth that contributes to economic growth, we’re also paying attention to how we mitigate or address the impacts on labor and work, which is where most people derive their livelihoods.

So, I don’t think you want to set up a set of system of incentives that slows down the technological innovation and product activity growth, because otherwise, we’re all going to be fighting over a diminishing economic pie. I think you want to invest in that and continue to drive that, but at the same time find ways and think about how to address some of the work implications of that, or the impacts on work and workers. And that’s been one of the challenges that we’ve had. I mean, we’ve all seen the hollowing out, if you like, of the middle class in advanced economies like America, where a big part of that is that much of that middle class or middle income workers have been working in these sectors and occupations where the impact of technology and productivity have actually had a huge impact on those jobs and those incomes.

And so, even though we have work in the economy, the occupations and jobs in sectors that are growing have tended to be in the service sectors and less in places like manufacturing. I mean, it’s the reason why I love it when politicians talk about good manufacturing jobs. I mean, they have a point in the sense that historically, those have been good, well paying jobs, but manufacturing today is only what? 8% of the labor force in America, right? It’s diminished, at its peak is probably at best close to the mid forties, 40% as a share of the labor markets back in the ’50s, right? It’s being going down ever since, right? And yet the service sector economy has been growing dramatically. And many, not all, but many of the jobs in the service sector economy don’t pay as much.

My point is, we have to think about not just incentives but the structure of our economy. So if you look forward, for example, over the next few decades, what of the jobs that are going to grow as a function of both demand for that work in the economy, but also as a result of what’s less likely to be automated by technology and AI and so forth? You end up with a list that includes care work, for example, and so forth. And even work that we say is valuable, which it is, like teachers and others that are harder to automate. But labor market system doesn’t reward and pay those occupations as much as some of the occupations that are declining. So those are some of what, when I talk about the changes in the structure of our economy, in a way that goes a little bit beyond just local incentives, is how do we address that? How do we make sure as those parts of our economy grow, which they will naturally. How do we make sure people are earning enough to live as they work in those occupations? And by the way, those occupations are many of the ones that in our current or recent COVID moment or period here, many are where the essential work and workers are by the way, people have come to rely on mostly in those service sector economies that we haven’t historically paid well. Those are real challenges.

Lucas Perry: There was that really compelling figure that you gave at the beginning of our conversation, where you said 6% of counties account for two thirds of our economic output. And so, there’s this change in dynamic between productivity and labor force. And the productivity you’re suggesting is what is increasing, and that is related to and contingent upon AI automation technology. Is that right?

James Manyika: Well, first of all, we need productivity to increase. It’s been kind of sluggish in the last several years. In fact, it’s one of the key questions that economists worry about, which is, how can we increase the growth of our economic productivity? It hasn’t been doing as well as we’d like it to do. So we’d like to actually increase it, partly because, as I said, we needed more than we’ve done in the last 50 years because of the labor supply pieces declining. So we actually would like productivity to go up even more.

Mike Spence and I just wrote a paper recently on the hopeful possibility that in fact we could see a revival in productivity growth coming out of COVID. We hope that happens, but it’s not assured. So we need more productivity growth. And the way you get productivity growth, technology and innovation is a big part of it. The other part of it is just managerial innovation that happens inside companies in sectors where those companies and sectors figure out ways to organize and do what they do in innovative, but highly productive ways. So it’s the combination of technology and those kinds of managerial and other innovations, usually in a competitive context, that’s what drives productivity.

Lucas Perry: Does that lead to us requiring less human labor?

James Manyika: It shouldn’t. One of the things about productivity is, it’s actually, in some ways, labor productivity is a very simple equation, right? It has on the numerator, value-added output, divided by hours worked or labor input, if you like. So you can have, what I think of as a virtuous version of productivity growth versus a vicious one. So let me describe the virtuous one. The virtuous one, which actually leads to job growth, is when in fact you expand the numerator. So in other words, there’s innovations, use of technology in the ways that I talked about before, that means to companies and sectors creating more valuable output, more of it and more valuable output. So you expand the numerator. So if you do that, and you expand the numerator much higher and faster than you’re reducing the denominator, which is the labor hours worked, you end up with a virtuous cycle in the sense that the economy grows, productivity grows, everything expands, the demand for work actually goes up. And that’s a virtuous cycle.

The last time we saw a great version of that was actually in the late ’90s. This is, if you recall before that, Bob Solow kind of framed what ended up being called the Solow Paradox, which is this idea that before the mid and late ’90s, you saw computers everywhere except in the productivity figures, right? And that’s because we hadn’t seen the kinds of deployment of technology, the managerial innovations do the kind of, what I call the numerator-driven productivity growth, which, when it did happen in the mid to late ’90s, it created this virtuous cycle.

Now let me describe the vicious cycle, which is the, if you like, the not so great version of productivity growth. It’s when you don’t expand the numerator, but what you do is simply reduce the denominator. So in other words, you reduce the hours worked. In other words, you become very efficient at delivering the same output or maybe even less of the output. So you reduce the denominator, that lead to productivity, but it’s off the vicious kind, right? Because you’re not expanding the output, it’s simply reducing the inputs, or the labor inputs. Therefore, you end up with less employment, fewer jobs, and that’s not great. That’s when you get what you asked about, which is, where you need less labor. And that’s the vicious version of productivity, you don’t want that either.

Lucas Perry: I see. How does the reality of AI and automation replacing human labor and human work essentially increasingly completely over time factor into and affect the virtuous and vicious versions of productivity?

James Manyika: Well, first of all, we shouldn’t assume that AI is simply going to replace work. I think we should think about this in this context of what you might call complements and substitutes. So if our AI technology is developed and then deployed in a way that is entirely substitutive of work, then you could have work decline. But there’s also other ways to deploy AI technology, where it’s complementary and it complements work. And in that case, you shouldn’t have to think about it as much about losing jobs.

Let me give you some specifics on this. So we’ve done research, and others have too, but let me describe what we’ve done, but I think the general consensus is emerging that it’s close to what at least we found in our research, which is that, so we looked at, so the Bureau of Labor Statistics kind of tracks in the US, tracks roughly 800 plus occupations. We looked at all those occupations in the economy. We also looked at the actual particular tasks and activities that people actually do, this because any of our jobs and occupations are not monolithic, right? They’re made up of several different tasks, right?

I spent part of my day typing, or talking to people, or analyzing things, so we’re all an amalgam of different tasks. And we looked at over 2000 tasks that go into these different occupations, but let me get to where we ended up. So where we ended up was, we looked at what current and expected AI technology and artificial technologies can do. And we came to the conclusion that at least over the next couple of decades at the task level, and I emphasize the task level, not the job level, these are tasks, I’ll come back to jobs, at the task level, these technologies look like they could automate as much as 50% of the tasks and activities that people do. And it’s important to, again, emphasize those are tasks, not jobs.

Now, when you take those highly automatable tasks back and map them to the occupations in the economy, what we concluded was that something like at most 10% of the occupations look like they have all of their constituent tasks automatable. And that’s a very important thing to note, right? 10% of all the occupations look like they have close to a hundred percent of their tasks that are automatable.

Lucas Perry: In what timeline?

James Manyika: This is over the next couple of decades.

Lucas Perry: Okay. Is that like two decades or?

James Manyika: We looked at this over two decades, right? We have scenarios around that because it’s very hard to be precise because you can imagine the rate of technologies development speeding up, I’ll come back to that, but the point is, it’s only 10% of the, in our analysis anyway, 10% of the occupations look like they have all of their constituent tasks that are automatable in that rough timeframe. But at the same time, what we also found is that something like 60% of the occupations have something like a third of their constituent tasks that are automatable in that same period. Well, what does that mean? What that actually means is that many more jobs and occupations are going to change than get fully automated away. Because what happens is, sure, some activity that I used to do myself, now that activity can be done in an automated fashion, but I still do other things too, right? So this effect of kind of the jobs that will change is actually a bigger effect than the jobs that will disappear completely.

Now that’s not to say there won’t be any occupations that will decline. In fact, what we found in our research, and we ended up kind of titling the research report, Jobs Lost and Jobs Gained. We probably should have fully titled the Jobs Lost, Jobs Gained, and Jobs Changed because all three phenomena will happen, right? Yes, there’ll be occupations that will decline, but there will also be occupations that’ll grow actually. And then there’ll be lots more occupations that will change. So I think we need to take the full picture into account. It’s a bit like, I guess a good example of the jobs changed portion is the one of the bank teller, right? Which is, if you had looked at what a bank teller did in 1968 versus what a bank teller does now, it’s very, very different, right? The bank teller back then spent all their time counting money either to take it from you or to give it back to you when you went up to the bank teller. Or the advent of ATM machines or the ATM machine automated much of that.

So we still have bank tellers today, the majority of the time isn’t spent doing that, right? They may do that on an exception basis, but their jobs have changed dramatically, but there’s still an occupation called a bank teller. And in fact, until about, I think the precise date is something like 2006, I think, what we actually had was a number of bank tellers in the US economy had actually grown since the early ’70s to about 2006. And that’s because the demand for bank tellers went up, not on a per bank basis, but on a economy-wide basis because we ended up opening up so many more branch banks by 2006 than we had in 1968. So the collective demand for banking actually drove the growth in the number of bank tellers, even though the number of bank tellers per branch might’ve gone down.

So that’s an example of where a growing economy can create its own demand for work back to this virtuous cycle that I was talking about as opposed to the vicious cycle that I was talking about. So this phenomenon of jobs changing is an important one that often gets lost in the conversation about technology and automation and jobs. And so, to come back to your original question about substitutes, we shouldn’t just think of technology substituting for jobs as the only thing that happens, but also that technology can complement work and jobs. In fact, one of the things to think about, particularly for AI researchers or people who develop these automation technologies, I think, on the one hand, while it’s actually certainly useful to think of human benchmarks when we say, how do we build machines and systems that match human vision or human dexterity and so forth? That’s a useful way to set goals and targets for technology development and AI development. But in an economic sense, it’s actually less useful because it’s less likely to lead to technologies that are more substitutes because we’ve built them to match what humans can do.

Imagine if we said, let’s build technology machines that can see around corners or do the kinds of things that humans can’t do, then we’re more likely in that case to build more complementing technologies than substituted technologies. I think that’s one of the things that we should be thinking about and doing a heck of a lot more to achieve.

Lucas Perry: This is very interesting. So you can think of every job as basically a list of tasks, and AI technology can automate say some number of tasks per job, but then the job changes in a sense that either you can spend more time on the tasks that remain and increase productivity by just focusing on those tasks, or the fact that AI technology is being integrated into the job process will create a few new tasks. The tension I see though is that we’re headed towards a generality with AI where we’re moving towards all tasks being automated. Perhaps over shorter timescales it seems like we’ll be able to spend more time on fewer tasks or our jobs will change in order to meet and work on the new tasks that AI technology demands of us, but generality is a movement towards the end of human level problem solving on work and objective-related tasks. So it seems like it would be increasingly shrinking. Is that a view that you share? Does that make sense?

James Manyika: Your observation makes sense. I don’t know if I fully share it, but just to back up a step, yeah, if you asked me over the next few decades, I mean, our research has looked at the next couple of decades, others have looked at this too by the way, and he’d come up with obviously slightly different numbers and views, but I think they’re generally in the same direction that I just described. So if you say over the next couple of decades, what do I worry about? I certainly don’t worry about the disappearance of work for sure. But that doesn’t mean that all is well, right? There’re still things that I worry about. So I still worry about, well, we’re going to have work, because I think, you know, what we found for example is the net of jobs lost and jobs gained and jobs changed, the net of all of that in the economies that we’ve looked at is still a net positive in the sense that there’s more work gained net then lost.

That doesn’t mean we should all then be, rest in our laurels and be happy that, hey, we’re not facing a jobless future. So I think we still have a few other challenges to deal with. And I want to come back to your future AGI question in a second. So one of the things to worry about, even in this stage where I say don’t worry about the disappearance of work, well, there’re still a few more things to worry about.

I think you want to worry about the transitions, right? The skill transitions. So if some jobs are declining, and some jobs are growing, and some jobs are changing, all of that is going to create a big requirement for skilling and reskilling, either to help people get into these new jobs that are growing, or if their jobs are changing, gain the new skills that work well alongside the task that the machines can do. So all of that says reskilling is a really big deal, which is why everybody’s talking about reskilling now, though, I don’t think we’re doing it fast enough or at scale enough, at the scale and pace that we should be doing it. But that’s one thing to worry about.

The other thing to worry about are the effects on wages. So even when you have enough work, if you look at the pattern of the jobs gained, most of them, not all of them, but many of them, many of them are actually jobs that pay less, at least in our current labor market structure, right? So care work is hard to fully automate because it turns out that, hey, it’s actually harder to automate somebody doing physical mechanical tasks than say somebody doing analytical work. But it turns out the person doing analytical work, where you can probably automate what they do a lot easier, also happens to be the person who’s earning a little bit more than the person doing the physical mechanical tasks. But by the way, that person is one that we don’t pay much in the first place. So you end up with physical mechanical activities that are hard to automate also growing and being demanded, but then we don’t pay much for them.

So the wage effects are something to worry about. Even in the example I gave you of complementing work, that’s great from the point of view of people and machines working alongside each other, but even that has interesting wage effects too, right? Because at one end, which I’ll call the happy end, and I’ll come back to the challenged end, the happy end is when we automate some of what you do, Lucas, and the combination of what the machine now does for you and what you still do yourself as a human, both are highly valuable, so the combo is even more productive. And this is the example that’s often given with the classic story of radiologists, right? So machines can maybe read some of those images way better than the radiologist, but that’s not all the radiologist does, there’s a whole other value-added activities and tasks that the radiologist does that the machine reading doesn’t understand them, MRI doesn’t do. But now you’ve got a radiologist partnered up with a machine, the combination is great. So that’s a happy example. Probably the productivity goes up, the wages of that radiologist go up. That’s a happy story.

Let me describe the less happy end of that complementing story. The less happy end of that is when the machine automates a portion of your work, but the portion that it automates is actually the value-added portion of that work. And what’s left over is even more commoditized, commoditized in the sense that many, many, many more people can do it, and therefore, the skill requirements for that actually go down as opposed to go up, because the hard part of what you used to do is now being done by a machine. The danger with that is that, that then potentially depresses the wages for that work given the way you’re complementing. So even the complementing story I described earlier, isn’t always in one direction from a wage effect and its impact.

So all of that step back is to say, if the first thing is reskilling, the second thing to worry about are these wage effects. And then the final thing to worry about, how we think about redesigning work itself and the workflows themselves. So all of that is to say, even in a world where we have enough work, that’s in the next few decades, we still are going to have to work these issues. Now, you are posing a question about, what about in the long, long future, because I should think it’s in the long future that we’re going to have AGI. I’m not one who thinks it’s as imminent as perhaps others think.

Lucas Perry: Do you have a timeline you’d be willing to share?

James Manyika: No, I don’t have a timeline, I just think that there’re many, many hard problems that we still seem like a long way from… Now, the reason I don’t have a timeline, is that, hey, we could have a breakthrough happen in the next decade that changes the timeline. So we haven’t figured out how to do causal reasoning, we haven’t figured out how to do what Kahneman called System 2 activities. We’ve solved System 1 tasks where we assisted… And so, there’s a whole bunch of things that, you know, we haven’t solved the issues of how we do a higher-level cognition or meta-level cognition, we haven’t solved through how we do meta learning, transfer learning. So there’s a whole bunch of things that we haven’t quite solved. Now we’re making progress on some of those things. I mean, some of the things that have happened with these large language universal models is really breathtaking, right?

But I think that, in my view, at least the collection of things that we have to solve before we get to AGI, there’s too many that still feel unsolved to me. Now we could have somebody breakthrough in a day. That’s why I’m not ready to give a prediction in terms of timeline, but these seem like really hard problems to me. And many of my friends who are working on some of these issues also seem to think these are hard problems. Although there are some of them who think that we’re almost there, that all we need to, you know, deep learning will get us to most places we need to get to and reinforcement learning will get us most of what we need. So those are my friends who think that, think that it’s more imminent-

Lucas Perry: In a decade or two away, sometimes they say.

James Manyika: Yeah, some of them say a decade or two. There’s a lot of real debate about this. In fact, you may have seen one of the things that I participated in a couple of years ago was, and Martin Ford put together a book that was a collection of interviews with a bunch of people, it’s his book, Architects of Intelligence. A wonderful range of people in that book, I was fortunate enough to be included, but there are many more people and way more interesting than me. People like Demis Hassabis and Yoshua Bengio and a whole bunch of people, it’s a really terrific collection. And one of the things that he asked that group who are in that book was to ask them to give a view as to when they think AGI would be achieved. And what came out of it is a very wide range from 2029, and I think that was Ray Kurzweil who stuck to his date, and all the way to something like 500 years from now. And that’s a group of people who are deep in the field, right? And you’d get that very wide range.

So I think, for me, I’m much more interested in the real things that we are going to need to break through, and I don’t know when we’ll make those breakthroughs, it could be imminent, it could be a long time from now, but they just seem to be some really hard problems to solve. But if you take the view, to follow your thought, if you take the view, you know, the thought experiment to say, okay, let’s just assume we’ll truly achieve AGI in all its sense, in both in the AGI and the, some people will say in the oracular.

I mean, it depends what form of the AGI it takes. If the AGI takes the form of both the cognitive part of that coupled with the embodiment of that of physical machines that can physically participate, and you truly have AGI in a fully-embodied sense as well in addition to the cognitive sense, what happens to humans and work in that case? I don’t know. I think that’s where presumably those machines allow us to create enormous surpluses and bounties in an economic sense. So presumably, we can afford to pay everybody, you know, to give everybody money and resources. And so, then the question is, in a world of true abundance, because presumably they’ll help us solve these, you know, these machines, AGIs will help us solve those things, in a world of true abundance, what do people do in that?

I guess it’s kind of akin, as somebody said, to Star Trek economy. What do people do in a Star Trek economy when they can replicate and do everything, right? I don’t know. I guess we explore the universe, we do creative things, I don’t know. I’m sure we’ll create some economic system that takes advantage of the things that people can still uniquely do even though they’ll probably have a very different economic value and purpose. I think humans will always find a way to create either literally or quasi economic systems of exchange of something or other.

Lucas Perry: So if we focus here on the next few decades where automation is increasingly taking over particular tasks and jobs, what is it that we can do to ensure positive outcomes for those that are beginning to be left behind by the economy that requires skill training and those whose jobs are soon to have many of the tasks automated?

James Manyika: Starting now, actually, in the next decade or two, I think there’re several things, there’s actually a pretty robust list of things we need to do actually to tackle this issue. I think one is just reskilling. We know that there’s already a shortage of skills. Even before we think about, we’ve had skill mismatches for quite a while before any of this fully kicks in. So this is a challenge we’ve had for a while. So this question of reskilling is a massive undertaking, and here, the question is really due to pace and scale, because while there are quite a lot of reskilling examples one can come across, and there are many of them that have been very successful. But I think the key thing to note about many of them, not all of them, but many of them is that they tend to be small.

One of the questions one should always ask about all the great reskilling examples we hear of is, how big is it, right? How many people well went through that program? And I think you’ll find that many of them, not all of them, many of them are relatively small. At least small relative to the scale of the reskilling that we need to do. Now, there’ve been a few big ones, I happen to like, for example, Walmart has had these Walmart academies, it’s been written about publicly quite a bit, but what’s interesting about that is, it’s one of the few really large scale reskilling, retraining programs through their academies. Then something like, I can’t remember reading this, but they’ve put something like 800,000 people through those academies. I like that example, simply because the numbers talked to sound big and meaningful.

Now I don’t know. I haven’t evaluated the programs, are they good? But I think the scale is about right. So, the reskilling at scale is going to be really important, number one.

The other thing we’re going to need to think about is, how do we address the wage question? Now, the wage question is important, for lots of reasons here.

One is, if you remember earlier in our conversation, we talked about the fact that over the last two decades, for many people, wages haven’t gone up, been relative wage stagnation, compared to rates of inflation, or the cost of living, and how things have gone up. Wages haven’t gone up.

The wage stagnation is one we already have, before we think about technology. But then, as we’ve just discussed, technology may even exacerbate that, even when there are jobs, and the continuing changing structure of our economy will also exacerbate that. So what do we do about the wage question?

One could consider raising minimum wage, right? Or one could consider ideas like UBI. I mean, we can come back and talk about UBI. I have mixed the views about UBI. What I like about it is the fact that it’s at least a recognition that we have a wage problem, that people don’t earn enough to live. So I like it in that sense.

Now the complication with it, in my view, is that while, of course, one of the primary things that work does for you, for the vast majority of people, that’s how they derive their livelihood, their income. So it’s important, but work also does other things, right? It’s a way to socialize, it’s a way to give purpose and meaning, et cetera.

So I think UBI, it may solve the income part of that, which is an important part of that. It may not address the other pieces of the other things that work does. So, we have to solve the wage problem.

I think we also have to solve this geographic concentration problem. We did some work where we looked at all the counties in America at the time that we did this, because the definition of what’s a county in America kind of changes a little bit year from year. But at the time that we did this work, which was back in 2019, though I think we looked at something like 3,149 counties across America.

What we’re looking at there was, it was a range of factors about economic investment, economic vibrancy, jobs, wage growth. We looked at 40 different variables in each county, but I’m just going to focus on one, which is job growth.

When we looked at job growth across those counties, while at the national level, we’re all celebrating the job growth that had happened, coming out of the 2008 recession, between 2008 and 2018 was the data set we looked at, first of all, at the national level, it was great. But when you looked at it at the county level, what you suddenly found is that a lot of that job growth was concentrated in places where roughly a third of the nation’s workers live.

The other two-thirds of the place where people live either saw flat or no job growth, or even continued job decline. All of that is to say, we also have to solve this question of, how do we get more even job growth and wage growth across the country, in the United States?

We’ve also done similar work with, we’ve looked at these micro regions in Europe, and you see similar patterns, although maybe not quite as extreme as the US, but you see similar patterns where some places get a lot of the job and wage growth, and some cases get less of it. It’s just a function of the structure of our economy. So we’d have to solve that, too.

Then the other thing we need to solve is the classic case of the hollowing out of the middle class. Because if you look at the pattern of, mostly driven by technology, to some extent, a lot of the job declines or the jobs lost as a result of technology have primarily been in the middle wage, middle-class jobs. And a lot of the job growth has been in the low wage jobs.

So this question of the hollowing out of the middle class is actually a really particular problem, which has all kinds of sociopolitical implications, by the way. But that’s the other thing to figure out. So let me stop there.

But I think these are some of the things we’re going to need to tackle in the near term. I’ve made that list mostly in the context of say, an economy like the United States. I think if you go outside of the United States, and outside of the advanced economies, there’s a different set of challenges.

I’m talking about places outside of the OECD countries and China. So you go to places like India, and lots of parts of Africa and Latin America, where you’ve got a very different problem, which is demographically young populations. China isn’t, but India and most of Africa is, and parts of Latin America are.

So there the challenge is, a huge number of people are entering the workforce. The challenge there is, how do you create work for them? That’s a huge challenge, right? When you’re looking at those places, the challenge is just, how do you create enough jobs in very demographically young countries?

The picture’s now gotten a little bit more complicated in recent years than perhaps in the past, because in the past, the story was, if you are a developing country, a poor developing country, your path to prosperity was to join the global economy, be part of either the labor supply or the cheap labor supply often, and go from being an agrarian country to an industrialized country. Then ultimately, maybe some day, you’ll become a service economy. Most advanced economies are.

That path of industrialization is less assured today than it used to be, for a bunch of reasons. Some of those reasons have to do with the fact that advanced economies now no longer seek cheap labor abroad as much as they used to. They still do for some sectors, but less so for many other sectors, I mean, we’re less likely to do that.

Part of that is technology, the fact that in some ways, manufacturing has changed. We can now, going forward, do things more like 3-D printing, and so forth. So the industrialization path is less available to poor countries than it used to be.

In fact, economists like Danny Roderick have written about this, and called it this kind of premature de-industrialization challenge which is facing many low income countries. So we have to think about what, is the path for those countries?

And by the way, these are countries, if you think about it from the point of view of technology and AI, in particular, the AI technological competition globally rapidly seems to come down to be a race between the US, led by the US, but increasingly by China, and others are largely being left behind. That includes in some cases, parts of Europe, but for sure, parts of the poor developing economies.

So the question is, in a future, in which capacity for technology’s developing a different pace, dramatically different paces for different countries, and the nature of globalization itself is changing, what is the path for these poor developing countries? I think that’s a very tough question that we don’t have very many good answers for, by the way.

But there have just been people who think about developing economies in developing economies themselves. That’s one of the tough challenges, I think, for the next several decades of the 21st century.

Lucas Perry: Yeah, I think that this is a really good job of explaining some of these really significant problems. I’m curious what the most significant findings of your own personal work, or the work more broadly being done at McKinsey are, with regards to these problems and issues. I really appreciate some of the figures that you’re able to share. So if you have any more of those, they’re really helpful, I think, for painting a picture of where things are at, and where they’re moving.

James Manyika: Well, I think the only other question on these kind of left behind countries and economies, as I said, these are topics that we’re trying to research and understand. I don’t think we have any kind of pat simple solutions to them.

We do know, though, that in fact, so if you look at the pattern, a lot of our work is very empirical. I mean, typically, I’m looking at what is actually happening on the ground. One of the things that you do see for developing economies is that the developing economies that are part of a regional ecosystem, either because of the value chains and supply chains.

Take the case of a country like Vietnam. It’s kind of in the value chain ecosystem around China, for example. So it benefits from being a participant or an input into the Chinese value chain.

When you have countries, and you could argue that’s what’s happened with countries like Mexico and a few others, so there’s something about being a participant in the value chains or supply chains of these that are emerging somewhat regionally, actually. That seems to be at least one path.

The other path that we’ve seen is that when you’ve got a developing countries that tend to have large and competitive private sectors, and emphasize “competitive,” that actually seems to make a difference. So we did some empirical work where we looked at something like 75 developing countries over the last 25 years, to see what are some of the patterns of which ones are those that have done well, because of their growth and development, and so forth?

Some of the factors that you see, we found in that research, is in fact, when the countries that happened to have, as I said, one is proximity to either all participants in the global value chains of other large ecosystems or economies did well.

Second, those that seem to have these large and vibrant and very competitive private sector economies also seem to do better. Also, those that had resource endowments did well, so that I don’t know, oil and natural resources, and those kinds of things, also seemed to do well.

Then we also find that those that seem to have more mixed economies, so they didn’t just rely on one part of their economy, but they had two or three different kinds of activities going on in their economy, they had maybe a little bit of a manufacturing sector, and a little bit of an agricultural sector, a little bit of a service sector, so the ones that had more mixed economies seem to do well. The other big thing was, the ones that seem to be reforming their economies seem to do well.

So those are some patterns. I don’t think those are guaranteed, in any of them, to be the recipe for the next few decades, partly because much of that picture on global supply chains is changing, and much of the role of technology and how it affects how people participate in the global economy is changing.

I think those are useful, but I don’t know if they’re any short recipe, going forward. There certainly have been the patterns for the last 25 years, but maybe that’s a place to start, if you look forward.

Lucas Perry: To pivot a bit here, I’m curious if you could explain what you see as the central forces that are currently acting on the global economy?

James Manyika: Well, I’ll tell you some of the things that are interesting, that we find interesting. One is, in fact, the fact that more and more and more and more, the role of technology in the global economy is getting bigger and bigger and bigger, in the sense that technology seems to have become way more general purpose in the sense that it’s foundational to every company, every sector and every country.

So the role of that is interesting. It also has these other outsize effects, because we know that technology often lead to the phenomenon of superstar firms and superstar returns, and so forth. You see that quite a bit, so the role of technology is an important one.

The other one that’s going on is what’s happening with globalization itself. And by globalization, I just mean that the movement of value and activity related to the global economy.

We did some work a few years ago, that we’ve tried to update regularly, where we looked at all the things of economic value. So we looked at, for example, the flow of products and goods across the world, the flow of money, finances, and other financing and other things, the flow of services, the movement of people, and even the movement of data, and data-related activities.

What was interesting is that one of the things that has changed is that the globalization in the form of the flow of goods and services, it actually kind of slowed down, actually. That’s why one of the reasons people were questioning is, is globalization dead, has it slowed down?

Well, it certainly looks that way. If you’re looking at it through the lens of the flow of products and goods, but not the case if you’re looking, necessarily, at the flow of money, for example, not necessarily if you’re looking at the flow of people, and for sure not the case, if you’re looking at the flow of data around the world.

One of the things that’s, I think, underappreciated is just how digitized the global economy has become, and just the massive amounts of data flows, digital data flows that now happen across borders between countries, and how much that is tied into globalization works. So if you’re looking at globalization through the lens of digitization, digital data flows, nothing has slowed down. In fact, if anything, it’s accelerated, actually.

That’s why, often, you will hear, people were looking at it through that lens, and say, “Oh no, it’s even more globalized than ever before.” But people who are looking at it through the flow of products and goods, for example, might say, “Oh, it seems, it looks like it is slowed down.” That’s one of the things that’s changing.

Also, the globalization of digital data flows is actually interesting, because one of the things that it does is it changed the participation map quite significantly. So we did some work, where if you look at it through that lens, you suddenly found that you have many more countries participating, and many more kinds of companies participating, as opposed to just a few countries and a few companies participating in the global economy. You have much more diversity of participation.

So you have very tiny companies, a two- or three-person company in some country, plugged into the global economy, using digital technology and digital platforms, in ways that wouldn’t have happened before, if you had a two- or three-person company 30 years ago. So this digitalization of the global economy is really quite fascinating.

The other thing that’s going on to the global economy is the rebalancing, where, with the emergence of China’s a big economy in its own right, that is changing the gravitational structure, if you like, of the global economy in some, in some very profound ways, in ways that we haven’t quite had before. Because, sure, in the past you’ve had other large economies like Germany and Japan and others, as large economies, but none of them were ever as big as the United States.

Also, all of them, whether it’s Japan or Germany or any of the European countries, largely operated in a framework, a globalization and a global framework, that was largely kind of Western-centric in a way. But now you have this very large economy that’s very different, is very, very large, will be the second largest economy in the world. That is quite different, but yet is tied into the, so that gravitational structural shift is very, very important.

Then, of course, the other thing that’s happening is, what’s happening with supply chains and global value chains. And that’s interesting, partly because we’re so intertwined with how supply chains and value chains work, but at the same time, it changes how we think about the resilience of economies. We’ve just seen that during this COVID last year, where, all of a sudden, everybody got concerned about the resilience of our supply chains with respect to, essential products and services like medical supplies and so forth.

I think people are now starting to rethink about how do we think about the structure of the global economy, in terms of these value chains. We should have at some point also mentioned other kinds of technologies that are happening. Because it’s not all AI and digital technologies, as much as I love that, and spend a lot of time on that.

I think other technological developments that are interesting include what’s happening in biosciences or the life sciences. We’ve just seen spectacular demonstrations of that with the MRNA vaccines that were rapidly developed.

But I think a lot more has been happening with just amazing progress, that we’re still at the very early stages of, with regards to the biotechnology and the life sciences. I think we’re going to see even more profound, societally, and profound impact from those developments in the coming decades.

So these are some of the things that I see happening in the global economy. Now, of course, climate change looms large over all of this as a thing that could really impact things in quite dramatic and quite existentially concerning ways.

Lucas Perry: In terms of this global economy, can you explain what the economic center of gravity is, where it’s been, and where it’s going?

James Manyika: Well, undoubtedly, the economic center of gravity has been the United States. If you look at the last 50 years, it’s been the largest economy on the planet, largest in every sense, right? As a market to sell into, as its own market. Everybody around the world for the last 50 years has been thinking about, “How do we access and sell to consumers and buyers in the United States?”

It’s been the largest market. It’s also been the largest developer and deployer of technologies and innovation. So, in all of those ways, it’s been the United States as the big gravitational pull.

But I think, going forward, that’s going to shift, because current course and speed, the Chinese economy will be as large. And you now start to have even other economies becoming large, too, like India.

So I think economic historians have created a wonderful map, where they showed the movement of the gravitational central to the global economy. I think they went back 1,000 years.

While it’s been in the Western Hemisphere, primarily in the United States, I think some of the mid-Atlantic, it’s been shifting east, mostly because of the Chinese economy, primarily, but also India and others that have come to grow. That’s clearly one of the big changes going on at the global economy, to its structure and its center of gravity.

Lucas Perry: With this increase of globalization, how do you see AI as fitting into and affecting globalization?

James Manyika: The impact on globalization? I don’t think that’s the way I would think about the impact of AI. Of course, it’ll affect globalization, because any time you have anything to do with products, goods, and services, because AI is going to effect all of those things.

To the extent that those things are playing out on the global economy landscape, AI will affect those things. I think the impact of AI, at least in my mind, is first and primarily about any economy, whether it’s the global economy or a national economy or a company, right? So I think it’s profoundly going to change many things about how any economic entity works.

Because we know the effect, the capital labor inputs, we know it’ll affect productivity, and we know it’ll change the rates of innovation. Because imagine, in this conversation, at least, we talked, I think mostly about AI’s impact on labor markets, we should not forget AI’s impact on innovation, on productivity, on the kinds of creation of products, goods, and services that we can imagine and how hopefully it’s going to accelerate those developments.

I mean, DeepMind dealing with AlphaFold, which is cracking a 50-year problem, that’s going to lead to all kinds of, hopefully, biomedical innovations and other things. I think one of the big impacts is going to be how AI affects innovation, and ultimately productivity, and the kinds of things we’re going to see, whether it’s products, goods, and services that we’re going to see in the economy.

Of course, any economy that takes advantage of that and embraces those innovations will obviously see the benefit to the growth of their economy. Of course, if on a global scale, on a global stage in the global economy, we have some countries do that more than others, then of course it’ll affect who gets ahead, who’s more competitive, and who potentially gets left behind.

One of the other things we’ve looked at is, what is the rate of AI participation, whether in terms of developments or contributing to developments, or just simply deploying technologies, or having the capacity to deploy technologies, or having the talent and people who can either both contribute to the deployment or the development, and also embrace it in companies and sectors? And you see a picture that’s very different around the world.

Again, you see the US and China, way ahead of everybody, and you’ll see some countries in Europe, and even Europe is not uniform, right, and some countries in Europe doing more of that than others, and then a whole bunch of others who are being left behind. Again, AI will impact the global economy in the sense of how it impacts each of the economies that participate, and each of the companies that participate in the global economy, in their products and services, and their innovations and outputs.

There are other things that’ll play out in the global stage related to AI. But from an economy standpoint, I think I see it through the lens of the participating companies and countries and economies, and how that then plays out on the global stage.

Lucas Perry: There’s also this facet of how this technological innovation and development, for example, with AI, and also technologies which may contribute to and mitigate climate change, all affect global catastrophic and existential risks. So I’m curious how you see global catastrophic and existential risks as potentially fitting into this evolution of the global economy of labor and society as we move forward?

James Manyika: Well, I think it depends whether you’re asking whether AI itself represents a catastrophic or existential risk. Many people have written about this, but I think that question is tied up with the view on how do we think about, how close we are to AI, or how close we are to AGI, and how close we are to superhuman AI capabilities.

As we discussed earlier, I don’t think we’re close yet. But there are other to think about, even as we progress in that direction. This include some of the safety considerations, the control considerations, and how we make sure we’re deploying and using AI safely.

We know that there’s particular problems with regards to things like, even with narrow AI, as it’s sometimes called, how do we think about reward and goal corruption, for example? How do we think about how we avoid the kind of interference, catastrophic interference, between, say, tasks and goals? How do we think about that?

There are all these kinds of safety related things, even on our way to AGI, that we still need to think about. In that sense, these are things to worry about.

I also think we should be thinking about questions of value and goal alignment. And these also get very complicated for a whole bunch of both philosophical reasons, but also quite practical reasons.

That’s why I love the work, for example, that Stuart Russell has been doing on how we think about human compatible AI, and how do we build these kinds of, the value alignment and goal alignment, that we should be thinking about? So these are, even on our way to AGI, both the safety control and these kinds of value alignment, and somewhat normative questions, about how we think about normativity, and what does it even mean, to think about normative things in the case of value alignment with the AI? These are important things.

Now, that’s if you’re thinking about catastrophic, or at least to existential risk, with regards to AI, even way before you get to AGI. Then you have the kinds of things, at that point, that Nick Bostrom and others have worried about.

I think, because those are non-zero probability concerns, we should invest all effort into working on those existential, potentially catastrophic problems. But I’m not super worried about those any time soon, but that doesn’t mean we shouldn’t invest and work on those, the kinds of concerns that Nick and others write about.

But they are also questions about AI governance, in the sense of we’re going to have many participating entities here. We’re going to have the companies that are leading the development of these technologies.

We’re going to have governments that are going to want to participate and use these technologies. We’re going to have issues around when to deploy these technologies, use and misuse. Many of these questions become particularly important when you think about the deployment of AI, especially in particular arenas.

Imagine if, once we have AI or AGI that’s capable of manipulation, for example, or persuasion, or those kinds of things, or capabilities that allow us to detect lies, or be able to interfere or play with signals’ intelligence, or even cryptography and number theory. I mean, our cryptographic systems rely on a lot of things in prime number theory, for example, or if you think about arenas like autonomous weapons.

So questions of governance become evermore important. I mean, they’re already important now, when we think about how AI may or may not be used for things like deep fakes and disinformation.

The closer we get to the kinds of areas that I was describing here, it becomes even more important to think about governance, and what’s permissible to deploy where and how, and how do we do that in a transparent way? And how do we deal with the challenges with AI about attribution?

One of the nice things about other potentially risky technologies or developments like nuclear science, or chemical weapons, and so forth, is at least those things, they’re easy to detect when they happen. And it’s relatively easy to do attribution, and verify that it happened, and it was used.

It’s much harder with the AI systems, so these questions of governance and so forth become monumentally important. So those are some of the things we should think about.

Lucas Perry: How do you see the way in which the climate change crisis arose, given human systems and human civilization? What is it about human civilizations and human systems that has led to the climate change crisis? And how do we not allow our systems to function and fail in the same way, with regards to AI and powerful technologies in the 21st century?

James Manyika: I’m not an expert on climate science, by the way. So I shouldn’t speculate as to how we got to where we are. But I think the way we’ve used certain technologies and fossil fuels, I think, is a big part of that, the way our economies have relied on that as our only mode of energy, is part of that.

The fact that we’ve done that in a relatively costless way, in terms of pricing the effects on our environment and our climate, I think, is a big part of it. The fact that we haven’t had very many as effective and as efficient alternatives, historically, I think, is a big part of that.

So I think all of that is part of how we got here in some ways, but I think others more expert than me can talk about that. I think if I think about AI, I think one of the things that is potentially challenging about AI, if, in fact, we think there’s a chance that we’ll get to these superhuman capabilities, and AGI, is that we may not have the opportunity to iterate our way there. Right?

I think, quite often, with a lot of these deployment of technologies, I think a practical thing that has served us well in the past has been this idea that, well, let’s try a few experiments, we’ll fix it if it fails. Or if it doesn’t work, and we’ll iterate and do better, and kind of iterate our way to the right answer. Well, if we believe that there is a real possibility of achieving AGI, we may not have the opportunity to iterate in that same way.

That’s one of the things that’s potentially different, perhaps, because we can’t undo that, as it were, if we truly get to AGI. Thinking about these existential things, so there’s maybe something of a similarity, or at least an analog with climate change, is that we can’t just undo what we’ve done, in a very simple fashion, right?

Look at how we’re now thinking about, how do we do carbon sequestration? How do we take carbon out of this, out of the air? How do we undo these things? And it’s very hard. It’s easy to go in one direction, it’s very hard to go in the other direction, in that sense, at least.

It’s always dangerous with the analysis. But at least in that sense, AI, on its way to AGI, may be similar in that sense, which is we can’t always quite get to undo it in a simple fashion.

Lucas Perry: Are you concerned or worried that we will use AI technology in the way that we’ve used fossil fuel technologies, such that we don’t factor in the negative effects or negative externalities of the use of that technology? With AI, there’s this deployment of single objective maximizing algorithms that don’t take account all of our other values, and that actually run over and increase human suffering.

For example, the ways in which YouTube or Facebook algorithms work to manipulate and capture attention. Do you have a concern that our society has a natural proclivity towards learning from mistakes, from ignoring negative externalities, until it reaches sort of a critical threshold?

James Manyika: I do worry about that. And then maybe, just to come back to one of your, I think, central concerns, back to the idea of incentives, I do worry about that, in the sense that there are going to be such overwhelming and compelling incentives to deploy AI systems, for both good reasons, and for the economic reasons that go with that. So there are lots of good reasons to deploy AI technology, right?

It’s actually great technology. Look at what it’s probably going to do to, in the case of health science, and breakthroughs we could make there in climate science itself, and scientific discovery and material science. So there’s lots of great reasons to get excited about AI.

And I am, because it’ll help us solve many, many problems, could create enormous bounty and benefits for our society. So we’re going to, people are going to be racing ahead to do that, for those reasons, for those very good and very compelling reasons.

There are also going to be a lot of very compelling economic reasons. The kinds of innovations that companies can make, the kind of contributions to the economic performance of companies, the kinds of economic benefits, and that possibly that AI will contribute to productivity growth as we talked about before.

There’s lots of reasons to want to go full steam ahead. And a lot of incentives will be aligned to encourage that, both the breakthrough innovations that are good for society. As I said, the benefits that companies will get from deploying and using AI until the innovations, the economy-wide productivity benefits, so, all good reasons.

And I think, in the rush to do that, we may in fact find that we’re not paying enough attention, not because anybody is out of malice or anything like that, but we just may not be paying enough attention to these other considerations that we should have alongside, considerations about, what does this mean for bias and fairness?

What does it mean for, potentially for inequality? We know these things have scale superstar effects. What does that mean for others who get left behind? What does this mean for the labor markets and jobs and so forth? So I think we’re going to need to find mechanisms to make sure that there’s continued, but substantial effort, at those kind of other sides of the side effects of AI, and some of the unintended consequences.

That’s why, at least, I think many of us are trying to think about this question, “What are the things we have to get right,” even as we race towards all the wonderful things we want to get out of it, what are the other things we need to make sure we’re getting right along the way?

How do we make sure these things… People are working on them, they’re funded, there’s support for people working on these other problems. I think that’s going to be quite important, and we should not lose sight of that. And that’s something I’m concerned about.

Lucas Perry: So let’s pivot here, then, into inequality and bias. Could you explain the risk and degree to which AI may contribute to new inequalities, or exacerbate existing inequalities?

James Manyika: Well, I think on the inequality point, it’s part of what we talked about before, right? Which is the fact that, even though we may not lose jobs in the near term, we may end up with creating jobs or complementing jobs in a way that have these wage effects, that could worsen the inequality question.

That’s one way in which AI could contribute to attain equality. The other way, of course, is the fact that because of the scale effects of these technologies, you could end up with a few companies or a few entities or a few countries having the ability to develop and deploy, and get the benefits of AI, while the other companies or countries and places don’t. So you’ve got that kind of inequality concern.

Now, some of that could be helped by the way as it is, because it was the case, it has been the case so far, that the kind of compute capacity needed to develop and deploy AI has been very, very large, and the data endowments needed to train algorithms has been very, very high, but we know the talent of people who are working on these things has been, up until now, relatively concentrated.

But we know that that picture’s changing, I think. The advent of cloud computing, which makes it easy for those who don’t have the compute capacity, is helping that. The fact that we now have ways to train algorithms, of pre-trained algorithms or other universal models and others, so that not everybody has to retrain everything every single time.

These scarcities and these kind of scale constraints, I think, in those particular ones, will get better as we go forward. But you do worry about those inequalities, both in a peoples sense, but also in a entity sense, where entities could be companies, countries, or whole economies.

I think the questions of bias are a little bit different. I think the set of questions of biases, just simply has to do with the fact that up until now, at least so far, anyway, most of the data sets that have been used to train these algorithms often come with societally derived biases. And I emphasize this, society derive bias. It’s just because of the way we collect data and the data that’s available and who’s contributing to it.

Often, you start out with data sets, training data sets that reflect society’s existing biases. Not that the technology itself has introduced the bias, but in fact, these come out of society. So what the technologies then do is kind of bake these biases in, into the algorithms and probably deploy them at scale.

That’s why I think this question bias is so important, but I think often it gets conflated with the fact that, well, proponents of using these technologies will say, but humans already have bias in them, anyway. We already make biased decisions, et cetera.

Of course, that’s a two-sided conversation. But at least to the case, the difference that I see between the biases we have already as human beings, versus the biases that could get baked into these systems, is that these systems could get deployed and scale in a way that, if I have biases that I have, and I’m in a room and I’m trying to hire somebody, and I’m making my biased decisions, at least, hopefully that only affects that one hiring decision.

But if I’m using an algorithm that has all these things baked in, and hundreds of millions of people are using the algorithm, then we’re kind of doing that in scale. So I think we need to keep that in mind, as we have the debate about, people already have biases and saturated biases, that’s true. So we need to do work on that.

But one of the things I like about the bias question, by the way, that these technologies are forcing us to confront is that it’s actually forcing us to really think about, what do we even mean when we say things are fair, quite aside from technology?

I think they’re forcing us, just like the UBI debate is forcing us to confront the question that people don’t earn enough to live, the bias question’s also forcing us to confront the question of, what is fair right? What counts as fairness? And I think all too often, in our society, we’ve tended to rely on proxies for fairness, right?

When we define it, we’ll say, “Well, let’s constitute the right group of people, a diverse enough group of people, and we will trust the decision that they make, because it’s a diverse group of people,” right? So yeah, if that group is diverse in the way we expect, then gender or racial or any other social income terms, and they make a decision, we’ll trust it, because the deciding group is diverse.

That’s just a fairness by proxy, in my view. Who knows what those people actually think, and how to make decisions? That’s a whole separate matter, but we trust it, because it’s a diverse group.

The other thing that we’ve tended to rely on is, we trust the process, right? If we trust the process that, “Hey, if it’s gone through a process like this, we will live with the results, because we think that the process like that is fair and unbiased.”

Who knows whether the process is actually fair, and that’s how we’ve typically done it with our legal system, for the most part. That if you follow through, if you’ve been given due process and you’ve gone through a jury trial, then it must be fair. We will live with the results.

But I think, in all of those cases, while they’re useful constructs for us in society, they still somewhat avoid defining what is actually fair. And I think, when we’ve started to deploy technologies, where, in the cases of AI, the process is somewhat opaque, because we have this kind of explainability challenge of these technologies. So the process is kind of black boxy, in that sense.

And if we automate the decisions with no humans involved, then we can’t rely on this constituent group that, “Hey, the group of people decided this, so it must be fair.” This is forcing us to come back to the age-old or even millennia-old question of what is fair? How do we define fairness?

I think there’s some work that was done before, where somebody is trying to come up with all kinds of definitions of fairness, and they came up with something like 21. So I think we now are having an interesting conversation about what constitutes fairness. Do we gather data differently? Do we code differently? Do we have reviews differently? Do we have different people that develop the technologies differently? Do we have different participants.

So we’re still grappling with this question, what counts as fair? I think that’s one of the key questions, as we rely more and more on these technologies to assist, in some cases, eventually take over some of our decision-making, of course, only when it’s appropriate, these questions will continue to persist, and will only grow, on how we think about fairness and bias.

Lucas Perry: In terms of fairness, bias, equality, and beneficial outcomes with technology and AI in the 21st century, how do you view the need for and path to integrating developing countries’ voices in the use and deployment of AI systems?

James Manyika: Well, I don’t know if there’s any magical answers, Lucas. At some level, at a base level, we should have them participate, right? I think any participation, both in the development and deployment, I think, is going to be important. And I think that’s true for developing countries. I think it’s true for parts of even US society that’s often not participating in these things.

I mean, it’s still striking to me how the lack of diversity, and diversity, in every sense of the term, who is developing AI and who’s deploying AI, whether they look within the United States or around the world, there are entities and places and communities and whole countries that are not really part of this. So I think we’re going to need to find a ways to do that.

I think part of doing that is at least for me, it starts out with the recognition that capabilities and intelligence are equally distributed everywhere. I don’t think there’s any one place or country or community that has a natural advantage to capability and intelligence.

On that premise, we just need to get people from different places participating in the development and deployment, and even the decision-making that’s related to AI, and not just go with the places where the money and the resources happen to be, and that’s, who’s racing ahead, both within countries, e.g., in the United States itself, or in other countries that are being left behind. I think participation, in these different ways, I think it’s going to be quite, quite important.

Lucas Perry: If there’s anything you’d like to leave the audience with, in terms of perspective on the 21st century on economic development and technology, what is it that you would share as a takeaway?

James Manyika: Well, I think, when I look ahead to the 21st century, I’m in two minds. On the one hand, I’m actually incredibly excited about the possibilities. I think we’re just at the beginning of what these technologies, both in AI and so forth, but also in the life sciences and biotech, I think that the possibilities in the 21st century are going to be enormous, possibilities for both improving human life, improving economic prosperity, growing economies.

The opportunities are just enormous, whether you’re a company, whether you’re a country, whether you’re a society, the possibilities are just enormous. I think there’s more that lies ahead than behind.

At the same time, though, I think, alongside pursuit of those opportunities are the really complicated challenges we’re going to need to navigate through, right? Even as we pursue the opportunities that AI and these technologies are going to bring us, we’re going to need to pay attention to some of these challenges that we just talked about, these questions of potential inequality and bias that comes out of the deployment of these technologies, or some of the superpower effects that could come out of that, even as we pursue economic opportunities around the world, we’re going to need to think about what happens to poor developing countries who may not keep up with that, or be part of that.

In every case, for all the things that I’m excited about the 21st century, which is plenty, there are also these challenges along the way we’re going to need to deal with. Also the fact that society, I think, demands more from all of us.

I think the demands for a more equal and just society are only going to grow. The demands or desires to have a more inclusive and participative economy are only going to grow, as they should. So we’re going to need to be working both sets of problems, pursuing the opportunities, because without them, these other problems only get harder, by the way.

I mean, try to solve the inequality when there’s no economic surpluses, right? Good luck with that. So we have to solve both. We can’t pick one side or the other, we have to solve both. At the same time, I think we also need to deal with some of the potentially existential challenges that we have, and may grow. I mean, we are living through one right now.

I mean, we’re going to have more pandemics in the future than we have had, perhaps, in the past. So we’re just going to need to be ready for that. We’ve got to deal with climate change. And these kinds of public health, climate change issues, I think, are global. They’re for all of us.

These are not challenges for any one country or any one community. We have to kind of work on all of these together. So that set of challenges, I think, is for everybody, for all of us. It’s on planet Earth, so we’re going to need to work on those things too. So that’s kind of how I think about what lies ahead.

We have to pursue the opportunities, there’s tons of them. I’m very excited about that. We have to solve the challenges that come along with questioning those opportunities, and we have to deal with these collective challenges that we have. I think those are all things to look forward to.

Lucas Perry: Wonderful, James, thank you so much. It’s really interesting and perspective shifting. If any of the audience is interested in following you or checking your workout anywhere, what are the best places to do that?

James Manyika: If you search my name and such McKinsey Global Institute, you will see some of the research and papers that I referenced. For those who love data, which I do, these are very data rich fact-based perspectives. So just look at the McKinsey Global Institute website.

Lucas Perry: All right. Thank you very much, James.

James Manyika: Oh, you’re welcome. Thank you.

Lucas Perry: Thanks for joining us. If you found this podcast interesting or useful, consider sharing it on social media with friends, and subscribing on your preferred podcasting platform. We’ll be back again soon, with another episode in the FLI Podcast.

Michael Klare on the Pentagon’s view of Climate Change and the Risks of State Collapse

  • How the US military views and takes action on climate change
  • Examples of existing climate related difficulties and what they tell us about the future
  • Threat multiplication from climate change
  • The risks of climate change catalyzed nuclear war and major conflict
  • The melting of the Arctic and the geopolitical situation which arises from that
  • Messaging on climate change

Watch the video version of this episode here

See here for information on the Podcast Producer position

Check out Michael’s website here


Lucas Perry: Welcome to the Future of Life Institute Podcast. I’m Lucas Perry. Today’s episode is with Michael Klare and explores how the US Department of Defense views and take action on climate change. This conversation is primarily centered around Michael’s book, All Hell Breaking Loose. In both this podcast and his book, Michael does an excellent job of making clear how climate change will affect global stability and functioning in our lifetimes through tons of examples of recent climate induced instabilities. I was also surprised to learn that despite changes in administrations, the DoD continued to pursue climate change mitigation efforts despite the Trump administration’s actions to remove mention and activism on climate change from the federal government. So, if you’ve ever had any doubts or if the impact of climate change and it’s significance has ever or does feel fuzzy or vague to you, this podcast might remedy that.

 I’d also like to make a final call for applications for the Podcast Producer role. If you missed it, we’re currently hiring for a Podcast Producer to work on the editing, production, publishing, and analytics tracking of the audio and visual content of this podcast. As the Producer you would be working directly with me, and the FLI outreach team, to help grow, and evolve this podcast. If you’re interested in applying, head over to the Careers tab on the Futureoflife.org homepage or follow the link in the description. The application deadline is July 31st, with rolling applications accepted thereafter until the role is filled. If you have any questions, feel free to reach out to socialmedia@futureoflife.org.

Michael Klare is a Five Colleges professor of Peace and World Security Studies. He serves on the board of directors of the Arms Control Association, and is a regular contributor to many publications including The Nation, TomDispatch and Mother Jones, and is a frequent columnist for Foreign Policy In Focus. Klare has written fourteen books and hundreds of essays on issues of war and peace, resource competition, and international affairs. You can check his work at Michaelklare.com. And with that I’m happy to present this interview with Michael Klare.

So to start things off here, I’m curious if you could explain at the highest level, how is it that the Pentagon views climate change and why is the Pentagon interested in climate change?

Michael Klare: So, if you speak to people in the military, they will tell you over and over again that their top concern is China. China, China, China followed by Russia and then maybe North Korea and Iran, and they spend their days preparing for war with China and those other countries. Climate change intercedes into this conversation because ultimately they believe that climate change is going to degrade their capacity to prepare for and to fight China and other adversaries down the road, that climate change is a complicating factor, a distraction that will undermine their ability to perform their military duties, and moreover, they see that the threat posed by climate change is increasing exponentially over time. So, the more they look into the future, the more they see that climate change will degrade their ability to carry out what they see as their primary function, which is to prepare for war with China. And so, it’s in that sense that climate change is critical. Now, then you go down in the specific ways in which climate change is a problem, but it’s ultimately because it will distract them from doing what they see as their primary responsibility.

Lucas Perry: I see, so there’s a belief in the validity of it and the way in which it will basically exacerbate existing difficulties and make achieving more important objectives more difficult.

Michael Klare: Something like that. Climate change they see as an intrusion into their work space. They’re trained as soldiers to carry out their military duties, which is combat related, and they believe that climate change is very real and getting more intense as time goes on and it’s going to hold them back, intrude on their ability to carry out their combat functions. It’s going to be a distraction on multiple levels. It’s going to create new kinds of conflicts that they would rather not deal with. It’s going to create emergencies around the world, humanitarian disasters at home and abroad, all of these are going to suck away resources, time, effort, energy, money that they believe should be devoted to their primary function of preparing for war with major enemies.

Lucas Perry: What would you say the primary interests of the Pentagon are right now other than climate change?

Michael Klare: Other than climate change, well the US Department of Defense at this time has a number of crises going on simultaneously. In addition to climate change, there’s COVID of course. Like every other institution in US society, the military was hampered by COVID, many service people came down with COVID and some died and it forced military operations to be restricted. Ships had to be brought back to port because COVID broke out on ships, so that was a problem. The military is also addressing issues of racism and extremism in the ranks. That’s become a major problem right now that they are dealing with, but they view climate change as the leading threat to national security of a non-military nature.

Lucas Perry: So, China was one of the first things that you mentioned. How would you also rank and relate the space of their considerations like Russia and a nuclear North Korea and Iran?

Michael Klare: Sure, the Department of Defense just released their budget for fiscal year 2022, and they rank the military threats and they say China is overwhelmingly the number one threat to US national security followed by Russia, followed by North Korea and Iran, and then down the list would be terrorist threats like Al-Qaeda and ISIS. But as you know, the administration has made a decision to leave Afghanistan and to downgrade US forces in that part of the world, so fighting terrorism and insurgency has been demoted as a major threat to US security, and even Russia has been demoted to second place. Over the past few years, Russia and China have been equated, but now China has been pushed ahead as the number one threat. The term they use is the pacing threat, which is to say that because China’s the number one threat, we have to meet that threat and if we can overcome China, the US could overcome any other threat that might come along, but China is number one.

Lucas Perry: So, there’s this sense of top concerns that the Department of Defense has, and then this is all happening in a context of climate change, which makes achieving its objectives on each of these threats more and more difficult. So, in the context of this interplay, can you describe the objectives of career officers at the Pentagon and how it’s related to and important for how they consider and work with climate change?

Michael Klare: Sure, so if you’re an aspiring general or admiral right now, as I say, you’re going to be talking about how you’re preparing your units, your division, your ship, your air squadron to be better prepared to fight China, but you also have to worry about what they call the operating environment, the OE, the operating environment in which your forces are going to be operating in, and if you’re going to be operating in the Pacific, which means dealing with China, then you have a whole set of worries that emerges. We have allies there that we count on: Japan, South Korea, the Philippines.

These countries are highly vulnerable to the effects of climate change and are becoming more so very rapidly. Moreover, we have bases in those places. Most of those bases, air bases and naval bases are at sea level or very close to sea level and are over and over again have been assaulted by powerful typhoons and have been disrupted, have had to be shut down for days or weeks at a time, and some of those bases like Diego Garcia in the Indian Ocean for example, or the Marianas Islands are not going to be viable much longer because they’re so close to sea level and sea level rise is just going to come and swamp them. So from an operating environment point of view, you have to be very aware of the impacts of climate change on the space in which you’re going to operate.

Lucas Perry: So, it seems like the concerns and objectives of career officers at the Pentagon can be distinguished in significant ways from the perspective and position of politicians, so there’s like some tension at least between career officers or the objectives of the Pentagon in relation to how some constituencies of the American political parties are skeptical of climate change?

Michael Klare: Yes, this was certainly the case during the Trump administration because the commander in chief, as one of his titles, President Trump forbad the discussion of climate change, and he was a denier. He called it a hoax and he forbad any conversation of that. So, the US military did have a position on climate change during the Obama administration. It had as early as 2010 the Department of Defense stated that climate change posed a serious threat to US security and was taking steps to address that threat. So when Trump came along, all of that had to go underground. It didn’t stop, but the Pentagon had to develop a whole lot of euphemisms, like changing climate or extreme weather events, all kinds of euphemisms used to describe what they saw as climate change, but that didn’t stop them from facing the consequences of climate change. During the Trump administration, US military bases in the US suffered billions and billions of dollars of damage from Hurricane Michael, from others that hit the East Coast and the Gulf of Mexico that did tremendous damage to a number of key US bases.

And, the military is still having to find the money to pay for that damage, and the Navy in particular is deeply concerned that its major operating bases in the United States… A Navy base by definition is going to be at the ocean, and many of these bases are of very low lying areas and already are being repeatedly flooded at very high tides, or when there are storms and the Navy is very aware that their ability to carry out their missions to reinforce American forces either in the Atlantic or Pacific are at risk because of rising seas, and they had to maneuver around Trump all during that period, trying to protect their bases, but calling it by different names, calling the danger they faced by different names.

Lucas Perry: Right, so there’s this sense of Trump essentially canceling mention of climate change throughout the federal government and its branches and the Pentagon responding by quietly still responding to what they see as a real threat. Is there anything else you’d like to add here about the Obama to Trump transition that helps to really paint the picture of how the Pentagon views climate change and what it did despite attempts to suppress thought and action around climate change?

Michael Klare: During the Obama administration, as I say, the Department of Defense acknowledged the reality of climate change number one. Number two said it posed a threat to US national security, and as a result said that the Department of Defense had an obligation to reduce its contribution to climate change to reduce its emissions and made all kinds of pledges that it was going to reduce its consumption of fossil fuels and increase its reliance on renewable energy, begin constructing solar arrays. A lot of very ambitious goals were announced in the Obama period, and although all of this was supposed to stop when Trump came into office because he said we’re not going to do anymore any of this anymore. In fact, the Pentagon continued to proceed with a lot of these endeavors, which were meant to mitigate climate change, but again, using different terminology that this was about base reliance, self-reliance, resiliency, and so on, not mentioning climate change, but nonetheless continued to proceed with efforts to actually mitigate their impact on climate.

Lucas Perry: All right, so is there any sense in which the Pentagon’s view of climate change is unique? And, could you also explain how it’s important and relevant for climate change and also the outcomes related to climate change?

Michael Klare: Yes, I think the Pentagon’s view of climate change, I think, is very distinctive and not well understood by the American public, and that’s why I think it’s so important, and that is that the Department of Defense sees climate change as… The term they use is as a threat multiplier. They say, look, we look out at the world and part of our job is to forecast ahead of time where our threat’s going to emerge to US security around the world. That’s our job, and to prepare for those threats, and we see that climate change is going to multiply threats in areas of the world that are already unstable, that are already suffering from scarcities of resources, where populations are divided and where resources are scarce and contested, and that this is going to create a multitude of new challenges for the United States and its allies around the world.

So, this notion of a threat multiplier is very much a part of the Pentagon’s understanding of climate change. What they mean by that is that societies are vulnerable in many ways and especially societies that are divided along ethnic and religious and racial lines as so many societies are, and if resources are scarce, housing, water, food, jobs, whatever, climate change is going to exacerbate these divisions within societies, including American society for that matter, but it’s going to exacerbate divisions around the world and it’s going to create a social breakdown and state collapse. And, the consequence of state collapse could include increased pandemics for example, and contribute to the spread of disease. It’s going to lead to mass migrations and mass migrations are going to become a growing problem for the US.

The influx of migrants on America’s Southern border, many of these people today are coming from Central America and from an area that’s suffering from extreme drought and where crop failure has become widespread, and people can’t earn an income and they’re fleeing to the United States in desperation. Well, this is something the military has been studying and talking about for a long time as a consequence of climate change, as an example of the ways in which climate change is going to multiply schisms in society and threats of all kinds that ultimately will endanger the United States, but it’s going to fall on their shoulders to cope with and creating humanitarian disasters and migratory problems.

And as I say, this is not what they view as their primary responsibility. They want to prepare for high-tech warfare with China and Russia, and they see all of this as a tremendous distraction, which will undermine their ability to defend the United States against its primary adversaries. So, it’s multiplying the threats and dangers to the United States on multiple levels including, and we have to talk about this, threats to the homeland itself.

Lucas Perry: I think one thing you do really well in your book is you give a lot of examples of natural disasters that have occurred recently, which will only increase with the existence of climate change as well as areas which are already experiencing climate change, and you give lots of examples about how that increases stress in the region. Before we move on to those examples, I just want to more clearly lay out all the ways in which climate change just makes everything worse. So, there’s the sense in which it stresses everything that is already stressed. Everything basically becomes more difficult and challenging, and so you mentioned things like mass migration, the increase of disease and pandemics, the increase of terrorism in destabilized regions, states may begin to collapse. There is, again, this idea of threat multiplication, so everything that’s already bad gets worse.

Lucas Perry: There’s loss of food, water, and shelter instability. There’s an increase in natural disasters from more and more extreme weather. This all leads to more resource competition and also energy crises as rivers dry up and electric dams stop working and the energy grid gets taxed more and more due to the extreme weather. So, is there anything else that you’d like to add here in terms of the specific ways in which things get worse and worse from the factor of threat multiplication?

Michael Klare: Then, you start getting kind of specific about particular places that could be affected, and the Pentagon would say, well this is first going to happen in the most vulnerable societies, poor countries, Central America, North Africa, places like that where society is already divided, poor, and the capacity to cope with disaster is very low. So, climate change will come along and conditions will deteriorate, and the state is unable to cope and you have breakdown and you have these migrations, but they also worry that as time goes on and climate change intensifies, that a bigger and bigger or richer and richer and more important states will begin to disintegrate, and some of these states are very important to US security and some of them have nuclear weapons, and then you have really serious dangers. For example, they worry a great deal about Pakistan.

Pakistan is a nuclear armed country. It’s also deeply divided along ethnic and religious lines, and it has multiple vulnerabilities to climate change. It goes between extremes of water scarcity, which will increase as the Himalayan glaciers disappear, but also we know that monsoons are likely to become more erratic and more destructive with more flooding.

All of these pose great threats to the ability of Pakistan’s government and society to cope with all of its internal divisions, which are already severe to begin with, and what happens when Pakistan experiences a state collapse and nuclear weapons begin to disappear into the hands of the Taliban or to forces close to the Taliban, then you have a level of worry and concern much greater than anything we’ve been talking before, and this is something that the Pentagon has started to worry about and to develop contingency plans for. And, there are other examples of this level of potential threat arising from bigger and more powerful states disintegrating. Saudi Arabia is at risk, Nigeria is at risk, the Philippines, a major ally in the Pacific is at extreme risk from rising waters and extreme storms, and I can continue, but from a strategic point of view, this starts getting very worrisome for the Department of Defense.

Lucas Perry: Could you also paint a little bit of a picture of how climate change will exacerbate the conditions between Pakistan, India, and China, especially given that they’re all nuclear weapon states?

Michael Klare: Absolutely, and this all goes back to water and many of us view water scarcity as the greatest danger arising from climate change in many parts of the world. In the case of India, China, Pakistan, not to mention a whole host of other countries depend very heavily on rivers that originate in the Himalayan mountains and draw a fair percentage of their water from the melting of the Himalayan glaciers and these glaciers are disappearing at a very rapid rate and are expected to lose a very large percentage of their mass by the end of this century due to warming temperatures.

And, this means that these critical rivers that are shared by these countries, the Indus River shared by India and Pakistan, the Brahmaputra River shared by India and China, these rivers, which provide the water for irrigation for hundreds of millions of people if not billions of people, depend on these rivers, the Mekong is another. As the water supply begins to diminish, this is going to exacerbate border disputes. All of these countries, Indian and China, Indian and Pakistan have border and territorial disputes. They have very restive agricultural populations to start with, that water scarcity is going to be the tipping point that will produce massive local violence that will lead to conflict between these countries, all of them nuclear armed.

Lucas Perry: So, to paint a little bit more of a picture of these historical examples of states essentially failing to be able to respond to climate events and the kind of destructive force that was to society and to the status of humanitarian conditions and the increasing need for humanitarian operations, so can you describe what happened in Tacloban for example, as well as what is going on in the Nigerian region?

Michael Klare: So, Tacloban is a major city on the island of Leyte in the Philippines, and it was a direct hit. It suffered a direct hit from Typhoon Haiyan in 2013. This was the most powerful typhoon to make landfall up until that point, an extremely powerful storm that created millions of homeless in the Philippines. Many people perished, but Tacloban was at the forefront of this. A city of several hundred thousand, many poor people living in low lying areas at the forefront of the storm. The storm surge was 10 or 20 feet high. That just over overwhelmed these low lying shanty towns, flooded them. Thousands of people died right away. The entire infrastructure of the city collapsed was destroyed, hospitals, everything. Food ran out, water ran out, and there was an element of despair and chaos. The Philippine government proved incapable of doing anything.

And, President Obama ordered the US Pacific Command to provide emergency assistance, and it sent almost the entire US Pacific fleet to Tacloban to provide emergency assistance on the scale of a major war, aircraft carrier, dozens of warships, hundreds of planes, thousands of troops to provide emergency assistance. Now, it was a wonderful sign of US aid. There are a number of elements of this crisis that are worthy of mention. In addition to all of this, one was the fact that there was anti-government rioting because of the failure of the local authorities to provide assistance or to provide it only to wealthy people in the town, and this is so often a characteristic of these disasters that assistance is not provided equitably, and the same thing was seen with Hurricane Katrina in New Orleans and this then becomes a new source of conflict.

When a disaster occurs and you do not have equitable emergency response, and some people are denied help and others are provided assistance, you’re setting the stage for future conflicts and anti-government violence, which is what happened in Tacloban And the US military had to intercede to calm things down, and this is something that has altered US thinking about humanitarian assistance because now they understand that it’s not just going to be handing out food and water, it’s also going to mean playing the role of a local government and providing police assistance and mediating disputes and providing law and order, not just in foreign countries, but in the United States itself and this proved to be the case in Houston with Hurricane Harvey in 2017 and in Puerto Rico with Hurricane Maria when local authorities simply disappeared or broke down and the military had to step in and play the role of government, which comes back to what I’ve been saying all along. From the military’s point of view, this is not what they were trained to do.

This is not what they want to do, and they view this as a distraction from their primary military function. So, here’s the Pacific fleet engaging in this very complex emergency in the Philippines, and what if there were a crisis with China that were to break out? The whole force would have been immobilized at that time, and this is the kind of worry that they have that climate change is going to create these complex emergencies they call them, or complex disasters that are going to require not just a quick in and out kind of situation, but a permanent or semi-permanent involvement in a disaster area and to provide services for which the military is not adequately prepared, but they see that climate change increasingly will force them to play this kind of role and thereby distracting them from what they see as their more important mission.

Lucas Perry: Right, so there’s this sense of the military increasingly being deployed in areas to provide humanitarian assistance. It’s obvious why that would be important and needed domestically in the United States and its territories. Can you explain why the military is incentivized or interested in providing global humanitarian assistance?

Michael Klare: This has always been part of American foreign policy, American diplomacy, winning friends, winning over friends and allies. So, it’s partly to make the United States look good particularly when other countries are not capable of doing that. We’re the one country that has that kind of global naval capacity to go anywhere and do that sort of thing. So, it’s a little bit a matter of showing off our capacity, but it’s also in the case of the Philippines, the Philippines plays a strategic role in US planning for conflict in the Pacific.

It is seen as a valuable ally in any future conflict with China and therefore its stability matters to the United States and the cooperation of the Philippine government is considered important and access to bases in the Philippines, for example, is considered important to the US. So, the fact that key allies of the US in the Pacific, in the Middle East and Europe are at risk of collapsing due to climate change poses a threat to the whole strategic planning of the US, which is to fight wars over there, in the forward area of operations off the coast of China, or off of Russian territory. So, we are very reliant on the stability and the capacity of key allies in these areas. So, providing humanitarian assistance and disaster relief is a part of a larger strategy of reliance on key allies in strategic parts of the world.

Lucas Perry: Can you also explain the conditions in Nigeria and how climate change has exacerbated those conditions and how this fits into the Pentagon’s perspective and interest in the issue?

Michael Klare: So, Nigeria is another country that has strategic significance for the US, not perhaps on the same scale as say Pakistan or Japan, but still important. Nigeria is a leading oil producer, not as important as it once was perhaps, but nonetheless important, but Nigeria is also a key player in peacekeeping operations throughout Africa and because the US doesn’t want to play that role itself, it relies on Nigeria for peacekeeping troops in many parts of Africa. And, Nigeria occupies a key piece of territory in Central Africa, which is it’s surrounded by countries, which are much more fragile and are threatened by terrorist organizations. So, Nigeria’s stability is very important in this larger picture, and in fact Nigeria itself is at risk from terrorist movements, especially Boko Haram and splinter groups, which continue to wreak havoc in Northern Nigeria despite years of effort by the Nigerian government to crush Boko Haram, it’s still a powerful force.

And, partly this is due to climate change. The Boko Haram operates in areas around Lake Chad, which is now a small sliver of what it once was. It has greatly diminished in size because of global warming and water mismanagement. And so, the farmers and fisher folk whose livelihood depended on Lake Chad has all been decimated. Many of them have become impoverished. The Nigerian government has proved inept and incapable of providing for their needs, and many of these people have therefore fallen prey to the appeals of recruitment by Boko Haram, young men without jobs. So, climate change is facilitating, is fueling the persistence of groups like Boko Haram and other terrorist groups in Nigeria, but that’s only part of the picture. There’s also growing conflict between pastoralists, these are herders, cattle herders whose lands are being devastated by desertification.

In this Sahel region, the southern fringe of the Sahara is expanding with climate change and driving these pastoralists into areas occupied by… These are all Muslim, the pastoralists are primarily Muslims and they’re moving into lands occupied by Christians, mainly Christian farmers, and there’s been terrible violence in the past few years, many hundreds of thousands of people displaced. Again, inept Nigerian response, and so I could go on. There’s violence in the Nigeria Delta region, the Niger Delta area in the south and in the area, their breakaway provinces. So, Nigeria is at permanent risk of breaking apart, and the US provides a lot of military aid to Nigeria and provides training. So, the US is involved in this country and faces a possibility of greater disequilibrium and greater US involvement.

Lucas Perry: Right, so I think this does a really good job of painting the picture of this factor of threat multiplication from climate change. So, climate change makes getting food, water, and shelter more difficult. There’s more extreme weather, which makes those things more difficult, which increases instability, and for places that are already not that stable, they get a lot more unstable and then states begin to collapse and you get terrorism, and then you get mass migration, and then there’s more disease spreading, so you get conditions for increased pandemics. Whether it’s in Nigeria or Pakistan and India or the Philippines or the United States and China and Russia, everything just keeps getting worse and worse and more difficult and challenging with climate change. So, could you describe the ladder of escalation of climate change related issues for the military and how that fits into all this?

Michael Klare: Well, now this is an expression that I made up to try to put this in some kind of context, drawing on the ladder of escalation from the nuclear era when the military talked about the escalation conflict from a skirmish to a small war, to a big war, to the first use of nuclear weapons, to all out nuclear war. That was the ladder of escalation of the nuclear age, and what I see happening is something of a similar nature where at present we’re still dealing mainly with these threat multiplying conditions occurring in the smaller and weaker states of Africa, Chad, Niger, Sudan and the Central American countries, Nicaragua and El Salvador, where you see all of these conditions developing, but not posing a threat to the central core of the major powers, but as climate change advances, the military expects and US intelligence agencies expect, as I indicated, that larger, stronger, richer states will experience the same kinds of consequences and dangers and begin to experience this kind of state disintegration.

So, what we’re seeing in places like Chad and Niger, which involves this skirmishing between insurgents, terrorists, and other factions in which the US is playing a remote role, is playing the role, but it’s remote to situations where a Pakistan collapses, a Nigeria collapses, a Saudi Arabia collapses would require a much greater involvement by American forces on a much larger scale and that would be the next step up the ladder of escalation arising from climate change, and then you have the possibility, as I indicated, where nuclear armed states would engage in conflict, would be drawn into conflict because of climate related factors like the melting of the Himalayan glaciers and Indian and Pakistan going to war or Indian and China going to war, or we haven’t discussed this, but another consequence of climate change is the melting of the Arctic and this is leading to competition between the US and Russia in particular for control of that area.

So, you go from disintegration of small states to disintegration of medium-sized states, to conflict between nuclear armed states, and eventually to conceivable US involvement in climate related conflicts. That would be the ladder of escalation as I see it, and on top of that, you would have multiple disasters happening simultaneously in the United States of America, which would require a massive US military response. So, you can envision, and the military certainly worries about this, a time when US forces are fully immobilized and incapable of carrying out what they see as their primary defense tasks because they’re divided. Half their forces are engaging in disaster relief in the United States and another half are dealing with these multiple disasters in the rest of the world.

Lucas Perry: So, I have a few bullet points here that you could expand upon or correct about this ladder of escalation as you describe it. So at first, there’s the humanitarian interventions where the military is running around to solve particular humanitarian disasters like in Tacloban. Then, there’s limited military operations to support allies. There’s disruptions to supply chains and the increase of failed states. There’s the conflict over resources. There’s internal climate catastrophes and complex catastrophes, which you just mentioned, and then there’s what you call climate shock waves, and finally all hell breaking loose where you have multiple failed states, tons of mass migration, a situation in which no state no matter how powerful is able to handle.

Michael Klare: Climate shock wave would be a situation where you have multiple extreme disasters occurring simultaneously in different parts of the world leading to a breakdown in the supply chains that keep the world’s economy afloat and keep food and energy supplies flowing around the world, and this is certainly a very real possibility. Scientists speak of clusters of extreme events, and we’ve begun to see that. We saw that in 2017 when Hurricane Harvey was followed immediately by Hurricane Irma in Florida, and then Hurricane Maria in the Caribbean and Puerto Rico and the US military responded to each of those events, but had some difficulty moving emergency supplies first from Houston to Florida, then to Puerto Rico. At the same time, the west of the US was burning up. There were multiple forest fires out of control and the military was also supplying emergency assistance to California, Washington State, and Oregon.

That’s an example of clusters of extreme events. Now looking into the future, scientists are predicting that this could occur in several continents simultaneously. And as a result, food supply chains would break down, and many parts of the world rely on imported grain supplies, or other food stuffs and imported energy. And in a situation like this, you could imagine a climate shockwave in which trade just collapses and entire states suffer from a major catastrophe, food catastrophes leading to state collapse and all that we’ve been talking about.

Lucas Perry: Can you describe what all hell breaking loose is?

Michael Klare: Well, this is my expression for the all of the above scenario. You have these multiple disasters occurring and one that we have not discussed at length is the threat to American bases and how that would impact on the military. So, you have these multiple disasters occurring that create a demand on the military to provide assistance domestically, like I say, many areas needing emergency assistance and not just of the obvious sort of handing out water bottles, but as I say, complex emergencies where the military is being called in to provide law and order, to restore infrastructure, to play the role of government. So, you need large capacity organizations to step in. At the same time, it’s being asked to do that in other parts of the world, or to intervene in conflicts with nuclear armed states happening simultaneously. But at the same time, its own bases have been immobilized by rising seas and flooding and fires. All of this is a very realistic scenario because parts of it have already occurred.

Lucas Perry: All right, so let’s make a little bit of a pivot here into something that you mentioned earlier, which is the melting of the Arctic. So, I’m curious if you could explain the geopolitical situation that arises from the melting of the Arctic Ocean… Sorry, the Arctic region that creates a new ocean that leads to Arctic shipping lanes, a new front to defend, and resource competition for fish, minerals, natural gas and oil.

Michael Klare: Yes, indeed. In a way, the Arctic is how the military first got interested in climate change, especially the Navy because the Navy never had much of an Arctic responsibility. It was covered with ice, so its ships couldn’t go there except for submarines on long-range patrols under the sea ice, but the Navy never had to worry about the Arctic. And then around 2009, the Department of the Navy created a climate change task force to address the consequences of a melting Arctic sea ice and came to the view that as you say, this is a new ocean that they would have to defend that they’d never thought about before, and for which they were not prepared.

Their ships were not equipped to operate, for the most part, in the Arctic. So ever since then, the Arctic has become a major geopolitical concern of the United States on multiple fronts. But two or three points in particular that need to be noted, first of all, the melting of the ice cap makes it possible to extract resources from the area, oil and natural gas, and it turns out there’s a lot of oil and natural gas buried under the ice cap, under the seabed of the Arctic and oil and gas companies are very eager to exploit those untapped reserves. So the area, what was once considered worthless, is now a valuable geo-economic prize and countries have exerted claims to the area, and some of these claims overlap. So, you have border disputes in the Arctic between Russia and the United States, Russia and Norway, Canada and Greenland, and so on. There are now border disputes because of the resources that are in these areas. And because of drilling occurring there, you now need to worry about spills and disasters occurring, so that creates a whole new level of Naval and Coast Guard operations in the Arctic. This has also led to shipping lanes opening up into the region, and who controls those shipping lanes becomes a matter of interest. Russia is trying to develop what it calls the Northern Sea Route from the Pacific to the Atlantic going across its Northern territory across Siberia, and potentially, this could save two weeks of travel for container ships, moving from Rotterdam say to Shanghai and could be commercially very important.

Russia wants to control that route but the U.S. and other countries says, “It’s not yours to control.” So, you have disputes over the sea routes. But then, more important than any of the above is that Russia has militarized its portion of the Arctic, which is the largest portion, and this has become a new frontier for U.S.-Russian military competition, and there has been a huge increase in military exercises, base construction. Now, from the U.S. point of view, the Arctic is a new front in the future war with Russia and they’re training for this all the time.

Lucas Perry: Could you explain how the Bering Strait fits in?

Michael Klare: The Bering Strait between the U.S. and Russia is a pretty narrow space and that’s the only way to get from the North Pacific into the Arctic region, whether you’re going to Northern Alaska and Northern Canada, or to across from China and Japan, across the Northern Sea Route to Europe. So, this becomes a strategic passage way, the way Gibraltar has been the past. And both the U.S. and Russia are fortifying that passageway and there’s constant tussling going on there. It doesn’t get reported much, but every other week or so, Russia will send up its war planes right to the edge of U.S. airspace in that region, or the U.S. will send its planes into the edge of Russian airspace to test their reflexes and their naval maneuvers happening all the time. So, this has become seen as a important strategic place on the global chessboard.

Lucas Perry: How does climate change affect the Bering Strait?

Michael Klare: Well, it affects it in the sense that it’s become the entry point to the Arctic and the climate change has made the Arctic a place you want to go that it wasn’t before.

Lucas Perry: All right. So, one point that you made in your book that I like to highlight is that the Arctic is seen as a main place for conflict between the great powers in the coming years. Is that correct?

Michael Klare: Yes. For the U.S. and Russia, it’s important, here we would focus more on the Barents Sea, the area above Norway, and just, it helps of course to have a map in your mind, but Russia shares the border with Norway in it’s extreme north. And that part of Russia is the Kola Peninsula, and it’s where the City of Murmansk is located, and that’s the headquarters of Russia’s Northern Fleet and where it keeps its nuclear or missile submarines are based there. So, that’s how, that’s one of Russia’s few ways of access into the Atlantic Ocean from its own territory, from its major naval port at Murmansk. The waters adjacent to Northern Norway and Russia, like on the other side, have become a strategic, very important strategic military location. The U.S. has started building military bases with Norway in that area close to the Russian border. We’ve now stationed B-1 bombers in that area, so it is seen as a likely first area of conflict in the event of a war between the U.S. and Russia is going to occur at that spot.

Climate change figures into this because Russia views its Arctic region as critical economically as well as strategically and is building up its military forces there. And therefore, from U.S. NATO point of view, it’s a more strategically important region. But you ask about China, and China has become very interested in the Arctic as a source of raw materials, but also as a strategic passageway from its east coast to Europe for the reason I indicated, if once the ice cap melts, they’ll be able to ship goods to Europe in much shorter space of time and bring goods back if they can go through the Arctic. But China also is very interested in drilling for energy at the Arctic and for minerals, there are a lot of valuable minerals believed to be in Greenland.

You can’t get to those now because Greenland is covered with ice. But as that ice melts, which it’s doing at a rapid rate, the ground is becoming exposed and mining activities have begun there for things like uranium, and rare earths, and other valuable minerals. China is very deeply interested in mining there and this has led to diplomatic maneuverings, didn’t Donald Trump once talk about buying Greenland, to geopolitical competition between the U.S. and China over Greenland and this area.

Lucas Perry: Are there any ongoing proposals for how to resolve territorial disputes in the Arctic?

Michael Klare: Well, the shorter answer is no, there’s talk, there is something called the Arctic Council and this is an organization of the states that occupy territory in the Arctic region and it has some very positive environmental agendas and had some success in addressing non-geopolitical issues. But it has not been given the authority to address territorial disputes that members have resisted that. So, you don’t have a, it’s not a forum that would provide for that. There is a mechanism under the United Nations Convention on the Law of the Sea that allows for adjudication of off shore territorial disputes and it’s possible that that could be a forum for discussion, but mostly, these disputes have remained unresolved.

Lucas Perry: I don’t know much about this. Does that have something to do with, you have so many, you have X many miles from your sea shelf or something having to do with like the tectonic plates or ocean something.

Michael Klare: I can… Yes, so under the UN Convention on the Law of the Sea, you’re allowed a 200 nautical mile exclusive economic zone off your coastline. Every, any coastal country can claim 200 nautical miles. But you’re also allowed an extra 150 miles if your outer continental shelf, if you can prove scientifically that your outer continental shelf extends beyond 200 nautical miles, then you can extend your EEZ another 150 nautical miles out to 350 nautical miles. And the Northern Arctic has islands and territories that have allowed contending states to claim overlapping EEZs-

Lucas Perry: Oh, okay.

Michael Klare: … on this bases.

Lucas Perry: I see.

Michael Klare: And Russia has claimed vast areas of the Arctic as part of its outer continental shelf. But the great imperial power of Denmark, which territorially, is one of the largest imperial powers on earth because it owns Greenland, and Greenland also has an extended outer continental shelf that overlaps with Russia’s, as does Canada’s. You have to picture the looking down, not on the kind of wall maps we have of the world in our classrooms that make the Arctic look huge, but from a global map, everything comes closer together up there. And so, these extended EEZs overlap and so Greenland, and Canada, and Russia are all claiming the North Pole.

Lucas Perry: Okay. So, I think that paints really well the picture of the already existing and conflict there and how it will likely only get worse in terms of the amount of conflict. It’d be great if we could focus a bit on nuclear weapons risk in climate change in particular. I’m curious if you could explain the DOD’s concerns an improving China, and a nuclear North Korea, and India, and Pakistan, and other nuclear states in this evolving situation of increasing territorial disputes due to climate change.

Michael Klare: From a nuclear war perspective, the two greatest dangers I think and I’ve mentioned these, one is the collapse or the disappearance of the Himalayan Glaciers, sparking a war between India and China that would go nuclear, or one between India and Pakistan that would go nuclear. That’s one climate-related risk of nuclear escalation. The other is in the Arctic, and here, I think the danger is the fact that Russia has turned the Arctic into a major stronghold for its nuclear weapons capabilities. It stations a large share of its nuclear retaliatory, warheads on submarines, and other forces that are based in the Arctic. And so, in the event of a conflict between the U.S. and Russia, this could very well take place in the Arctic region and trigger the use of nuclear weapons as a consequence.

Lucas Perry: I think we’ve done a really good job of showing all of the bad things that happen as climate change gets worse. The Pentagon has perspective on everything that we’ve covered here, is that correct?

Michael Klare: Yes.

Lucas Perry: So, how does the Pentagon intend to address the issue of climate change and how it affects its operations?

Michael Klare: The Pentagon has multiple responses to this, and this began as early as 2010 in the Quadrennial Defense Review of 2010. This is a every four-year strategic blueprint released by the Joint Chiefs of Staff and the Secretary of Defense. And that years was the first one that, number one, identified climate change as a national security threat and spelled out the responses that the military should make, and there were three parts to that. One part is, I guess you would call it hardening U.S. bases to the impacts of climate change, increasing resiliency and seawalls to protect low-lying bases, but otherwise, enhancing the survivability of U.S. bases in the face of climate change. That’s one response. A second response is in mitigating the department’s own contributions to climate change by reducing its reliance on fossil fuels. And I could talk what specifically they’re doing in that area.

The third is, and I think this is very interesting, they said that we should not only, that because climate change is a global problem, this was specific, climate change is a global problem, affects our allies and friends, and therefore, we should work with our allies and with the military forces of our allies and friends to do the same things we’re doing at home to do in their countries as well, that is to build resilience, to prepare for climate change, to reduce impacts so that this would be a global cooperative effort, military to military which has gotten very little attention, I think, from the media and from Congress and elsewhere, but a very important part of American foreign policy with respect to climate change.

Lucas Perry: So, there’s hardening our own bases and systems, I believe in your book you mentioned, for example, turning bases into operational islands such that their energy and material needs are disconnected from supply lines. The second was reducing the greenhouse emissions of the military, and the third is helping allies with such efforts. I’m curious if you could describe a bit more the first and the second of these, the hardening of our own systems and bases and becoming more green. Because I mean, it is interesting and at least a little bit surprising that the military is trying to become green in order to improve combat readiness through independence of a foreign and domestic fuel needs and sources. So, could you explain a little bit more this, for example, the drive to create a green fleet in the Navy?

Michael Klare: Sure. Now, but this began during the Obama administration and then went semi-underground during the Trump administration, so the information we have is mainly pre-Trump. Now, under president Biden, climate change has been elevated to a national security threat as per an executive order he issued shortly after taking office, and our new Secretary of Defense Lloyd Austin has said, has issued a complementary statement that climate change is a departmental-wide Department of Defense concern, so activities that were prohibited by the Trump administration will now be revived. So, we will now hear a lot more about this in the months ahead, but there is a four-year blackout of information on what was being done. But during the Obama administration, the Department of Defense was ordered to, as I say, to work on adaptation and mitigation both as part of its responsibilities, the adaptation affected particularly bases in low low-lying coastal areas.

And there are a lot of U.S. bases for historic purposes, for historic reasons are located along the East Coast of the U.S., that’s where they started out. Most important of them is the Norfolk Naval Station in Virginia, the most important naval base in the United States. It’s at sea level and it’s on basically reclaimed swamplands and it’s subsiding into the ocean at the same time sea level is rising. But there are many other bases along the East Coast and Florida, and in the Gulf coast that are at equal risk. And so, part of what the military is doing is to build seawalls to protect them against sea surges, moving critical equipment from areas that are in high flood prone areas to areas that are at higher elevation, adopting codes, any new buildings built on these bases have to be hardened against hurricanes, and sensitive equipment, electronic equipment has to be put in the higher stories so that if they are flooded they won’t be damaged.

There’s a lot of very concrete measures that have to do with base construction that have been undertaken to enhance the resilience of bases in response to extreme storms and flooding. That’s one aspect of this. The mitigation aspect is to reduce reliance on fossil fuels and to convert as wherever possible, vehicles, air, ground, and sea vehicles to use alternative fuels. So, the Navy, the Army, the Air Force are converting their non-tactical vehicle fleets, they all have huge numbers of ordinary sedans, and vans, and trucks. Increasingly, these will be hybrids or electric vehicles. And the Air Force is experimenting with alternative fuels produced by algae, and the Navy has experimented with alternative fuels derived from agricultural products, and so on. So, there’s a lot of experimentation going on, a lot of, some of the biggest solar arrays in the U.S. are on U.S. military bases or constructed at the behest of U.S. military bases by private energy companies. Those are some of the activities that are underway.

Lucas Perry: In addition to threatening U.S. military bases and the bases of our allies, climate change will also affect the safety and security of, for example biosafety level 4 labs and also nuclear power plants. So, I’m curious how you view the risks of climate change affecting crucial infrastructure, should it fail, could create global catastrophe, for example, from nuclear power plants melting down or pathogens being released from biosafety labs that fail under the stresses of climate change.

Michael Klare: I have not seen the literature on the bio labs in the Pentagon literature. What they do worry about is the fragility of the U.S. energy infrastructure in particular, in part because they depend on the same energy infrastructure as we do for their energy needs, for electricity transmission, pipelines and the like to supply their bases and their other facilities. And they’re very aware that the U.S. energy infrastructure is exceedingly vulnerable to climate change, either a lot of it, a very large part of our infrastructure is on the East Coast and the West Coast, very close to sea level, very exposed to storm damage and a lot of it is just fragile. A clearer example of that is Hurricane Maria in Puerto Rico when the electric system collapsed entirely and the Army Corps of Engineers had to come in and were there for almost an entire year rebuilding the energy infrastructure of Puerto Rico.

They’ve had to do this and other places as well. So, they are very worried that climate change disasters, multiple disasters, will knock out the power in the U.S. causing major cascading failures. So, when energy fails, then petrochemical facilities fail. And that’s what happened in Houston in Hurricane Harvey. The power failure went out and these petrochemical facilities, which Houston has many of, failed and toxic chemicals spilled out, and also the sewer system collapsed. So, you have, cascading failures producing toxic threat. And the military had to issue toxic protective clothing to its personnel in doing rescue operations because the water in flooded areas of Houston was poisonous. So, it’s the cascading effects that they worry about. This happened in New York City with Hurricane Sandy in 2012 where power went out, then gas stations couldn’t operate and hospitals and nursing homes couldn’t function. Well, I’m going on here, but you get a sense of the interrelationship between these critical elements of infrastructure. Fires are another aspect of this, as we know from California. A lot of US bases in California are at risk from fires and the transmission lines that carry the energy. I was going to mention the Colonial Pipeline disaster, which was a cyber attack, not climate related, but that exposes the degree to which our energy infrastructure is fragile.

Lucas Perry: If it rains or snows just enough, we’ve all experienced losing power for six hours or more. The energy grid seems very fragile even to relatively normal weather.

Michael Klare: Yes, but with climate change and these multiple, simultaneous disasters where the whole systems break down.

Lucas Perry: Do you see lethal autonomous weapons as fitting into the risks of escalation in a world stressed by climate change?

Michael Klare: Well, I see lethal autonomous weapons as a major issue and problem, which I’ve written about and I worry about a great deal. Now, what is their relationship to climate change? I couldn’t say. I think the military in general is facing the world in which they feel that humans are increasingly unable to cope with the demands of time compression and decision-making and the complexity of the environment in which decision-makers have to operate and that’s partly technological, it’s partly just the complexity of the world that we’ve been discussing.

And so, there’s ever increasing sense among the military that commanders have to be provided with computer assisted decision-making and autonomous operations because they can’t process the amount of data that’s coming into them simultaneously. This is behind not just autonomous weapons systems, but autonomous weapons systems’ decision-making. The new plans for how the Army, Navy, and Air Force will operate will be fewer human decision-makers and more machine information processors and decision-makers, and which humans will be given a menu of possible choices, but they will be strike this set of targets or that set of targets, but not stop and think about this, and maybe we should de-escalate. They’re going to be militarized options.

Lucas Perry: So, some sense of lethal autonomous weapons is potentially exacerbating or catalyzing the speed at which the ladder of escalation is moved through.

Michael Klare: No question about it. Many factors are contributing to that. The speed of weaponry, the introduction of hypersonic missiles, which cuts down flight time from 30 minutes to five minutes, the fact that wars are being conducted in what they call multiple domains simultaneously: cyber, space, air, sea, and ground, that no commander can know what’s happening in all of those domains and make decisions. So, you have to have what they want to create, a super brain called the Joint All-Domain Command and Control System, the JADC2 system, which will collect data from sensors all over the planet and compress it into simplified assessments of what’s happening, and then tell commanders, here are your choices, one, two, and three, and you have five seconds to choose, and if not, we’ll pick the best one and we’ll be linked directly to the firers to launch weapons. This is what the future will look like, and they’re testing this now. It’s called Project Convergence.

Lucas Perry: So, how do you see all of this affecting the risks of human extinction and of existential risks?

Michael Klare: I’m deeply concerned about this inclination to rely more on machines to make decisions of life and death for the planet. I think everybody should be worried about this, and I don’t think enough attention is being paid to these dangers of automating life and death decision-making, but this is moving ahead very rapidly and I think it does pose enormous risks. The reason that I’m so worried is that I think the computer assisted decision-making will have a bias towards military actions.

Humans are imperfect and sometimes we make mistakes. Sometimes we get angry and we go in the direction of being more violent and brutal. There’s no question about that, but we also have a capacity to say, stop, wait a minute, there’s something wrong here and maybe we should think twice and hold back. And, that’s saved us on a number of occasions from nuclear extinction. I recommend the book Gambling with Armageddon by Martin Sherwin, a new account of the Cuban Missile Crisis day by day, hour by hour account, and which it was clear that the US and Russia came very close, extremely close to starting a nuclear war in 1962, and somebody said, “Wait a minute, let’s just think about this. Let’s not rush into this. Let’s give it another 24 hours to see if we can come up with a solution.”

Adlai Stevenson apparently played a key role in this. I fear that the machines we designed are not going to have that kind of thinking built into them, that kind of hesitancy, that second thinking. I think the machines are going to be designed… The algorithms that inhabit them are going to reflect the most aggressive possible outcomes, and that’s why I fear that we move closer to human extinction in a crisis than before, and because of the time of decision-making is going to be so compressed that humans are going to have very little chance to think about this.

Lucas Perry: So, how do you view the interplay of climate change and autonomous weapons as affecting existential risk?

Michael Klare: Climate change is just going to make everything on the planet more stressful in general. It’s going to create a lot of stress, a lot of catastrophes occurring simultaneously and creating a lot of risk events happening that people are going to have to be dealing with, and they’re going to create a lot of hard, difficult choices. Let’s say you’re the president, you’re the commander in chief, and you have multiple hurricanes striking and fires striking the United States, that’s hardly an unlikely outcome, at the same time that there’s a crisis with China and Russia occurring where war would be a possible outcome. There’s a naval clash at sea in the South China Sea or something happening on the Ukraine border, and meanwhile, Nigeria is breaking apart and India and Pakistan are at the verge of war.

These are very likely situations in another 10 to 20 years if climate change proceeds the way it is. So, just the complexity of the environment, the stress that people will be under, the decisions they’re going to have to make swiftly between do we save Miami or do we save Tokyo? Do we save Los Angeles or do we save New York, or do we save London? We only have so many resources. In these conditions, I think the inclination is going to be to rely more on machines to make decisions and to carry out actions, and that I think has inherent dangers in it.

Lucas Perry: Do you and/or the Pentagon have a timeline for… How much and how fast is the instability from climate change coming?

Michael Klare: This is a progression. We’re on that path, so there’s no point at which you could say we’ve reached that level. It’s just an ever increasing level of stress.

Lucas Perry: How do you see the world in five or 10 years given the path that we’re currently on?

Michael Klare: I’m pessimistic about this, and the reason I am pessimistic is because if you go back and read the very first reports of the Intergovernmental Panel on Climate Change, the IPCC, their very first reports, and they would give a series of projections based on their estimates of the pace of greenhouse gas emissions. If they go this high, then you have these projections. If they go higher, then these projections out to 2030, 2040, 2050, we’ve all seen these charts.

So, if you go back to the first ones, basically we’re living in 2021 what they said were the worst case projections for 2040 to 2050 by and large. So, we’re moving into the danger zone. So, what I’m saying is we’re moving into the danger zone much, much faster than the most worst case scenarios that scientists were talking about 10 years ago, or 20 years ago, and if that’s the case, then we should be very, very worried about the pace at which this is occurring because we’re off the charts now from those earlier predictions of how rapidly sea level rise was occurring, desertification was occurring, heat waves. We’re living in a 2050 world now. So, where are we going to be in a 2030? We’re going to be in a 2075 world and that world was a pretty damn scary world.

Lucas Perry: All right, so I’m mindful of the time here. So, just a few more questions about messaging would be nice. So, do you think that tying existential risks to national security issues would benefit the movement towards reducing existential risks, given that climate change is elevated in some sense by the DOD taking it seriously on account of national security?

Michael Klare: So, let me explain why I wrote this book, and this is very much a product of the Trump era, but I think it’s still true today that you have a country that’s divided between environmentalists and climate deniers, and this divide has prevented forward movement in Congress to pass legislation that will make a significant difference in this country, and I believe this has to come from national level, the kind of changes we need, the massive investments in renewables and charging stations for electric vehicles, all these things require national leadership, and right now that’s impossible because of the fundamental divide between the Democrats and Republicans or denialists and environmentalist, however you want to put it. Some of my friends in the environmental community, dear friends, think if we could only get across the message that things are getting worse that those deniers will finally wake up and change their views.

I don’t think that’s going to happen. I think more scientific evidence about climate change is not going to win over more people. We’ve tried that. We’ve done everything we can to make the scientific evidence known. So, the way to win, I believe the military perspective that this is a threat to the national security of the United States of America, are you a patriotic American or not? Do you care about the security of this country or not?

This is not a matter of environmentalism or anti-environmentalism. This is about the national security of this country. Where do you stand on that? That this is a third approach that could possibly win over some segment of that population that until now has resisted action on climate change, that’s not going to listen to an environmentalist or green argument. There is evidence that this approach is making a difference, that Republicans who won’t even talk about the causes of climate change, but who acknowledge that their communities are at risk or the country is at risk on a national security basis, and therefore are willing to invest in some of the changes that are necessary for that reason. So, I do believe that making this argument, it could win over enough of that resistant population to make it possible to actually achieve forward momentum.

Lucas Perry: Do you think that relating climate change to migration issues is helpful for messaging?

Michael Klare: I’m not sure because I think people who are opposed to migration don’t care what the cause is, but I do think that it might feed into the argument that I was just making that our security would be better off by emphasizing climate change and therefore taking steps to reduce the pressures that lead climate migrants to migrate. The military certainly takes that view, so it could be helpful, but I think it’s a difficult topic.

Lucas Perry: All right, so given everything we’ve discussed here today, how would you characterize and summarize the Pentagon’s interest, view, and action on climate change and why that matters?

Michael Klare: So, now we have a new test because as I’ve indicated, we had a blackout period of four years during the Trump administration when all of this was hidden and couldn’t be discussed. So, we don’t know how much was accomplished. Now, this is an explicit priority for the Department of Defense and the defense budget, and other documents say that this is a priority for the department and the Armed Forces, and they are required to take steps to adapt to climate change and to mitigate their role in climate change.

So, we have to see how much actually is accomplished in this new period before really you can make any definitive assessment, but I think that you can see that the language adopted by the Biden administration and Lloyd Austin at the Department of Defense is so much stronger and more vigorous than what the Pentagon was saying in the Obama administration. So, even though there was a four year blackout period, there was a learning curve going on, and what they’re saying today is much more advanced and the sense of recognizing the severity of the risks posed by climate change and the necessity of making this a priority.

Lucas Perry: All right, so as we wrap up, are there any final words or anything you’d like to share that you feel is left unsaid or any parting words for the audience?

Michael Klare: As I started out, we mustn’t forget that if you asked anybody in the military what their job is, they’re going to come back to China number one. So, we shouldn’t forget that, defending against China. It’s only after you peel away the layers of how they’re going to operate in a climate altered world that all of these other concerns start spilling out, but it’s not going to be the first thing that they’re likely to say. I think that has to be clear, based on my conversations, but there is a real awareness that in fact climate change is going to have an immense impact on the operations of the military in the years ahead, and that its impact is going to grow exponentially.

Lucas Perry: All right. Well, thank you very much for coming on Michael, and for sharing all of this with us. I really appreciated your book and I recommend others check it out as well, it’s All Breaking Loose. I think it does a really good job of showing the ways in which the world is going to get worse through climate change. There’s a lot of really great examples in there. So, also the audiobook has a really great narrator, which I very much liked. So, thank you very much for coming on. If people want to check or follow you out on social media, where and how can they do that?

Michael Klare: Oh, I’m at michaelklare.com and let’s start there.

Lucas Perry: All right. Do you also have a place for you where you list publications?

Michael Klare: At that site.

Lucas Perry: At that site? Okay.

Michael Klare: And, it’s K-L-A-R-E, Michael Klare, K-L-A-R-E.

Lucas Perry: All right, thank you very much, Michael.

Avi Loeb on UFOs and if they’re Alien in Origin

  • Evidence counting for the natural, human, and extraterrestrial origins of UAPs
  • The culture of science and how it deals with UAP reports
  • How humanity should respond if we discover UAPs are alien in origin
  • A project for collecting high quality data on UAPs

Watch the video version of this episode here

See here for information on the Podcast Producer position

See the Office of the Director of National Intelligence report on unidentified aerial phenomena here


Lucas Perry: Welcome to the Future of Life Institute Podcast. I’m Lucas Perry. This is a follow up interview to the main release that we’ve done with Avi Loeb. After our initial interview, the US Government released a report on UFOs, otherwise now known as UAPs, titled, Preliminary Assessment: Unidentified Aerial Phenomena. This report is a major and significant event in the history and acknowledgement of UFOs as a legitimate phenomena. As our first interview with Avi focused on Oumuamua and its potential alien origin, we also wanted to get his perspective on UFOs, this report, his views on whether they’re potentially alien in origin, and what this all means for humanity.

In case you missed it in the main episode, we’re currently hiring for a Podcast Producer to work on the editing, production, publishing, and analytics tracking of the audio and visual content of this podcast. As the Producer you would be working directly with me, and the FLI outreach team, to help grow, and evolve this podcast. If you’re interested in applying, head over to the Careers tab on the Futureoflife.org homepage or follow the link in the description. The application deadline is July 31st, with rolling applications accepted thereafter until the role is filled. If you have any questions, feel free to reach out to socialmedia@futureoflife.org.

And with that, I’m happy to present this bonus interview with Avi Loeb.

The Office of the Director of National Intelligence has released a preliminary assessment on unidentified aerial phenomena, which is a new word that they’re using for UFO, so now it’s UAP. So can you summarize the contents of this report and explain why the report is significant?

Avi Loeb: The most important statement made in the report is that some of the objects that were detected are probably real and that is based on the fact that they were detected in multiple instruments using radar systems, or infrared cameras, or optical visual cameras, or several military personnel seeing the same thing, doing the same thing at the same time. And so that is a very significant statement because the immediate suspicion is that unusual phenomena occur when you have a smudge on your camera, when there is a malfunction of some instruments, and the fact that there is corroborating evidence among different instruments implies that it must be something real happening. That’s the first significant statement.

And then there were 144 incidents documented but it was also mentioned there is a stigma on reporting because there is a taboo on discussing extraterrestrial technologies, and as a result only a small minority of all events were reported. But nevertheless, the Navy established in March 2019 a procedure for reporting, which was not available prior to that and the Air Force followed on that in December 2020. So it’s all very recent that there is this procedure or formal path through which reports can be obtained. And of course, that helps in the sense that it provides a psychological support system for those who want to report about unusual things they have witnessed, and prior to that they had to dare to speak given the stigma.

And so the second issue is of course we have objects, if some of them are real. And by the way, we only saw a small fraction of the evidence most of it is classified and the reason is that the government owns the sensors that were used to obtain this evidence. And these sensors are being used to monitor the sky, and therefore our national security importance and we don’t want to release information about the quality of the sensors to our adversaries, to other nations. And so the data itself is being classified because the instruments are classified, but nevertheless one can think of several possible interpretations of these real objects.

And I should say that CIA Directors, say like Brennan, and Woolsey, and President Barack Obama spoke about these events as serious matters so that, to me, it all implies that we need to consider them seriously. So there are several possible interpretations, one of course is that they are human made and some other nation produced them, but some of the objects behaved in ways that supersede our technologies, the limits of the technologies we have in the U.S. and we have some intelligence on what other nations are doing. And moreover, if there was another nation with far better technologies they would find themselves in the consumer market because there would be a huge financial benefit to using them or we would see some evidence for them in the battlefield. And at the moment we have a pretty good idea, I would argue, as to what other nations are doing technologically speaking.

So if there are real objects behaving in ways that exceed human technologies, then the question is what could it be? And there are two possibilities, either these are natural phenomena that occur in the atmosphere that we did not expect or they are of extraterrestrial origin, some other technological civilization produced these objects and deployed them here. And of course, both of these possibilities are exciting because we learned something new. So the message I take from this report is that the evidence is sufficiently intriguing for the subject to move away from the talking points of politicians, national security advisors, military personnel that were really not trained as scientists. It should move into the realm of science where we use state-of-the-art equipment, such as cameras installed on wide field telescopes that scan the sky. These are telescopes you can buy off the shelf and you can position them in similar geographical locations, and monitor the sky with open data, and analyze the data using the best computers we have in a completely transparent way like a scientific experiment.

And so that’s what we should do next and instead what I see is that a lot of scientists just ridicule the significance of the report or say business as usual, that there is no need to attend to these statements. And I think it’s an opportunity for science to actually clarify this matter and clear up the fog and this is definitely a question that is of great interest to the public. What is the nature of these unidentified objects or phenomena? Are they natural in origin or maybe extraterrestrial? And I’m very much willing to address this with my research group given proper funding for it.

Lucas Perry: Let’s stick here with this first point, right? So I actually heard Neil deGrasse Tyson on MSNBC just before we started talking and he was mentioning that he thought that there could have been hardware or software artifacts that could have been generating artifacts on these systems. And I think your first point very clearly refutes that because you have multiple different systems plus eyewitness reports all corroborating the same thing. So it’s confusing to me why he would say that.

Avi Loeb: Well, because he is trying to maximize the number of likes he has on Twitter and doubting the reality of these reports appears to be popular among people in academia, among scientists, among some people in the public and so he’s driven by that. My point is, an intelligent culture is driven or actually is following the guiding principles of science and those are sharing evidence based knowledge. What Neil deGrasse Tyson is doing is not sharing evidence based knowledge, but rather dismissing evidence. And my point is, this evidence that is being reported in Washington, D.C. is intriguing enough to motivate us to collect more evidence rather than ignore it. So obviously, if you look at the history of science, we often make discoveries when we find anomalies, things that do not line up with what we expected.

And the best example is quantum mechanics that was discovered a century ago, nobody expected it. It was forced upon us by experiments and actually, Albert Einstein at the time resisted one of its fundamental facets, entanglement, or what he called spooky action at a distance. That the quantum system knows about its different parts even if they are separated by a large distance such that light signals cannot propagate over the time of the experiment. And he argued that this cannot be the case and wrote a paper about it, it’s with his postdocs, it’s called the Einstein-Podolsky-Rosen experiment. That experiment was done and demonstrated that he was wrong and even a century later we are still debating the meaning of quantum mechanics.

So it’s sort of like a bone stuck in the throat of physicists, but nevertheless, we know that quantum mechanics holds and applies to reality. And in fact the reason the two of us are conversing is because of our understanding of quantum mechanics, the fact that we can use it in all the instruments that we use. For example, the speaker that the two of us are using, and the internet, and the computers we are using, all of these use principles of quantum mechanics that were discovered over the century. And my point is, that very often when you do experiments there are situations where you get something you don’t expect, and that’s part of the learning experience. And we should respect deviations from our expectations because they carve a new path to new understanding, new learning about reality and about nature rather than ridiculing it, rather than always thinking that what we will find will match what we expected.

And so if the government among all comes along with reports about unusual phenomena, the government is very conservative very often, you would expect the scientific community to embrace that as an exciting topic to investigate because the government is saying something, let’s figure out what is it about. Let’s clarify the nature of these phenomena that appear anomalous rather than saying business as usual, I don’t believe it, it could be a malfunction of the instrument. So if it is a malfunction of the instrument, why did many instruments show the same thing? Why did many pilots show the same thing? And I should clarify that in the courtroom if you have two eyewitness testimonies that corroborate each other you can put people in jail as a result. So we believe people in the legal system and somehow when it comes to pilots, who are very serious people that serve our country, then Neil deGrasse Tyson dismisses their testimony.

So my point is not that we must believe it, but rather that it’s intriguing enough for us to collect more data and evidence and let’s not express any judgment until we collect that evidence. The only sin we can make is to basically ignore the reports and do business as usual, which is pretty much what he is preaching for. And my point is, no instead we should invest funds in new experiment or experiments that will shed more light on the nature of these objects.

Lucas Perry: So the second point that you made earlier was that the government was establishing a framework and system for receiving reports about UFOs. So part of this and part of this document, is it true to say then there is also a confirmation that the government does not know what these are and that they are not a secret U.S. project?

Avi Loeb: Yeah, they stated it explicitly in the report. They said the data is not good enough, the evidence is not good enough to figure out the nature of these objects so we don’t know what they are. And by the way, you wouldn’t expect military personnel or politicians to figure out the nature of anomalous objects because they were not trained as scientists. So when you go to a shoemaker, you won’t expect the shoemaker to bake you a cake. These are not people that were trained to analyze data of this type or to collect new data such that the nature of these objects will be figured out.

That is what scientists are supposed to do and that’s why I’m advocating for moving this subject away from Washington, D.C. into an open discussion in the scientific community where we’ll collect open data, analyze it in a transparent way, not with government owned sensors or computers, and then it will be all clear. It will be a transparent process. We don’t need to rely on Washington, D.C. to tell us what appears in our sky. The sky is unclassified in most locations. We can look up anytime we want and so we should do it.

Lucas Perry: So with your third point, there’s this consideration that people are more likely to try and give a conventional explanation of UAPs as coming from other countries like Russia or China, and you’re explaining that there are heavy commercial incentives. For example, that if you had this kind of technology it could for example, revolutionize your economy and you wouldn’t just be using it to pester U.S. Navy pilots off the coast, right? It could be used for really significant economical reasons. And so it seems like that also counts as evidence against it being conventional in origin or a secret government project. What is your perspective on that?

Avi Loeb: Yes and it would not only find its place in the consumer market, but also in the battlefield. And we have a sense of what other nations are doing because the U.S. has its own intelligence and we pretty much know what the status of their science and technology is. So it’s not as if we are completely in the dark here and I would argue if the U.S. government reports these objects there is good evidence that they are not made by those other nations because if our intelligence would tell us that they were potentially made by other nations then we would try to develop the same technologies ourselves. And another way to put it is if a scientific inquiry into the nature of these objects allows us to get a high resolution photograph of one of them and then we see the label “Made in China” or “Made in Russia,” then we would realize that there was a major failure of national intelligence and that would be a very important conclusion of course, that would have implications for our national security.

But I doubt that this is the case because we have some knowledge of what other nations are doing and the data would not have been released this way in this kind of a report if there was suspicion that these objects are human made.

Lucas Perry: So the report makes clear then that it’s not U.S. technology and there’re also reasons that count against it being for example, Russian or Chinese technology because the incentives are aligned for them to just deploy it already and use it in the public sector. So before we get into more specifics about thinking about whether or not these are human or extraterrestrial in origin, I’m curious if you could explain a bit more the flight and capability characteristics of these UAPs and UFOs and what you feel are the most significant reports of them and their activity.

Avi Loeb: Well, I didn’t have direct access to the data, especially not the classified one and I would very much want to see the full dataset before expressing an opinion, but at least some of the videos that were shown indicated motions that cannot be reproduced by the kind of crafts that we own. But what I would like to know is whether when the object moves faster than the speed of sound for example in air, whether it produces a sonic boom, a shockwave that we see for example, when jets do the same because that would be an indication that indeed there is a physical object that is compressing air as it moves around. Or if it moves through water I want to see the splash of water and from that I can infer some physical properties.

And of course, I would like to have a very high resolution image of the object so that I can see if it has screws, if there is something written on it, either Made in China or Made on Planet X. Either messages would be of great importance. So what I’m really after is access to the best data that we have and obviously, it will not be released from the government because the best data is probably classified. But I would like to collect it through using scientific instrumentation, which by the way could be far better than the instruments that were on airplanes that the pilots were using or on Navy because these were designed to be in combat situations and they were not optimal for analyzing such objects. And we can do much better if we choose our scientific instruments carefully and design the experiment in a way that would reproduce the results with a much higher fidelity of the data.

Lucas Perry: There is so much about this that is similar to Oumuamua in terms of there being just barely… The imaging is not quite enough to really know what it is and then there being lots of interesting evidence that counts for extraterrestrial in origin. Is that a perspective you share?

Avi Loeb: Well, yes I wrote a Scientific American article where I said one thing we know about Oumuamua is that it probably had a flat shape, pancake like, and also if its push away from the sun was a result of reflecting sunlight it must have been quite thin and the size of a football field. And in that case, I thought maybe it serves as a lightsail, but potentially it could also be a receiver intended to detect information or signals from probes that were sprinkled on planets in the habitable zone around the sun. So if for example the UAP are probes transmitting signals, then the passage of such a receiver near Earth was meant to obtain that information. And Oumuamua for example, was tumbling every eight hours, was looking in all directions in principle for such signals, so that could be one interpretation that it was thin not because it was a lightsail, but because it served a different purpose.

And in September 2020, we saw another object that also exhibited an excess push away from the sun by reflecting sunlight and had no cometary tail. It was given the name 2020 SO and it was a rocket booster from a 1966 mission. It had thin walls for a completely different purpose, not having anything to do with it being a lightsail. So I would argue that perhaps Oumuamua had these weird properties because it served a different purpose and that’s why we should both try to get a better image of an object like Oumuamua and of the unidentified objects we find closer to Earth. And in both cases, a high resolution photograph is better than a 1,000 words, in my case better than 66,000 words, the number of words in my book.

Lucas Perry: In both cases, a little bit better instrumentation would have seemingly made a huge difference. So let’s pivot into again, this area of conventional explanations. And so we talked a little bit earlier about one conventional explanation being that this is some kind of advanced secret military technology of China or Russia that’s used for probing our aerial defenses. And the argument that counts against that again, was that there are military and economic incentives to deploy it more fully, especially because the flight characteristics that these objects are expressing are so much greater than anything that America has in terms of the speed and the agility. So one theory is that instead of the technology being actual, like they actually have the technology that goes that fast and is that agile, that this is actually some form of spoofing technology, so some kind of way of an adversary training electronic countermeasures to simulate, or emulate, or create the illusion of what we witnessed in terms of the U.S. instruments. So do you think that such an explanation is viable?

Avi Loeb: I mean, it’s possible and that’s why we need more data. But it’s not easy to produce an illusion in multiple instruments, both radar, infrared, and optical sensors because you can probably create an illusion for one of these sensors, but then for all of them it would require a great deal of ingenuity and then you would need a good reason to do that. Why would other nations engage in deceiving us in this way for 20 years? I mean, that would look a bit off and also, we would have probably found something, some clue about them trying to do that because they would have trained such probes or such objects first in their own facilities and we would see some evidence for that. So I find it hard to believe, I would think it’s either some natural phenomena that we haven’t yet experienced or suspected or it’s this unusual possibility of an extraterrestrial origin.

And either way we will learn something new by exploring it more. We should not have any prejudice. We should not dismiss it. That would be the worst we can do, just dismiss it, and ridicule it, and continue business as usual because actually it’s exciting to try and figure out a puzzle. That’s what detectives often do and I just don’t understand the spirit of dismissing it and not looking into it at all.

Lucas Perry: So you just mentioned that you thought that it might have some kind of natural explanation. There are very strange things in nature. I’m not sure if this is for example, real or not, but there’s a Wikipedia page for example, for ball lightning, and there’s also really weird phenomena that you can have in the sky if the lighting in the sky is just right, and where the sun is where you get weird halos and things. And throughout history there are reports of dancing lights in the sky or things that might have been collective hallucinations or actually real. In terms of it being something natural that we understand or it being something human made that we’re not aware of, what to you is the most convincing natural or conventional explanation of these objects? An explanation that is not extraterrestrial in origin.

Avi Loeb: Well, if it’s dancing lights it wouldn’t produce a radar echo. So as I said, I don’t have access to the data in each and every incident, but there are some fundamental logic that one can use for each of these datasets and figure out if it could be an illusion. If not, if it must be a real object, somehow nature has to produce a real object that behaves this way and until I get my own data and reproduce those I won’t make any statement. But I’m optimistic that given the appropriate investment of funds, which I’m currently discussing with private sector funders, that we can do it. And just to give you an example, if you wanted to get a high resolution image, like a megapixel image of a one meter size object at a distance of a kilometer, you just need a one meter telescope for that and observing it at optical light. And you will be able to see millimeter size features on it, like the head of a pin.

People ask why didn’t we see it already in iPhone images of the sky? Well, the iPhone camera is a millimeter or a few millimeters in aperture size and it’s too small. You can’t get anything better than a fuzzy image of a very distant object. So you really need to have a dedicated experiment, and I think one can do it, and I’m happy to engage in that.

Lucas Perry: You would also wonder that these things are probably… So if they were extraterrestrial in origin, you would expect that they would be pretty intelligent and that they might understand what our sensor capabilities are. So, I think perhaps that might count as evidence for why given that there are billions of camera phones around the planet that there aren’t any good pictures. What is your perspective on that?

Avi Loeb: If I had to guess, I would think of these systems as equipped with artificial intelligence. We already have systems of artificial intelligence that are capable of superseding human abilities and within a decade they will be more intelligent than people, we’ll be able to learn from machine learning, and adapt to changing circumstances, and behave very intelligently. So in fact, I can imagine that if another civilization had that technological development of more than a century, more than we did, they could have produced systems that are autonomous. It doesn’t make any sense to communicate with the sender because the nearest star is four light years away. It takes four years for a signal to reach even the nearest star and it takes tens of thousands of years to reach the end of the galaxy, the Milky Way.

And so there is no sense of a piece of equipment communicating with its senders in order to get guidelines as to what to do. Instead, it should be autonomous, it has its own intelligence, and it could outsmart us. We already almost have such systems. So in fact, we may need to use our own artificial intelligence systems in order to interpret the actions of those artificial intelligence systems. So it will resemble the experience of asking our children to interpret the content that we find on the internet because they have better computer skills. We need our computers to tell us what their computers are doing. And that’s the way I think about it and these systems could be so intelligent that they do things that are subtle. They don’t appear and start a conversation with us. They are sort of gathering information of interest to them and acting in a way that reflects the blueprint that guided whoever created them.

And the question is, what is their agenda? What is their intent? And that will take us a while to figure out. We have to see what kind of information they’re seeking, how do they respond to our actions, and eventually we might want to engage with them. But the point is, many people think of contact as being a very sort of abrupt and interaction of extraordinary proportion that is impossible to deny, but in fact, it could be very subtle because they are very intelligent. If you look at intelligent humans, they’re not aggressive, they’re often thinking about everything they do and select their actions appropriately. They don’t get into very violent confrontations often. We need to rely on evidence rather than prejudice and the biggest mistake we can make is the mistake made by philosophers during the days of Galileo. They said, “We don’t want to look through at telescope because we know that the sun moves around the Earth.”

Lucas Perry: We spoke a little bit earlier about Bayesian reasoning and Oumuamua. Do you have the same feelings about not having priors about the UAPs being extraterrestrial or human in origin? Or is there a credence that you’re able to assign to it being extraterrestrial in origin?

Avi Loeb: The situation is even better in the context of UAP because they are here, not far from us and we can do the experiment with a rather modest budget. And therefore, I think we can resolve this issue with no need to have a prejudice. Often you need a prior if the effort requires extraordinary funds, so you have to say, okay is it really worth investing those funds? But my point is, that finding the answer to the nature of UAP may cost us much less than we already spent in the search for dark matter. We haven’t found anything. We don’t know where the dark matter is. We spent hundreds of millions of dollars and at a cost lower than that, maybe by an order of magnitude, we can try and figure out the nature of UAP. So given the price tag, let’s not make any assumptions. Let’s just do it and figure it out.

Lucas Perry: If these are extraterrestrial in origin, one might expect that they are here for probing or information gathering. So you said there are reports going back 20 years, if they are extraterrestrial in origin who knows how long they’ve been here. They could have been sent out through nanoscale shots out into the cosmos that land, and then grow, and replicate on some planet, and act as scouts. So if this were the case, that they were here as information gathering probes, one might wonder why they don’t use much more advanced technology. So for example, why not use nanotechnology that we would have no hope of detecting? In one report for example though, the pilot explains it following him and they’re kind of like… It comes right in front of him and then it disappears, so that disappearing seems a bit more like magic, right? Any sufficiently powerful technology is indistinguishable to magic to a less advanced civilization. But the other characteristics seem maybe like 100 or 200 years away of human technological advancement, so what’s up with that?

Avi Loeb: Well, yeah so for us to figure out what’s going on we need more data and it may well be that there are lots of things happening that we haven’t yet realized, because they are done in a way that is very subtle, that we cannot really detect easily, because our technologies are quite limited to the century and a half that we developed them. So you are right, there may be a hidden reality that we are not aware of, but we are seeing traces of things that attract our attention. That’s why when we see something intriguing we should dig into that. It could be clues for something that we have never imagined.

So for example, if you were to present a cellphone to a caveman, obviously the cellphone would be visible to the caveman and the caveman would think, oh it’s probably a rock, a shiny rock. So the caveman will recognize part of reality, that there is an object reflecting light, that is a bit more shiny than a typical rock because the caveman is used to playing with rocks. But the caveman, initially at least, will not figure out the features on the cellphone and the fact that he can speak to other people through this rock. That’s what will take us a while to be educated about. And the question is, among the things that are happening around us, which fraction are we aware of with the correct interpretation? And maybe we are not.

Lucas Perry: Moving on a bit here, so if these UAPs, or Oumuamua itself, or some new interstellar object that we were able to find were fairly conclusively shown to be extraterrestrial technology, what do you think our response should be? It seems like on one hand this would clearly potentially be an existential threat, which then makes it relevant to the Future of Life Institute. On the other hand, it’s likely that we could do nothing to counter such a threat. We couldn’t even counter humanity probably 50 years from now if we had to defend ourselves against a 50 year old, wiser, more technologically advanced version of ourself. And on cosmological timescales you would expect that even a 1,000, 2,000 year lead would be pretty common, but also indefensible. So there’s a sense that also an antagonistic attitude would probably make things worse, but also that we couldn’t do anything. So how do you think humanity should react?

Avi Loeb: The question of intent is indeed the next question after you identify an object that appears to be of extraterrestrial technological origin. We should all remember the story about the Trojan horse that looked very innocent to the citizens of Troy, but ended up serving a different purpose. That of course implies that we should collect as much evidence as possible about the objects that we find at first and see how they behave, what kind of information they are seeking, how do they respond to our actions, and ultimately we might want to engage with them. But I should mention, if you look at human history, nations that traded with each other benefited much more than nations that went into war with each other. And so a truly intelligent species might actually prefer to benefit from the interaction with us rather than kill us or destroy us, and perhaps take advantage of the resources, use whatever we are able to provide them with. So it’s possible that they are initially just spectators trying to figure out what are the things that they can benefit from.

But from our perspective, we should obviously be suspicious, and careful, and we should speak in one voice. Humanity as a whole, there should be an international organization perhaps related to the United Nations or some other entity that makes decisions about how to interact with whatever we find. And we don’t want one nation to respond in a way that would not represent all of humanity because that could endanger all of us. In that forum that makes decisions about how to respond, there should be of course physicists that figure out the physical properties of these objects and there should be policymakers that can think about how best to interact with these objects.

Lucas Perry: So you also mentioned earlier that you were talking with private funders about potentially coming up with an action plan or a project for getting more evidence and data on these objects. So I guess, there’s a two part question here. I’m curious if you could explain a little bit about what that project is about and more generally what can the scientific, non-governmental, and amateur hobbyist communities do to help investigate these phenomena? So are there productive ways for citizen scientists and interested listeners to contribute to the efforts to better understand UAPs?

Avi Loeb: Well, my hope is to get a high resolution photograph. It’s a very simple thing to desire. We’re not talking about some deep philosophical questions here. If we had a megapixel image, an image with a million resolution elements of an object, if it has a size of a meter that would mean each pixel is a millimeter in size, the size of the head of a pin, you can pretty much see all the details on the object and try and figure out, reverse engineer what it’s meant to do and whether it’s human made or not. So even a kid can understand my ambition. It’s not very complicated. Just get a megapixel image of such an object. That’s it.

Lucas Perry: They seem common enough that it wouldn’t be to difficult if the-

Avi Loeb: Well, the issue is not how common they are but what device you are using to image them, because if you use an iPhone the aperture on the iPhone will give you only fuzzy image. What you need is a meter sized telescope collecting the information and resolving an object of a meter size at the distance of a kilometer down to a millimeter resolution.

Lucas Perry: Right, right. I mean that Navy pilots, for example, have reported seeing them every day for years, so if we had such a device then you wouldn’t have to wait too long to get a really good picture of it.

Avi Loeb: So that’s my point, if these are real objects we can resolve them, and that’s what I want to have, a high resolution image. That’s all. And it will not be classified because it’s being taken by off the shelf instruments. The data will be open and here comes the role that can be played by amateurs, once the data is available to the public anyone can analyze it. Nothing is classified about the sky. We can all look up and if I get that high resolution image, believe me that everyone will be able to look at it.

Lucas Perry: Do you have a favorite science fiction book? And what are some of your favorite science fiction ideas?

Avi Loeb: Well, my favorite film is Arrival and in fact, I admired this film long ago, but a few months ago the producer of that film had a Zoom session with me to tell me how much he liked my book Extraterrestrial. And I told him, “I admired your film long before you read my book.” The reason I like this film is because it deals with the deep philosophical question of how to communicate with an alien culture. In fact, even the medium through which the communication takes place in the film is unusual and the challenge is similar to code breaking, sort of like the project that Alan Turing led during the Second World War of the enigma, trying to figure out, to break the code of the Nazis. So if you have some signal and you want to figure out the meaning of it, it’s actually a very complex challenge depending on how the information is being encoded. And I think the film addresses it in a very genuine and original fashion and I liked it a lot.

Lucas Perry: So do you have any last minute thoughts or anything you’d just really like to communicate to the audience and the public about UAPs, these reports, and the need to collect more evidence and data for figuring out what they are?

Avi Loeb: My hope is that with a high resolution image we will not only learn more about the nature of UAP but change the culture of the discourse on this subject. And I think that such an image would convince even the skeptics, even people that are currently ridiculing it, to join the discussion, the serious discussion about what all of this means.

Lucas Perry: And if there are any private funders or philanthropists listening that are interested in contributing to the project to capture this data, how is it best that they get in contact with you?

Avi Loeb: Well, they can just send me an email to aloeb@cfa.harvard.edu and I would be delighted to add them to the group of funders that are currently showing interest in it.

Lucas Perry: All right, thank you very much Avi.

Avi Loeb: Thank you for having me.

Avi Loeb on ‘Oumuamua, Aliens, Space Archeology, Great Filters, and Superstructures

  • Whether ‘Oumuamua is alien or natural in origin
  • The culture of science and how it affects fruitful inquiry
  • Looking for signs of alien life throughout the solar system and beyond
  • Alien artefacts and galactic treaties
  • How humanity should handle a potential first contact with extraterrestrials
  • The relationship between what is true and what is good

3:28 What is ‘Oumuamua’s wager?

11:29 The properties of ‘Oumuamua and how they lend credence to the theory of it being artificial in origin

17:23 Theories of ‘Oumuamua being natural in origin

21:42 Why was the smooth acceleration of ‘Oumuamua significant?

23:35 What are comets and asteroids?

28:30 What we know about Oort clouds and how ‘Oumuamua relates to what we expect of Oort clouds

33:40 Could there be exotic objects in Oort clouds that would account for ‘Oumuamua

38:08 What is your credence that ‘Oumuamua is alien in origin?

44:50 Bayesian reasoning and ‘Oumuamua

46:34 How do UFO reports and sightings affect your perspective of ‘Oumuamua?

54:35 Might alien artefacts be more common than we expect?

58:48 The Drake equation

1:01:50 Where are the most likely great filters?

1:11:22 Difficulties in scientific culture and how they affect fruitful inquiry

1:27:03 The cosmic endowment, traveling to galactic clusters, and galactic treaties

1:31:34 Why don’t we find evidence of alien superstructures?

1:36:36 Looking for the bio and techno signatures of alien life

1:40:27 Do alien civilizations converge on beneficence?

1:43:05 Is there a necessary relationship between what is true and good?

1:47:02 Is morality evidence based knowledge?

1:48:18 Axiomatic based knowledge and testing moral systems

1:54:08 International governance and making contact with alien life

1:55:59 The need for an elite scientific body to advise on global catastrophic and existential risk

1:59:57 What are the most fundamental questions?


See here for information on the Podcast Producer position

See the Office of the Director of National Intelligence report on unidentified aerial phenomena here


Lucas Perry: Welcome to the Future of Life Institute Podcast. I’m Lucas Perry. Today’s episode is with Avi Loeb, and in it, we explore ‘Oumuamua, an interstellar object that passed through our solar system and which is argued by Avi to potentially be alien in origin. We explore how common extraterrestial life might be, how to search for it through the space archaeology of bio and techno-signatures they might create. We also get into Great Filters and how making first contact with alien life would change human civilization.

This conversation marks the beginning of the continuous uploading of video content for all of our podcast episodes. For every new interview that we release, you will also be able to watch the video version of each episode on our YouTube channel. You can serach for Future of Life Institute on YouTube to find our channel or check the link in the description of this podcast to go directly to the video version of this episode. There is also bonus content to this episode which has been released speararetly on both our audio and visual feeds.

After our initital interview, the U.S. government released a report on UFOs, otherwise now known as UAPs, titled “Prelimiary Assessment: Unidentified Aerial Phenomena”. Given the release of this report and the relevance of UFOs to ‘Oumuamua, both in terms of the culture of science surrounding UFOs and their potential relation to alien life, I sat down to interview Avi for a second time to explore his thoughts on the report as well as his assessment of unidentified aerial phenomena. You can find this bonus content wherever you might be listening.

We’re also pleased to announce a new opportunity to join this podcast and help make existential risk outreach content. We are currently looking to hire a podcast producer to work on the editing, production, publishing, and analytics tracking of the audio and visual content of this podcast. You would be working directly with me, and the FLI outreach team, to help produce, grow, and evolve this podcast. If you are interested in applying, head over to the “Careers” tab on the FutureofLife.org homepage or follow the link in the description. The application deadline is July 31st, with rolling applications accepted thereafter until the role is filled. If you have any questions, feel free to reach out to socialmedia@futureoflife.org. 

Professor Loeb received a PhD in plasma physics at the age of 24 from the Hebrew University of Jerusalem and was subsequently a long-term member at the Institute for Advanced Study in Princeton, where he started to work in theoretical astrophysics. In 1993, he moved to Harvard University where he was tenured three years later. He is now the  and is a former chair of the Harvard astronomy department. He also holds a visiting professorship at the Weizman Institute of Science and a Sackler Senior Professorship by special appointment in the School of Physics and Astronomy at Tel Aviv University. Loeb has authored nearly 700 research articles and four books. The most recent of which is “Extraterrestial: The First Sign of Intelligent Life Beyond Earth”. This conversation is centrally focused on the contents of this work. And with that, I’m happy to present this interview with Avi Loeb.

To start things off here, I’m curious if you could explain what ‘Oumuamua’s wager is and what does it mean for humanity in our future?

Avi Loeb: ‘Oumuamua was the first interstellar object that was spotted near earth. And by interstellar, I mean an object that came from outside the solar system. We knew that because it moved too fast to be bound to the sun. It’s just like finding an object in your backyard from the street. And this saves you the need to go to the street and find out what’s going on out there. In particular, from my perspective, it allows us to figure out if the street has neighbors, if we are the smartest kid on the block, because this object looked unusual. It didn’t look like any rock that we have seen before in the solar system. It exhibited a very extreme shape because it changed the amount of reflected sunlight by a factor of 10 as it was tumbling every eight hours.

It also didn’t have a cometary tail. There was no gas or dust around it, yet it showed an excess push away from the sun. And the only possible interpretation that came to my mind was a reflection of sunlight. And for that, the object had to be very thin, sort of like a sail, but being pushed by sunlight rather than by the wind. This, you often find on a boat. And the nature doesn’t make sail, so in a scientific paper, we propose that maybe it’s artificial in origin. And since then in September 2020, there was another object found that was pushed away from the sun by reflecting sunlight. And without the cometary tail, it was discovered by the same telescope in Hawaii, Pan-STARRS, and was given the name 2020 SO. And then, the astronomers realized actually it’s a rocket booster that we launched in 1966 in a lunar landing mission. And we know that this object had very thin walls, and that’s why it had a lot of area for its mass and it could be pushed by reflecting sunlight.

And we definitely know that it was artificial in origin, and that’s why it didn’t show cometary tail because we produced it. The question is, who produced ‘Oumuamua? And my point is that just like Blaise Pascal, the philosopher, argued that we cannot ignore the question of whether God exists, because Pascal was a mathematician and he said, okay, logically the two possibilities either God exists or not. And we can’t ignore the possibility that God exists because the implications are huge. And so, my argument is very similar. The possibility that ‘Oumuamua is a technological relic carries such great consequences for humanity, that we should not ignore it. Many of my colleagues in academia dismiss that possibility. They say we need extraordinary evidence before we even engage in such a discussion. And my point is requiring extraordinary evidence is a way of brushing it aside.

It’s a sort of a self-fulfilling prophecy if you’re not funding research that looks for additional evidence, it’s sort of like stepping on the grass and claiming the grass doesn’t grow. Because for example, to the tape gravitational waves required an investment of $1.1 billion by the National Science Foundation. We would never discover gravitational waves unless we invest that amount. To search for dark matter, we invested hundreds of millions of dollars so far. We didn’t find what the dark matter is. It’s a search in the dark. But without the investment of funds, we will never find. So on the one hand, the scientific community puts almost no funding towards the search for technological relics, and at the same time argues all the evidence is not sufficiently extraordinary for us to consider that possibility in the first place. And I think that’s a sign of arrogance. It’s a very presumptuous statement to say, we are unique and special. There is nothing like us in the universe.

I think a much more reasonable down to earth kind of approach is a modest approach. Basically saying, look, the conditions on earth are reproducing in tens of billions of planets within the Milky Way galaxy alone. We know that from the capital satellite, about half of the sun-like stars have a planet the size of the earth, roughly at the same separation. And that means that not only we are not at the center of the universe, like Aristotle argued, we are also what we find in our backyard is not privileged. There are lots of sun-earth systems out there. And if you arrange for similar circumstances, you might as well get similar outcomes. And actually most of the stars formed billions of years before the sun. And so that to me indicates that there could have been a lot of technological civilization like ours that launched equipment into space just like we launched the Voyager 1, Voyager 2, New Horizons, and we just need to look for it.

Even if these civilizations are dead, we can do space archeology. And what they mean by that is when I go to the kitchen and I find an ant, I get alarm because there must be many more ants out there. So we found ‘Oumuamua, to me, it means that there must be many more out there and weird objects that do not look like a comet or an asteroid that we have seen before within the solar system. And we should search for them. And for example, in a couple of years, there would be the Vera Rubin Observatory that would be much more sensitive than the Pan-STARRS telescope and could find one such ‘Oumuamua like object every month. So when we find one that approaches us and we have an alert of a year or so, we can send a spacecraft equipped with a camera that will take a close up photograph of that object and perhaps even land on it, just like OSIRIS-REx landed on the asteroid Bennu recently and collected a sample from it, because they say a picture is worth a thousand words.

In my case, a picture is worth 66,000 words. The number of words in my book. If we had the photograph, I will need to write the book. It would be obvious whether it’s a rock or an artificial object. And if it is artificial and we land on it, it can read off the label made on Planet “X” and even import the technology that we find there to earth. And if it’s a technology representing our future, let’s say a million years into our future, it will save us a lot of time. It will give us a technological leap and it could be worth a lot of money.

Lucas Perry: So, that’s an excellent overview. I think of a really good chunk of the conversation, right? So there’s this first part of an interstellar object called ‘Oumuamua, entering the solar system in 2017. And then there are lots of parameters about and properties of this object, which are not easily or readily explainable as an asteroid or as a comet. Some of these things that we’ll discuss are for example, its rotation, its brightness variation, its size, its shape, how it was accelerating on its way out. And then the noticing of this object is happening in a scientific context, which has some sense of arrogance of not being fully open to exploring hypotheses that seem a bit too weird or too far out there. People are much more comfortable trying to explain it as some kind of like loose aggregate of a cosmic dust bunny or other things which don’t really fit or match the evidence.

And so then you argue that if we look into this with epistemic humility, then if we follow the evidence, it takes us to having a reasonable amount of credence that this is actually artificial in origin rather than something natural. And then that brings up questions of other kinds of life, and the Drake equation, and what it is that we might find in the universe, and how to conduct space archeology. So to start off, I’m curious if you could explain a bit more of these particular properties that ‘Oumuamua had and why it is that a natural origin isn’t convincing to you?

Avi Loeb: Right. I basically follow the evidence. I didn’t have any agenda. And in fact, I worked on the early universe and the black holes throughout most of my career, and then came along this object that was quite unusual. A decade earlier, I predicted how many rocks from other stars should we expect to find. And that was the first paper predicting that. And we predicted the Pan-STARRS telescope that discovered the ‘Oumuamua will not find anything. And the mere detection of ‘Oumuamua was a surprise by all this with magnitude, I should say. And it is still a surprise given what we know about the solar system, the number of rocks that the solar system produce. But nevertheless, that was the first unusual fact, but it still allowed for ‘Oumuamua to be a rock. And then, it didn’t show any cometary tail. And the Spitzer Space Telescope then put very tight limits on any carbon-based molecules in its vicinity or any dust particles.

And it was definitely clear that it’s not a comet because if you wanted to explain the excess push that it exhibited away from the sun through cometary evaporation, you needed about 10% of the mass of this object to be evaporated. And that’s a lot of mass. We would have seen it. The object size is of over the size of the football field, the 100 to 200 meters. And we would see such evaporation easily. So, that implied that it’s not a comet. And then if it’s not the rocket effect that is pushing it through evaporation, the question arose as to what actually triggers that push. And the suggestion that we made in the paper is that it’s the reflection of sunlight. And for that to be effective, you needed the object to be very thin. The other aspect of the object that was unusual is that as it was tumbling, every eight hours, the amount of sunlight reflected from it changed by a factor of 10.

And that implied that the object has an extreme shape, most likely pancake-shaped, flat and not cigar-shaped. Depiction of the object that’s cigar was based on the fact that projected on the sky as it was tumbling, the area that it showed us changed by a factor of 10. So then of course, if you look at the piece of paper tumbling in the wind and you look at it when it’s sideways, it does look like a cigar, but intrinsically it’s flat. And that is at the 90% confidence when trying to model the amount of light reflected from it as it was tumbling. The conclusion was at the 90% confidence that it should be pancake-shaped, flat, which again is unusual. You don’t get such objects very often in the context of rocks. And the most that we have seen before was of the order of a factor of three in length versus width. And then came the fact that it originated from a special frame of reference called the local standard of rest, which is sort of like the local parking lot of the Milky Way galaxy.

If you think about it, the stars are moving relative to each other in the vicinity of the sun, just like cars moving relative to each other in the center of a town. And then there is a parking lot that you can get to when you average over the motions of all of the stars in the vicinity of the sun, and that is called the local standard of rest. And ‘Oumuamua originated at rest in that frame. And that’s very unusual because only one in 500 stars is so much at rest in that frame as ‘Oumuamua was. So firstly, it tells you it didn’t originate from any of the nearby stars. Also, not likely from any of the far away stars because they are moving even faster relative to us, if they’re far away because of the rotation around the center of the Milky Way galaxy.

So it was not a natural result yet, a very small likelihood to have an object that is so rare. But then, or sort of like a buoy sitting at rest on the surface of the ocean and the sun bumped into it like a giant ship. And the question is if it’s artificial in origin, why would it originate from that frame? And one possibility is that it’s a member of objects on a grid that’s for navigation purposes. If you want to know your coordinates as you’re navigating an interstellar space, you find your location relative to this grid. And obviously you want those objects to be stationary, to be addressed relative to the local frame of the galaxy. And another possibility is that it’s a member of relay stations for communication. So to save on the power needed for transmission of signals, you may have relay stations like we have on earth and it’s one of them.

We don’t know the purpose of this object because we don’t have enough data on it. That’s why we need to find more of the same. But my basic point is there were six anomalies of this object that I detail in my book, Extraterrestrial, and I also wrote about in Scientific American. And these six anomalies make it very unusual. If you assign a probability of 1% to the object having each of these anomalies, when you multiply them, you get the probability of one in a trillion that this object is something that we have seen before. So clearly, it’s very different from what we’ve seen before. And response from the scientific community was to dismiss the artificial origin. And there were some scientists that took the scientific process more seriously and tried to explain the origin of  from a natural source. And they suggested four possibilities after my paper came out.

And one of them was maybe it’s a hydrogen iceberg, a chunk of frozen hydrogen that we’ve never seen before by the way. And then the idea is that when hydrogen evaporates, you don’t see the cometary tail because it’s transparent. The problem with that idea is that hydrogen evaporates very easily. So, we showed in a follow-up paper that such a chunk of frozen hydrogen the size of a football field would not survive the journey through interstellar space from its birth site to the solar system. And then there was another suggestion, maybe it’s a nitrogen iceberg that was chipped off the surface of a planet like Pluto. And then we showed in a follow-up paper that in fact you need more mass in heavy elements than you find in all the stars in the Milky Way galaxy by orders of magnitude more just to have a large enough population of nitrogen icy objects in space to explain the discovery of ‘Oumuamua.

And the reason is that there is a very thin layer of nitrogen, solid nitrogen on the surface of Pluto. And that makes a small fraction of the mass budget of the solar system. And so you just cannot imagine making enough chunks, even if you rip off all the nitrogen on the surface of exo-Plutos. It just doesn’t work out this scenario. And then there was a suggestion, maybe it’s a dust bunny as you mentioned it, a cloud of dust particles very loosely bound. And it needs to be a hundred times less dense than air so that when reflecting sunlight, it will be pushed like a feather. And the problem with that idea is that such a cloud would get heated by hundreds of degrees when it gets close to the sun and they would not maintain its integrity. So, that also has a problem.

And the final suggestion was maybe it’s a fragment, a shrapnel from a bigger object that pass close to a star. And the problem with that is the chance of passing close to a star is very small, most objects do not. So, why should we see the first interstellar object is belonging to that category? And the second is when you tidally disrupt a big object when passing through nearest star, the fragments usually get elongated and not pancake-shaped. You get often a cigar-shaped object. So, all of these suggestions have major flows. And my argument was simple. If it’s nothing like we have seen before, we better leave on the table the possibility that it’s artificial. And then, take a photograph of future objects that appears weirdest as this one.

Lucas Perry: So you mentioned the local standard of rest, which is the average velocity of our local group of stars. Is that right?

Avi Loeb: Yes. Well, it’s the frame that you get to after you average over the motions of all the stars relative to the sun, yes.

Lucas Perry: Okay. And so ‘Oumuamua was at the local standard of rest until the sun’s gravitation pulled it in, is that right?

Avi Loeb: Well, no. So the way to think of it, it was sitting at rest in that frame and just like buoy on the surface of the ocean. And then the sun happened to bump into it, the sun simply intercepted it along. And as a result, gave it a kick just like a ship gives a kick to a buoy. The sun acted on it through its gravitational force primarily. And then in addition, there was this excess push which was a smaller fraction of the gravitational force, just a fraction of a percent.

Lucas Perry: Right. And that’s the sun pushing on it through its suspected large surface area and structure.

Avi Loeb: Yeah. So in addition to gravity, there was an extra force acting on it, which was a small correction to the force of gravity, the other 10%. But it’s still, it was detected at very high significance because we monitored the motion of ‘Oumuamua. And to explain this force given that there was no cometary evaporation, you needed a thin object. And as I said, there was another thin object discovered in September 2020 called , that also exhibited an excess push by reflecting sunlight. So, it doesn’t mean necessarily that ‘Oumuamua was a light sail. It just means that it had the large area for its mass.

Lucas Perry: Can you explain why the smooth acceleration of ‘Oumuamua is significant?

Avi Loeb: Yeah. So what we detected is an excess acceleration away from the sun that declines inversely with distance squared in a smooth fashion. And first of all, the inverse-square law is indicative of a force that acts on the surface of the object. And the reflection of sunlight is exactly giving you that. And the fact that it’s smooth cannot be easily mimicked by cometary evaporation because often you had jets. These are spots on the surface of a comet from where the evaporation takes off. And that introduces jitter as the object tumbles, there is a jitter introduced to its motion because of the localized nature of these jets that are pushing it. You can think of the jets as the jets in a plane that push the airplane forward by ejecting gas backwards. But in the case of a comet, the comet is also tumbling and spinning.

And so, that introduces some jitter because the jets are exposed to sunlight at different phases of the spin of the object. And moreover, beyond a certain distance, water does not sublimate, does not evaporate anymore. You have water ice on the surface and beyond a certain distance, it doesn’t get heated enough to evaporate. So the push that you get from cometary evaporation has a sharp cutoff beyond a certain distance, and that was not observed. In the case of ‘Oumuamua, there was a smooth push that didn’t really cut off, didn’t show an abrupt change at the distance where water ice would stop evaporate. And so, that again is consistent with the reflection of sunlight being the origin of the excess push.

Lucas Perry: Can you explain the difference between comets and asteroids?

Avi Loeb: Yeah. So, we’re talking about the bricks that were left over from the construction project of the solar system. So the way that the planets form is that first you make a star like the sun, and you make it from a cloud of gas that condenses and collapses under the influence of itself gravity, its own gravitational force contracted and it pulls, and makes a star in the middle. But some of the gas has rotation around the center. And so when you make a star like the sun, a small fraction of the gas of the other for few percent or so remains in the leftover disks around the star that was just formed. And that debris of gas in the disks is the birthplace of the planets. And that disks of gas that is leftover from the formation process of the sun of course includes hydrogen and helium, the main elements from which the sun is made, but also includes heavy elements.

And they condensed in the mid-plane of the disks and make dust particles that stick to each other, get bigger and bigger over time. And they make the so-called planetesimals. These are the building blocks, the bricks that come together in making planets like the earth or the core of Jupiter that created also hydrogen and helium around the central rocky region. So, the idea is that you have all these bricks that just like Lego pieces make up the planets. And some of them get scattered during the formation process of the planets and they remain as rocks in the outer solar system. So, the solar system actually extends a thousand times farther than the location of the most distant planet in a region called the  that extends to a 100,000 times the earth-sun separation. And that is a huge volume. It goes halfway to the nearest star.

So in fact, if you imagine each star having an Oort cloud of these bricks, these building blocks that were scattered out of the construction process of the planets around the star, then these Oort clouds are touching each other, just like densely packed billiard balls. So just imagine the spherical region of planetesimals, these rocks. And so, comets are those rocks that are covered with water ice. So since they’re so far away from the sun, the ice freezes, the water freezes on their surface. But some of them have orbits that bring them very close to the sun. So when they get close to the sun, the water ice evaporates and creates a cloud of gas, a water vapor and some dust that was embedded in this rock that creates this appearance of a cometary tail. So what you see is the object is moving and then its surface layers get heated up by absorbing sunlight and the gas and dust evaporate and create this halo around the object and a tail, but always points away from the sun because it’s calmed by the solar wind, the wind coming from the sun.

And so you end up with a cometary tail, that’s what the comet is. Now, some rocks remain closer to the sun and are not covered with ice whatsoever. So, they’re just bare rocks. And when they get close to the sun, there is no ice that evaporates from them. These are called asteroids. And they’re just rock without any ice on the surface. And so, we see those as well. There is actually a region where asteroids, it’s called the main belt of asteroids, that’s we don’t know what the origin of that is. It could be a planet that was disintegrated, or it could be a region that didn’t quite make a planet and you ended up with fragments floating there. But at any event, there are asteroids, bare rocks without ice on them because they were close enough to the sun that the ice evaporated and we don’t have the water there.

And then these objects are also seen in the vicinity of the earth every now and then, these are called asteroids. And we see basically two populations. Now, ‘Oumuamua was not a comet because we haven’t seen a cometary tail around it. And it wasn’t an asteroid because there was this excess push. If you have a piece of rock, it will not be pushed much by reflecting sunlight because it’s area is not big enough relative to its mass. So it gets a push, but it’s too small for it to exhibit it in its trajectory.

Lucas Perry: Right. So, can you also explain how much we know about the composition of Oort clouds and specifically the shape and size of the kinds of objects there? And how ‘Oumuamua relates to our expectation of what exists in the Oort cloud of different stars?

Avi Loeb: Yeah. So, the one thing that I should point upfront is when scientists that try to attend to the anomalies of ‘Oumuamua suggests that it’s a hydrogen iceberg or a nitrogen iceberg. By the way, that notion gathered popularity in the mainstream. People said, oh, they had a sigh of relief. We can explain this object with something we know. But the truth is, it’s not something we know. We’ve never seen a nitrogen iceberg that was chipped off Pluto in our solar system. The Oort cloud does not have nitrogen icebergs that we witnessed. So claiming that ‘Oumuamua, the first interstellar object is a nitrogen iceberg or a hydrogen iceberg implies that there are nurseries out there around other stars or in molecular clouds that are completely different than the solar system in the sense that they produce most of the interstellar objects because ‘Oumuamua was the first one we discovered.

So they produce a large fraction of the interstellar objects, yet they are completely different from the solar system. It’s just like going to the hospital and seeing a baby that looks completely different than any child you have seen before. It’s your home from any child you had. And it implies that the birthplace of that child was quite different, but yet that child appears to be the first one you see. So, that’s to me an important signal from nature that you have to rethink what the meaning of this discovery is. And the other message is we will learn something new no matter what, so we need to get more data on the next object that belongs to this family. Because even if it’s a naturally produced object, it will teach us about environments that produce objects that are quite different from the ones we find in the solar system.

And that means that we miss something about the nature. And even if it’s natural in origin, we learn something really new in the process of gathering this data. So we should not dismiss this object and say, business as usual, we don’t have to worry about it, rather we should attempt to collect as much data as possible on the next weird object that comes along. I should say there was a second interstellar object discovered by an amateur astronomer from Russia that called Gennady Borisov. And it was given the name Borisov discovered in 2019. That one looked just like a comet. And I was asked, does that convince you that ‘Oumuamua was also natural because this one looks exactly like the comets we have seen? And I reclined, when you go along the beach and most of the time you see rocks and suddenly you see a plastic bottle. And after that you see rocks, the fact that you found rocks afterwards doesn’t make the plastic bottle a rock.

Each object has to be considered on its own merit. And therefore, it makes ‘Oumuamua even more unusual. The fact that we see Borisov as a natural comet. So in terms of the object that come from the Oort cloud, our own Oort cloud, there is a size distribution that there are objects that are much smaller than ‘Oumuamua and objects that are much bigger. And of course, the bigger objects are more rare. And then roughly speaking, there is equal amount of mass per logarithmic size bin. So, there are many more small objects. And most of them we can’t see because ‘Oumuamua was roughly at the limit of our sensitivity with Pan-STARRS. And that means that objects much smaller than the size of a football field cannot be noticed within a distance comparable to the distance to the sun. The sun acts as a lamppost that illuminates the darkness around us.

And so, an object is detected when it reflects enough sunlight for us to detect with our telescopes. And so small objects do not reflect enough sunlight, and we will notice them. But I calculated that in fact, if there are probes moving very fast through the solar system, let’s say at a fraction of the speed of light that were sent by some alien civilizations, we could detect the emission from them, the infrared emission from them with the James Webb Space Telescope. They would move very fast across our sky, so we just need to be ready to detect them.

Lucas Perry: Do you think given our limited knowledge of Oort clouds that there are perhaps exotic objects or rare objects, which we haven’t encountered yet, but that are natural in origin that may account for ‘Oumuamua?

Avi Loeb: Of course, there could be. As I mentioned, there are people suggested the hydrogen iceberg and nitrogen iceberg, dust bunny. These were suggestions that were already made and each of them has its own challenges. And it could be something else, of course. And the way to find out, that’s the way science operates. The science is guided by evidence by collecting data. And the way science should be done is you leave all possibilities on the table and then you collect enough data to rule out all but one interpretation that looks most plausible. And so, my argument is we should leave the artificial origin possibility on the table, because all the other possibilities that were contemplated invoke something that we’ve never seen before. So, we cannot argue based on speculations that it’s something that we’ve never seen before. We cannot argue that proves the point that it’s not artificial. So, it’s a very simple point that I’m making, and I’m arguing for collecting more data. I mean, I would be happy to be proven wrong, but it’s not artificial in origin, and then move on. The point is that science is not done by having a prejudice, knowing the answer in advance. It’s done by collecting data, and the mistake that was made by the philosophers during Galileo’s time is not to look through his telescope and argue that they know that the sun moves around the Earth. And that only maintained their ignorance.

The reality doesn’t care whether we ignore it. The Earth continued to move around the sun. If we have neighbors that exists out there, and it doesn’t really matter whether we shut down the curtains on our windows and claim, “No, we’re unique and special, and there is nobody out there on the street.” The fact that we say that, we can get a lot of likes on Twitter saying that, and then we can ridicule anyone that argues differently, but that would not change the fact whether we have neighbors or not. That’s an empirical fact. And, in order for us to improve our knowledge of reality, I’m talking about reality, not about philosophical arguments, just figuring out whether we have neighbors, whether we are the smartest kid on the block, that’s within the realm of science, and finding out the answer to this question is not a matter of debate.

It’s a matter of collecting evidence. But of course, if you are not willing to find wonderful things, you will never discover them. So, my point is, we should consider this possibility as real, as very plausible, as mainstream activity, just like the search for dark matter or the search for gravitational waves. We exist. There are many planets out there just like the Earth. Therefore, we should search for things like us that existed or exist on them. That’s a very simple assumption to make, an argument to make, and to me, it sounds like this should be a mainstream activity. But then, I realize that my colleagues do not agree, and I failed to understand this dismissal, because it’s a subject of great interest to the public, and the public fund science. So, if you go back a thousand years, there were people saying the human body has a soul, and therefore anatomy should be forbidden.

So imagine if scientists would say, “Oh, this is a controversial subject. The human body could have a soul. We don’t want to deal with that, because some people are claiming that we should not operate the human body,” where would modern medicine be? My argument is, if science has the tool to address the subject of great interest to the public, we have an obligation to address it and clear it up. Let’s do it bravely, with open eyes. And by the way, there is an added bonus. If the public cares about it, there will be funding for it. So, how is it possible that the scientific community ridicules this subject, brushes it aside, claims, “We don’t want to entertain this unless we have extraordinary evidence,” yet fails to fund at a very substantial level the search for that extraordinary evidence? How is that possible in the 21st century?

Lucas Perry: So, given the evidence and data that we do have, what is your credence that ‘Oumuamua is alien in origin?

Avi Loeb: Well, I have no certainty in that possibility, but I say, it’s a possibility that should be put left on the table, with at least as high likelihood as a nitrogen iceberg or a hydrogen iceberg or a dust bunny. That’s what I consider as the competing interpretations. I don’t consider statements like, “It’s always rocks. It’s never aliens,” as valid scientific statements, because they remind me of the possibility. If you were to present a cell phone to a caveman, and the caveman is used to playing with rocks all of his life, the caveman would argue that the cell phone is just a shiny rock. And, just basing your assertions on past experience is no different than what the philosophers were arguing. We don’t want to look through Galileo’s telescope because we know that the sun moves around the Earth. So, this mistake was made over and over again, throughout human history. I would expect modern scientists to be more open-minded to thinking outside the box, to entertain possibilities that are straightforward.

And what I find is, the strange thing is not so much that there is conservatism regarding this subject. But at the same time, in theoretical particle physics, you have whole communities of hundreds of people entertaining ideas that have no experimental verification, no experimental tests in the foreseeable future whatsoever, ideas like the string theory landscape or the multiverse. Or some people argue we live in a simulation, or other people talk about supersymmetry. And awards were given to people doing mathematical gymnastics, and these studies are part of the mainstream. And I ask myself, “How is it possible that this is considered part of the mainstream and the search for technological signatures is not?” And my answer is, that these ideas provide a sandbox for people to demonstrate that they’re smart, that they are clever, and a lot of the country, the academia is about that. It’s not about understanding nature. It’s more about showing that you’re smart and getting honors and awards. And that’s unfortunate, because physics and science is a dialogue with nature. It’s a learning experience. We’re supposed to listen to nature. And the best way to listen to nature is to look at anomalies, things that do not quite line up with what we expected. And by the way, whether Oumuamua is artificial or not, that doesn’t require very fancy math. It’s a very simple fact that any person can understand. I mean, nature is under no obligation to reveal its most exciting secrets without fancy math. It doesn’t need to be sophisticated.

Aristotle had this idea of the spheres surrounding us, that we are at the center of the universe, and there are these beautiful spheres around us. That was a very sophisticated idea that many people liked, because it flattered their ego to be at the center of the universe, and it also had this very clever arrangement. But it was wrong. So, who cares how sophisticated an idea is? Who cares if the math is extremely complicated? I mean, of course, it demonstrates that you are smart if you’re able to maneuver through these complicated mathematical gymnastics. But that doesn’t mean that it’s reflecting reality. And my point is, we better pay attention to anomalies that nature gives us, than to promoting our image.

Lucas Perry: Right. So it seems like there’s this interesting difference between the extent to which the scientific community is willing to entertain ‘Oumuamua as being artificial and origin, whereas at the same time, there is a ton of theories that, at least at the moment, are unfalsifiable. Yet, here we have a theory that is simple, matches the data, and can be falsified.

Avi Loeb: Right. And the way to falsify it, I mean, it’s not by chasing ‘Oumuamua, because by now, it’s a million times fainter than it was close to the sun. But then, it’s by finding more objects that look as weird as it was. And this was the first object we identified. There must be many more. If we found this object over by serving the sky for a few years, we will definitely find more by serving the sky for a few more years, because of the Copernican principle. Copernicus discovered that we are not positioned in a special location, in a privileged location in the universe. We’re not at the center of the universe, and you can extend it also, not just space, but also time. And, when you make an observation over a few years time, the chance of these few years being special and privileged is small.

I mean, most likely, it’s a typical time, and you would find it if you were to look at the previous three years, so then the following three years… That’s the Copernican principle and I very much subscribe to it, because again, the one thing I learned from practicing astronomy over the decades was a sense of modesty. We are not special. We are not unique. We are not located at the center of the universe. We don’t have anything special in our backyard. The Earth-sun system is very common. So, that’s the message that nature gives us. And, we are born into the world like actors put on a stage. And first thing we see is the stage is huge. It’s 10 to the power 26 times larger than our body. And the second thing we see is that the play has been going on for 13.8 billion years since the big bang, and we just arrived at the end of it.

So, the play is not about us. We are not the main actors. So let’s get a sense of modesty, and let’s look for other actors that may have been around for longer than we did. There’s a technological civilization. Maybe they have a better sense of what the play is about. So, I think it all starts from a sense of modesty. My daughters, when they were young, they were at home and they had the impression that they are the center of the world, that they are the smartest, because they haven’t met anyone else outside the family. And then, when we took them to the kindergarten, they got the better sense of reality by meeting others and realizing that they’re not necessarily the smartest kid on the block. And so, I think our civilization has yet to mature, and the best way to do that is by meeting others.

Lucas Perry: So before we move on to meeting others, I’m curious if you’re willing to offer a specific credence. So, you said that there are these other natural theories, like the dust bunny and the iceberg theories. If we think of this in terms of Bayesian reasoning, what kind of probability would you assign to the alien hypothesis?

Avi Loeb: Well, the point is that these objects that were postulated for natural origin of ‘Oumuamua were never seen before. So, there is no way of assigning likelihood to something that we’ve never seen before. And it needs to be the most common object in interstellar space. So, what I would say is that, we should approach it without a Bayesian prior. Basically, we should leave all of these possibilities on the table, and then get as much data as possible on the next object that shows the same qualities as ‘Oumuamua. By these qualities, I mean, not having a cometary tail, so not being a comet, and showing an excess push away from the sun.

And as I mentioned, there was such an object, 2020 SO, but it was produced by us. So, we should just look for more objects that come from interstellar space that exhibit these properties, and see what the data tells us. It’s not a matter of a philosophical debate. That’s my point. We just need a close up photograph, and we can easily tell the difference between a rock and an artificial object. And I would argue that anyone on Earth should be convinced when we have such a photograph. So, if we can get such a photograph in the next few years, I would be delighted, even if I’m proven wrong, because we will learn something new no matter what.

Lucas Perry: So, there’s also been a lot of energy in the news around UFO sightings and UFO reports recently. I’m curious how the current news and status of UFO interest in the United States and the world, how that affects your credence of ‘Oumuamua being alien in origin, and if you have any perspective or thoughts on UFOs.

Avi Loeb: Yeah, it’s a completely independent set of facts that is underlying the discussion on UFOs. But of course, again, it’s the facts, the evidence that we need to pay attention to. I always say, “Let’s keep our eyes on the ball, not on the audience.” Because if you look at the audience, the scientists are responding to these UFO reports in exactly the same way as they responded to ‘Oumuamua. They dismiss it. They ridicule it. And, that’s unfortunate, because the scientists should ask, “Who do we have access to the data? Could we analyze the data? Could we see the full data? Or could we collect new data on these objects, so that we can clear up the mystery?” I mean, science is about evidence. It’s not about prejudice. But instead, the scientists know the answer in advance. They say, “Oh, these reports are just related to human-made objects, and that’s it.”

Now, let’s follow the logic of Sherlock Holmes. Basically, Sherlock Holmes, as I mentioned in my book Extraterrestrial, Sherlock Holmes made the statement that you put all possibilities on the table, and then, whatever remains after you sought out all the facts must be the truth. That’s the way he operated as a detective. So, that’s the way we should operate as scientists. And what do we know about the latest UFO report, from the Pentagon and Intelligence agencies? So far, a few weeks before it’s being released, we know from leaks that there is a statement that some of the objects that were found are real. Okay? They are not artifacts of the cameras. They are not illusions of the people who saw them, because they were detected by multiple instruments, including infrared cameras, radar systems, optical cameras, and a lot of people from different angles.

And, when you consider that statement coming from the Pentagon, you have to take it seriously, because it’s just the tip of the iceberg. That the data that will be released to the public, presumably, is partial, because they will never released the high quality data, because it will inform other nations of the capabilities, the kind of sensors that the US has in monitoring the sky. Okay? So, I have no doubt that a lot of data is being hidden for national security reasons, because otherwise, it will expose the capabilities of these sensors that are being routinely used to monitor the sky. But, if people that had access to the full data, and that includes officials such as former president Barack Obama, former CIA director James Woolsey and others, that saw the data, and they make the case that these objects are real, then these objects may very well be real.

Okay? And I take that at face value. Of course, as a scientist, I would like to see the full data, or collect new data. There is no difference, because science is about reproducibility of results. So, if the data is classified, I would much rather place state-of-the-art cameras that you can buy in the commercial sector, or scientific instrumentation that we can purchase, and just place those in the same locations and record the sky. The sky is not classified. In principle, anyone could collect data about the sky. So, I would argue that, if all data is classified, we should collect new data that would be open to the public. And it’s not a huge investment of funds to have such an experiment. But the point of the matter is, that we can infer if the objects are real using the scientific method, then let’s assume that they are real, like the people that saw the full data claim.

So, if they’re real, then there are three possibilities. Either they were produced, manufactured by other nations, because we certainly know what we are doing, the US. So, if they were produced by other nations, like China or Russia, then humans have the ability to produce such objects, and they cannot exceed the limits of our technology. And, if the maneuvering of these objects look as if they exceed, substantially, the limits of the technologies we possess, then we would argue it’s not made by humans, because there is no way that the secret about an advanced technology would be preserved on Earth by humans. And because it has huge benefits commercially, so it would appear in the market, in the commercial sector because you can sell it for a lot of money, or it would appear in the battlefield, if it’s being used by other nations.

And we pretty much know what humans are capable of producing. We are also probably getting intelligence on other nations. So, we know what are the limits of human technology. I don’t think we can leave that possibility vague. If there is an object behaving in a way that far exceeds what we are able to produce, then that looks quite intriguing. But the remaining possibilities are, that somehow it’s a phenomenon that occurs in the Earth atmosphere. There is something that happens that we didn’t identify before, or that these are objects that came from an extraterrestrial origin. Okay? And, once again, I make the case that, the way to make progress on this is not to appear on Twitter and claim we know the answer in advance and ridicule the other side of the argument. This is not the way by which we make progress, but rather collect better evidence, better clues and figure it out, clear up the fog.

It’s not the mystery that should be unraveled by philosophical arguments. It’s something that you can measure and get data on and reproduce with future experiments. And once we get that, we will have a clear view of what it means. And then, that’s how mysteries get resolved in science. So, I would argue, for a scientific experiment that will clear up the fog. And the way that we would not do that is if the scientific community would ridicule these reports, and the public would speculate about the possible interpretations. That’s the worst situation you can be in, because you’re basically leaving a subject of great interest to the public unresolved.

And, that’s not the right way. Again, in the 21st century, to treat the subject of interest to the public, that obviously reaches the Congress, it’s not an eyewitness on the street that says, “I saw something unusual.” It’s military personnel. We have to take it seriously, and we have to get to the bottom of it. So that’s the way I look at it. Then, it may well be that it’s not the extraterrestrial in origin, but I think the key is by finding evidence.

Lucas Perry: So, given the age of the universe and the age of our galaxy and the age of our solar system, would you be surprised if there were alien artifacts almost everywhere or in many places, but we were just really bad at finding them? Or those artifacts were really good at hiding?

Avi Loeb: No, I wouldn’t be surprised, because as I said, most of the stars formed billions of years before the sun. And, if there were technological civilizations around them, many of these stars died by now and these civilizations may have perished, but if they send equipment, that equipment may operate, especially if it’s being operated by artificial intelligence or by things that we haven’t invented yet. It may well survive billions of years and get to our environment. Now, one thing you have to realize is, when you go in the wilderness, you better be quiet. You better not make a sound, and listen, because there may be predators out there. Now, we have not been careful in that sense, because we have been broadcasting radio waves for more than a century. So, these radio signals reached a hundred light years by now.

And, if there is another advanced civilization out there with radio telescopes of the type that we possess, they may already know about us. And then, if they use chemical rockets to get back to us, it would take them a million years to traverse a hundred light years. But if they use much faster propulsion, they may be already here. And the question is, are we noticing them? There was this Fermi paradox, formulated 70 years ago by Enrico Fermi, a famous physicist, who said that, “Where is everybody?” And of course, that’s a presumptuous statement, because it assumes that we are sufficiently interesting for them to come and visit us. And, when I met my wife, she had a lot of friends that were waiting for prince charming on a white horse to make them a marriage proposal, and that never happened, and then they compromise.

We, as a civilization, would be presumptuous in assuming that we are sufficiently interesting for others to have a party in our backyard. But nevertheless, it could be that it already happened. As you said, that we didn’t notice. One thing to keep in mind is full geological activity. Most of the surface of the Earth gets mixed with the interior of the Earth, over a hundred million years time scales. So, it could be that some of the evidence was buried by the geological activity on Earth, and that’s why we don’t see it.

But the moon, for example, is like a museum, because it doesn’t have geological activity, and also, it doesn’t have an atmosphere that would burn up an object that is smaller than the size of a person, like the Earth’s atmosphere does, say, for meteors. So in principle, once we establish a sustainable base on the moon, we can regard it as an archeological site, and survey the surface of the moon to look for artifacts that may have landed, may have crashed on it. Maybe we will find a piece of equipment that we never sent, that came from somewhere else that crashed on the surface of the moon.

Lucas Perry: So, it’d be wonderful if we could pivot into Great Filters and space archeology here, but before we do that, you’re talking about the Fermi paradox and whether or not we’re sufficiently interesting to merit the attention of other alien civilizations. I wonder if interesting is really the right criteria, because if advanced civilizations converge on some form of ethics or beneficence, then whether or not we’re interesting is not perhaps the right criteria for whether or not they would reach out. We have people on earth who are interested in animal ethics, like how the ants and bees and other animals are doing. So, it could be the same case with aliens, right?

Avi Loeb: Right. I completely agree. One thing I should say… Well, actually, two things is that, first, that you mentioned before the Drake’s equation. It doesn’t apply to relics. It doesn’t apply to objects. And the Drake equation talks about the likelihood of detecting radio signals. And, that has been the method we used over the past 70 years in searching for other civilizations. And, I think it’s misguided, because in order to get a signal, it’s just like trying to have a phone conversation. You need the counterpart to be alive. And it’s quite possible that most of the civilizations are dead by now. So, that’s the Great Filter idea that there is a narrow window of opportunity for us to communicate with them. But, on the other hand, they may have sent equipment into space, and we can search for it through space archeology, and find relics from civilizations that are not around anymore, just like we find relics from cultures that existed on the surface of Earth through archeological digs.

So I think a much more promising approach to find evidence for dead civilizations is looking for objects floating in space. And, the calculation of what’s the likelihood of finding them, is completely different from the Drake equation. It resembles more the calculation of what’s the chance that you would have stumbled across a plastic bottle on the beach or on the surface of the ocean. And, you just need to know how many plastic bottles are per unit area on the surface of the ocean, and then you will know what’s the likelihood of crossing one of them. And, the same is true for relics in space. You just need to know the number of such objects per unit volume, and then you will figure out what’s your chance of bumping into one of them.

And that’s a completely different calculation than the Drake equation, which talks about receiving radio signals. This is one point that should be born. And the other point that I would like to mention is that, during our childhood, we always have a sense of adults looking over our shoulders, and then making sure that everything goes well, and they often protect us. And then, as we become independent and grow up, we encounter reality on our own. There is this longing for a higher power that overlooks our shoulder. And, that is provided by the idea of God in a religion. But interestingly enough, it’s also related to the idea of some unidentified flying objects that are looking over our shoulders, because if a UFO was identified to be of extraterrestrial origin, it may imply that there is an adult wiser than we are in the room, looking over our shoulder. The question of whether that adult is trying to protect us is still open, remains open, but we can be optimistic.

Lucas Perry: All right. So, let’s talk a little bit about whether or not there might be adults in the room. So, you defined what Great Filter was. So, when I think of Great Filters, I think of there being potentially many of them, rather than a single Great Filter. So, there’s the birth of the universe, and then you need generations of stars to fuse heavier elements. And then there’s the number of planets and Goldilocks zones. And then there’s abiogenesis or the arising of life on Earth. And then there’s moving from single to multicellular life. And then there’s intelligent life and civilization, et cetera. Right? So, it seems like there’s a lot of different places where there could be Great Filters. Could you explain your perspective on where you think the most likely Great Filters might be?

Avi Loeb: Well, I think it’s self destruction, because I was asked by Harvard Alumni, how much longer do I expect our civilization to survive? And I said, “When you look at your life, and you just select a random day throughout your life, what’s the chance that it’s the first day after you are born. That probability is tens of thousands of times smaller, than the probability that the day you select would be during your adulthood, because there are tens of thousands of days in the life of an adult.” So, we existed for about a century as an advanced technological civilization. And you ask yourself, “Okay. Well, if we are in our adulthood, which is the most probable state for us to be in?” As I mentioned before, we’re just sampling randomly a time, and most likely during your adulthood, then that means that we have only a few more centuries left, because the likelihood that we will survive for millions of years, is tens of thousands of times smaller.

It would imply that we are in the first day of our life. And that is unlikely. Now, the one caveat I have for this statement is, that the human spirit can defy all odds. So, I believe that in principle, if we get our act together, we can be an outlier, in the statistical likelihood function. And, that’s my hope. I’m an optimist. And I hope that we will get our act together. But if we continue to behave the way we are, not to care so much about the climate. You can even see it in world politics nowadays. Even when you have administrations that care about climate, they cannot really convince the commercial sector to cooperate. And, suppose our civilization is on a path to self destruction, then we don’t have more than a few centuries left. So, that is a Great Filter. And of course, there could be many other Great Filters, but that seems to me as the most serious one.

And, then you ask yourself, “Okay, so which civilization is more likely to survive?” It’s probably the dumber civilization that doesn’t create the technologies that destroy it. If you have a bunch of crocodiles swimming on the surface of a planet, they will not create an atomic weapon. They would not change the climate. So, they may survive for billions of years. Who knows? So maybe the most common civilizations are the dumb ones. But, one thing to keep in mind is that, when you create technological capabilities, you can create equipment that will reproduce itself, like Von Neumann machines, or you can send it to space. You can escape from the location.

Or you can send it to space. You can escape from the location that you were born on. And so that opens up a whole range of opportunities in space. And that’s why I say that once a civilization ventures into space, then everything is possible. Then you can fill up space with equipment that reproduces itself. And there could be a lot of plastic bottles out there. And we don’t know. We shouldn’t assume anything. We should just search for them. And ‘Oumuamua, as far as I’m concerned, was the wake up call. And the other thing I would like to say is if I imagine a very advanced civilization that understands how to unify quantum mechanics with gravity, something we don’t possess at the moment… there’s such a unification scheme that we know works… perhaps they know how to irritate the vacuum and create a baby universe that would lead to more civilizations.

So it’s just like having a baby that can make babies that can make babies, and you would get many generations as a result of that. So this could be an origin of the Big Bang. Maybe the umbilical cord of the Big Bang started in a laboratory. And by the way, it would say that intelligence, technological advance is an approximation to God because in the religious stories, God created the universe. We can imagine a technology that would create a baby universe. And then the same is true for life. We don’t know if life was seeded, the origins of life was seeded in a laboratory somewhere. And so that remains a possibility. And that’s what’s so fascinating about the search for intelligent life out there, because it may provide answers to the most fundamental questions we have, like the meaning of life.

Lucas Perry: Would you consider your argument there about human extinction? Given what we are currently observing, is that like the doomsday argument?

Avi Loeb: Yeah. Well, you can call it the doomsday. I would call it risk assessment. And then I don’t think we are statistical systems in the sense that there is no escape from a particular future, because I think that once we recognize the risk in a particular future, we can respond and avoid it. The only question is whether as a civilization, we will be intelligent enough. And frankly, I’m worried that we are not intelligent enough. And it may be just like a Darwinian principle where if you are not intelligent enough, you will not survive and we will never be admitted to the club of intelligent civilizations in the Milky Way Galaxy unless we change our behavior. And it’s yet to be been whether we will change our behavior accordingly. One way to convince people to change their behavior is to find evidence for other civilizations that didn’t, and perished as a result. That would be a warning for us, a history lesson.

Now, one caveat I should mention is we always imagined things like us. And when we go to meet someone, it’s a fair assumption to assume that that person has eyes and nose and ears the way we have. And the reason it’s a reasonable assumption is because we share the same genetic heritage as the person that we are meeting. But if you think about life on a planet that had no causal contact with Earth, it could be very different.

And so calculating the likelihood of self-destruction, the likelihood of life of one form versus another, the likelihood of intelligence, all of these very often assume something similar to us, which may not be the case. I think it might be shocking to us to find the creatures from another planet or technologies from another planet. And so my solution to this ambiguity is to be an observer. Even though I’m a theorist, I would argue, let’s be modest. Let’s not try to predict things in this context. Let’s just explore the universe. And the biggest mistake we are making over and over again is to argue about the answer before seeing the evidence. An