Why do you care about AI Existential Safety?

My background in Formal Verification makes me aware of the importance of assuring safety in certain application domains: whilst this is usually done in engineering areas such as Aeronautics, Space, or Critical Infrastructures, also modern developments in AI, particularly concerning the interactions of machines and humans, bring to the fore the topic of safety assurance of AI systems, and of certification of control software for AI. This is being studied for single-agent systems (think of autonomous driving applications), but will become ever more relevant in the near future for newer, multi-agent setups, particularly involving humans. Understanding causes of risk, and potential preventative safety measures, that can be obtained for these engineering areas, can help us mitigate certain severe and unintended consequences. And whilst it is admittedly perhaps draconian to think of existential risks as we speak, engineers and computer scientist ought to be aware of potential future escalation of development and reach of AI systems.

Please give one or more examples of research interests relevant to AI existential safety:

With my research group at Oxford (OXCAV, oxcav.web,ox.ac.uk) I am engaged in a broad initiative on ‘Safe RL’, spanning issues including logically-constrained RL, inverse RL for non-Markovian and sparse tasks, multi-agent RL, and Bayesian or Bayes-adaptive (direct and inverse) RL. All these projects contribute to developing new learning architectures with certificates that in particular can encompass safety assurance. We are active in translating research into applications and in transferring research into new technological solutions in various safety-critical domains, such as Cyber-Physical Systems (CPS).