AI Safety Research

Michael Wellman

Lynn A. Conway Professor of Computer Science and Engineering

Professor, Electrical Engineering and Computer Science

University of Michigan

Project: Understanding and Mitigating AI Threats to the Financial System

Amount Recommended:    $200,000

Project Summary

The devastation of the 2008 financial crisis remains a fresh memory seven years later, and its effects still reverberate in the global economy. The loss of trillions of dollars in output, and associated tragedy of displacement for millions of people demonstrate in the most vivid way the crucial role of a functional financial system for modern civilization. Unlike physical disasters, financial crises are essentially information events: shocks in the beliefs and expectations of individuals and organizations–about asset values, ability of counterparties to meet obligations, etc.–that nevertheless have real consequences for everyone.

This pivotal and fragile sector also happens to be at the leading edge of autonomous computational (AI) decision making. For large classes of financial assets, trading is dominated by algorithms, or “bots”, operating at speeds well beyond the scale of human reaction times. This regime change is a fait accompli, despite our unresolved debates and generally poor understanding of its implications for fundamental market stability as well as performance and efficiency.

We propose a systematic in-depth study of AI risks to the financial system. Our goals are to identify the main pathways of concern and generate constructive solutions for making financial infrastructure more robust to interaction with AI participants.

Technical Abstract

The financial system presents a critical sector of our society, at the leading-edge of AI engagement and especially vulnerable to impact from near-term AI advances. Algorithmic and high-frequency trading now dominate financial markets, yet their implications for market stability are poorly understood. In this project we undertake a systematic investigation of how AI traders can impact market stability, and how extreme movements in securities markets in turn can impact the real economy. We develop a general framework for automated trading based on a flexible architecture for arbitrage reasoning. Through agent-based simulation combined with game-theoretic strategy selection, we search for vulnerabilities in financial markets, and characterize the conditions that enable or prevent their exploitation. A new approach to modeling complex networks of financial obligations is applied to the study of contagion between asset-pricing anomalies and panics in the broader financial system. Results from this study will be employed to design market rules, monitoring technologies, and regulation techniques that promote stability in a world of algorithmic traders.


  1. Wellman,  MP and Rajan, U. Ethical issues for autonomous trading agents. IJCAI-16 Workshop on Ethics for Artificial Intelligence, July 2016.
    • Wellman gave a presentation on this paper, as well.


  1. “Understanding and Mitigating AI Threats to the Financial System”, MP Wellman and Uday Rajan. Center for Finance, Law, and Policy. University of Michigan, 4 Jun 2015.
  2. “Do Trading Algorithms Threaten Financial Market Stability?” MP Wellman.  Conference on Interdisciplinary Approaches to Financial Stability. University of Michigan Law School, 22 Oct 2015.
  3. “Autonomous Agents: Threat or Menace?” MP Wellman. Collegiate Professorship Lecture. University of Michigan, 5 May 2016.
  4. “Autonomous Agents in FInancial Markets: Implications and Risks” MP Wellman. Machine Intelligence Research Institute Colloquium on Robust and Beneficial AI. Berkeley, CA, 15 Jun 2016.