In a blog post yesterday, AI giant Anthropic sounded the alarm on massive societal risks from recursive self-improvement, and urged companies to consider slowing down or pausing development.

In March 2023, the Future of Life Institute made headlines around the world when they published an open letter calling for a similar pause in giant AI experiments, signed by giants like Elon Musk and Yoshua Bengio. It was not heeded. The letter explained what was at stake:

“Should we let machines flood our information channels with propaganda and untruth? Should we automate away all the jobs, including the fulfilling ones? Should we develop nonhuman minds that might eventually outnumber, outsmart, obsolete and replace us? Should we risk loss of control of our civilization?”

In response to Anthropic’s echo, FLI President and CEO Anthony Aguirre released the following statement:

“We are approaching a runaway to superintelligence that could threaten our shared human future. Both publicly and privately, AI companies are recognizing that a pause or slowdown in certain developmental pathways is crucial to protect lives and livelihoods everywhere. This should give everyone hope, and we stand ready to work with anyone who agrees.”