I. Background on FLI

The Future of Life Institute (FLI) is an independent non-profit that works to steer transformative technology towards benefiting life and away from extreme large-scale risks. We work through policy advocacy at the UN, in the EU and the US, and have a long history of grants programmes supporting such work as AI existential safety research and investigations into the humanitarian impacts of nuclear war. This current request for proposals is part of FLI’s Futures program, which aims to guide humanity towards the beneficial outcomes made possible by transformative technologies. The program seeks to engage a diverse group of stakeholders from different professions, communities, and regions to shape our shared future together.

II. Request for Proposal

Call for proposed designs for global institutions governing AI

FLI is calling for research proposals with the aim of designing trustworthy global governance mechanisms or institutions that can help stabilise a future with 0, 1, or more AGI projects. These proposals should outline the specifications needed to reach or preserve a secure world, taking into account the myriad threats posed by advanced AI. Moreover, we expect proposals to specify and justify whether global stability is achieved by banning the creation of all AGIs, enabling just one AGI system and using it to prevent the creation of more, or creating several systems to improve global stability. There is the option for proposals to focus on a mechanism not dependent on a particular scenario, or one flexible enough to adapt to 0, 1 or more AGI projects. Nonetheless, it is recommended that applicants consider in which scenario their proposed mechanism would best perform, or be most valued. In that sense, as well as pitching a particular control mechanism, each proposal is also making a case for how humanity is kept safe in a specific future scenario.

FLI’s rationale for launching this request for proposal

Reaching a future stable state may require restricting AI development such that the world has a.) no AGI projects; b.) a single, global AGI project, or c.) multiple AGI projects. By AGI, we refer to Shane Legg’s definition of ‘ASI’: a system which outperforms 100% of humans at a wide range of non-physical tasks, including metacognitive abilities like learning new skills (see grid below, taken from Legg’s paper on this). A stable state would be a scenario that evolves at the cautious timescale determined by thorough risk assessments rather than corporate competition.

a. No AGI projects

As recently argued by the Future of Life Institute’s Executive Director, Anthony Aguirre, if there is no way to make AGI – or superhuman general-purpose AI – safe, loyal, and beneficial, then we should not go ahead with it. In practice, that means that until we have a way of proving that an AGI project will not, upon completion or release, take control away from humanity or cause a catastrophe, AGI models should be prevented from being run. Equally, harnessing the benefits and preventing the risks of narrow AI systems and controllable general purpose models still requires substantial global cooperation and institutional heft, none of which currently exists. Proposals focusing on such initiatives might include mechanisms for:

• Risk analysis;

• Predicting, estimating, or evaluating the additive benefit of continuing AI capability development;

• Ensuring no parties are cheating on AGI capability development;

• Building trust in a global AI governance entity; or,

• Leveraging the power of systems to solve bespoke problems that can yield societal and economic benefits.

b. One AGI project

As Aguirre points out, even if an AGI is ‘somehow, made both perfectly loyal/subservient to, and a perfect delegate for, some operator’, that operator will rapidly acquire far too much power – certainly too much for the comfort of existing powers. He notes that other power structures ‘will correctly think… that this is an existential threat to their existence as a power structure – and perhaps even their existence period (given that they may not be assured that the system is in fact under control.)’ They may try to destroy such a capability. To avoid this imbalance or its disastrous results, the single AGI will either need to be created by a pre-existing cooperation of the great powers, or brought under the control of a new global institution right after it is produced. Again, the world lacks these institutions, and forming them will require extensive research and thought. Alongside variations on the ideas listed above, proposals for such establishments and connected questions needing answers could also include:

• Mechanisms for distributing the benefits of the centralised capabilities development efforts: how is this done from a technical perspective? Is it all of humanity? Only the signatories of an international treaty? How are economic benefits shared and is that the same or different than new knowledge?

• Mechanisms for preventing authoritarian control? How can power be centralised without corruption?

• Intellectual property structures that would enable and incentivize commercialization of breakthroughs discovered in an AGI system.

• Mechanisms for determining if a new discovery should be developed by a private entity or if it is for the common good (e.g. radical climate intervention)

• Potential governance structures for such an entity: who makes the decisions about development? Risks? How is the concentration of power kept in check and accountable to the globe? How is capture by special interests, spies, or geopolitical blocs prevented?

• Verification mechanisms to ensure no one is cheating on AGI capability development-How to prevent others from cheating? Penalties? Surveillance mechanisms? How much does this need to vary by jurisdiction?

• How is such an organization physically distributed?

• Mechanisms for ensuring the security of such an entity, preventing leaks and accidents.

• Can realpolitik support centralised global AGI development? What would it take to actually convince the major states that they wouldn’t effectively be giving up too much sovereignty or too much strategic positioning?

c. Multiple AGI projects

Similar concerns arise if multiple AGI systems emerge. A delicate balance of power must be strenuously maintained. Furthermore, with all of these scenarios there will ensue the significant dual problem of on the one hand limiting the associated risks of such powerful AI systems, and, on the other, distributing the associated benefits – of which we can expect many. Some problems along those lines in need of solutions:

• How much power should an assembly of represented states have vs. bureaucratic managers vs. scientists?

• What is an equitable, fair, and widely agreeable distribution of voting power among the represented states? Should simple- or super-majorities be required? Should Vanuatu have the same weight as Japan?

• Who gets to decide how or whether AI should influence human values? What and whose values get enshrined in a new global institution for governing AI? And if no one’s in particular, what mechanisms can help to maintain room for personal convictions, free thinking, community traditions such as religion, careful decision-making and contradicting values? (see Robert Lempert’s new RAND paper)

d. Flexible to different numbers of AGI projects

Dividing scenarios into the three above groups will hopefully yield a balanced sample of each outcome. Equally, urging primary investigators to select just one of these categories may help to encourage concrete scenarios for what a well-managed future with advanced AI could look like – in short, push them to pick only their preferred future. However, some applicants may wish to submit proposals for mechanisms flexible enough to adjust to varying numbers of AGI systems. These applicants will need to address how their mechanism can ensure stability in a world where the number of AGI systems can keep changing. Alternatively, there may be mechanisms or institutions proposed whose function is not dependent on a particular number of AGI projects. In such cases, it is still recommended that applicants consider in which scenario their proposed mechanism would best perform, or be most valued; nonetheless, we leave this option available for those cases where such a consideration proves to be unenlightening.

The success of all of these case groups depends upon a diligent and consistent consideration of the risks and benefits of AI capability development. Any increase in the power of intelligent systems must proceed in accordance with an agreed acceptable risk profile, as is done in the development of new drugs. However, it is not clear what the structure of such an organization would look like, how it would command trust, how it would evade capture, or how it could endure as a stable state. This request for proposals can be summarised as a search for this clarity. A better, more informed sense of where we wish to be in a few years, of what institutions will best place us to tackle the upcoming challenges, will be invaluable for policymakers today. Such a well-researched north star can help the world to reverse engineer and work out what governments should be doing now to take humanity in a better, safer direction.

Without a clear articulation of how trustworthy global governance could work, the default narrative is that it is impossible. This RFP is thus borne both of an assessment of the risks we face, and of sincere hope that the default narrative is wrong, a hope that if we keep it under control and use it well, AI will empower – rather than disempower – humans the world over.

Existing projects

FLI is by no means creating a new field here. Promising initiatives already in the works, which may inspire researchers applying to this program, include the following:

• FLI’s international AI governance organization proposal

• Davidad’s Open Agency Architecture

• The Oxford Martin AI Governance Initiative’s whitepaper: International Governance of Civilian AI: A Jurisdictional Certification Approach

• Conjecture’s Multinational AGI Consortium (MAGIC) – a ‘CERN for AI’

III. Evaluation Criteria & Project Eligibility

Proposals will be evaluated according to the track record of the researcher, the proposal’s originality or potential to be transformative, the potential for the proposed activity to advance knowledge of the coordination problems for mitigating AGI risk, and how convincingly the proposal accounts for the range of AGI risks.

Grants applications will be subject to a competitive process of external and confidential peer review. We intend to support several proposals. Accepted proposals will receive a one-time grant of $15,000, to be used at the researcher’s discretion. Grants will be made to nonprofit organizations, with institutional overhead or indirect costs not exceeding 15%.

IV. Application process

All applications should be submitted electronically through this form. We will accept applications internationally. But all applicants should have a nonprofit organization with which they are associated to accept the funding. We will not make grants directly to individuals.

Applications deadline: 1st April 2024.

External reviewers invited by FLI will then evaluate all the proposals according to the above criteria, and decisions will be shared by mid to late May. Completed research papers are due by 13th September.

All questions should be sent to grants@futureoflife.org.