Exploring the constraints on artificial general intelligence: A game-theoretic model of human vs machine interaction

• Published in: Mathematical social sciences Vol. 129; pp. 70 - 76
• Main Author: Ismail, Mehmet S.
• Format: Journal Article
• Language: English
• Published: Elsevier B.V 01-05-2024
• Subjects: Artificial general intelligence; Cooperation; Non-cooperative games; Superhuman performance; Non-cooperative games; Artificial general intelligence; Superhuman performance; Cooperation
• ISSN: 0165-4896; 1879-3118
• DOI: 10.1016/j.mathsocsci.2024.03.004

The potential emergence of artificial general intelligence (AGI) systems has sparked intense debate among researchers, policymakers, and the public due to their potential to surpass human intelligence in all domains. This note argues that for an AI to be considered “general”, it should achieve superhuman performance not only in zero-sum games but also in general-sum games, where winning or losing is not clearly defined. In this note, I propose a game-theoretic framework that captures the strategic interactions between a representative human agent and a potential superhuman machine agent. Four assumptions underpin this framework: Superhuman Machine, Machine Strategy, Rationality, and Strategic Unpredictability. The main result is an impossibility theorem, establishing that these assumptions are inconsistent when taken together, but relaxing any one of them results in a consistent set of assumptions. This note contributes to a better understanding of the theoretical context that can shape the development of superhuman AI.
•A game between a human vs an Artificial General Intelligence system is introduced.•An AI system is general if it is successful in both zero-sum and general-sum games.•Assumptions are Superhuman AI, Machine Strategy, Rationality, and Unpredictability.•An impossibility theorem shows that these assumptions lead to inconsistency.•The result is tight; relaxing any of the assumptions resolves the inconsistency.