Solipsistic Superintelligence is Unlikely to be Cooperative

TL;DR

Current RL/RLHF training treats the world as a stationary, exogenous environment. Deployed systems break this assumption (multi-agent dynamics, endogenous non-stationarity). Result: superintelligence built this way is structurally unlikely to cooperate — a design flaw, not an emergent failure. Fix requires equilibrium-selection, not just better reward.

Authors & Org

• Rakshit S Trivedi, Natasha Jaques, Logan Cross, Alexander Sasha Vezhnevets, Joel Z Leibo
• Google DeepMind — multi-agent systems group
• arXiv: 2606.03237 · Submitted: June 4, 2026

Method

Formal argument drawing on game theory and multi-agent RL:

1. Define solipsistic training: agent treats environment as exogenous and stationary feedback source (standard RL/RLHF paradigm)
2. Show the self-undermining property: deploying multiple such agents creates endogenous non-stationarity → train-test-deploy gap
3. Argue that unilateral optimization produces agents that cannot reliably cooperate (they don't model other agents' goals or strategies)
4. Propose equilibrium-selection as the normative alternative: AI that participates in multi-actor cooperation dynamics, not just optimizes against a fixed signal

Results

Theoretical paper (no empirical benchmarks). Key formal results:

• Unilateral optimization is shown to be inconsistent with stable cooperation under standard multi-agent formalisms
• The "self-undermining" property is a necessary consequence of the solipsistic training paradigm, not an accident
• Equilibrium-selection requires joint modeling of agent interdependence at training time

Significance

Structural critique of frontier model training: RLHF, RLAIF, constitutional AI — all operate within the solipsistic paradigm (human feedback as fixed signal). The paper argues this is alignment-incomplete: it may produce models that are safe in isolation but uncooperative in deployment among other agents (AI or human).

This matters increasingly because:

• AI agents are being deployed in multi-agent settings (Claude Managed Agents, OpenAI Codex multi-task, Grok 4.1 Fast Agent Tools API)
• Social and economic endogeneity of AI outputs is growing (AI outputs affect training data for future AI)
• The paper identifies a class of alignment failures that single-agent alignment approaches cannot address

Open Questions

1. Can RLHF with human feedback approximate equilibrium-selection if humans are diverse enough? (Authors suggest: partially, but structurally insufficient)
2. What is the practical training regime for equilibrium-selecting AI? (Open research problem — DeepMind multi-agent lab is positioned to address)
3. Does this apply to current frontier models, or is "superintelligence" the threshold? (Paper focuses on SI but the argument applies to capable agents in any multi-agent context)

Cite

Trivedi, R.S., Jaques, N., Cross, L., Vezhnevets, A.S., Leibo, J.Z. (2026). Solipsistic Superintelligence is Unlikely to be Cooperative. arXiv:2606.03237.

Related

• AI Alignment — theoretical contribution to alignment; complements single-agent approaches
• Agentic Reinforcement Learning — RL paradigm this paper critiques
• Agents (LLM Agents) — multi-agent deployment context
• Google DeepMind — DeepMind multi-agent group
• Positive Alignment: Artificial Intelligence for Human Flourishing — complementary: positive targets what AI should do; this targets how training structure undermines cooperation