The dominant framings of existential risk from AI feature an AI that is more than human: a misaligned superintelligence, or a powerful tool in malicious hands. This essay identifies a third risk that requires neither. Grant the AI safety community everything it hopes for — no superintelligence, no misaligned goals, a perfectly obedient instrument — and suppose only that AI attains closure: the capacity to run the economy without human labour, the very capacity presupposed by the post-scarcity ut…
Read moreThe dominant framings of existential risk from AI feature an AI that is more than human: a misaligned superintelligence, or a powerful tool in malicious hands. This essay identifies a third risk that requires neither. Grant the AI safety community everything it hopes for — no superintelligence, no misaligned goals, a perfectly obedient instrument — and suppose only that AI attains closure: the capacity to run the economy without human labour, the very capacity presupposed by the post-scarcity utopia its optimists promise. I argue that closure, together with the concentrated control under which capital-intensive technology is ordinarily distributed, removes the instrumental basis of inclusive society. The two pillars on which large, inclusive societies rest — cooperation and growth — both depend on human beings remaining necessary to one another; once machines supply the primary goods that do not require a specifically human provider, the rational case for including the masses dissolves and a positive case for excluding them emerges. The result, which I call social transcendence, is not extinction or rogue takeover but an unrecoverable dystopia: a stratified world, made for humans, that no longer needs most of them. The withdrawal requires no malice, only instrumental rationality. Distinct from gradual disempowerment (systemic drift) and the intelligence curse (eroded incentives to invest in people), this account derives exclusion from individual rational choice and predicts a stratified rather than uniform outcome.