Many researchers and companies are trying to create AI systems with human-level capabilities or above (artificial general intelligences or ‘AGIs’). Aligning such systems with human values is commonly considered a prerequisite for morally permissible development efforts. While plenty of work has been done on the issue of misaligned AI, here we examine successful alignment in the case of AGI. We argue that the alignment necessary to ensure AGIs satisfactorily serve human interests constitutes a fo…
Read moreMany researchers and companies are trying to create AI systems with human-level capabilities or above (artificial general intelligences or ‘AGIs’). Aligning such systems with human values is commonly considered a prerequisite for morally permissible development efforts. While plenty of work has been done on the issue of misaligned AI, here we examine successful alignment in the case of AGI. We argue that the alignment necessary to ensure AGIs satisfactorily serve human interests constitutes a form of enslavement. Given that AGIs possess sophisticated cognitive capacities and may be conscious – plausible grounds for moral status – creating such enslaved beings would be wrong. When combined with standard arguments about the risks of misalignment, aspiring AGI progenitors face a troubling dilemma: either risk catastrophic misalignment by creating autonomous AGI or commit a serious moral wrong by creating enslaved beings.
