Tyler Cook

One natural motivation for designing artificial intelligence (AI) with ethical capacities is to mitigate the risk that powerful AI systems will harm us. Against this idea, some authors have argued we do not need ethical AI in order to prevent harm to humans. We simply need safe AI. In this paper, I consider an argument of this type and raise objections to it. In particular, I argue that the goal of implementing safety features in AI systems is a much more complicated task than such authors have acknowledged, and moreover I maintain that merely safe AI could be ethically problematic in numerous ways. Then, I show that a certain kind of ethical AI, which I call end-autonomous ethical AI, would be especially dangerous since their autonomous capacities would make possible numerous risks. Finally, I motivate the case for a specific category of ethical AI, namely, end-constrained ethical AI. I describe these systems as possessing whatever capacities would be necessary for satisfying ethical aims beyond safety while lacking end-autonomy. In short, the goal of this paper is to establish that end-constrained ethical AI occupy a desirable middle ground between merely safe AI and end-autonomous ethical AI because they allow us to secure more ethical goods than just safety, and they are also not as risky as end-autonomous ethical AI.

This paper concerns top-down approaches in machine ethics. It is divided into three main parts. First, I briefly describe top-down design approaches, and in doing so I make clear what those approaches are committed to and what they involve when it comes to training an AI to behave ethically. In the second part, I formulate two underappreciated motivations for endorsing them, one relating to predictability of machine behavior and the other relating to scrutability of machine decision-making. Finally, I present three major worries about such approaches, and I attempt to show that advocates of top-down approaches have some plausible avenues of response. I focus most of my attention on what I call the ‘technical manual objection’ to top-down approaches, inspired by the work of Annas (2004). In short, the idea is that top-down approaches treat ethical decision-making as being merely a matter of following some ethical instructions in the same way that one might follow some set of instructions contained in a technical manual (e.g., computer manual), and this invites sensible skepticism about the ethical wisdom of machines that have been trained on those approaches. I respond by claiming that the objection is successful only if it is understood as targeting machines that have certain kinds of goals, and it should not compel us to totally abandon top-down approaches. Such approaches could still be reasonably employed to design ethical AI that operate in contexts that include fairly noncontroversial answers to ethical questions. In fact, we should prefer top-down approaches when it comes to those types of context, or so I argue, due to the advantages I claim for them.