Jen Semler

Many views of moral agency include, implicitly or explicitly, a consciousness requirement—namely, the claim that phenomenal consciousness is a necessary condition of moral agency. This paper casts doubt on the consciousness requirement. I argue that consciousness is not necessary for instantiating four key capacities necessary for moral agency: action, moral concept possession, responsiveness to moral reasons, and moral understanding. I defend my picture of nonconscious moral agency as a plausible account of an entity that can act for moral reasons and can be morally responsible. Lastly, I discuss broader implications of my argument, especially on the possibility of artificial moral agency.

Those who worry about a superintelligent AI destroying humanity often appeal to the instrumental convergence thesis—the claim that even if we don’t know what a superintelligence’s ultimate goals will be, we can expect it to pursue various instrumental goals which are useful for achieving most ends. In this paper, we argue that one of these proposed goals is mistaken. We argue that instrumental goal preservation—the claim that a rational agent will tend to preserve its goals—is false on the basis of the timing problem: an agent which abandons or otherwise changes its goal does not thereby fail to take a required means for achieving a goal it has. Our argument draws on the distinction between means-rationality (adopting suitable means to achieve an end) and ends-rationality (choosing one’s ends based on reasons). Because proponents of the instrumental convergence thesis are concerned with means-rationality, we argue, they cannot avoid the timing problem. After defending our argument against several objections, we conclude by considering the implications our argument has for the rest of the instrumental convergence thesis and for AI safety more generally.