Rhys Southan

Those who worry about a superintelligent AI destroying humanity often appeal to the instrumental convergence thesis—the claim that even if we don’t know what a superintelligence’s ultimate goals will be, we can expect it to pursue various instrumental goals which are useful for achieving most ends. In this paper, we argue that one of these proposed goals is mistaken. We argue that instrumental goal preservation—the claim that a rational agent will tend to preserve its goals—is false on the basis of the timing problem: an agent which abandons or otherwise changes its goal does not thereby fail to take a required means for achieving a goal it has. Our argument draws on the distinction between means-rationality (adopting suitable means to achieve an end) and ends-rationality (choosing one’s ends based on reasons). Because proponents of the instrumental convergence thesis are concerned with means-rationality, we argue, they cannot avoid the timing problem. After defending our argument against several objections, we conclude by considering the implications our argument has for the rest of the instrumental convergence thesis and for AI safety more generally.

According to the metaethics of R. M. Hare, we determine morality objectively by making a moral judgment, committing to the moral principle underlying that judgment, and then logically extending that moral principle to all relevantly similar cases. This metaethical system called universal prescriptivism had a major impact on Peter Singer, whose arguments for radically improving animal welfare and alleviating global suffering frequently rely on Hare-ian appeals to logical consistency. Hare’s work in metaethics is largely rejected now, but Singer’s popularity has kept Hare’s prescriptivism alive through the many animal welfarists and effective altruists who have borrowed Singer’s style in their own logic-based calls for the obligation to reduce suffering impartially. In this paper, I will describe Hare’s metaethics, show how this has served as Singer’s own metaethics for most of his academic career, and then I will describe a problem for Hare’s system that is particularly relevant to effective altruists who have been influenced by Singer’s early writings and may be repeating the mistakes that Hare bequeathed to Singer.

Animal rights advocates have sometimes illustrated a moral problem with animal farming through a thought experiment in which aliens who are more powerful and intelligent than we are come to Earth and farm us for food. If we think it is wrong for aliens to do that to us, then we should similarly think it is wrong for us to do that to animals. This paper applies this alien analogy to how vegan humans treat non-human animals, by imagining vegan aliens who come to Earth and treat us as badly as vegans treat non-human animals, through the side effects of industrial civilization. If it is wrong for aliens to unintentionally but foreseeably harm us by building a vegan civilization on Earth, it is similarly wrong for humans to do that to animals.