Tan Zhi-Xuan

Recent theoretical work has argued that moral psychology can be understood through the lens of “resource rational contractualism.” The view posits that the best way of making a decision that affects other people is to get everyone together to negotiate under idealized conditions. The outcome of that negotiation is an arrangement (or “contract”) that would lead to mutual benefit. However, this ideal is seldom (if ever) practical given the resource demands (time, information, computational processing power) that are required. Instead, the theory proposes that moral psychology is organized around a series of resource-rational approximations of the contractualist ideal, efficiently trading off between more resource-intensive, accurate mechanisms and less. This paper presents empirical evidence and a cognitive model that test a central claim of this view: when the stakes of the situation are high, then more resource-intensive processes are engaged over more approximate ones. We present subjects with a case that can be judged using virtual bargaining—a resource-intensive process that involves simulating what two people would agree to—or by simply following a standard rule. We find that about a third of our participants use the resource-rational approach, flexibly switching to virtual bargaining in high-stakes situations, but deploying the simple rule when stakes are low. A third of the participants are best modeled as consistently using the strict rule-based approach and the remaining third as consistently using virtual bargaining. A model positing the reverse resource-rational hypothesis (that participants use more resource-intensive mechanisms in lower stakes situations) fails to capture the data.

The dominant practice of AI alignment assumes (1) that preferences are an adequate representation of human values, (2) that human rationality can be understood in terms of maximizing the satisfaction of preferences, and (3) that AI systems should be aligned with the preferences of one or more humans to ensure that they behave safely and in accordance with our values. Whether implicitly followed or explicitly endorsed, these commitments constitute what we term a preferentist approach to AI alignment. In this paper, we characterize and challenge the preferentist approach, describing conceptual and technical alternatives that are ripe for further research. We first survey the limits of rational choice theory as a descriptive model, explaining how preferences fail to capture the thick semantic content of human values, and how utility representations neglect the possible incommensurability of those values. We then critique the normativity of expected utility theory (EUT) for humans and AI, drawing upon arguments showing how rational agents need not comply with EUT, while highlighting how EUT is silent on which preferences are normatively acceptable. Finally, we argue that these limitations motivate a reframing of the targets of AI alignment: Instead of alignment with the preferences of a human user, developer, or humanity-writ-large, AI systems should be aligned with normative standards appropriate to their social roles, such as the role of a general-purpose assistant. Furthermore, these standards should be negotiated and agreed upon by all relevant stakeholders. On this alternative conception of alignment, a multiplicity of AI systems will be able to serve diverse ends, aligned with normative standards that promote mutual benefit and limit harm despite our plural and divergent values.