Paul de Font-Reaulx

As AI systems progress, we rely more on them to make decisions with us and for us. To ensure that such decisions are aligned with human values, it is imperative for us to understand not only what decisions they make but also how they come to those decisions. Reasoning language models, which provide both final responses and (partially transparent) intermediate thinking traces, present a timely opportunity to study AI procedural reasoning. Unlike math and code problems which often have objectively correct answers, moral dilemmas are an excellent testbed for process-focused evaluation because they allow for multiple defensible conclusions. To do so, we present MoReBench: 1,000 moral scenarios, each paired with a set of rubric criteria that experts consider essential to include (or avoid) when reasoning about the scenarios. MoReBench contains over 23 thousand criteria including identifying moral considerations, weighing trade-offs, and giving actionable recommendations to cover cases on AI advising humans moral decisions as well as making moral decisions autonomously. Separately, we curate MoReBench-Theory: 150 examples to test whether AI can reason under five major frameworks in normative ethics. Our results show that scaling laws and existing benchmarks on math, code, and scientific reasoning tasks fail to predict models’ abilities to perform moral reasoning. Models also show partiality towards specific moral frameworks (e.g., Benthamite Act Utilitarianism and Kantian Deontology), which might be side effects of popular training paradigms. Together, these benchmarks advance process-focused reasoning evaluation towards safer and more transparent AI.

Critics have argued that expected utility theory fails as a theory of rational choice for diachronic agents who expect their preferences to change in response to temptations. According to this criticism, such agents cannot rationally commit to executing a sequence of actions, even when doing so would produce outcomes they consistently prefer. Call this the Commitment Problem for expected utility theory. In this paper, I argue that the Commitment Problem is based on a mistake. I show that expected utility maximizers who care about their future will form and execute plans, even when they are nearly certain of facing preference reversals. In doing so, they learn that they can trust themselves, and that allows their future selves to rationally pursue outcomes that they prefer even today. By modeling an agent's successive selves as players in a Bayesian reputation game, I show that this holds in most everyday circumstances where a person expects to have at least a short future that they care about. In cases where it does not hold, they should intuitively not trust themselves. I conclude that there is no Commitment Problem for expected utility theory.

Most of us intuitively take discrimination based on gender or ethnicity to be impermissible because we have a right to be treated on the basis of merit and capacity rather than e.g. ethnicity or gender. I call this suggestion the Impermissibility Account. I argue that, despite how the Impermissibility Account seems intuitive to most of us with a humanist outlook, it is indefensible. I show that well-informed discrimination can sometimes be permissible, and even morally required, meaning we cannot have a strict right not to be discriminated against. I then propose an alternative and more plausible account which I call the Fairness and Externalities Account, arguing that acts of discrimination are wrong partly because they are unfair and partly because they create harmful externalities which—analogously to pollution—there is a collective responsibility to minimize. Both of these factors are however defeasible, meaning that if the Fairness and Externalities Account is correct, then discrimination is sometimes permissible. These results are counterintuitive, and suggest that the ethics of discrimination requires further attention.