Jake Iain Stone

Progress in machine learning and artificial intelligence has spurred the widespread adoption of automated decision systems (ADS). An extensive literature explores what conditions must be met for these systems’ decisions to be fair. However, questions of legitimacy—why those in control of ADS are entitled to make such decisions—have received comparatively little attention. This paper shows that when such questions are raised theorists often incorrectly conflate legitimacy with either public acceptance or other substantive values such as fairness, accuracy, expertise or efficiency. In search of better theories, we conduct a critical analysis of the philosophical literature on the legitimacy of the state, focusing on consent, public reason, and democratic authorisation. This analysis reveals that these theories require careful consideration before being applied to legitimate ADS. Specifically, we find that consent and public reason theories can legitimate some automated decision systems, but encounter serious difficulties which limit their applicability. Democratic authorisation, though not without its own challenges, emerges as a more promising and broadly applicable means of legitimation.

Optimism about our ability to enhance societal decision‐making by leaning on Machine Learning (ML) for cheap, accurate predictions has palled in recent years, as these ‘cheap’ predictions have come at significant social cost, contributing to systematic harms suffered by already disadvantaged populations. But what precisely goes wrong when ML goes wrong? We argue that, as well as more obvious concerns about the downstream effects of ML‐based decision‐making, there can be moral grounds for the criticism of these predictions themselves. We introduce and defend a theory of predictive justice, according to which differential model performance for systematically disadvantaged groups can be grounds for moral criticism of the model, independently of its downstream effects. As well as helping resolve some urgent disputes around algorithmic fairness, this theory points the way to a novel dimension of epistemic ethics, related to the recently discussed category of doxastic wrong.