Alejandro Fernández-Roldán

Fact-checking agencies assess and score the truthfulness of politicians’ claims to foster their electoral accountability. Fact-checking is sometimes presented as a quasi-scientific activity, based on reproducible verification protocols that would guarantee an unbiased assessment. We will study these verification protocols and discuss under which conditions fact-checking could achieve effective reproducibility. Through an analysis of the methodological norms in verification protocols, we will argue that achieving reproducible fact-checking may not help much in rendering politicians accountable. Political fact-checkers do not deliver either reproducibility or accountability today, and there are reasons to think that traditional quality journalism may serve liberal democracies better.

Political fact-checkers evaluate the truthfulness of politicians’ claims. This paper contributes to an emerging scholarly debate on whether fact-checkers treat political parties differently in a systematic manner depending on their ideology (bias). We first examine the available approaches to analyze bias and then present a new approach in two steps. First, we propose a logistic regression model to analyze the outcomes of fact-checks and calculate how likely each political party will obtain a truth score. We test our model with a sample of fact-checks from Newtral, a major Spanish fact-checker. Our model would signal bias under two assumptions: a) all political parties are on average equally accurate in their statements; b) the verification method gives precise instructions and is implemented systematically. We investigate this second assumption with a series of interviews with Newtral fact-checkers. We show that standard verification protocols are so loosely implemented that fact-checks reflect a set of journalistic decisions, rather than a bias in the statistical sense. We call for a more rigorous definition of verification methods as a pre-requisite for an unbiased assessment of politician’s claims.