Remco Heesen

Knowledge brokers, usually conceptualized as passive intermediaries between scientists and policymakers in evidence-based policymaking, are understudied in philosophy of science. Here, we challenge that usual conceptualization. As agents in their own right, knowledge brokers have their own goals and incentives, which complicate the effects of their presence at the science-policy interface. We illustrate this in an agent-based model and suggest several avenues for further exploration of the role of knowledge brokers in evidence-based policy.

Are scientists incentivized to share intermediate results? Some recent work has argued that credit-maximizing scientists would share intermediate results, whereas other work has argued that sharing too much too quickly can hinder a community’s achievement of epistemic goals. This suggests a tradeoff between scientists’ credit-based self-interest and what is epistemically best. We make this tradeoff explicit in an agent-based NK landscape model, where credit-maximizing scientists face a prisoner’s dilemma due to collective epistemic and credit benefits from withholding. Multiple coders independently implemented the model, leading us to identify several coding errors. We reflect on this code comparison experience.

Are scientists incentivized to share intermediate results? Some recent work has argued that credit-maximizing scientists would share intermediate results, whereas other work has argued that sharing too much too quickly can hinder a community’s achievement of epistemic goals. This suggests a tradeoff between scientists’ credit-based self-interest and what is epistemically best. We make this tradeoff explicit in an agent-based NK landscape model, where credit-maximizing scientists face a prisoner’s dilemma due to collective epistemic and credit benefits from withholding. Multiple coders independently implemented the model, leading us to identify several coding errors. We reflect on this code comparison experience.

Social scientists use many different methods, and there are often substantial disagreements about which method is appropriate for a given research question. In response to this uncertainty about the relative merits of different methods, W. E. B. Du Bois advocated for and applied “methodological triangulation”. This is to use multiple methods simultaneously in the belief that, where one is uncertain about the reliability of any given method, if multiple methods yield the same answer that answer is confirmed more strongly than it could have been by any single method. Against this, methodological purists believe that one should choose a single appropriate method and stick with it. Using tools from voting theory, we show Du Boisian methodological triangulation to be more likely to yield the correct answer than purism, assuming the scientist is subject to some degree of diffidence about the relative merits of the various methods. This holds even when in fact only one of the methods is appropriate for the given research question.

This article investigates how the credit incentive to engage in questionable research practices (up to and including fraud) interacts with cumulative advantage, the process whereby high-status academics more easily increase their status than low-status academics. I use a mathematical model to highlight two dynamics that have not yet received much attention. First, because of cumulative advantage, questionable research practices may pay off over the course of an academic career even if they are not attractive at the level of individual publications. Second, because of the role of bottleneck moments in academic careers, questionable research practices may be selected for even if they do not provide a benefit in expectation. I also observe that within the model, the most successful academics are the most likely to have benefited from fraud.

What differentiates scientific research from non-scientific inquiry? Philosophers addressing this question have typically been inspired by the exalted social place and intellectual achievements of science. They have hence tended to point to some epistemic virtue or methodological feature of science that sets it apart. Our discussion on the other hand is motivated by the case of commercial research, which we argue is distinct from (and often epistemically inferior to) academic research. We consider a deflationary view in which science refers to whatever is regarded as epistemically successful, but find that this does not leave room for the important notion of scientific error and fails to capture distinctive social elements of science. This leads us to the view that a demarcation criterion should be a widely upheld social norm without immediate epistemic connotations. Our tentative answer is the communist norm, which calls on scientists to share their work widely for public scrutiny and evaluation.

Prepublication peer review should be abolished. We consider the effects that such a change will have on the social structure of science, paying particular attention to the changed incentive structure and the likely effects on the behaviour of individual scientists. We evaluate these changes from the perspective of epistemic consequentialism. We find that where the effects of abolishing prepublication peer review can be evaluated with a reasonable level of confidence based on presently available evidence, they are either positive or neutral. We conclude that on present evidence abolishing peer review weakly dominates the status quo.

Peer review is often taken to be the main form of quality control on academic research. Usually journals carry this out. However, parts of maths and physics appear to have a parallel, crowd-sourced model of peer review, where articles are posted on the arXiv to be publicly discussed. In this article we argue that crowd-sourced peer review is likely to do better than journal-solicited peer review at sorting articles by quality. Our argument rests on two key claims. First, crowd-sourced peer review will lead on average to more reviewers per article than journal-solicited peer review. Second, due to the wisdom of the crowds, more reviewers will tend to make better judgements than fewer reviewers will. We make the second claim precise by looking at the Condorcet jury theorem as well as two related jury theorems developed specifically to apply to peer review.

Evidence-based policy (EBP) has become widely embraced for its commitment to greater uptake of scientific knowledge in policymaking. But what legitimizes EBP and in what respect are evidence-based policymaking practices better than other policymaking practices? In this article, we distinguish and refine three potential legitimizers of EBP. We suggest that evidence-based policymaking practices are better because they “follow the science,” because they focus on “what works,” or because they “follow the rules.” We discuss some consequences, for advocates of EBP, of consciously adopting one or other of these legitimizers. Finally, we examine whether it is appropriate to switch from advocating for EBP to advocating for evidence-informed policy.

When evidence‑based policymaking is so often mired in disagreement and controversy, how can we know if the process is meeting its stated goals? We develop a novel mathematical model to study disagreements about adequate knowledge utilization, like those regarding wild horse culling, shark drumlines and facemask policies during pandemics. We find that, when stakeholders disagree, it is frequently impossible to tell whether any party is at fault. We demonstrate the need for a distinctive kind of transparency in evidence‑based policymaking, which we call transparency of reasoning. Such transparency is critical to the success of the evidence‑based policy movement, as without it, we will be unable to tell whether in any instance a policy was in fact based on evidence.

Peer review is often taken to be the main form of quality control on academic research. Usually journals carry this out. However, parts of maths and physics appear to have a parallel, crowd-sourced model of peer review, where articles are posted on the arXiv to be publicly discussed. In this article we argue that crowd-sourced peer review is likely to do better than journal-solicited peer review at sorting articles by quality. Our argument rests on two key claims. First, crowd-sourced peer review will lead on average to more reviewers per article than journal-solicited peer review. Second, due to the wisdom of the crowds, more reviewers will tend to make better judgements than fewer reviewers will. We make the second claim precise by looking at the Condorcet jury theorem as well as two related jury theorems developed specifically to apply to peer review.

Whether or not any particular policy is adequate by EBP's own standard—being evidence‐based—cannot be decided without appeal to value‐based considerations. We support this claim in two steps. First, we argue that which evidence gets used in policy‐making depends on our value commitments, which are rarely made explicit, let alone being the subject of critical and transparent reflection. In other words, value commitments are not just important at the point of spelling out specific policy details and choosing between policy options but they are absolutely essential right from the very beginning of the policy‐making process: all the way from deciding which problem should be addressed by policy or regulation to determining which evidence to use and where to look for it. Second, in order to determine when we have enough evidence, we need to take into account relevant value‐based considerations.

Whether or not any particular policy is adequate by EBP's own standard—being evidence-based—cannot be decided without appeal to value-based considerations. We support this claim in two steps. First, we argue that which evidence gets used in policy-making depends on our value commitments, which are rarely made explicit, let alone being the subject of critical and transparent reflection. In other words, value commitments are not just important at the point of spelling out specific policy details and choos¬ing between policy options but they are absolutely essential right from the very beginning of the policy-making process: all the way from deciding which problem should be addressed by policy or regulation to determining which evidence to use and where to look for it. Second, in order to determine when we have enough evidence, we need to take into account relevant value-based considerations.

Recent empirical work has shown that many scientific results may not be reproducible. By itself, this does not entail that there is a problem (or a "reproducibility crisis"). However, I argue that there is a problem: the reward structure of science incentivizes scientists to focus on speed and impact at the expense of the reproducibility of their work. I illustrate this using a well-known failure of reproducibility: Fleischmann and Pons' work on cold fusion. I then use a rational choice model to identify a set of sufficient conditions for this problem to arise, and I argue that these conditions plausibly apply to a wide range of research situations. In the conclusion I consider possible solutions and implications for how Fleischmann and Pons' work should be evaluated.

Determining appropriate mechanisms for transferring and translating research into policy has become a major concern for researchers (knowledge producers) and policymakers (knowledge users) worldwide. This has led to the emergence of a new function of brokering between researchers and policymakers, and a new type of agent called Knowledge Broker. Understanding these complex multi-agent interactions is critical for an efficient knowledge brokering practice during any given policymaking process. Here, we present (1) the current diversity of knowledge broker groups working in the field of biosecurity and environmental management; (2) the incentives linking the different agents involved in the process (knowledge producers, knowledge brokers and knowledge users); and (3) the gaps, needs and challenges to better understand this social ecosystem. We also propose alternatives aimed at improving transparency and efficiency, including future scenarios where the role of artificial intelligence (AI) technologies may become predominant in knowledge-brokering activities.

It might seem obvious that the scientific process should not be biased. We strive for reliable inference, and systematically skewing the results of inquiry apparently conflicts with this. Publication bias—which involves only publishing certain types of results—seems particularly troubling and has been blamed for the replication crisis. While we ultimately agree, there are considerable nuances to take into account. Using a Bayesian model of scientific reasoning we show that a scientist who is aware of publication bias can (theoretically) interpret the published literature so as to avoid acquiring biased beliefs. Moreover, in some highly specific circumstances she might prefer not to bother with policies designed to mitigate or reduce the presence of publication bias—it would impose a cost in time or effort that she would not see any benefit in paying. However, we also argue that science as a social endeavour is made worse off by publication bias. This is because the social benefits of science are largely secured via go-between agents, various non-experts who nonetheless need to make use of or convey the results of scientific inquiry if its fruits are to be enjoyed by society at large. These are unlikely to be well-informed enough to account for publication bias appropriately. As such, we conclude, the costs of having to implement policies like mandatory pre-registration are worth imposing on scientists, even if they would perhaps not view these costs as worth paying for their own sake. The benefits are reaped by the go-between agents, and we argue that their perspective is quite properly favoured when deciding how to govern scientific institutions.

Evidence-based policy (EBP) has become widely embraced for its commitment to greater uptake of scientific knowledge in policymaking. But what legitimizes EBP and in what respect are evidence-based policymaking practices better than other policymaking practices? In this article, we distinguish and refine three potential legitimizers of EBP. We suggest that evidence-based policymaking practices are better because they “follow the science,” because they focus on “what works,” or because they “follow the rules.” We discuss some consequences, for advocates of EBP, of consciously adopting one or other of these legitimizers. Finally, we examine whether it is appropriate to switch from advocating for EBP to advocating for evidence-informed policy.

When evidence-based policymaking is so often mired in disagreement and controversy, how can we know if the process is meeting its stated goals? We develop a novel mathematical model to study disagreements about adequate knowledge utilization, like those regarding wild horse culling, shark drumlines and facemask policies during pandemics. We find that, when stakeholders disagree, it is frequently impossible to tell whether any party is at fault. We demonstrate the need for a distinctive kind of transparency in evidence-based policymaking, which we call transparency of reasoning. Such transparency is critical to the success of the evidence-based policy movement, as without it, we will be unable to tell whether in any instance a policy was in fact based on evidence.

Marine ecosystems are under increasing pressure from human activity, yet successful management relies on knowledge. The evidence-based policy (EBP) approach has been promoted on the grounds that it provides greater transparency and consistency by relying on ‘high quality’ information. However, EBP also creates epistemic responsibilities. Decision-making where limited or no empirical evidence exists, such as is often the case in marine systems, creates epistemic obligations for new information acquisition. We argue that philosophical approaches can inform the science-policy interface. Using marine biosecurity examples, we specifically examine the epistemic challenges in the acquisition and acceptance of evidence to inform policy, discussing epistemic due care and biases in consideration of evidence.

Theoretical arguments and empirical investigations indicate that a high proportion of published findings do not replicate and are likely false. The current position paper provides a broad perspective on scientific error, which may lead to replication failures. This broad perspective focuses on reform history and on opportunities for future reform. We organize our perspective along four main themes: institutional reform, methodological reform, statistical reform and publishing reform. For each theme, we illustrate potential errors by narrating the story of a fictional researcher during the research cycle. We discuss future opportunities for reform. The resulting agenda provides a resource to usher in an era that is marked by a research culture that is less error-prone and a scientific publication landscape with fewer spurious findings.

What differentiates scientific research from non-scientific inquiry? Philosophers addressing this question have typically been inspired by the exalted social place and intellectual achievements of science. They have hence tended to point to some epistemic virtue or methodological feature of science that sets it apart. Our discussion on the other hand is motivated by the case of commercial research, which we argue is distinct from (and often epistemically inferior to) academic research. We consider a deflationary view in which science refers to whatever is regarded as epistemically successful, but find that this does not leave room for the important notion of scientific error and fails to capture distinctive social elements of science. This leads us to the view that a demarcation criterion should be a widely upheld social norm without immediate epistemic connotations. Our tentative answer is the communist norm, which calls on scientists to share their work widely for public scrutiny and evaluation.

This paper investigates how the credit incentive to engage in questionable research practices interacts with cumulative advantage, the process whereby high-status academics more easily increase their status than low-status academics. I use a mathematical model to highlight two dynamics that have not yet received much attention. First, due to cumulative advantage, questionable research practices may pay off over the course of an academic career even if they are not attractive at the level of individual publications. Second, because of the role of bottleneck moments in academic careers, questionable research practices may be selected for even if they do not provide a benefit in expectation. I also observe that, within the model, the most successful academics are the most likely to have benefited from fraud.

This paper investigates how the credit incentive to engage in questionable research practices interacts with cumulative advantage, the process whereby high-status academics more easily increase their status than low-status academics. I use a mathematical model to highlight two dynamics that have not yet received much attention. First, due to cumulative advantage, questionable research practices may pay off over the course of an academic career even if they do not appear attractive at the level of individual publications. Second, because of the role of bottleneck moments in academic careers, questionable research practices may be selected for even if they do not provide a benefit in expectation. I also observe that, within the model, the most successful academics are the most likely to have benefited from fraud. This paper has been published under a different title. See PhilPapers record HEECAA or DOI 10.1086/716235

This paper offers a new angle on the common idea that the process of science does not support epistemic diversity. Under minimal assumptions on the nature of journal editing, we prove that editorial procedures, even when impartial in themselves, disadvantage less prominent research programs. This purely statistical bias in article selection further skews existing differences in the success rate and hence attractiveness of research programs, and exacerbates the reputation difference between the programs. After a discussion of the modeling assumptions, the paper ends with a number of recommendations that may help promote scientific diversity through editorial decision making.

Peer review is often taken to be the main form of quality control on academic research. Usually journals carry this out. However, parts of maths and physics appear to have a parallel, crowd-sourced model of peer review, where papers are posted on the arXiv to be publicly discussed. In this paper we argue that crowd-sourced peer review is likely to do better than journal-solicited peer review at sorting papers by quality. Our argument rests on two key claims. First, crowd-sourced peer review will lead on average to more reviewers per paper than journal-solicited peer review. Second, due to the wisdom of the crowds, more reviewers will tend to make better judgments than fewer. We make the second claim precise by looking at the Condorcet Jury Theorem as well as two related jury theorems developed specifically to apply to peer review.

Peer reviewers at many funding agencies and scientific journals are asked to score submissions both on individual criteria and overall. The overall scores should be some kind of aggregate of the criteria scores. Carole Lee identifies this as a potential locus for bias to enter the peer review process, which she calls commensuration bias. Here I view the aggregation of scores through the lens of social choice theory. I argue that, when reviewing grant proposals, it is in many cases impossible to avoid commensuration bias.