•  18
    Peer review is often taken to be the main form of quality control on academic writings. Usually this is carried out by journals. Parts of math and physics appear to have now set up a parallel, crowd-sourced model of peer review, where papers are posted on the arXiv to be publicly discussed. In this paper we argue that crowd-sourced peer review is likely to do better than journal-solicited peer review at sorting papers by quality. Our argument rests on two key claims. First, crowd-sourced peer re…Read more
  •  6
    Peer reviewers at many funding agencies and scientific journals are asked to score submissions both on individual criteria and overall. The overall scores should be some kind of aggregate of the criteria scores. Carole Lee identifies this as a potential locus for bias to enter the peer review process, which she calls commensuration bias. Here I view the aggregation of scores through the lens of social choice theory. I argue that in many situations, especially when reviewing grant proposals, it i…Read more
  •  9
    Review of the volume "Scientific Collaboration and Collective Knowledge: New Essays", edited by Thomas Boyer-Kassem, Conor Mayo-Wilson, and Michael Weisberg.
  •  664
    Why the Reward Structure of Science Makes Reproducibility Problems Inevitable
    Journal of Philosophy 115 (12): 661-674. 2018.
    Recent philosophical work has praised the reward structure of science, while recent empirical work has shown that many scientific results may not be reproducible. I argue that the reward structure of science incentivizes scientists to focus on speed and impact at the expense of the reproducibility of their work, thus contributing to the so-called reproducibility crisis. I use a rational choice model to identify a set of sufficient conditions for this problem to arise, and I argue that these cond…Read more
  •  11
    Recent empirical work has shown that many scientific results may not be reproducible. By itself, this does not entail that there is a problem. However, I argue that there is a problem: the reward structure of science incentivizes scientists to focus on speed and impact at the expense of the reproducibility of their work. I illustrate this using a well-known failure of reproducibility: Fleischmann and Pons' work on cold fusion. I then use a rational choice model to identify a set of sufficient co…Read more
  •  23
    The Credit Incentive to Be a Maverick
    Studies in History and Philosophy of Science Part A. forthcoming.
    There is a commonly made distinction between two types of scientists: risk-taking, trailblazing mavericks and detail-oriented followers. A number of recent papers have discussed the question what a desirable mixture of mavericks and followers looks like. Answering this question is most useful if a scientific community can be steered toward such a desirable mixture. One attractive route is through credit incentives: manipulating rewards so that reward-seeking scientists are likely to form the des…Read more
  •  138
    Is Peer Review a Good Idea?
    British Journal for the Philosophy of Science. forthcoming.
    Prepublication peer review should be abolished. We consider the effects that such a change will have on the social structure of science, paying particular attention to the changed incentive structure and the likely effects on the behaviour of individual scientists. We evaluate these changes from the perspective of epistemic consequentialism. We find that where the effects of abolishing prepublication peer review can be evaluated with a reasonable level of confidence based on presently available …Read more
  •  11
    I discuss a game-theoretic model in which scientists compete to finish the intermediate stages of some research project. Banerjee et al. have previously shown that if the credit awarded for intermediate results is proportional to their difficulty, then the strategy profile in which scientists share each intermediate stage as soon as they complete it is a Nash equilibrium. I show that the equilibrium is both unique and strict. Thus rational credit-maximizing scientists have an incentive to share …Read more
  •  42
    This paper raises the problem of judgment aggregation in science. The problem has two sides. First, how do scientists decide which propositions to assert in a collaborative document? And second, how should they make such decisions? The literature on judgment aggregation is relevant to the second question. Although little evidence is available regarding the first question, it suggests that current scientific practice is not in line with the most plausible recommendations from the judgment aggrega…Read more
  •  96
    A Role for Judgment Aggregation in Coauthoring Scientific Papers
    with Liam Kofi Bright and Haixin Dang
    Erkenntnis 83 (2): 231-252. 2018.
    This paper addresses the problem of judgment aggregation in science. How should scientists decide which propositions to assert in a collaborative document? We distinguish the question of what to write in a collaborative document from the question of collective belief. We argue that recent objections to the application of the formal literature on judgment aggregation to the problem of judgment aggregation in science apply to the latter, not the former question. The formal literature has introduce…Read more
  •  87
    Vindicating Methodological Triangulation
    with Liam Kofi Bright and Andrew Zucker
    Synthese 1-15. forthcoming.
    Social scientists use many different methods, and there are often substantial disagreements about which method is appropriate for a given research question. A proponent of methodological triangulation believes that if multiple methods yield the same answer that answer is confirmed more strongly than it could have been by any single method. Methodological purists, on the other hand, believe that one should choose a single appropriate method and stick with it. Using formal tools from voting theory…Read more
  •  50
    The Bounded Strength of Weak Expectations
    with J. Sprenger and R. Heesen
    Mind 120 (479): 819-832. 2011.
    The rational price of the Pasadena and Altadena games, introduced by Nover and Hájek (2004 ), has been the subject of considerable discussion. Easwaran (2008 ) has suggested that weak expectations — the value to which the average payoffs converge in probability — can give the rational price of such games. We argue against the normative force of weak expectations in the standard framework. Furthermore, we propose to replace this framework by a bounded utility perspective: this shift renders the p…Read more
  •  130
    A Game-Theoretic Approach to Peer Disagreement
    Erkenntnis 81 (6): 1345-1368. 2016.
    In this paper we propose and analyze a game-theoretic model of the epistemology of peer disagreement. In this model, the peers' rationality is evaluated in terms of their probability of ending the disagreement with a true belief. We find that different strategies---in particular, one based on the Steadfast View and one based on the Conciliatory View---are rational depending on the truth-sensitivity of the individuals involved in the disagreement. Interestingly, the Steadfast and the Conciliatory…Read more
  •  12
    It is well-known that some scientists are more prominent than others. But what makes one scientist more prominent than another? I propose a possible mechanism that produces differences in prominence: scientists' desire for information. In a model of a scientific community exchanging information, I show that this mechanism indeed produces the kind of patterns of prominence that are actually observed. I discuss the implications of this result for three possible explanations of an individual scient…Read more
  •  97
    Academic superstars: competent or lucky?
    Synthese 194 (11): 4499-4518. 2017.
    I show that the social stratification of academic science can arise as a result of academics’ preference for reading work of high epistemic value. This is consistent with a view on which academic superstars are highly competent academics, but also with a view on which superstars arise primarily due to luck. I argue that stratification is beneficial if most superstars are competent, but not if most superstars are lucky. I also argue that it is impossible to tell whether most superstars are in fac…Read more
  •  16
    In the second volume of the "Handbuch der physiologischen Optik", published in 1860, Helmholtz sets out a three-receptor theory of color vision using coterminal response curves, and shows that this theory can unify most phenomena of color mixing known at the time. Maxwell had publicized the same theory five years earlier, but Helmholtz barely acknowledges this fact in the "Handbuch". Some historians have argued that this is because Helmholtz independently discovered the theory around the same ti…Read more
  •  114
    Communism and the Incentive to Share in Science
    Philosophy of Science 84 (4): 698-716. 2017.
    The communist norm requires that scientists widely share the results of their work. Where did this norm come from, and how does it persist? Michael Strevens provides a partial answer to these questions by showing that scientists should be willing to sign a social contract that mandates sharing. However, he also argues that it is not in an individual credit-maximizing scientist's interest to follow this norm. I argue against Strevens that individual scientists can rationally conform to the commun…Read more
  •  90
    When journal editors play favorites
    Philosophical Studies 175 (4): 831-858. 2018.
    Should editors of scientific journals practice triple-anonymous reviewing? I consider two arguments in favor. The first says that insofar as editors’ decisions are affected by information they would not have had under triple-anonymous review, an injustice is committed against certain authors. I show that even well-meaning editors would commit this wrong and I endorse this argument. The second argument says that insofar as editors’ decisions are affected by information they would not have had und…Read more
  •  193
    How much evidence should one collect?
    Philosophical Studies 172 (9): 2299-2313. 2015.
    A number of philosophers of science and statisticians have attempted to justify conclusions drawn from a finite sequence of evidence by appealing to results about what happens if the length of that sequence tends to infinity. If their justifications are to be successful, they need to rely on the finite sequence being either indefinitely increasing or of a large size. These assumptions are often not met in practice. This paper analyzes a simple model of collecting evidence and finds that the prac…Read more