•  6
    This dissertation is a study of how methodological issues in psychology can have significant implications for philosophical accounts of interpretation, justification, and psychological explanation. In the first chapter, I analyze traditional philosophical accounts of interpretation with an eye to identifying the ways in which philosophers have used rationality as a methodological tool. I argue that these forms of methodological rationalism do not successfully cope with the challenge from the heu…Read more
  •  4
    NIH Peer Review: Criterion Scores Completely Account for Racial Disparities in Overall Impact Scores
    with Elena A. Erosheva, Sheridan Grant, Mei-Ching Chen, Mark D. Lindner, and Richard K. Nakamura
    Science Advances 6 (23). 2020.
    Previous research has found that funding disparities are driven by applications’ final impact scores and that only a portion of the black/white funding gap can be explained by bibliometrics and topic choice. Using National Institutes of Health R01 applications for council years 2014–2016, we examine assigned reviewers’ preliminary overall impact and criterion scores to evaluate whether racial disparities in impact scores can be explained by application and applicant characteristics. We hypothesi…Read more
  •  10
    When Zero May Not be Zero: A Cautionary Note on the Use of Inter-Rater Reliability in Evaluating Grant Peer Review
    with Elena A. Erosheva and Patrícia Martinková
    Journal of the Royal Statistical Society: Series A 184 904-19. 2021.
    Considerable attention has focused on studying reviewer agreement via inter-rater reliability (IRR) as a way to assess the quality of the peer review process. Inspired by a recent study that reported an IRR of zero in the mock peer review of top-quality grant proposals, we use real data from a complete range of submissions to the National Institutes of Health and to the American Institute of Biological Sciences to bring awareness to two important issues with using IRR for assessing peer review q…Read more
  • Refinement: Measuring informativeness of ratings in the absence of a gold standard
    with Sheridan Grant, Marina Meilă, and Elena Erosheva
    British Journal of Mathematical and Statistical Psychology 75 (3): 593-615. 2022.
    We propose a new metric for evaluating the informativeness of a set of ratings from a single rater on a given scale. Such evaluations are of interest when raters rate numerous comparable items on the same scale, as occurs in hiring, college admissions, and peer review. Our exposition takes the context of peer review, which involves univariate and multivariate cardinal ratings. We draw on this context to motivate an information-theoretic measure of the refinement of a set of ratings – entropic re…Read more
  •  25
    Certified Amplification: An Emerging Scientific Norm and Ethos
    Philosophy of Science 89 (5): 1002-1012. 2022.
    Merton envisioned his norms of science at a time when peer-reviewed journals controlled scientific communication. Technologies for sharing and finding content have since divorced the certification and amplification of science, generating systemic vulnerabilities. Certified amplification—a new Mertonian-styled norm—enjoins their recoupling and introduces a taxonomy of strategies adopted by institutions to close the certification-amplification gap, including the proportioning of the one to the oth…Read more
  •  1
    Alternative Funding Models Might Perpetuate Black-White Funding Gaps
    with Sheridan Grant and Elena A. Erosheva
    The Lancet 396 955-6. 2020.
    The White Coats for Black Lives and #ShutDownSTEM movements have galvanised biomedical practitioners and researchers to eliminate institutional and systematic racism, including barriers faced by Black researchers in biomedicine and science, technology, engineering, and mathematics. In our study on Black–White funding gaps for National Institutes of Health Research Project grants, we found that the overall award rate for Black applicants is 55% of that for white applicants. How can systems for al…Read more
  •  73
    The Reference Class Problem for Credit Valuation in Science
    Philosophy of Science 87 (5): 1026-1036. 2020.
    Scholars belong to multiple communities of credit simultaneously. When these communities disagree about a scholarly achievement’s credit assignment, this raises a puzzle for decision and game theor...
  •  38
    Gender-based homophily in collaborations across a heterogeneous scholarly landscape
    with Y. Samuel Wang, Jevin D. West, Carl T. Bergstrom, and Elena A. Erosheva
    PLoS ONE 18 (4). 2023.
    Using the corpus of JSTOR articles, we investigate the role of gender in collaboration patterns across the scholarly landscape by analyzing gender-based homophily--the tendency for researchers to co-author with individuals of the same gender. For a nuanced analysis of gender homophily, we develop methodology necessitated by the fact that the data comprises heterogeneous sub-disciplines and that not all authorships are exchangeable. In particular, we distinguish three components of gender homophi…Read more
  •  16
    Promoting Gender Equity in Grant Making: What Can a Funder Do?
    with Sindy N. Escobar Alvarez, Reshma Jagsi, Stephanie B. Abbuhl, and Elizabeth R. Myers
    The Lancet 393 (10171). 2019.
    The Doris Duke Charitable Foundation's competitive career development award selects awardees annually. This paper describes changes DDCF made to its grants making process to improve gender representation in its applicant and awardee pools.
  •  24
    Promote Scientific Integrity via Journal Peer Review Data
    Science 357 (6348): 256-257. 2017.
    There is an increasing push by journals to ensure that data and products related to published papers are shared as part of a cultural move to promote transparency, reproducibility, and trust in the scientific literature. Yet few journals commit to evaluating their effectiveness in implementing reporting standards aimed at meeting those goals (1, 2). Similarly, though the vast majority of journals endorse peer review as an approach to ensure trust in the literature, few make their peer review dat…Read more
  •  308
    A Dispositional Account of Aversive Racism
    Proceedings of the 40th Annual Cognitive Science Society. 2018.
    I motivate and articulate a dispositional account of aversive racism. By conceptualizing and measuring attitudes in terms of their full distribution, rather than in terms of their mode or mean preference, my account of dispositional attitudes gives ambivalent attitudes (qua attitude) the ability to predict aggregate behavior. This account can be distinguished from other dispositional accounts of attitude by its ability to characterize ambivalent attitudes such as aversive racism at the attitudin…Read more
  •  423
    Collective Implicit Attitudes: A Stakeholder Conception of Implicit Bias
    Proceedings of the 40th Annual Cognitive Science Society. 2018.
    Psychologists and philosophers have not yet resolved what they take implicit attitudes to be; and, some, concerned about limitations in the psychometric evidence, have even challenged the predictive and theoretical value of positing implicit attitudes in explanations for social behavior. In the midst of this debate, prominent stakeholders in science have called for scientific communities to recognize and countenance implicit bias in STEM fields. In this paper, I stake out a stakeholder conceptio…Read more
  •  84
    is normative in the sense that it aims to make recommendations for improving human judgment; it aims to have a practical impact on morally and politically significant human decisions and actions; and it studies normative, rational judgment qua rational judgment. These nonstandard ways of understanding ACP as normative collectively suggest a new interpretation of the strong replacement thesis that does not call for replacing normative epistemic concepts, relations, and inquiries with descriptive,…Read more
  •  83
    A Kuhnian Critique of Psychometric Research on Peer Review
    Philosophy of Science 79 (5): 859-870. 2012.
    Psychometrically oriented researchers construe low inter-rater reliability measures for expert peer reviewers as damning for the practice of peer review. I argue that this perspective overlooks different forms of normatively appropriate disagreement among reviewers. Of special interest are Kuhnian questions about the extent to which variance in reviewer ratings can be accounted for by normatively appropriate disagreements about how to interpret and apply evaluative criteria within disciplines …Read more
  •  93
    On the surface, developing a social psychology of science seems compelling as a way to understand how individual social cognition – in aggregate – contributes towards individual and group behavior within scientific communities (Kitcher, 2002). However, in cases where the functional input-output profile of psychological processes cannot be mapped directly onto the observed behavior of working scientists, it becomes clear that the relationship between psychological claims and normative philosophy…Read more
  •  64
    Gricean charity: The Gricean turn in psychology
    Philosophy of the Social Sciences 36 (2): 193-218. 2006.
    Psychologists' work on conversational pragmatics and judgment suggests a refreshing approach to charitable interpretation and theorizing. This charitable approach—what I call Gricean charity —recognizes the role of conversational assumptions and norms in subject-experimenter communication. In this paper, I outline the methodological lessons Gricean charity gleans from psychologists' work in conversational pragmatics. In particular, Gricean charity imposes specific evidential standards requiring …Read more
  •  138
    Bias in Peer Review
    with Cassidy R. Sugimoto, Guo Zhang, and Blaise Cronin
    Journal of the American Society for Information Science and Technology 64 (1): 2-17. 2013.
    Research on bias in peer review examines scholarly communication and funding processes to assess the epistemic and social legitimacy of the mechanisms by which knowledge communities vet and self-regulate their work. Despite vocal concerns, a closer look at the empirical and methodological limitations of research on bias raises questions about the existence and extent of many hypothesized forms of bias. In addition, the notion of bias is predicated on an implicit ideal that, once articulated, rai…Read more
  •  139
    Social Biases and Solution for Procedural Objectivity
    with Christian D. Schunn
    Hypatia 26 (2): 352-73. 2011.
    An empirically sensitive formulation of the norms of transformative criticism must recognize that even public and shared standards of evaluation can be implemented in ways that unintentionally perpetuate and reproduce forms of social bias that are epistemically detrimental. Helen Longino’s theory can explain and redress such social bias by treating peer evaluations as hypotheses based on data and by requiring a kind of perspectival diversity that bears, not on the content of the community’s know…Read more
  •  568
    Reclaiming Davidson’s Methodological Rationalism as Galilean Idealization in Psychology
    Philosophy of the Social Sciences 40 (1): 84-106. 2010.
    In his early experimental work with Suppes, Davidson adopted rationality assumptions, not as necessary constraints on interpretation, but as practical conceits in addressing methodological problems faced by experimenters studying decision making under uncertainty. Although the content of their theory has since been undermined, their methodological approach—a Galilean form of methodological rationalism—lives on in contemporary psychological research. This article draws on Max Weber’s verstehen to…Read more
  •  93
    Commensuration Bias in Peer Review
    Philosophy of Science 82 (5). 2015.
    To arrive at their final evaluation of a manuscript or grant proposal, reviewers must convert a submission’s strengths and weaknesses for heterogeneous peer review criteria into a single metric of quality or merit. I identify this process of commensuration as the locus for a new kind of peer review bias. Commensuration bias illuminates how the systematic prioritization of some peer review criteria over others permits and facilitates problematic patterns of publication and funding in science. Com…Read more
  •  597
    Under the traditional system of peer-reviewed publication, the degree of prestige conferred to authors by successful publication is tied to the degree of the intellectual rigor of its peer review process: ambitious scientists do well professionally by doing well epistemically. As a result, we should expect journal editors, in their dual role as epistemic evaluators and prestige-allocators, to have the power to motivate improved author behavior through the tightening of publication requirements. …Read more
  •  31
    Asian Americans, positive stereotyping, and philosophy
    American Philosophical Association Newsletter on Asian and Asian-American Philosophers and Philosophies 14 (2-7). 2014.
    What is the current status of Asian Americans in philosophy? How do Asian Americans fare in comparison to other minority groups? And, what professional strategies might they use (more or less successfully) in response to their counterstereotypical status in philosophy? In this piece, I will address these questions empirically by extrapolating from available demographic, survey, and experimental studies. This analysis will be too fast and loose, but I offer it in the spirit of constructing a broa…Read more
  •  72
    The representation of judgment heuristics and the generality problem
    Proceedings of the 29th Annual Cognitive Science Society 1211-6. 2007.
    In his debates with Daniel Kahneman and Amos Tversky, Gerd Gigerenzer puts forward a stricter standard for the proper representation of judgment heuristics. I argue that Gigerenzer’s stricter standard contributes to naturalized epistemology in two ways. First, Gigerenzer’s standard can be used to winnow away cognitive processes that are inappropriately characterized and should not be used in the epistemic evaluation of belief. Second, Gigerenzer’s critique helps to recast the generality problem …Read more
  •  48
    Philosophy journal practices and opportunities for bias
    with Christian D. Schunn
    American Philosophical Association Newsletter on Feminism and Philosophy. 2010.