Deborah Mayo

After some general remarks about the interrelation between philosophical and statistical thinking, the discussion centres largely on significance tests. These are defined as the calculation of p-values rather than as formal procedures for ‘acceptance‘ and ‘rejection‘. A number of types of null hypothesis are described and a principle for evidential interpretation set out governing the implications of p- values in the specific circumstances of each application, as contrasted with a long-run interpretation. A number of more complicated situ- ations are discussed in which modification of the simple p-value may be essential.

While this chapter and Achinstein agree that an account of evidence should be objective, not subjective, and empirical, not a priori, Achinstein has argued that we may reach conflicting assessments of evidence. There are cases where little has been done to rule out threats of error to H—as severity requires—that Achinstein construes as good evidence for H. Conversely, data x may fail to count as evidence for H, according to Achinstein's epistemic probabilist, even where H has passed a severe test by dint of x. We may call this the “highly probed vs. highly probable” conflict. This chapter argues, based on Achinstein's most recent installment to this debate, that the severity account is more in sync with the Achinstein's goals and the special features of his brand of Bayesianism. This chapter also considers how Achinstein's defense of Mill's account of induction gives further grounds for viewing his objective epistemologist as a severe tester.

The crisis of replication has led many to blame statistical significance tests for making it too easy to find impressive looking effects that do not replicate. However, the very fact it becomes difficult to replicate effects when features of the tests are tied down actually serves to vindicate statistical significance tests. While statistical significance tests, used correctly, serve to bound the probabilities of erroneous interpretations of data, this error control is nullified by data-dredging, multiple testing, and other biasing selection effects. Arguments claiming to vitiate statistical significance tests attack straw person variants of tests that commit well-known fallacies and misinterpretations. There is a tension between popular calls for preregistration – arguably, one of the most promising ways to boost replication – and accounts that downplay error probabilities: Bayes Factors, Bayesian posteriors, likelihood ratios. By underscoring the importance of error control for well testedness, the replication crisis points to reformulating tests so as to avoid fallacies and report the extent of discrepancies that are and are not indicated with severity.

Discussions of science and values in risk management have largely focused on how values enter into arguments about risks, that is, issues of acceptable risk. Instead this volume concentrates on how values enter into collecting, interpreting, communicating, and evaluating the evidence of risks, that is, issues of the acceptability of evidence of risk. By focusing on acceptable evidence, this volume avoids two barriers to progress. One barrier assumes that evidence of risk is largely a matter of objective scientific data and therefore uncontroversial. The other assumes that evidence of risk, being "just" a matter of values, is not amenable to reasoned critique. Denying both extremes, this volume argues for a more constructive conclusion: understanding the interrelations of scientific and value issues enables a critical scrutiny of risk assessments and better public deliberation about social choices. The contributors, distinguished philosophers, policy analysts, and natural and social scientists, analyze environmental and medical controversies, and assumptions underlying views about risk assessment and the scientific and statistical models used in risk management.

The field of quantified risk assessment is a new field, only about 20 years old, and already it is considered to be in a crisis. As Funtowicz and J.R. Ravetz (1985) put it:The concept of risk in terms of probability has proved to be so elusive, and statistical inference so problematic, that many experts in the field have recently either lost hope of finding a scientific solution or lost faith in Risk Analysis as a tool for decisionmaking. (p.219)Thus the ‘art’ of the assessment of risks… is at an impasse. The early hopes that it could be reduced to a science are frustrated. …[O]thers are tending to introduce the ‘human’ and ‘cultural’ factors. The question now becomes, to what extent should these predominate? Would it be to the reduction or exclusion of the ‘scientific’ aspects? For, …if the perceived phenomena of ‘risks’ are interpreted as lacking all objective content or being merely a small part of some total cultural configuration, then there is no basis for dialogue between opposed positions on such problems, (pp.220-221)

In How the Laws of Physics Lie (1983)2 Cartwright argues for being a realist about theoretical entities but non-realist about theoretical laws. Her reason for this distinction is that only the former involves causal explanation, and accepting causal explanations commits us to the existence of the causal entity invoked. “What is special about explanation by theoretical entity is that it is causal explanation, and existence is an internal characteristic of causal claims. There is nothing similar for theoretical laws.” (p. 93). For, according to Cartwright, the acceptability of a theoretical explanation is a matter of its ability to satisfy such criteria as prganizing and simplifying, and in her view, “success at organizing, predicting, and classifying is never an argument for truth.” (p. 91). In contrast, Cartwright claims, “When I infer from an effect to a cause, I am asking what made the effect occur, what brought it about.

While the common procedure of statistical significance testing and its accompanying concept of p-values have long been surrounded by controversy, renewed concern has been triggered by the replication crisis in science. Many blame statistical significance tests themselves, and some regard them as sufficiently damaging to scientific practice as to warrant being abandoned. We take a contrary position, arguing that the central criticisms arise from misunderstanding and misusing the statistical tools, and that in fact the purported remedies themselves risk damaging science. We argue that banning the use of p-value thresholds in interpreting data does not diminish but rather exacerbates data-dredging and biasing selection effects. If an account cannot specify outcomes that will not be allowed to count as evidence for a claim—if all thresholds are abandoned—then there is no test of that claim. The contributions of this paper are: To explain the rival statistical philosophies underlying the ongoing controversy; To elucidate and reinterpret statistical significance tests, and explain how this reinterpretation ameliorates common misuses and misinterpretations; To argue why recent recommendations to replace, abandon, or retire statistical significance undermine a central function of statistics in science: to test whether observed patterns in the data are genuine or due to background variability.

The growing availability of computer power and statistical software has greatly increased the ease with which practitioners apply statistical methods, but this has not been accompanied by attention to checking the assumptions on which these methods are based. At the same time, disagreements about inferences based on statistical research frequently revolve around whether the assumptions are actually met in the studies available, e.g., in psychology, ecology, biology, risk assessment. Philosophical scrutiny can help disentangle 'practical' problems of model validation, and conversely, a methodology of statistical model validation can shed light on a number of issues of interest to philosophers of science

Philosophy of statistics meets modal epistemology: Severity and sensitivity In this paper I take up a challenge raised by Gardiner and Zaharatos (2022) to link severity in philosophy of statistics and sensitivity in modal epistemology. Severity is based on using the error probabilities of statistical methods, not to ensure they rarely fail in a series of use, but to capture their capability to probe mistakes in the case at hand. A claim C is severely tested by passing a test that it (probably) would have failed if it were false. Analogously, a judgement that p is sensitive iff were p false, the agent would not have judged that p. Severe testing is a sensitivity account in the epistemologist’s sense, but with important qualifications. Notably, I will argue that severity avoids a difficulty long thought to bedevil sensitivity accounts: their inability to rule out skeptical challenges to claims regarded as warranted by evidence. This problem has led many to appeal instead to the notion of safety: a claim is safe if it could not easily be false. I compare the severity solution to the appeal to safety. I argue that severity reaches the safety theorist’s destination by a different route: a severely warranted claim could “not easily be false”, since overturning C would require discounting the network of tests that established its warrant. Rather than setting aside skeptical challenges as irrelevant, severity shows—through auditing assumptions and error-probing—that they collapse under severe scrutiny.

Although both philosophers and scientists are interested in how to obtain reliable knowledge in the face of error, there is a gap between their perspectives that has been an obstacle to progress. By means of a series of exchanges between the editors and leaders from the philosophy of science, statistics and economics, this volume offers a cumulative introduction connecting problems of traditional philosophy of science to problems of inference in statistical and empirical modelling practice. Philosophers of science and scientific practitioners are challenged to reevaluate the assumptions of their own theories - philosophical or methodological. Practitioners may better appreciate the foundational issues around which their questions revolve and thereby become better 'applied philosophers'. Conversely, new avenues emerge for finally solving recalcitrant philosophical problems of induction, explanation and theory testing.

Philosophers of science have long been concerned with the question of what a given scientific theory tells us about the contents of the world, but relatively little attention has been paid to how we set out to build theories and to the relevance of pre-theoretical methodology on a theory’s interpretation. In the traditional view, the form and content of a mature theory can be separated from any tentative ontological assumptions that went into its development. For this reason, the target of interpretation is taken to be the mature theory and nothing more. On this view, positions on ontology emerge only once a theory is to hand, not as part of the process of theory building. Serious attention to theory creation suggests this is too simple. In particular, data collection and experimentation are influenced both by theory and by assumptions about the entities thought to be the target of study. Initial reasoning about possible ontologies has an influence on the choice of theoretical variables as well as on the judgments of the methodology appropriate to investigate them.

In empirical modeling, an important desiderata for deeming theoretical entities and processes as real is that they can be reproducible in a statistical sense. Current day crises regarding replicability in science intertwines with the question of how statistical methods link data to statistical and substantive theories and models. Different answers to this question have important methodological consequences for inference, which are intertwined with a contrast between the ontological commitments of the two types of models. The key to untangling them is the realization that behind every substantive model there is a statistical model that pertains exclusively to the probabilistic assumptions imposed on the data. It is not that the methodology determines whether to be a realist about entities and processes in a substantive field. It is rather that the substantive and statistical models refer to different entities and processes, and therefore call for different criteria of adequacy.