Mark L. Taper

Most of the claims we make, nowhere more so than in the empirical sciences, outrun the information enlisted to support them. Such claims are never more than probable/likely. Intuitively, even obviously, some claims are more probable/likely than others. Everyone agrees that scientific claims in particular are probable/likely to the extent that they are confirmed by experimental evidence. But there is very little agreement about what “confirmation by empirical evidence” involves or how it is to be measured. A central thesis of this monograph is that a source of this disagreement is the near-universal tendency to conflate the two different concepts—“confirmation” and “evidence”—used to formulate the essence of the methodology. There is no doubt that the words signifying them are used interchangeably. But as we will go on to argue, failure to make the distinction leads to muddled thinking in philosophy, statistics, and the empirical sciences themselves. Two examples, one having to do with the testing of traditional psychotherapeutic hypotheses, the other with determining the empirical superiority of the wave or particle theories of light, make it clear how data can confirm a hypothesis or theory, yet fail, in context, to provide evidence for it.

Very possibly the most famously intractable epistemological conundrum in the history of modern western philosophy is Descartes’ argument from dreaming. It seems to support in an irrefutable way a radical scepticism about the existence of a physical world existing independent of our sense-experience. But this argument as well as those we discussed in the last chapter and many others of the same kind rest on a conflation of evidence and confirmation: since the paradoxical or sceptical hypothesis has as much “evidence” going for it as the conventional or commonly accepted hypothesis, it is equally well supported by the data and there is nothing to choose between them. By this time, however, we understand very well that data that fail to discriminate hypotheses do not constitute “evidence” for any of them, i.e., that “data” and “evidence” are not interchangeable notions, that it does not follow from the fact that there is strong evidence for a hypothesis against one or more of its competitors that it is therefore highly confirmed, and that it does not follow from the fact that a hypothesis is highly confirmed that there is strong evidence for it against its rivals.

Clark Glymour’s “bootstrap” account of confirmation rightly stresses the importance of selective confirmation of individual hypotheses, on the one hand, and the determination of theoretical constants, on the other. But in our view it is marred by a failure to deal with the problem of confounding, illustrated by the demonstration of a causal link between smoking and lung cancer, and by the apparent circularity of bootstrap testing (which is distinguished from statistical bootstrapping). Glymour’s proper insistence on a variety of evidence is built into our account of evidence and not added on as a way of handling the apparent circularity in his account. We discuss and dissolve his well-known charge against Bayesian theories of confirmation, that they lead to the paradox of “old evidence,” in Chap. 9.

Several non-Bayesian and non-Likelihood accounts of evidence have been worked out in interesting detail. One such account has been championed by the philosopher Deborah Mayo and the statistician Ari Spanos. Following Popper, it assumes from the outset that to test a hypothesis is to submit it to a severe test. Unlike Popper it relies on the notion of error frequencies central to Neyman-Pearson statistics. Unlike Popper as well, Mayo and Spanos think that global theories like Newtonian mechanics are tested in a piecemeal way, by submitting their component hypotheses to severe tests. We argue that the error-statistical notion of severity is not adequately “severe,” that the emphasis on piecemeal testing procedures is misplaced, and that the Mayo-Spanos account of evidence is mistakenly committed to a “true model” assumption. In a technical Appendix we deflect Mayo’s critique of the multiple-model character of our account of evidence.

In our view, data confirm a hypothesis just in case they increase its probability; they constitute evidence for one hypothesis vis-à-vis others just in case they are more probable on it than on its available rivals. In subsequent chapters, we go on to clarify and amplify the confirmation/evidence distinction. Before doing so, however, we need to consider various objections that might be made, not to the distinction itself but to the way in which we have formulated its principal elements. Four of these objections are standard in the literature. The first, third, and fourth raise questions concerning our analyses of both confirmation and evidence; the second has to do more narrowly with the application of Bayesian methods. Each suggests a different way in which our intentions in this monograph might be misunderstood.

Like the error-statisticians, Glymour, and us, Peter Achinstein rejects an account of evidence in traditional Bayesian terms. Like the error-statisticians and Glymour, but unlike us, his own account of evidence incorporates what we have called the “true-model” assumption, that there is a conceptual connection between the existence of evidence for a hypothesis and having a good reason to believe that the hypothesis is true. In this connection, and unlike any of the other views surveyed, Achinstein does not so much analyze the concept of evidence per se as provide a taxonomy of conceptions of evidence—subjective, objective, potential, “epistemic-situational,” and “veridical.” He then both argues that only “veridical evidence” captures exemplary scientific practice and explains why this should be the case. In the first half of this chapter, we set out Achinstein’s criticisms of the Bayesian account, then develop and in turn criticize his own account of “veridical evidence.” In the second half of the chapter, we contrast “veridical” with “misleading” evidence and show how two key theorems concerning the probability of misleading evidence allow us to measure it.

Our object in this monograph has been to offer analyses of confirmation and evidence that will set the bar for what is to count as each and at the same time provide guidance for working scientists and statisticians. Philosophy does not sit in judgment on other disciplines nor can it dictate methodology. Instead, it forces reflection on the aims and methods of these disciplines in the hope that such reflection will lead to a critical testing of these aims and methods, in the same way that the methods themselves are used to test empirical hypotheses with certain aims in view. In the Appendix we discuss an application of the confirmation/evidence distinction to an important problem in current ecological research and in the process suggest ways of settling some outstanding problems at the intersection of statistics and the philosophy of science.

It can be demonstrated in a very straightforward way that confirmation and evidence as spelled out by us can vary from one case to the next, that is, a hypothesis may be weakly confirmed and yet the evidence for it can be strong, and conversely, the evidence may be weak and the confirmation strong. At first glance, this seems puzzling; the puzzlement disappears once it is understood that confirmation is of single hypotheses, in which there is an initial degree of belief which is adjusted up or down as data accumulate, whereas evidence always has to do with a comparison of one hypothesis against another with respect to the data and is belief-independent. Confusing them is, we suggest, a plausible source of the so-called “base-rate fallacy” identified by Kahneman and Tversky which leads most of us to make mistaken statistical inferences. It is also in the background, or so we argue in some detail, of the important policy controversies concerning human-induced global warming.

We contend that Bayesian accounts of evidence are inadequate, and that in this sense a complete theory of hypothesis testing must go beyond belief adjustment. Some prominent Bayesians disagree. To make our case, we will discuss and then provide reasons for rejecting the accounts of David Christensen, James Joyce, and Alan Hàjek. The main theme and final conclusions are straightforward: first, that no purely subjective account of evidence, in terms of belief alone, is adequate and second, that evidence is a comparative notion, applicable only when two hypotheses are confronted with the same data, as has been suggested in the literature on “crucial experiments” from Francis Bacon on.

The first step is to distinguish two questions: 1. Given the data, what should we believe, and to what degree? 2. What kind of evidence do the data provide for a hypothesis H 1 as against an alternative hypothesis H 2, and how much? We call the first the “confirmation”, the second the “evidence” question. Many different answers to each have been given. In order to make the distinction between them as intuitive and precise as possible, we answer the first in a Bayesian way: a hypothesis is confirmed to the extent that the data raise the probability that it is true. We answer the second question in a Likelihoodist way, that is, data constitute evidence for a hypothesis as against any of its rivals to the extent that they are more likely on it than on them. These two simple ideas are very different, but both can be made precise, and each has a great deal of explanatory power. At the same time, they enforce corollary distinctions between “data” and “evidence”, and between different ways in which the concept of “probability” is to be interpreted. An Appendix explains how our likelihoodist account of evidence deals with composite hypotheses.

There is a debate in Bayesian confirmation theory between subjective and non-subjective accounts of evidence. Colin Howson has provided a counterexample to our non-subjective account of evidence: the counterexample refers to a case in which there is strong evidence for a hypothesis, but the hypothesis is highly implausible. In this article, we contend that, by supposing that strong evidence for a hypothesis makes the hypothesis more believable, Howson conflates the distinction between confirmation and evidence. We demonstrate that Howson’s counterexample fails for a different pair of hypotheses.

Models carry the meaning of science. This puts a tremendous burden on the process of model selection. In general practice, models are selected on the basis of their relative goodness of fit to data penalized by model complexity. However, this may not be the most effective approach for selecting models to answer a specific scientific question because model fit is sensitive to all aspects of a model, not just those relevant to the question. Model Structural Adequacy analysis is proposed as a means to select models based on their ability to answer specific scientific questions given the current understanding of the relevant aspects of the real world.

The major competing statistical paradigms share a common remarkable but unremarked thread: in many of their inferential applications, different probability interpretations are combined. How this plays out in different theories of inference depends on the type of question asked. We distinguish four question types: confirmation, evidence, decision, and prediction. We show that Bayesian confirmation theory mixes what are intuitively “subjective” and “objective” interpretations of probability, whereas the likelihood-based account of evidence melds three conceptions of what constitutes an “objective” probability.

It can be demonstrated in a very straightforward way that confirmation and evidence as spelled out by us can vary from one case to the next, that is, a hypothesis may be weakly confirmed and yet the evidence for it can be strong, and conversely, the evidence may be weak and the confirmation strong. At first glance, this seems puzzling; the puzzlement disappears once it is understood that confirmation is of single hypotheses, in which there is an initial degree of belief which is adjusted up or down as data accumulate, whereas evidence always has to do with a comparison of one hypothesis against another with respect to the data and is belief-independent. Confusing them is, we suggest, a plausible source of the so-called “base-rate fallacy” identified by Kahneman and Tversky which leads most of us to make mistaken statistical inferences. It is also in the background, or so we argue in some detail, of the important policy controversies concerning human-induced global warming.