Clifford Miller

ABSTRACT: Abstract Most modern knowledge is not science. The physical sciences have successfully validated theories to infer they can be used universally to predict in previously unexperienced circumstances. According to the conventional conception of science such inferences are falsified by a single irregular outcome. And verification is by the scientific method which requires strict regularity of outcome and establishes cause and effect. Medicine, medical research and many “soft” sciences are concerned with individual people in complex heterogeneous populations. These populations cannot be tested to demonstrate strict regularity of outcome in every individual. Neither randomised controlled trials nor observational studies in medicine are science in the conventional conception. Establishing and using medical and other “soft science” theories cannot be scientific. It requires conceptually different means: requiring expert judgement applying all available evidence in the relevant available factual matrix. The practice of medicine is observational. Prediction of outcomes for the individual requires professional expertise applying available medical knowledge and evidence. Expertise in any profession can only be acquired through experience. Prior cases are the fundament of knowledge and expertise in medicine. Case histories, studies and series can provide knowledge of extremely high reliability applicable to establishing reliable general theories and falsifying others. Their collation, study and analysis should be a priority in medicine. Their devaluation as evidence, the failure to apply their lessons, the devaluation of expert professional judgement and the attempt to emulate the scientific method are all historic errors in the theory and practice of modern medicine.

Medicine, like law, is a pragmatic, probabilistic activity. Both require that decisions be made on the basis of available evidence, within a limited time. In contrast to law, medicine, particularly evidence-based medicine as it is currently practiced, aspires to a scientific standard of proof, one that is more certain than the standards of proof courts apply in civil and criminal proceedings. But medicine, as Dr. William Osler put it, is an "art of probabilities," or at best, a "science of uncertainty." One can better practice medicine by using other evidentiary standards in addition to the "scientific." To employ only the scientific standard of proof is inappropriate, if not impossible; furthermore, as this review will show, its application in medicine is fraught with bias. Evidence is information. It supports or undermines a proposition, whether a hypothesis in science, a diagnosis in medicine, or a fact or point in question in a legal investigation. In medicine, physicians marshal evidence to make decisions on how to best prevent, diagnose, and treat disease, and improve health. In law, courts decide the facts and render justice. Judges and juries assess evidence to establish liability, to settle custody and medical issues, and to determine a defendant's guilt or innocence.

Eyal Shahar’s essay review [1] of James Penston’s remarkable book [2] seems more inspired playful academic provocation than review or essay, expressing dramatic views of impossible validity. The account given of modern biostatistical causation reveals the slide from science into the intellectual confusion and non-science RCTs have created: “…. the purpose of medical research is to estimate the magnitude of the effect of a causal contrast, for example the probability ratio of a binary outcome …” But Shahar’s world is simultaneously not probabilistic, but of absolute uncertainty: “We should have no confidence in any type of evidence ….. We should have no confidence at all”. Shahar’s "Causal contrast" is attractive. It seems to make sense, but bypasses in two words the means of establishing causation by the scientific method. This phrase assumes a numeric statistically significant “contrast” is causal rather than a potential correlation requiring further investigation. The concept of “causal contrast” is a slippery slope from sense into biostatistical non-science. This can be illustrated with an hypothetical RCT where 100% of interventions exhibit a posited treatment effect and 0% of placebo controls. Internal validity is seemingly quite reasonably assumed satisfied (common-sense dictating the likelihood of an awesome magnificent fraud, bias or plain error of the magnitude required is infinitesimal). Scientific method appears satisfied. The RCT demonstrates: (1) strict regularity of outcome in the presence of posited cause; (2) the absence of outcome in its absence and (3) an intervention (experiment) showing the direction of causation is from posited cause to posited effect. Now travel further down the slope from science. Assume 50% of interventions and 0% of controls are positive. We compromise scientific method, but justify this by assuming a large subgroup which we say surely must on these figures be exhibiting the posited treatment effect. But what of 10% of interventions and 9% of placebo controls exhibiting the posited treatment effect? Our biostatistician says the 1% “causal contrast” is statistically significant. But we have: (1) minimal evidence of regularity; (2) the posited outcome irrespective of presence of posited cause and (3) our intervention is at the highest equivocal in demonstrating any form of causation. This is not science. It is, however, where biostatistics has unthinkingly taken us, as Penston has shown comprehensively [2]. We, the audience of published medical research, are now for the 10% / 9% example well down the slope from science. An unattractive hypothesis results requiring numerous assumptions similar to these:- "There is a 'contrast' which is ‘causal’, albeit the method employed is not scientific. An effect of the intervention has been observed in a very small subgroup. This subgroup is susceptible to treatment. The similar number of placebo controls exhibiting the outcome sought is irrelevant, because the 1% difference between intervention and controls is statistically significant. The statistical analysis is valid and reliable. The RCT’s internal validity is sufficiently satisfied. No funding or bias or fraud has affected the results or their analysis.” As Penston notes: “Confirming and refuting the results of research is crucial to science …. But … there’s no way of testing the results of any particular large-scale RCT or epidemiological study. Each study … is left hanging in the air, unsupported.” It gets worse. To identify a rare serious adverse reaction of a frequency of 1:10,000 can require a trial of 200,000 or larger split between controls and interventions. This is not done. But for every 100 who prospectively benefit from the intervention, 9,900 also receive it. And for every 100 benefiting one person (who likely gains no benefit) will suffer a serious unidentified adverse reaction. This is also without taking account of more common adverse reactions whether serious or otherwise. References [1] Shahar, E. (2011) Research and medicine: human conjectures at every turn. International Journal of Person Centered Medicine 1 (2), 250-253. [2] Penston, J. (2010). stats.con: How we’ve been fooled by statistics-based research in medicine. London: The London Press, UK.

Intuitively, Berger [1], commenting on Penston [2,3], must be right that we should not abandon RCTs. But here, intuition is no guide. Penston's strength is comprehensively marshalling the evidence, clarity of exposition, intellectual rigour and depth of analysis [4]. Dr Berger suggests some solutions: "needing better trials, more careful consideration of what can go wrong in trials, better reporting of trial results and better statistical input, we also need a more educated and involved general public". This has the ring of a political manifesto. In 1997 Britain, Tony Blair's New Labour government's emblematic theme was "Things can only get better". They didn't. With over fifty years of failure and tens of billions of dollars spent annually on medical research, is not suggesting "better" anything, too late? And how is better to be achieved? Is what exists now the best we will ever get? Berger attributes the "serious problems that occur with randomized trials" to how "they are currently conducted in practice". Penston, however shows that the problems are fundamental [4]. Large scale RCT's claiming treatment effects in small sub-groups are systematic, but not scientific. In this context, RCTs alone only ever demonstrate correlation; never causation. The answer to Berger's question "Can we attribute this failure to the involvement of statisticians, without even considering other possible explanations" is, therefore, "No". Berger asks "Is the answer, then, to dismiss statistical involvement altogether?" No. Berger is right that "statistical input is an asset, not a hindrance", but only in its rightful context and attributed an appropriate level of evidential reliability. I have little option but to agree with Penston. Large scale RCTs for small treatment effects are weak evidence of correlation and not scientific. Fraud, corruption and incompetence aside, they probably rank with early reports calling for detailed clinical investigations. The latter in turn call for real scientific peer review, rigorous replication or falsification. If that is right, are large scale RCTs to find small treatment effects worth the money or the effort? If not, what do we do? The textbooks, so far as I am aware, are yet to be written. The larger issue is the reliability of the medical evidence base. A recent paper in the IJPCM addresses the failure of EBM [5]. Its working title was Evidence Based Medicine or Ignorance Based Evidence. The medical evidence base is contradictory, conflicting and a source of a substantial body of ignorance believed to be truth, which is misdirecting medical practice daily. It is corrupt in all senses beyond tainted or adulterated [4]. As a body of evidence it, like large scale RCTs, is neither science nor scientific. Berger readily concedes "much statistical work … is grossly incompetent", but says "This does not justify condemning all statisticians with a broad brush ….. nor does it justify condemning all statistics-based research." This is the I'm only a little bit pregnant approach. It does not work. When a practitioner turns to the published medical evidence base, which papers does he or she trust? This goes well beyond statistics. James Penston has shown EBM has no roof. Neither does it possess foundations [5]. So, where to now? Given the opportunity in an academic environment, that is a question I am willing to attempt to address, if there is an institution brave enough to undertake and resource that journey.