A Summary of the Failures of Trick or Treatment?

by Simon Singh and Edzard Ernst

Introduction Trick or Treatment? by Simon Singh and Edzard Ernst claims to “examine the various alternative therapies in a scrupulous manner” (p.3). In Halloween Science we offer a full critique of their book, and assess the validity of this claim, both in general and specifically in respect of homeopathy, by analysing the authors’ own arguments and evidence for accuracy, consistency and reliability. The present article is a summary with examples of nineteen major faults exhibited in Trick or Treatment?. The faults are grouped under four headings: Evidence, Science, Definitions and Analytical Tools. These headings reflect the main areas of failure, and sub-headings relate to specific issues. Evidence 1. Unsupported evidence Many of the figures, trials, events, quotations, statements, opinions and explanations presented in Trick or Treatment are unreferenced, making it difficult to verify the information, despite the fact that some of these form a significant part of their argument. For example, the authors provide insufficient support for the following statements:

Figures: “Indeed, it is estimated that the annual global spend on all alternative medicines is in the region of £40 billion, making it the fastest-growing area of medical spending.” (p. 2) [In this case the information (amount spent) does not even support the concluson drawn from it (rate of growth in spending).] Trials: “In fact, a major study in 2006 confirmed numerous previous investigations showing that fears over mercury fillings were groundless.” (p. 265) [This actually appears to have been two separate studies.] Events: “This success was repeated during a cholera epidemic in London in 1854, when patients at the London Homoeopathic Hospital had a survival rate of 84 per cent, compared to just 47 per cent for patients receiving more conventional treatment at the nearby Middlesex Hospital.” (p. 107) Quotations: “ ‘A therapeutic agent cannot be employed with any discrimination or probability of success in a given case, unless its general efficacy, in analogous cases, has been previously ascertained’.” (p. 23) [This is ascribed to Pierre Louis. No support is offered for its basic assumption.] Statements: “These treatments are piled high in every pharmacy, written about in every magazine, discussed on millions of web pages and used by billions of people, yet they are regarded with scepticism by many doctors.” (p.1) [The scale of these figures requires supporting evidence. For example, with a world population of approximately seven billion people, “billions” means more than 28% of people.] Opinions: “Homeopaths would argue that the remedy has some memory of the original ingredient, which somehow influences the body, but this makes no scientific sense.” (p.100) [No justification is offered.] Explanations: “This would involve giving daily doses of a homeopathic remedy to several healthy people and then asking them to keep a detailed diary of any symptoms that might emerge over the course of a few weeks.” (p. 96) [This is actually an inaccurate description of a homeopathic proving.]

2. Information out of context Not only is information unreferenced, but it is often without context. Thus the figure quoted above for “the annual global spend on all alternative medicines” is not put into the context of the estimated $4.1 trillion (£2.8 trillion) global spend on medicine as a whole. (1) Similarly the statement that

The bottom line is that none of the above [alternative] treatments is backed by the sort of evidence that would be considered impressive by the current standards of medical research. (p. 238)

is not compared with the British Medical Journal’s Clincal Evidence report that

Of around 2500 [commonly used NHS] treatments covered 13% are rated as beneficial, 23% likely to be beneficial, 8% as trade off between benefits and harms, 6% unlikely to be beneficial, 4% likely to be ineffective or harmful, and 46%, the largest proportion, as unknown effectiveness.(2)

3. Double-standards for evidence The authors accept material which supports their argument despite its failure to meet the standards they set for material which supports an opposing view. For example, they complain about the Bristol Homeopathic Hospital outcome survey (2005) that

The study had no control group, so it was impossible to determine whether these patients would have improved without any homeopathic treatment. (p. 140)

Yet they claim that it is possible to determine consequential harm without a control group:

There are numerous reports of patients with serious conditions (e.g. diabetes, cancer, AIDS) suffering harm after following irresponsible advice from alternative practitioners instead of following the advice of a doctor. (p. 186)

Similarly they refer to a spoof story about “DiHydrogen MonOxide” (H2O), alleged to show that

‘You can give people this totally accurate (but emotionally laden, and sensationalist) information about water. When you then survey these people, about three quarters of them will willingly sign a petition to ban it.’ (p. 267)

But they do not mention whether this research has been replicated and confirmed, although they point out that “independent replication is a vital part of how science progresses.” (p. 125). Nor do they provide any information about what medium was used to publish the article, what size of population was involved, how they were selected, what control was used, how the responses were surveyed, nor, crucially, what relationship the population sample has to the population using alternative medicine. In other words, it is purely anecdotal and satisfies none of the requirements they insist are necessary for a valid trial.

Science

4. Confusion of absence of proof with proof of absence The authors repeatedly assume that lack of evidence can be taken as proof that such evidence is unobtainable, and that a theoretical explanation is wrong. Thus they claim that “the traditional principles of acupuncture are deeply flawed, as there is no evidence at all to demonstrate the existence of Ch’i or meridians” (p. 83). Similarly, in the case of homeopathy the authors allege that “hundreds of trials have failed to deliver significant or convincing evidence to support the use of homeopathy for the treatment of any particular ailment” (p. 139), yet they then go on to use this alleged lack of evidence to claim that “the scientific evidence indicates that homeopathy is wholly ineffective” (p. 231). In each case there is no evidence against the therapeutic approach, but a mixture of evidence for it(i) which is good, ambiguous or insufficient. This suggests that there is a problem of lack of research, not lack of validity of the therapeutic approach.

5. Disregard for the importance of theory On the very first page, the authors state that “science employs experiments, observations, trials, argument and discussion in order to arrive at an objective consensus on the truth” (p. 1). They go on to state that “Chapter 1 provides an introduction to the scientific method. It explains how scientists, by experimenting and observing, can determine whether or not a particular therapy is effective” (p. 4). At no point, however, do they mention the importance of theory to science, despite the fact that an essential part of the scientific method is the interaction of experiment with theory. Nor do they discuss the relative merits and justificatons for different medical theories. Instead the authors refer to their “scientific evidence” in the abstract, as though it were independent of its specific context of randomised controlled trials based on the pharmaceutical research model. Thus, they fail to acknowledge the existence of their own theoretical assumptions, and fail to question the appropriateness of those assumptions when assessing alternative medicine.

6. Assumption that orthodox medicine is scientific No justification of orthodox medicine as a science is ever provided, and yet all references to the “evidence”, “testing”, “trials”, “studies”, “investigations”, “research”, “information”, “criticisms”, “methods”, “foundations”, “rigour”, “approach”, “attitude”, “thinking”, “understanding” or “point of view” are described as “scientific” when based on the pharmaceutical model of orthodox clinical trials. (3) In this way the authors appear to be trying to establish, through repetition rather than reason, the idea that this approach alone is the “scientific” means of testing the validity of alternative medicine. They also refer to alternative therapies as “unscientific” (p. 163), or as having “no scientific sense” (pp. 100 and 226).

Similarly, without any explanation of the meaning in this context of “philosophies” (that is, ‘theories’) or of the alleged “conflict” the authors state that

These other therapies have struggled to be accepted by mainstream medicine, partly because their underlying philosophies conflict with our scientific understanding of anatomy, physiology and pathology. (p. 196)

7. Failure to understand orthodox medicine Curiously, the authors make mistakes about orthodox medicine, such as claiming that “the term ‘vitamin’ describes an organic nutrient that is vital for survival, but which the body cannot produce itself” (p. 15), when the body can produce vitamins A, B3, D and K; or appearing to confuse chronic pancreatitis with acute pancreatitis (p. 186). They also generalise “the ability of oranges and lemons to cure scurvy” (p. 18) into evidence that the RCT can be used “to decide hat works (lemons for scurvy)” (p. 36). In fact, vitamin C (and fruit containing it) successfully treat scurvy because scurvy is simply a result of a deficiency of vitamin C. As such, this treatment has no similarity with orthodox or alternative treatments for infections and chronic diseases, and to confuse the two types of treatment suggests a general failure to understand the nature of medicine.

Definitions

8. Four different definitions of alternative medicine The authors initially define alternative medicine as (our emphasis)

… any therapy that is not accepted by the majority of mainstream doctors, and typically this also means that these alternative therapies have mechanisms that lie outside the current understanding of modern medicine. (p. 1). In Chapter 4 the primacy of this lack of acceptance later changes to that of lack of understanding of the mechanism of action, since “chiropractors have become part of the medical mainstream” (p. 147), but their therapy allegedly “makes no sense at all from a modern scientific point of view. That is why chiropractic treatment is still considered by many as an alternative medicine” (p. 147).

In Chapter 5 the authors again note that “other therapies have struggled to be accepted by mainstream medicine” (p. 196), but in the case of herbal medicine … plants contain a complex cocktail of pharmacologically active chemicals, so it is not surprising that some of them can impact on our wellbeing. Consequently, herbal medicine has been embraced by science to a far greater extent than the other treatments above. (p. 196) They add that “there is general agreement that much of modern pharmacology has evolved out of the herbal tradition” (p. 196). As a result acceptance is now based on understanding rather than being contrasted with it, this understanding being specifically the chemical action recognised by pharmacology.

Finally, in Chapter 6, acceptance becomes dependent on testing according to the procedures used by pharmacology, tests which have been repeatedly called “scientific”. Thus the authors claim that this brings us to an interesting situation: any provably safe and effective alternative medicine is ot really an alternative medicine at all, but rather it becomes a conventional medicine. Therefore, alternative medicine, by definition, seems to consist of treatments that are untested, or unproven, or disproven, or unsafe, or placebos, or only marginally beneficial. (p. 287) However the principles on which these “scientific” tests of pharmacology are based have never been properly explained or scientifically justified. Furthermore, while alternative therapies may not be accepted because they have not been proven effective according to criteria external to those therapies, many common orthodox treatments are accepted whether or not they have been proven effective according to the uthors’ own criteria, since (as we have already pointed out) orthodox medicine includes 64% to 87% of commonly used treatments which have not been proven safe and effective.

9. Failure to define significant terms We have noted the authors’ failure to define ‘science’ properly and their constant reference to orthodox medicine as “scientific”. In addition, their terms for orthodox medicine do not refer to its theoretical principles but only to its official status, such as “mainstream”, “conventional” or “establishment”. Some essential terms are not defined at all, such as ‘disease’, ‘cure’ and ‘effective’. Given that the whole of their examination rests on proving whether alternative medicine is effective or not, failure to define this term seriously undermines their argument. The authors have an ambivalent attitude to some other terms, such as ‘holistic’ and ‘individualisation’. On the one hand they refer to them as “impressive buzzwords” (p. 2), but on the other they use them as legitimate terms (pp. 138 and 223). They also refer to “the fundamental question: ‘Is alternative medicine effective for treating disease?’” (p. 3), and then note that “when unpacked it becomes somewhat complicated and has many answers” (p. 3). Indeed it ‘unpacks’ to no less than 25,900 questions, which makes their definition of ‘fundamental’ unusual at the least.

10. Arguments based on readers’ preconceptions Because the authors fail to define their terms, readers are frequently left to assume meanings for them. The effect of this is that readers rely on preconceptions which are unquestioned and unjustified but assumed to be valid. For example, the term ‘effective’ is undefined, but it is constantly used by the authors with reference to RCTs. In the absence of any definition of the criteria being used or of the appropriateness of these criteria to specific trials, the reader assumes a ‘valid’ definition. This means that conclusions based on these trials are accepted, depite the lack of evidence for the validity of these trials and the reliability of their results. In other words, the authors’ are presenting their arguments so as to deliberately exploit "confirmation bias, which is the tendency to interpret events in a way that confirms preconceptions” (p. 234). 11. Failure to take into account different definitions of terms The failure of the authors to define their terms has even greater significance given that some alternative therapies (especially homeopathy) have definitions of ‘disease’, ‘cure’ and ‘effectiveness’ which are significantly different from those used by orthodox medicine. In Halloween Science we present eleven examples of how these differences can impact on clinical trials, causing the results to range from ambiguous to meaningless. The authors ignore these issues, yet their own comment about trials of homeopathy indicates that such a failure may be having real consequences, since “over and over again, the evidence is either non-exstent or shaky” (p. 139). Furthermore, other therapies also exhibit ambiguity in RCT evidence, indicating that the problem of erroneous definitions may affect trials of those therapies too.

12. Failure to present the principles of evidence-based medicine accurately (EBM) Just as the authors leave theory out of their definition of science, so they leave clinical expertise based on experience out of their definition of EBM. They quote David Sackett as stating that “Evidence-based medicine is the conscientious, explicit, and judicious use of current best evidence in making decisions about the care of individual patients” (p. 24), but they do not add that he went on to state that “without clinical expertise, practice risks becoming tyrannised by evidence, for even excellent external evidence may be inapplicable to or inappropriate for an individual patient”. (4) Instead they rely on RCTs alone and attack evidence from clinical experience. In the case of the Bristol Homeopathic Hospital outcome survey, for example, “As far as the public was concerned, this appeared to be an extraordinarily positive result” (p. 140), but the authors claim that this “70 per cent improvement rate” (p. 140) is “largely meaningless” (p. 140), and justify their opinion with explanations which are incompatible with the facts or their other statements.

13. Failure to present homeopathy accurately The authors describe the nature and development of homeopathy so inadequately and inaccurately as to make it impossible to assess the validity of their arguments. They state that “after the dilution, the mixture is vigorously shaken, which completes the potentization process” (p. 98), yet immediately afterwards refer to (our emphasis) “further dilution and potentization” (p. 98). They refer to “the remedy that offers a perfect match with the patient’s symptoms” (p. 101), yet show a perfect match is impossible as their example has symptoms which cannot coexist in one person at one time. They state that the homeopathic term “miasmatic” (p. 255) refers to “poisonous vapours” (p. 255) and that homeopaths “tend to reject … the role of bacteria as agents of disease” (p. 105) even though Hahnemann himself stated that the cholera-miasm finds a favourable element for its multiplication, and grows into an enormous brood of those excessively minute, invisible, living creatures, so inimical to human life, of which the contagious matter of the cholera most probably consists. (5) With errors of this magnitude in their explanation of one therapy, it is reasonable to distrust their explanations of all the other therapies too, but we do not have the expertise to recognise if such errors do actually occur in other cases.

14. Doubts about the validity of orthodox drug therapy The authors show that orthodox drug therapy attempts to find “the active ingredient of each plant and isolate it” (p. 197), and yet they also accept that in some cases the effects may be “due to a combination of chemicals, each one working to enhance the effect of the others” (p. 200), and further that “we now accept that almost every medical intervention carries a risk of side-effects” (p. 205). In other words they acknowledge that the goal of a single chemical with a single effect is illusory, a point confirmed by knowledge of the variable action of chemicals at cellular level. (6) The authors consider (our emphasis) “meticulously documenting its impact on a total of 156 patients” (p. 194) (that is, people suffering from diseases) to be a good testing regime for a drug, yet The Merck Manual of Medical Information notes that “many factors influence drug response”, including “disease”. (7) In other words testing drugs on the sick is an inherently flawed approach. The authors claim that the general results of trials are essential for determining treatment for individual patients, yet state that there is “no guarantee that a treatment that had succeeded during a set of trials would cure a particular patient” (p. 23). These issues raise serious questions about the validity of the approach used by orthodox medicine, and about the validity of using its tests to assess alternative medicine. The authors also attack the majority of front-line orthodox practitioners as unscientific, alleging that they are “ignorant” (p. 269), “lazy” (p. 269) or “convinced … despite all the lack of evidence” (p. 270) when it comes to alternative medicine.

Analytical tools 15. Failure of the Randomised Controlled Trial (RCT) While the authors demonstrate that the RCT is an appropriate tool for identifying harmful interventions, they fail to offer evidence of its validity as a test of beneficial interventions. They also note that after trials have been completed “doctors are encouraged to continue to monitor and report any adverse incidents … [so] we can, if risks emerge, withdraw a drug” (p. 178). In other words, RCTs by themselves are not even reliable guides to the extent of harm produced by drugs. As has been stated above, there is also “no guarantee that a treatment that had succeeded during a set of trials would cure a particular patient” (p. 23), so the RCT does not produce evidence valid for an individual case. In other words, the RCT is not an appropriate tool for identifying whether alternative medicine is effective.

16. Failure of the meta-analysis The authors rely on meta-analyses of RCTs for their conclusions about homeopathy and chiropractic therapy, yet they point out that, for example, “Not surprisingly, Linde’s conclusion was questioned by opponents of homeopathy. Critics argued that his meta-analysis had been too lax” (p. 134). Similarly the meta-analysis by Shang et al. (8) met with great criticism, particularly as regards its lack of information about the criteria used for selecting the final fourteen trials out of 110 ‘matched pairs’. (9) In short, not only are such trials based on RCTs which may not be valid, but they are also liable to subjectivity in the choice of selection criteria.

17. Failure of the placebo effect The authors explain the effects of alternative medicine primarily by reference to the placebo effect without any scientific justification. Not only do they admit that “scientists strive to establish the scientific basis of the placebo effect” (p. 62), but they acknowledge that it is variable (p. 244); individual (p. 64); may be stimulated by completely opposite circumstances, such as “novelty” (p. 57) or “tradition” (p. 223); and can be an “ineffective treatment that can nevertheless be consoling” (p. 57), or produce “real physiological changes” (p. 60) without any explanation for these different consequences. They also fail to show that there is any consistent similarity between the placebo effect and the observed effects of alternative therapies.

18. Denial of the importance of individuality The authors acknowledge that the curative process is individually determined. In the case of drug treatments they note that, despite the RCT, “there was still no guarantee that a treatment … would cure a particular patient” (p. 23), whilst in the case of the placebo effect they allege that “the actual placebo effect for a particular patient depends entirely on the belief system and personal experiences of that individual” (p. 62). This indicates that Ernst and Singh should be aware that any scientific system of medicine needs to take individuality into account. However, when testing alternative therapies which consider individualisation essential, the authors devalue its importance. For example, when discussing homeopathy they note that “most trials have not been individualized” (p. 138), but they do not point out that those trials are therefore not valid or at least suspect. Furthermore, in the examples they then give of “individualized” trials, the individualisation is wholly inadequate.

19. Denial of the importance of clinical experience The limited version of evidence-based medicine used by the authors relies exclusively on controlled clinical trials, even though these provide only generalised evidence of effectiveness. As has been pointed out above, the full approach insists that expertise derived from clinical experience is necessary for the selection of the correct treatment in a particular case because of the individuality of patients. Clinical experience is also crucial in revealing some of the harmful effects of drugs tested by RCT. In the case of homeopathy the definition of effectiveness recognises and uses the individuality of the patient as a basis for both determining treatment and assessing the results. As a result there may be “conflict between personal experience and scientific research” (p. 231) because of a failure to define the research protocols correctly. Such is the authors’ reliance on RCTs and devaluation of clinical experience, however, that they do not challenge the trial protocols in order to explain this “conflict” even though eleven possible failures can be identified in these protocols. Instead they challenge the validity of experience. In doing so they attribute the higher rates of success in clinical practice to “the Hawthorne effect” (p. 65), “natural healing processes” (p. 140), “regression to the mean” (p. 233), remedies “contaminated, perhaps with steroids or other conventional pharmaceuticals” (p. 232), “other treatments” (p. 140), “coincidence” (p. 232), “the placebo effect” (p. 140), and even “patients being reluctant to disappoint whoever was interviewing them” (p. 140). None of these ‘explanations’ is supported by research evidence demonstrating that they have a significant impact on outcomes.

Conclusions What is clear from the points listed above, is that Ernst and Singh have failed to provide a secure theoretical or evidential base for their argument. They have not defined their basic terms, they have not presented a theoretical relationship between evidence and practice, and they have arbitrarily rejected evidence. They have also used analytical tools which are either inherently inadequate for achieving objective and reliable conclusions, or which have been rendered inadequate for such a purpose by the limitations the authors set on their use. Furthermore, they rely heavily on unsupported statements, preconceptions, perjorative language, hyperbole, double standards, and facts which are misrepresented, juxtaposed and removed from context in order to support their argument. Such a biased and wholly inadequate examination of alternative medicine by two “trained scientists” (p. 3) is damaging to the credibility of orthodox medical researchers and science in general. Trick or Treatment? also encourages a hazardous therapeutic environment for patients. By exploiting prejudice whilst claiming to offer “an unparalleled level of rigour, authority and independence” (p. 3), the authors help to alienate doctors and alternative therapists from each other. As a result patients are faced with the increased likelihood of orthodox and alternative practitioners being unable to communicate with each other or learn from the outcomes of different therapeutic approaches. They may even be faced with contradictory treatments which cannot be reconciled or even discussed with the relevant practitoners. In brief, Trick or Treatment? has no validity as a scientific examination of alternative medicine, but is damaging to the credibility of science, and a threat to the safe and effective practice of medicine.

Notes i) Paolo Bellavite and Andrea Signorini, The Emerging Science of Homeopathy: Complexity, biodynamics, and nanopharmacology (Berkley: North Atlantic Books, 2002).

Bibliography 1. Spending On Health: A Global Overview, World Health Organization, 2007, http://www.who.int/mediacentre/factsheets/fs319/en/index.html, accessed 21 February 2008. 2. BMJ Clinical Evidence website, http://clinicalevidence.bmj.com/ceweb/about/knowledge.jsp, accessed 23 February 2009. 3. See pp. 4, 34, 35, 100, 116, 122, 147, 194, 196, 197, 199, 223, 231, 234, 239, 244, 269, 276, 284 for examples. 4. David L Sackett, William M C Rosenberg, J A Muir Gray, R Brian Haynes, W Scott Richardson, ‘Evidence based medicine: what it is and what it isn't’, BMJ, 312 (1996), 71-72 (13 January), at http://www.bmj.com/cgi/content/full/312/7023/71, accessed 6 December 2008. 5. Samuel Hahnemann, ‘Appeal to Thnking Philanthropists Respecting the Mode of Propagation of the Asiatic Cholera’, (Leipzic: the author, 1831) in Samuel Hahnemann (trans. R E Dudgeon MD), The Lesser Writings of Samuel Hahnemann, 1851 edn (New Delhi: B. Jain Publishers, repr. edn 2002), p. 758. 6. See Paolo Bellavite MD and Andrea Signorini MD, The Emerging Science of Homeopathy: Complexity, biodynamics, and nanopharmacology (Berkley: North Atlantic Books, 2002), p. 141. 7. Robert M.D. (Editor in Chief) Berkow, The Merck Manual of Medical Information: Home edition (New York: Simon and Schuster Inc., 2000), p. 35. 8. Aijing Shang MD, Karin Huwiler-Müntener MD, Linda Nartey MD, Peter Jüni MD, Stephan Dörig, Jonathan AC Sterne PhD, Daniel Pewsner MD, Prof Matthias Egger MD, ‘Are the clinical effects of homoeopathy placebo effects? Comparative study of placebo-controlled trials of homoeopathy and allopathy’, The Lancet, 366 (2005),726-732 at http://www.thelancet.com/journals/lancet/article/PIIS0140-6736(05)67177-2/fulltext, accessed 3 March 2009. 9. Klaus Linde, Wayne B Jonas, ‘Meta-analysis of homoeopathy trials’ (letter to the editor), The Lancet, 9503 (2005) at http://www.thelancet.com/journals/lancet/article/PIIS0140-6736(05)67878-6/fulltext, accessed 3 March 2009; Peter Fisher, Brian Berman, Jonathan Davidson, David Reilly, Trevor Thompson and 29 others, Letter to the editor, The Lancet, 9503 (2005) at http://www.thelancet.com/journals/lancet/article/PIIS0140-6736(05)67879-8/fulltext, accessed 3 March 2009; and R Lüdtke and A L B Rutten, ‘The conclusions on the effectiveness of homeopathy highly depend on the set of analyzed trials’, J. Clin. Epidemiol., (2008) at ; and A L B Rutten and C F Stolper, ‘The 2005 meta-analysis of homeopathy: the importance of post-publication data’, Homeopathy, 2008 at .

Author: William Alderson RSHom LLSCH, on behalf of H:MC21 (Homeopathy: Medicine for the 21st Century). March 2009.

Source: http://www.homeopathyworkedforme.org/#/halloweenscience/4533482584