Not surprisingly, Linde’s conclusion was questioned by opponents of homeopathy. Critics argued that his meta‑analysis had been too lax, inasmuch as it had included too many trials of relatively poor quality, and they feared that these might have biased the overall conclusion in favour of homeopathy. Homeopaths responded that there had been a quality threshold, which Linde had implemented specifically in order to exclude poor‑quality trials. Remember, Linde had included only those trials that were placebo‑controlled or randomized. Nevertheless, the critics were still unhappy, as they maintained that the quality threshold had not been set high enough.
Because poorer‑quality trials are more likely to produce misleading results, researchers have developed techniques for assessing quality and weeding out those trials that should not be taken seriously. For example, the Oxford quality scoring system, developed in 1996 by Alejandro Jadad and his colleagues at Oxford University, can be used to give a trial a score between 0 points (very poor) and 5 points (rigorous). The system works by awarding or deducting points according to what appears in the published version of the trial. So if the research paper confirms that there was randomization of patients then it receives a point, yet this point can be deducted if the randomization appears to have been inadequate. Or the trial can score a point if the paper describes how the researchers dealt with the data from patients who dropped out from the trial. If the researchers have thought about this in detail and bothered to describe it in their research paper, then it is a good indication of their overall level of rigour.
Critics pointed out that sixty‑eight out of the eighty‑nine trials in Linde’s meta‑analysis scored only 3 or less on the Oxford scale, which meant that three‑quarters of the trials were substandard. Moreover, critics pointed out that restricting the meta‑analysis to the higher‑quality trials (4 or 5 points) drastically reduced the apparent efficacy of homeopathy. In fact, the conclusion of the twenty‑one higher‑quality trials was that homeopathy offered either a small benefit for patients or no benefit at all. Despite the amount of data available from these twenty‑one trials, it was still not possible to distinguish between these two possibilities.
In time, Linde and his colleagues agreed that their critics had a valid point, and in 1999 they published a follow‑up paper that re‑examined the same data with a special emphasis on the quality of the individual trials. Linde wrote: ‘We conclude that in the study set investigated, there was clear evidence that studies with better methodological quality tended to yield less positive results.’ Then, referring back to the original meta‑analysis, he stressed: ‘It seems, therefore, likely that our meta‑analysis at least over‑estimated the effects of homeopathic treatments.’
Linde’s original 1997 paper had supported homeopathy, yet his revised 1999 paper was much more equivocal. His re‑analysis of his own meta‑analysis obviously disappointed the alternative medicine community, yet it was also frustrating for the medical establishment. Everyone was dissatisfied because Linde was neither able to claim that homeopathy was effective, nor was he able to dismiss it as a mere placebo.
Despite the lack of clear‑cut evidence in either direction, the public was increasingly turning towards homeopathy, either consulting practitioners or buying over‑the‑counter remedies. This gave researchers a renewed sense of urgency to test the therapy via larger, more rigorously conducted trials. Hence, homeopathy was subjected to a much higher level of scrutiny from the late 1990s onwards.
This eventually prompted Dr Aijing Shang and his colleagues at the University of Berne, Switzerland, to undertake a fresh meta‑analysis of all the trials published up to January 2003. The medical research group at Berne, which is led by Professor Mathias Eggers, has a world‑wide reputation for excellence and the Swiss government had provided the team with adequate funding for a fully rigorous meta‑analysis. Hopes were high that Shang would at last be able to deliver a reliable conclusion. Indeed, after two centuries of bitter dispute between homeopaths and mainstream medics, Shang’s meta‑analysis was destined to decide, at last, who was right and who was wrong.
Shang was ruthless in his demand for quality, which meant that his meta‑analysis included only those trials with large numbers of participants, decent blinding and proper randomization. In the end, he was left with only eight homeopathy trials. After studying the data from these eight trials–the best available trials on homeopathy–his meta‑analysis reached its momentous conclusion. On average, homeopathy was only very marginally more effective than placebo. So, did this tiny marginal average benefit suggest that homeopathy actually healed patients?
Before answering this question, it is important to realize that the results of every scientific analysis are always associated with a level of uncertainty. For example, analysing the age of the Earth gives a result of 4,550 million years, and the error on this age is give or take 30 million years. The uncertainty associated with Shang’s estimate on the efficacy of homeopathy was such that his conclusion was wholly compatible with the judgement that homeopathy acted as nothing more than a placebo. In fact, the most sensible interpretation of the meta‑analysis was that homeopathy was indeed nothing more than a placebo.
This interpretation becomes more convincing if we bear in mind another aspect of his research. While conducting his meta‑analysis on homeopathy, he also conducted a meta‑analysis for a whole variety of new, conventional pharmaceuticals. These pharmaceuticals had been tested on the same illnesses that had been considered for the homeopathy meta‑analysis. In this secondary meta‑analysis, Shang scrupulously applied exactly the same selection criteria to these conventional drug trials as he had done in his homeopathy meta‑analysis. The result of his meta‑analysis on conventional drug trials was that on average they worked. Although this result also had an uncertainty associated with it, the average benefit was so large that the effectiveness of these new conventional drugs was not in any doubt.
The contrast between the homeopathic trials and the conventional drug trials was striking. Homeopathy had failed to show a clear benefit for patients and the result was compatible with homeopathy acting as a placebo, whereas conventional drugs had shown a clear benefit for patients, which suggested that they do indeed have a genuine physiological impact on the body. This illustrates the stark difference between pseudo‑medicine and real medicine.
Shang published his results in the Lancet in August 2005. Based on his meta‑analysis, he concluded: ‘This finding is compatible with the notion that the clinical effects of homeopathy are placebo effects.’ Reinforcing this point, the Lancet ran an editorial entitled ‘The end of homeopathy’ in which they argued that ‘doctors need to be bold and honest with their patients about homeopathy’s lack of benefit’. This sparked major news stories around the world, angering homeopaths who refused to accept the conclusions of Shang’s meta‑analysis and the Lancet ’s accompanying statement. They attempted to undermine the research by pointing out four key issues, but in fact each of their criticisms can be easily addressed.
1. Homeopaths might argue that Shang’s paper indicates a positive effect for homeopathy, and that his meta‑analysis therefore supports homeopathy.
There is indeed a positive effect for homeopathy, but it is very small and entirely compatible with the treatment being a placebo. Shang’s paper is the most comprehensive analysis of homeopathy during its 200‑year history, and such a paucity of positive evidence has to be interpreted as a blow for homeopathy. Crucially, Shang’s analysis confirms the results of about a dozen other meta‑analyses and systematic reviews published over the last decade, all of which fail to show that homeopathy offers any benefit beyond placebo.