Although the term meta‑analysis might be unfamiliar to many readers, it is a concept that crops up in a range of familiar situations where it is important to make sense of lots of data. In the run‑up to a general election, for instance, several newspapers might publish opinion polls with conflicting results. In this situation it would be sensible to combine all the data from all the polls, which ought to lead to a more reliable conclusion than any single poll, because the meta‑poll (i.e. poll of polls) reflects the complete data from a much larger group of voters.
The power of meta‑analysis becomes obvious if we examine some hypothetical sets of data concerning astrology. If your astrological sign determined your character, then an astrologer should be able to identify a person’s star sign after an interview. Imagine that a series of five experiments is conducted around the world by rival research groups. In each case, the same astrologer is simply asked to identify correctly a person’s star sign based on a five‑minute conversation. The experiments range in size from 20 to 290 participants, but the protocol is the same in each case. Chance alone would give rise to a success rate of one correct identification (or hit) in twelve, so the astrologer would have to do significantly better than this to give credence to the notion of astrology. The five experiments lead to the following success rates:
| Experiment 1 | 12 | hits out of | 170 | (equivalent to 0.85 hits out of 12) |
| Experiment 2 | 5 | hits out of | 50 | (equivalent to 1.20 hits out of 12) |
| Experiment 3 | 5 | hits out of | 20 | (equivalent to 3.00 hits out of 12) |
| Experiment 4 | 6 | hits out of | 70 | (equivalent to 1.03 hits out of 12) |
| Experiment 5 | 21 | hits out of | 290 | (equivalent to 0.87 hits out of 12) |
On its own, the third experiment seems to suggest that astrology works, because a hit rate equivalent to 5 out of 20 is much higher than chance would predict. Indeed, the majority of experiments (three out of five) imply a higher than expected hit rate, so one way to interpret these sets of data would be to conclude that, in general, the experiments support astrology. However, a meta‑analysis would come to a different conclusion.
The meta‑analysis would start by pointing out that the number of attempts made by the astrologer in any one of the experiments was relatively small, and therefore the result of any single experiment could be explained by mere chance. In other words, the result of any one of these experiments is effectively meaningless. Next, the researcher doing the meta‑analysis would combine all the data from the individual experiments as though they were part of one giant experiment. This tells us that the astrologer had 49 hits out of 600 in total, which is equivalent to a hit rate of 0.98 out of 12, which is very close to 1 out of 12, the hit rate expected by chance alone. The conclusion of this hypothetical meta‑analysis would be that the astrologer has demonstrated no special ability to determine a person’s star sign based on their personality. This conclusion is far more reliable than anything that could have been deduced solely from any one of the small‑scale experiments. In scientific terms: a meta‑analysis is said to minimize random and selection biases.
Turning to medical research, there are numerous treatments that have been tested by meta‑analysis. For example, in the 1980s researchers wanted to know if corticosteroid medication could help reduce respiratory problems in premature babies. They designed a trial which involved giving the treatment to pregnant women likely to have premature births and then monitoring the babies born to these mothers. Ideally, the researchers would have conducted one trial in a single hospital with a large number of cases, but it was only possible to identify a few suitable cases each year per hospital, so it would have taken several years to accumulate sufficient data in this manner. Instead, the researchers conducted several trials across several hospitals. The results of each individual trial varied from hospital to hospital, because the numbers of babies in each trial was small and random influences were large. Yet a meta‑analysis of all the trials showed with certainty that corticosteroid medication during pregnancy benefited premature babies. This treatment is part of the reason why the number of infant deaths due to respiratory distress syndrome has fallen dramatically–there were 25,000 such deaths in America in the early 1950s and today the number is fewer than 500.
The meta‑analysis in the premature baby study was fairly straightforward, because the individual trials were similar to each other and so they could be merged easily. The same is true of the hypothetical example concerning astrology. Unfortunately, conducting a meta‑analysis is often a messy business, because the individual trials have generally been conducted in different ways. Trials for the same medication might vary according to the dose given, the period of monitoring, and so on. In Linde’s case, the meta‑analysis was particularly problematic. In order to draw a conclusion about the efficacy of homeopathy, Linde was attempting to include homeopathy trials investigating a huge variety of remedies, across a range of potencies, being used to treat a wide range of conditions, such as asthma and minor burns.
Linde trawled through the computer databases, attended numerous homeopathic conferences, contacted researchers in the field and eventually found 186 published trials on homeopathy. He and his colleagues then decided to exclude from his meta‑analysis those trials that failed to meet certain basic conditions. For example, in addition to a group of patients being treated with homeopathy and a control group of patients, an acceptable trial had to have a placebo for the control‑group patients, or there had to be random allocation of patients to the treatment and control groups. This left eighty‑nine trials. What followed was months of careful statistical analysis, so that each trial contributed appropriately to the final result. For example, the result of a very small trial would carry very little weight in the overall conclusion, because the reliability of a trial’s result is closely linked to the number of participants in the trial.
The meta‑analysis was eventually published in September 1997 in the Lancet. It was one of the most controversial medical research papers of the year, because its conclusion endorsed exactly what homeopaths had been saying for two centuries. On average, patients receiving homeopathy were much more likely to show signs of improvement than those patients in the control groups receiving placebo. The paper concluded: ‘The results of our meta‑analysis are not compatible with the hypothesis that the clinical effects of homeopathy are completely due to placebo.’ In other words, according to the meta‑analysis, homeopathy was genuinely effective.