Believing that your hypothesis must be correct before all the evidence is gathered encourages you to interpret the evidence selectively. This is human nature. It is also precisely what the scientific method tries to avoid. It does so by requiring that scientists not just test their hypotheses, but try to prove them false. “The method of science is the method of bold conjectures and ingenious and severe attempts to refute them,” said Karl Popper, the dean of the philosophy of science. Popper also noted that an infinite number of possible wrong conjectures exist for every one that happens to be right. This is why the practice of science requires an exquisite balance between a fierce ambition to discover the truth and a ruthless skepticism toward your own work. This, too, is the ideal albeit not the reality, of research in medicine and public health.
In 1957, Keys insisted that “each new research adds detail, reduces areas of uncertainty, and, so far, provides further reason to believe” his hypothesis. This is known technically as selection bias or confirmation bias; it would be applied often in the dietary-fat controversy. The fact, for instance, that Japanese men who lived in Japan had low blood-cholesterol levels and low levels of heart disease was taken as a confirmation of Keys’s hypothesis, as was the fact that Japanese men in California had higher cholesterol levels and higher rates of heart disease. That Japanese men in California who had very low cholesterol levels still had more heart disease than their counterparts living in Japan with similarly low cholesterol was considered largely irrelevant.
Keys, Stamler, and their supporters based their belief on the compelling nature of the hypothesis supplemented only by the evidence in support of it. Any research that did not support their hypothesis was said to be misinterpreted, irrelevant, or based on untrustworthy data. Studies of Navajo Indians, Irish immigrants to Boston, African nomads, Swiss Alpine farmers, and Benedictine and Trappist monks all suggested that dietary fat seemed unrelated to heart disease. These were explained away or rejected by Keys.
The Masai nomads of Kenya in 1962 had blood-cholesterol levels among the lowest ever measured, despite living exclusively on milk, blood, and occasionally meat from the cattle they herded. Their high-cholesterol diets supplied nearly three thousand calories a day of mostly saturated fat. George Mann, an early director of the Framingham Heart Study, examined the Masai and concluded that these observations refuted Keys’s hypothesis. In response, Keys cited similar research on the Samburu and Rendille nomads of Kenya that he interpreted as supporting his hypothesis. Whereas the Samburu had low cholesterol—despite a typical diet of five to seven quarts of high-fat milk a day, and twenty-five to thirty-five hundred calories of fat—the Rendille had cholesterol values averaging 230 mg/dl, “fully as high as United States averages.” “It has been estimated,” Keys wrote, “that at the time of blood sampling the percentage of calories from fats may have been 20–25 percent of calories from fat for the Samburu and 35–40 percent for the Rendille. Such diets, consumed at a bare subsistence level, would be consistent with the serum cholesterol values achieved.” Keys, however, had no reason to assume that either the Samburu or the Rendille were living at a bare subsistence level. To explain away Mann’s research on the Masai, Keys then evoked more recent research suggesting that the Masai, living in nomadic isolation for thousands of years, must have somehow evolved a unique “feedback mechanism to suppress endogenous cholesterol synthesis.” This mechanism, Keys suggested, would bestow immunity on the Masai to the cholesterol-raising effects of fat.
To believe Keys’s explanation, we would have to ignore Mann’s further research reporting that the Masai indeed had extensive atherosclerosis, despite their low cholesterol, without suffering heart attacks or any other symptoms of coronary heart disease. And we’d have to ignore still more research reporting that when the Masai moved into nearby Nairobi and began eating traditional Western diets, their cholesterol rose considerably. By 1975, Keys had relegated the Masai, and even the Samburu and the Rendille, to the sidelines of the controversy: “The peculiarities of those primitive nomads have no relevance to diet-cholesterol-CHD [coronary heart disease] relationships in other populations,” he wrote.
Once having adopted firm convictions about the dangers of dietary fat based on his own limited research among small populations around the world, Keys repeatedly preached against the temptation to adopt any firm contrary convictions based on the many other studies of small populations that seemed to repudiate his hypothesis. “The data scarcely warrant any firm conclusion,” he would write about such contradictory evidence. When a 1964 article in JAMA, The Journal of the American Medical Association, for instance, reported that the mostly Italian population of Roseto, Pennsylvania, ate copious animal fat—eating prosciutto with an inch-thick rim of fat, and cooking with lard instead of olive oil—and yet had a “strikingly low” number of deaths from heart disease, Keys said it warranted “few conclusions and certainly cannot be accepted as evidence that calories and fats in the diet are not important.”
The Framingham Heart Study was an ideal example of this kind of selective thinking at work. The study was launched in 1950 under Thomas Dawber’s leadership to observe in a single community aspects of diet and lifestyle that might predispose its members to heart disease—risk factors of heart disease, as they would come to be called. The factory town of Framingham, Massachusetts, was chosen because it was what Dawber called a “reasonably typical” New England town. By 1952, fifty-one hundred Framingham residents had been recruited and subjected to comprehensive physicals, including, of course, cholesterol measurements. They were then re-examined every two years to see who got heart disease and who didn’t. High blood pressure, abnormal electrocardiograms, obesity, cigarette smoking, and genes (having close family with heart disease) were identified as factors that increased the risk of heart disease. In October 1961, Dawber announced that cholesterol was another one. The risk of heart disease for those Framingham men whose cholesterol had initially been over 260 mg/dl was five times greater than it was for men whose cholesterol had been under 200. This is considered one of the seminal discoveries in heart-disease research. It was touted as compelling evidence that Keys’s hypothesis was correct.
But there were caveats. As the men aged, those who succumbed to heart disease were ever more likely to have low cholesterol (as had Eisenhower) rather than high cholesterol. The cholesterol/heart-disease association was tenuous for women under fifty, and nonexistent for women older. Cholesterol has “no predictive value,” the Framingham investigators noted in 1971. This means women over fifty would have no reason to avoid fatty foods, because lowering their cholesterol by doing so would not lower their risk of heart disease. None of this was deemed relevant to the question of whether Keys’s hypothesis was true.
The dietary research from Framingham also failed to support Keys’s hypothesis. This never became common knowledge, because it was never published in a medical journal. George Mann, who left the Framingham Study in the early 1960s, recalled that the NIH administrators who funded the work refused to allow publication. Only in the late 1960s did the NIH biostatistician Tavia Gordon come across the data and decide they were worth writing up. His analysis was documented in the twenty-fourth volume of a twenty-eight-volume report on Framingham released in 1968. Between 1957 and 1960, the Framingham investigators had interviewed and assessed the diet of a thousand local subjects. They focused on men with exceedingly high cholesterol (over 300) and exceedingly low cholesterol (under 170), because these men “promised to be unusually potent in the evaluation of dietary hypotheses.” But when Gordon compared the diet records of the men who had very high cholesterol with those of the men who had very low cholesterol, they differed not at all in the amount or type of fat consumed. This injected a “cautionary note” into the proceedings, as the report noted. “There is a considerable range of serum cholesterol levels within the Framingham Study Group. Something explains this inter-individual variation, but it is not diet (as measured here).”