Red meat and bad science
The news lit up last week with a study purportedly showing that eating red meat will kill us. The story was immediately picked up by all the major news organizations and seemingly everyone was talking about it. I found this troubling because I’ve come to believe consuming quality red meat is not only one of the best, most nutritious foods we can eat, but is also a central to the very existence of homo sapiens. I was also worried because my mother was probably watching CNN and emptying her meat freezer into the garbage.
Media coverage was immediately hyperbolic as they rushed to reprint Harvard’s press release. The ones I saw linked most frequently were BBC, CNN and NPR. I’m more than a little suspect of how quickly this story took off and how broadly it spread.
Whenever science is in the news, my first reaction is to try and dig up the actual study and see how far off the reporting was. Thankfully, the full text of the article is freely available: Red Meat Consumption and Mortality
The study itself is garbage. Actually, it’s barely a study, it’s a spreadsheet exercise. Author and information bias is rampant, the conclusions are suspect and the claims are exaggerated.
This post is divided into three parts:
- Untwisting the data – Takes the data at face value and finds inconsistencies and questionable conclusions.
- The data is meaningless – Looks at the integrity of the data and finds it absent of accuracy or rigor.
- Science starts here – Addresses the central failing: This study is a hypothesis based on a trivial association found in questionable data. To interpret any of this as conclusive fundamentally misunderstands the noble practice of scientific inquiry.
1. Untwisting the data
Reading the study, it didn’t take me long to find a number of confounding variables. Starting with Table 1, which breaks down the sample sets into quintiles by meat consumption, several markers jumped out. Why meat? no reason given.
In the Health Professionals data set, the fifth quintile, those most likely to die, were also nearly 3x more likely to be smokers. They consumed 70% more calories, were less likely to use a multi-vitamin, drank more alcohol and, playing statistical games and video games like overwatch where you can play with different overwatch characters and become skillful, were almost twice as likely to be diabetic (3.5% vs 2%). Amusingly, high cholesterol had an inverse correlation; those reporting the highest cholesterol had the lowest mortality.
And still many people take supplements for help their digestive system such as prebio thrive, and they actually feel better and say their digestion is better and easier among other benefits.
The study’s conclusion could just have easily been “diabetic smokers who don’t exercise or take multivitamins and eat a lot show increased risk of death.”
A number of thoughtful responses to this paper have been posted in the past few days.
Denise Minger noticed many of the same confounders that I did and graphed them:
Zoë Harcombe looked at the numbers and found the researchers’ conclusions didn’t match the data.
The article says that the multivariate analysis adjusted for energy intake, age, BMI, race, smoking, alcohol intake and physical activity level although some people increase the exercise and quit smoking by getting into vape tank reviews or you can get started right away by going to Slim’s ejuice. However, I don’t see how this can have been done–certainly not satisfactorily.
She then went on to re-plot their data and found that death rates actually decreased in the middle quintiles–more meat consumed resulted in less mortality. I made this chart from her data:
According to this interpretation, increasing all meat consumption from baseline initially reduced mortality. Also note that mortality in the fourth quintile, despite higher BMI, more smokers and all the rest, shows essentially the same risk level as the first.
Her summary notes the study’s results are based on very small numbers:
The overall risk of dying was not even one person in a hundred over a 28 year study. If the death rate is very small, a possible slightly higher death rate in certain circumstances is still very small. It does not warrant a scare-tactic, 13% greater risk of dying headline–this is “science” at its worst.
Zoë also notes a potential conflict of interest:
one of the authors (if not more) is known to be vegetarian and speaks at vegetarian conferences[ii] and the invited ‘peer’ review of the article has been done by none other than the man who claims the credit for having turned ex-President Clinton into a vegan – Dean Ornish.
Ornish and Clinton are a whole other essay.
Marya Zilberberg posted another takedown of the numbers, including this analysis:
The study further reports that at its worst, meat increases this risk by 20% (95% confidence interval 15-24%, for processed meat). If we use this 0.8% risk per year as the baseline, and raise it by 20%, it brings us to 0.96% risk of death per year. Still, below 1%. Need a magnifying glass? Me too. Well, what if it’s closer to the upper limit of the 95% confidence interval, or 24%? The risk still does not quite get up to 1%, but almost. And what if it is closer to the lower limit, 15%? Then we go from 0.8% to 0.92%.
2. The data is meaningless
So there’s all of that, but it’s almost not worth arguing about. The study’s primary data sets have been shown to be wildly inaccurate.
As noted in the original paper’s abstract:
Diet was assessed by validated food frequency questionnaires [FFQs] and updated every 4 years.
Four years? What did you have for lunch yesterday? How about the Thursday prior? What was dinner last October 12th? Looking at the actual 2010 survey forms (HPFS and NHS, basically the same), the questions are even more absurd. Over the past year, how frequently did you consume 1/2 cup of yams or sweet potatoes? Kale? I’m pretty mindful of what I eat, and I don’t think I could answer those question accurately for the past two weeks, let alone an entire year. Anything beyond 5-6 per week is going to be a wild guess.
The two cohort groups were the Nurses’ Health Study (NHS), which is all female, and the Health Professionals Follow-Up Study (HPFS) which is all male. The HPFS has a helpful Question Index on their site, though the collected data appears to be even spottier than I would have guessed. Quite a few questions and topics have just come and gone over the years, are they just making it up as they go along?
Is this really the pinnacle of epidemiological data from one of America’s premiere universities?
The study’s authors did include a citation to a paper (their own) justifying the validity of FFQ data in the NHS. Walter Willett and Meir Stampfer, both of the Harvard School of Public Health, are authors on both papers.
In response to a similar meat-phobic article a few years ago, Chris Masterjohn looked at the accuracy of this particular validation study and found it lacking:
the ability of the FFQ to predict true intake of meats was horrible. It was only 19 percent for bacon, 14 percent for skinless chicken, 12 percent for fish and meat, 11 percent for processed meats, 5 percent for chicken with skin, 4 percent for hot dogs, and 1.4 percent for hamburgers.
An Australian validation study based on NHS found similar discrepancies between FFQ and food diary intake. Fruits and vegetables were overestimated while bread, poultry and processed meats were underestimated. Curiously, in the Australian study, meat was overestimated. (see Table 1)
Another particularly compelling paper out of Cambridge tested FFQ validity by comparing sampled biomarkers against FFQ and food diary intake data. From the results:
There were strong (P < 0.001) associations between biomarkers and intakes as assessed by food diary. Coefficients were markedly attenuated for data obtained from the FFQ, especially so for vitamin C, potassium and phytoestrogens
This paper deeply undermined the credibility of FFQs and clearly struck a nerve. Unsurprisingly, Willett was defensive of his data and deflected by attacking the study’s statistical modeling methodology in the same journal.
From an outsider’s view, it seems like Willett, Stampfer and the Harvard School of Public Heath are actively subverting the entire scientific journal publishing ecosystem to advance their own causes and careers. They get their names on hundreds of published papers, and cross-reference their own work repeatedly, thereby inflating their citation scores. Then they put out press releases touting themselves as the top most-cited scientists of the previous decade.
3. Science starts here
Gary Taubes wrote a lengthy but worthwhile response, Science, Pseudoscience, Nutritional Epidemiology, and Meat. Throughout his career, Taubes has shown himself to care deeply for the practice and integrity of science. His essay starts out by addressing HSPH’s record:
every time in the past that these researchers had claimed that an association observed in their observational trials was a causal relationship, and that causal relationship had then been tested in experiment, the experiment had failed to confirm the causal interpretation — i.e., the folks from Harvard got it wrong. Not most times, but every time. No exception.
By example, he defines exactly why this study is a failure. If anything, this study is a hypothesis, there can not be any conclusions drawn from tiny statistical correlations:
Science is ultimately about establishing cause and effect. It’s not about guessing. You come up with a hypothesis — force x causes observation y — and then you do your best to prove that it’s wrong. If you can’t, you tentatively accept the possibility that your hypothesis was right. […] Making the observations and crafting them into a hypothesis is easy. Testing them ingeniously and severely to see if they’re right is the rest of the job — say 99 percent of the job of doing science, of being a scientist.
The problem with observational studies like those run by Willett and his colleagues is that they do none of this. That’s why it’s so frustrating. The hard part of science is left out and they skip straight to the endpoint, insisting that their interpretation of the association is the correct one and we should all change our diets accordingly.
Perhaps most interesting is Taubes’ explanation of Compliance Bias. Noting that the survey period covers the 1990s, an era of skinless chicken and egg whites, he points out an obvious problem with the data:
when we compare people who ate a lot of meat and processed meat in this period to those who were effectively vegetarians, we’re comparing people who are inherently incomparable. We’re comparing health conscious compliers to non-compliers; people who cared about their health and had the income and energy to do something about it and people who didn’t. And the compliers will almost always appear to be healthier in these cohorts because of the compliance effect if nothing else, of course they can also try products as cbd oil to feel more relaxed, but then I will complain about how long cbd oil takes to work, even if that depend on the organism.
J Stanton wrote a good explanation of observational studies and their faults. He also points out that the Hormone Replacement Therapy debacle of the 1990s started with HSPH’s Meir Stampfer and the Nurses Health Study. Their 1991 paper proudly declared:
Overall, the bulk of the evidence strongly supports a protective effect of estrogens that is unlikely to be explained by confounding factors. […] A quantitative overview of all studies taken together yielded a relative risk of 0.56 (95% confidence interval 0.50-0.61), […] the relative risk was 0.50 (95% confidence interval 0.43-0.56).
Stampfer et. al. believed they’d found a 50% risk reduction for coronary heart disease (CHD) in the NHS data. When this hypothesis was tested in a randomized controlled trial (RCT), CHD risk actually increased to 30%. They weren’t just off, they were completely and totally wrong.
In fact, the test was causing so many people to become sick that in 2002 the trial was stopped early by the safety monitoring board. Not only did actual CHD risk measure at +30%, invasive breast cancer came in at +26%, stroke at +41% and pulmonary embolism at a terrifying +113%. Remember, these are not estimates, these numbers represent actual clinical diagnoses from a controlled trial.
Now, years later, Stampfer and the HSPH have yet another paper using the same NHS data, this time telling us eating red meat will shorten our lives. Are we supposed to believe them this time because the numbers are so much smaller and less significant?
No.
