A new study from researchers at the University of South Carolina took a closer look at the numbers from the National Health and Nutrition Examination Survey, or NHANES. NHANES surveys Americans to find out what they’ve been eating, if they’ve been sick, that kind of thing. It’s used mostly to learn about diseases and health risk factors throughout the country–and this new study says it’s wrong. And has been wrong since it began. In 1971.
The study was led by epidemiologist Edward Archer, a researcher with a penchant for controversial surveys–he was previously best known for this examination of calories burned by housewives. The latest survey could have an even broader impact: NHANES is a 40-year-old survey, and Archer and his team says it is “physiologically implausible” for the numbers to be correct. The paper concludes that “the ability to estimate population trends in caloric intake and generate empirically supported public policy relevant to diet-health relationships from U.S. nutritional surveillance is extremely limited.”
NHANES is a combination of a physical checkup with a survey of what each surveyed American ate in the 24 hours preceding the survey. But that list of food eaten is entirely self-administered; there’s nobody following people around to make sure they’re telling the truth. And it seems like people weren’t.
This new study says that the majority of the data can’t possibly be accurate. Basically, people claimed they ate far less food than their physical checkup indicates they’d need. If an obese person with a very high body mass percentage–body mass and weight being empirical data which the subject can’t lie about–reports that he or she consumes 1,500 calories a day, the study gives that person a high “disparity value,” a new point of data that the NHANES had not calculated. An obese person may be able to survive on 1,500 calories a day, but that person won’t be obese for long, which makes that calorie count unlikely to be true. En masse, if a huge number of people in a certain category (like, say, the obese) have a high disparity number, it’s likely that something’s wrong here.
The worst offenders were the obese, who underreported their calorie intake by 25 and 41 percent in men and women, respectively. A possible culprit is “social desirability bias,” in which a survey subject lies in order to provide answers that he or she feels are more acceptable.
The introduction to the paper refers to the methodology of the survey as “indirect” and “pseudo-quantitative.”
This is a big deal. “The nation’s major surveillance tool for studying the relationships between nutrition and health is not valid. It is time to stop spending tens of millions of health research dollars collecting invalid data and find more accurate measures,” says Archer. The implications are troubling; policymakers could be much less likely to allocate adequate funds to treat obesity, a serious health problem, if it’s underreported. Programs like implementing calorie counts in fast-food restaurants could have been put into place sooner or more comprehensively. Public education may not have addressed the problem of obesity with the weight it deserves. Hopefully this study will provoke a closer examination of our health data and the methods used to obtain it.
You can read the paper over at PLoS One.
