I don’t spend much time in D.C. so I rarely know what, if any, research papers are making waves, however small. But I got a tip that Louise Sheiner’s "Why the Geographic Variation in Health Care Spending Can’t Tell Us Much about the Effciency or Quality of our Health Care System" (PDF) was getting some attention. From the title alone, I can see why.
It’s an attempted takedown of much of the Dartmouth work on regional spending variations. There’s a lot in it, a great deal of which I don’t agree with, and more than I could cover in one blog post. Fortunately, the key results hinge on one methodological choice, offering an opportunity for a parsimonious response. That methodological choice will be my focus.
Backing up a bit, the key question is this: to what extent is geographic variation in health care spending a signal of variation in health versus variation in health systems and practice patterns? Or, more simply, is it the patients or the providers that drive spending differences? Finding out is important because it suggests different policy interventions. If spending variation is largely due to regional differences in underlying health, then we need to focus on causes of poor health, including lifestyle factors, but also access to health care, public health investment, and educational and work opportunities.
If the variation is largely due to health systems and practice pattern differences, then we need to focus more on things like putting evidence into practice, payment incentives to motivate change, and fostering organizational structures that promote high quality, yet efficient, care.
For a long time, and over many publications, Dartmouth researchers, and others, have suggested organizational inefficiencies are the chief culprit, though variations in health play a role. Even controlling for underlying health, a lot of variation in health spending remained unexplained in their work. Meanwhile, large variations – correlated with those in spending – in how systems operated and how doctors practiced were observed. Though not everyone agrees, based on this body of work, many health systems experts believe that a lot of wasteful, overuse of care is driven by patterns of practice.
Enter Sheiner who says this is all wrong. In a cross-sectional, state-level analysis, she finds that almost all the variation in Medicare spending can be explained by just a few aggregate measures of health and well being.
[T]he combination of per capita income and age distributions explains only about 30 percent of the variation in acute Medicare spending across states. However, including measures of health—in particular, the obesity rate in the state and the percent sedentary, increases the explained share of spending to 48 percent. Adding in the percent uninsured raises the explained share of spending to 75 percent, and adding in the percent black raises it to 81 percent.
With so much of spending variation – 81 percent of it – explained by age distribution, per capita income, percent uninsured, rates of obesity and sedentary lifestyles, there’s not much left to be explained by organizational factors and practice pattern variation. Did Dartmouth get it all wrong?
No.
Sheiner’s findings are due to her selection of unit of analysis, the state. Much of the Dartmouth work over the last decade is at the individual level. This makes all the difference. That one can obtain disparate results due to level of aggregation alone has a name: the ecological fallacy. Responding to Sheiner, Dartmouth researchers Jonathan Skinner and Elliot Fisher wrote,
Why should individual-level risk-adjustment data be preferred to state-level data? The answer is: the ecological fallacy problem. It occurs when a researcher makes an inference about individual behavior based on group-level averages. One example is the following: In 2004, the Republican candidate, George W. Bush, won the 10 states with the lowest income, while the Democratic candidate, John Kerry won 9 of the 10 highest-income states. Thus one might conclude from the state-level data that low income voters tend to vote Republican and high-income voters vote Democrat (with a stunningly high R2). Yet as Gelman and colleagues have noted, high income voters are more than 10% more likely to vote Republican than are low-income voters – precisely the opposite pattern. Only by use of individual level data is the extent of the ecological fallacy revealed. [Hyperlink added.]
For this reason, we should question Sheiner’s conclusions. Indeed, Richard Kronick and Todd Gilmer have documented the role that level of aggregation plays in analyses of health spending variation. Interestingly, in an individual-level analysis of correlates of Medicare spending variation, Stephen Zuckerman and colleagues (PDF) found that some of the key factors that explained variation in Sheiner’s state-level analysis – income, most measures of race, and BMI – were not statistically significant. More generally, there’s just no reason to believe aggregate measures that explain a lot of variation at the state level are the same factors that drive variation at the level at which health care and any intervention thereof ultimately takes place: the individual. Consequently, Sheiner’s state-level findings in no way invalidate the individual-level analyses by Dartmouth scholars and others.
I wish Sheiner’s conclusions could be believed. It would really simplify things if the vast majority of health spending variation was due to obesity, sedentary lifestyles, lack of insurance, and a few other things. Were that true, we’d know better how to shape policy. However, based on the body of evidence, it’s far more likely the story is more complex and also includes the efficiency with which care is delivered.
Delivery system variation still matters, despite what they read in the papers in D.C.
--Austin