Look, I'm a big fan of randomized controlled trials. I even made a video talking about how critical they are. But they have their limits. Many of these were ignored in a recent article in the New York Times by Gina Kolata:
The idea seemed transformative. The Affordable Care Act would fund a new research outfit evocatively named the Innovation Center to discover how to most effectively deliver health care, with $10 billion to spend over a decade.But now that the center has gotten started, many researchers and economists are disturbed that it is not using randomized clinical trials, the rigorous method that is widely considered the gold standard in medical and social science research. Such trials have long been required to prove the efficacy of medicines, and similarly designed studies have guided efforts to reform welfare-to-work, education and criminal justice programs.
But they have rarely been used to guide health care policy — and experts say the center is now squandering a crucial opportunity to develop the evidence needed to retool the nation’s troubled health care system in a period of rapid and fundamental change.
When you want to find out if one thing causes another, there is simply no better tool than a randomized controlled trial. If you want to determine the efficacy of a treatment in a specified population, there is no better design than an randomized controlled trial. But those studies have limits.
They almost always have strict inclusion and exclusion criteria to make sure that there is the best chance of seeing a significant result. Patients who are the least likely to comply are often prohibited from taking part. They often involve incentives and environments that help the study, but bear no resemblance to the real world.
Randomized controlled trials are great at determining efficacy. In other words, they are fantastic as seeing whether a certain therapy has the potential to produce a desired effect.
What they aren't so good at is determining effectiveness. In other words, they aren't nearly as good at telling us how these therapies work in the real world.
This is because in the real world, we rarely exclude patients. We want to treat as many as possible. We can't deny therapy to those least likely to comply; they are, in fact, those who likely need the most help. And we can't create perfect environments. We often have to care for people in under-resources settings.
Even the example Kolata uses to shore up support for the RCT makes my point. She discussed the RAND Health Insurance Experiment:
In health care, a seminal, large randomized study by the RAND Corporation in 1982 found that people used health care less, but that their health was not affected, when they had to pay a small amount — as compared with nothing — for doctor visits.
It's true that this was the major finding of the RAND HIE. But what it misses is that, for the most part, the RAND HIE studies pretty healthy people who had jobs. That's not necessarily who needs health care the most. In the real world, people are often poor and have chronic illnesses. And what the RAND HIE also found is that poorer people with high blood pressure who had to pay more had significantly higher mortality rates. (I made a video about this, too)
In other words, the efficacy of cost-sharing was determined by the RAND HIE. But the effectiveness of it in the real world was not necessarily the same thing.
Moreover, even randomized controlled trials are sometimes wrong.
The Innovation Center is focused on effectiveness. They want to know how we can change the delivery of health care in order to make it work better in the real world. They are focusing much of their efforts to designs tht are not randomized controlled trials.
Some, including the NYT piece, argue that this is "one-sided". This is true only if you ignore the fact that the vast majority of NIH money goes not to effectiveness research, but to efficacy work, like RCTs.
The Innovation Center is trying to correct the balance in some small way. We should give it a chance to succeed. Randomized controlled trials are awesome, but they're not perfect, and we also need to know what works in actual practice.