The White House Blog has a post up on "How Low-cost Randomized Controlled Trials Can Drive Effective Social Spending":
The Office of Science and Technology Policy and the Coalition for Evidence-Based Policy convened leaders from the White House, Federal agencies, Congress, philanthropic foundations, and academia this week to explore an important development in the effort to build credible evidence about “what works” in social spending: low-cost randomized controlled trials (RCTs). The goal of the conference was to help advance a broader Administration effort to promote evidence-based policy, described in the evaluation chapter of the 2014 Economic Report of the President, and the Performance and Management section of the President’s budget.Large and rigorous RCTs are widely regarded as the most valid method of evaluating program effectiveness, but they are often perceived as too costly and burdensome for practical use in most contexts. The conference showcased a new paradigm: by measuring key outcomes using large administrative data sets already collected for other purposes – whether it be student test scores, hospitalization records, or employment and earnings data – sizeable RCTs can be conducted at low cost and low burden.
The conference showcased a number of RCTs that were conducted for between $50,000 and $350,000 (a fraction of the usual multimillion dollar cost of such studies), yet produced valid evidence that informed important policy decisions.
First of all, I'm thrilled at the idea, in general, of using data, especially those derived from randomized controlled trials, to make policy decisions. Using evidence to drive policy has always been one of the goals of this blog, as well as others I write for.
But this post focused on how it's possible to do randomized controlled trials at an amazingly low cost. It describes, for instance, a study of Recovery Coach services to substance-abusing parents who had temporarily lost custody of their children. The cost of the nine-year trial was about $100,000, which is stunning.
But these types of opportunities are the exception, not the rule. I think it highlights a misunderstanding of where grant money goes much of the time.
If you looked at the budgets of any R01 funded randomized controlled trial I've had funded, the number one cost, by far, is the salaries of those who are conducting the study. Even if it's just 10-20% of someone's time, and it's just a few people, the salaries add up. When you add in benefits and overhead, you can get into the hundreds of thousands of dollars pretty quickly. Also, research comes with indirect costs. The rent for people's offices, the electricity, their phone lines, support staff, taxes, etc. There are a lot of costs.
You often need to pay for a number of people because you're setting up some new intervention! That takes time, it takes effort, and it takes money to pay for it.
The opportunities like those described in the White House blog are relatively rare. All of the interventions discussed were already being run by personnel who were likely employed through other public lines of support. That's great, but that doesn't happen often. If the intervention is already paid for, if those running it are already salaried, then most of the costs of a grant are wiped clean. If the data are already collected as part of administrative data, that's awesome, but that wasn't free. It was paid for by someone else. Yes, that makes the "grant" cost less, but not the research itself.
It appears that all that had to be paid for in the studies presented was the cost of the analysis. That may easily be $100,000. But that's only the last part of an RCT, usually.
It's totally worth it to try and do research on the cheap when you can. It's incredibly efficient. But we shouldn't be under the illusion that research is cheap. Lots of great things can't be done this way, and it's important to do those things, too.