A seminal moment for evidence-based policy was when the Foundations for Evidence-Based Policymaking Act of 2018 (or the Evidence Act) was enacted. That Act required that across the federal government, agencies should figure out how to evaluate their own activities to see what works (and what doesn’t). A key requirement is that federal agencies produce an evaluation plan for the next year as well as a broader learning agenda for the next several years. The Office of Management and Budget (OMB) recently released a federal-wide dashboard to explore all of these plans.
This all seems like progress towards evidence-based policy across the federal government. But as usual, the devil is in the details.
Consider the National Institutes of Health (NIH). While the many offices/institutes at NIH are in fact doing a great deal of evidence-building activities, hardly any of them were captured in HHS’s “FY 2023 Evaluation Plan” or its “FY 2023-2026 Evidence-Building Plan.”
Indeed, the HHS documents barely mention NIH at all. First, in the HHS 2023 plan, NIH is mentioned for a mere two evaluation questions:
(1) assessing research outcomes for people who receive a particular pre-doc award to see whether this funding is “effective in preparing graduate students to transition to the next step in their pathway to a research career,” and
(2) assessing how to plan and monitor the “Environmental influences on Child Health Outcomes (ECHO)-wide Cohort,” a program that is trying to collect data from over 50,000 children and families.
Second, there’s the longer-term plan. From OMB’s site, we can see that the National Science Foundation’s longer-term plan was 37 pages long. The entirety of HHS, which includes not just NIH but the Food and Drug Administration (FDA), Centers for Disease Control (CDC), Center for Medicare and Medicaid Services (CMS), and much more, merely has a 44-page plan for the next four years. As one might expect, NIH’s portion is fairly small: the bottom of page 39 to the top of page 42 (about 3 pages of text in total). Over the next four years, NIH plans to assess
1) a set of recommendations that came out of a 2014 workshop on opioid treatment,
2) the Cancer Moonshot (NIH estimates an end date of 2027 for this), and
3) the data science portfolio at the National Institute of Diabetes and Digestive and Kidney Diseases.
Again, this seems underwhelming. And unlike other agencies, there is no description of any NIH-wide strategic priorities and goals that might help decide which evaluation questions are important.
An organization that spends $47+ billion a year requires far more evaluation and research than this.
But is NIH to blame? Not at all! Indeed, there are many more evaluation activities going on at NIH than were mentioned in the HHS document.
Just to take a few examples, the National Institute of Allergy and Infectious Diseases (NIAID) has its own evaluation program that is currently evaluating four programs (more than what you would know from the overall HHS/NIH agenda). The National Institute of General Medical Sciences (NIGMS) conducts evaluations of its programs. The NIH Office of Portfolio Analysis conducts many evaluations of various kinds, including a major evaluation of racial disparities and on whether gender affects scientific mentoring relationships (its 29-page plan for 2021-2025 is far more detailed than the HHS evaluation plan). As well, NIH’s Office of Evaluation, Performance, and Reporting (OEPR), which was established in 2018, has a 26-page strategic plan for 2023-2027.
Why didn’t any of that make it into the HHS evaluation plan? I’ve heard that it is because the HHS divisions (like NIH) were only asked to provide a “sample” of the evaluation activities that are occurring.
In turn, we might wonder why HHS had to rely on just a small sample of evaluation activities, and how such a cursory document would be useful to anyone. The Evidence Act itself is arguably to blame here. Under the Evidence Act, each agency is required to develop evaluation plans, etc. But not all agencies are equal. The Department of the Interior has about an $18 billion budget. But HHS has a budget of $1.6 trillion and includes many operating divisions (or op-divs) such as NIH, FDA, CDC, and the big one: CMS.
Why would we treat HHS the same as agencies that are around one percent as big, such as Interior or Commerce? Even if it were feasible to produce an HHS-wide learning agenda that is reasonably complete, it would likely be a thousand pages or more, and would cover many disparate activities about everything from drug approvals at FDA, to hospital reimbursement and quality measurement at CMS, to biomedical research at 28 NIH Institutes and Centers (plus the Office of the Director).Few readers would find this sort of document useful either!
Instead, these major divisions should be directly responsible under the Evidence Act for fully participating in organization-wide evaluation and data collection. That is, HHS should have an overall Chief Evaluation Officer, but each major operating division should have to hire its own Chief Evaluation Officer, its own Chief Data Officer, etc., and should have to develop its own yearly evaluation plan and longer-term agenda.
Indeed, CMS has long had the Center for Medicare and Medicaid Innovation, which spends $1 billion a year on piloting new ideas and evaluating them.As Kushal Kadakia and I recently argued in a Health Affairs article, the NIH should establish a Center for Experimentation/Innovation so as to evaluate important questions about science funding. But short of that, the NIH’s Office of Evaluation, Performance, and Reporting should likely take more of a cross-NIH leading role here. In particular, it needs to be given a budget sufficient to conduct (or contract for) dozens of major evaluations that are independent from any influence by the Institutes, Centers or Offices whose work is being evaluated. That would help fulfill the vision of the Evidence Act.
The opinions expressed in this blog post are the author's own and do not necessarily reflect the view of AcademyHealth.