At AcademyHealth, we’re committed to working with our members and the field to support the production and use of rigorous, timely evidence. From advancing the latest methods and data to brokering connections among the research, policy, and practice communities, we undertake a broad range of activities to help ensure that health services and policy research is high quality, responsive to the needs of policymakers, health system leaders, patients, and other stakeholders, and available in forms that decision makers can understand and use.
Yet as many research-focused organizations know all too well, establishing whether, how, and why a particular study has directly impacted health policy or health care practice can be a challenging endeavor. It’s an issue facing scientists in many fields and in many different countries, especially in settings in which budget constraints put particular pressure on researchers and their institutions to demonstrate the real-world return on research investments. In some cases, research funders have begun to incorporate measures of societal impact into funding decisions, a notable example being the United Kingdom’s Research Excellence Framework, undertaken for the first time in 2014 with the next iteration planned for 2021.
AcademyHealth had an opportunity to examine the impact issue further in a pilot project for the Robert Wood Johnson Foundation (RWJF), with whom we have worked for many years to support research grantmaking and dissemination activities. The purpose of the pilot was twofold: 1) to help RWJF better understand the policy impact of a subset of its investments on the topic of health insurance coverage and health reform, and 2) to help inform how the foundation may more systematically track and measure the impact of the research it funds. Conducted over 18 months, the pilot was not a large-scale program evaluation, but rather an effort to formulate practical recommendations for how foundation staff might collect, organize, and interpret key measures of policy impact on an ongoing basis.
A recent article in The Foundation Review chronicles AcademyHealth’s experience with the pilot, including lessons learned that may be useful to other funders that support research and policy analysis. Among our findings, evidence generated by these grants – including investigator-initiated research studies and policy analyses, survey research, and data set creation, analysis, and dissemination – was used by a diverse set of stakeholders interested in understanding the effects of health reform at the state and federal levels, including policymakers, advocacy organizations, the media, and others. More than a dozen policymakers mentioned or used research findings from this portfolio and findings from one study were even cited in a Supreme Court decision.
Perhaps just as important as these impact examples are insights we gained into potential strategies for capturing indicators of policy impact moving forward. AcademyHealth’s approach to the pilot combined the use of free and paid online tracking tools and services with semi-structured interviews with grantees and research end-users, resulting in a rich set of data relevant to grant products, related meetings and briefings, grantees’ social media presence, and much more. In The Foundation Review article, we reflect on the pros and cons of this approach, including feasibility considerations for both research funders and their grantees. One takeaway is that while grantees are a crucial source of information about their project and its impact, they would require resources and support to perform the level of tracking and reporting reflected in the pilot. Identifying a designated grant monitor, either within a funding organization or a third party, is one way to ensure that impact tracking is consistent across a portfolio of projects and not overly burdensome for grantees.
Capturing the impact of research on policy is a complex and evolving undertaking, and AcademyHealth’s pilot project for RWJF is simply one of many efforts to better understand opportunities and challenges in this area. However, through this project and in future work, we hope to contribute to important conversations on the topic of research impact assessment and serve as a resource for researchers, funders, and others interested in doing this work more effectively.