Measurement resources for dissemination and implementation research in health

Measurement issues often threaten the evolution of new fields [1]. Recent reviews suggest that fewer than 50 % of existing dissemination and implementation (D&I) research measures are psychometrically validated (i.e., in many cases, no data exists on whether the measure assesses the construct it is intended to address; [2–4]). In addition to psychometric quality, information about pragmatic quality, including clinical or operational utility, is gaining ground as an important measurement dimension [5], particularly for advancing the practice of D&I [6]. Additional challenges include the dearth of measures available for certain D&I constructs (e.g., context, adaptation) [7]. However, perhaps the most critical challenge is the combination of these issues: the apparent lack of pragmatic and high-quality measures for key constructs and the disparate use of measures across studies, which inhibits the integration of results from observational and interventional studies conducted across multiple sectors that examine health behaviors and outcomes.

D&I research is especially vulnerable to communication barriers that may exacerbate these measurement issues given the rapid spread of the field across numerous disciplines within and outside of healthcare (e.g., healthcare, mental health, public health, education). A likely issue is that D&I scientists working in clinical medicine or healthcare may not be aware of measures available from those working in public health or mental health, for example. Without shared measures resources, the field is vulnerable to redundancies in measure development and missed opportunities to use common measures across studies, with the ultimate consequence being an artificially fractured knowledge base and inefficient efforts to advance the field.

From 2007 to 2012, the USA (US) National Institutes of Health (NIH) held five large conferences on D&I research, choosing themes that would call attention to aspects of the field for which advances were particularly needed. While the large meetings, including one specifically focusing on research methods and measurement, were able to spotlight the “state-of-the-science” across different domains to guide development, there was limited opportunity to directly fill the gaps that speakers had identified. In 2013, the NIH developed a series of three meetings with the purpose to convene working groups of leaders in D&I research to identify gaps, articulate key next steps, and locate potential tools for the field related to (1) training, (2) study design, and (3) measurement and standardized reporting. On October 23–24, 2013, the working group on measurement and standardized reporting, including 23 representatives from large-scale efforts to synthesize and evaluate D&I measures—including from the Society for Implementation Research Collaboration (SIRC), the US National Cancer Institute (NCI) Grid-Enabled Measures D&I campaign, and the affiliated NIH Clinical and Translational Science Award Community-Engaged Research and Comparative Effectiveness Research measurement effort—took on the challenge of assessing the state of D&I measures and of identifying mechanisms to improve standardized reporting across studies. In bringing together this group of scientists, it became quickly apparent that cross-talk between different research areas was lacking, not only with respect to the use of similar measures but also in terms of knowledge of measure resources that could promote ease of measure identification, selection, and harmonization. Thus, a subgroup of scientists from this meeting aimed to locate existing measure resources to share with D&I-engaged scientists and to reveal action steps that emerged from the meeting with a focus on measure development grants as a potential avenue for filling the obvious gaps in the field.

Three resources came from mental health, one from business, and the remaining from health disciplines. Eleven reported identifying measures from databases, of which five combined this search strategy with a second approach (e.g., expert review, snowball sampling); the remaining drew from experts, snowball sampling, crowd sourcing, or did not specify their search approach (n = 6). Ten provided information on quantitative measures only, seven on both quantitative plus qualitative. All resources were developed for use by researchers with implementation practitioners as the next most commented target audience (n = 9).

Fourteen resources were organized according to an established D&I theory or framework. Most notably, the Consolidated Framework for Implementation Research (CFIR) [9] was represented in three of the resources. The number of constructs ranged from 1 to 359. A similar broad range was observed in the number of measures: 1 to >450. Although only nine resources provided the measure, 11 resources provided information regarding measure citations to promote ease of access. The amount and type of information provided regarding both the measures themselves (i.e., metadata) and the development and validation of the measures was also quite variable. For example, six out of the 17 resources included reliability and validity information and only three resources provided information about pragmatic measure qualities. Ten provided measures targeting consumers and 14 targeting providers.

In this short report, we described 17 resources that synthesized and evaluated existing measures for D&I research. These resources included four web-based repositories and 13 static reviews or tools, each providing varying levels of measure metadata. The summary of the resources provided herein can be used as a starting point for researchers and other stakeholders (e.g., implementation practitioners, administrators) intending to identify measures for D&I studies. In the case of interactive resources in which crowd sourcing of data and experiences is encouraged, end-users can share experiences and additional measure metadata. Taken together, these can facilitate a culture of measure harmonization and data comparison across studies [11]. For example, the SIRC Instrument Review Project provides expert-informed rating of measures organized along the CFIR [9] and the Implementation Outcomes Framework [8] making the identification of scientifically sound D&I measurement convenient [2]. Another interactive instrument, the D&I Workspace in the Grid-enabled Measures Database uses a crowd sourcing approach to populate, update, rate, and comment on measures that are organized around critical D&I constructs [12]. Active participation from researchers and practitioners in the development and refinement of these and other interactive resources for D&I measures is critical for achieving their ultimate goal of being living and relevant resources for the D&I community.

Advancing and strengthening measurement approaches for D&I research are critical to building a cumulative scientific knowledge base and offering tools for informing the real-world practice of D&I. A number of existing measurement resources can provide a starting point to researchers and stakeholders for the identification of appropriate measures and harmonization of measurement use across studies. However, additional work needs to take place to advance and strengthen the field. Critical areas for development include the following: additional high quality, pragmatic measures for key D&I-related constructs for which measures do not exist; a core set of brief measures that can be used efficiently across pragmatic clinical trials and practice-based observational studies; and a rapid-cycle measure development process. Researchers and stakeholders are encouraged to take advantage of a number of funding mechanisms supporting this type of work. Implementation of a set of core measures across multiple studies would facilitate future synthesis enabling the examination of the impact of various D&I constructs on clinical and population health outcomes.