Table 8 Agenda for implementation outcomes research: 2022–2032

1

2011 agenda argued for consistency of terminology

This issue persists. * Most articles used terms from the 2011 taxonomy but some used new and unclear terms

Use consistent terminology, specify the referent, and report the level of measurement for each implementation outcome. *

Investigators should capture implementation outcome changes over time and speed of their attainment. **

    • Does attainment of a given outcome sustain over time?

    • How long does it take to achieve such outcomes as fidelity, acceptability?

2

2011 agenda stated that “researchers should report the referent for all implementation outcomes measured” (pg. 71)

This issue persists. * Coding was challenged by a lack of clarity

3

2011 agenda drew our attention to the fact that “Currently, very few studies reporting implementation outcomes specify the level of measurement, nor do they address issues of aggregation within or across levels” (pg. 71)

This issue persists. * Coding was challenged by limited and confusing reporting of the level of measurement for implementation outcomes

Few studies reported clear time frames for observation windows

4

2011 agenda noted that “The actual words used by stakeholders may or may not reflect the terms used in academic literature and reflected in our proposed taxonomy (acceptability, appropriateness, feasibility, adoption, fidelity, penetration, sustainability, and costs). But such research can identify the terms and distinctions that are meaningful to implementation stakeholders” (pgs. 71–72)

Eight percent (n = 32) of manuscripts coded implementation outcomes that were not in the original taxonomy. Some of these terms (e.g., provider skill) may reflect determinants of implementation; others—notably usefulness, usability, and utility—were identified in the 2011 paper as “other terms in the literature.”

Advance the identification of new implementation outcomes

    • Identify and define—conceptually and operationally—new implementation outcomes, demonstrating and explaining their distinction from outcomes in existing typologies

5

2011 paper suggested implementation outcomes’ salience by stage but did not offer specific recommendations

Most manuscripts indicated the stage or phase of implementation investigated, which we coded using the EPIS framework. Few reported observation windows

Capture the time to achieve implementation outcomes

    • Define and capture theory-relevant phases

    • Identify meaningful observation periods; establish and report metrics

6

2011 agenda asserted that “Measurement development is needed to enhance the portability and usefulness of implementation outcomes in real-world settings of care” (pg. 72)

More than half (52%) of the included manuscripts examined acceptability, followed by fidelity (38.8%), feasibility (36.9%), adoption (24.0%), and appropriateness (20.1%). Penetration (15.4%), sustainability (15.1%), and cost (7.5%) were examined less frequently (Table 2). This mirrors the difference in availability of measures for different outcome types (explained in our Discussion)

Advance precision in measurement

    • Leverage measurement resources (e.g., SIRC measurement project)

    • Capture implementation outcomes using a range of sources: e.g., agency records, participant observation, and other ethnographic methods

    • Report data sources, frequency of measurement, and metrics. *

    • Perform and report psychometric properties of measures used

7

2011 paper presented the original taxonomy

Most studies in our review used the original terms or synonyms presented in the 2011 paper. Eight percent (n = 32) of manuscripts coded identified implementation outcomes that were not in the original taxonomy

Conduct deeper analyses of conceptual and operational definitions for each of the eight outcomes, capturing variation and shedding light on a range of measurement approaches

8

Not explicitly mentioned in 2011 agenda

Most papers in scoping review employed observational designs and reported only descriptive statistics. Very few employed experimental designs. Several used mixed methods

Leverage design innovations in studying implementation outcomes

    • Employ optimization approaches and intervention mapping to unpack mechanisms of “getting to implementation outcomes.”

    • Utilize newly developed research approaches (including rapid ethnography and human/user-centered designs) that can amplify and clarify participant perspectives on salience of implementation outcomes

    • Employing adaptive designs; test changes in implementation outcomes as a result of adapting evidence-based interventions

    • Leverage “SMART” designs to enable real-time adaptation or change in strategies employed in response to real-time data about implementation outcome attainment

#2 Theory-building research

Item #

What 2011 agenda said

Progress evident in this scoping review

Priorities/recommendations for the 2022 agenda

9

2011 agenda called for the field to: build a theory that includes the identification and testing of change mechanisms; model and test interrelationships among implementation outcomes; examine implementation outcomes in relation to service and clinical outcomes

Most studies in the scoping review used observational approaches. Very few studies employed experimental designs capable of modeling relationships between various implementation outcomes, between implementation strategies and implementation outcomes, or between implementation outcomes and subsequent outcomes (e.g., service, clinical)

Conduct theory-based and theory-building studies, especially in relation to implementation phases

    • Leverage existing toolboxes of frameworks, methods, and measures to facilitate theory-building research

    • Test and accrue evidence about causal relationships with implementation outcomes (including measurable hypotheses about how and why a strategy works to change an implementation outcome)

    • Examine the salience of outcomes by phase and time to attainment

    • Examine interrelationships among implementation outcomes, capturing temporal aspects and dynamic iterative and/or interactive effects

    • Test the role of implementation outcomes in attaining service outcomes, especially equity (see Item #12)

10

2011 agenda discussed the importance of modeling interrelationships among implementation outcomes

Fifteen percent of included manuscripts examined relationships between implementation outcomes and other outcomes

Examine the impact of implementation outcomes on service system outcomes

    • Empirically test and model relationships, measuring over time and phases

11

2011 agenda discussed the importance of modeling the attainment of implementation outcomes

Implementation outcomes were treated as dependent variables in only one-quarter (n = 97) of included manuscripts

Examine the effectiveness of implementation strategies for attaining implementation outcomes, recognizing that many routes may lead to their attainment (equifinality)

    • Conduct more research to rigorously test theoretically derived predictions of the role of implementation outcomes in mechanisms of change

12

Equity is mentioned only in Fig. 1 in 2011 paper

No articles in the scoping review examined the role of implementation outcomes in relation to equity. The field has scant empirical evidence about how to improve equity in implementation

Map inequity and equity by including measures of implementation outcomes

    • For various subgroups, is the evidence acceptable? Are evidence-based interventions and implementation strategies feasible? Is an evidence-based intervention sustainable in specific contexts (why and why not)?

    • Examine implementation outcomes as signals or harbingers of inequity

    • Posit and test various strategies to improve acceptability, feasibility, sustainability

    • Posit and test interrelationships between implementation outcomes (e.g., feasibility → adoption)