Table 1 Statements by cluster by bridging
Cluster 1: RQ: organizational issues | ||||||||
83 | Paying special attention to issues related to the organizational learning curve and other issues related to evolution or drift of the practices | .41 | ||||||
77 | Understanding when innovation or the natural life cycle of a program takes precedence over sustainability | .43 | ||||||
34 | Identifying organizational adopter/sustainer archetypes | .45 | ||||||
30 | Considering the organization as a learning organization rather than sustainability as an end-point | .52 | ||||||
76 | Developing guidance for how organizations should decide to end a program and instead adopt a newer/better/more effective one | .69 | ||||||
5 | Understanding the “rapid learning” or problem solving skills needed by key individuals/leaders and organizations in order to respond to changing environmental challenges | .78 | ||||||
Count: | 6 | Std. Dev.: | 0.14 | Minimum | .041 | Average: | .55 | |
Variance: | 0.02 | Maximum | .078 | Median: | 0.49 | |||
Cluster 2: research and program funding | ||||||||
70 | Funding | .09 | ||||||
67 | Finding funding sources for conducting sustainability research | .22 | ||||||
86 | Understanding and commitment on the part of funders for such work | .46 | ||||||
82 | The availability of RFAs that explicitly call for sustainability research | .48 | ||||||
6 | Determining why implementing agencies do/do not seek continued funding | .51 | ||||||
19 | Fully exploring the role of adequate funding in sustainability | .62 | ||||||
41 | Funds available to sustain a program | .70 | ||||||
36 | Making sure that the research addresses required infrastructure and a viable business model to provide longstanding revenue support for the program | .73 | ||||||
Count: | 8 | Std. Dev.: | 0.21 | Minimum | 0.09 | Average: | .47 | |
Variance: | 0.04 | Maximum | 0.73 | Median: | 0.49 | |||
Cluster 3: research practice and training | ||||||||
22 | Establishing training in sustainability research | .20 | ||||||
15 | Developing and implementing graduate curriculum relevant for sustainability researchers | .29 | ||||||
56 | Training and capacity building in a public health agency | .42 | ||||||
91 | Training and technical support to providers/deliverers of the program/intervention | .49 | ||||||
45 | Identifying a broad set of journals and professional conferences that are good places for dissemination of sustainability research | .68 | ||||||
Count: | 5 | Std. Dev.: | 0.16 | Minimum | 0.20 | Average: | .42 | |
Variance: | 0.03 | Maximum | 0.68 | Median: | 0.42 | |||
Cluster 4: research question: ROI | ||||||||
78 | Determining the return-on-investment of sustainability given a variety of likely parameters (e.g., number of individuals reached; number of infections averted) | .43 | ||||||
50 | Conducting ROI (return on investment) studies to make it clear to stakeholders and funders how much is actually gained when effective programs are sustained | .43 | ||||||
33 | Determining the return-on-investment of sustainability during different time periods (e.g., 6-months, 12-months, 24-months) | .46 | ||||||
9 | Investigating to what extent benefits (e.g. cost-savings, improved clinical outcomes) are sustained along with sustained use/behavior | .47 | ||||||
84 | Conducting longitudinal cost-benefit analysis comparing implementation vs. sustainability | .50 | ||||||
46 | Considering cost and economic issues from multiple perspectives | .66 | ||||||
23 | Taking advantage of research already conducted in other fields and industries regarding sustained implementation of technologies and practices | .75 | ||||||
74 | Creating survey of disciplines most involved in sustainability research | 1.00 | ||||||
Count: | 8 | Std. Dev.: | 0.19 | Minimum | 0.43 | Average: | .59 | |
Variance: | 0.04 | Maximum | 1.00 | Median: | 0.48 | |||
Cluster 5: research question: factors affecting sustainability | ||||||||
13 | Figuring out how to predict sustainability based on the experience or knowledge of context gained through implementation | .23 | ||||||
39 | Identifying key features of Evidenced Based Practice associated with variations in sustainability (e.g., strength of evidence; consistency with established practice; size of difference between prior and the new/evidence-based practices; staffing/other) | .25 | ||||||
53 | Understanding which variables and factors are more important for sustainability than others | .25 | ||||||
7 | Identifying key or core program sustainability components | .25 | ||||||
75 | Investigating the relationship between sustained use, routinization and resistance to change | .29 | ||||||
12 | Exploring whether the factors influencing sustainability differ from those influencing implementation | .30 | ||||||
60 | Understanding if certain types of programs are less likely to be sustained | .30 | ||||||
69 | Identifying common and independent factors that drive adoption vs. initial implementation vs. long-term use | .35 | ||||||
42 | Exploring the supporting interventions (e.g., feedback) that are needed to sustain behaviors/use, for how long, and what intensity | .35 | ||||||
88 | Understanding the reasons why strategies are/are not sustained | .39 | ||||||
Count: | 10 | Std. Dev.: | 0.05 | Minimum | 0.23 | Average: | .30 | |
Variance: | 0.00 | Maximum | 0.39 | Median: | 0.30 | |||
Cluster 6: research question: adaptation | ||||||||
64 | Understanding the tension between fidelity and adaptation as it pertains to sustained use or feasibility of continued use | .13 | ||||||
21 | Documenting adaptations and their impact on the effectiveness of evidence-based practices | .14 | ||||||
47 | Determining the core vs. peripheral or adaptable components of interventions | .14 | ||||||
55 | Figuring out how to characterize adaptations | .16 | ||||||
54 | Determining the point at which the intervention or program can no longer be considered sustained because of extensive adaptations | .18 | ||||||
66 | Considering how the evidence-based program may change over time | .19 | ||||||
38 | Understanding the implications of “partial” sustainability and adaptations | .24 | ||||||
28 | Identifying interventions that are effective and cost-effective in fostering sustainability | .33 | ||||||
4 | Framing sustainability as a partnership in which participants continue to adapt an intervention in response to changing conditions, while trying to remain true to its core principles | .34 | ||||||
Count: | 9 | Std. Dev.: | 0.08 | Minimum | 0.13 | Average: | .21 | |
Variance: | 0.01 | Maximum | 0.34 | Median: | 0.18 | |||
Cluster 7: research question: environment | ||||||||
17 | Defining the key attributes of organizations and systems that successfully sustain effective practice (e.g., ongoing leadership attention, ongoing measurement, systematic hardwiring of effective innovation, etc.) | .20 | ||||||
49 | Identifying the key contextual factors (e.g., organizational characteristics) associated with variations in sustainability | .21 | ||||||
58 | Defining and assessing the multiplicity of environmental variables that are likely to affect sustainability | .26 | ||||||
51 | Discerning situations in which sustained use may be at odds with adopters’ (e.g., organizations) best interests | .31 | ||||||
8 | Characterizing the context or environment of the intervention to be sustained | .33 | ||||||
65 | Understanding cultural barriers to adoption | .33 | ||||||
62 | Identifying which factors that advance or inhibit sustainability are amenable to management intervention | .53 | ||||||
63 | Understanding how to sustain programs/policies in low resource settings | .71 | ||||||
Count: | 8 | Std. Dev.: | 0.16 | Minimum | 0.20 | Average: | .36 | |
Variance: | 0.03 | Maximum | 0.71 | Median: | 0.32 | |||
Cluster 8: research stage: measurement | ||||||||
2 | Developing methods for studying sustainability across the complexity dimension (e.g., sustainability of a specific clinical treatment vs. sustainability of a complex state-level chronic disease program) | .00 | ||||||
3 | Using multilevel measurement | .00 | ||||||
37 | Developing measures of sustainability (overall and sub-dimensions) | .01 | ||||||
18 | Determining which analytic methods are most appropriate for sustainability research | .02 | ||||||
48 | Identifying appropriate study designs for measuring sustainability | .04 | ||||||
85 | Considering the role of self-reported data in assessing sustainability outcomes | .09 | ||||||
68 | Identifying valid data sources for assessing sustainability | .10 | ||||||
81 | Deciding how long to follow-up on a newly implemented program to determine whether it has been sustained | .10 | ||||||
27 | Developing and validating fidelity measures for assessing adherence of a program to an evidence-based model | .11 | ||||||
89 | Constructing reliable and validated tools to measure core sustainability constructs | .13 | ||||||
61 | Having an operational definition of sustainability, with measurable criteria | .14 | ||||||
44 | Determining “what” should measured as an outcome (patient level outcomes? fidelity? program activities? capacity?) | .26 | ||||||
14 | The ability to move rapidly and use innovative research methods to learn from emerging opportunities and examples | .29 | ||||||
Count: | 13 | Std. Dev.: | 0.09 | Minimum | 0.00 | Average: | .10 | |
Variance: | 0.01 | Maximum | 0.29 | Median: | 0.10 | |||
Cluster 9: research stage: design and analysis | ||||||||
72 | Discussing how systems science methods such as modeling and network analysis can be used to study important sustainability questions | .03 | ||||||
59 | Increasing use of non-experimental study designs in sustainability research | .08 | ||||||
43 | Using a participatory approach to research that values the perspective of developers of original idea as well as the target group | .14 | ||||||
35 | Clearly defining research hypotheses and goals linked to real-world application outcomes | .19 | ||||||
87 | Identifying the indicators of sustained use so that we will know it when we see it | .25 | ||||||
29 | Developing ways researchers can better integrate their documentation needs into the agency, so that it creates a smaller burden on those who deliver care (e.g., integrated data collection with electronic records) | .39 | ||||||
90 | Conducting observational research of implemented programs to identify barriers/facilitators to sustainability | .42 | ||||||
Count: | 7 | Std. Dev.: | 0.14 | Minimum | 0.03 | Average: | .21 | |
Variance: | 0.02 | Maximum | 0.42 | Median: | 0.19 | |||
Cluster 10: research stage: frameworks | ||||||||
11 | Discussing whether to study sustainability separately from implementation (e.g., How are the two related? not related?) | .28 | ||||||
10 | Testing of theories/frameworks for sustainability | .40 | ||||||
31 | Creating greater distinction between predictors of sustainability (e.g., organizational capacity) and sustainability outcomes (sustained programming) | .42 | ||||||
26 | Studying a greater variety of sustained activities, including interventions, programs, and policies | .43 | ||||||
16 | Assessing the dynamic processes underlying sustained use | .50 | ||||||
57 | Developing case studies to identify key characteristics of those that do sustain vs. do not | .63 | ||||||
Count: | 6 | Std. Dev.: | 0.11 | Minimum | 0.28 | Average: | .44 | |
Variance: | 0.01 | Maximum | 0.63 | Median: | 0.42 | |||
Cluster 11: research stage: definitions | ||||||||
24 | Conceptualizing and defining “sustainability”, its sub-dimensions, and related concepts (e.g., fidelity, routinization, institutionalization) | .08 | ||||||
79 | Defining sustainability outcomes | .10 | ||||||
73 | Creating a visual depiction of a sustainability model | .16 | ||||||
32 | Identifying dimensions and degrees of sustained use | .16 | ||||||
80 | Clarifying terminology (e.g., assimilation, institutionalization, continued use) | .17 | ||||||
71 | Developing a formal conceptual model that links dissemination, implementation, and sustainability | .17 | ||||||
40 | Balancing the use of pertinent theory, evidence, and experience | .29 | ||||||
20 | Determining whether there is a standard level of initial “success” necessary before a project or organization or community is “eligible” to be considered for a “sustainability” evaluation | .32 | ||||||
Count: | 8 | Std. Dev.: | 0.08 | Minimum | 0.08 | Average: | .18 | |
Variance: | 0.01 | Maximum | 0.32 | Median: | 0.17 | |||