Measurement of sustainment of prevention programs and initiatives: the sustainment measurement system scale

Background Enhancing the sustainability of evidence-based prevention programs for mental and behavioral health requires tools for measuring both sustainability determinants and sustainment outcomes. The aim of this study was to develop the Sustainment Measurement System Scale (SMSS) and to assess its reliability and construct validity for measuring both determinants and outcomes of efforts to sustain prevention programs and initiatives. Methods A 42-item scale comprised of items identified from qualitative data collected from 45 representatives of 10 programs and 8 SAMHSA program officers was administered to 186 representatives of 145 programs funded by 7 SAMHSA prevention grant initiatives. Cronbach’s alphas were used to determine inter-item reliability. Convergent validity was assessed by comparisons of a global measure of sustainment with current SAMHSA-funding status and continued operation in the same form. Discriminant validity was assessed by comparisons of sustainability determinants with whether or not the program had undergone adaptations. Results Confirmatory factor analysis provided support for a 35-item model fit to the data. Cronbach’s alpha was .84 for the sustainment outcome construct and ranged from .70 to .93 for the sustainability determinant constructs. All of the determinant constructs were significantly associated with sustainment outcome individual and global measures for the entire sample (p < 0.01 to 0.001) and for community-based programs and programs with a substance abuse focus (p < 0.05 to 0.001). Convergent validity was supported by significant associations between the global sustainment measure and current SAMHSA funding status and continued operation in the same form (p < 0.001). Four of the sustainability determinant constructs (responsive to community needs; coalitions, partnerships, and networks; organizational staff capability; and evaluation, feedback, and program outcomes) were also significantly associated with current SAMHSA funding status (p < 0.5 to 0.01). With the exception of organizational staff capability, all sustainability determinants were unrelated to program adaptation as predicted. Conclusions The SMSS demonstrated good reliability and convergent and discriminant validity in assessing likelihood of sustainment of SAMHSA funded prevention programs and initiatives. The measure demonstrates potential in identifying predictors of program sustainment and as a tool for enhancing the likelihood of successful sustainment through ongoing evaluation and feedback.


Background
Sustainment is considered to be the final stage of the process of implementation of evidence-based practices, policies, and programs (EBPs) [1,2]. Defined as "the continued use of program components and activities for the continued achievement of desirable program and population outcomes," [3] sustainment is considered to have occurred when, after a defined period of time, a program, clinical intervention, and/or implementation strategies continue to be delivered and/or individual behavior change (i.e., clinician, patient) is maintained, either as originally planned or with some degree of adaptation, while continuing to produce benefits for individuals/systems [4]. However, what is to be sustained differs from one program to the next [5,6]. For instance, with respect to the community coalitions supporting drug and suicide prevention activities, some definitions of sustainment focus on the continued existence of the coalition itself while others focus on the activities and impacts of the coalition [7]. Moreover, sustainability, defined as the capacity to maintain EBP components [8], is increasingly being viewed as a dynamic process with shifting outcomes that represents increased likelihood that the program or infrastructure will continue, while sustainment refers to the continued operation of a program or initiative, often relating to the time after initial funding has ended [8,9].
In recent years, there has been a proliferation of frameworks that focus specifically on sustainability [3,4,[7][8][9]. However, despite the growing consensus as to how sustainability should be defined [3,4,9], the underdeveloped state of measurement of sustainment poses one of the most serious methodological challenges to understanding and facilitating sustainability of evidence-based practices and programs [9][10][11]. Some instruments like the Stages of Implementation Completion (SIC) [12] have been developed to measure the implementation outcomes of a specific intervention [13]. The SIC is an 8-stage assessment tool developed as part of a largescale randomized implementation trial that contrasted two methods of implementing an EBP for youth with serious behavioral problems in the juvenile justice and child welfare systems. The eight stages range from Engagement (stage 1) with the developers/purveyors in the implementation process, to achievement of Competency in program delivery (stage 8) that span three phases of implementation including pre-implementation, implementation, and sustainability. The SIC was developed to measure a community or organization's progress and milestones toward successful implementation of Treatment Foster Care Oregon (TFCO) regardless of the implementation strategy utilized. Within each of the eight stages, subactivities are operationalized and completion of activities are monitored, along with the length of time taken to complete these activities. Competency in program delivery (stage 8) is considered to be a measure of program sustainment [13], although continued use of the EBP once competency has been attained is not assessed. Data on the validity and reliability of the SIC to assess sustainment outcomes is lacking. Other instruments like the Program Sustainability Assessment Tool (PSAT) [14][15][16], Sustained Implementation Support Scale (SISS) [17], and Program Sustainability Index (PSI) [18], take a broader ecological approach to sustainability and are used primarily to assess capacity and to plan for sustainability by measuring determinants or factors that influence sustainability. The PSAT is a 40-item scale, containing 8 sustainability domains (Environmental Support, Funding Stability, Partnerships, Organizational Capacity, Program Evaluation, Program Adaptation, Communications, and Strategic Planning), with 5 items per domain [15], and designed to measure capacity for sustainability a public health program's capacity for sustainability. The SISS is a 28-item scale composed of five subscales (Program Benefits, Program Burdens, Workplace Support, Workplace Cohesion, and Leadership Style) and designed to measure capacity to maintain implementation of components of an EBP for parent training. The PSI is a 53-item scale reflecting 7 sustainability elements (leadership competence, effective collaboration, understanding the community, demonstrating program results, strategic funding, staff involvement and integration, and program responsivity). All three instruments have demonstrated validity and reliability as measures of a program's capacity for sustainability. However, none of these instruments assess both sustainability determinants and sustainment outcomes.
One of the reasons for developing a valid and reliable measure of sustainment is to provide a means of conducting an audit and providing feedback to organizations engaged in implementing innovative and evidence-based programs and practices. While monitoring and feedback

Contributions to the Literature
Understanding of evidence-based practice sustainment requires improved tools for measurement of determinants and outcomes.
The Sustainment Measurement System Scale (SMSS) is the first tool of its kind that is designed to measure both sustainment determinants and outcomes.
The SMSS demonstrates good reliability and convergent and discriminant validity.
The SMSS can be used to conduct audits and provide feedback to grantees committed to sustaining prevention programs and initiatives once initial funding has ended. are recognized as important for prevention [19,20], much of the relevant science on feedback in health has involved improvement in clinical performance [21][22][23][24][25][26][27]. This includes clinical supervision and use of technology like electronic dashboards in measurement-based quality improvement (MBQI) strategies that monitor patient behavior and clinician activity [28][29][30], while prevention has a more limited history of using computational technologies for monitoring [31][32][33][34]. Such feedback offers the clinician a better understanding of whether they are on course to achieve a successful outcome or need to alter their treatment in order to improve the likelihood of a successful outcome. MBQI strategies also hold great promise for facilitating implementation of evidencebased practices [35]. Audit and feedback has demonstrated that it can be an effective strategy for implementation [36,37], but there is little evidence to suggest that it is equally effective in achieving the sustainment stage of implementation [38].
Federal agencies responsible for wide-scale delivery of prevention programs, including the Substance Abuse and Mental Health Services Administration (SAMHSA), routinely collect information from their grantees to monitor progress toward completion of goals and objectives. SAMHSA supports a wide array of prevention grant programs targeting mental, emotional, and behavioral disorders including illicit substance use, suicide, and antisocial behavior. Each of SAMHSA's prevention initiatives have specific sets of goals and objectives, and each have different prevention approaches that the administration expects will be sustained once support from SAMHSA is no longer available. As part of their initial proposal for funding, all SAMHSA grantees are required to submit a plan for sustainment of the grantee's activities once the federal funding has come to an end. SAMHSA programs currently rely on electronic data collection systems including the Transformation Accountability (TRAC) data collection system for SAMH SA's Center for Mental Health Services (CMHS) programs, and the Coalition Online Management and Evaluation Tool (COMET) and the Performance Management Reporting Tool (PMRT) used by SAMHSA's Center for Substance Abuse Prevention (CSAP). This information is used to provide feedback to grantees when there is evidence of failure to achieve goals and objectives. However, there is no empirical evidence that such feedback leads to an improvement in performance or increases the likelihood of sustainment.
While SAMHSA remains deeply concerned about sustainment, its mission does not allow monitoring of its grantees after funding has ended. In building a partnership with SAMHSA, the National Institute on Drug Abuse (NIDA)-funded Center for Prevention Implementation Methodology (Ce-PIM, P30DA027828) responded to this need by undertaking the development of a sustainability measure across diverse prevention programs and test its predictive validity over time [39]. This measure was intended for use in a Sustainment Measurement System (SMS), a process designed to evaluate likelihood toward successful sustainment of SAMHSA-funded programs and initiatives and to provide feedback to grantees that would enhance the likelihood of successful sustainment. The SMS combined existing sources of information obtained from SAMHSA grantees described above with information gathered by an instrument that assessed determinants as well as outcomes of sustainment to identify and support both the unique requirements for improving sustainment for individual funding initiatives as well as for developing a generalizable framework of sustainment across diverse prevention approaches, thereby bringing precision to monitoring the structures and processes for sustaining each prevention approach and improving the likelihood of achieving sustainment of any grantee's prevention efforts, regardless of source of funding [39].
The objective of the current study was to explore and identify dimensions of the Sustainment Measurement System Scale (SMSS) that was designed to assess the sustainment of prevention programs and initiatives by generating quantitative items from qualitative content domains of sustainment outcomes and sustainability determinants and subjecting them to confirmatory factor analysis in order to discern their factor structure. Our goal was to develop a scale that focused on the determinants and outcomes of sustainment of prevention programs and initiatives and to examine its factor structure, reliability, and construct validity.

Background
Representatives from grantees supported by the following seven SAMHSA programs took part in the study: (1) the Sober Truth on Preventing Underage Drinking Act (STOP-Act) grants committed to prevent and reduce alcohol use among youth and young adults; (2) the Implementing Evidence-based Prevention Practices in Schools (PPS) grants designed to address the prevention of early childhood behavioral disorders; (3) the Strategic Prevention Framework State Incentive (SPF-SIG) grants to prevent the onset of substance use amongst youth; (4) the Garrett Lee Smith (GLS) State and Tribal Youth Suicide Prevention grants to address depression, suicidal attempts, and behavioral health problems linked to suicide; (5) the Substance Abuse and HIV Prevention Navigator Program for Racial/Ethnic Minorities (Prevention Navigator, PN) grants serving minority populations at-risk for substance use and HIV; (6) Minority Serving Institutions Partnerships with Community-Based Organizations (MSI-CBOs) grants serving communities at risk for substance use, HIV, and hepatitis-C infections; and (7) the Capacity Building Initiative for Substance Abuse (SA) and HIV Prevention Services for At-Risk Racial/Ethnic Minority Youth and Young Adults (HIV-CBI) grants to prevent and reduce the onset of SA and transmission of HIV/AIDS among at-risk populations ages 13-24.

Participants
Program Officers representing the 7 SAMHSA funding initiatives provided names and email addresses of all project directors and key personal for each organization currently or previously funded under these initiatives. Researchers identified other people to contact from information provided by project directors of all PPS grantees funded by RFA SM-10-017. Names and email addresses for two to three contacts (i.e., director, coordinator, local evaluator) per GLS grantee site were obtained from ICF international, the GLS contracted evaluators of the grant program, for grantees belonging to Cohorts 8 through 12, per recommendation of SAMHSA program officers. Invitations to participate in the study were sent to 528 representatives of 306 grantee sites supported by the 7 SAMHSA grants between November 2017 and March 2019. One-hundred eighty-six representatives of 145 grantees agreed to participate, resulting in an individual representative response rate of 35.2% and a grantee response rate of 47.5%.

Procedure
The study was approved by the appropriate Institutional Review Boards prior to participant recruitment, and informed consent was obtained prior to administering surveys. Study data were collected and managed using REDCap electronic data capture tools hosted at Northwestern University. REDCap (Research Electronic Data Capture) is a secure, web-based application designed to support data capture for research studies, providing (1) an intuitive interface for validated data entry, (2) audit trails for tracking data manipulation and export procedures, (3) automated export procedures for seamless data downloads to common statistical packages, and (4) procedures for importing data from external sources [40].
Each participant was emailed an invitation to participate including a link to the web-based survey. Participants reviewed informed consent and after agreeing to participate were able to access the survey and proceed to the survey items. Once participants logged in to the online survey, they were able to answer questions and could pause and resume at any time. The online survey took approximately 15 to 20 min to complete.

The Sustainment Measurement System Scale (SMSS)
Item development for the SMSS is described in detail elsewhere [41]. Briefly, interviews were conducted with 45 representatives of 10 SAMHSA grantees and 9 SAMHSA government program officers that was comprised of three parts: (1) a series of semistructured questions relating to experience with implementing and sustaining the grantee's program; (2) a free list exercise [42] asking participants to respond to whatever came to mind when asked what was meant by the term sustainment or sustainability, what components of their programs they most wanted to sustain, and what it would take to sustain those components; and (3) a template of Consolidated Framework for Implementation Research (CFIR) domains and components [43] in which participants were asked to rate each of the domains and elements as being unimportant (0), somewhat important (1), important (2), or very important (3) to sustainability of their program and to explain the basis for their assessment of each component to sustainability. The CFIR template was used to determine whether the components believed to be associated with successful implementation are also associated with successful sustainment. Templates of the PSAT and SIC sustainability measures informed the semi-structured interview questions and analysis of the free-list exercise.
The data sets containing sustainability elements from each of the three components of the interview (semi-structured interview, free lists, and CFIR checklist) were then compared through a process of data triangulation (i.e., determining consistency of findings obtained from different sources of data) to identify items that were elicited from more than one data set. Items were then placed into three groups: (1) those that appeared in only one of the three data sets, (2) those that appeared on two of the three data sets, and (3) those that appeared on all three data sets. Four sustainability elements were identified by all three data sets: (1) ongoing coalitions, collaborations, networks, and partnerships; (2) infrastructure and capacity to support sustainability; (3) community need for program; and (4) ongoing evaluation of performance and outcomes). An additional 11 elements were identified by two of three data sets: (1) availability of funding; (2) consistency with organizational culture; (3) evidence of positive outcomes; (4) development of a plan for implementation and sustainment; (5) presence of a champion; (6) institutionalization and integration of program; (7) institutional support and commitment; (8) community buy-in and support; (9) program continuity; (10) supportive leadership; and (11) opportunities for staff training [41].
To construct this survey, the fifteen elements were condensed to create a 42-item scale divided into the following subscales: Sustainment outcomes: 4 items assessing the continued operation of the program or initiative, including delivering prevention services to intended population that are evidence-based as described in the original application for funding and periodically measuring service fidelity. Financial stability: 6 items measuring funding from federal, state, or local governments and non-profit and non-governmental sources, combination of earmarked and discretionary funding, sustained funding, financial support from diverse community interests, and financial solvency. Responsiveness to community needs and values: 7 items measuring meeting needs of intended target populations and behavioral health needs of communities/populations being served, adaptability to meet these needs, consistency with norms and values of participating organizations, fit with values of sustaining organizations and communities, shared perception of project importance by participating organizations, and unacceptability of public health problem addressed by project. Coalitions, partnerships, and networks: 8 items measuring networking of grantee organization with other organizations committed to program sustainability, community engagement in development of project goals, community access to knowledge and information about the project, project support by a coalition/partnership/network of community organizatons, network expansion, commitment to continued operation of project, level of networking, and communications within organizations responsible for sustaining the project. Infrastructure and capacity to support sustainment: 9 items measuring available resources for project implementation and sustainment, integration into operations of the organization and partners, advanced development of plans for implementing and sustaining the project, execution of the project according to these plans, adequacy of staff to sustain program goals and activities, sufficiency of training available to staff and community members, staff knowledge and supportive beliefs, and staff selfefficacy to implement the project. Implementation leadership: 5 items measuring active engagement of leaders in project implementation and sustainment, involvement of community leaders in the project, appointment of someone responsible for coordinating project implementation and sustainment, support from a program champion, and process in place for sustainment in the event the champion leaves. Monitoring, evaluation, and program outcomes: 3 items measuring ongoing evaluation of progress made toward sustainment, sufficiency, and timeliness of feedback about project delivery and quality improvement, and evidence of positive outcomes.
Study participants were asked to indicate level of agreement with a series of statements using a Likert scale ranging from 1 (not at all) to 5 (all the time) for the four items contained in one subscale measuring sustainment outcomes and seven different domains of determinants of sustainment; lower scores on all subscales indicate lower levels of agreement, while higher scores indicate higher levels of agreement with the respective statements. Each subscale score is represented as an average of the scores for each item included in the subscale. The average of the scores for each item included in the Sustainment Outcomes subscale was defined as Global Sustainment.

Program characteristics
Two sets of variables measuring characteristics of the funded programs and initiatives were included in the analysis to determine whether the association between sustainability determinants and sustainment outcome varied by one or more program characteristics and to assess the convergent and discriminant validity of the SMSS. The first set included a categorical variable describing program type (community or state) and a categorical variable describing program focus (mental health or substance use). The second set also included two variables, one describing whether the grantee is currently funded by one of the seven SAMHSA grant initiatives (yes or no), and if not, a variable describing the current status of the grantee's program (the project no longer exists, the project continues to exist and it has been substantially adapted, and the project continues to exist in much the same form as it did when funded by the SAMHSA program).

Statistical analyses
To evaluate the psychometric properties of the SMSS, confirmatory factor analyses (CFA) with maximum likelihood estimation were conducted using EQS statistical software [44,45]. The aim of the CFA was to determine if the data fit the hypothesized model of sustainability determinants and sustainment outcomes based on the qualitative research conducted in earlier study described above [41,45]. CFA is commonly used to confirm a hypothesized model based on theory or prior empirical research, as well as evaluate the reliability and validity of measures employed in implementation research [15,17,18,[46][47][48]. Initially, confirmatory factor analysis was applied to all 42 items to identify poorly performing items and test our hypothesized sustainment domain structure, followed by subsequent modifications based on resulting model modification indices and theoretical justification [49]. Poor items were those that had low reliability or poor fit (i.e., factor loadings below 0.300) with the intended latent factor (or subscale). Principal axis factoring (PAF) with Promax oblique rotation was also conducted to provide guidance on item reduction and construct reclassification. Principal axis factoring was selected for factor extraction because it allows for consideration of both systematic and random error [50], and Promax oblique rotation was utilized as we assumed that the derived factors would be correlated [51]. Three criteria were used to determine the number of factors to retain: (1) examination of the oblique rotated factor pattern matrix, (2) parallel analysis [52], and (3) interpretability of the factor structure as indicated in the rotated solution. Examination of the rotated factor structure included identification of eigenvalues above 1.0 and Scree test results, as well as absence of multicollinearity and presence of outliers [53]. We then used four measures of model fit to assess model adequacy for the initial model and revised models: chi-square statistic, the comparative fit index (CFI), root mean square error of approximation (RMSEA), and the standardized root mean square residual (SRMR). CFI values greater than 0.95, RMSEA values less than 0.06, and SRMR values less than 0.08 indicate model fit that is deemed acceptable [49].
Reliability of the SMSS was assessed by examining Cronbach's alpha internal consistency for each of the subscales and the total scale. Cronbach's alpha of .70 and above is considered good internal consistency for a newly developed scale [54]. Corrected item total correlation tests were also conducted in order to check each item's contribution to the total scale. An item to total correlation higher than .4 was considered acceptable [55]. Item analyses were also conducted, including an examination of inter-item correlations and alpha if the item is removed.
Construct validity of an instrument is typically assessed by comparing constructs or measurements that theoretically should be related to one another to determine if that is the case (convergent validity) and by comparing concepts or measurements that are not supposed to be related to determine if they are actually unrelated (discriminant validity) [56]. In this study, convergent and discriminant validity were assessed by computing Pearson productmoment correlations of SMSS determinant subscales and global sustainment outcome scores, and one-way analysis of variance of mean outcome and determinant subscale scores by current funding from the original SAMHSA initiative (yes/no) and the program status (no longer in operation, exists but has been extensively adapted, exists pretty much in the same form), respectively. Only 5 programs were no longer in operation at the time the study was conducted and therefore removed from the comparison, even though their mean values were lower than those for the other two types of programs. Current funding by the original SAMHSA initiative was chosen to assess the convergent validity of the SMSS as it was hypothesized that grantees currently funded would have higher mean scores of sustainment outcomes than grantees no longer funded. Program status was chosen to assess the divergent validity of the SMSS because the determinants were hypothesized to be unrelated to whether the sustained program was adapted or not as long as it continued to exist.

Results
There were no missing values for any of the 4 sustainment outcome variables; however, missing data for each of the determinant variables ranged from 15.8 to 17.9%. The same 29 to 31 individual grantee representatives did not provide responses to any of the determinant items and were removed from analyses of sustainability determinants by sustainment outcomes or determinants by current funding status or program status. As a result, 113 (77.9%) of grantees were represented by one individual, 26 (17.9%) grantees were represented by two individuals, and 6 grantees were represented by 3 (4.1%) individuals. A flow chart describing the numbers of participants from recruitment to data collection to data analysis is presented in Fig. 1. The effects of nesting of individuals within grantees was confirmed by the absence of any differences in associations between determinants and outcomes by grantee and associations between determinants and outcomes by individual participants.

Confirmatory factor analysis
Results of the CFA analyses are presented in Table 2. In the initial model, we gave no consideration of program type or program focus affecting either the measurement model (e.g., loadings and unique variances) or the structural model (e.g., means, variances, and covariances of the latent variables). Although the 6 sustainment subscales and 1 sustainment outcome subscale exhibited good to excellent reliability, CFA analyses indicated poor fit of the 42-item nine-factor model (labeled Initial Model) across the four major indices (chi-square, CFI, RMSEA, SRMR). Elimination of items with a factor The project has a formally appointed person responsible for coordinating the process of implementing and sustaining the project. The project is also supported by a champion who is actively engaged in the process of implementing and sustaining the project.
.  correlation of less than .300 resulted in 36 items. Results of an exploratory factor analysis revealed 10 domains that included a segmentation of the responsiveness to community needs and values subscale into two subscales-responsiveness to community needs (2 items) and responsiveness to community values (3 items)-and a segmentation of infrastructure and capacity subscale into two subscales-organizational capacity (4 items) and organizational staff capability (3 items). Only 1 item (community leadership) loaded onto the 10th factor, which was also eliminated for further analysis. One of the items from the leadership subscale also exhibited a much higher loading in the coalition, partnerships, and networks factor (.660) than on the leadership factor (.129). The resulting 35-item nine factor model (labeled Final Model) provided better fit with an absolute model fit of χ 2 (517) = 960.23, p < .001, an RMSEA of .076 and an SRMR of .066 meeting criteria for acceptability; however, fit was not supported by a CFI of .95 or greater. The standardized factor loadings ranged from .56 to .88, and all were statistically significant (all p levels < 0.001). Separating the responsiveness to community needs and values subscale into two separate subscales also increased the Cronbach's alpha coefficient from .60 to .70 for both subscales. The 35-item version of the SMSS is provided in the Appendix. Table 3 shows the correlations among the SMSS subscales. In general, the correlations between the responsiveness to community values subscale and the other eight subscales (average r = .19) were lower than the correlations among the other eight subscales (average r = .42). Of particular note, sustainment outcome was significantly associated with all 8 determinants (average r = .43), with significance levels < 0.001 except for financial stability (p < 0.01). Table 4 shows the results of correlations between the global sustainment outcome score and the 8 determinant subscale scores stratified by program type and focus. Global sustainment was significantly associated with all 8 determinants in community-based programs and programs targeting substance use prevention. Global sustainment was associated with responsiveness to community needs and values and organizational capacity and staff capability in statewide prevention programs, and with six of the eight determinants (all but financial stability and responsiveness to community needs) in programs that focused on mental health.

Convergent and discriminant validity
A comparison of sustainment determinants and outcomes by program current funding status in provided in Table 5 The discriminant validity of the SMSS was assessed in comparisons of total sustainment outcomes and determinants by program status. The results are presented in Table 6 below. The global sustainment outcome score and the individual measures of sustainment were significantly greater in programs that continued to exist in the same form. However, only one determinant was found to be significantly greater in such programs (organizational staff capability; F = 4.52, d.f. = 146, p = 0.035), suggesting strong support for the discriminant validity of the SMSS.

Discussion
Although there exist other measures designed to evaluate sustainability of evidence-based programs and interventions from the point of view of the determinants of sustainability or sustainment as an outcome, the Sustainment Measurement System Scale (SMSS) is the first instrument designed to assess both sustainability determinants and sustainment outcomes. Further, the SMSS is designed to assess different prevention programs of different types and foci. As such, it contains elements that are specific to particular programs but enable comparisons across different types of programs. It also demonstrates some potential as a tool for providing feedback to organizations, enabling them to monitor their trajectory towards achieving the final stage of state of implementation and increase the likelihood of successfully doing so. The SMSS draws from and shares several features of existing measures of sustainability. For instance, like the Stages of Implementation Completion Scale [12], the SMSS measures sustainment as an outcome and can be adapted for use as a tool for monitoring progress toward sustainment [13]. Unlike the SIC, it measures characteristics of programs and their inner and outer settings that  predict these outcomes and excludes earlier phases of implementation. Like the Program Sustainability Assessment Tool [15], Sustained Implementation Support Scale [17], and Program Sustainability Index [18], the SMSS measures determinants of sustainability and exhibits similar internal consistency, model fit, and construct validity. With the possible exception of responsiveness to community values, the domains are highly correlated with one another. SMSS has similar chi-square significance, RMSS, and SRMR as the PSAT. All of these instruments have less than desirable CFI of .95, suggesting room for improvement. The SMSS is the product of a standardized process of eliciting determinants and outcomes of sustainability that are both specific to the program being sustained and generalizable to other types of programs and initiatives [41]. This approach is also more consistent with the Dynamic Sustainability Framework [5] and the growing consensus of sustainability as a dynamic process with changes in both determinants and outcomes over time.
In this study, correlations between sustainability determinants and sustainment outcomes were stronger with community-based programs and programs targeting substance use than with state-based programs and programs targeting mental health. However, there was consistency across programs by type and focus with respect to certain determinants such as responsiveness to community values (which included adaptability), organizational capacity, and organizational staff capability.
The design and evaluation of the SMSS also provides some important insights into sustainment as the final stage of the implementation of evidence-based prevention programs and initiatives. The process of achieving sustainment is the product of eight domains of sustainability determinants: financial stability; responsiveness to community needs; responsiveness to community values; coalitions, partnerships, and networks; organizational capacity; organizational staff capability; implementation leadership; and evaluation and feedback and positive program outcomes. Luke and colleagues [15] found similar associations between program manager and staff perceptions of sustainability of their programs and the determinants of funding stability, partnerships, organizational capacity, and program evaluation. Hodge and colleagues [17] found similar associations between sustained implementation of the Triple P parent training program and the determinants of program benefits, workplace support, and leadership style and. Mancini and Marek [18] found similar associations between meeting at risk needs and the determinants of leadership competence, effective collaboration, demonstrating program results strategic funding, staff involvement and integration, and program responsivity. In our study, these associations are consistent across community level programs and/or a focus on substance use prevention, less so with state level programs, and/or a focus on mental health. Many of these determinants decline once funding that supported their implementation comes to an end, which is to be expected since many of them like coalitions, partnerships and networks, staff capability (e.g., training), and the capacity for conducting evaluations and providing feedback are resource-dependent. As predicted, however, it makes no difference whether the program continues to exist in same form as originally proposed or has been adapted to improve fit, also consistent with the Dynamic Sustainability Framework. This may be because many programs are not necessarily implementing a specific practice, and some are just trying to sustain coalitions.
Nevertheless, there are a number of factors that limit the findings of this study. The findings reflect the experience of specific prevention programs and initiatives that have been or are currently funded by SAMHSA; thus, their generalizability to prevention programs funded by other sources (e.g., state or local funds, research grants), to programs targeting treatment, or event to other SAMHSA-funded programs, is limited. The confirmatory factor analysis was constrained by the small number of programs and program representatives. Only two items loaded onto one of the identified constructs (responsiveness to community needs), while only three items loaded onto the construct of responsiveness to community values. Further research is required to determine whether or not they represent two distinct constructs.
With these limitations in mind, future steps include the following: (1) evaluation of the utility of the SMSS with other types of programs in other settings, (2) use of larger samples to confirm fit of the data to the model, (3) development of guidelines for providing feedback to organizations seeking to sustain programs and initiatives based on ongoing monitoring efforts using the SMSS, and (4) consistent with the SMS process, ongoing revision of the tool itself.

Conclusion
The SMSS is innovative in three specific respects. First, it draws upon the experience of evaluating sustainability in different types of prevention programs and initiatives with different aims and areas of emphases. This enabled us to identify a set of common elements of sustainment that can be used to generate a model and set of testable hypotheses that apply to a broad array of substance use/mental disorder/suicide prevention programs, practices and initiatives, regardless of objectives, outcomes, and infrastructure to achieve these outcomes. Second, although we did not assess the effectiveness of the various programs, practices, and initiatives participating in this study, a measure like the SMSS offers the potential to determine whether the extent to which a program or practice is evidence-based or evidence-informed determines whether it can be sustained. Such information would be invaluable to determining whether the program or practice merits initial or continued funding. Third, this measure can be used to monitor progress toward sustainment and provide feedback to stakeholders as to how to increase the likelihood of sustainment. The SMSS in particular and the SMS in general can be used as a tool for program management as well as research purposes. Although the SMSS was based on programs funded by SAMHSA, the instrument should have general applicability across diverse federal, statewide, and local prevention implementation initiatives.