Associations between implementation characteristics and evidence-based practice sustainment: a study of the Adolescent Community Reinforcement Approach

In order to improve health care quality, many government agencies and community organizations have invested significant resources to encourage evidence-based practice (EBP) implementation with the hope that adoption of EBPs will lead to improved health care outcomes. This is particularly relevant for the treatment of adolescent substance use disorders for several reasons. First, the quality of adolescent treatment services has been known to be highly variable [1, 2] with fewer than half of those receiving treatment achieving recovery a year following treatment [3]. Second, treatment for substance use disorders has been slow to be integrated into mainstream medical care settings and has heavily relied on community and peer-based approaches that lack a strong theoretical and empirical base. Moreover, developmentally appropriate treatments for adolescents are often not incorporated into practice settings [4, 5]. Providing EBPs may mitigate the future consequences for involved adolescents and help to reduce the public burden of caring for a chronic relapsing condition that is estimated to cost in excess of $600 billion a year [6–8].

In order to address this critical gap, the Substance Abuse and Mental Health Services Administration’s (SAMHSA) Center for Substance Abuse Treatment (CSAT) has offered discretionary grant funding to community-based treatment organizations in order to help facilitate the delivery of EBPs for adolescent substance use disorders. Among one of the largest investments to date has been support for an EBP called the adolescent community reinforcement approach (A-CRA). A-CRA is a behavioral intervention that seeks to replace environmental contingencies that encourage substance use with pro-social activities and behaviors that promote recovery. Three randomized controlled studies of A-CRA have yielded positive outcomes across alcohol use, mental health, and social domains [3, 9, 10]. Since 2006, SAMHSA has provided multi-year funding to 84 community-based treatment organizations located in over 27 states/districts/territories across the USA and has totaled more than 150 million dollars in funding. The discretionary grants that treatment organizations received were 3 years of implementation support, which included approximately $300,000 annually, 2.5 days of in-person A-CRA training followed by a standardized clinical and supervision certification process that incorporated technology-assisted performance feedback [11].

Despite this large investment to support EBP implementation, little is currently known about how effective these grant mechanisms are in supporting the sustainment of A-CRA following the SAMHSA funding period. That is, the long-term effectiveness of these investments in improving longer-term EBP implementation is unknown. To address this question, we looked to the burgeoning field of implementation science to inform our research hypotheses. We found a diversity in the definitions of practice sustainment and sustainability [12–18] and conceptual frameworks [12, 19–26] regarding implementation; however, across this literature, we found four main factors theorized to influence sustainment: (1) the broader community environment, external to the organization implementing the practice (i.e., “outer setting”), for example, policy and fiscal factors; (2) the level or extent of implementation (or implementation quality) during the funded and/or initial period; (3) factors within the organizational or “inner” setting, such as strategic planning, program evaluation, and program adaptation; and (4) elements of the practice itself, that is, the intervention characteristics and staff perceptions of the treatment. Although each of these factors may conceptually play a role in EBP sustainment, we posit that such factors as external funding and political factors may trump such factors as inner setting activities which include strategic planning, evaluation, and communication.

Using data collected from key clinical staff, we found that the sustainment of an evidence-based adolescent treatment protocol was likely to eventually be discontinued by a majority of programs within 3–4 years after the initial federal funding ended. More specifically, about two thirds of the organizations reported A-CRA sustainment at 12–24 months following the end of funding. However, fewer than half the programs continued to sustain A-CRA 36–48 months after funding ended. These findings are comparable to those reported by Cooper et al. [32] regarding seed funded prevention programs at 2 years post-funding (69 %) and by Scheirer who conducted a review of community coalition health-related program studies [44] (60 %). We are not aware of any comparable studies that focus on the sustainment of substance use treatment or more specifically, adolescent treatment programs which may be more likely to sustain themselves using a different organizational structure and funding mix than programs operated by community coalitions or prevention programming that is often embedded in schools settings.

In terms of factors that appear related to A-CRA sustainment, we found that setting-, implementation-, and intervention-related characteristics were important. The type of organization that was funded, the level of implementation during the funding period, funding stability, political support for the program, and staff perceptions of A-CRA’s complexity and implementation difficulty were related to A-CRA sustainment. These findings are consistent with previous theories that suggest that existing infrastructure, such as the intervention being consistent with the overall mission of the organization, is key to adoption, implementation and sustainment [45, 20, 19]. More specifically, one of the most compelling findings was staff that primarily described their organization as substance use focused, rather than mental health, general health care, or some other focused were more likely to report sustaining A-CRA. Although A-CRA has been demonstrated to be effective among adolescents with co-occurring mental health disorders and the majority of adolescents in these programs are referred from criminal justice settings [46–48], it appears that substance use treatment-oriented organizations are better equipped to continue delivering an adolescent substance use treatment program than organizations with a mental health or other type of focus. The findings appear to also be consistent with the Tibbitts et al. [29] who found that organizational (in this case, school) support and program “fit” were related to 1- to 3-year program sustainment and several previous studies that emphasize that funding stability is the primary element to program sustainment [44]. For example, following the federal initiatives that supported the participating organizations to deliver A-CRA, federal support for adolescent treatment has been allocated to single state agencies for substance use that can then grant funds to local community-based organizations that have existing capacity to deliver substance use treatment (e.g., SAMHSA CSAT’s Cooperative Agreements for State Adolescent and Transitional Aged Youth Treatment Enhancement and Dissemination [49]). Therefore, organizations with a demonstrated track record for delivering substance use treatment may be better positioned to receive future funding to support adolescent treatment.

It is also somewhat surprising that a host of factors typically associated with program adoption and implementation, such as staff perceptions of organizational capacity, program planning, communication, and evaluation activities were not found to be significantly associated with sustainment. Although there often efforts by funders to build the evaluation capacity of their grantees and assist them with strategic planning and communication about their program as well as help form partnerships among local program stakeholders, it appears that existing in a stable funding environment, regardless of evaluation and planning efforts, is critical to a program’s long-term survival.

The results may appear somewhat inconsistent with the previous research. For example, we found that the level of implementation at the end of the funding period (as assessed by the number of staff certified and employed at grant end and the number of youth served during the grant period) was associated with sustainment, whereas Peterson et al. [30] reported that few indicators of implementation quality were related to sustainment. It is important to note the Peterson et al. [30] study used assessments that represented longer time points (i.e., 2 years apart) and included implementation measures that were different from ours (e.g., staff turnover). We selected a more refined capacity measure than simply staff turnover, as we took into account the organization’s capacity at the end of the funding period (i.e., number of staff certified to deliver the treatment), which may have been more closely associated to sustainment than attrition rates. Given the lack of empirical studies on this topic, it is not unusual that the few existing studies use different time periods and assessment tools. More research is needed to help replicate these findings.

The study findings also diverged from those found by Cooper et al. [32] in that community support, and external partnerships were not found as an important factor associated with program sustainment. This inconsistency may be due to the differences between studied programs; Cooper et al. [32] focused on newly funded prevention programs which may require more community level support than established treatment programs as prevention programs are often built by community coalitions rather than being primarily operated within one particular organization, like found with many substance treatment programs.

In sum, we found that organizational focus, funding stability, political support, implementation quality during the funding period, and perceptions of the intervention in terms of complexity and implementation difficulty by key staff are important factors associated with program sustainment. These factors appear to be more critical to sustainment than many of the organizational factors often hypothesized as important to implementation (and thus sustainment) including staff perceptions of other organizational factors, such as program planning, communication, and evaluation activities.

We address several limitations to previous research on health care program sustainment. For example, to advance what is known about program sustainability, researchers have argued that it is necessary to study sustainability over several years rather than just a single point in time [28]. In this study, some organizations that we queried were just ending their initial federal funding support and others that had lost funding up to 4 years ago. Moreover, few past studies have used conceptual frameworks to inform their work. This study was developed taking into account several existing conceptual approaches to program implementation and sustainment, and therefore, we assessed factors both external to and internal to the organization along with intervention-specific components to sustainment. Also, few studies have employed longitudinal, prospective methods. Given the extensive data collected during the implementation period, we were able to examine several variables prospectively and demonstrate that such factors as the number of adolescents exposed to the treatment and the number of clinical supervisors and clinicians certified to deliver the treatment and still employed at the end of the funding period were both predictive of later A-CRA sustainment. These implementation factors appear especially important to predicting sustainment and represent more “objective” characteristics than staff perceptions and attitudes collected in the post-funding period. In sum, this study addresses many important gaps in previous research.

It is important to note that this study represents the first of 3 years of longitudinal data that are being collected. In the future, we plan to report on whether these findings are stable using a larger sample of organizations that have matured several years beyond the federal funding period. In addition, we will incorporate both qualitative and quantitative data to better explain the relationships between theorized factors and A-CRA sustainment. For example, thus far, we have learned from staff that competing funding requirements, that is, situations where local, state, or federal funding are tied to a different EBP sometimes shift the organization to a different treatment regime even though there is an organizational support for the treatment, and staff perceive it as an effective approach [50].

Some limitations to this study are important to acknowledge. First, this study relied on self-reported sustainment from key clinical staff charged with treatment delivery. We focused on reports from clinical staff, as they are most likely to be aware of the treatments offered and what adolescents received, as compared to administrative staff. However, self-reports may be subject to bias and may not represent the extent to which a treatment is delivered with fidelity. Our future work is to better explicate A-CRA sustainment through the assessment of core elements of the treatment as specified by the treatment developer (e.g., delivery of an adequate number of sessions and ongoing clinical supervision that is aligned with the treatment developer’s approach used during the funded implementation period). A second limitation to this study is the relatively small sample. We targeted the entire population of organizations that were funded and achieved over a 80 % response rate; however, to study the multitude of hypothesized factors and the potential interaction effects, a larger sample would be required; this is a commonly noted challenge in implementation research where the main analysis is often conducted at the organizational rather than individual level.