Predicting evidence-based treatment sustainment: results from a longitudinal study of the Adolescent-Community Reinforcement Approach

Previous literature

In the past decade, there has been a growing body of evidence on measuring EBT sustainment and the hypothesized predictors [6]. However, recent reviews suggest that research within the health care field is underdeveloped [7, 8]. One issue that has resulted from this work is the diversity of definitions regarding how to assess program sustainment. Generally, it has been assessed by a simple self-reported dichotomous indicator of whether a program is still in operation or not following an initial funding period [6]. Although a dichotomous variable of whether a treatment was delivered or not may be appropriate for medication trials or to understand utilization rates (e.g., [9]), findings from systematic reviews emphasize the need to better capture the extent or quality of sustainment because partial sustainment may not lead to the same desired outcomes observed in research trials, especially in relation to more complex treatments such as psychotherapy. Moreover, the research to date suggests that “partial” sustainment over full sustainment is commonly found after implementation support ends [7, 8, 10].

Systematic literature reviews also suggest that program sustainment is an intricate process, and the determinants or predictors of sustainment are mixed and vary across health care domains. As this study was conceptualized, two major implementation frameworks were published, the Exploration, Practice, Implementation, and Sustainment model (EPIS; [11]) and the Consolidated Framework For Implementation Research (CFIR; [12]). These frameworks both suggest that characteristics related to the broader community environment, external to the organization (i.e., “outer” setting or context), and characteristics operating within the organization or “inner” setting or context, such as organizational leadership, climate and culture, adequate staffing, and organizational processes, that help support EBT implementation such as strategic planning are relevant. In addition, characteristics of individuals charged with implementing the intervention and characteristics of the intervention itself are important. In sum, these conceptual frameworks suggest the need to take into account multiple factors operating at the system, organization, and treatment provider level to understand EBT adoption, implementation, and sustainment [13].

Taking into account these conceptual framework perspectives, findings from recent empirical studies have been inconsistent [1] and limited. For example, the use of small samples are common (e.g., prospective data from 32 programs [14]; 11 systems operating in two states [15]), studies that have been limited to the examination of demonstration projects (e.g., [10]) rather than programs operated by established organizations, and some studies that have only examined a subset of hypothesized factors, such as inner context characteristics (e.g., [5, 16]) and few rigorous studies have been conducted that use longitudinal data and measure the quality or extent of EBT sustainment. Therefore, a continued need exists for comprehensive empirical studies assessing the extent or quality of sustainment and the numerous hypothesized factors that predict it. Moreover, there is also little information about EBT sustainment in relation to psychosocial treatments for substance use which is typically provided in community-based settings that are often low or under-resourced and experience high staff turnover [17–19]. The current study helps address these gaps by examining the extent to which an EBT is sustained and what implementation factors, including a comprehensive set of potential factors that span both the inner and outer setting along with implementation- and intervention-related characteristics, are associated with sustainment.

Current study

Substance use disorder treatment and implementation support are largely funded in the USA through public sources [20], such as discretionary grants that enhance organizational capacity to utilize EBTs. In this study, we investigated one of the largest investments to date by the Substance Abuse and Mental Health Services Administration’s (SAMHSA) Center for Substance Abuse Treatment (CSAT) to address adolescent substance use disorders, by providing multiple year support to community-based organizations to implement the Adolescent-Community Reinforcement Approach (A-CRA). A-CRA has been found in five randomized controlled studies to yield positive outcomes across alcohol use, mental health, and social domains [21–25], and it is listed in SAMHSA’s National Registry of Evidence-based Programs and Practices (N-REPP). A-CRA supports youth and parents by enhancing family, social, educational, and/or vocational reinforcers to aid in recovery from substance use. As of January 2016, over 270 organizations received support to deliver A-CRA [26].

A-CRA implementation support model

The discretionary grants that treatment organizations received from SAMHSA during its initial dissemination efforts included approximately $300,000 annually for 3 years, plus three and half days of in-person A-CRA training for up to five clinical supervisors and clinicians followed by ongoing telephone coaching calls. The training processes were designed based on effective methods for disseminating EBTs [27, 28] and are described in greater detail in Godley et al. [6]. For example, individual clinicians received numeric and narrative feedback specific to reviews of recorded therapy sessions based on a detailed coding manual from external A-CRA experts [29] until they demonstrated competence and then randomly thereafter. A supervisor training process was designed to train supervisors to become internal A-CRA experts and included requirements that supervisors were able to reliably rate recorded therapy sessions and conduct regular internal supervision.

A-CRA sustainment

As with many EBT dissemination efforts, prior to the current study little was known about how effective the A-CRA implementation support effort was in promoting long-term use of A-CRA. We operationalized sustainment by assessing the extent to which core treatment elements were maintained following the end of the implementation support period and adequate organizational capacity to continue maintenance of these core elements was demonstrated. This definition is consistent with the recommendations from a recent systematic review [8], and this operationalization was also recently employed by Aarons and colleagues [15, 30]. The elements align with the definitions of implementation fidelity (e.g., [31, 32]) in that they included measurement of treatment dose, quality (as measured by staff knowledge, certification, access to high-quality training, and supervision), and penetration (i.e., proportion of staff trained and youth treated in A-CRA).

We attempted to capture factors related to external and internal support for A-CRA implementation and take into account assessments of the extent or quality of implementation during the funding period to capture how well organizations had built the capacity to continue to deliver A-CRA following the loss of implementation support. This approach is consistent with implementation theories that suggest that program sustainment only can occur within an organization that has first committed to adopting and implementing a treatment or practice [11]. In sum, the purpose of this study was to examine the extent of A-CRA sustainment following loss of federal funding among community-based organizations and identify what hypothesized implementation factors were empirically related to sustainment.

Study context

This project examined A-CRA sustainment among 82 community-based organizations that were awarded an implementation support grant by SAMHSA/CSAT between 2006 and 2010. During that period, there were four program cohorts called the “Assertive Adolescent Family Treatment” initiatives that encouraged the use of A-CRA as the treatment approach (e.g., [33]). In addition, other SAMHSA funding opportunities were offered during this period including the “Juvenile Drug Court” and “Juvenile Drug Treatment Court,” the “Offender Reentry Project,” and the “Targeted Capacity Expansion” initiatives. For these initiatives, the community-based grantee was required to identify an EBT, and several of the funded organizations selected A-CRA and therefore were included in our study sample.

Sample

Data sources

We used administrative data collected from organizations during their respective funding period to assess the implementation-related factors. We also collected three waves of interview and survey data approximately 9 months apart starting in Fall 2013 and ending in Spring 2015.

Primary data collection procedures

We used an administrative dataset containing the contact information of staff who were trained during the 3-year implementation period. Consistent with effective tailored survey methods [34], the recruitment strategy employed multiple methods (i.e., mail, phone, and email) to introduce and remind potential participants about the study. Once the phone interviews were completed, participants were sent an email with instructions on how to access an online survey.

The staff from the grantee organizations were re-contacted approximately 9 months following their first interview (and 9 months after the second interview for the third wave of interviews) to complete another interview and survey describing their experience with delivering A-CRA. If a participant reported that their organization was no longer delivering A-CRA at the proceeding wave, the organizations’ status regarding A-CRA discontinuation was confirmed at the following interview wave.

A-CRA sustainment

Sustainment period and self-reported status

We used administrative data about when the organization stopped receiving the last SAMHSA/CSAT funding (i.e., grant end date) to calculate the time between the funding end date and the study data collection dates. We utilized the number of months since the last grant end date to characterize the time since funding loss and used it as a control variable in the main regression analyses.

Participants were asked at the beginning of the interview whether their organization was currently delivering A-CRA and if the organization had stopped using A-CRA, to report when it had stopped being delivered. A dichotomous indicator of whether the organization was sustaining A-CRA or not was calculated and used to address missing data (see Analytic Plan).

Implementation characteristics

Analytic plan

First, data collected from staff was aggregated to the organization by wave level. The hypothesized outer setting, inner setting, and intervention-related variables were constructed using assessments collected from what we considered the “critical period”, that is, the first 18 months following the end of the implementation support period.

Prior to the longitudinal analyses, each of the 10 elements was standardized to 0–1 and summed for a score ranging from 0 to 10 at the site by wave level. Missing data in elements was imputed by using the means of observed data in each study wave and by self-reported sustainment status. The summed score is our main sustainment outcome in statistical analysis.

As planned [45], we conducted exploratory analysis to find potential clustering of sustainment trajectories. The organizations were grouped into two sustainment patterns according to the cluster analysis. The first pattern is consisted of organizations which reported stopping A-CRA delivery within 12 months at the end of the funding period. The second pattern is consisted of organizations which reported sustaining A-CRA longer than 12 months in the post-funding period (i.e., either stopped A-CRA after the 12th month in the post-funding period or continued sustaining A-CRA without observed stoppage). Since organizations in the first pattern had a shorter observation period (no more than two waves) than organizations in the second pattern, it is necessary to adjust for the mixing patterns in order to unbiasedly estimate the effects of predictors [46, 47]. Lastly, we fitted a set of pattern-mixture longitudinal models to estimate the marginal relationship between the sustainment outcome and each predictor separately. In these pattern-mixture models, we also controlled for time after the funding ended (in months). We used random effects to account for intra-site correlations among the longitudinal measures of sustainment scores. The mixed-effect model also allows for unsynchronized measurement times so that we could consistently estimate the marginal association. To account for multiple comparisons, we applied the step-up methods to adjust p values to control the false discovery rate at the .05 [48].

Sample characteristics

The average age of participants was 41.74 years (SD = 11.57). Sixty-five percent of the sample was female. Thirty-three percent of the sample reported that they were of Hispanic origin. The racial composition reported by respondents was 70% White; 13% Black/African-American; 5% Native American or Alaskan; 2% Asian, Native Hawaiian, or other Pacific Islander; 1% more than one race; and 10% did not endorse a race. Sixty-five percent of the sample reported a master’s level of education, 22% reported a bachelor’s level education, 5% reported an associate’s level of education, 4% reported a doctoral degree, and 4% reported some college. The average number years of substance use disorder counseling experience among the sample was 9.44 (SD = 7.84).

Extent of A-CRA sustainment

Factors associated with A-CRA sustainment

Limitations

Some limitations to this study are important to acknowledge. First, this study has a relatively small sample. We targeted the entire population of organizations that were funded and achieved over a 90% response rate; however, to study the multitude of hypothesized factors and the potential interaction effects, a larger sample would be required; this is a commonly noted challenge in implementation research where the main analyses are often conducted at the organizational level rather than at the patient level [32]. Due to the sample size, we were unable to reliably estimate the effects using all of the hypothesized predictors in one model; therefore, our findings cannot address questions about the relative impact of the different hypothesized predictors compared to one another on the extent of sustainment. For example, we cannot address whether external factors are more or less important than staff experience with or perceptions of A-CRA. Also relevant, the CFIR contains 39 domains, and we did not examine all of these. Moreover, we were not able to examine clinical program outcomes and whether the extent of sustainment limits the impact of A-CRA on youth substance use and related outcomes. We also did not verify some of the A-CRA elements (e.g., frequency and content of supervision); staff may have endorsed higher levels than provided on some of these activities. Future research is warranted on whether varying levels of A-CRA sustainment observed following funding loss influences the effectiveness of A-CRA.