Study context
This project examines A-CRA sustainment among community-based adolescent substance use treatment programs that were awarded funding by the SAMHSA/CSAT between 2006 and 2010. During that period, there were four program cohorts funded by the SAMHSA/CSAT called the “Assertive Adolescent Family Treatment” initiative in 2006, 2007, 2009, and 2010 (e.g., [34]). For these initiatives, grantees were required to utilize A-CRA as the treatment approach. In addition, other SAMHSA funding opportunities were offered during this period including the “Juvenile Drug Court” and “Juvenile Drug Treatment Court,” the “Offender Reentry Project,” and the “Targeted Capacity Expansion” initiatives. For these initiatives, the community-based grantee was required to identify an EBP, and several of the funded organizations selected A-CRA and therefore were included in our study sample. At the time of data collection for this study (Fall 2013), the cohorts varied in the time since the loss of federal funding from approximately 48 months to 1 month.
Sample
Organizations
The study sample was composed of nonprofit treatment providers located across the country representing 27 states. Using recent data available from the National Survey of Substance Abuse Treatment programs [35], 88 % of adolescent treatment is provided in outpatient settings, and 66 % of that treatment is delivered by nonprofit providers, similar to the proposed study sample. The funder also specified that applicants were required to demonstrate (a) program operation in the same geographical location(s) for at least 2 years prior to the proposed project period and (b) compliance with local and state/tribal licensing, accreditation, and certification requirements. These specifications indicated SAMHSA’s intent to build existing substance use treatment program capacity rather than to support new programs. In some cases, the grantee was a non-substance use treatment provider (e.g., school, court, community organization) that partnered with an existing substance use treatment provider in order to deliver the services.
Staff
Clinical supervisors and clinicians at the treatment organizations were recruited to participate. We aimed to enroll at least two individuals from each program that were familiar with A-CRA and had experience providing A-CRA services directly to youth and/or supervising staff providing A-CRA services.
Data collection
Data sources
Primary data was collected from interviews and surveys with organization staff. Implementation data collected during the funding period was also used.
Data collection procedures
We used an administrative dataset that provided the research team with the contact information of the funded treatment organizations and the staff at the treatment organizations who were trained during the 3-year funding period to implement A-CRA. Our recruitment strategy employed multiple methods (i.e., mail, phone, and email) to introduce and remind potential participants about the study, consistent with effective tailored survey methods [36]. We first introduced the study via an email to clinical supervisors and clinical staff. We followed up the introductory email with phone calls to request and/or confirm an interview time. Once an interview was completed, participants were sent an email with instructions on how to access the online survey. Final attempts to contact participants for those that did not respond by email or phone were sent by FedEx.
Measures
A-CRA sustainment
For the analyses reported in this paper, we used staff reports of whether the organization currently utilized the A-CRA treatment or not at the time of the interview. For treatment organizations that reported no longer using A-CRA, we asked when they stopped using it to document the length of time following the federal funding period that the organization utilized A-CRA. From administrative data regarding implementation, we had information about when the organization first received federal funding to deliver A-CRA (i.e., grant start date) and when the organization stopped receiving federal funding (i.e., grant end date) so that we could calculate the time since the federal funding grant end date that the A-CRA treatment was sustained. The funding period was typically 3 years, but some organizations were granted no-cost extensions for 6 to 12 months. We utilized the number of months since grant end date to characterize A-CRA sustainment.
Implementation characteristics
Consistent with several theories of implementation [19, 26], we selected measures that characterized the four main factors theorized to be related to sustainment: inner and outer setting characteristics, implementation factors, and intervention characteristics. As few empirical studies exist in the area of program sustainment, there is an increasing literature on developing valid assessment tools related to the hypothesized constructs. As noted in more detail in the following sections, we attempted to assess these four main constructs using measures that have been recently developed for this purpose (e.g., Program Sustainability Assessment Tool (PSAT)) or have been used in previous studies related to program implementation (e.g., the Steckler and O’Loughlin tools) while ensuring we capture aspects that were particularly relevant to the study context (i.e., adolescent substance use treatment). Data for these measures were collected from staff participants as part of an online survey following a phone interview where they discussed whether their organization was still implementing A-CRA treatment with adolescents.
Setting characteristics
Both Greenhalgh et al. (2004) [21] and Damschroder et al.’s (2009) [19] theories specify that structural characteristics, such as the size and architecture of an organization may influence implementation. We operationalized this by examining the comprehensiveness of the organizations, by evaluating a count of services offered at the organization using a survey question from the National Survey of Substance Abuse Treatment Services [37]. The range on this variable was from 0–17.
“Innovation-fit” has also been nominated as an important factor in implementation [19, 21] and sustainment theories [26]. We operationalized this construct at the organizational level by asking what the primary focus of the organization at the location in question was with the following response options: substance use treatment services, mental health services, a mix of mental health and substance use services (neither is primary), general health care, and other. A binary variable was used that indicated whether staff reported that substance use treatment was the primary service (coded “1”) as compared to all other options (coded “0”).
To meet the goals of developing measurement tools related to program sustainability, a team at Washington University recently conducted a literature review, expert panel, and concept mapping to develop the Program Sustainability Assessment Tool [38, 39] which consists of eight subscales: communication (e.g., “The program has communication strategies to secure and maintain public support”), funding stability (e.g., “The program has a combination of stable and flexible funding”), organizational capacity (e.g., “The program is well integrated into the operations of the organization”), partnerships (e.g., “Diverse community organizations are invested in the success of the program”), political support (e.g., “Political champions advocate for the program”), program adaptation (e.g., “The program adapts strategies as needed”), program evaluation (e.g., “Evaluation results inform program planning and implementation”), and strategic planning (e.g., “The program plans for future resource needs”). Each subscale consisted of five items and the alphas among our samples ranged from 0.84–0.95. Responses ranged from “to little or no extent” (scored as a “1”) to “to a great extent” (scored as a “7”), we summed responses (for a range of 5–35) and calculated mean values for each scale. This assessment tool encompasses both outer and inner setting characteristics.
Implementation characteristics
The extent to which an organization has consistently implemented the intervention during the funding period is likely to influence how well it can be sustained post the initial support period [26, 40]. In order to examine this, we utilized data collected during the funding period that included (1) the number of adolescents that received A-CRA during the funding period, (2) the number of clinical supervisors certified in A-CRA and still employed at each organization at grant end, and (3) the number of clinicians certified in A-CRA at each organization and still employed at the end of the grant period. More specifically, the number of participating adolescents was based on the number of adolescents who completed a baseline interview. The number of staff who completed the A-CRA clinical supervisor and clinician certification process and were still employed at the end of the grant period at each organization was drawn from data collected by the technical assistance group that provided the A-CRA training and monitored implementation for the SAMHSA/CSAT. We intentionally selected assessments of the number certified staff employed at grant end, rather than the number of staff trained during the funding period or staff turnover, to more accurately account for the organizational resources at grant end that may influence sustainment.
Intervention characteristics
Several theories suggest that perceptions of a particular innovation in terms of its ease of use and benefit over alternative options will influence its adoption, use, and presumably longer-term sustainment [19–21, 26]. We included assessments of staff perceptions of A-CRA’s complexity (e.g., “A-CRA is hard for therapists to understand”; using a scale of five items) and relative advantage (e.g., “A-CRA is more effective in reducing substance use by clients than our current treatment practices”; using a scale of four items) using items from Steckler et al. [41] (alphas = 0.88 and 0.83, respectively). We included staff perceptions of implementation difficulty (e.g., “Recruiting staff/participants for A-CRA is difficult”) and perceived success (e.g., “A-CRA had an impact on participants”) using five-item scales developed from O’Loughlin et al. [16] (alphas = 0.57 and 0.91, respectively). All of these survey items had response options on a five-point scale ranging from “strongly disagree” (scored as a “1”) to “strongly agree” (scored as a “5”) for a summed score range of 4–20 (for the relative advantage scale) or 5–25 (for the complexity, difficulty, and success scales).
Analytic plan
We conducted a set of discrete-time survival analyses. The self-reported termination of A-CRA was defined as the event occurrence in the survival analysis. The time-to-event was the number of months between the end of federal funding and the time an organization stopped implementing A-CRA. The time-to-event was right censored by our interview time, and the censoring was independent of the time-to-event. We first fitted a Kaplan-Meier survival curve (i.e., the probability that an organization sustains A-CRA longer than a given length of time). Next, we examined the marginal proportional hazards (i.e., the ratio in hazards between two levels of a binary factor or a unit change of a continuous predictor, where the hazard is the conditional probability that an organization ceases to sustain A-CRA at the end of a month given it sustains A-CRA at the beginning of the month) for a list of hypothesized factors. Organization-level values for the hypothesized outer, inner setting, implementation quality, and intervention characteristics were computed by taking the mean (for scale variables) or mode value across the participant responses at an organization. Missing data at the organizational-level was imputed by mean imputation conditioned on the sustainment statuses. The discrete-time marginal proportional hazard was estimated by a logistic regression following the standard approach to recode the time-to-event data to binary outcomes [42, 43]. To account for multiple comparisons, we apply the step-up methods to adjust p values to control the false discovery rate at the .05 and .10 levels [42].