Table 1 Ten core treatment elements of A-CRA sustainment

1. Clinical knowledge of A-CRA

Supervisor and clinician survey

10 multiple-choice items that were derived from the A-CRA treatment manual which covered the theoretical underpinnings and the content of selected A-CRA procedures [53]. During the implementation phase, clinicians and supervisors were required to demonstrate proficiency on these items in order to obtain A-CRA certification.

Each item included five response options, and answers were coded as correct (scored as a “1”) or incorrect (scored as a “0”) for a total score range from 0 to 10. In organizations where more than one respondent delivered A-CRA, scores were averaged across respondents.

2. A-CRA dosage

Clinician interview

One open-ended item that asked how many A-CRA treatment sessions clients were told that they would receive

If all clinicians at an organization endorsed a value within the treatment developers’ recommended range of sessions (i.e., 11–15 sessions), the organization received a score of “1;” otherwise, the organization scored “0”.

3. Clinical staff penetration

Supervisor interview and verification with certification documentation

Supervisors were asked (a) how many clinicians were on staff in the adolescent substance use treatment program and (b) the number and name of those clinicians who were certified to deliver A-CRA

Responses were verified using the treatment developer records that documented who had successfully completed the certification process. Each organization received a score from 0 to 1 based on the proportion of clinicians in the youth treatment program that were certified to deliver A-CRA.

4. Presence of certified A-CRA supervisor(s)

Supervisor interview and verification with certification documentation

Supervisors were asked whether their organization currently had any A-CRA-certified clinical supervisor(s) within their adolescent substance use treatment program

Responses were verified using the treatment developer records that documented who had successfully completed the supervision certification process. Each organization received a score of “1” for having an A-CRA-certified clinical supervisor on staff at the time of the interview or a “0” if no current staff were certified to provide A-CRA clinical supervision.

5. Recommended frequency of clinical supervision

Clinician interview

Clinicians were asked how often they received clinical supervision. Bi-weekly clinical supervision is emphasized as part of the A-CRA training protocol [20].

Organizations where clinicians reported receiving clinical supervision every 2 weeks or more were coded a “1” and organizations where clinicians reported receiving clinical supervision less frequently were coded a “0.” If more than one clinician responded per organization, the responses were averaged for an organization-level score.

6. Recommended clinical supervision content

Clinician interview

Six items based on the rating manual used for coding-recorded supervision sessions during the supervision certification process [54] were used. Example items were, “My supervisor reviews my A-CRA case review report” with “yes” and “no” as response options and “My supervisor role plays the correct way to do a procedure.”

Responses to the six items were scored correct (coded as “1”) or incorrect (coded as “0”). Two items were weighted more due to their relative importance and the average proportion correct was used for a range of 0–1.5. If more than one clinician responded per organization, the responses were averaged for an organization-level score.

7. Review and feedback on treatment sessions

Clinician interview

Clinicians were asked whether their clinical supervisor reviewed recorded treatment sessions and (a) told the clinician what they had done well and (b) gave suggestions about how the clinician could improve treatment delivery. These items were based on the guidelines in the A-CRA treatment manual.

Responses were scored “0” if clinicians reported that their clinical supervisors did not review sessions, “2” if the sessions were reviewed and clinicians were told they had done well, or “4” if session review also included suggestions on how treatment delivery could be improved. If more than one clinician responded per organization, the responses were averaged for an organization-level score.

8. Clinical supervisors’ knowledge of A-CRA training and certification process

Supervisor interview

Supervisors who had reported training staff in the A-CRA model were asked 15 true or false statements about the certification process. These statements were drawn from a list of requirements of the A-CRA certification process and outlined in the A-CRA coding manual [29] that supervisors were required to master as part of the A-CRA treatment certification process. An example question was, “When I decide to pass a clinician on a procedure, it is based on ratings of 1 or more on every component of a procedure.” Clinical supervisors were also asked an open-ended interview question about how many therapy session recordings were needed to reach basic clinical certification.

Organizations where supervisor(s) correctly answered the recording question (i.e., the correct answer was 8 or more recordings) received a “1” and organizations where respondents incorrectly answered this question received a “0.” The score on the certification process questions and recordings question were combined by taking the average of these two measures to form the score on this element. In organizations with more than clinical supervisor, the percentage correct was averaged to create an organization-level value that ranged from 0 to 100%.

9. On-site A-CRA training quality

Supervisor interview and documentation

Supervisors who reported providing A-CRA training at their organization were asked to submit a training agenda. Submission of a training agenda was one of the required elements for the A-CRA Clinical Supervisor Certification process and supervisors were provided feedback until their agendas met specified criteria

Submitted agendas were rated by a member of the A-CRA implementation support team for its coverage of key A-CRA training aspects. Scores ranged from 0 to 1 representing the proportion of A-CRA training aspects covered (e.g., inclusion of half of the training elements would be scored 0.5).

10. Youth penetration

Supervisor interview

Supervisors were asked (a) how many youth clients had received substance use treatment at their organization in the past 6 months and (b) how many youth received A-CRA during that same period.

A proportion of youth clients receiving A-CRA (of those receiving any substance use treatment) was calculated for each organization (range 0–1).