- Open Access
- Open Peer Review
Pay-for-performance as a cost-effective implementation strategy: results from a cluster randomized trial
Implementation Sciencevolume 13, Article number: 92 (2018)
Pay-for-performance (P4P) has been recommended as a promising strategy to improve implementation of high-quality care. This study examined the incremental cost-effectiveness of a P4P strategy found to be highly effective in improving the implementation and effectiveness of the Adolescent Community Reinforcement Approach (A-CRA), an evidence-based treatment (EBT) for adolescent substance use disorders (SUDs).
Building on a $30 million national initiative to implement A-CRA in SUD treatment settings, urn randomization was used to assign 29 organizations and their 105 therapists and 1173 patients to one of two conditions (implementation-as-usual (IAU) control condition or IAU+P4P experimental condition). It was not possible to blind organizations, therapists, or all research staff to condition assignment. All treatment organizations and their therapists received a multifaceted implementation strategy. In addition to those IAU strategies, therapists in the IAU+P4P condition received US $50 for each month that they demonstrated competence in treatment delivery (A-CRA competence) and US $200 for each patient who received a specified number of treatment procedures and sessions found to be associated with significantly improved patient outcomes (target A-CRA). Incremental cost-effectiveness ratios (ICERs), which represent the difference between the two conditions in average cost per treatment organization divided by the corresponding average difference in effectiveness per organization, and quality-adjusted life years (QALYs) were the primary outcomes.
At trial completion, 15 organizations were randomized to the IAU condition and 14 organizations were randomized to the IAU+P4P condition. Data from all 29 organizations were analyzed. Cluster-level analyses suggested the P4P strategy led to significantly higher average total costs compared to the IAU control condition, yet this average increase of 5% resulted in a 116% increase in the average number of months therapists demonstrated competence in treatment delivery (ICER = $333), a 325% increase in the average number of patients who received the targeted dosage of treatment (ICER = $453), and a 325% increase in the number of days of abstinence per patient in treatment (ICER = $8.134). Further supporting P4P as a cost-effective implementation strategy, the cost per QALY was only $8681 (95% confidence interval $1191–$16,171).
This study provides experimental evidence supporting P4P as a cost-effective implementation strategy.
Illicit drug use has been estimated to cost the USA an approximately $200 billion annually as a result of direct and indirect costs associated with crime, health, and productivity , which surpasses the estimated costs for other chronic health issues like diabetes  and obesity . Although a number of evidence-based treatments (EBTs) to address substance use disorders (SUDs) have been developed , implementation of EBTs for SUDs within practice settings has been limited [5, 6]. This gap has lowered the return-on-investment of both research-related dollars spent developing EBTs and service-related dollars spent providing treatment to individuals and families that are in need of SUD services.
Beyond the relatively low levels of EBT implementation found within the SUD treatment field [5, 6], EBT implementation has been identified as a significant challenge for other areas of health [7,8,9,10]. In 2007, as part of a broad effort to improve the quality of care delivered within the USA (including greater implementation of EBTs), the Institute of Medicine recommended pay-for-performance (P4P) as a promising strategy to improve implementation of high-quality care . This recommendation, combined with the limited empirical research evidence supporting P4P as a method to improve quality of care, motivated an experimental test of the effectiveness and cost-effectiveness of P4P as a strategy to improve the implementation and effectiveness of the Adolescent Community Reinforcement Approach (A-CRA)—an EBT shown to be effective and cost-effective in treating SUDs for adolescents [12,13,14,15,16,17,18]. The rationale for using a cluster randomized trial design was that primary interest was to examine P4P as an organizational-level strategy, as well as that validity threats are possible from the randomization of patients within therapists (e.g., contamination) or of therapists within treatment organizations (e.g., compensatory rivalry and resentful demoralization).
Main effectiveness findings suggested P4P had a direct impact on improving the two primary measures of implementation (one measured at the therapist level and one measured at the patient level), as well as an indirect impact on patient substance use . That is, relative to therapists in the implementation-as-usual (IAU) condition , therapists in the IAU+P4P condition were significantly more likely to demonstrate monthly A-CRA competence (event rate ratio = 2.24). Additionally, relative to patients treated in the IAU condition, patients treated in the IAU+P4P condition were significantly more likely to receive an empirically supported level of A-CRA treatment exposure (target A-CRA; odds ratio = 5.19), which was positively associated with post-treatment recovery status (odds ratio = 1.91). According to a recent systematic review of quality improvement, implementation, and dissemination strategies to improve mental health care for children and adolescents , Garner and colleagues  findings provided “The strongest evidence in the review” for their key question (i.e., What is the effectiveness of quality improvement, implementation, and dissemination strategies).
Consistent with the original study protocol , recommendations for evaluating the cost-effectiveness of P4P [23, 24], and the importance of SUD research addressing cost issues , the primary purpose of the current study was to report on the incremental cost-effectiveness of P4P as a discrete implementation strategy. We hypothesized that relative to the IAU condition, the IAU+P4P condition would have significantly higher total implementation costs (hypothesis 1), significantly lower incremental cost-effectiveness ratio (ICER) per therapist month of A-CRA competence (hypothesis 2), significantly lower ICER per patients receiving target A-CRA (hypothesis 3), and significantly lower ICER per day of abstinence at follow-up (hypothesis 4). Additionally, we hypothesized (hypothesis 5) that the cost per quality-adjusted life year (QALY) would be less than the USA per capita gross domestic product, which is the standard recommended by the World Health Organization for an intervention to be highly cost-effective .
All study procedures were conducted under Institutional Review Board approval. The study protocol paper for this cluster randomized trial was published in Implementation Science in 2010 [19, 27, 28]. The current article was written in accordance with the 2012 CONSORT guidelines for cluster randomized trials , which were published 1 month after publication of the effectiveness paper . Additionally, in appreciation of the importance of implementation strategy specification, Table 1 specifies each of the discrete implementation strategies included as part of the IAU condition (see A–J of Table 1) and the P4P strategy (see K of Table 1) included as an adjunct to the IAU condition. Table 1 was developed to be consistent with the list of discrete implementation strategies compiled by Powell and colleagues , as well as with Proctor and colleagues  recommendations regarding implementation strategy specification.
Twenty-nine community-based treatment organizations (97% participation rate) were assigned via urn randomization to either the control IAU condition (n = 15) or the experimental IAU+P4P condition (n = 14). It was not possible to blind organizations, therapists, or all research staff to condition assignment. As part of a national EBT implementation initiative conducted between October 2006 and October 2010 by the Substance Abuse and Mental Health Services Administration’s Center for Substance Abuse Treatment (SAMHSA/CSAT) [4, 20], treatment organizations in both conditions received similar levels of funding to implement A-CRA. Additionally, funded by SAMHSA/CSAT as part of a separate contract to A-CRA model developers, treatment organizations in both conditions received a multifaceted implementation strategy to assist with their implementation of A-CRA (see A–J of Table 1). The level of standardization provided by the SAMHSA/CSAT initiative provided a unique opportunity to test the incremental effectiveness and cost-effectiveness of P4P as a discrete implementation strategy (see K of Table 1).
In September 2008, via funding from the National Institute on Alcohol Abuse and Alcoholism, Garner and colleagues initiated the reinforcing therapist performance (RTP) experiment . With enrollment from November 17, 2008, through January 12, 2009, this cluster randomized experiment involved 29 treatment organizations, 105 therapists, and 1173 patients. As noted above, IAU for the current cluster randomized trial included a multifaceted implementation strategy (A–J of Table 1) . In addition to the discrete implementation strategies included in the IAU condition, therapists who worked at one of the 14 treatment organizations randomized to the IAU+P4P condition and who provided voluntary written consent to participate in the study were offered the opportunity to earn monetary bonuses for achievement of two predefined treatment implementation performance measures (i.e., P4P; see K of Table 1). First, therapists could earn $200 for each of their patients who received target A-CRA (defined as at least 10 of 12 specific A-CRA procedures delivered within the first 14 weeks of treatment and in no fewer than 7 sessions). Several studies found target A-CRA to be significantly associated with decreased post-treatment substance use [13, 19, 22]. Second, to reinforce delivery of treatment procedures with high quality, therapists could earn $50 for each month that they demonstrated A-CRA competence (defined as competent delivery of all components of at least one A-CRA treatment procedure during the same treatment session). Demonstration of target A-CRA and A-CRA competence was determined by a trained rater based on review of session audio recordings. Overall, there was a 95% level of agreement between the study’s primary rater and monthly fidelity ratings conducted by a second rater blinded to study condition . Each month, participating therapists working at one of the treatment organizations assigned to the IAU+P4P condition received emails regarding any monthly bonuses earned. Participating therapists received their bonuses monthly either through check or direct deposit. Conducted from the perspective of the healthcare system, the cost-effectiveness study focused on costs incurred by each participating treatment organization.
There were three categories of system costs; all used 2010 US dollars. The first category, termed Training & Coaching Costs, included costs related to the training, coaching, certification, and fidelity monitoring of therapists delivering A-CRA treatment sessions to patients as part of the SAMHSA/CSAT initiative. These costs were estimated for each treatment organization by multiplying the treatment organization’s total number of participating therapists by $5717 (i.e., the overall average cost per therapist involved according to administrative training records).
The second category, termed Treatment Costs, included costs related to the delivery of A-CRA treatment sessions to patients. Treatment costs were calculated for each treatment organization by multiplying the treatment organization’s total number of treatment sessions by $125.26, which represents the inflation-adjusted (adjusted to 2010 dollars) average cost of a counseling hour as reported by the Alcohol and Drug Services Cost Study , a nationally representative study.
The third category, termed P4P Costs, included costs related to the monetary bonuses paid to therapists and costs associated with the administration of the P4P methods (e.g., staff time to determine achievement of the two performance measures and to issue monetary bonuses to therapists). Each treatment organization’s P4P cost associated with target A-CRA was computed by summing each treatment organization’s total amount of target A-CRA bonuses ($200 per target A-CRA) plus the product of the number of patients who were identified as having met target A-CRA eligibility criteria times $44.21 (i.e., the average fully loaded cost [including fringe and indirect costs] of 1.5 h of staff time to review each session’s audio recordings in order to verify each patient’s target A-CRA). Similarly, the treatment organization’s P4P cost associated with A-CRA competence was calculated by adding the amount of A-CRA competence bonuses ($50 per A-CRA competence) plus the product of the number of months therapists met the monthly eligibility criteria of recording at least 80% of their face-to-face A-CRA treatment sessions times $29.47 (i.e., the average fully loaded cost [including fringe and indirect costs] of 1.0 h of staff time to review a randomly selected session audio recording for A-CRA competence). Total P4P costs were computed for each treatment organization by summing all of their respective target A-CRA and A-CRA competence costs.
Effectiveness was assessed by implementation measures and health outcome measures. The implementation measures included the number of months therapists demonstrated A-CRA competence and the number of patients receiving target A-CRA. The first health outcome measure was the percent of days patients were abstinent from alcohol and other drugs at the 6-month follow-up. Data from both of the study conditions were combined to model the impact on patient abstinence as a function of receipt of target A-CRA. The sample was all patients (n = 600) with known outcome status at the 6-month follow-up assessment. Follow-up rates were similar (χ2 = 0.12, p = .73) in patients with (63%) and without target A-CRA (64%). Results indicated that patients who received target A-CRA reported a significantly greater percentage of days of abstinence from alcohol and drug use (coefficient = 0.153 [SE = 0.076], p < .05). Converting by 365 to years gave the added person-years of abstinence. The second outcome measure was the estimated number of quality-adjusted life years (QALYs) gained. Our QALY measure was calculated using the results of a study by Daley and colleagues (2005) , which extended the QALY measure to substance abuse. In the Daley et al. study, participants gained 0.19 QALYs over 1 year while their rates of abstinence improved by 55.6 percentage points, indicating that each percentage point improvement in abstinence (and concurrent improvements in other dimensions) over 1 year was associated with a gain of 0.00342 QALYs. This implies a disability weight per year for substance use overall of 0.342 (i.e., 0.00342/1%), with this weight measuring the amount lost from perfect health. Additionally, given that Daley and colleagues’ study represents only a single investigation of the overall QALY burden of substance abuse, we also calculated QALY using the results of a global study that estimated disability weights separately by substance and severity . Among the five relevant substances, the median weights (across severity levels) ranged from 0.153 for cannabis to 0.516 for heroin, with the midpoint value (0.335) similar to the value derived using the Daley et al. study (0.342).
Consistent with standard economic theory , we assessed the cost-effectiveness of the P4P methods using incremental cost-effectiveness ratios (ICER). For each comparison, the ICER is the difference between the two arms in average yearly cost per treatment organization divided by the corresponding average difference in effectiveness per organization over 1 year. The comparison between the IAU condition and the IAU+P4P condition calculated the costs and impact of adding P4P onto the existing SAMHSA/CSAT program. We estimated the confidence interval (CI) of the cost per QALY by treating target A-CRA by treatment organization as a normal variable. The lower (most favorable) bound for the CI used the highest disability weight (0.516), where treatment would confer the greatest benefit. The upper (least favorable) bound for the CI used the lowest disability weight (0.153), where treatment would confer the least benefit. Finally, a preliminary cost-benefit analysis of IAU + P4P vs. IAU valued each QALY at the 2010 U.S. Gross National Income per capita of $48,950 .
Figure 1 shows the flow of treatment organizations, therapists, and patients through the experiment. No adverse events were reported. Table 2 shows implementation outcomes and costs by condition. Relative to treatment organizations assigned to the IAU condition, treatment organizations assigned to the IAU+P4P condition had a significantly higher average number of months that therapists demonstrated A-CRA competence (IAU = 8.62; IAU+P4P = 18.64; p < 0.001; 116% increase) and a significantly higher average number of patients who received target A-CRA (IAU = 2.27; IAU+P4P = 9.64; p < 0.001; 325% increase).
In terms of costs, the IAU+P4P condition had a significantly higher average Training & Coaching Cost (IAU = $18,844; IAU+P4P = $23,483; p < 0.001; 24.6% increase), but also a significantly lower average treatment cost (IAU = $44,073; IAU+P4P = $39,838; p < 0.001; 9.6% decrease) per organization. Additionally, when these costs were combined with P4P costs, which averaged $2935 (SD = $3103) for the treatment organizations assigned to the IAU+P4P condition, the total cost per organization was significantly higher for the IAU+P4P condition (IAU = $62,917; IAU+P4P = $66,256; p = .034; 5% increase).
Treatment costs represented the largest percentage of total cost for both conditions (70% for IAU and 58% for IAU+P4P), with Training & Coaching Costs representing the second largest percentage of total cost (30% for IAU and 34% for IAU+P4P). Within the IAU+P4P condition, the average P4P cost represented only 3% of the average total cost. The average amount of monetary bonuses paid per treatment organization was $1076 (SD = $1013) for A-CRA competence and $1859 (SD = $2273) for target A-CRA. Additionally, administration costs added 30% to the costs for the P4P incentives.
Table 3 presents the incremental costs, incremental effectiveness, and incremental cost-effectiveness ratios (ICERs). Relative to the IAU condition, the ICERs for therapist months of A-CRA competence, patients receiving target A-CRA, and days of abstinence per patient for the IAU+P4P condition were $333, $453, and $8.134, respectively. Table 3 also presents the incremental cost per QALY for patients, which was $8681 per QALY for the IAU+P4P vs. IAU. Using the range of disability weights from different substances , the ICER could range from $5754 (derived from the heroin disability) to $19,405 (derived from the cannabis disability). The value of $8681 is substantially below the 2010 US per capita GDP ($48,950) .
P4P has been recommended as a promising strategy to improve the quality of care delivered within the United States ; however, empirical support for P4P based on rigorous experimental research has been limited to date [24, 37,38,39,40,41]. In an effort to address this gap, the current study examined the incremental cost-effectiveness of a P4P implementation strategy previously shown to be highly effective at improving the implementation and effectiveness of an EBT for adolescent SUDs .
Overall, study results suggest P4P can be a cost-effective implementation strategy. Supporting hypothesis 1, the addition of P4P led to a significantly higher average total cost relative to the IAU condition. Supporting hypotheses 2 through 4, despite the increase in average total cost, the IAU+P4P condition was found to be more cost-effective than the IAU condition regarding therapist months of A-CRA competence, as patient’s receipt of target A-CRA, and patient days of abstinence at follow-up. More specifically, the average increase of 5% in total costs resulted in a 116% increase in the average number of therapist months of A-CRA competence (ICER = 333), a 325% increase in the average number of patients who received target A-CRA (ICER = $453), and a 325% increase in the number of days of abstinence per patient in treatment (ICER = $8.134). According to a recent systematic review of economic evaluations of P4P in health care, only two full economic evaluations of P4P approaches had been evaluated through randomized trials . The first study found that a 16% increase in cost raised the receipt of flu vaccination by 9% , while the second study found that a 971% increase in cost raised the number of referrals to a tobacco quitline by 236% . Thus, results of the present study are unique as they represent the first known P4P experiment to find that the percentage increase in implementation outcomes was greater than the percentage increase in costs. Finally, supporting hypothesis 5, the cost per QALY ($8655) was more favorable (i.e., lower) than two benchmarks—the comparably calculated pooled average from Dennis and colleagues’ interventions ($11,641)  and the USA per capita GDP ($46,616) , the latter of which is the standard recommended by the World Health Organization for an intervention to be highly cost-effective .
Our use of the perspective of the healthcare system is increasingly recommended for its relevance to resource constraints of many decision makers . An analysis from the societal perspective, the other option, would require complete societal costs, such as value of participants’ time and transportation expense. Given our study lacked direct information about such societal costs, our costs are underestimated. However, because therapists encouraged adolescents to engage in pro-social activities with low or no out-of-pocket costs to the adolescents (e.g., high-school event), as well as that the opportunity costs of the time of adolescents with substance abuse disorders are likely minimal, this omission is assumed to be minor.
Valuing each QALY at the per capita GDP of the USA based on a human capital approach , we conducted a preliminary benefit-cost analysis of IAU+P4P vs. IAU. It gave a ratio of 5.4 with a range, based on the disability weights for alternative substances, of 2.4 to 8.1, with all values above the cutoff of 1.0. These preliminary benefit-cost ratios are likely conservative, as we did not include possible reductions in crime nor the value that adolescents might place on participating in encouraged activities, such as watching high-school sports with their friends.
Strengths and limitations
Important strengths of the current study include the use of a randomized design, independent verification of behaviors that resulted in receipt of P4P incentives, the inclusion of administration costs as part of the P4P costs, and links to QALY outcome measures. Important limitations of the current study, however, were reliance on patient self-report of substance use and substance-related problems, as well as the lack of biometric data (e.g., breathalyzer or urine test results).
Implications and directions for future research
McCarty and colleagues  identified five priorities for policy research on treatment for substance use disorders, which included: (a) organization and delivery of care, (b) quality of care, (c) EBTs, (d) access to care, and (e) financing, costs, and value of care. With regard to “financing, costs, and value of care,” McCarty and colleagues , who specifically highlighted P4P as an emerging strategy, noted “policy makers are often more attentive to cost-effectiveness estimates than overall spending estimates.” Thus, one key implication of the current study is it provides estimates of the cost-effectiveness of P4P strategies. Given the limited number of studies that have experimentally tested the cost-effectiveness of P4P , in addition to having important implications for the SUD treatment field, current findings also may have important implications for other areas of healthcare that have struggled to identify effective and cost-effective models of P4P [24, 38, 39, 47, 48]. Nonetheless, additional estimates regarding the cost-effectiveness of P4P remain needed.
With regard to EBTs as an area of priority for policy research, McCarty and colleagues  noted the need to identify strategies to help facilitate use of EBTs. Thus, a second implication of the current study is it provides strong evidence that P4P can be a cost-effective strategy to help improve implementation of EBTs. This is important given identification of effective and cost-effective implementation strategies is an important topic for several areas of health . Given the limited number of effective and cost-effective implementation strategies, additional research examining the effectiveness and cost-effectiveness of P4P as a discrete strategy or component of a multifaceted implementation strategy is warranted.
Finally, with regard to research on quality of care, McCarty and colleagues  noted “the most critical policy issues for research on quality of care pertain to the link between performance measures and evidence of enhanced treatment outcomes.” Thus, given the patient-level performance measure used in this study (target A-CRA) has been shown to be a significant predictor of patient outcomes [13, 19, 22], a third implication of the current study is that it supports focusing implementation strategies on improving evidenced-based measures of implementation , which are distinguished from other “implementation outcomes” that have not been shown to be associated with improved treatment outcomes (e.g., acceptability, appropriateness, feasibility).
The present study provides experimental evidence supporting P4P as a cost-effective implementation strategy to improve implementation of EBTs for SUD treatment. The precise reasons why our P4P study had a large effect, in contrast to other large P4P studies that found P4P to have modest  to no impact , are not known; however, consistent with research highlighting the importance of the design elements of P4P [38, 48, 51,52,53], we believe at least three design issues were important. First, in contrast to P4P designs that focused on numerous performance measures, many of which had little room for improvement due to base rates above 90% , we designed the current study to focus on two clinically relevant performance measures that had considerable room for improvement due to their relatively low base rates [19, 22]. Second, rather than the P4P incentives being directed toward the organization, which is how most P4P studies to date have been designed , our study design directed the P4P incentives at the individual therapists. In addition to being consistent with our prior research that had used therapist-directed P4P incentives to improve client retention in SUD continuing care , this design choice is consistent with Van Herck and colleagues’ conclusion that “targeting the individual has generally better effects than not to do so” . Third, in contrast to P4P designs where P4P incentives were provided on a relatively infrequent annual basis [39, 50], our study design provided P4P incentives on a relatively frequent monthly basis, which is consistent with key principles of operant conditioning [55, 56]. Future research is clearly needed, including the optimal bonus amounts and the extent to which effects are sustained. Our hope, however, is that the current P4P experiment will not only help inform and improve P4P research, but also will help inform and improve implementation research focused on identifying effective and cost-effective strategies for improving implementation of EBTs in practice settings.
Adolescent community reinforcement approach
Center for Substance Abuse Treatment
Incremental cost-effectiveness ratios
Quality-adjusted life years
Reinforcing therapist performance
Substance Abuse and Mental Health Services Administration
Substance use disorders
National Drug Intelligence Center. The economic impact of illicit drug use on American Society. Washington, DC: United States Department of Justice; 2011.
Centers for Disease Control and Prevention. National diabetes fact sheet: national estimates and general information on diabetes and prediabetes in the United States. Atlanta, GA: U.S. Department of Health and Human Services, centers for disease control and Prevention; 2011.
Finkelstein EA, Trogdon JG, Cohen JW, Dietz W. Annual medical spending attributable to obesity: payer-and service-specific estimates. Health Aff (Millwood). 2009;28(5):w822–31.
Substance Abuse and Mental Health Services Administration. NREPP: SAMHSA's National Registry of Evidence-based Programs and Practices. 2014; https://nrepp.samhsa.gov/landing.aspx. Accessed 27 Jan 2014.
Garner BR. Research on the diffusion of evidence-based treatments within substance abuse treatment: a systematic review. J Subst Abus Treat. 2009;36(4):376–99.
Institute of Medicine. Bridging the gap between practice and research: forging partnerships with community-based drug and alcohol treatment. Washington, DC: National Academy Press; 1998.
Institute of Medicine. Crossing the quality chasm: a new health system for the 21st century. Washington, DC: National Academy Press; 2001.
Institute of Medicine. Improving the quality of health care for mental and substance use conditions: quality chasm series. Washington, DC: National Academy Press; 2006.
Department of Health and Human Services. Dissemination and implementation research in health (R01). PAR-16-238: NIH Funding Opportunities and Notices. Washington, DC: Department of Health and Human Services; 2017. https://grants.nih.gov/grants/guide/pa-files/PAR-16-238.html.
Hogan MF. The President's new freedom commission: recommendations to transform mental health care in America. Psychiatr Serv. 2003;54(11):1467–74.
Institute of Medicine. Rewarding provider performance: aligning incentives in Medicare. Washington, DC: National Academy Press; 2007.
Dennis M, Godley SH, Diamond G, et al. The Cannabis Youth Treatment (CYT) Study: main findings from two randomized trials. J Subst Abus Treat. 2004;27(3):197–213.
Garner BR, Godley SH, Funk RR, Dennis ML, Smith JE, Godley MD. Exposure to adolescent community reinforcement approach treatment procedures as a mediator of the relationship between adolescent substance abuse treatment retention and outcome. J Subst Abus Treat. 2009;36(3):252–64.
Godley SH, Hedges K, Hunter B. Gender and racial differences in treatment process and outcome among participants in the adolescent community reinforcement approach. Psychol Addict Behav. 2011;25(1):143–54.
Slesnick N, Prestopnik JL, Meyers RJ, Glassman M. Treatment outcome for street-living, homeless youth. Addict Behav. 2007;32(6):1237–51.
Godley SH, Hunter BD, Fernandez-Artamendi S, Smith JE, Meyers RJ, Godley MD. A comparison of treatment outcomes for adolescent community reinforcement approach participants with and without co-occurring problems. J Subst Abus Treat. 2014;46(4):463–71.
Henderson CE, Wevodau AL, Henderson SE, et al. An independent replication of the adolescent-community reinforcement approach with justice-involved youth. Am J Addict. 2016;25(3):233–40.
Godley MD, Passetti LL, Subramaniam GA, Funk RR, Smith JE, Meyers RJ. Adolescent community reinforcement approach implementation and treatment outcomes for youth with opioid problem use. Drug Alcohol Depend. 2017;174:9–16.
Garner BR, Godley SH, Dennis ML, Hunter BD, Bair CM, Godley MD. Using pay for performance to improve treatment implementation for adolescent substance use disorders: results from a cluster randomized trial. Arch Pediatr Adolesc Med. 2012;166(10):938–44.
Godley SH, Garner BR, Smith JE, Meyers RJ, Godley MD. A large-scale dissemination and implementation model for evidence-based treatment and continuing care. Clin Psychol (New York). 2011;18(1):67–83.
Forman-Hoffman VL, Middleton JC, McKeeman JL, et al. Quality improvement, implementation, and dissemination strategies to improve mental health care for children and adolescents: a systematic review. Implement Sci. 2017;12(1):93.
Garner BR, Godley SH, Dennis ML, Godley MD, Shepard DS. The reinforcing therapist performance (RTP) experiment: study protocol for a cluster randomized trial. Implement Sci. 2010;5:5.
Emmert M, Eijkenaar F, Kemter H, Esslinger AS, Schoffski O. Economic evaluation of pay-for-performance in health care: a systematic review. Eur J Health Econ. 2012;13(6):755–67.
Greene SE, Nash DB. Pay for performance: an overview of the literature. Am J Med Qual. 2009;24(2):140–63.
McCarty D, McConnell KJ, Schmidt LA. Priorities for policy research on treatments for alcohol and drug use disorders. J Subst Abus Treat. 2010;39(2):87–95.
World Health Organization (WHO). Cost-effectiveness thresholds. 2013; http://www.who.int/bulletin/volumes/94/12/15-164418/en/. Accessed 2 Sept 2013.
Garner BR, Godley MD, Funk RR, Lee MT, Garnick DW. The Washington circle continuity of care performance measure: predictive validity with adolescents discharged from residential treatment. J Subst Abus Treat. 2010;38(1):3–11.
Campbell MK, Elbourne DR, Altman DG. CONSORT statement: extension to cluster randomised trials. BMJ. 2004;328(7441):702–8.
Campbell MK, Piaggio G, Elbourne DR, Altman DG. Consort 2010 statement: extension to cluster randomised trials. BMJ. 2012;345:e5661.
Powell BJ, McMillen JC, Proctor EK, et al. A compilation of strategies for implementing clinical innovations in health and mental health. Med Care Res Rev. 2012;69(2):123–57.
Proctor EK, Powell BJ, McMillen JC. Implementation strategies: recommendations for specifying and reporting. Implement Sci. 2013;8(1):139.
Shepard DS, Beaston-Blaakman A, Horgan CM. The ADSS cost study: costs of substance abuse treatment in the specialty sector. DHHS publication no. SMA 03-3762. Rockville, MD: Substance Abuse and Mental Health Services Administration, Office of Applied Studies; 2003.
Daley M, Shepard DS, Bury-Maynard D. Changes in quality of life for pregnant women in substance user treatment: developing a quality of life index for the addictions. Subst Use Misuse. 2005;40(3):375–94.
Salomon JA, Haagsma JA, Davis A, et al. Disability weights for the Global Burden of Disease 2013 study. Lancet Glob Health. 2015;3(11):e712–23.
Drummond MF, Sculpher MJ, Torrance GW, O'Brien BJ, Stoddart GL. Methods for the economic evaluation of health care programmes. 3rd ed. Oxford: Oxford University Press; 2005.
World Bank Group. United States GDP per capita. 2010; http://data.worldbank.org/country/united-states. Accessed 2 Sept 2013.
Bremer RW, Scholle SH, Keyser D, Houtsinger JV, Pincus HA. Pay for performance in behavioral health. Psychiatr Serv. 2008;59(12):1419–29.
Eijkenaar F, Emmert M, Scheppach M, Schoffski O. Effects of pay for performance in health care: a systematic review of systematic reviews. Health Policy. 2013;110(2–3):115–30.
Jha AK, Joynt KE, Orav EJ, Epstein AM. The long-term effect of premier pay for performance on patient outcomes. N Engl J Med. 2012;366(17):1606–15.
Ryan AM, Blustein J, Casalino LP. Medicare’s flagship test of pay-for-performance did not spur more rapid quality improvement among low-performing hospitals. Health Aff (Millwood). 2012;31(4):797–805.
Scheffler RM. Pay for performance (P4P) programs in health services: What is the evidence. World Health Report Background Paper, No 31. Geneva: World Health Organization; 2010.
Kouides RW, Bennett NM, Lewis B, Cappuccio JD, Barker WH, LaForce FM. Performance-based physician reimbursement and influenza immunization rates in the elderly. The Primary-Care Physicians of Monroe County. Am J Prev Med. 1998;14(2):89–95.
An LC, Bluhm JH, Foldes SS, et al. A randomized trial of a pay-for-performance program targeting clinician referral to a state tobacco quitline. Arch Intern Med. 2008;168(18):1993–9.
World Bank Group. United States GDP per capita. 2013; http://data.worldbank.org/country/united-states. Accessed 2 Sept 2013.
Neumann PJ, Sanders GD, Russell LB, Siegel JE, Ganiats TG. Cost-effectiveness in health and medicine. 2nd ed. New York: Oxford University Press; 2017.
Drummond MF, O’Brien JO, Stoddart GL, Torrance GW. Methods for the economic evaluation of health care programmes. 2nd ed. New York: Oxford University Press; 1997.
Ryan A, Blustein J. Making the best of hospital pay for performance. N Engl J Med. 2012;366(17):1557–9.
Van Herck P, De Smedt D, Annemans L, Remmen R, Rosenthal MB, Sermeus W. Systematic review: effects, design choices, and context of pay-for-performance in health care. BMC Health Serv Res. 2010;10:247.
Garner BR, Hunter SB, Funk RR, Griffin BA, Godley SH. Toward evidence-based measures of implementation: examining the relationship between implementation outcomes and client outcomes. J Subst Abus Treat. 2016;67:15–21.
Lindenauer PK, Remus D, Roman S, et al. Public reporting and pay for performance in hospital quality improvement. N Engl J Med. 2007;356(5):486–96.
Eijkenaar F. Pay for performance in health care: an international overview of initiatives. Med Care Res Rev. 2012;69(3):251–76.
Jha AK. Time to get serious about pay for performance. JAMA. 2013;309(4):347–8.
Mehrotra A, Sorbero ME, Damberg CL. Using the lessons of behavioral economics to design more effective pay-for-performance programs. Am J Manag Care. 2010;16(7):497–503.
Shepard DS, Calabro JA, Love CT, McKay JR, Tetreault J, Yeom HS. Counselor incentives to improve client retention in an outpatient substance abuse aftercare program. Admin Pol Ment Health. 2006;33(6):629–35.
Skinner BF. The behavior of organisms: an experimental analysis. New York: Appleton-Century; 1938.
Staddon JE, Cerutti DT. Operant conditioning. Annu Rev Psychol. 2003;54:115–44.
Squires DD, Gumbley SJ, Storti SA. Training substance abuse treatment organizations to adopt evidence-based practices: the addiction technology transfer center of New England science to service laboratory. J Subst Abus Treat. 2008;34(3):293–301.
Davis D, Barrington T, Phoenix U, et al. Evaluation and technical assistance for successful HIV program delivery. AIDS Educ Prev. 2000;12(Suppl A 5):115–25.
Wandersman A, Chien VH, Katz J. Toward an evidence-based system for innovation support for implementing innovations with quality: tools, training, technical assistance, and quality assurance/quality improvement. Am J Community Psychol. 2012;50(3–4):445–59.
Chaple M, Sacks S. The impact of technical assistance and implementation support on program capacity to deliver integrated services. J Behav Health Serv Res. 2016;43(1):3–17.
Godley SH, Meyers RJ, Smith JE, et al. The Adolescent Community Reinforcement Approach (ACRA) for adolescent cannabis users, Cannabis Youth Treatment Manual Series Vol. 4. Rockville, MD: Center for Subst Abus Treatment, Subst Abus and Mental Health Services Administration; 2001. DHHS Publication No. (SMA) 01–3489
Smith JE, Lundy SL, Gianini L. Community reinforcement approach (CRA) and adolescent community reinforcement approach (A-CRA) coding manual. Albuquerque, NM: University of New Mexico; 2007.
Gibbons CJ, Carroll KM, Ball SA, Nich C, Frankforter TL, Martino S. Community program therapist adherence and competence in a motivational interviewing assessment intake session. Am J Drug Alcohol Abuse. 2010;36(6):342–9.
Martino S, Ball S, Nich C, Frankforter TL, Carroll KM. Correspondence of motivational enhancement treatment integrity ratings among therapists, supervisors, and observers. Psychother Res. 2009;19(2):181–93.
Martino S, Ball SA, Nich C, Frankforter TL, Carroll KM. Community program therapist adherence and competence in motivational enhancement therapy. Drug Alcohol Depend. 2008;96(1–2):37–48.
Madson MB, Loignon AC, Lane C. Training in motivational interviewing: a systematic review. J Subst Abus Treat. 2009;36(1):101–9.
Lane C, Hood K, Rollnick S. Teaching motivational interviewing: using role play is as effective as using simulated patients. Med Educ. 2008;42(6):637–44.
Martino S, Haeseler F, Belitsky R, Pantalon M, AHt F. Teaching brief motivational interviewing to year three medical students. Med Educ. 2007;41(2):160–7.
Haeseler F, AHt F, Pfeiffer C, Walters C, Martino S. Assessment of a motivational interviewing curriculum for year 3 medical students using a standardized patient case. Patient Educ Couns. 2011;84(1):27–30.
Haine-Schlagel R, Brookman-Frazee L, Janis B, Gordon J. Evaluating a learning collaborative to implement evidence-informed engagement strategies in community-based services for young children. Child Youth Care Forum. 2013;42(5):457–73.
Roosa M, Scripa JS, Zastowny TR, Ford JH 2nd. Using a NIATx based local learning collaborative for performance improvement. Eval Program Plann. 2011;34(4):390–8.
Stephan SH, Connors EH, Arora P, et al. A learning collaborative approach to training school-based health providers in evidence-based mental health treatment. Child Youth Serv Rev. 2013;35(12):1970–8.
Farmer AP, Legare F, Turcot L, et al. Printed educational materials: effects on professional practice and health care outcomes. Cochrane Database Syst Rev. 2008;3(3):CD004398.
Giguere A, Legare F, Grimshaw J, et al. Printed educational materials: effects on professional practice and healthcare outcomes. Cochrane Database Syst Rev. 2012;(10):CD004398.
Schneider EC, Zaslavsky AM, Landon BE, Lied TR, Sheingold S, Cleary PD. National quality monitoring of Medicare health plans: the relationship between enrollees’ reports and the quality of clinical care. Med Care. 2001;39(12):1313–25.
Miller WR, Moyers TB, Arciniega L, Ernst D, Forcehimes A. Training, supervision and quality monitoring of the COMBINE Study behavioral interventions. J Stud Alcohol. 2005;(Suppl 15):188–95. discussion 168–189
McCormack L, Sheridan S, Lewis M, et al. Communication and Dissemination Strategies To Facilitate the Use of Health-Related Evidence. Rockville, MD: Evidence Report/Technology Assessment No. 213. (Prepared by the RTI International-University of North Carolina Evidence-based Practice Center under Contract No. 290–2007-10056-I.) November 2013. AHRQ Publication No. 13(14)-E003-EF.
Miller WR, Yahne CE, Moyers TB, Martinez J, Pirritano M. A randomized trial of methods to help clinicians learn motivational interviewing. J Consult Clin Psychol. 2004;72(6):1050–62.
Sholomskas DE, Syracuse-Siewert G, Rounsaville BJ, Ball SA, Nuro KF, Carroll KM. We don’t train in vain: a dissemination trial of three strategies of training clinicians in cognitive-behavioral therapy. J Consult Clin Psychol. 2005;73(1):106–15.
Jamtvedt G, Young JM, Kristoffersen DT, et al. Audit and feedback: effects on professional practice and health care outcomes. Cochrane Database Syst Rev. 2006;2(2):CD000259.
Hysong SJ. Meta-analysis: audit and feedback features impact effectiveness on care quality. Med Care. 2009;47(3):356–63.
Ivers N, Jamtvedt G, Flottorp S, et al. Audit and feedback: effects on professional practice and healthcare outcomes. Cochrane Database Syst Rev. 2012;(6):CD000259.
Colquhoun HL, Brehaut JC, Sales A, et al. A systematic review of the use of theory in randomized controlled trials of audit and feedback. Implement Sci. 2013;8:66.
The content is solely the responsibility of the authors and does not necessarily represent the official views of the government. The authors thank Dr. Susan Godley and her EBT Coordinating Center staff for delivering the SAMHSA-funded implementation strategies, Christin Bair for coordinating the delivery of the NIDA-funded implementation strategy, and Emily Callot, Clare L. Hurley, and Sidney Amankwah for editorial assistance.
This work was supported by the National Institute on Alcohol Abuse and Alcoholism (NIAAA; R01-AA017625; Principal Investigator: Bryan R. Garner).
Availability of data and materials
Upon reasonable request, which should be made to the corresponding author, study data or materials may be made available.
Ethics approval and consent to participate
The current study was conducted under the auspices of Chestnut Health System’s Institutional Review Board.
Consent for publication
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.