Testing use of payers to facilitate evidence-based practice adoption: protocol for a cluster-randomized trial
© Molfenter et al.; licensee BioMed Central Ltd. 2013
Received: 8 March 2013
Accepted: 29 April 2013
Published: 10 May 2013
More effective methods are needed to implement evidence-based findings into practice. The Advancing Recovery Framework offers a multi-level approach to evidence-based practice implementation by aligning purchasing and regulatory policies at the payer level with organizational change strategies at the organizational level.
The Advancing Recovery Buprenorphine Implementation Study is a cluster-randomized controlled trial designed to increase use of the evidence-based practice buprenorphine medication to treat opiate addiction. Ohio Alcohol, Drug Addiction, and Mental Health Services Boards (ADAMHS), who are payers, and their addiction treatment organizations were recruited for a trial to assess the effects of payer and treatment organization changes (using the Advancing Recovery Framework) versus treatment organization changes alone on the use of buprenorphine. A matched-pair randomization, based on county characteristics, was applied, resulting in seven county ADAMHS boards and twenty-five treatment organizations in each arm. Opioid dependent patients are nested within cluster (treatment organization), and treatment organization clusters are nested within ADAMHS county board. The primary outcome is the percentage of individuals with an opioid dependence diagnosis who use buprenorphine during the 24-month intervention period and the 12-month sustainability period. The trial is currently in the baseline data collection stage.
Although addiction treatment providers are under increasing pressure to implement evidence-based practices that have been proven to improve patient outcomes, adoption of these practices lags, compared to other areas of healthcare. Reasons frequently cited for the slow adoption of EBPs in addiction treatment include, regulatory issues, staff, or client resistance and lack of resources. Yet the way addiction treatment is funded, the payer’s role—has not received a lot of attention in research on EBP adoption.
This research is unique because it investigates the role of payers in evidence-based practice implementation using a randomized controlled design instead of case examples. The testing of the Advancing Recovery Framework is designed to broaden the understanding of the impact payers have on evidence-based practice (EBP) adoption.
http://NCT01702142 (ClinicalTrials.gov registry, USA)
KeywordsEvidence-based practice implementation Buprenorphine Addiction treatment Innovation
The use of evidence-based clinical practices in health care is suboptimal [1, 2]. The addiction treatment field adopts evidence-based practices (EBPs) at an even lower rate than general health care [3–5]. For example, use of medication-assisted therapy for addiction disorders is not standard practice, despite the evidence that medication improves clinical, societal and financial outcomes through increased abstinence rates , reduced relapse rates , and reduced criminal costs . A survey of 354 private addiction treatment centers reported that 34 percent of programs provided medication for opioid dependence and 24% used medication for treatment of alcohol dependence . Within treatment programs that do offer medication, only 40% of the treatment plans for opioid dependence included medication .
Buprenorphine received Food and Drug Administration approval for treating opioid dependence in 2002 but remains underutilized. Less than one-fifth of patients with an opioid dependence diagnosis are offered buprenorphine [11, 12]. Its adoption has been impeded by behavioral resistance, lack of or low reimbursement, insufficient access to personnel needed to prescribe the medication, licensure and contractual rules that prohibit the use of any pharmacotherapies in some settings, and insufficient organizational systems to encourage its use. Buprenorphine was selected as a candidate for EBP implementation and dissemination research because the broad set of behavioral and systemic barriers that limit its use incorporates factors that affect other EBPs with poor use rates.
Payer policy and organizational interventions can address barriers to the adoption of medication (and evidence-based practices more generally) [13, 14]. Policy makers and local organizations collaborate to disseminate school reform, alternative energy use, and third-world agricultural practices [15, 16]. Few trials in healthcare have tested the impact of aligning payer policy and treatment organization interventions to enhance implementation of treatment innovations . However, from 2006 through 2010, NIATx (formerly the Network for the Improvement of Addiction Treatment), a process improvement research center based at the University of Wisconsin–Madison, conducted a demonstration project titled ‘Advancing Recovery: State and Treatment Organization Partnerships for Quality Addiction Care’. Funded by the Robert Wood Johnson Foundation, Advancing Recovery promoted the use of evidence-based practices such as medication-assisted treatment by building relationships between treatment organizations and the payers that fund their services. These partnerships made system changes to regulatory, clinical and administrative practices to increase the implementation of EBPs . Advancing Recovery involved payer-treatment organization partnerships in 12 states, but was not a randomized controlled trial. The Advancing Recovery demonstration project did develop a framework for systems change to be tested in this randomized trial, calling it the Advancing Recovery Framework.
The primary research question is to determine if the Advancing Recovery Framework increases the use of buprenorphine in addiction treatment organizations. The secondary research question is to determine if county characteristics (e.g., percentage of opioid abusers, existing addiction treatment medication availability, and payment policy) and organizational characteristics (e.g., organization size, access to physicians, number of nurses, percentage of clinicians with graduate degrees, and patient demographic characteristics) can moderate the delivery of buprenorphine.
The Advancing Recovery Framework builds upon the multi-organizational learning collaborative method used in the NIATx 200 national trial to implement process improvement strategies in over 200 treatment organizations [22, 23]. The Advancing Recovery Framework delivers technical assistance through learning sessions and coaching. Learning sessions are face-to-face meetings that occur once in the NIATx-only arm and three times in the Advancing Recovery arm to facilitate partnering between the ADAMHS boards and their treatment organizations. In these sessions, change teams from the ADAMHS board area (Arm 2 only) and each of the treatment organizations convene in face-to-face meetings to support and encourage each other. Outside experts offer advice on changes to make and ways to make them. Coaching assigns an expert in medication-assisted treatment adoption and process improvement to help the ADAMHS board and its treatment organizations make, sustain and spread buprenorphine adoption efforts. Coaches help the ADAMHS boards’ counties in Arm 2, and organizations in both arms think through key issues, broker relationships with other organizations, offer process improvement training, and suggest changes to make and how to do so. Participants in both arms receive coaching through monthly group teleconference calls, through e-mail, and at in-person learning sessions.
Payer intervention model of the advancing recovery framework (applied in Arm 2 only)
Pre-work for Leadership
To be completed as part of Executive Briefing by the Executive Director (or designee)
Executive sponsor (County Director) personal responsibilities checklist
Coach call to Executive Sponsor to review primer for Executive Sponsor
1. Identify an Executive Sponsor for payer portion of the project
Change leader training on how to apply and interpret the 5 Levers Barriers Assessment
2. Develop the strategic aim for buprenorphine adoption.
3. Select change leader and team members who are committed and make sure they have adequate time to participate in the project.
Preparing the Change Leader & Team
4. Change Leader & Team Training on organizational and policy change techniques.
NIATx Workbook (Change Leader Training Guide – Administrative Version)
Provided in Learning Session 1 and reinforced by coaches throughout
Planning (in conjunction with the treatment organization community)
5. Conduct the 5 Levers Barriers Assessment. (Customer Impact, Financial & Purchasing, Regulatory & Policy, Operational, & Inter-organizational) to identify list of potential barriers to remove
5-Levers Policy-Barriers Assessment
Provided in Learning Session 1 and reinforced by coaches throughout
6. Develop tactics to apply
Implementation charter form
7. Pilot test tactics
8. Develop list of tactics to implement (based on planning)
Implementation charter form
Change leader training in Learning Sessions on the implementation and sustainability process
9. Assign a process owner, and timeline with deliverables for each tactic
Apply payer-based sustainability assessment form
10. Implement changes
11. Sustain gains
The five levers described in the planning phase provide a mechanism for payers to identify systemic barriers to EBP use and promote change. The ‘customer impact analysis’ examines how the EBP benefits the customer, affects the number or type (i.e., demographic) of customers involved in treatment, and which customers are needed to advocate for the change. A ‘financing and purchasing analysis’ determines how financing, reimbursement and payment provide incentives for EBP use. The ‘regulatory and policy analysis’ uncovers regulatory barriers. The ‘operations analysis’ assesses the training and skill-building payers and treatment organizations need for EBP implementation. Finally, an ‘inter-organizational analysis’ identifies what other organizations need to be involved to remove regulatory or financing barriers.
During project months 1 to 5, the study team will identify and recruit counties (through the Ohio Association of County Behavioral Health Authorities) and treatment organizations. In months 6 to 11, the study team will collect baseline data on buprenorphine use rates, conduct buprenorphine training workshops for treatment organizations, and update NIATx training materials to reflect the study conditions. In months 12 to 36, the study team will implement the study arm interventions and collect outcome data. During months 37 to 50, the study team will collect quantitative data on the sustainability of the changes. Data analysis, publication development, and dissemination of findings will occur at several points, but primarily during months 51 to 60.
To determine sample size, we fit a linear mixed-effects model to the monthly results for the performance measure of percent opioid dependent patients receiving buprenorphine from each of the participating organizations and estimate the ‘payer-treatment organization’ effect. A mixed-effect model was selected instead of conventional regression models because, first, the mean outcomes for a given treatment organization are expected to be correlated from month to month. A mixed-effect model allows for auto-correlated error terms. Second, a number of unobservable treatment organization characteristics are likely to influence the key measures. Random effects at the treatment organization level can be employed within the mixed-effect model to capture the correlation introduced by such characteristics. Proper reflection of auto-correlation and random effects allows for more reliable estimates of the fixed effects, starting in months 6 to 11 (for baseline), then during months 12 to 36 (intervention stage), and through months 37 to 50 (follow-up stage). The power of the study design is determined by the anticipated standardized effect size based on effects experienced in the Advancing Recovery pilot project. The Advancing Recovery pilot data found that the payer-treatment organization intervention increased buprenorphine use rates from 6% to 43%, with an estimated standard deviation of 23.5% based on a sample size of 1,016. This yields an effect size of 0.935. Intraclass correlation (ICC) among counties affects the power of cluster-randomized trials . An estimate of ICC is around 0.05. For this study, a dropout rate of 20% is being proposed, giving a total of 40 organizations available for analysis. With a total of 40 treatment organizations, each with an average of 91 patients, the study will achieve at least a power of 0.82 with a one-tailed type I error rate of 0.05. Power and sample size calculations were performed using Optimal Design Software by Raudenbush (Version 3.01) .
Randomization & implementation
Matched-pair blocks were created on the basis of the county characteristics of: percentage of opiate admissions; population covered; and ratio of opiate addiction treatment admissions to physicians licensed to prescribe buprenorphine . The cut-off criteria used for matching, based on the median measures from the 2011 Ohio Department of Alcohol and Drug Addiction Services treatment organization dataset, were: 20% opiate admissions (above/below); 200,00 people covered (above/below); and 50 treatment admissions per physicians in the county registered to prescribe buprenorphine per annum (above/below). As a result, seven county board matched-pairs were identified. A random number generator was utilized within the seven county board pairs to assign organizations into one of the two study conditions. Study staff completed enrollment and received consent approval from the county boards as well as providers. No visible concealment was applied, and blinding of participants and researchers was considered infeasible due to logistical reasons.
Because the number of treatment organizations within an ADAMHS Board is small (n = 2 to 8) we intend to use an optimal design in which treatment arms are allocated following a multi-site cluster randomization procedure [27, 28]. The initial exploratory analyses of both outcomes and covariates will assess standard summary statistics and graphical presentations. Scatter plots and correlation analyses identify possible associations between outcome and predictor variables. The analysis will represent the multi-site cluster randomized trial as a mixed-effects model; percentage of buprenorphine use by opioid dependent patients is nested within cluster (treatment organization); and clusters are nested within site (ADAMHS board). In this mixed-effects model, there will be random effects due to county board and organization; fixed effect due to study arm and time. For the percent of opioid admissions receiving buprenorphine, a logistic regression analysis will be performed. We will examine the Advancing Recovery effect using a separate model at each time point and in an omnibus growth curve model that includes the main effects of Advancing Recovery and Time as well as the interaction between Advancing Recovery and Time. The percent of opioid patients receiving buprenorphine will be measured monthly for each organization, and it is expected that these values will be correlated over time. Therefore, instead of assuming an independence (i.e., zero correlation) or a compound symmetry covariance structure (i.e., a constant correlation regardless of the proximity of measurement time points), we will allow errors to be first-order auto-correlated, denoted as AR (1). This can be done by allowing an additional parameter in the mixed-effects model that represents the correlation between the two most adjacent measures of Opioid Patients Receiving Buprenorphine. This correlation is reduced exponentially as the measures become further apart.
For the ADAMHS board and organizational covariates, at each time point we will assess moderation by including each potential moderator as an independent variable and as an interaction with the Advancing Recovery effect. Statistical significance of the interaction effect will indicate the presence of moderation. The estimate for the effect size of moderation, which is cross-level interaction, is between 0.17 and 0.19.
The study received approval from the institutional review boards at the University of Wisconsin–Madison, Ohio Department of Health’s Human Subjects Institutional Review Board, and Oregon Health and Science University.
The recruitment phase of the trial has just been completed and baseline data collection has begun.
Conceptual multi-level models of EBP adoption are present in the literature and are based on correlates of EBP adoption discovered at the larger systems/environment, organizational, and individual levels [13, 17, 29]. These models identify which factors EBP implementation efforts should attempt to influence at the different levels in the healthcare system, but do not address how to change these factors. Accordingly, this research represents a movement from correlate identification to implementation model testing. This progression is described in McGovern’s ‘Implementation Research Flow’ paradigm, which explains how the purpose of implementation models is to positively influence the implementation and maintenance trajectories of the EBP adoption .
At present, the implementation research on how administrative practices at the payer level promote use of clinical practices at the treatment organization level primarily relies on case examples. The described research protocol has the unique aspect of randomizing by payer. The use of implementation research randomizing by county has been limited [31, 32], and no instances of randomization at the state level were found. Research studying use of payer administrative practices to promote clinical evidence-based practice implementation should begin to use randomized trials to augment and validate findings found in payer case-examples.
This trial is taking lessons from past implementation research through case examples from the Advancing Recovery pilot [33–39], and packages them into an implementation model . The research trial testing the Advancing Recovery Framework will build on several implementation research questions: What are the roles of payers with planning and stakeholder engagement in the provision of EBPs? What policy levers have an impact on EBP implementation and sustainability rate? What are the synergistic effects of having multiple levels focused on implementing a targeted EBP? What is the role of leadership at the payer and provider levels in the broader use of EBPs? What training and technical assistance offerings in the learning collaborative model are most helpful in a multi-level design?
Ohio was selected as a research site because it was considered to be a representative governmental healthcare environment to test this implementation model. A total of 34% of the states in the United States have county or region boards to distribute SAPT block grant funds. Each Ohio ADAMHS board represents a unique payer environment. The state has a mix of urban and rural counties. Within these counties, buprenorphine use varies from 7% to 58%. Limitations could be present due to the research environment. This trial addresses governmental payers securing healthcare services. Governmental healthcare is a considerable enterprise, where an effective framework for evidence-based practice implementation could prove to be extremely helpful. While this trial may not be generalizable to governmental payers for other community services, the Advancing Recovery framework could give these sectors interventions to consider in their efforts to improve services.
In conclusion, this trial will contribute to the evolving implementation science field over the next five years. Future results of this multi-level approach will be published as they become available.
Alcohol, Drug Addiction and Mental Health Services
Network for the Improvement of Addiction Treatment
Substance Abuse and Prevention Treatment.
The study is funded by the National Institutes on Drug Abuse (NIDA) (R01 DA030431-01A1). NIDA has played no role in the study design or preparation of this manuscript.
- Baker TB, McFall RM, Shoham V: Current status and future prospects of clinical psychology: toward a scientifically principled approach to mental and behavioral health care. Psychol Sci Public Interest. 2008, 9 (2): 67-103.PubMedGoogle Scholar
- Bodenheimer T: The American health care system–the movement for improved quality in health care. N Engl J Med. 1999, 340 (6): 488-492. 10.1056/NEJM199902113400621.View ArticlePubMedGoogle Scholar
- Institute of Medicine: Bridging the gap between practice and research: forging partnerships with community based drug and alcohol treatment. 1998, Washington, DC: National Academy PressGoogle Scholar
- Willinbring ML, Kivlahan D, Kenny M, Grillo M, Hagedorn H, Postier A: Beliefs about evidence-based practices in addiction treatment: a survey of veterans administration program leaders. J Subst Abuse Treat. 2004, 26 (2): 79-85. 10.1016/S0740-5472(03)00161-2.View ArticleGoogle Scholar
- Garner B: Research on the diffusion of evidence-based treatments within substance abuse treatment: a systematic review. J Subst Abuse Treat. 2009, 36 (4): 376-399. 10.1016/j.jsat.2008.08.004.View ArticlePubMedGoogle Scholar
- Moore TJ, Cohen MR, Furberg CD: Serious adverse drug events reported to the food and drug administration, 1998–2005. Arch Intern Med. 2007, 167 (16): 1752-10.1001/archinte.167.16.1752.View ArticlePubMedGoogle Scholar
- O’Brien CP: Recent developments in the pharmacotherapy of substance abuse. J Consult Clin Psychol. 1996, 64 (4): 677-686.View ArticlePubMedGoogle Scholar
- Harris AH, Gospodarevskaya E, Ritter AJ: A randomised trial of the cost effectiveness of buprenorphine as an alternative to methadone maintenance treatment for heroin dependence in a primary care setting. PharmacoEconomics. 2005, 23 (1): 77-91. 10.2165/00019053-200523010-00007.View ArticlePubMedGoogle Scholar
- Knudsen HK, Abraham AJ, Roman PM: Adoption and implementation of medications in addiction treatment programs. J Addict Med. 2011, 5: 21-27. 10.1097/ADM.0b013e3181d41ddb.View ArticlePubMedPubMed CentralGoogle Scholar
- Knudsen HK, Roman PM: Financial factors and the implementation of medications for treating opioid use disorders. J Addict Med. 2012, 6 (4): 280-286. 10.1097/ADM.0b013e318262a97a.View ArticlePubMedPubMed CentralGoogle Scholar
- Wallack S, Thomas C, Martin T, Chilingerian J, Reif S: Substance abuse treatment organizations as mediators of social policy: slowing the adoption of a congressionally approved medication. J Behav Health Serv Res. 2010, 37 (1): 64-78. 10.1007/s11414-008-9132-4.View ArticlePubMedGoogle Scholar
- Stein BD, Gordon AJ, Sorbero M, Dick AW, Schuster J, Farmer C: The impact of buprenorphine on treatment of opioid dependence in a Medicaid population: recent service utilization trends in the use of buprenorphine and methadone. Drug Alcohol Depend. 2012, 123 (1–3): 72-78.View ArticlePubMedGoogle Scholar
- Ferlie EW, Shortell SH: Improving the quality of health care in the United Kingdom and the United States: a framework for change. Milbank Q. 2001, 79 (2): 281-315. 10.1111/1468-0009.00206.View ArticlePubMedPubMed CentralGoogle Scholar
- Grol R: Improving the quality of medical care: building bridges among professional pride, payer profit, and patient satisfaction. JAMA. 2001, 286 (20): 2578-2585. 10.1001/jama.286.20.2578.View ArticlePubMedGoogle Scholar
- Slavin RE: Research on achievement outcomes of success for all. Phi Delta Kappan. 2000, 82 (1): 38.View ArticleGoogle Scholar
- Rogers EM: Diffusion of innovations. 2003, New York: Free PressGoogle Scholar
- Greenhalgh T, Robert G, Macfarlane F, Bate P, Kyriakidou O: Diffusion of innovations in service organizations: systematic review and recommendations. Milbank Q. 2004, 82 (4): 581-629. 10.1111/j.0887-378X.2004.00325.x.View ArticlePubMedPubMed CentralGoogle Scholar
- Schmidt LA, Rieckmann T, Abraham A, Molfenter T, Capoccia V, Roman P, Gustafson DH, McCarty D: Advancing Recovery: implementing evidence-based treatment for substance use disorders at the systems level. J Stud Alcohol Drugs. 2012, 73 (3): 413-422.View ArticlePubMedPubMed CentralGoogle Scholar
- Simpson DD: A conceptual framework for transferring research to practice. J Subst Abuse Treat. 2002, 22 (4): 171-182. 10.1016/S0740-5472(02)00231-3.View ArticlePubMedGoogle Scholar
- Proctor EK: Leverage points for the implementation of evidence-based practice. Brief Treat Crisis Interv. 2004, 4 (3): 227-242. 10.1093/brief-treatment/mhh020.View ArticleGoogle Scholar
- Mark TL, Levit KR, Coffey RM, McKusick DR, Harwood HJ, King EC, Bouchery E, Genuardi JS, Vandivort-Warren R, Buck JA, Ryan K: National expenditures for mental health services and substance abuse treatment: 1993-2003.Vol SAMHSA Publication No. SMA 07-4227. 2007, Rockville, MD: Substance Abuse and Mental Health ServicesAdministrationGoogle Scholar
- Quanbeck AR, Gustafson DH, Ford JH, Pulvermacher A, French MT, McConnell KJ, McCarty D: Disseminating quality improvement: study protocol for a large cluster randomized trial. Implement Sci. 2011, 6 (1): 44-10.1186/1748-5908-6-44.View ArticlePubMedPubMed CentralGoogle Scholar
- Gustafson DH, Quanbeck AR, Robinson JM, Ford JH, Pulvermacher A, French MT, McConnell KJ, Batalden PB, Hoffman KA, McCarty D: Which elements of improvement collaboratives are most effective? a cluster-randomized trial. Addiction. in press
- Hsieh FY, Lavori PW, Cohen HJ, Feussner JR: An overview of variance inflation factors for sample-size calculation. Eval Health Prof. 2003, 26 (3): 239-257. 10.1177/0163278703255230.View ArticlePubMedGoogle Scholar
- Raudenbush SW, Spybrook J, Congdon R, Liu X, Martinez A, Bloom H, Hill C: Optimal design software for multi-level and longitudinal research (Version 3.0). [http://www.wtgrantfoundation.org/resources/consultation-service-and-optimal-design]
- Knudsen HK, Ducharme LJ, Roman PM: Early adoption of buprenorphine in substance abuse treatment centers: data from the private and public sectors. J Subst Abuse Treat. 2006, 30 (4): 363-373. 10.1016/j.jsat.2006.03.013.View ArticlePubMedGoogle Scholar
- Spybrook J, Raudenbush SW, Liu X, Congdon R, Martinez A: Building capacity to evaluate group-level interventions.http://sitemaker.umich.edu/group-based,
- Raudenbush SW, Liu X: Statistical power and optimal design for multisite randomized trials. Psychol Methods. 2000, 5 (2): 199-213.View ArticlePubMedGoogle Scholar
- Aarons GA, Hurlburt M, Horwitz SM: Advancing a conceptual model of evidence-based practice implementation in public service sectors. Adm Policy Ment Health. 2011, 38 (1): 4-23. 10.1007/s10488-010-0327-7.View ArticlePubMedGoogle Scholar
- McGovern MP, Saunders EC, Kim E: Substance abuse treatment implementation research. J Subst Abuse Treat. 2013, 44 (1): 1-3. 10.1016/j.jsat.2012.09.006.View ArticlePubMedGoogle Scholar
- Gilbert CM, O’Donnell E: The hospital library’s role in recruitment and retention of medical staff. J Hosp Librariansh. 2004, 4 (4): 7-30. 10.1300/J186v04n04_02.View ArticleGoogle Scholar
- Chamberlain P, Brown CH, Saldana L, Reid J, Wang W, Marsenich L, Sosna T, Padgett C, Bouwman G: Engaging and recruiting counties in an experiment on implementing evidence-based practice in California. Adm Policy Ment Health. 2008, 35 (4): 250-260. 10.1007/s10488-008-0167-x.View ArticlePubMedPubMed CentralGoogle Scholar
- Magnabosco JL: Innovations in mental health services implementation: a report on state-level data from the National Evidence-Based Practices Project. Implement Sci. 2006, 1: 13-10.1186/1748-5908-1-13.View ArticlePubMedPubMed CentralGoogle Scholar
- Bruns EJ, Hoagwood KE, Rivard JC, Wotring J, Marsenich L, Carter B: State implementation of evidence-based practice for youths, Part II: recommendations for research and policy. J Am Acad Child Adolesc Psychiatry. 2008, 47: 499-504. 10.1097/CHI.0b013e3181684557.View ArticlePubMedGoogle Scholar
- Shea MA, Callis BP, Cassidy-Stewart H, Cranston K, Tomoyasu N: Diffusion of effective HIV prevention interventions--lessons from Maryland and Massachusetts. AIDS Educ Prev. 2006, 18: 96-107. 10.1521/aeap.2006.18.supp.96.View ArticlePubMedGoogle Scholar
- Raghavan R, Bright CL, Shadoin A: Toward a policy ecology of implementation of evidence-based practices in public mental health settings. Implement Sci. 2008, 3 (1): 26-10.1186/1748-5908-3-26.View ArticlePubMedPubMed CentralGoogle Scholar
- Ganju V: Implementation of evidence-based practices in state mental health systems. Schizophr Bull. 2003, 29: 125-131. 10.1093/oxfordjournals.schbul.a006982.View ArticlePubMedGoogle Scholar
- Rapp CA, Bond GR, Becker DR, Carpinello SE, Nikkel RE, Gintoli G: The role of state mental health authorities in promoting improved client outcomes through evidence-based practice. Community Ment Health J. 2005, 41: 347-363. 10.1007/s10597-005-5008-8.View ArticlePubMedGoogle Scholar
- Isett KR, Burnam MA, Coleman-Beattie B, Hyde PS, Morrissey JP, Magnabosco JL, Rapp C, Ganju V, Goldman HH: The role of state mental health authorities in managing change for the implementation of evidence-based practices. Community Ment Health J. 2008, 44: 195-211. 10.1007/s10597-007-9107-6.View ArticlePubMedGoogle Scholar
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.