This article has Open Peer Review reports available.
Brief cognitive behavioral therapy in primary care: a hybrid type 2 patient-randomized effectiveness-implementation design
© Cully et al; licensee BioMed Central Ltd. 2012
Received: 10 April 2012
Accepted: 20 June 2012
Published: 11 July 2012
Despite the availability of evidence-based psychotherapies for depression and anxiety, they are underused in non-mental health specialty settings such as primary care. Hybrid effectiveness-implementation designs have the potential to evaluate clinical and implementation outcomes of evidence-based psychotherapies to improve their translation into routine clinical care practices.
This protocol article discusses the study methodology and implementation strategies employed in an ongoing, hybrid, type 2 randomized controlled trial with two primary aims: (1) to determine whether a brief, manualized cognitive behavioral therapy administered by Veterans Affairs Primary Care Mental Health Integration program clinicians is effective in treating depression and anxiety in a sample of medically ill (chronic cardiopulmonary diseases) primary care patients and (2) to examine the acceptability, feasibility, and preliminary outcomes of a focused implementation strategy on improving adoption and fidelity of brief cognitive behavioral therapy at two Primary Care-Mental Health Integration clinics. The study uses a hybrid type 2 effectiveness/implementation design to simultaneously test clinical effectiveness and to collect pilot data on a multifaceted implementation strategy that includes an online training program, audit and feedback of session content, and internal and external facilitation. Additionally, the study engages the participation of an advisory council consisting of stakeholders from Primary Care-Mental Health Integration, as well as regional and national mental health leaders within the Veterans Administration. It targets recruitment of 320 participants randomized to brief cognitive behavioral therapy (n = 200) or usual care (n = 120). Both effectiveness and implementation outcomes are being assessed using mixed methods, including quantitative evaluation (e.g., intent-to-treat analyses across multiple time points) and qualitative methods (e.g., focus interviews and surveys from patients and providers). Patient-effectiveness outcomes include measures of depression, anxiety, and physical health functioning using blinded independent evaluators. Implementation outcomes include patient engagement and adherence and clinician brief cognitive behavioral therapy adoption and fidelity.
Hybrid designs are needed to advance clinical effectiveness and implementation knowledge to improve healthcare practices. The current article describes the rationale and challenges associated with the use of a hybrid design for the study of brief cognitive behavioral therapy in primary care. Although trade-offs exist between scientific control and external validity, hybrid designs are part of an emerging approach that has the potential to rapidly advance both science and practice.
KeywordsPrimary care Hybrid effectiveness-implementation designs Cognitive behavioral therapy Mental health Veterans Anxiety Depression
Traditional cognitive behavioral therapy (CBT), consisting of 12 to 16 sessions over three to six months of treatment, is an efficacious treatment for depression and anxiety[1–5]. Much of the research on CBT has been conducted in academic trials and within specialty mental healthcare settings[6, 7]. Less is known about the utilization of brief CBT (bCBT), consisting of four to six sessions over a maximum of four months, and its ability to be adopted within non-mental healthcare settings such as primary care. Treatment modalities like bCBT can dramatically enhance the reach of mental illness therapy, given the prevalence and morbidity of depression and anxiety among medically ill patients.
Although psychotherapies such as CBT improve patient outcomes, such treatments are infrequently used within integrated healthcare settings, especially in non-specialty mental healthcare settings such as primary care[8–11]. In an effort to improve the availability of psychotherapy, the Department of Veterans Affairs (VA) issued the Uniform Mental Health Services Handbook in 2008, which mandated that VA medical centers embed mental health services into primary care settings and make available evidence-based psychotherapies for all veterans with depression and anxiety. Since 2008, the VA has promoted a national dissemination program called Primary Care-Mental Health Integration (PC-MHI) for all its hospitals and large community-based clinics and has also dedicated significant resources toward clinician training in traditional evidence-based psychotherapies, including CBT, using a 12- to 16-session approach[4, 13, 14].
Unfortunately, mental health providers in the primary care setting face challenges to implementation of traditional evidence-based psychotherapies. Evidence suggests that patients treated in primary care are distinct from specialty mental healthcare patients (e.g., present with physical health concerns and fewer chronic and severe mental health difficulties) and require modifications to traditional approaches[15, 16]. Data also suggest that primary care patients may benefit from mental health interventions that are less intensive and focus treatment around physical as well as emotional health concerns[17–19]. These patients may also prefer and engage more fully in care co-located in primary care[20, 21]. Lastly, VA PC-MHI providers require assistance to determine how best to modify traditional evidence-based therapies to better align with the practice demands of the primary care setting, which are increasingly involving collaborations between mental and physical health providers (e.g., medical home or the Patient-Aligned Care Team initiative within the VA). For example, in the VA PC-MHI program, clinicians are asked to provide evidence-based care (e.g., 12–16 sessions of CBT) within an integrated primary care model that encourages psychotherapy services in the range of four to six sessions of care[11, 22, 23]. Presently, mental health providers in primary care are struggling to adapt traditional evidence-based approaches, and they need data to support their delivery of high-quality care.
Initial studies provide evidence that bCBT is an effective treatment within primary care for depression, panic, and generalized anxiety disorders[17, 18, 24–27]. Although studies support the general efficacy of bCBT, additional effectiveness data are needed to determine the impact of these treatments under real-world conditions and to understand the potential for bCBT to be integrated within primary care.
Using blended effectiveness-implementation designs to improve the provision of evidence-based psychotherapy
It is imperative that research focus on increasing use of evidence-based interventions in frontline practice. Unfortunately, significant time and resources are required by the traditional sequential scientific approach that moves from clinical efficacy to effectiveness and then implementation, and this approach is often plagued by practice or translational barriers because of the mismatch between the scientific intervention and the demands of frontline practice. Hybrid designs are increasingly being used to move science beyond an excessive attention to internal validity. Clinical-intervention hybrid designs are commonly employed on the efficacy/effectiveness continuum[9, 30] and generally provide more relaxed internal controls to improve generalizability. Hybrid effectiveness/implementation (E-I) designsare relatively new, with distinct methodological opportunities and challenges. Hybrid E-I designs address clinical effectiveness but target the methods and procedures necessary to deliver and sustain interventions in real-world care settings. As a prerequisite, hybrid E-I designs require a minimum of “indirect evidence” supporting a clinical intervention. Without preexisting evidence to support the clinical intervention, implementation evaluation is premature.
Prototypical clinical and implementation research designs do not share many design features. Thus, a series of compromises are needed when creating and conducting a hybrid E-I study to balance the demands associated with scientific control or internal validity and factors associated with implementation, such as evaluation and improvement of processes and systems of care. Although compromise between these competing constructs may prove challenging, hybrid designs have the potential to (1) shorten the existing lag time between research discovery and uptake in care settings, (2) provide additional knowledge of clinical processes and system factors critical to adoption and long-term utilization, and (3) reduce cost by streamlining and combining elements of the traditional step-wise progression of research (efficacy to effectiveness to implementation). Most importantly, hybrid E-I designs that use strong and broad-reaching partnerships improve the collective shared vision among stakeholders and, ultimately, improve the odds that interventions developed will be adopted into practice.
Evaluation of hybrid E-I designs involves careful attention to the assessment of outcomes related to both clinical effectiveness and implementation success. RE-AIM is a commonly used framework to aid in addressing the unique challenges of blended E-I designs[32–36]. RE-AIM seeks to enhance evaluation and reduce the disparities between research, applied clinical practice, and sustainability of evidence-based practices over time[28, 34] by focusing on five components: R each (the participation rate of the targeted population), E fficacy/E ffectiveness (the impact of the intervention on outcome criteria), A doption (number or proportion of sites/clinicians who provide the clinical intervention), I mplementation (intervention integrity, quality, and consistency of delivery), and M aintenance (sustainability of the intervention over time).
This protocol article describes a hybrid type 2 study, entitled “Cognitive Behavioral Therapy in Primary Care: Treating the Medically Ill.” Consistent with the hybrid E-I design and informed by the RE-AIM framework, the project focuses on the effectiveness of bCBT for patient outcomes related to depression, anxiety, and physical health, while also seeking to simultaneously pilot test the acceptability, feasibility, and preliminary outcomes of a multifaceted implementation strategy to enhance patient engagement, as well as clinician adoption and fidelity. The article highlights study procedures and critical methodological decisions related to conducting a hybrid E-I study for bCBT.
The Adjusting to Chronic Conditions Using Education, Support, and Skills (ACCESS) study was developed to examine the clinical effectiveness and implementation potential of an evidence-based, patient-centered bCBT intervention for depressed and/or anxious medically ill veterans. The project is being conducted within the PC-MHI programs of two large VA medical centers in the south central United States. The current study was approved by the Baylor College of Medicine Institutional Review Board (H-27082) and local VA Research and Development review committee. As such the study is in compliance with the Helsinki Declaration and all study participants provided informed consent including permissions for publication of this report.
ACCESS uses a patient-level, randomized design and focuses on veterans with chronic obstructive pulmonary disease (COPD) and/or heart failure (HF). COPD and HF are highly prevalent and burdensome to patients and the healthcare system[37–39]. These conditions are exacerbated by clinically elevated symptoms of depression and anxiety, which are common in approximately 50% of this patient population[40–43].
Study outcomes and objectives
#1: To determine whether a bCBT treatment group as provided by VA PC-MHI clinicians is superior to a usual-care control group at post treatment and 8- and 12-month follow-ups, as measured by:
a) depression and anxiety scores (Patient Health Questionnaire-9 and Beck Anxiety Inventory)
b) cardiopulmonary disease outcomes (Chronic Respiratory Questionnaire and Kansas City Cardiomyopathy Questionnaire).
#2: To assess bCBT adoption and fidelity, as measured by:
a) bCBT patient engagement (one or more sessions) and adherence (four or more sessions)
b) PC-MHI clinician bCBT adherence and competency ratings as evaluated by expert audio session reviews.
Objective #1: clinical effectiveness of brief cognitive behavioral therapy
Comparative effectiveness trials suggest a general equivalence between CBT and other therapeutic modalities, with CBT being particularly well-suited for implementation within primary care settings, given its skill-based approach and ability to function with a restricted time and session framework[45, 46]. Recent evidence suggests the general efficacy of bCBT for patients in primary care and patients with comorbid health conditions[17, 18, 47, 48]. However, limited data exist on the real-world effectiveness of bCBT within the primary care setting, especially when used for the chronically medically ill.
The ACCESS intervention is a manualized bCBT protocol that provides a flexible, patient-centered approach to increase patient engagement and adherence, while addressing both the mental and physical health needs of veterans with COPD and HF. ACCESS consists of six weekly treatment sessions and two brief (10- to 15-minute) telephone “booster” sessions within a four-month time frame. Participants are asked to attend the first session in person and can participate in subsequent sessions by telephone or in person.
The ACCESS intervention was developed based on prior studies and input from a variety of stakeholders, including patients, providers, PC-MHI directors, and national CBT experts. Particular emphasis has been placed on maximizing intervention potency and minimizing session intensity and duration to maximize patient engagement and adoption within PC-MHI settings. Patients initially receive core modules focused on increasing awareness and controlling physical and emotional symptoms and subsequently are able to select skills from a series of module choices best aligned with their most pressing needs. Therapists follow a structured intervention manual, yet retain the ability to streamline and tailor the bCBT intervention collaboratively with patients when selecting treatment modules and goals. A patient workbook is provided to structure session content, guide telephone interactions, and provide opportunities to guide patient practice of skills between sessions.
Following an iterative developmental process[19, 50], the ACCESS intervention was modified to be consistent with the PC-MHI model of the VA of brief four to six sessions of psychotherapy. ACCESS provides clinicians with structure and details that seek to ensure high fidelity to the intervention, while retaining flexibility in administration so as to mirror traditional psychotherapy practices. Data from a previously conducted open trial suggest that ACCESS improves both physical and emotional health outcomes and results in high engagement and adherence. Detailed descriptive information about the content and processes of the ACCESS intervention can be found elsewhere.
Identification and recruitment of participants
Study participants are being recruited from the Houston and Oklahoma City VA medical centers, using VA administrative databases to identify COPD and HF patients who received care in the prior year. Houston and Oklahoma City VAs were chosen for their size and integration of primary care mental health within the Veterans Integrated Service Network 16. Opt-out letters are mailed to participants to ensure the widest catchment possible. Interested participants complete a telephone screening process. Participants are included in the study if they are confirmed to have patient-reported functional limitations associated with their COPD and/or HF and have clinically significant symptoms of anxiety and/or depression. Participants are excluded only for clinical factors (e.g., ongoing psychotherapy, concurrent specialty mental healthcare) or patient factors (e.g., cognitive, bipolar, psychotic, or substance use disorders) that would render a PC-MHI bCBT intervention or PC-MHI treatment setting inappropriate. Patients receiving psychotropic medications are not excluded, although they have their medication use monitored during the course of the study.
Sample size was established to ensure adequate power to detect differences in the primary patient outcomes for depression, anxiety, and physical health. To ensure 80% power to detect differences at the 0.05 level for all study measures, adjusting for repeated measures, a sample size of 180 randomized patients was necessary. However, this sample-size calculation was inflated because of the potential for intraclass correlations related to the clustering of patients within bCBT providers. Using an established inflation factor and controlling for attrition of 25%, the final sample-size calculation required 200 participants for the ACCESS arm and 120 for UC (320 total). Unequal randomization, an accepted practice in clinical trials, was used to reduce unnecessary recruitment without significant impact on power. Randomization was stratified by medical condition (COPD only, HF only, both COPD and HF) and by site, using random block sizes of 5 or 10. Because of the higher number of patients in Houston, 200 of the 320 total patients will be randomized at Houston and 120 at Oklahoma City. Random allocation lists were generated using SAS statistical software (SAS Institute, Inc., Cary, NC, USA). Randomization numbers are drawn from envelopes stuffed by non-project staff and opened only after a participant is deemed fully eligible and ready for group assignment.
Clinical effectiveness: outcome measures
Clinical effectiveness will be evaluated posttreatment (at 4 months) and at 8- and 12-month follow-ups by trained independent evaluators not associated with other aspects of the study. Assessment measures were selected to evaluate both physical and emotional health constructs related to COPD/HF patient-centered outcomes (Chronic Respiratory Questionnaire, Kansas City Cardiomyopathy Questionnaire), anxiety, and depression (Patient Health Questionnaire-9 (PHQ-9)). These effectiveness measures have strong psychometric properties and possess a high degree of external validity for clinical use in the primary care setting. For example, within the VA, the PHQ-9 is the standard measure for depression screening and intervention outcome evaluation. Other study variables include demographic variables, COPD/HF severity, presence/absence of coexistent psychiatric and medical conditions, psychotropic medication use, and health-services use, including ambulatory and inpatient care, primary care, and specialty care. Treatment intensity (“dose”) will be measured by total number of bCBT sessions.
In addition to the above quantitative measures, semistructured-interview data are being collected from patients and providers regarding the impact of the intervention on physical and emotional health and identification of salient aspects of the treatment. Interviews will be conducted with all enrolled clinicians and a purposeful sample of 5 to 10 patients identified during the later stages of the trial. Patients are selected based on ACCESS clinician recommendations. Clinicians also refer ACCESS patients who recently completed treatment and might be able to share additional information about the intervention (whether positive or negative and regardless of treatment engagement).
Clinical effectiveness: analyses
All quantitative analyses will be done on an intention-to-treat basis, and participants will be analyzed in the group to which they were randomized. Absolute differences in outcome measures will be examined between baseline and 12-month follow-up, and the effect sizes of the bCBT group will be compared with those of participants in the UC group. We will also examine effect sizes for the bCBT group baseline versus posttreatment and eight-month follow-up. To compare changes between the two groups over time, we will use longitudinal, mixed-model analyses that allow nesting of patients by medical center site.
Qualitative data will be collected using audiotaped interactions of the semistructured interview, transcribed and coded for themes, in combination with interviewer (field) notes. Using content-analysis methods, we will code textual data from the patients’ and providers’ perspectives to uncover patterns in the experiences of the bCBT intervention[59–62]. Coding and analysis will be conducted using formal qualitative-research methods. It is anticipated that qualitative data will provide details about the meaning of the intervention and its context within primary care that are unattainable using survey methods.
Objective #2: implementation strategy and evaluation of adoption and implementation
Objective #2 seeks to pilot test and evaluate the acceptability, feasibility and preliminary outcomes of a comprehensive implementation strategy designed to increase bCBT engagement, adherence, and quality. The development and administration of these procedures are viewed as important to understanding the barriers and facilitators associated with access, adoption, and fidelity of bCBT. Given the developmental nature of this work, a case-study design was used without a clinician control group.
Partnership development and implementation framework
Implementation planning for this study began with the construction of multiple stakeholder partnerships within the VA PC-MHI program. Study investigators with knowledge and prior practice experience in PC-MHI actively worked with PC-MHI clinicians and directors at both intervention sites to examine interests and needs related to evidence-based bCBT practices. The project team worked with these frontline stakeholders to identify needs and practice barriers to help inform the current project.
The ACCESS implementation strategy was informed by the Promoting Action on Research Implementation in Health Services (PARiHS) framework[63, 64]. According to PARiHS, successful implementation (SI) is a function (f) of Evidence (E), Context (C), and Facilitation (F). Evidence refers to sources of knowledge and the importance of this knowledge as perceived by stakeholders. Context encompasses the environment or setting in which the improvement program is implemented. Facilitation refers to a technique by which an individual or group makes adoption or implementation easier for others, achieved through support in terms of attitudes, habits, skills, and ways of thinking and working.
Based on stakeholder feedback and project-team experiences, the implementation strategy for this trial was developed to include three separate but interrelated interventions—online clinician training, clinician audit and feedback, and internal and external facilitation. Each element was developed within the PARiHS model and augmented by the available implementation literature. The project team focused on competing demands for depression treatment in the primary care setting—namely, constructs related to patients, providers, and the larger primary care clinical setting. To address these three areas in accordance with the PARiHS framework, the implementation strategy included (1) online clinician training to address confidence and competency, as well as potential patient barriers related to presenting concerns, attitudes about mental health treatment, and initial goal-setting strategies to increase treatment engagement[45, 66]; (2) audit and feedback of clinician session content, as provided by expert review[26, 67] (audit and feedback was viewed as a critical element to not only assess treatment fidelity but also to assist clinicians by providing timely feedback about performance both positive and negative); and (3) internal and external facilitation to support CBT use, including identification of system/clinic barriers. Facilitation that represents a package of well-known change strategies employed by the facilitator to address individual or site-specific issues at the right time to promote change adoption was seen as the primary implementation intervention to address barriers at the system level, including difficulties with patient scheduling and provider time allowances for therapy.
In addition to these elements, ACCESS engages the participation of an advisory council consisting of various stakeholders from PC-MHI programs, as well as regional and national mental health leaders within the VA. Given the pilot nature of this implementation strategy, particular emphasis has been placed on understanding stakeholder perspectives, using formative (pre-implementation) and process (during implementation) evaluations such that the implementation interventions could be modified as needed during the trial.
The study has targeted the inclusion of 12 PC-MHI clinicians between the two sites. The project team is working collaboratively with local stakeholders and clinic directors to identify clinicians best suited to provide psychotherapy services. Clinician enrollment is not influenced by prior psychotherapy or CBT knowledge and skill. However, clinicians are expected to have an interest in using bCBT in their daily practice and are asked to voluntarily participate and formally consent to participate. Nurses, social workers, psychologists, counselors, nurse practitioners, and physician assistants will potentially be included in the trial. Pretrial data will be collected for all study providers to help identify unequal entry-level bCBT expertise in care providers. Follow-up fidelity data will be collected throughout the project to document provider skill and intervention adherence.
Implementation intervention #1: ACCESS (bCBT) online clinician training
All study therapists participate in a comprehensive bCBT training program based on methods developed in a prior study. Print-based intervention materials (e.g., clinician manual and patient workbook) provide a foundational knowledge of the intervention. Print-based materials are augmented by an online training course (www.vaprojectaccess.org) consisting of narrated audio slides and audio vignettes to elaborate on critical intervention elements (detailed in a separate article—Cully et al.).
Implementation intervention #2: audit and feedback
Following the introductory training, clinicians receive audit and feedback on their first ACCESS therapy patient. Clinicians audiotape each session and have those audiotapes reviewed by a bCBT expert for adherence and skillfulness, using a standard rating scale[19, 26, 68]. Feedback from these evaluations is provided to the clinician after session 2 (completion of the core modules) and again at the conclusion of treatment. After completion of the first patient, clinicians continue to audiotape sessions, with audit and feedback occurring for a random selection and feedback on no less than two sessions every four months. While clinicians are not required to maintain minimal levels of adherence or skillfulness to remain in the trial (thus allowing for the evaluation of the intervention under real-world circumstances), those with scores falling below an a priori minimum performance standard are provided with increased frequency of audit and feedback.
Implementation intervention #3: internal and external facilitation
External facilitation is provided to both intervention sites by bCBT experts/trainers as pilot tested by Kauth et al.. The exact nature and content of external facilitation varies between sites, depending on clinician needs and logistical constraints. In general, external facilitators engage clinicians in regular individual or group meetings and less formally through telephone and email communication. The purpose of external facilitation is to provide clinicians with opportunities to address questions or concerns about the intervention, disseminate key information related to implementation, and create a sense of colleageality and community among the clinicians. Although external facilitators will address topics related to ACCESS intervention procedures and strategies, they focus on assisting clinicians to adopt and maintain fidelity of bCBT in their daily practice.
Internal facilitation efforts, although not a robust implementation strategy for this study, consist of engaging PC-MHI directors to facilitate the adoption of bCBT within each site’s PC-MHI clinic. Study external facilitators work with PC-MHI directors (internal facilitators) to share information related to the project and encourage the directors to provide resources to clinicians to facilitate training and use of bCBT. External facilitators and study personnel also engage internal facilitators at both sites to streamline clinical processes and increase adoption, as dictated by the unique needs at each site.
Study advisory council
A team of researchers and “end users,” including VA clinical managers and regional and national VA mental health leaders, have agreed to participate as study advisors to monitor progress and provide feedback on adoption and implementation of the bCBT intervention. The advisory council will meet semiannually throughout the study. During these meetings, the advisory council will receive updates on study progress, and notable occurrences will be discussed. The primary purpose of these meetings is to identify and address challenges and effective practice patterns to improve adoption of bCBT in other primary care settings within the VA. The advisory council also serves as an advocate for change and as a conduit for dissemination of study findings.
Implementation evaluation plan and analyses
Engagement and adherence will be measured for all participants randomized into bCBT. Patient engagement is defined as the percentage of randomized patients that attend at least one CBT session. Adherence is defined as the completion of four or more active-treatment sessions. We will compare the percentage of patients with engagement and adherence by bCBT clinician and site and subsequently examine overall rates to those found in similar brief therapy trials. The study advisory committee will review these outcomes and make recommendations as to the “clinical significance” of these findings relative to VA policy and current initiatives in PC-MHI.
We will also use clinician survey data to examine implementation success according to principles contained in the PARiHS framework, as detailed by Stetler et al., 2011. Survey data will elicit provider Likert-style responses to domains related to consistency of ACCESS procedures with available evidence-based mental health treatments (PARiHS evidence) and suitability of ACCESS for PC-MHI (PARiHS context), as well as detailed questions eliciting feedback and perceptions of the various implementation interventions (PARiHS facilitation). Although not the focus of the study, amount of time spent conducting the implementation interventions will be collected.
Qualitative methods (individual interviews) with bCBT stakeholders (patients, clinicians, and clinic directors and managers) will be used to elicit information on perceptions of bCBT importance, potential for adoption, and potential barriers to implementation. Individual interviews will be conducted using a semistructured interview format and conducted by an experienced psychotherapist. The semistructured interview format will allow for flexibility to explore each individual stakeholder’s perspectives. These data, which are not attainable through traditional survey methods, will provide another level of data on acceptability, feasibility, and impact of the implementation interventions to help determine the potential for wider use of the implementation strategy in future trials.
Finally, taking advantage of the longer-term posttreatment patient follow-up period, we will examine clinician bCBT utilization rates six months postcompletion of all study implementation efforts. Using a self-report survey, we will measure frequency of bCBT procedures and techniques used by clinicians to document perceived practice changes during and after implementation.
Hybrid E-I designs have the potential to rapidly advance the utilization of evidence-based practices for complex mental health treatments such as bCBT. However, impact of these designs is strongly tied to the project team’s ability to form meaningful partnerships with stakeholders, including patients, clinicians, administrators, and policy makers. Effective partnerships are more likely to translate into a collaborative team-based approach and a shared vision. From this shared vision, the team can more effectively address the logistical and methodological challenges associated with administering interventions and changing practice patterns. Although the “upfront cost” of building these partnerships is significant, these collaborations are more likely to generate high-quality treatments that are feasible for frontline practice settings and may avoid common pitfalls associated with post hoc adaptations to efficacy-based interventions, especially the threat to validity that commonly occurs when translating evidence to practice. Although hybrid designs may aid in mitigating the current science-to-service gaps, the application of these designs requires thoughtful planning and potentially difficult methodological compromises.
The ACCESS project began with grant preparations that engaged local, regional, and national stakeholders. Stakeholders unanimously agreed that bCBT interventions were needed for the PC-MHI setting and that clinicians would benefit from having additional resources to learn and apply these treatments. Given this shared vision, the project team worked actively with stakeholders to create a clinical intervention and implementation strategy that was viewed as feasible and meaningful. The project team was then able to focus on addressing the methodological challenges of blending a study with both effectiveness and implementation goals. For the ACCESS project, the team thought it critical to test a set of clinical and implementation interventions that would provide data applicable for the scientific, clinical, administrative, and policy communities. At its core, the ACCESS project seeks to provide knowledge that will lead to improvements in the provision of mental healthcare for the primary care setting.
The following sections provide examples of the important methodological compromises identified and addressed in the current study, focusing on the balance between internal validity and improvement of care processes. Consistent with other hybrid studies, ACCESS requires a longer-term study period (four years) and considerable financial and human capital resources. The trial also targets a large number of study participants (N = 320), which allows greater statistical power to examine mediators and moderators to balance the need for effectiveness and implementation outcomes simultaneously.
Methodological challenge #1: patient identification, recruitment, and inclusion criteria
Traditional efficacy designs are conducted with strict internal controls and rigorous patient inclusion and exclusion criteria, often at the expense of restricting applicability to clinical practice. The current trial uses relaxed inclusion and exclusion criteria to more closely align study participants with typical patients seen by PC-MHI clinicians in their daily practice. The study also uses symptom (rather than diagnostic) assessment tools commonly used in PC-MHI.
Patient evaluations are conducted by independent evaluators and allow separation of the clinical and evaluation components of the trial. Most importantly, these evaluations serve as a method for obtaining detailed clinical information about patients, which can be used during secondary mediator and moderator analyses without posing unnecessary restrictions on participant inclusion criteria.
Methodological challenge #2: clinical intervention and interventionists
Clinical interventions must possess replicable procedures but also have high levels of external validity and be feasible within the predetermined practice environment. The clinical intervention for the current trial was developed as a brief technique to closely align with PC-MHI care models. To increase flexibility and acceptability, the clinical intervention includes options for telephone sessions and allows clinicians and patients to select modules (content) to best address the patient’s individual needs. Although intervention flexibility is viewed as congruent with real-world practice, it also creates variation in the content and delivery. To retain these core treatment aspects while retaining moderate internal controls, the project uses comprehensive data-collection strategies, such as medical-record reviews and fidelity ratings of session content, to describe and statistically account for variability in treatment delivery in secondary analyses.
Methodological challenge #3: implementation strategy and modification of clinic procedures
The immediate aim of the implementation strategy is to provide PC-MHI clinicians and clinics with the necessary support to conduct bCBT. In the current trial, a developmental implementation strategy is being pilot tested for acceptability, feasibility, and preliminary outcomes. Unlike the clinical intervention, which will remain unchanged throughout the study, the implementation interventions are conceptualized as developmental, given the potential for variability in practice patterns and needs within the two PC-MHI sites. In developing the implementation interventions, the project team (including stakeholders) targeted learning opportunities that could be feasibly embedded within the existing PC-MHI structure. We also sought to create objective procedures to be tested and refined for future use in a more comprehensive implementation trial. Lastly, the evaluation team intends to rely heavily on stakeholders and the advisory council to direct implementation efforts. As such, stakeholders will receive regular implementation updates, and we will elicit feedback from these groups to identify needed changes. The long-term implementation aim is to better understand the impact of a refined implementation strategy using a cluster randomized design. Ultimately, this work seeks to aid in the identification of best-practice guidelines for psychotherapy use in the primary care setting.
Opportunities for knowledge transfer and dissemination of project results
Transfer of information, including research findings, training, and clinical-care practices from this study, will be ongoing and multifaceted. The dual approach of the hybrid type 2 design will allow the project to meaningfully target frontline practitioners and managers, national scientific research journals and professional organizations, as well as national, regional, and local VA and other academic hospital leaders and policy makers.
The project team intends to disseminate published training and intervention materials to frontline practitioners both within and outside the VA through the established ACCESS website (www.vaprojectaccess.org), as well as through focused dissemination channels within the VA, including regional and national distributions through the VA Veteran Integrated Service Networks and Central Office administration and leadership. The advisory council will assist in dissemination of study findings, both within and beyond the VA.
Preliminary data from the trial will focus on implementation and training process characteristics, the continued advancement of partnerships, and baseline clinical information on patient functioning. Final study data will address the study objectives related to clinical effectiveness of bCBT and the utility and success of the piloted implementation strategy to enhance bCBT adoption. Dissemination of these project findings will not only target traditional scientific publication avenues and national presentations but will also seek to inform clinical leaders at the national, regional, and local levels of the VA.
Although hybrid E-I designs are often complicated by methodological trade-offs, the opportunities for scientific and clinical practice advancements are significant. In the current trial, the dual focus on bCBT effectiveness and developmental implementation strategies will afford the project team a unique opportunity to simultaneously understand patient outcomes and clinical practices challenges. Further, given the structure of the research design, the project is likely to provide meaningful implementation data and scientific findings, regardless of the patient clinical outcomes. The implementation approach, constructed with active involvement of multiple stakeholders and end users, was broadly constructed around increasing the utilization of evidence-based psychotherapies within primary care. As such, the formative aspect of the implementation strategy will generate data that are distinct from the patient clinical-effectiveness aspects of the study and may inform future, more rigorous implementation studies.
Corresponding to its desire to expand access to evidence-based mental health treatments for veterans, the VA is ideally situated and currently needs identification of feasible evidence-based treatments for use within the primary care setting. Hybrid designs like the one described in this study have the potential to rapidly advance the knowledge of both the clinical effectiveness of bCBT and the possible implementation of best practices necessary to support the use of this complex intervention in real-world care.
The authors would like to thank Sonora Hudson for her assistance in preparation of this manuscript. We would also like to thank our clinical partners, including Candy Smith, Pamela Fischer, and all of the mental health practitioners at the PC-MHI study sites.
This material is based upon work supported by the Department of Veterans Affairs (HSR&D Grant IIR 09-088). It was also partly supported by the Veterans Health Administration, Office of Research and Development and the Houston VA Health Services Research and Development Center of Excellence (HFP90-020) and the South Central Mental Illness, Research, Education, and Clinical Center. The views expressed in this article are those of the authors and do not necessarily reflect the position or policy of the Department of Veterans Affairs, the US government, or Baylor College of Medicine.
- Clark DM, Salkovskis PM, Hackmann A, Middleton H, Anastasiades P, Gelder M: A comparison of cognitive therapy, applied relaxation and imipramine in the treatment of panic disorder. Br J Psychiatry. 1994, 164: 759-769. 10.1192/bjp.164.6.759.View ArticlePubMedGoogle Scholar
- Gloaguen V, Cottraux J, Cucherat M, Blackburn IM: A meta-analysis of the effects of cognitive therapy in depressed patients. J Affect Disord. 1998, 49: 59-72. 10.1016/S0165-0327(97)00199-7.View ArticlePubMedGoogle Scholar
- Stanley MA, Beck JG, Novy DM, Averill PM, Swann AC, Diefenbach GJ: Cognitive-behavioral treatment of late-life generalized anxiety disorder. J Consult Clin Psychol. 2003, 71: 309-319. Retrieved from PM:12699025View ArticlePubMedGoogle Scholar
- Wenzel A, Brown GK, Karlin BE: Cognitive behavioral therapy for depression in veterans and military service members: therapist manual. 2011, Washington, D.C.: U.S. Department of Veterans AffairsGoogle Scholar
- Whitfield G, Williams C: The evidence base for cognitive-behavioural therapy in depression: delivery in busy clinical settings. Adv Psychiatr Treat. 2003, 9: 21-30. 10.1192/apt.9.1.21. Retrieved fromhttp://apt.rcpsych.org/content/9/1/21.abstract,View ArticleGoogle Scholar
- Cuijpers P, van Straten A, van Schaik A, Andersson G: Psychological treatment of depression in primary care: a meta-analysis. B J Gen Pract. 2009, 59: e51-e60.View ArticleGoogle Scholar
- Wolf NJ, Hopko DR: Psychosocial and pharmacological interventions for depressed adults in primary care: A critical review. Clin Psychol Rev. 2008, 28: 131-161. 10.1016/j.cpr.2007.04.004. Retrieved fromhttp://www.sciencedirect.com/science/article/pii/S0272735807000785,View ArticlePubMedGoogle Scholar
- Cully J, Tolpin L, Jimenez D, Kunik M, Petersen L: Psychotherapy in the Veterans Health Administration: missed opportunities?. Psychol Serv. 2008, 55: 320-331.View ArticleGoogle Scholar
- Glasgow RE, Lichtenstein E, Marcus AC: Why don't we see more translation of health promotion research to practice? Rethinking the efficacy-to-effectiveness transition. Am J Public Health. 2003, 93: 1261-1267. 10.2105/AJPH.93.8.1261.View ArticlePubMedPubMed CentralGoogle Scholar
- Alexander CL, Arnkoff DB, Glass CR: Bringing psychotherapy to primary care: innovations and challenges. Clin Psychol. 2010, 17: 191-214. 10.1111/j.1468-2850.2010.01211.x. Retrieved fromhttp://dx.doi.org/10.1111/j.1468-2850.2010.01211.x,Google Scholar
- Funderburk JS, Sugarman DE, Labbe AK, Rodrigues A, Maisto SA, Nelson B: Behavioral health interventions being implemented in a VA primary care system. J Clin Psychol Med Settings. 2011, 18: 22-29. 10.1007/s10880-011-9230-y. Retrieved fromhttp://www.springerlink.com/index/10.1007/s10880-011-9230-y,View ArticlePubMedGoogle Scholar
- Veterans Health Administration: Uniform mental health services in VA medical centers and clinics (VHA Handbook 1160.01). 2008, Washington, DC: Department of Veterans AffairsGoogle Scholar
- Karlin BE, Ruzek JI, Chard KM, Eftekhari A, Monson CM, Hembree EA, Resick PA, Foa EB: Dissemination of evidence-based psychological treatments for posttraumatic stress disorder in the Veterans Health Administration. J Trauma Stress. 2010, 23: 663-673. 10.1002/jts.20588. Retrieved fromhttp://dx.doi.org/10.1002/jts.20588,View ArticlePubMedGoogle Scholar
- McHugh RK, Barlow DH: The dissemination and implementation of evidence-based psychological treatments. A review of current efforts. Am Psychol. 2010, 65: 73-84.View ArticlePubMedGoogle Scholar
- Unützer J, Katon W, Sullivan M, Miranda J: Treating depressed older adults in primary care: narrowing the gap between efficacy and effectiveness. Milbank Q. 1999, 77: 225-256. 10.1111/1468-0009.00132.View ArticlePubMedPubMed CentralGoogle Scholar
- Bower P, Gilbody S: Managing common mental health disorders in primary care: conceptual models and evidence base. BMJ. 2005, 330 (7495): 839-842. 10.1136/bmj.330.7495.839.View ArticlePubMedPubMed CentralGoogle Scholar
- Nieuwsma JA, Trivedi RB, McDuffie J, Kronish I, Benjamin D, Williams JW: Brief psychotherapy for depression in primary care: a systematic review of the evidence (VA-ESP Project #09-010). 2011, Washington, DC: Department of Veterans Affairs, Veterans Health AdministrationGoogle Scholar
- Cape J, Whittington C, Buszewicz M, Wallace P, Underwood L: Brief psychological therapies for anxiety and depression in primary care: meta-analysis and meta-regression. BMC Med. 2010, 8: 38-10.1186/1741-7015-8-38.View ArticlePubMedPubMed CentralGoogle Scholar
- Cully JA, Stanley MA, Deswal A, Hanania NA, Phillips LL, Kunik ME: Cognitive-behavioral therapy for chronic cardiopulmonary conditions: Moving beyond mental health outcomes. Prim Care Companion J Clin Psychiatry. 2010, 12 (4): e1-e6.Google Scholar
- Bartels SJ, Coakley EH, Zubritsky C, Ware JH, Miles KM, Areán PA, Chen H, Oslin DW, Llorente MD, Costantino G, Quijano L, McIntyre JS, Linkins KW, Oxman TE, Maxwell J, Levkoff SE, PRISM-E Investigators: Improving access to geriatric mental health services: a randomized trial comparing treatment engagement with integrated versus enhanced referral care for depression, anxiety, and at-risk alcohol use. Am J Psychiatry. 2004, 161: 1455-1462. 10.1176/appi.ajp.161.8.1455.View ArticlePubMedGoogle Scholar
- Lester H, Tritter JQ, Sorohan H: Patients' and health professionals' views on primary care for people with serious mental illness: focus group study. BMJ. 2005, 330: 1122-10.1136/bmj.38440.418426.8F.View ArticlePubMedPubMed CentralGoogle Scholar
- Robinson P, Reiter J: Behavioral consultation and primary care: a guide to integrating services. 2007, New York: SpringerView ArticleGoogle Scholar
- Robinson P: Adapting empirically supported treatments to the primary care setting: a template for success. Behavioral Integrative Care: Treatments That Work in the Primary Care Setting. Edited by: O'Donahue W, Byrd M, Cummings N, Henderson D. 2005, New York: Brunner-Routledge, 53-71.Google Scholar
- Roy-Byrne P, Craske MG, Sullivan G, Rose RD, Edlund MJ, Lang AJ, Bystritsky A, Welch SS, Chavira DA, Golinelli D, Campbell Sills L, Sherbourne C, Stein MB: Delivery of evidence-based treatment for multiple anxiety disorders in primary care. JAMA. 2010, 303: 1921-1928. 10.1001/jama.2010.608. Retrieved fromhttp://jama.ama-assn.org/content/303/19/1921.abstract,View ArticlePubMedPubMed CentralGoogle Scholar
- Roy-Byrne PP, Craske MG, Stein MB, Sullivan G, Bystritsky A, Katon W, Golinelli D, Sherbourne CD: A randomized effectiveness trial of cognitive-behavioral therapy and medication for primary care panic disorder. Arch Gen Psychiatry. 2005, 62: 290-298. 10.1001/archpsyc.62.3.290. Retrieved fromhttp://archpsyc.ama-assn.org/cgi/content/abstract/62/3/290,View ArticlePubMedPubMed CentralGoogle Scholar
- Stanley MA, Wilson NL, Novy DM, Rhoades HM, Wagener PD, Greisinger AJ, Cully JA, Kunik ME: Cognitive behavior therapy for generalized anxiety disorder among older adults in primary care. JAMA. 2009, 301: 1460-1467. 10.1001/jama.2009.458. Retrieved fromhttp://jama.ama-assn.org/content/301/14/1460.abstract,View ArticlePubMedPubMed CentralGoogle Scholar
- Willemse GRWM, Smit F, Cuijpers P, Tiemens BG: Minimal-contact psychotherapy for sub-threshold depression in primary care. Br J Psychiatry. 2004, 185: 416-421. 10.1192/bjp.185.5.416. Retrieved fromhttp://bjp.rcpsych.org/content/185/5/416.abstract,View ArticlePubMedGoogle Scholar
- Institute of Medicine: Crossing the quality chasm: a new health system for the 21st century. 2001, Washington, D.C.: National Academies PressGoogle Scholar
- Flay BR: Efficacy and effectiveness trials (and other phases of research) in the development of health promotion programs. Prev Med. 1986, 15: 451-474. 10.1016/0091-7435(86)90024-1.View ArticlePubMedGoogle Scholar
- Wells KB: Treatment research at the crossroads: the scientific interface of clinical trials and effectiveness research. Am J Psychiatry. 1999, 156: 5-10.View ArticlePubMedGoogle Scholar
- Curran GM, Bauer M, Mittman B, Pyne JM, Stetler C: Effectiveness-implementation hybrid designs: combining elements of clinical effectiveness and implementation research to enhance public health Impact. Med Care. 2012, 50: 217-226. 10.1097/MLR.0b013e3182408812.View ArticlePubMedPubMed CentralGoogle Scholar
- Farris RP, Will JC, Khaviou O, Finkelstein EA: Beyond effectiveness: evaluating the public health impact of the WISEWOMAN program. Am J Public Health. 2007, 97 (4): 641-647. 10.2105/AJPH.2005.072264.View ArticlePubMedPubMed CentralGoogle Scholar
- Glasgow RE, Vogt TM, Boles SM: Evaluating the public health impact of health promotion interventions: the RE-AIM framework. Am J Public Health. 1999, 89: 1322-1327. 10.2105/AJPH.89.9.1322.View ArticlePubMedPubMed CentralGoogle Scholar
- Glasgow RE, McKay HG, Piette JD, Reynolds KD: The RE-AIM framework for evaluating interventions: what can it tell us about approaches to chronic illness management?. Patient Educ Couns. 2001, 44: 119-127. 10.1016/S0738-3991(00)00186-5. Retrieved from PM:11479052View ArticlePubMedGoogle Scholar
- Hallfors D, Cho H: Moving behavioral science from efficacy to effectiveness. Int J Behav Consult Ther. 2007, 3: 236-250.View ArticleGoogle Scholar
- Jenkinson KA, Naughton G, Benson AC: The GLAMA (Girls! Lead! Achieve! Mentor! Activate!) physical activity and peer leadership intervention pilot project: a process evaluation using the Re-AIM framework. BMC Publ Health. 2012, 12: 55-10.1186/1471-2458-12-55.View ArticleGoogle Scholar
- Jemal A, Ward E, Hao Y, Thun M: Trends in the leading causes of death in the United States, 1970-2002. JAMA. 2005, 294: 1255-1259. 10.1001/jama.294.10.1255.View ArticlePubMedGoogle Scholar
- O’Connell JB: The economic burden of heart failure. Clin Cardiol. 2000, 23 (3Suppl): III6-III10.PubMedGoogle Scholar
- World Health Organization: Chronic Respiratory Disease. Burden of COPD. 2012, Available athttp://www.who.int/respiratory/copd/burden/en/index.html. Accessed 4/9/12,
- Cully JA, Graham DP, Stanley MA, Ferguson CJ, Sharafkhaneh A, Souchek J, Kunik ME: Quality of life in patients with chronic obstructive pulmonary disease and comorbid anxiety or depression. Psychosomatics. 2006, 47: 312-319. 10.1176/appi.psy.47.4.312.View ArticlePubMedGoogle Scholar
- Cully J, Jimenez D, LeDoux T, Deswal A: Recognition and treatment of depression and anxiety symptoms in heart failure. Prim Care Companion J Clin Psychiatry. 2009, 11: 103-109. 10.4088/PCC.08m00700.View ArticlePubMedPubMed CentralGoogle Scholar
- Kunik ME, Roundy K, Veazey C, Souchek J, Richardson P, Wray NP, Stanley MA: Surprisingly high prevalence of anxiety and depression in chronic breathing disorders. Chest. 2005, 127: 1205-1211. 10.1378/chest.127.4.1205.PubMedGoogle Scholar
- Cully JA, Johnson M, Moffett M, Khan M, Deswal A: Depression and anxiety in ambulatory patients with heart failure. Psychosomatics. 2009, 50: 592-598.View ArticlePubMedGoogle Scholar
- DeRubeis RJ, Crits-Christoph P: Empirically supported individual and group psychological treatments for adult mental disorders. J Consult Clin Psychol. 1998, 66: 37-52.View ArticlePubMedGoogle Scholar
- Cully JA, Teten AT, Benge J, Sorocco K, Kauth M: Multidisciplinary cognitive behavioral therapy training for the VA primary care setting. Prim Care Companion J Clin Psychiatry. 2010, 12 (3): e1-e8.Google Scholar
- Kauth MR, Sullivan G, Blevins D, Cully JA, Landes RD, Said Q, Teasdale TA: Employing external facilitation to implement cognitive behavioral therapy in VA clinics: a pilot study. Implement Sci. 2010, 5: 75-10.1186/1748-5908-5-75.View ArticlePubMedPubMed CentralGoogle Scholar
- Kunik ME, Veazey C, Cully JA, Souchek J, Graham DP, Hopko D, Carter R, Sharafkhaneh A, Goepfert EJ, Wray N, Stanley MA: COPD education and cognitive behavioral therapy group treatment for clinically significant symptoms of depression and anxiety in COPD patients: a randomized controlled trial. Psychol Med. 2008, 38: 385-396.View ArticlePubMedGoogle Scholar
- Lustman PJ, Griffith LS, Freedland KE, Kissel SS, Clouse RE: Cognitive behavior therapy for depression in type 2 diabetes mellitus. A randomized, controlled trial. Ann Intern Med. 1998, 129: 613-621.View ArticlePubMedGoogle Scholar
- Mignogna JM, Cully JA: Depression and anxiety in patients with COPD: a focus on psychological treatments in ambulatory care settings. Curr Respir Med Rev. 2012, 8: 137-144.View ArticleGoogle Scholar
- Cully JA, Paukert A, Falco J, Stanley MA: Cognitive-behavioral therapy: innovations for cardiopulmonary patients with depression and anxiety. Cogn Behav Pract. 2009, 16: 394-407. 10.1016/j.cbpra.2009.04.004.View ArticleGoogle Scholar
- Sheehan DV, Lecrubier Y, Sheehan KH, Janavs J, Weiller E: Comparison of the Mini International Neuropsychiatric Interview (MINI) with the SCID-P and the CIDI: a validity study. Psychopharmacol Bull. 1995, 31: 616-Google Scholar
- Schnurr PP, Friedman MJ, Engel CC, Foa EB, Shea MT, Resick PM, James KE, Chow BK: Issues in the design of multisite clinical trials of psychotherapy: VA Cooperative Study No. 494 as an example. Contemp Clin Trials. 2005, 26: 626-636. 10.1016/j.cct.2005.09.001.View ArticlePubMedGoogle Scholar
- Dumville JC, Hahn S, Miles JNV, Torgerson DJ: The use of unequal randomization ratios in clinical trials: a review. Contemp Clin Trials. 2007, 27: 1-12.View ArticleGoogle Scholar
- Guyatt GH, Berman LB, Townsend M, Pugsley SO, Chambers LW: A measure of quality of life for clinical trials in chronic lung disease. Thorax. 1987, 42: 773-778. 10.1136/thx.42.10.773.View ArticlePubMedPubMed CentralGoogle Scholar
- Green CP, Porter CB, Bresnahan DR, Spertus JA: Development and evaluation of the Kansas City Cardiomyopathy Questionnaire: a new health status measure for heart failure. J Am Coll Cardiol. 2000, 35: 1245-1255. 10.1016/S0735-1097(00)00531-3.View ArticlePubMedGoogle Scholar
- Beck AT, Steer RA: Beck anxiety inventory: manual. 1990, San Antonio: Harcourt Assessment, Inc.Google Scholar
- Spitzer RL, Kroenke K, Williams JB: Validation and utility of a self-report version of PRIME-MD: the PHW primary care study. Primary Care Evaluation of Mental Disorders. Patient Health Questionnaire. JAMA. 1999, 282: 1734-1744.View ArticleGoogle Scholar
- Oslin DW, Ross J, Sayers S, Murphy J, Kane V, Katz IR: Screening, assessment and management of depression in VA primary care clinics. J Gen Intern Med. 2006, 21: 46-50. 10.1111/j.1525-1497.2005.0267.x.View ArticlePubMedPubMed CentralGoogle Scholar
- Patton MQ: Qualitative evaluation and research methods. 2002, Thousand Oaks: Sage, 3Google Scholar
- Lincoln YS, Guba G: Naturalistic inquiry. 1985, Beverly Hills: SageGoogle Scholar
- Gubrium JF, Holstein JA: Handbook of interview research. 2002, Thousand Oaks: SageGoogle Scholar
- Miles MB, Huberman AM: Qualitative data analysis. 1994, Thousand Oaks: Sage, 2Google Scholar
- Kitson A, Rycroft-Maline J, Harvey G, McCormack B, Seers K, Titchen A: Evaluating the successful implementation of evidence into practice using the PARIHS framework: theoretical and practical challenges. Implement Sci. 2008, 3 (1): 1-10.1186/1748-5908-3-1.View ArticlePubMedPubMed CentralGoogle Scholar
- Stetler CB, Damschroder LJ, Helfrich CE, Hagedorn HJ: A guide for applying a revised version of the PARIHS framework for implementation. Implement Sci. 2011, 6: 99-10.1186/1748-5908-6-99.View ArticlePubMedPubMed CentralGoogle Scholar
- Klinkman MS: Competing demands in psychosocial care: a model for the identification and treatment of depressive disorders in primary care. Gen Hosp Psychiatry. 1997, 19: 98-111. 10.1016/S0163-8343(96)00145-4.View ArticlePubMedGoogle Scholar
- Cully JA, Curry AD, Ryan SR, Malik A, Zeno D, Willcockson I: Develoment of a computer-aided training program for brief cognitive-behavioral therapy in primary care. Acad Psychiatry. 2011, in pressGoogle Scholar
- Jamtvedt G, Young JM, Kristoffersen DT, O’Brien MA, Oxman AD: Audit and feedback: effects on professional practice and health care outcomes. Cochrane Database Syst Rev. 2006, CD000259-2Google Scholar
- Cully JA, Mignogna J, Stanley MA, Davila J, Wear J, Amico KR, Giordano TP: Development and pilot testing of a standardized training program for a patient-mentoring intervention to increase adherence to outpatient HIV care. AIDS Patient Care STDS. 2012, 26 (3): 165-172. 10.1089/apc.2011.0248.View ArticlePubMedPubMed CentralGoogle Scholar
- Kauth M, Sullivan G, Cully JA, Blevins D: Facilitating mental health practice changes: a practical guide for developing implementation plans. Prof Psychol Res Pr. 2011, 8: 36-47.Google Scholar
- Naik AD, Isaac TT, Street RL, Kunik ME: Understanding the quality chasm for hypertension control in diabetes: a structured review of “comaneuvers” used in clinical trials. J Am Board Fam Med. 2007, 20: 469-478. 10.3122/jabfm.2007.05.070026.View ArticlePubMedPubMed CentralGoogle Scholar
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.