Improving care quality through hybrid implementation/effectiveness studies: Best practices in design, methods, and measures
- Amy N Cohen1, 2Email author,
- Alison B Hamilton2, 3,
- Mona Ritchie4, 5,
- Brian S Mittman6, 7,
- JoAnn E Kirchner4, 5,
- Gail E Wyatt2,
- John C Fortney8, 9,
- Gerhard Hellemann10,
- Honghu Liu11, 12,
- Geoffrey M Curran13,
- Fiona Whelan10,
- Alicia M Eccles2,
- Louise E Parker4, 14,
- Kirk McNagny15, 16,
- Craig S Hutchinson2,
- Annapurni B Teague17,
- Christopher Reist16, 18 and
- Alexander S Young1, 19
© Cohen et al. 2015
Published: 14 August 2015
Panel overview abstract
Implementation research is the scientific study of methods that promote systematic uptake of research findings and other evidence-based practices into routine practice, thereby improving the quality and effectiveness of health services. As the field has progressed over the past decades, substantial advances continue in the development and application of implementation-related theories as well as innovative implementation strategies and methods.
This session will provide three examples of implementation research studies that are designed to improve care through the use of evidence. The examples come from three health services areas: primary care, mental health, and HIV; from funded NIH and VA studies; and from community and hospital settings in rural and urban sites. Across the three examples, six different theories were used-providing ample examples of linking conceptual models and frameworks to study design, implementation strategies, and measurements. Each presentation will highlight the ways in which the conceptual theory was configured into the overall research design and evaluation, as well as common challenges and lessons learned while conducting implementation research.
Blended facilitation to enhance PCMH program implementation: conceptual, design, and measurement considerations
Integrated primary care mental health evidence-based programs improve care. The Department of Veterans Affairs mandated and provided limited national level implementation support for Primary Care - Mental Health Integration (PC-MHI) but VA facilities were slow to implement them. The Blended Facilitation study was funded by the VA to implement and evaluate an innovative implementation facilitation (IF) strategy that included a national external expert facilitator with expertise in implementation science and PC-MHI who mentored and worked with two internal regional facilitators to help clinics implement PC-MHI.
According to the PARIHS framework, successful implementation is a function of the dynamic interaction between evidence, context and facilitation. This presentation will describe how PARIHS guided the application of the IF strategy and the study's design. For this quasi-experimental, Hybrid Type 3 study, we used mixed methods to test the effectiveness of the IF strategy and assess organizational context, perceptions of evidence, and facilitation activities. We used a consensus matching approach to select sixteen clinics that were unlikely to implement PC-MHI without assistance across four VA regions. The RE-AIM framework guided our test of IF's effectiveness using administrative data and program component interviews to measure RE-AIM dimensions. We conducted monthly debriefing interviews with and collected time data from facilitators and measured site level contextual and evidence factors through key informant interviews. We also conducted intensive case studies at four IF clinics to assess stakeholders' perception of IF's processes and value.
There are limited examples of Hybrid Type 3 studies and this is an excellent large-scale example of one that also details how the conceptual frameworks guide design, strategies, and measurement. This presentation will discuss the strengths and weaknesses of the conceptual framework, the IF strategy and the study design we selected and lessons we learned about the challenges of conducting implementation research within the context of a VA policy initiative.
Implementing high priority evidence-based practices for individuals with schizophrenia: conceptual, design, and measurement considerations
Most individuals with schizophrenia receive about half of the indicated treatments that have been shown to improve quality of life and health. As a result, outcomes in routine practice are much worse than in state-of-the-art care. This gap in implementation- both in delivery and uptake of services-was addressed in the VA HSR&D QUERI-funded EQUIP study.
This Hybrid Type 2 study integrated two conceptual frameworks. The Simpson Transfer Model (STM) is a program change model that examines readiness to change at the organizational and provider levels and posits that research moves into practice through 5 phases: exposure, adoption, implementation, practice, and sustainability. Since the STM does not recommend specific behavior change tools, the PRECEDE model guided the choice of a multifaceted implementation strategy to influence the adoption of behavior changes necessary for implementation success and positive patient outcomes.
Across four states, eight specialty mental health programs were assigned to implementation or usual care. Two evidence-based services were targeted: Wellness and Supported Employment. Veterans with schizophrenia (n = 801) and clinicians (n = 201) were enrolled. At implementation sites, organizational readiness data were used to tailor implementation. The implementation strategy included patient-facing kiosks for routine assessment, evidence-based quality improvement methods, social marketing, opinion leaders, provider and patient education, and continual feedback to staff. Mixed methods were used to evaluate implementation and effectiveness. Patients and clinicians were surveyed and interviewed at baseline and 15 months. Intervention clinicians were also interviewed mid-study. A cost effectiveness evaluation was included.
This presentation will discuss the key decisions made in selecting the conceptual frameworks and how those decisions impacted the study design including preparation and implementation, measure choice, and assessment timing. This is one of the first studies to improve care quality in specialty mental health in diverse VA settings and therefore provides an excellent example for the application of integrated conceptual frameworks and a multifaceted implementation strategy.
Implementation and effectiveness of an evidence-based intervention in community-based organizations: conceptual, design, and measurement considerations
The HIV/AIDS epidemic continues to disproportionately affect African-American communities in the US, particularly those located in urban, resource-constrained areas. This five-year Hybrid Type 2 study investigates community-based implementation, effectiveness, and sustainability of 'Eban II,' an evidence-based risk reduction intervention for African-American heterosexual, serodiscordant couples.
Key decisions in the design of this study included how to balance investigating implementation and effectiveness (and resources associated with each), which models/theories to apply in order to guide organization-level implementation and couples-level behavioral change, which implementation strategies to select for optimal uptake, how to measure implementation, how to analyze organizational-level data, how to capture implementation costs, and how to conceptualize and measure sustainability. Each of these decisions will be described in this presentation. Specifically, we will describe our: hybrid design and dynamic waitlisted effectiveness study, application of the Program Change Model (PCM, a phased model of organizational change) and social cognitive theory, selection of implementation strategies and tools for each phase of the PCM, quantitative and qualitative implementation and effectiveness measures/instruments, multi-level hierarchical modeling with a multi-level nested structure analysis, measures of implementation costs and potential cost savings, and approach to sustainability as defined by research-independent delivery of the intervention and relative use of technical assistance.
This study provides an excellent example of a funded study using a hybrid design in resource-constrained community settings. The study provides an illustration of the value of community supports and expertise throughout the study to shape the design, implementation strategies, and measurements. The impact of the organizational feedback and contingencies will be briefly articulated with regard to lessons learned for future community-based implementation studies.
- Armstrong NP, Cohen AN, Hellemann G, Reist C, Young AS: Validating a brief version of the mental health recovery measure for individuals with schizophrenia. Psychiatr Serv. 2014, 65 (9): 1154-1159. 10.1176/appi.ps.201300215.View ArticlePubMedGoogle Scholar
- Brown AH, Cohen AN, Chinman MJ, Kessler C, Young AS: EQUIP: Implementing Chronic Care Principles and Applying Formative Evaluation Methods to Improve Care for Schizophrenia. Implement Sci. 2008, 3: 9-10.1186/1748-5908-3-9.View ArticlePubMedPubMed CentralGoogle Scholar
- Cohen AN, Chinman MJ, Hamilton AB, Whelan F, Young AS: Using Patient-Facing Kiosks to Support Quality Improvement at Mental Health Clinics. Med Care. 2013, 51 (3 Suppl 1): S13-S20.View ArticlePubMedPubMed CentralGoogle Scholar
- Hamilton AB, Mittman BS, Williams JK, Liu HH, Eccles AM, Hutchinson CS, Wyatt GE: Community-based implementation and effectiveness in a randomized trial of a risk reduction intervention for HIV-serodiscordant couples: study protocol. Implement Sci. 2014, 9: 79-10.1186/1748-5908-9-79.View ArticlePubMedPubMed CentralGoogle Scholar
- Hamilton AB, Cohen AN, Young AS: Organizational Readiness in Specialty Mental Health Care. J Gen Intern Med. 2010, 25 (Suppl 1): 27-31.View ArticlePubMedPubMed CentralGoogle Scholar
- Hamilton AB, Cohen AN, Glover DL, Whelan F, Chemerinski E, McNagny KP, et al: Implementation of Evidence-Based Employment Services in Specialty Mental Health: Mixed Methods Research. Health Serv Res. 2013, 48 (6 Pt 2): 2224-2244.View ArticlePubMedPubMed CentralGoogle Scholar
- Kirchner JE, Ritchie MJ, Pitcock JA, Parker LE, Curran GM, Fortney JC: Outcomes of a partnered facilitation strategy to implement primary care - mental health. J Gen Intern Med. 2014, 29 Suppl 4: 904-912.View ArticlePubMedGoogle Scholar
- Kirchner JE, Kearney LK, Ritchie MJ, Dollar KM, Swensen AB, Schohn M: Lessons learned through a national partnership between clinical leaders and researchers. Psychiatr Serv. 2014, 65 (5): 577-579. 10.1176/appi.ps.201400054.View ArticlePubMedGoogle Scholar
- Kirchner JE, Ritchie MJ, Dollar KM, Gundlach P, Smith JL: Implementation facilitation training manual: Using external and internal facilitation to improve care in the Veterans Health Administration. [http://www.queri.research.va.gov/tools/implementation/Facilitation-Manual.pdf]
- Ritchie MJ, Dollar KM, Kearney LK, Kirchner JE: Responding to needs of clinical operations partners: Transferring implementation facilitation knowledge and skills. Psychiatr Serv. 2014, 65 (2): 141-143. 10.1176/appi.ps.201300468.View ArticlePubMedGoogle Scholar
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.