Proceedings from the 9^th annual conference on the science of dissemination and implementation

A1 Introduction

David Chambers¹, Lisa Simpson², Gila Neta¹

¹Division of Cancer Control and Population Sciences, National Cancer Institute, Rockville, MD, 20850, USA; ²AcademyHealth, Washington, DC, 20009, USA

At a time of significant upheaval in American health policy, maintaining a focus on a “North Star” is critical. For implementation science, this star is the knowledge base on how to optimally disseminate evidence related to health and health care, how to implement interventions to improve care within the many settings where people receive health care and make health-related decisions, and how to improve the health of the global population. To that end, the end of 2016 brought over 1100 engaged and activated “disciples of D & I” to Washington, DC for the 9^th Annual Conference on the Science of Dissemination and Implementation in Health. Once again, the accompanying abstracts in this issue demonstrate the breadth, depth and vigor of this continually expanding and evolving subset of health research. During three dynamic plenaries with rows and rows of filled seats and packed concurrent sessions presenters and attendees shared findings, raised methodologic and other challenges, and discussed future priorities, trends, and next steps for this community of research.

For the third year in a row, we were buoyed by a strong partnership, co-led by AcademyHealth and the National Institutes of Health (NIH), with co-sponsorship from others committed to implementation science: the Agency for Healthcare Research and Quality (AHRQ), the Patient Centered Outcomes Research Institute (PCORI), the Robert Wood Johnson Foundation (RWJF), and the US Department of Veterans Affairs (VA). The multidisciplinary program planning committee informed the development of the key themes for the conference, identified the plenary sessions topics and speakers, established track leads to manage the review process for concurrent panels, papers, and posters, and convened a scientific advisory panel to advise on the overall conference, thus ensuring a robust, inclusive, and rigorous process.

Together, the opening keynote address and the three plenary panel sessions set a tone of innovation and dialogue, raised critical issues, surfaced different perspectives, and ensured that follow on lunchtime and hallway discussions delved deeper into thorny challenges facing the field. Roy Rosin, Chief Innovation Officer for the University of Pennsylvania’s Perelman School of Medicine, introduced the audience to a range of methods for rapid testing, innovation in healthcare delivery, and lessons learned from other industries to maximize potential of new practices to be scaled-up. Each of the three plenary panels presented a general discussion on a high priority challenge for dissemination and implementation (D & I) research. A panel on the balance between intervention and implementation fidelity and local adaptation touched on the very real dynamic that is playing out in communities across this country as policy and payment changes are driving providers and others to seek new ways to solve the challenges in their particular contexts. A panel on the longer-term decisions around sustainment or de-implementation of interventions could not be more timely given the “improvement fatigue” of some systems and providers and the very real limits on providers’ time and focus. Too often, the imperative is to “do more”; much more attention needs to be about stopping what is not working, particularly in light of estimates that 30 percent of care provided is either unnecessary, of low value or wasteful (Institute of Medicine, 2013). The third plenary panel brought different perspectives on the enduring and evolving challenges in the dissemination of evidence and evidence-based practices as well as the opportunities emerging from innovations in the digital health sector. The plenary sessions were complemented by facilitated lunchtime discussions on these topics, as well as additional research priorities, which enabled more in-depth discussions, additional question and answer time, and brainstorming of future directions. Synopses of the lunchtime discussions are included in this supplement.

The concurrent sessions were once again organized by tracks. Last year’s tracks—Behavioral Health, Big Data and Technology for Dissemination and Implementation Research, Clinical Care Settings, Global Dissemination and Implementation, Promoting Health Equity and Eliminating Disparities, Health Policy Dissemination and Implementation, Prevention and Public Health, and Models, Measures and Methods—were maintained, and a new track on Precision Medicine was added, built upon the significant interest that emerged from last year’s plenary and subsequent discussions at NIH, National Academy of Medicine, and beyond. The tracks again enabled conference participants to follow a consistent theme across the multiple sessions of the conference and to better group thematically the individual papers and posters submitted by the conference participants. This supplement also is organized by these track themes.

The call for abstracts, including individual paper presentations, individual posters and panel presentations, resulted in 601 submissions, spread across the nine thematic tracks. Over one hundred reviewers from multiple disciplines, sectors, settings and career stage devoted their time to ensuring a comprehensive and expert review, and reviews were conducted within each track and coordinated by the track leads. For the final program, 19 oral abstract sessions, 9 panels, and 334 posters were presented over the two-day meeting, in addition to a “poster slam”. Slides for the oral presentations and panels (with the agreement of the authors) were posted on the conference website (https://academyhealth.confex.com/academyhealth/2016di/meetingapp.cgi/Home/0) and all abstracts were included on the conference webapp (https://academyhealth.confex.com/academyhealth/2016di/meetingapp.cgi). New this year was a presentation format that combined the efficiency of posters with the benefits of oral presentation—the poster slam. Eighteen of the top scoring posters were selected for this format. This new format enabled the lead authors of the top scoring posters to present their key findings in three minutes each, forcing clarity of communication on the vital message of their research. This skill of the “elevator” pitch is often not well developed amongst researchers.

This supplement has compiled the abstracts for presented papers, panel sessions, and lunchtime discussions from the 9th Annual Meeting on the Science of Dissemination and Implementation in Health: Mapping the Complexity and Dynamism of the Field. We are pleased to have the combined proceedings from the conference together in one volume once again, and look forward to the 10th Annual meeting, scheduled for December 4-5, 2017 in Arlington, VA. We look forward to next year’s conference which will also be the 10th anniversary of this convening, marking just how much implementation science has grown over the last decade.

D1 Balancing adaptation and fidelity: Exploring the continuum

Ulrica von Thiele Schwarz¹, Antoinette Percy-Laurry², Gregory A. Aarons³

¹Department of Learning, Informatics, Management, and Ethics, Karolinska Institutet, Stockholm, 17177, Sweden; ²Division of Cancer Control and Population Sciences, National Cancer Institute, Rockville, MD, 20850, USA; ³Department of Psychiatry, University of California San Diego, San Diego, CA, 92093, USA

Efforts to implement evidence-based interventions (EBIs) must address the challenges of remaining faithful to the original design of an intervention (i.e., fidelity) while considering whether and how adapt to local populations, contexts, or delivery models. Similarly, there may also be a need for adaptation of implementation strategies and adaptation of systems and organizations in which EBIs are being implemented. It is increasingly recognized that adaptation processes go beyond simple either/or decisions in considering when, how, and why adaptations are necessary. This session provided an opportunity for participants to address the challenges of, and strategies for, studying and achieving adaptation without sacrificing meaningful fidelity. There were approximately 100 participants in this 90 minute session that included small group discussions and large group interactive discussions facilitated by the authors.

The key themes identified included the importance of determining how much adaptation indicates a significant departure from the EBI, types of EBI or contextual adaptations that might be needed for patients with comorbidities, how adaptations can be tailored within complex health systems, and optimal methods and measures for assessing fidelity and adaptation. Other central areas of focus were how to scale adaptation across health systems, identifying contextual influences on adaptation, and the tension in moving from adaptation to innovation while avoiding drift that may compromise EBI or implementation strategy effectiveness.

Recommendations included examining the degree to which adaptation is occurring, discriminating between intentional vs. unintentional adaptation, using metrics to determine when drift occurs, and having a plan to facilitate and monitor appropriate adaptation and drift.

A call was made for researchers to put more emphasis on identifying core EBI and implementation strategy components, and it was highlighted that in order to manage fidelity and adaptations, both the EBI and the clinical environment needs to be understood. This calls for collaborative approaches between researchers and practitioners. Also, there is a need to consider the outer system context and inner organizational context factors that can constrain or allow adaptation, as well as prioritizing scaling-up and spreading EBIs that are robust enough to survive some adaptation. Another recommendation was to use technology to assess fidelity in real-time and to be sure that fidelity assessment and source (e.g., clinician report, patient report, observation, etc.) is reliable and valid. Lastly, the importance of how adaptations or reinvention impact quality of care and outcomes was voiced. In conclusion, the discussion highlighted that often, the question is not a matter of either fidelity or adaptation, but rather fidelity and adaptation.

D2 Novel directions in dissemination research

Gila Neta¹, Ross Brownson²

¹Division of Cancer Control and Population Sciences, National Cancer Institute, Rockville, MD, 20850, USA; ²George Warren Brown School of Social Work, Washington University in St. Louis, St. Louis, MO, 63130, USA

Within funded health-related research, far greater emphasis seems to be placed on implementation of evidence-based practices than on effective dissemination processes to get evidence and evidence-based interventions spread to target audiences. With great advances in communications technology and new sources of data, this lunch discussion forum brought together over 150 conference attendees to discuss a broad range of challenges, share experiences, and explore future directions for dissemination research.

Participants began the discussion by distinguishing features of dissemination research as opposed to implementation research (i.e., directive spread, less active, communicating vs. adopting, not setting-specific) and noting that dissemination is not spontaneous and not effective when passive. Aspects of successful dissemination were highlighted, including stakeholder engagement, team science, community based participatory research, and coalition building. Participants also explored what makes research innovative, reflecting on the keynote by Roy Rosin and highlighting the value of rethinking the problem. Other aspects of innovation included: solving a problem that has not been solved before, addressing a problem in a new context, applying technology in new ways, exploring the significance of the changing environment, engaging a new sector or partners in addressing a problem, applying a theory from an outside sector, transferring technology in new ways, and rethinking strategies needed for scale up. Strategies to identify novel topics were explored, including identifying evidence gaps, using knowledge brokers, engaging partnerships, spending time with practitioners, and concept mapping.

Examples of innovative and scalable research opportunities were discussed, including in primary prevention and the Affordable Care Act, the integration of public health, literacy, tracking behavior in real time, social determinants of health, and value-based care and how to incorporate patient perspectives. Participants also emphasized research challenges, including in finding funding for cross cutting health issues, review processes, and data credibility particularly in the eyes of providers. And future directions were highlighted, including in policy dissemination, examination of ways in which evidence is moving, longitudinal evaluation of the impact of dissemination research, and understanding mechanism through which partnerships work and how best to measure that.

The discussion concluded with brief presentations about resources in dissemination research, including from Maureen Dobbins of the National Collaborative Centre for Methods and Tools who highlighted their capacity building activities, support for change initiatives, and mechanisms to disseminate reviews. A brief overview of resources at the National Cancer Institute in dissemination research was also highlighted and can be found at https://cancercontrol.cancer.gov/is/.

D3 Planning for the long-term: considering sustainment

Amanda Vogel¹, Shannon Wiltsey Stirman²

¹Center for Global Health, National Cancer Institute, Rockville, MD, 20850, USA; ²Dissemination and Training Division, VA National Center for PTSD, Palo Alto, CA, 94305, USA

The last ten years have shown increasing interest in sustainment of interventions in population health and healthcare. In this lunchtime discussion forum, participants from academia, community-based organizations, and research corporations discussed challenges, strategies, and goals in sustainment research.

A key theme of the conversation was challenges and strategies around funding for sustainment research. Participants discussed the limitations of the five-year research grant to cover both implementation and sustainment research. It was suggested that this was enough time to include both types of research only if an intervention could be implemented in full in year 1 (e.g., a minor change in EMRs). The group discussed the related challenge of being unsure at what point to shift from implementation research to sustainment research. Participants shared related strategies such as refining and implementing interventions in early waves of funding, and studying adaptation and/or sustainment in later waves of funding. A number of persons discussed the opportunities created by observational approaches to sustainment research. The importance of theory-driven research was also noted.

Another key theme of the conversation was the importance of adaptation for effective dissemination and sustainment. The group discussed a range of challenges related to effectively adapting interventions to suit a wide range of new contexts, including: balancing fidelity and adaptation in diverse contexts, adapting complex multi-part interventions to limited resource settings, and successfully implementing and sustaining an intervention in the absence of its original champions. The group highlighted related needs in D&I science, including: a need for intervention designers to identify and specify the core of the intervention versus the adaptable periphery; a need to document how peripheral aspects are adapted as context for reporting results; a need to capture multilevel influences on sustainability; the value of implementation tools and playbooks to support effective adaptation without reinventing the wheel; and the use of SMART designs, A/B designs, and observational approaches to compare the effectiveness of various adaptations in different populations/patient groups/contexts. The conversation also highlighted the need to understand how to routinize new interventions so they become part of “the way things are done” (but not to the extent that the interventions become entrenched and difficult to replace if more effective practices are identified), with an emphasis on training individuals in multiple implementing and leadership roles to facilitate routinization.

Overall, the group was highly engaged and committed to this area of D&I science. They framed sustainment research as an essential component of D&I science, and necessary to generate the data on long-term costs, population health and patient outcomes that are essential to public health and healthcare decision making. They believed these factors, along with interest in ROI and efficiency, were powerful reasons for funding agencies to invest in sustainment research.

D4 Addressing education and training needs in implementation science

Kenneth Sherr¹, Rachel Sturke²

¹Department of Global Health, University of Washington, Seattle, WA, 98105, USA; ²Center for Global Health Studies, Fogarty International Center, National Institutes of Health, Bethesda, MD, 20912, USA

Over the past decade, there has been remarkable increased interest in the field of implementation science, with commensurate increases in demand for training. A number of programs and resources have been created to address this need. In this discussion forum, participants representing researchers and practitioners from academia, industry, community-based organizations, and research funders shared and discussed existing training and education opportunities in implementation science. Participants discussed barriers and facilitators to establishing and accessing training programs, and priorities for training in implementation science.

Multiple themes emerged through the discussion. Participants described an array of existing training opportunities in implementation science globally and domestically, ranging from short courses to long-term degree programs, and targeting professionals in practice and/or researchers. Despite increases in training opportunities, demand continues to outpace supply, especially for low- and middle-income country researchers. Participants identified a number of constraints in implementation science training. The field lacks a standardized taxonomy for implementation science, including its definition and methodologic components. Furthermore, training competencies are not well formed, nor differentiated for multiple end users (e.g. practitioners vs. researchers, domestic vs. global implementation scientists, undergraduate vs. graduate/doctoral trainees), which impedes assessment of training programs success. Participants noted that there is a lack of critical mass of expertise in implementation science, including insufficient mentors and support for mentors, which combined with limited funding opportunities to support training, constrains efforts to build a broad base for the field. Finally, participants noted the need to educate the value of implementation science among practitioner organizations, grant reviewers and journal reviewers.

Participants identified priorities to address implementation science training gaps. First, a crosswalk of existing training programs to catalog competencies for different end users, metrics for these competencies, and core methodologies and tools covered in these training programs could improve consensus and identify specific gaps to inform future training development. Efforts are needed to engage practitioner organizations to enhance training programs to meet their priorities, and generate support for training in implementation science; a clear understanding of return on investment for implementation science, perhaps illustrated through a set of case studies would support these efforts. Further engagement with disciplines outside of the health sciences would support infusing implementation science into other training programs. Finally, participants noted that diverse funding for implementation science training is needed to match the needs of different end users, including funding to support dedicated mentorship to build a critical mass in implementation science expertise.

D5 De-implementation

Wynne E. Norton¹, Allyson Varley², David Chambers¹

¹Division of Cancer Control and Population Sciences, National Cancer Institute, Rockville, MD, 20850, USA; ²School of Public Health, University of Alabama at Birmingham, Birmingham, AL, 35243, USA

De-implementation—reducing or stopping the use of ineffective, harmful, low-value, and/or unproven interventions, practices, and programs—is an important but understudied area of research. To advance the field, we need to develop and test frameworks, methods, measures, outcomes and strategies that address issues specific to de-implementation issues. The purpose of the forum was to discuss these issues, raise awareness of and interest in the need for advancing the science, and identify next steps for moving in such a direction.

During the forum, researchers, funders, and practitioners with various health care and public health perspectives identified key questions and issues in de-implementation, including: terminology, definitions, multi-level factors, frameworks, challenges in conducting research on de-implementation, and priorities for advancing the field. Common questions focused on how to incentivize de-implementation among multi-level stakeholders, including patients, providers, care teams, organizations, systems, and policies. Other challenges included mapping terms and processes, identifying and testing appropriate frameworks, and utilizing diverse methodologies (e.g., direct observation) to understand de-implementation processes, and testing financial and other de-implementation strategies. Participants discussed ways to capitalize on ongoing initiatives to reduce overuse or inappropriate care (e.g., Choosing Wisely; http://www.choosingwisely.org/) to study de-implementation. Participants also noted commonly-used terms broadly used to describe the field of de-implementation (e.g., reassessment, disinvestment, de-adoption, and decreased use), and the need for standardized definitions, operationalization, and validated measures. Finally, processes involved in de-implementation were perceived as similar to but distinct from those involved in implementation, warranting novel conceptualizations, outcomes, research designs, and practical considerations.

Consistent with the need to advance this emerging discipline, recommendations from attendees to advance the field were numerous and multi-pronged. To begin, participants suggested organizing stakeholder meetings with relevant health researchers, practitioners, funders, payers, and policymakers to discuss definitions, conceptualizations, and prioritize next steps. Participants also encouraged efforts to raise awareness of the field, including publishing data-driven and perspective pieces on terminology, standards for warranting de-implementation, outcomes, and incentives for de-implementation. Finally, participants suggested ways in which the field could use natural experiments to study policy changes aimed at reducing overuse as well as collecting data on de-implementation from ongoing implementation studies.

D6 Designing for dissemination and implementation

Cynthia Vinson¹, Lisa Klesges^1,2, Suzanne Heurtin-Roberts¹

¹Division of Cancer Control and Population Sciences, National Cancer Institute, Rockville, MD, 20850, USA; ²School of Public Health, University of Memphis, Memphis, TN, 38152, USA

Behavioral researchers have evidence of effective health interventions that work, but this evidence is not widely used in practice. Programs to disseminate effective interventions exist (Small Business Innovative Research Program (SBIR), NSF I-Corps program, etc.) to help move biomedical research into practice, but there is limited assistance for behavioral researchers to move their interventions into practice. This lunch discussion forum provided participants the opportunity to explore how the field can better support efforts to design interventions with dissemination and implementation in mind.

The forum began with brief statements by NCI leaders regarding the concept of designing for implementation. Several considerations in designing for implementation were discussed including giving serious attention to ease of future implementation in practice, considering the range of contexts in which an intervention is likely to be implemented, as does implementation feasibility. Researchers, practitioners and funders engaged in discussions that raised a number of issues and lessons learned from their own experience.

Concern about minimizing the intrusiveness and influence on implementation sites was raised, since the trial of an implementation is supposed to occur in environments as they exist. Significant modification of an implementation site by research activities can diminish the validity and utility of results. It can obscure findings regarding the resources generally needed and available to implement an intervention.

Hybrid designs were discussed as a means to allow the development of an intervention in the same study that implementation research questions were asked. Further discussion focused on which designs might be appropriate at which stage of intervention development.

The importance of researcher, practitioner, and other stakeholder collaboration was emphasized. The utility of a participatory approach to the design of an implementable intervention was noted.

Career issues were discussed, with special concern for early researchers. Developing effective practices doesn’t necessarily build academic careers in the traditional model. Novel efficacious interventions are rewarded in publications and further research funding which are necessary for success in academic careers. The need for publishable intervention outcomes as well as successful implementation outcomes and strategies are challenging for those seeking tenure and career advancement.

The discussion was an interactive and supportive one. As participants asked questions and raised issues, other participants engaged to respond to questions with accountings of previous experiences and lessons learned.

D7 Learning from improvement

M. Rashad Massoud, Leighann Kimble

USAID Applying Science to Strengthen and Improve Systems (ASSIST) Project, Bethesda, Maryland 20814, USA

The purpose for the lunch discussion on “Learning from Improvement” was to report out on and discuss insights from the Salzburg, Global Seminar Session 565, “Better Health Care: How do we learn about improvement?”, which was convened from July 10th – 15th, 2016. During the lunch discussion a panel of participants from the seminar shared insights from the session as well as next steps.

The session is designed to discuss the need for research methods, such as mixed methods in order to address the complex issues posed by health care delivery. Understanding that the contextual nature of health care delivery in the real-world setting does not easily allow for replicability in research, the Salzburg Global Seminar discussed issues such as:

• Can we attribute the improvements we are measuring to the changes we are testing and implementing?

• How do we know that no other factors are influencing the results?

• If other factors are also affecting the results, how do we know what part is attributable to the changes we are making?

• Why did the changes which yielded improvements work, and how?

• How can we incorporate the effects of local context into improvements?

• How should we design improvement efforts to answer different learning objectives?

• How do we optimize data collection that simultaneously serves to drive quality improvement, inform evaluation efforts, and fulfil performance reporting requirements?

As a result of deliberating on these questions during the seminar, a framework was created to tackle these issues and encourage a “marriage” between implementation and research and evaluation. The framework describes that all improvement and evaluation must start with an aim specific detailing what is to be done and for whom. The aim is interwoven with a “theory of change”. The theory of change itself is closely related to “what” is to be done and “how” it is to be done in the improvement. Each of these factors, “what”, “how”, and the “theory of change” are related to context. In consideration of all of these complex factors, evaluation design and methods must be selected and integrated. In this way, evaluation design must occur alongside improvement, not after an improvement initiative is conducted. Such a change will allow for both better improvement design to allow for better learning and better evaluation design to inform implementation.

After insights from the seminar were shared regarding these issues, interaction, questions, and comments from participants was encouraged through group discussion.

Behavioral Health

S1 Large-scale implementation of collaborative care management for depression and diabetes and/or cardiovascular disease

Arne Beck¹, Claire Neely², Jennifer Boggs¹

¹ Institute for Health Research, Kaiser Permanente, Denver, CO, 80231, USA; ²ICSI, Institute for Clinical Systems Improvement, Bloomington, MN, 55425, USA

Correspondence: Claire Neely

Background

The effectiveness of collaborative care management for patients with depression and comorbid chronic medical conditions is influenced by numerous implementation factors. The objective of this study was to describe the variation in implementation across medical groups of the Care of Mental, Physical, and Substance-use Syndromes (COMPASS) initiative, a large-scale implementation of an evidence-based collaborative care management model to improve health outcomes in patients with depression and diabetes and/or cardiovascular disease.

Methods

Ten health care systems participated in the COMPASS initiative, representing 18 medical groups with over 172 clinics across eight states, and 3,854 patients. Eligible patients had active depression (PHQ9 > =10) and one of the following poorly controlled medical conditions: diabetes mellitus with a HgbA1c > 8% or systolic blood pressure (SBP) > 145; and/or cardiovascular disease with (SBP > 145 [SBP > 165 for patients over 65]). COMPASS entailed 1) a clearly defined care management process, a care team that included a consulting physician and psychiatrist, and weekly systematic case reviews focusing on treat-to-target guidelines; 2) A registry to track patients for follow-up contacts; and 3) monitoring of hospital and emergency department utilization. Descriptive data on COMPASS implementation were obtained from annual site visit reports and supplemental site surveys. Site visit reports were analyzed with Atlas.ti software to identify emergent themes regarding implementation. The Consolidated Framework for Implementation Research (CFIR) was used to generalize themes to broader implementation constructs.

Findings

Nine specific implementation themes were identified that related to six main CFIR constructs: 1) Length of patient enrollment in COMPASS and 2) registry use (CFIR intervention characteristics); 3) patients’ social needs and 4) challenges to health systems’ organizational environments (CFIR outer setting), 5) primary care physician engagement and 6) experiences with care coordination (CFIR inner setting); 7) care manager characteristics (CFIR characteristics of individuals); and 8) COMPASS care team dynamics and 9) quality improvement and outcomes monitoring reports (CFIR process).

Implications for D&I Research

Substantial variation across COMPASS medical groups was observed in the emergent implementation themes. Understanding such variation may provide important data with which to increase successful large-scale dissemination of similar models.

Primary Funding Source

Centers for Medicare and Medicaid Services - The project described was supported by Grant Number 1C1CMS331048-01-00 from the Department of Health and Human Services, Centers for Medicare & Medicaid Services

S2 Cost analysis of the collaborative care model for behavioral health in an urban, African-American, Medicare population

Carmel Nichols, Wen Wan, Erin Staab, Neda Laiteerapong

Medicine, University of Chicago, Chicago, IL, 60637, USA

Correspondence: Carmel Nichols

Background

The Collaborative Care Model (CCM) for behavioral health has been studied in more than 70 randomized controlled trials and has been shown to be more cost-effective than usual care in the primary care setting. However, it has not been evaluated in an urban, predominately African-American, Medicare population. We evaluated the change in inpatient and outpatient provider costs before and after implementation of the CCM in this population.

Methods

The Medicare Advantage Program (MAP) CCM program for Behavioral Health was implemented in January 2016 at the primary care clinic of an urban academic medical center. The MAP CCM program involved routine and targeted mental health screening, outreach, and comprehensive care management. We performed a retrospective analysis of the total costs of patients who were continually enrolled in Medicare Advantage from July 2015 to June 2016. Changes in total cost before and after the start date of the CCM were compared using a Wilcoxon signed rank test, as well as changes in emergency room, inpatient and outpatient costs.

Findings

Of the 762 patients in MAP, 460 patients were continuously enrolled during the study period. Median (range) age was 69 years (29-95), 67% were female and over 85% were African-American. There was a significant decrease in median health care costs after (vs. before) the CCM implementation (difference = -$1,899, interquartile range (IQR) (-11,921, 5193), p = 0.005), mostly due to a decrease in outpatient (difference = -$946, IQR (-8283, 4501), p = 0.01), and emergency room (p = 0.002) costs. There was also a decrease, although not significant, in inpatient costs (p = 0.26).

Implications for D&I Research

These results provide evidence of potential cost-saving attributable to the Collaborative Care Model for Behavioral Health in an urban, predominately African-American, Medicare population. Future studies will include a more detailed evaluation of the CCM program in order to evaluate the cost-effectiveness of the program in this population, both for patients with depression and for those with depression and diabetes.

S3 Barriers and facilitators to implementing the New York state collaborative care initiative for depression in academic primary care settings: using a theoretical framework to inform policy

Nathalie Moise¹, Ravi Shah², Susan Essock², Margaret Handley³, Amy Jones⁴, Jay Carruthers⁴, Karina Davidson¹, Lauren Peccoralo⁵, Lloyd Sederer⁴

¹Medicine, Columbia University Medical Center, New York, NY, 10032, USA; ²Psychiatry, Columbia University, New York, NY, 10032, USA; ³General Internal Medicine, Epidemiology and Biostatistics, University of California San Francisco, San Francisco, CA, 94110, USA; ⁴New York State, Office of Mental Health, New York, NY, 10001, USA; ⁵Medicine, Icahn School of Medicine at Mount Sinai, New York, NY, 10029, USA

Correspondence: Nathalie Moise

Background

In 2012, the New York State (NYS) Office of Mental Health, in partnership with the NYS Department of Health, implemented Collaborative Care (CC) for depression in 19 academic medical centers (32 clinics serving 1 million patients). We aimed to assess barriers and facilitators to its implementation and to use a theoretical framework to propose corresponding policy and behavioral change interventions for future successful spread of CC.

Methods

Of 19 centers, 17 agreed to be interviewed. We completed 30 semi-structured interviews (6 psychiatrists, 8 clinic administrators, 8 primary care physicians, and 8 care managers) representing 8 healthcare systems before reaching saturation. We conducted a hybrid process of both inductive and deductive thematic analysis (NVIVO 11.1), before using the Capability, Opportunity and Motivation-Behavior (COM-B) model to characterize the themes and to propose related policy recommendations that could address identified barriers and facilitate site-level behavioral change.

Findings

Clinics ranged in size from 6,000 to 70,000 patients (5% to 37% depression screen positive rates) with 0.5 to 6.0 care manager full time equivalents. Themes related to major implementation barriers concerned personnel-resources (e.g., competing care manager demands, inadequate staff) (77% of respondents), patient engagement (e.g., no shows/non-adherence) (63%), team engagement (e.g., physicians/residents) (50%), and external factors (e.g., competing state/national initiatives, psychosocial resources) (40%). Major facilitators involved patient engagement (e.g, warm handoffs, personalization) (83%), team engagement (e.g., culture change, training, accountability) (80%), and personnel/resources (e.g., hiring paraprofessionals for registry/billing) (73%). Themes were predominantly classified under opportunity and motivation (COM-B). Corresponding policy categories for future opportunities included optimizing billing infrastructure and instituting value-based purchasing (fiscal supports, legislation), incorporating paraprofessionals and information technology platforms into the CC framework (environmental/social planning, service provision), and providing educational and decision support tools to improve staff, physician and patient engagement (marketing/communication, guidelines).

Implications for D&I Research

This is one of the first studies to identify barriers and facilitators for implementing a large, state-wide initiative for depression CC in academic centers and to use a theoretical framework to provide concrete suggestions for its future dissemination and successful implementation. Our study suggests that engagement strategies with optimization of registry and billing infrastructures are key to success.

Primary Funding Source

National Institutes of Health - NHLBI 3 R01 HL114924-03S1

S4 Improving transitions from detox to continuing care: using a care transition implementation system

Todd Molfenter

Center for Health Enhancement System Studies, University of Wisconsin, Madison, WI, 53706, USA

Background

With successful transitions from detoxification to continuing care, patients are more likely to remain drug-free or sober, making this is a key transition in the addiction treatment continuum. Yet, just 11-27% of individuals receiving emergency detoxification services for alcohol or drug abuse receive additional treatment or continuing care. This is occurring at a time when drug overdose deaths, primarily due to opioids, are the leading cause of accidental death in the United States.

Methods

The system to be tested to improve transitions from detoxification to continuing care integrated three evidence-based implementation science approaches: a) a practice bundle of replicable evidence-based practices found to increase detoxification to continuing care transition rates, b) a checklist to review use of practice bundle elements, and c) an established standardized organizational change model, called the NIATx model, to facilitate use of the practice bundle and checklist. The mixed methods analysis compared baseline to post-intervention detoxification to continuing care transition, the efficacy of different practice bundle combinations, and observed the use of the checklist.

Findings

The integrated implementation approach improved detoxification to continuing care transition rates from 20% (baseline average) to 43% (post-intervention). The bundle combination that demonstrated the highest transition rates was: a) transitioning patients from detoxification to continuing care in <3 days; b) conducting warm/orchestrated hand-offs (or transfers) between levels of care when possible; and c) collecting data in the detoxification unit to monitor continuing care performance. The process checklists were modified from being used per patient to being used to guide systems change.

Implications for D&I Research

The intervention uses an innovative approach based on practice bundles and process checklists, which have been successfully applied in acute care hospital settings, with varied approaches to implementation. The system offers a standardized implementation approach and examines how using bundles and checklists is generalizable to a community-based behavioral health setting. Should the system prove to be generalizable and beneficial, it could provide a tool to address other long-standing public health challenges that are worsened by poor transitions between levels of care.

S5 A mixed-methods study of system-level sustainability of an evidence-based practice following 12 large-scale implementation initiatives

Ashley Scudder¹, Sarah Taber-Thomas^1,2, Kristen Schaffner¹, Amy Herschell^1,3

¹Department of Psychiatry, University of Pittsburgh, Pittsburgh, PA, 15213, USA; ²Psychology, University at Buffalo, Buffalo, NY, 14260, USA; ³Department of Psychology, West Virginia University, Morgantown, WV, 26506, USA

Correspondence: Amy Herschell

Background

As evidence-based practices (EBPs) become more widespread in community behavioral health systems, determining how to systematically sustain (facilitate and optimize) EBPs in complex, multi-level service systems has broad implications for public health. The current study sought to understand the process of sustaining a particular EBP, Parent-Child Interaction Therapy (PCIT), based on the knowledge and experiences of individuals working within existing service systems across states that have employed large-scale implementation initiatives.

Methods

This mixed methods study examined rates of sustainment following 12 large-scale initiatives to implement PCIT, factors impacting sustainability, and strategies used to enhance sustainment. Participants were recruited from large-scale PCIT training initiatives using a snow-ball sampling approach. A mixed-methods approach to data collection was utilized; qualitative and quantitative data were sequentially collected and analyzed, using each type of data to answer specific questions and to provide depth and breadth of understanding related to sustainability outcomes and processes.

Findings

Sustainment strategies fell into nine categories, including infrastructure, training, marketing, integration, and building partnerships. Strategies involving integration of PCIT into existing practices and quality monitoring predicted sustainment, while financing also emerged as a key factor. The relations among demographic variables, barriers, and strategies were examined, with regression results indicating two strategies (integration into existing practices and monitoring quality) and one barrier (lack of financial funding) as significant predictors of sustainment outcomes. In conjunction with the quantitative data, a grounded theory approach was used to qualitatively identify system-level factors, implementation and initiative factors, and intervention factors related to the facilitation and impediment of the sustainability of PCIT over time.

Implications for D&I Research

Authors will discuss implications for the system-level sustainability of EBPs, such as the dynamic interaction among challenges and specific strategies utilized, as well as the reported need of innovation in systems and practice.

Primary Funding Source

National Institutes of Health - This research was supported by funding from the National Institute of Mental Health (R01 MH095750)

S6 Implementing integrated primary care in late adopter sites: the impact of key events on repeated measures of uptake

Eva Woodward¹, Jeffery Pitcock², Mona Ritchie², JoAnn Kirchner^2,3

¹MIRECC, VA, North Little Rock, AR, 72114, USA; ²QUERI for Team-Based Behavioral Health, Central Arkansas Veterans Healthcare System, North Little Rock, AR, 72114, USA; ³Department of Psychiatry and Behavioral Sciences, University of Arkansas for Medical Sciences, Little Rock, AR, 72205, USA

Correspondence: Eva Woodward

Background

Integrating behavioral health care into primary care is a complex behavioral health innovation and often warrants facilitation—an umbrella implementation strategy—to increase its adoption, especially in late adopter sites. We conducted a secondary data analysis from a two-year hybrid Type II study using facilitation to promote implementation of integrated primary care across seven VA medical clinics. Our aim was to (a) identify patterns of adoption across clinics and (b) identify associations between key events that occurred during implementation and adoption.

Methods

For each clinic, we synthesized quantitative and qualitative data. First, we utilized a quantitative B-spline function to map out a pattern of continuous, linear combination of multiple repeated (monthly) measures of adoption—this accounted for correlations between time points. Second, we used qualitative data from program records, debriefing interviews with facilitators, and facilitator time logs to identify key events that occurred during implementation facilitation (e.g., staff turnover). Two coders categorized key events into domains of the theoretical framework that informed the study—Integrating Promoting Action on Research for Health Services (i-PARIHS). Then, we created a visual overlay of key events on to mapped patterns of adoption.

Findings

Adoption increased over time, although with great variability—it was more rapid for smaller clinics. Events that inhibited adoption over time included staff loss and hiring of staff and leadership not trained in the innovation. Events that promoted adoption over time included facilitator site visits, positive changes in leadership, and creating and reviewing implementation-planning checklists.

Implications for D&I Research

When using facilitation for complex innovations in healthcare, preliminary results suggest that site visits, continuous assessment of implementation-planning checklists, and staff loss or mismatch to the innovation are most impactful on adoption. Though it presents methodological challenges, an advantage of using repeated measures is that, rather than having only a snapshot of adoption at a few, discrete time points, we were able to assess variability in adoption by using a nearly continuous measure of repeated assessments over two years. One challenge is validity of associations between adoption and key events, due to the multifactorial nature or “shocks” of the data over time.

Primary Funding Source

Department of Veterans Affairs - Health Services Research and Development and QUERI grants

S7 Enhancing organizational capacity in implementation science: an evaluation of the practicing knowledge translation course

Julia E. Moore¹, Sobia Khan¹, Shusmita Rashid¹, Jamie Park¹, Melissa Courvoisier¹, Sharon Straus^1,2

¹ Knowledge Translation (KT) Program, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, ON, M5B 1 T8, Canada; ² Geriatric Medicine, University of Toronto, Toronto, ON, M5G 1 V7, Canada

Correspondence: Julia E. Moore

Background

Effective evidence implementation can help health care organizations optimize quality of care. A challenge is lack of organisational capacity in implementation. Building on a previous evaluation, we developed a 6-month course, Practicing Knowledge Translation (PKT), designed to teach best practices in high-quality implementation and equip participants with the core competences required to integrate evidence-informed practices. We conducted an outcome evaluation of the course.

Methods

The 15 PKT course participants included professionals working in healthcare, government, and non-government organizations that support healthcare delivery or funding. The primary outcome was an increase in knowledge and self-efficacy across six core competencies (developing an evidence-informed, theory-driven program (ETP); implementation; evaluation; sustainability, spread, scale up; contextual assessment; stakeholder engagement) measured through surveys (likert 1-7) completed before (time 1), immediately after (time 2), and 7 months after (time 3) the course. Secondary outcomes included use of implementation (likert 1-5) and satisfaction with the course based on semi-structured interviews. Descriptive statistics and repeated-measures ANOVA and Bonferroni post-hoc models were used for quantitative data analysis to evaluate changes in knowledge and self-efficacy in SPSS v.20. Data from qualitative interviews will be analyzed in NVivo.

Findings

Survey response rate was 60-80% across all time points (Time 1, n = 12; Time 2, n = 12; Time 3, n = 9). Participants’ knowledge and self-efficacy increased in four and five of the six core competencies, respectively: developing an ETP (p = .06; p = .01), implementation (p = .002; p = .001), evaluation (p = .003; p = .03), sustainability/scale/spread (p = .002; p = .02), contextual assessment (p = .012; p = .002), and stakeholder engagement (p = .06; p = .13). There was no change in current implementation practice after the course was delivered (p = .12). Qualitative data are being analyzed and will be available for presentation.

Implications for D&I Research

Quantitative findings indicate that the PKT course increased participant knowledge and self-efficacy across 5 of 6 core competencies. There is a need to evaluate the impact of implementation training on participants’ knowledge of and self-efficacy in using implementation science in their projects and organizations. Findings from this evaluation can be used to inform future training initiatives.

S8 Implementation barriers and facilitators of treatments for criminogenic thinking in the Veterans Health Administration

Daniel Blonigen^1,2, Allison Rodriguez³, Luisa Manfredi¹, Andrea Nevedal^1,4, Joel Rosenthal⁵, David Smelson^6,7, Christine Timko¹

¹HSR&D Center for Innovation to Implementation, Department of Veterans Affairs Palo Alto Health Care System, Menlo Park, CA, 94025, USA; ²Clinical Psychology PhD Program, Palo Alto University, Palo Alto, CA, 94304, USA; ³VA Palo Alto Health Care System, National Center for PTSD, Menlo Park, CA, 94025, USA; ⁴Health Services Research and Development, Veterans Health Administration, Menlo Park, CA, 94025, USA; ⁵Office of Homelessness/Veterans Justice Programs, Department of Veterans Affairs, Menlo Park, CA, 94025, USA; ⁶Center for Healthcare Organization and Implementation Research (CHOIR), VA Health Services Research & Development, Bedford/Boston, MA, 01730, USA; ⁷Department of Psychiatry, University of Massachusetts Medical School, Worcester, MA, 01655, USA

Correspondence: Daniel Blonigen

Background

With the rise of specialty courts, behavioral health services are increasingly called upon to treat criminal offenders in order to reduce offenders’ risk for recidivism. Criminogenic thinking (antisocial cognitions/attitudes) is the risk factor with the strongest association with criminal recidivism. Cognitive-behavioral treatments for criminogenic thinking–e.g., Moral Reconation Therapy (MRT); Thinking 4 a Change (T4C)–were developed within correctional settings and are regarded as best practices for reducing recidivism. However, knowledge of their implementation potential within behavioral health services is unknown. To address this gap, we identified barriers to implementation of cognitive-behavioral treatments for criminogenic thinking in the Veterans Health Administration (VHA), and facilitators that could serve as solutions to these barriers.

Methods

Specialists from the VHA’s Veterans Justice Programs–a nationwide outreach and linkage service for justice-involved veterans–were recruited to participate in a semi-structured phone interview to describe their practices regarding treatment of criminal recidivism (n = 63). Participants who had been trained in MRT or T4C (n = 22) were asked questions regarding the implementation potential of these interventions in VHA. Thematic coding and pile-sorting techniques were used to identify barrier and facilitator themes at the patient-, program-, and system levels.

Findings

The time-intensiveness of these treatments emerged as a barrier to implementation. Potential solutions identified were patient incentives for treatment engagement, streamlining the curriculum, and implementing the treatments within long-term/residential programs. At the program level, providers’ stigma/bias towards patients with antisocial tendencies was seen as a barrier to implementation, as were time/resource constraints on providers. To address the latter, use of peer providers to deliver the treatments and partnerships between justice programs and behavioral health services were suggested. At the system level, lack of recognition of criminogenic treatments as evidence based, and uncertainty of sustained funds to support ongoing costs of these treatments emerged as implementation barriers. To address the latter, a “train-the-trainers” model was suggested.

Implications for D&I Research

Our findings serve as a guide for implementation of criminogenic treatments for providers/policymakers in VHA and other large healthcare systems that are increasingly called upon to provide care to justice-involved adults–a large and highly vulnerable subgroup of the behavioral health population.

Primary Funding Source

Department of Veterans Affairs - VA Health Services Research & Development (RRP 12-507)

S9 Therapist and leader attitudes towards evidence-based practices: influence of practice, staff role, and perceptions of organizational climate

Nicole Stadnick¹, Jennifer Regan², Miya Barnett³, Anna Lau⁴, Lauren Brookman-Frazee¹

¹Psychiatry, University of California, San Diego, La Jolla, CA, 92093, USA; ²Clinical Training and Evidence-Based Practice, Hathaway-Sycamores Child and Family Services, Pasadena, CA, 91105, USA; ³Counseling, Clinical, & School Psychology, University of California, Santa Barbara, Santa Barbara, CA, 93106-9490, USA; ⁴Psychology, University of California, Los Angeles, Los Angeles, CA, 90095, USA

Correspondence: Nicole Stadnick

Background

Organizational factors, such as implementation climate and workforce burnout, play an influential role on attitudes towards implementation of evidence-based practices (EBPs) in mental health systems. Both line staff and administrators are integral to EBP implementation efforts, and it is important to understand whether attitudes toward specific EBPs vary between agency leaders and therapists. Further, it is plausible that distinctive organizational factors differentially impact leader versus therapist attitudes toward EBPs.

Methods

This study examined leader and therapist attitudes towards EBPs within the context of a system-level reform mandating the implementation of multiple EBPs through reimbursement policies in children’s mental health services in Los Angeles County. Study objectives were to: 1) determine whether leaders and therapists differed in their attitudes toward six EBPs, and 2) identify organizational factors associated with leaders’ and therapists’ practice-specific attitudes. Data were obtained from 181 leaders and 763 therapists who completed an online survey that assessed perceptions of workplace burnout and organizational functioning using subscales from the Organizational Climate Measure and Organizational Readiness for Change. Hierarchical linear models were specified with practice (L1) nested within leader/therapist (L2) nested within agency (L3) to predict practice-specific attitudes (e.g., ease of use, adaptability to fit to service setting).

Findings

There were significant cross-level interactions between practice and informant, indicating that leaders and therapists reported differing preferences for EBPs. Furthermore, distinctive organizational factors interacted with practice to predict leader and therapist attitudes. For example, leader perceptions of staff cohesion were associated with more positive appraisals for most practices. Therapist perceptions of burnout were associated with more negative evaluation of some practices but more positive evaluation of others.

Implications for D&I Research

Understanding the unique perspectives of different types of staff towards EBPs can inform EBP implementation efforts. Findings indicate that leaders and therapists differ in their preferences for practices and these differences may be shaped by their perceptions of organizational functioning. Results highlight the importance of considering organizational characteristics such as staff cohesion and burnout when selecting and implementing EBPs to promote uptake and sustainment in service systems.

Primary Funding Source

National Institutes of Health - Funding for this research project was supported by the following grants from NIMH: R01 MH100134 and R01 MH100134-S.

S10 The influence of leadership at executive and middle management levels on the implementation of evidence-based behavioral health care practices

Erick Guerrero¹, Karissa Fenwick², Yinfei Kong³, Gregory Aarons⁴

¹Communities, Organizations and Business Innovation, School of Social Work, and Marshall School of Business, University of Southern California, Los Angeles, CA, 90015, USA; ²School of Social Work, University of Southern California, Los Angeles, CA, 90015, USA; ³Mihaylo College of Business and Economics, California State University, Fullerton, CA, 90089, USA; ⁴Psychiatry, UC San Diego, La Jolla, CA, 92083-0812, USA

Correspondence: Erick Guerrero

Background

Leadership plays a critical role in the implementation of evidence-based practices (EBPs) in behavioral health settings. While most research focuses on a single leader or a broad leadership style (e.g. transformational approach), this study examined the impact of both broad and focused leadership and multiple leadership levels on implementation of EBPs. Guided by social exchange theory, we proposed that mid-level, implementation-focused leadership would enhance how executive level transformational leadership affects 1) staff attitudes toward EBPs, and 2) implementation of contingency management treatment (CMT) and medication-assisted treatment (MAT).

Methods

We collected multilevel data in 2013 from 427 employees embedded in 112 behavioral health treatment programs in the largest addiction health services system in the United States. We measured transformational leadership using the Survey of Transformational Leadership, implementation-focused leadership using the Implementation Leadership Scale, and attitudes towards EBPs using the Evidence-Based Practice Attitudes Scale. We assessed implementation of CMT and MAT using a 5-point Likert Scale indicating how often the EBP is used in the program. We tested a multilevel path analysis with bootstrap standard errors using Stata, 13.

Findings

Results showed a positive relationship between transformational executive leadership and mid-level implementation-focused leadership (standardized direct effect = .173, bootstrap p < .03). As expected, transformational leadership was indirectly associated with employee attitudes toward EBPs via mid-level implementation leadership (standardized indirect effect = .090, bootstrap p < .01). Also, transformational executive leadership was indirectly associated with delivery of CMT through mid-level implementation-focused leadership (standardized indirect effect = .006, bootstrap p < .09), though this relationship was partially significant. The relationships between leadership types and MAT were not statistically significant.

Implications for D&I Research

This study offers theoretical contributions to social exchange and leadership theories as applied to implementation of EBPs in behavioral health settings. Findings highlight the importance of testing leadership process using mediators in order to unpack how leadership may cascade to different levels. Results highlight the importance of training mid-level management on implementation leadership to enhance the uptake of CMT in addiction health services.

Primary Funding Source

National Institutes of Health - R33DA035634-03, PI: Erick Guerrero, R25 MH080916, PI: Enola Proctor.

S11 A multilevel case study of act implementation

Rebecca Lengnick-Hall, Karissa Fenwick, Benjamin Henwood

School of Social Work, University of Southern California, Los Angeles, CA, 90015, USA

Correspondence: Rebecca Lengnick-Hall

Background

Commonly used implementation frameworks—such as the Consolidated Framework for Implementation Research (CFIR) and the Exploration, Preparation, Implementation, and Sustainment (EPIS) framework—describe the multilevel nature of evidence-based practice (EBP) implementation. These frameworks highlight the ways in which inner and outer contextual features overlap, interact, and influence implementation success. This study uses a multilevel case study method to understand the implementation of Assertive Community Treatment (ACT) in Los Angeles County.

Methods

Focusing on fidelity as a key implementation outcome, we conducted semi-structured interviews to explore ACT implementation experiences at three levels: county, agency leadership, and frontline staff. The sample included county level experts as well as 35 providers and managers at three organizations currently implementing ACT in L.A. County. Interview questions assessed knowledge of the ACT model, history of ACT implementation, fidelity, and the use of ACT in everyday clinical practice. Experts and managers addressed outer contextual features such as legislative decisions, funding structures, and contracting arrangements. Frontline staff and managers addressed inner contextual features such as organizational characteristics, employee attitudes, and EBP fit.

Findings

Findings revealed important relationships and differences among the three levels of ACT implementation, in particular relating to definition of the ACT model and conceptualization of fidelity. At the provider level, frontline staff offered many examples of 'doing' ACT, yet reported little explicit knowledge of model components or fidelity measurement. This was consistent with county administrator interviews describing the California programs as 'ACT like' and designed to be flexible, but also contrasted with administrator and director statements about the importance of achieving fidelity to core components.

Implications for D&I Research

This study demonstrates how a multilevel case study approach can capture and describe the nuances and complexity of EBP implementation. First, this design enabled us to link specific outer contextual features to behaviors and perspectives among those who are using ACT on a day-to-day basis. Second, this design allowed us to explore and assess differences across the three levels of analysis. Findings offer concrete examples of the relationships described in implementation frameworks.

S12 Organizational factors differentiating VA PTSD outpatient teams with high and low delivery of evidence-based psychotherapy

Nina Sayer^1,2, Craig Rosen^3,4, Robert Orazem¹, Brandy Smith^3,5

¹Center for Chronic Disease Outcomes Research, Minneapolis VA Health Care System, Minneapolis, MN, 55417, USA; ²Medicine and Psychiatry, University of Minnesota, Minneapolis, MN, 55454, USA; ³VA Palo Alto Health Care System, National Center for PTSD Dissemination & Training Division, Menlo Park, CA, 94025, USA; ⁴Department of Psychiatry and Behavioral Sciences, Stanford University School of Medicine, Stanford, CA, 94304, USA; ⁵VA Palo Alto Health Care System, Center for Innovation to Implementation, Menlo Park, CA, 94403, USA

Correspondence: Nina Sayer - Medicine and Psychiatry, University of Minnesota, Minneapolis, MN, 55454, USA

Background

As part of a broad effort to implement evidence-based care, the Veterans Health Administration (VHA) has been working over the past decade to expand use of Cognitive Processing Therapy (CPT) and Prolonged Exposure (PE) for treatment on posttraumatic stress disorder (PTSD). VHA's primary strategy has been national training initiatives efforts that target individual clinicians. Less attention has been given to team-level processes, and identifying local organizational factors that promote higher levels of use of CPT and PE.

Methods

This mixed methods study used automated processing of chart notes to select PTSD clinics with high and low use of CPT and PE. Rapid Assessment Process methodology was then used to compare teams along dimensions known to affect implementation and sustainability of a practice improvement program. The primary method of data collection was semi-structured interview with 100 staff from nine VA medical centers which had PTSD teams that differed in reach of CPT and PE to therapy patients with PTSD. Qualitative analysis involved constant comparison of high, medium and lower reach teams in terms of the identified themes.

Findings

High reach teams embraced a mission to deliver evidence-based psychotherapies for PTSD. This mission was reflected in staff beliefs about CPT and PE, enacted through clinic operations and supported by resources and leadership outside the teams. Lower reach teams were less specialized and situated in practice environments with less support and movement of patients between mental health teams. Medium reach teams shared characteristics of both high and lower reach teams.

Implications for D&I Research

Widespread implementation of PE and CPT requires not only training clinicians to deliver these treatments, but also addressing team and facility-level organizational factors. High reach teams prioritized evidence-based treatments and either de-implemented or shifted to other clinics delivery of other types of care. Teams that added EBPs without de-implementing other clinical responsibilities had lower reach.

Primary Funding Source

Department of Veterans Affairs

S13 Engaging clinicians and veterans in efforts to decrease benzodiazepines in posttraumatic stress disorder (PTSD): de-implementing through academic detailing

Craig Rosen

VA Palo Alto Health Care System, National Center for PTSD, Dissemination and Training Division, Menlo Park, CA, 94025, USA; Psychiatry and Behavioral Sciences, Stanford University School of Medicine, Menlo Park, CA, 94025, USA; Veterans Affairs Palo Alto Healthcare System, Center for Innovation to Implementation, Menlo Park, CA, 94025, USA

Correspondence: Craig Rosen

Background

Benzodiazepines are not recommended by practice guidelines for the treatment of PTSD. Despite these recommendations, providers routinely prescribe benzodiazepines in patients diagnosed with PTSD. Our research group has identified subgroups of Veterans with PTSD in whom the rate of prescribing of benzodiazepines is higher and to whom benzodiazepine use poses increased risk of harm. Resulting non-adherence to the guideline recommendations suggests that increased education, training, and consultation to both clinicians and Veterans can help ‘de-implement’ the use and harm of these agents.

Methods

Our team conducted research locally and in Northern California to examine the use of an Academic Detailing (AD) intervention to share developed decision support tools (brochures for both providers and patients). A clinical pharmacist met with providers in individualized outreach visits at their practice site to promote evidence-based treatment options. A dashboard developed by VA’s Pharmacy Benefits Management identified providers and patient-level prescribing data and allowed a focus on high-risk patients on a provider’s panel. Brochures were shared, key clinical messages about benzodiazepines in PTSD were discussed, and safer treatment alternatives were reviewed. Direct-to-consumer strategies were used by mailing patient brochures to identified subgroups of Veterans and were asked to discuss the content with their provider at upcoming visits.

Findings

Not all clinical pharmacists are strong detailers. We experienced variability in the skill of our pharmacists to engage providers in discussions of decreased benzodiazepine use. We also learned that clinical providers are typically too busy to have time to use performance dashboards designed to improve care. So the detailer has to offer brief key messages, clinical shortcuts, and resources to engage prescribing clinicians in a dialogue. Decreases in incidence and prevalence rates of benzodiazepine prescribing have been variable. More success has been observed locally due to repeated visits by the detailer.

Implications for D&I Research

AD is a promising and potentially efficacious de-implementation strategy, but it requires interpersonal skill and repeated contacts for it to be effective.

Primary Funding Source

Department of Veterans Affairs

S14 Participatory system dynamics: triangulating electronic health records, stakeholder expertise and simulation modeling to expand evidence-based practices

Lindsey Zimmerman^1,2, David Lounsbury^3,4, Craig Rosen^1,5,6, Rachel Kimerling¹, Jodie A. Trafton^5,6,7, Steven Lindley^5,8

¹VA Palo Alto Health Care System, National Center for PTSD, Dissemination and Training Division, Menlo Park, CA, 94025, USA; ²Psychiatry and Behavioral Sciences, University of Washington School of Medicine, Seattle, WA, 98195, USA; ³Epidemiology and Population Health, Albert Einstein College of Medicine, Bronx, NY, 10467, USA; ⁴Division of Community Collaboration and Implementation Science, Albert Einstein College of Medicine of Yeshiva University, Bronx, NY, 10467, USA; ⁵Psychiatry and Behavioral Sciences, Stanford University School of Medicine, Menlo Park, CA, 94025, USA; ⁶Veterans Affairs Palo Alto Healthcare System, Center for Innovation to Implementation, Menlo Park, CA, 94025, USA; ⁷VA Program Evaluation and Resource Center, VA Office of Mental Health Operations, Menlo Park, CA, 94025, USA; ⁸Outpatient Mental Health, Veterans Affairs Palo Alto Healthcare System, Menlo Park, CA, 94025, USA

Correspondence: Lindsey Zimmerman

Background

In some settings, evidence-based practices (EBPs) may be adopted by providers, prioritized by leadership and supported by health system infrastructure, yet still not reach an adequate proportion of patients. In these systems, local staff expertise and operations data can be synthesized in a participatory system dynamics (PSD) model, developed for selecting strategies to improve EBP reach. PSD modeling simulation empowers stakeholders by enabling them see the potential yield of implementation plans prior to implementation.

Methods

Our team worked with Veteran patients, local leadership, psychiatrists, psychologists, social workers, nurses and clerks to develop a PSD model of Veterans Health Administration (VA) outpatient mental health care. Our team coordinated with national VA offices responsible for disseminating evidence-based psychotherapy and pharmacotherapy, and conducting quality assurance. Qualitative group modeling sessions with frontline staff included developing the model structure, reviewing data inputs for validity, and model calibration checks against the historical behavior of the clinical system. Stakeholders provided a list of restructuring scenarios intended to improve EBP reach, which were evaluated with simulation.

Findings

Using a model parameterized with local clinic data, stakeholders viewed the operational mechanisms in their settings that they believed could expand EBP reach. Interdependent relationships among model parameters, such as patient factors (e.g., missed appointment rates), and system capacities (e.g., staffing) were examined. Stakeholders determined which implementation plan was likely to increase EBP reach the most, drilling down to examine the entire EBP continuum: referral to EBP, scheduling EBPs, initiating an EBP, completing a therapeutic dose of an EBP and patient benefit from an EBP.

Implications for D&I Research

PSD models evaluate stakeholders’ theories of clinic operation, testing explanatory mechanisms (i.e., local policies and procedures) by which EBP reach could be improved. PSD capitalizes on health record data, stakeholder expertise and simulation to optimize implementation. PSD may be a strategy for more timely patient access to EBPs even when no new system resources (e.g., new staff, new EBP trainings) are available. The PSD process increases staff general capacity for quality improvement, while the PSD modeling tool identifies implementation strategies tailored to local EBP-specific capacities and constraints.

Primary Funding Source

Department of Veterans Affairs

Big Data and Technology for Dissemination & Implementation Research

S15 Media influence and framing the public health narrative around unarmed deaths, race, and gun violence: mapping the Trayvon Martin case

Rahul Bhargava^1,2

¹Media Lab, Massachusetts Institute of Technology, Cambridge, MA, 02139, USA; ²Berkman Klein Center for Internet & Society, Harvard Law School, Cambridge, MA, 02138, USA

Correspondence: Rahul Bhargava

Background

Restrictions on gun-related research in the US create barriers to fully understanding the health implications of gun violence on racial disparities. Media coverage of gun related deaths and violence, particularly incidents that result in the death of unarmed African Americans, helps illustrate the gun violence landscape. The killing of unarmed teen Trayvon Martin offers a potent case study for examining the potential influence of participatory media on framing gun violence and race as a public health issue.

Methods

We collected data on the Trayvon Martin case from Media Cloud sources to analyze the media coverage volume and network. Conceiving of the media ecosystem as a network demanded a network analysis approach, for which we used Gephi and the PageRank algorithm. We complemented and informed the direction of our quantitative analysis through qualitative interviews with media activists involved in the early stages of the Trayvon Martin story. Peak coverage and the linked network of publishers were mapped from the date of his death, February 26, 2012, to April 30, 2012.

Findings

News coverage started as a short-lived, local Florida news piece. Media pressure grew after Martin’s family hired a civil rights attorney, and a few large influential media outlets generated the racial framing of the story. 8643 total sources linked to one another and the civil rights narrative was carried across the national media network through hyperlinks. Social media clicks (n = 1.2million) provided a proxy for readership and resulting action offline. Broadcast media remained an amplifier and gatekeeper of the narrative, but participatory public media activism co-created the framing of the story’s ultimate message about race

Implications for D&I Research

These findings show the utility of viewing online media influence metrics in the context of a network of diverse actors. Our results suggest that this network influenced the public discourse, sentiment, and response to gun violence and race. Mapping the network of online media sources that create, spread, and reinforce messages illustrates how the general public shapes the conversation through their own narratives. Network mapping provides opportunities for identifying professional and public influencers who can assist in framing gun violence and racial disparities as public health issues

Primary Funding Source

John D. and Catherine T. MacArthur Foundation, Open Society Foundation, and Ford Foundation

S16 Vaccine hesitancy and digital networks: the influence of social proof

Hal Roberts

Berkman Klein Center for Internet and Society, Harvard University, Cambridge, MA, 02138, USA

Background

Health experts recognize childhood vaccinations as one of the most important public health achievements of the twentieth century. The Internet, directly linked to vaccine hesitancy, is a well-documented health information source. The rise in numbers and combinations of vaccines along with globalized, non-hierarchical communication make today’s challenge of overcoming vaccine hesitancy particularly complex. Despite a variety of efforts to address this challenge, online vaccine hesitant communities remain robust.

Methods

We used Media Cloud’s searchable archive of over 350 million stories from 50 thousand media sources, along with tools to analyze that archive, to conduct quantitative and qualitative network analysis based on key word selection and community detection. We generated a network map of sources who published about vaccines from June 1, 2014 to March 1, 2015. We measured influence via hyperlink degree centrality, betweenness centrality, degree of freedom of speech, and sentiment for the top 400 sources. Pearson chi squared tests determined magnitudes of associations.

Findings

4,187 identified sources formed distinct networked communities based on hyperlink sharing. Those link-sharing communities separated cleanly into pro and vaccine hesitant groups. Pro vaccine sources were directly linked to most frequently (p = 0.027), but highly connected sources within the network were twice as likely to portray vaccine hesitant sentiment contrary to evidence (p = 0.03). Sources that were more successful in spreading their information through linked connections were non-evidence based (p = 0.001) with low or no degree of scientific peer review. (p = 0.033).

Implications for D&I Research

The social structure and function of the Internet means network science and visualization mapping is essential in understanding relationships and connections between various publishers. These findings demonstrate an online environment where scientific evidence may not be as influential as social proof, a form of imitation where individuals ascribe to social behaviors of others to resolve uncertainty, such as link sharing. A challenge for childhood vaccinations communication may not only be poor quality information online but also the norms and values of the most influential online social networks sharing that information. Understanding online influence and network sociology is key to maximizing the effectiveness of digital and social communications necessary for sustaining optimal vaccine acceptance and reducing vaccine hesitancy.

S17 Could fragmented communication networks reshape the narrative?: Evidence from tobacco and e-cigarette media framing

Laura Gibson

Tobacco Center of Regulatory Science, University of Pennsylvania School for Communication, Philadelphia, PA, 19104, USA

Background

Youth smoking behavior in 2003 could be predicted from content analysis of the public communication environment around tobacco. The more frequently tobacco appeared in the news (e.g., describing negative health effects and anti-smoking policies), the less youth used tobacco. However since 2003, media platforms have diversified – as have tobacco products.

Methods

Text from websites popular among 12-24 year olds were pulled from the Media Cloud database via tobacco terms. Three traditional media sources. (Associate Press [AP], top 50 U.S. newspapers, and broadcast news transcripts) were pulled by searching the Lexis-Nexis database for the same set of tobacco terms. Supervised learning was used to classify this initial sample of potentially related texts (n = 79,020) as truly tobacco-related (n = 62,232). We calculated the three-day frequency of mentions of each topic for each source (n =136). Online and traditional media were compared.

Findings

E-cigarette mentions accounted for (n = 4,167, 7%) while tobacco only mentions were the majority (n = 58,065, 93%). Results for both topics showed moderate coordination between websites with broadcast news and AP. Coordination was significant for e-cigarette coverage across all sources (r = .44), suggesting that different platforms are picking up on the same e-cigarette narrative of events. For tobacco, message coordination across websites and newspapers was absent.

Implications for D&I Research

Comparing new online networked messaging with traditional media sources allows us to evaluate how the narrative is changing as media evolve and new tobacco products emerge over time. Implications of this approach include visualizing general trends in narratives across the publishing network (rather than single issues or events), mapping of all data available from every source, and the inclusion of many examples of sources for each platform (i.e., multiple newspapers and websites). However e-cigarettes in particular are a controversial product, so there could still be fragmentation in type of coverage. Follow-up analysis of specific content of e-cigarette news coverage across sources will provide further information about whether or not the public communication environment is fragmented on this topic. Network science and visualization models can assist in improved hyperlinking and generating new connections for dissemination of optimally coordinated messaging around tobacco and e-cigarettes.

Primary Funding Source

National Institutes of Health - National Cancer Institute (NCI) of the National Institutes of Health (NIH) and FDA Center for Tobacco Products (CTP) under Award Number P50CA179546.

S18 Implementation of real time clinical decision support: the challenge of evaluation

Gabriel J Escobar¹, Vincent Liu², Benjamin Turk², Arona Ragins², Patricia Kipnis³

¹Division of Research, Kaiser Permanente, Oakland, CA, 94612, USA; ²Health Care Effectiveness, Kaiser Permanente Division of Research, Oakland, CA, 94612, USA; ³Decision Support, Kaiser Permanente Northern California, Oakland, CA, 94612, USA

Correspondence: Gabriel J Escobar

Background

Comprehensive electronic medical records (EMRs) permit access to large datasets with granular clinical data and can serve as platforms for clinical decision support (CDS). In theory, this combination should facilitate testing, implementation and evaluation of CDS tools to improve patient outcomes. In practice, demonstration of clinical benefit remains difficult. We describe challenges encountered in evaluation of two inpatient CDS systems deployed by Kaiser Permanente Northern California.

Methods

Using data from >1.0 M patients, we developed an early warning system for patient deterioration outside the intensive care unit (ICU) and a risk score for non-elective rehospitalization. The early warning system, deployed at 2 hospitals in 2013-2014, provides discrete, patient-specific probability estimates every 6 hours. The rehospitalization risk score, deployed in 7 hospitals in 2016, generates patient-specific probability estimates every morning. Implementation teams developed clinician workflows for both systems. Given KPNC’s 22 hospitals and unitary information capture (single EMR across the entire enterprise), we can assign probability estimates to patients at the pilot sites and virtual probability estimates at the other sites. This permits evaluating the CDS by comparing patient outcomes from pilot and non-pilot hospitals (adjusting for risk differences via multivariate matching, since technical limitations precluded randomization).

Findings

Based on observed mortality, rehospitalization, and cost decreases at the pilot sites, KPNC will deploy these two CDS systems in all its hospitals. We encountered these challenges in the evaluation process: selection of outcomes measures - including the problem of employing mortality as an outcome, since patients near the end of life may not desire further intervention; inability to access real time data for model development; lack of concordance between real time data used by CDSs and data that are stored for retrospective analyses; computational challenges involving extremely large datasets; need for balancing measures (e.g., possible increases in admissions to the ICU); and differential effects across study sites

Implications for D&I Research

Although the use of EMRs as platforms for real time CDS holds great promise, quantification of benefit is likely to remain challenging in the near future. Greater attention should be given to methodological aspects of the evaluation of CDS impact.

Primary Funding Source

Other (please specify below) - Gordon and Betty Moore Foundation

S19 Reducing readmissions through improving care transitions (RRTICT): development of a tailored approach to improve veteran outcomes

Ashley Ketterer Gruszkowski¹, Michael W. Kennedy¹, Emily Rentschler Drobek¹, Lior Turgeman², Aleksandra Sasha Milicevic², Terrence L. Hubert¹, Larissa Myaskovsky^1,3,4, Youxu C. Tjader², Robert J. Monte¹, Kathryn G. Sapnas⁵

¹Veterans Engineering Resource Center, VA Pittsburgh Healthcare System, Pittsburgh, PA, 15125, USA; ²Joseph M. Katz Graduate School of Business, University of Pittsburgh, Pittsburgh, PA, 15260, USA; ³Center for Health Equity Research and Promotion, VA Pittsburgh Healthcare System, Pittsburgh, PA, 15240, USA; ⁴University of Pittsburgh School of Medicine, University of Pittsburgh, Pittsburgh, PA, 15240, USA; ⁵Office of the Assistant Deputy Under Secretary of Health for Patient Care Services, VA Central Office, Washington, DC, 20420, USA

Correspondence: Ashley Ketterer Gruszkowski

Background

The Pittsburgh Veterans Engineering Resource Center (VERC) partnered with the University of Pittsburgh and the VA Office of Patient Care Services to develop the Reducing Readmissions through Improving Care Transitions (RRTICT) program. RRTICT is a suite of care strategies that combines 32 best practices from Veterans Administration Medical Centers (VAMC) and health services literature. The use of a near real-time readmission risk predictive model to assist clinical decision making and equitable provisions of care differentiates RRTICT from other programs.

Methods

The RRTICT model was derived using a set of over 7000 patients and has an 80% accuracy rate. It uses administrative, clinical, and patient descriptive data from the VA Corporate Data Warehouse to predict risk of hospital readmission (within 30-days of discharge, 30DRR) at initial hospital admission. Six VAMCs volunteered to pilot RRTICT for six months in FY15. Participating sites had to agree to engage a discharge team, participate on monthly calls, and submit a midpoint and final report. The model provided daily risk values for each patient on the pilot wards. At the conclusion of the pilot, the VERC used the VA Inpatient Evaluation Center (IPEC) all-cause readmission rate definition to calculate the 30DRR on the patients in the pilot sites. Calculated rates were compared to rates from the same six-month period in FY 2014 using the two proportions Z-test.

Findings

Sites that implemented RRTICT demonstrated an overall decrease in 30DRR by 28% (Table 1).

Implications for D&I Research

RRTICT provides a structured framework that helps translate evidence based interventions into clinical practice. By using “Big Data,” the RRTICT predictive model allows clinical teams to maximize their use of best practices to achieve the greatest equity of care. In addition, the success of reducing readmissions is encouraging further interest to expand RRTICT throughout the VHA.

Primary Funding Source

Department of Veterans Affairs

Table 1 RRTICT 30DRR Results

S20 Work system redesign and implementation of evidence-based hypertension protocol in specialty clinics using electronic health record prompts

Edmond Ramly¹, Diane R Lauver², Christie M Bartels³

¹Industrial and Systems Engineering, University of Wisconsin-Madison College of Engineering, Madison, WI, 53704, USA; ²Nursing, University of Wisconsin School of Nursing, Madison, WI, 53704, USA; ³Medicine, Rheumatology Division, University of Wisconsin School of Medicine and Public Health, Madison, WI, 53705, USA

Correspondence: Edmond Ramly

Background

Hypertension protocols improve hypertension control and CVD prevention in primary care but have not been tailored for implementation in specialty clinics, although specialty visits outnumbered US primary care visits in 2013. We aimed to engage clinic staff in work system redesign to tailor and implement a blood pressure (BP) protocol for specialty clinics using electronic health record (EHR) prompts.

Methods

Over six months, our multidisciplinary study team (specialty MD, nursing, systems engineering) engaged staff from three specialty clinics in focus groups, specialty-relevant education, EHR prompted workflows, and audit feedback. EHR alerts prompted routine re-measurement and EHR order set assisted with primary care follow up orders after elevated BPs. Four monthly interactive 1-on-1 audit and feedback sessions with staff used a Self Determination Theory-informed process to continuously identify and overcome barriers to improving protocol fidelity and reach. EHR data was used to monitor implementation outcomes: re-measurement of elevated BP, follow up orders, and timely follow up. We also administered a retrospective anonymous 15 item staff questionnaire.

Findings

We compared implementation outcomes for all visits with elevated BP at baseline and in the intervention study months. BP re-measurement improved from <2% of visits to 83%. Follow up orders for patients with confirmed elevated BP improved from 0% to 73%. Monthly patient-completed timely follow up improved from 29% to 71%, doubling timely follow-up in multivariable analysis (OR 2.1, CI 1.4-3). All groups benefited and black patients showed additional gains compared to white patients (OR 1.7, CI 1.1-2.5). Gains were sustained at 18 months. Additionally, the percentage of staff self-reporting high or extremely high BP care confidence (self-efficacy) rose from 20% before to 90% after implementation.

Implications for D&I Research

Work system redesign with EHR-prompted workflows resulted in success tailoring and implementing a hypertension protocol for specialty clinic staff. Results suggest sound intervention reach, feasibility, and acceptability including improved BP re-measurement, follow up orders, patient-completed timely follow up, and staff self-efficacy.

Primary Funding Source

Peer reviewed QI funding

Clinical (i.e., Primary, Specialty, Hospital, etc.) Care Settings

S21 Diffusion of Excellence initiative: accelerating the implementation of best practices in America's largest integrated healthcare delivery system

Shereef Elnahal¹, Andrea Ippolito², Hillary Peabody³, Carolyn Clancy¹

¹Office of the Under Secretary for Health, Organizational Excellence, US Department of Veterans Affairs, Washington, DC, 20005, USA; ²Office of Under Secretary for Health, Veterans Health Administration, Washington, DC, 20005, USA; ³Healthcare Performance Improvement and Innovation, Atlas Research, Washington, DC, 20005, USA

Correspondence: Shereef Elnahal

Background

The Veterans Health Administration (VHA) is America’s largest integrated healthcare system with over 1,700 sites of care, serving 8.76 million Veterans each year. Despite numerous accomplishments, VHA previously lacked a systematic mechanism for identifying and scaling best practices.

Methods

The model has five steps: ⁽¹⁾ identify promising practices; ⁽²⁾ find the champions; ⁽³⁾ adapt and replicate; ⁽⁴⁾ establish consistency and standardize; and ⁽⁵⁾ sustain and improve. Three guiding principles were governance, process, and technology.

Findings

Launched in October 2015, this initiative has already resulted in over 280 adaptations of gold status practices at over 70 facilities nationwide. Key accomplishments include:

• Facilitated adaptation provided to over 15 facilities to adapt 12 practices.

• Identified select processes for national rollout

• Prioritized vulnerable populations facing mental health, economic, and chronic disease related challenges:

  • o One practice that includes the use of tablets for patients to complete screening tools from the waiting room reduced data entry time by 17 minutes per patient. This process improves care for Veterans with mental health conditions, as it increased screening for suicide risk from 50% to 90% and cut in half the time to document suicide risk.

  • o One practice brings Veterans together in social worker facilitated group visits to discuss advance care planning. 85% of all participants took concrete steps to develop more comprehensive care plans, especially important as patients with lower socioeconomic status are less likely to have Advance Directives.

  • o One practice leverages pharmacists to support the primary care team, saving primary care providers 15 minutes per new patient appointment. This practice could save up to 2,000 hours of primary care providers’ time if implemented just for patients with diabetes.

Implications for D&I Research

The VHA diffusion initiative model offers early evidence that it is an engaging solution that can scale best practices that serve vulnerable patient populations.

Primary Funding Source

Department of Veterans Affairs

S22 Reducing disparities in a regional health improvement collaborative: a positive deviance approach

Randall Cebul¹, Thomas Love^1,2, Douglas Einstadter^1,3, Shari Bolen^1,4

¹Center for Health Care Research and Policy, Case Western Reserve University School of Medicine, Cleveland, OH, 44109, USA; ²Medicine and Epidemiology and Biostatistics, Better Health Partnership, Cleveland, OH, 44109, USA; ³MetroHealth Medical Center, Cleveland, OH, 44109, USA; ⁴Medicine, Better Health Partnership, Cleveland, OH, 44109, USA

Correspondence: Randall Cebul

Background

In northeast Ohio, our regional health improvement collaborative uses a positive deviance (PD) approach with public reporting to identify and disseminate best practices in primary care. PD identifies subgroups in a community whose strategies enable better solutions to problems than their peers, despite having similar challenges. We examine reductions in disparities in diabetes (DM) care and hypertension (HBP) control using the PD approach.

Methods

Observational study of disparities in DM care (four measures, over 7 years, by race/ethnicity) and HBP control (<140/90, over 2 years, by several social factors) among patients in diverse urban clinics. PD protocols were identified from clinic performance data, confirmed by interviews, and disseminated using practice coaching and biannual Learning Collaboratives. For DM, we modeled annualized region-wide changes in race/ethnicity gaps (highest minus lowest performing category). For HBP, we compared improvement in control across numerous disparities factors, including non-safety net vs. safety net clinics (SNC). For DM, 53 clinics reported DM achievement regularly across 2008-2014. In 2014, there were 34,185 DM patients: 46.3% non-white, 17.5% uninsured or Medicaid, and over 30% in the region’s lowest tertile for household income and education. For HBP, 35 clinics reported in both 2014 and 2015. In 2015, 42% of these clinics’ 122,740 patients were treated at SNCs.

Findings

For patients with DM in 2008, there was a 16.3 percentage point gap between white and Hispanic patients meeting all four care measures (achievement: 50.4% white, 47% black, and 34.1% Hispanic). In 2014, the gap was reduced to 4.4 points, an 11.9 point reduction. In a weighted regression, gap reduction was 1.72 points per year (95% CI: -2.74, -0.71; p = 0.002). For HBP, the top 9 improvers (2015 vs 2014) of 35 clinics were SNCs using PD with coaching. Together the 25 SNCs improved by 2.9 percentage points; non-SNCs improved by 0.5 point.

Implications for D&I Research

A PD approach in a primary care regional health improvement collaborative was associated with community-wide improvement and reductions in disparities in care and outcomes for important chronic conditions. Similar models for accelerating improvement should be tested in other regional improvement collaboratives that collectively serve more than 120 million Americans.

Primary Funding Source

The Robert Wood Johnson Foundation - Aligning Forces for Quality

S23 Building synergy among national health system initiatives and a regional health improvement collaborative: a case study of the Cleveland Veterans Affairs medical center

Brook Watts^1,2

¹ Informatics and Analytics, Louis Stokes Cleveland VA Medical Center, Cleveland, OH, 44106, USA; ² Department of Medicine, Case Western Reserve University, Cleveland, OH, 44106, USA

Correspondence: Brook Watts

Background

The Veterans Health Administration, the largest integrated health system in the U.S., has made substantial efforts in recent years to ensure system-wide provision of high-quality ambulatory care. These efforts include full implementation of the Patient-Centered Medical Home, increased use of non-face-to-face encounters, and tools to support population management strategies that emphasize care coordination and quality improvement. On a local level, the Cleveland VA Medical Center (Cleveland VAMC) developed and implemented a variety of best practices to ensure high quality primary care. In addition, 13 VA clinics recently joined the main clinic at Cleveland VAMC in becoming members of Better Health Partnership (BHP), a northeast Ohio regional health improvement collaborative (RHIC) that publicly reports clinic-level data twice yearly.

Methods

Leadership at the Cleveland VAMC emphasizes the use of clinical process and outcome data for internal and external benchmarking as a foundational tool for quality improvement efforts. Local population health management tools and regular performance reports at the provider and clinic level were developed as a partnership between operational, analytic, and clinical staff. By participating with BHP in public reporting and twice yearly regional Learning Collaboratives, the Cleveland VAMC learns from and collaborates with health care “competitors” to improve care for the region’s residents with chronic medical conditions. Exceptional achievement (top decile performance in the region) is recognized by BHP with publicly awarded “Gold Stars”.

Findings

The Cleveland VAMC has maintained a “5-Star Quality” rating on internal VA benchmarking metrics since 2010. Public reporting of all fourteen Cleveland VAMC primary care clinics through BHP began in 2016, including results for 10,146 VA patients with diabetes and 40,910 patients with hypertension. All fourteen clinics received “Gold Star Practice” awards for care and/or outcome achievement in 2015.

Implications for D&I Research

Local initiatives in a hospital system synergized with national system-wide quality improvement efforts and a regional health improvement collaborative. Public reporting and the sharing of best practices with provider systems outside the VA will benefit the community and strengthen even high performing VA facilities.

Primary Funding Source

Department of Veterans Affairs

S24 Applying lean principles to improve hepatitis C testing in VA primary care community clinics

Vera Yakovchenko^1,2, Angela Park³, William Lukesh³, Donald R. Miller^1,2, David Thornton⁴, Mari-Lynn Drainoni^1,2,5, Allen L. Gifford^1,2,7

¹Department of Health Law, Policy and Management, Boston University School of Public Health, Boston, MA, 02118, USA; ²Department of Veterans Affairs, VA HSR&D Center for Healthcare Organization and Implementation Research (CHOIR), Bedford, MA, 01730, USA; ³Boston VA Healthcare System, New England Veterans Engineering Resource Center, Boston, MA, 02132, USA; ⁴VA Boston Healthcare System, Boston, MA, 02139, USA; ⁵Department of Medicine, Boston University School of Medicine, Boston, MA, 02118, USA; ⁷Internal Medicine, Boston University School of Medicine, Boston, MA, 02215, USA

Correspondence: Vera Yakovchenko

Background

The proliferation of Lean and team-based quality improvement (QI) has been relatively unexplored at Department of Veterans Affairs (VA) community-based outpatient clinics (CBOCs), which often serve rural areas and socioeconomically underserved neighborhoods. Hepatitis C (HCV) testing is recommended for all those born between 1945-1965. Nevertheless, 42,008 (38%) VA New England 1945-1965 birth cohort Veterans have never been tested. We describe a primary care-based QI initiative using Lean principles, external facilitators, and local champions to increase HCV testing in CBOCs.

Methods

Four intervention and four matched control CBOCs across different healthcare systems were selected based on primary care volume (medium/large), HCV tested population (29.5-77.1%), and geographic location. The intervention consisted of: 1) collaborative on-site improvement sessions, 2) ongoing external facilitation, 3) champion-led Plan-Do-Study-Act cycles, and 4) audit and feedback. Using interrupted time series we analyzed pre- and post-intervention HCV testing rates, adjusting for clinic volume, implementation climate scores, and HCV prevalence.

Findings

Veterans with a first HCV test increased (p = 0.016) at intervention CBOCs by an average of 20.4% (4.4-36.2%) and without significant change (p = 0.204) at matched controls [5.3% (-0.5-15.1%)] over a three-month period. In the month prior to intervention, Lean intervention CBOCs were 8% of regional HCV testing and by intervention month three were 29%. Clinic staff participated in training sessions (N = 68) and responded (N = 45) to an organizational context instrument. At baseline, 43% of staff were satisfied with current HCV testing processes and 53% were familiar with QI. While 84% agreed that they have a sense of personal responsibility for improving patient care and outcomes only 31% agreed that they would receive appreciation for HCV testing Veterans. Higher improvement sites had more rapid uptake and integration of changes into workflow. The lowest baseline HCV testing performance site had the lowest implementation climate, and highest testing improvement with a 20-fold increase (1.9 to 38.1%) of newly tested.

Implications for D&I Research

This Lean and team-based multi-modal QI intervention activated HCV testing improvement in CBOCs. Future study will use Qualitative Comparative Analysis to identify the necessary context, organizational and interventional conditions conducive to adoption and sustainability before being scaled across CBOCs.

Primary Funding Source

Department of Veterans Affairs - QUERI

S25 What does the facilitation implementation strategy look like in the real world?

Shawna Smith¹, Julia Kyle², Mark S Bauer^3,4, Daniel Eisenberg⁵, Celeste Liebrecht^2,6, Michelle Barbaresso^2,6, Amy Kilbourne^2,7,8

¹Internal Medicine, Division of General Medicine, University of Michigan, Ann Arbor, MI, 48109, USA; ²Psychiatry, University of Michigan Medical School, Ann Arbor, MI, 48109, USA; ³Center for Healthcare Organization and Implementation Research, VA HSR&D, Bedford/Boston, MA, 02130, USA; ⁴Department of Psychiatry, Harvard Medical School, Boston, MA, 02115, USA; ⁵Health Management and Policy, University of Michigan School of Public Health, Ann Arbor, MI, 48109, USA; ⁶Center for Clinical Management Research, VA Ann Arbor Health System, Ann Arbor, MI, 48109, USA; ⁷Dept of Veterans Affairs, VA Quality Enhancement Research Initiative (QUERI), Washington, DC, 20420, USA; ⁸VA Health Services Research and Development, VA Quality Enhancement Research Initiative, Washington, DC, 20420, USA

Correspondence: Shawna Smith

Background

Evidence-based collaborative care programs can improve physical and mental health outcomes. However, barriers to implementation of effective programs in community-based practices are significant. External facilitators (EF), or change agents outside of the study site, have been highlighted as a key factor in successful implementation. Yet, few studies have studied the content and impact of EF as it relates to implementation.

Methods

49 community-based practices participating in a health systems trial of implementation strategies to improve the uptake of a psychosocial intervention, Life Goals, were randomized to receive either External Facilitation (EF) or EF augmented with Internal Facilitation (EF + IF) for 6-12 months. The EF logged their tasks, categorizing mode, personnel interaction, duration, and primary focus of each task. We examine the content of EF tasks, describing predominant themes, and testing differences in content and quantity of tasks across sites receiving EF vs. EF + IF.

Findings

1,037 tasks were logged by the EF between January 2015 and July 2016. The EF logged a median of 28 minutes (IQR: 24-41) per site per month. 64% of interactions were done via email and 34% via phone, with email requiring a mean 6 minutes of EF time and phone calls 23 minutes. Interactions at EF sites (N = 564) involved site administrators (31%), supervisors (29%) and providers (20%); at EF + IF sites (N = 473), 70% of interactions were with the assigned IF. Mean interaction times did not differ across EF and EF + IF (t = 0.47, p = 0.64), however content did. EF sites focused more on education (EF: 55%; EF + IF: 45%), while EF + IF sites focused more on strategic tools (EF: 10%; EF + IF: 18%), including leveraging resources, marketing, and developing implementation plans, and reinforcement (EF: 26%; EF + IF: 36%).

Implications for D&I Research

Our uniquely-detailed purview of EF tasks provides insight into the mechanisms of EF as it impacts implementation efforts. Our results show that the focus of EF activity becomes more strategic when augmented with IF. Future work will link EF data to data collected by IFs to further describe differences in mechanisms of EF vs. EF + IF implementation strategies, and also explore site-level differences in EF interactions as they relate to implementation success and mediators of implementation success.

Primary Funding Source

National Institutes of Health - R01MH099898

S26 Lesson learned from NCI SPRINT entrepreneurs: commercializing a tobacco treatment for cancer center implementation

Elyse Park¹, Giselle Perez¹, Jamie Ostroff²

¹MIHP, MGH, Boston, MA, 02114, USA; ²Psychiatry, Memorial Sloan Kettering Cancer Center, New York, NY, 10022, USA

Correspondence: Elyse Park

Background

As clinical researchers investigating the integration of evidence-based, tobacco treatment into cancer care, we participated in an NCI Dissemination and Implementation Research Program initiative beginning in July 2016, Speeding Research-tested INTerventions (SPRINT). SPRINT’s trains researchers in the business of real-world transformation of behavioral interventions into commercially viable and scalable products or services. We have worked towards creating and refining a business plan for our promising intervention. Our goal is to market tobacco treatment, as a 3-part product, to a variety of buyers (i.e., hospitals, health care systems, employers).

Methods

The SPRINT course consists of didactic learning, interactive presentations, expert consultation, collaborative learning and completion of targeted interviews focusing on our potential customers’ preferences and needs. We were tasked to complete 30 interviews with potential "end users" (e.g., patients, clinicians, administrators, insurers, and businesses). Our initial value propositions, are as follows: 1) Providers and patients need a detailed tobacco treatment manual/guide; 2) Cancer patients have unique tobacco treatment needs; 3) Oncology nurses will be eager to deliver tobacco treatment; and 4) There are specialized training needs for tobacco treatment in cancer patients.

Findings

To date we have completed 20 interviews, all of which have played vital information supporting or disproving our value propositions. We developed a Value Proposition Canvas to assess interest in our products (e.g., manuals, certification and training), perceptions of customer “Pains” (reducing no shows) this product could address, and customer “Gains” (engaging patients in tobacco treatment) that our product would achieve. We also identified three types of customer archetypes (oncology providers, tobacco treatment specialists, and cancer patients) representing our potential end users. Currently, we are exploring channel distributions (e.g., publishing and training companies) as well as payment and reimbursement venues. Primary interview themes will be presented.

Implications for D&I Research

Lessons learned from our training experience- both its process and results- will guide our ability to transform our research-tested intervention into a marketable product. Complementary to traditional behavior change theory and strategies, business concepts and methods, introduced through SPRINT and explored through a series of interviews, can shape the translation of behavioral intervention into real world settings.

Primary Funding Source

National Institutes of Health - NCI SPRINT initiative

S27 Dissemination and implementation in health systems: more GPS than self-driving car

Sarah Greene¹, Michael Parchman², Brian Austin², Eric Larson²

¹Research, Health Care Systems Research Network, Seattle, WA, 98115, USA; ²MacColl Center for Health Care Innovation and Senior Investigator, Group Health Research Institute, Seattle, WA, 98101, USA

Correspondence: Sarah Greene

Background

Effective dissemination of research findings into practice is a critical element of the research lifecycle, but one that is often constrained by lack of resources, time, and/or skill. Moreover, spreading evidence into organizations requires substantial effort, as implementation is rarely a turn-key process. Overcoming these barriers and enhancing availability and usability of evidence is a critically important step in realizing the promise of true learning healthcare systems. Using the example of an embedded research center in a care delivery system, we explored the barriers and potential facilitators to more effective dissemination and implementation (D&I).

Methods

Through key informant interviews and an environmental scan, we explored the barriers and facilitators to researchers' willingness and ability to engage in D&I activities beyond conference presentations and journal publications. We assessed the potential business model(s) for effective D&I, and also discussed with interviewees whether there is a moral imperative to engage in D&I. The environmental scan identified case studies of exemplar entities that sustained their research through strategic D&I activities.

Findings

Through 28 semi-structured interviews, we identified a range of challenges and opportunities related to researchers' dissemination and implementation capabilities. Notably, some researchers felt compelled to intensify their D&I skills and activities, whereas others felt that good science and publications were sufficient end products. Challenges impeding D&I included lack of funding, lack of protected time for building connections and nurturing relationships with health system leaders, and incomplete understanding of the market potential for research products. Use of digital and social media was identified as an opportunity, but many researchers are not proficient in these newer approaches. Fifteen exemplar organizations were included in the environmental scan, and illustrate a range of business models and approaches to supporting translation of research into practice.

Implications for D&I Research

Dissemination and implementation of research results often amounts to an "unfunded mandate" for many researchers. Moreover, many would benefit from training on how to effectively communicate their results to healthcare decision-makers, policymakers, and other stakeholders. The entire research community would benefit from addressing the structural, interpersonal, financial, and technical aspects that invigorate and incentivize more effective D&I.

Primary Funding Source

GHRI Director's Development Fund

S28 Implementation of an integrated care management program in community pharmacies: identifying barriers and employing implementation strategies

Stefanie Ferreri¹, Chris Shea², Megan Smith³, Kea Turner⁴

¹Division of Practice Advancement and Clinical Education, University of North Carolina Eshelman School of Pharmacy, Chapel Hill, NC, 27599, USA; ²Health Policy and Management, The University of North Carolina at Chapel Hill, Chapel Hill, NC, 27599, USA; ³Pharmacy Practice, University of Arkansas for Medical Sciences College of Pharmacy, Little Rock, AR, 72205, USA; ⁴Health Policy and Management, University of North Carolina Gillings School of Global Public Health, Chapel Hill, NC, 27599, USA

Correspondence: Stefanie Ferreri

Background

Preliminary evidence and expert guidance suggest integrated care management (ICM) programs in community pharmacies contribute to improved outcomes and lower cost of care. However, existing payment models prioritize drug dispensing rather than integrated care services. In this study, we identified and categorized barriers to implementing ICM services among a network of community pharmacies and documented implementation strategies employed to assist uptake.

Methods

The Community Pharmacy Enhanced Services Network (CPESN) of North Carolina focuses on implementation of a payment model to support delivery of enhanced services. At launch, 123 community pharmacies joined CPESN. We conducted interviews with pharmacists representing 73 pharmacies to identify implementation barriers within the ICM program: ⁽¹⁾ Initiating; ⁽²⁾ accessing internal information; ⁽³⁾ accessing external information; ⁽⁴⁾ scheduling patients; ⁽⁵⁾ performing patient interviews; ⁽⁶⁾ developing assessment; ⁽⁷⁾ developing a care plan; ⁽⁸⁾ documenting; ⁽⁹⁾ coordinating care; and ⁽¹⁰⁾ performing follow-up. We also conducted 23 interviews with CPESN staff and collaborators who were employing implementation strategies to assist pharmacies. We used a semi-structured interview guide informed by Proctor et al.’s (2013) recommendations for reporting implementation strategies and Powell et al.’s (2015) implementation strategy compilation.

Findings

Most pharmacists reported barriers around “Initiating” and “Documenting”. Many pharmacists lacked a planning process for incorporating new services, underestimated resources needed, and/or did not ensure timely availability of resources. The most commonly reported barriers were around adequate staffing and time. Also, documentation in the web-based platform was problematic due to unclear requirements, poor system design, time, and system connectivity/responsiveness. Regarding implementation strategies, several strategies from the Powell et al. compilation were reported, such as conducting educational outreach and audit and feedback. The strategies have been focused on utilizing staff time more efficiently, facilitating documentation in the web-based platform, and monitoring performance.

Implications for D&I Research

Our project highlights challenges with implementing new care processes within a complex program. Identifying barriers within an ICM program facilitates targeted interventions for pharmacists’ needs; however, systematic evaluation of each strategy is needed. Finally, documenting implementation strategies promotes more efficient use of implementation resources and reveals barriers not currently being addressed.

Primary Funding Source

Centers for Medicare and Medicaid Services - CMS Demonstration Grant

S29 Implementation strategies for patient care services in community pharmacy: a systematic review

Jennifer Bacci¹, Kyle Bigham¹, Geoffrey Curran², Stefanie Ferreri³, Caity Frail⁴, Cory Hamata¹, Terry Jankowski⁵, Wendy Lantaff⁶, Melissa Somma McGivney⁷, Margie Snyder⁶

¹Pharmacy Practice, University of Washington School of Pharmacy, Seattle, WA, 98195, USA; ²Department of Pharmacy Practice, University of Arkansas for Medical Sciences, Little Rock, AR, 72205, USA; ³Division of Practice Advancement and Clinical Education, University of North Carolina Eshelman School of Pharmacy, Chapel Hill, NC, 27599, USA; ⁴Pharmacy Practice, University of Minnesota College of Pharmacy, Minneapolis, MN, 55455, USA; ⁵Library Science, University of Washington, Seattle, WA, 98195, USA; ⁶Pharmacy Practice, Purdue University College of Pharmacy, Indianapolis, IN, 46202, USA; ⁷Pharmacy Practice, University of Pittsburg School of Pharmacy, Pittsburg, PA, 15261, USA

Correspondence: Jennifer Bacci

Background

Studies have identified the positive impact of patient care services in community pharmacy; yet, widespread implementation remains limited. Identifying the most effective implementation strategies is critical to sustaining evidence-based patient care in this setting. This scoping systematic review aims to describe strategies used to ⁽¹⁾ implement and ⁽²⁾ evaluate the implementation of patient care services in community pharmacies.

Methods

PubMed/ MEDLINE, EMBASE, and International Pharmaceutical Abstracts were searched until March 9, April 20, and April 5, 2016, respectively. Innovations in Pharmacy, Implementation Science, and reference lists of excluded articles were hand searched. The primary investigator reviewed each article’s abstract to assess eligibility for full-text review. At least 2 investigators conducted full-text reviews. Discrepancies were resolved by a third investigator. To be included, articles had to ⁽¹⁾ be written in English and published in 1985 or later; ⁽²⁾ describe a peer-reviewed empirical study or practice experience implementing a patient care service in a community pharmacy; and, ⁽³⁾ describe the strategies used to implement and/ or evaluate the implementation of the service. Data are being extracted from each article by 2 investigators using a standardized form, including pharmacy type(s), patient care service type(s), workflow and financial models, pharmacy staff roles, implementation strategies, implementation outcomes, evaluation method, and conceptual model/ framework. Implementation strategies are being categorized using terminology from the Expert Recommendations for Implementation Change (ERIC) study by Waltz and colleagues.

Findings

Of the 3,046 articles retrieved, 237 were included. The majority describe implementation of medication management, disease state management education, and risk reduction services in the US, Europe, and Australia. Preliminary results indicate an emphasis on adapting and tailoring to local context and training and educating stakeholders as implementation strategies and qualitative and process evaluation. Final results will be completed in October 2016.

Implications for D&I Research

This review is one of the first efforts to comprehensively identify implementation strategies for patient care services in community pharmacies and is a critical step in identifying the most effective implementation approaches. The results of this work will enable implementation scientists and clinicians to refine and test strategies that result in sustainable and scalable patient care service implementation.

S30 Testing implementation strategies for anticoagulation improvement in clinical pharmacy clinics: a qualitative study

Megan McCullough^1,2, Chris Gillespie¹, Beth Ann Petrakis¹, Ellen Jones³, Angela Park⁴, Carol VanDeusen Lukas^5,6, Adam Rose^1,7

¹Center for Healthcare Organization and Implementation Research (CHOIR), VA Health Services Research & Development, Bedford, MA, 01730, USA; ²Department of Health Law, Policy and Management, Boston University School of Public Health, Boston, MA, 02118, USA; ³Pharmacy, Central Western Massachusetts VA Healthcare System, Leeds, MA, 01053, USA; ⁴Boston VA Healthcare System, New England Veterans Engineering Resource Center, Boston, MA, 02130, USA; ⁵Research, VA Boston Healthcare System, Center for Healthcare Organization and Implementation Research, Boston, MA, 02130, USA; ⁶Health Policy and Management, Boston University School of Public Health, Boston, MA, 02118, USA; ⁷Internal Medicine, Boston University Medical Center, Boston, MA, 02119, USA

Correspondence: Megan McCullough

Background

Implementation science (IS) has recently concentrated efforts on assessing what implementation strategies are effective and in what circumstances. In complex interventions, a “bundle” of implementation strategies are often applied. In a four-year pharmacist-focused Anticoagulation Care Improvement Initiative (ACCII), we implemented a strategy bundle (audit and feedback, blended internal-external facilitation, small tests of cyclical change, and ongoing consultation) to improve anticoagulation across 8 Veterans Health Administration (VHA) medical centers in a single region. This study describes the significance and impact of these combined strategies on successful implementation.

Methods

We conducted an average of 50 semi-structured interviews with all frontline ACC staff annually for 4 years with a special focus on implementation strategies. Interviews were conducted with the pharmacy leadership twice (N = 22) and the External Facilitation Team (delivered intervention) once (N = 5). Analysis drew deductively on Promoting Action on Research Implementation in Health Services (PARIHS) as well as an emergent thematic analysis. Analysis focused on identifying factors related to the impact and effectiveness of strategies employed.

Findings

Sites with greater improvement: 1) Integrated and used all IS strategies. 2) Had support from managers so full implementation of strategies could occur. 3) Spread strategies throughout entire clinical team (i.e., made quality improvement everybody’s job) as opposed to having a designated person. Sites with less successful implementation of the ACC II had less leadership support and lower engagement but most important they divided the bundle of strategies (i.e., pursued some but not all of them).

Implications for D&I Research

While the value of bundling IS strategies has been recognized, this study suggests how and why bundling can be effective in successful implementation as well as where it might not. Our findings can help guide researchers who seek to choose and then incorporate a bundle of implementation strategies. This study contributes to a growing literature regarding if and how bundles of implementation strategies impact successful implementation.

Primary Funding Source

Department of Veterans Affairs - Research Grant from the Department of Veterans Affairs

S31 An implementation research model for pharmacy: application of the Consolidated Framework for Implementation Research (CFIR) to community pharmacy

Sarah J. Shoemaker¹, Geoffrey Curran², Jeremy Thomas², Benjamin Teeter², Holly Swan¹

¹US Health, Abt Associates, Cambridge, MA, 02138, USA; ²Department of Pharmacy Practice, University of Arkansas for Medical Sciences, Little Rock, AR, 72205, USA

Correspondence: Sarah J. Shoemaker

Background

Pharmacists are providing direct patient care (e.g., medication therapy management) and delivering important prevention and screening (e.g., immunizations, rapid HIV testing) interventions. Community pharmacies are an increasingly important health care setting with opportunities for improving quality and safety, yet little is understood about determinants of implementation. Given community pharmacy’s unique features for consideration in implementation research, we developed a model for implementation research for this setting based on the Consolidated Framework for Implementation Research (CFIR).

Methods

We conducted a critical review examining literature on services and interventions provided in community pharmacies, including MTM services, rapid HIV testing, and immunizations provided in community pharmacies. We scanned select titles and abstracts to identify implementation studies or studies that sought to understand barriers and facilitators to implementing these types of services in community pharmacies. We found few systematic implementation studies in these settings. To synthesize findings and develop a model, we chose the CFIR because of its ability to situate potential implementation determinants across a wide range of constructs.

Findings

While Intervention Characteristics depend on individual interventions, of note here are “relative advantage” and “complexity”. The former because implementation of services can pose a cost-benefit challenge where dispensing is the primary role and the latter because of the challenge of integrating services into the dispensing workflow. In terms of Outer Setting, pharmacies are subject to external policy and incentives (e.g., CMS star ratings, payor incentives) but to a lesser extent than other health settings. For Inner Setting, important structural characteristics include pharmacy type (e.g., chain, independent, safety net), extent of “patient-centeredness”, and relationships with schools of pharmacy-- which facilitate social networks, communications, and “in-kind” staffing (e.g., students, residents). Key Characteristics of Individuals include training, preparedness, and self-efficacy of the pharmacist for providing new services (e.g., MTM). Finally, Process constructs of note from pharmacy implementation studies include the importance of champions/active local change agents.

Implications for D&I Research

As pharmacists’ roles in health care expand, models to inform implementation research in community pharmacy (and other) settings are crucially needed. Additionally, further application of the CFIR to other healthcare settings expands the field’s understanding of the utility of the framework.

S32 Adaptation of a motivational interviewing intervention in community pharmacies: application of the Wiltsey Stirman framework to characterize modifications in a 4-site implementation demonstration study

Benjamin Teeter¹, Jeremy Thomas¹, Geoffrey Curran¹, Appathurai Balamurugan²

¹Department of Pharmacy Practice, University of Arkansas for Medical Sciences, Little Rock, AR, 72205, USA; ²Center for Health Advancement, Arkansas Department of Health, Little Rock, AR, 72205, USA

Correspondence: Benjamin Teeter

Background

As community pharmacies expand their role in healthcare through the implementation of cognitive services into their dispensing workflows, identifying the different modifications made to these services for implementation and the impact of these modifications are important to determining their success or failure. Previous research by Wiltsey Stirman and colleagues (2013) identified 4 types of contextual modifications, 12 types of content modifications, and 7 levels at which modifications occur, all of which have the potential to impact the desired benefits of the intervention. This study seeks to apply their framework in the community pharmacy setting to determine the modifications that occur during implementation of cognitive services.

Methods

In a 4-site demonstration project, pharmacists were trained to provide a brief Motivational Interviewing (MI) intervention to at least 50 patients who were non-adherent to antihypertensive medications. Training included a three-hour online course in MI and in-pharmacy training on identifying eligible patients and documenting the intervention. Observations and semi-structured interviews took place in 4 community pharmacies that implemented the MI intervention. Interviews covered modifications to the process of identifying eligible patients, MI interventions, and documenting the intervention. Data was coded using Wiltsey Stirman and colleagues’ framework.

Findings

Contextual modifications were made to the format of the intervention (e.g., telephone instead of in-pharmacy). All 4 pharmacies reported conducting at least some of their interventions via telephone. The most common content modification was ‘Loosening the Structure’ (e.g., cancelling use of computer alerts) followed by ‘Drifting or Departing’ (e.g., stopped MI with defensive patients). Although less frequent, ‘Adding Elements’ (e.g., reminder cards) and ‘Repeating Elements’ (e.g., identification of patients) were also mentioned. Modifications to the MI intervention typically occurred at the provider/facilitator level. Interestingly, modifications seemed to be made to fit the pharmacists’ needs, not the needs of their patients.

Implications for D&I Research

This study demonstrates the utility of Wiltsey Stirman and colleagues’ framework in the pharmacy setting. In contrast to previous research, individual level content modifications were uncommon. Data from this study will guide future research within a larger sample of pharmacies in an effort to determine the modifications that are detrimental and beneficial to patient outcomes and sustainability of services.

S33 Evidence supporting clinician acceptance of a standardized handoff process: findings from a hybrid effectiveness-implementation study of operating room to intensive care unit handoffs

Meghan Lane-Fall^1,2,3, Rinad Beidas^2,4, Laura Di Taranti¹, Sruthi Buddai¹, Enrique Torres Hernandez⁵, Jerome Watts⁶, Lee Fleisher^1,2, Frances Barg^1,7

¹Anesthesiology and Critical Care, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, 19104-4865, USA; ²Leonard Davis Institute of Health Economics, University of Pennsylvania, Philadelphia, PA, 19104, USA; ³Center for Healthcare Improvement and Patient Safety, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, 19104, USA; ⁴Psychiatry, University of Pennsylvania, Philadelphia, PA, 19104, USA; ⁵School of Science, Engineering and Technology, St. Mary's University, San Antonio, TX, 78228, USA; ⁶Haverford College, Haverford, PA, 19041, USA; ⁷Family Medicine and Community Health, University of Pennsylvania, Philadelphia, PA, 19104, USA

Correspondence: Meghan Lane-Fall

Background

Hospitalized patients whose care is transferred from one healthcare team to another experience a “handoff”, during which patient information and accountability are transferred. After implementing a standardized handoff protocol for patients being transferred from a surgical operating room (OR) to the intensive care unit (ICU), we sought to determine clinician acceptance of the new process.

Methods

As part of the Handoffs and Transitions in Critical Care (HATRICC) study, we conducted interviews, focus groups, collected surveys, observed patient handoffs, and surveilled the health system’s event reporting system from 2014 to 2016. We defined “process adherence” as following our standardized handoff protocol's steps in a pre-specified order. Qualitative data were analyzed with a grounded theory approach while quantitative data were analyzed with descriptive and bivariate statistics.

Findings

We conducted 109 interviews, 8 focus groups (71 participants), and collected 445 responses across 3 surveys from clinicians participating in OR to ICU handoffs, including physicians, nurses, and advanced practice providers. In interviews and focus groups before the intervention, a “perfect handoff” was variably described, but this definition coalesced afterward to become consistent with the new handoff protocol. In surveys, 143/152 (94.1%) and 136/152 (89.5%) respondents found the new process to be appropriate and acceptable, respectively. 125/160 (78.1%) respondents said that they usually or always used the new process in their handoffs. 107/156 (68.7%) said that the new process made patient care “better” or “much better”. We observed 61 patient transfers before and 156 transfers after standardized handoff implementation. In the postintervention period, 156/156 transfers included an observable handoff, compared to 53/61 (86.9%) before the intervention (p < 0.001). Clinicians adhered to the new handoff process in 110/156 (70.5%) handoffs. Review of the hospital’s event reporting system revealed safety reports including the term “HATRICC” (study acronym) used as both a noun and a verb, indicating familiarity with the process.

Implications for D&I Research

Multiple data sources may be used to assess clinician acceptance of a process change. In this study, we demonstrated through interviews, focus groups, surveys, direct observation, and event report review that clinicians were accepting of a standardized OR to ICU handoff process.

Primary Funding Source

Safety Scientist Career Development Award

S34 Defying expectations for care management practice uptake: a mixed methods analysis of positive deviants in a national sample of physician organizations

Isomi Miake-Lye¹, Tanya Olmos², Emmeline Chuang¹, Hector Rodriguez³, Gerald Kominski⁴, Becky Yano^1,5, Stephen Shortell⁶

¹Health Policy and Management, UCLA Fielding School of Public Health, Los Angeles, CA, 90024, USA; ²Health Policy and Management, University of California, Los Angeles, San Fernando, CA, 91340, USA; ³Health Policy and Management, University of California - Berkeley, School of Public Health, Berkeley, CA, 94720, USA; ⁴Department of Health Policy and Management, UCLA Center for Health Policy Research, Los Angeles, CA, 90272, USA; ⁵HSR&D Center for the Study of Healthcare Innovation, Implementation & Policy (CSHIIP), VA Greater Los Angeles Healthcare System, Sepulveda, CA, 91343, USA; ⁶School of Public Health, University of California, Berkeley, Berkeley, CA, 94720, USA

Correspondence: Isomi Miake-Lye

Background

Evidence-based practices are often slow to diffuse from the settings in which they were initially developed. Given the differing contexts of these later adopter organizations, new strategies may be needed to support adoption. This study seeks to identify organizational characteristics associated with non-adoption of care management practices (CMPs) for diabetes and to examine the role of culture, leadership, and organizational priorities in non-adopters and positive deviants.

Methods

In this explanatory sequential mixed methods study, we first use quantitative analyses to identify organizational characteristics associated with non-adopter status in cross-sectional data from the National Survey of Physician Organizations (NSPO3, 2012-2013; n = 1,328), a nationally representative sample of physician organizations. We then conducted semi-structured interviews with key stakeholders (i.e., primary care provider, nurse, administrator) in a purposive sample of positive deviant and non-adopter organizations. Qualitative directed content analysis builds from this work to explore the role of culture, leadership, and organizational priorities in the adoption decisions for diabetes CMPs among non-adopter and positive deviant organizations.

Findings

We identified organizational characteristics including using electronic medical records, using quality improvement systems, making investments in quality of care, having physician ownership, and being in California. These factors were then used as sampling criteria in the qualitative phase. Two non-adopter sites and two positive deviant sites were included in interviews. All sites shared similarities in culture that resembled the laggard characteristics described by Diffusion of Innovation, including aversion to change. The main differences identified in qualitative analyses between non-adopter and positive deviant sites were if they considered diabetes management to be within their scope of practice and if they described support from outside organizations with CMPs.

Implications for D&I Research

Non-adopters and positive deviants had cultural similarities in addition to being similar on quantitative measures. While non-adopters may require more outside support to adopt diabetes CMPs, they may also be non-adopters for key strategic reasons, such as focusing solely on specialty care. Lessons learned from positive deviants may be key in building strategies to combat variations in care, and more attention to these organizations is warranted.

Primary Funding Source

Agency for Healthcare Research and Quality - AHRQ R36 HS024176-01; National Study of Physician Organizations III was funded by the Robert Wood Johnson Foundation (Award No. 68847)

S35 Using implementation theory to evaluate the impact of technology on nurses’ knowledge and use of best practices in acute care

Mary Hook

Center for Nursing Research and Practice, Aurora Health Care, Milwaukee, WI, 53233, USA

Background

Advances in health information technology (HIT) and the use of electronic clinical decision support (CDS) tools to support nurses to know and use evidence-based practices (EBP) hold great promise, but relatively untested in acute care.

Methods

This pre/post mixed methods study was conducted to evaluate the impact of embedding EBP recommendations into policy and the electronic health record (EHR) to support nurses to know and use best practices to improve patient outcomes. The Dissemination of Evidence-based Policy Framework (Dobson, Brownson, & Weiss, 2012) was adapted and guided the study, proposing that the impact of an EBP innovation is influenced by how it is deployed. The study was conducted with consenting inpatient nursing units (N = 28 units) from 3 diverse facilities where the policies and technology were deployed. Non-participant observations, audits, nurse surveys, patient surveys, process and outcome metrics were gathered to describe context, nurse knowledge, use of EBP behaviors, and achievement of nurse-sensitive patient outcomes. A multimodal implementation intervention was delivered with audit/feedback, optimization training, and support for unit-based implementation related to 6 nurse-sensitive phenomena: pain, falls, pressure ulcers, medication adherence, delirium, and depression/suicide over a 6 month period.

Findings

Baseline findings revealed a supportive culture, functioning technology, with gaps in knowledge (N = 536, Mean Score = 56.3% correct, SD 8.4) and use of EBP practices using dissemination-based deployment conditions. Baseline results were used to create the feedback-based optimization training curriculum. Training Sessions (n = 57) were attended by 89% of staff and 100% of leaders with high ratings for goal achievement and commitment to implementation. Units were observed to have limited capacity to monitor and maintain EBP behaviors during the implementation phase. Post assessment revealed some improvement in knowledge (N = 523, Mean Score = 61.8% correct, SD, 8.8, p < 0.001) and use of EBP behaviors with little change in patient outcomes.

Implications for D&I Research

Gaps in knowledge and use of technology-supported EBP were identified and addressed with a multimodal implementation intervention. Despite high training participation and commitment, units had limited capacity to implement and maintain best practices over time. These findings suggest that the impact of EBP-based technology will be limited when deployment is based on dissemination training alone.

Primary Funding Source

US Army Medical Research and Materiel Command (USAMRMC) Grant No. W81XWH-13-1-0034

S36 Integrating emergent mhealth apps into pediatric practice: a mixed methods implementation project

Linda Fleisher^1,2, Alexander Fiks³, Katie Halkyard¹, Rachel Gruver¹, Emily Sykes¹

¹Research Institute, Children's Hospital of Philadelphia, Philadelphia, PA, 19104, USA; ²Population Science, Fox Chase Cancer Center, Philadelpia, PA, 19015, USA; ³General Pediatrics, University of Pennsylvania, Philadelphia, PA, 19066, USA

Correspondence: Linda Fleisher

Background

The explosion of digital/mHealth tools for consumers coupled with their increased integration into health care systems has prompted the need for systematic organizational processes to evaluate and determine which tools should be provided or recommended. The iAPP (Integrating Apps into Pediatric Practice) initiative was created to develop a strategy to integrate evidence-based health apps for patients and families into pediatric care.

Methods

This 2-year project includes: an implementation framework-guided environmental scan of leading pediatric hospitals, parent surveys , provider surveys, development of a decision framework and governance process, and validation through a number of strategically selected pilot projects. This abstract focuses on the environmental scan (including surveys and interviews) results and the resulting implementation tools (decision tree, driver diagram, and process documents) to guide mHealth integration. Leading pediatric hospitals (N = 7) were recruited based on their reputation in innovation and mHealth. The appropriate contact (Director-level) completed a brief online survey prior to the interview. The interview guide, developed based on four implementation frameworks (e.g. TAM, Rodgers’ Innovation) included questions on strategic leadership, culture, regulatory, economic, technical, legal and governance factors within the context of mHealth tools for patients and families.

Findings

Findings from both the survey and interviews highlighted that although these hospitals are entering the digital health/mobile health space (90% using digital health tools), only 3 had policies either in development/implementation or relied on external organizations (e.g. HIMSS) for guidance. The most frequent topics for these mHealth tools were patient education and chronic disease management and most are deployed through the app store or institutional websites. Respondents indicated that the 3 top factors to determining whether to use a 3^rd-party app were cost, credibility and level of clinician support.

Implications for D&I Research

This presentation focuses on the findings of this environmental scan that have applicability to the integration of mHealth into practice as well as our resulting decision making and implementation process tools addressing the evidence of efficacy, perceived clinical utility/impact on quality, interest of clinicians and families in using a particular app, the fit of the app in workflows, age appropriateness, and sensitivity to culture, cost and the potential for adverse consequences/safety.

Primary Funding Source

Pediatric Chair's Initiative

S37 Promoting change in the genitourinary care of postmenopausal women through clinician education and electronic health record tools: a cluster randomized trial

Kimberly Vesco^1,2, Kate Beadle³, Joanna Bulkley¹, Ashley Stoneburner¹, Michael Leo¹, Amanda Clark^1,2

¹Science Programs Department, Kaiser Permanente Center for Health Research, Northwest, Portland, OR, 97227, USA; ²Obstetrics & Gynecology, Northwest Permanente, Clackamas, OR, 97015, USA; ³Obstetrics & Gynecology, Kaiser Permanente, Northwest, Portland, OR, 97227, USA

Correspondence: Kimberly Vesco

Background

Data from our health care system (2011-2012) suggested that the genitourinary syndrome of menopause (GSM), an easily treated condition that progresses in severity with age, was underdiagnosed in our patient population (observed 4%, expected 20-45%). To promote greater detection and treatment of GSM, we created a clinician-focused intervention which included a suite of evidence-based electronic health record (EHR) tools and clinician education. We conducted a cluster-randomized trial to test the efficacy of the intervention.

Methods

We randomized Primary Care (PC) and Gynecology (GYN) clinics to intervention (8 PC, 3 GYN) or control (8 PC, 3 GYN). From September to November 2014, we provided training about GSM diagnosis, treatment, and the EHR tools through face-to-face presentations at each intervention clinic and through an on-line video. Control clinicians received no training or notification about the tools. Our primary outcome was the proportion of well visits with GSM-related diagnoses and prescriptions from 11/15/14 through 11/15/15. We also assessed use of the electronic tools. There was PC and GYN departmental support for the intervention but no prioritization within the health care system to drive new performance measures or incentives for change related to GSM care.

Findings

There were 199 intervention and 208 control clinicians who performed 15,062 well visits for women aged 55 and older. Among intervention clinicians, 107 (53.7%) completed educational training. Intervention clinicians were more likely (p < .001) to use the Smart Set (2.8 vs 0.1 uses) and Smart Text (1.1 vs 0.1 uses). However, the proportion of visits that included a GSM-related diagnosis (9.2% vs 8.2%) or prescription (6.5% vs 5.8%) did not differ between study arms. There was a significant interaction for PC and Gyn, suggesting an increase in GSM diagnosis in Gyn but not PC intervention clinics (OR = 1.57, p = .01).

Implications for D&I Research

EHR tools are more likely to be used if clinician education is included as part of their implementation. However, without simultaneous departmental and organizational initiatives to promote, enforce, or sustain a clinical practice change, the overall efficacy of EHR tools and education alone for improving patient care is limited and competing patient care priorities may take precedence.

Primary Funding Source

Other (please specify below) - North American Menopause Society & Pfizer Independent Grant for Learning & Change #10319

S38 Provider-developed clinical decision support and education to optimize and sustain timely and accurate treatment of hypoxic ischemic encephalopathy in the neonatal intensive care unit

Joan Smith¹, Christopher Smyser², Maggie Wolf¹, Shamik Trivedi³, Brian Hackett⁴, Rakesh Rao⁴, F. Sessions Cole⁴, Rose McGonigle⁵, Ann Donze⁶, Enola Proctor⁷, Amit Mathur⁸

¹Nursing, St. Louis Children's Hospital, St. Louis, MO, 63110, USA; ²Neurology, Washington University School of Medicine, St. Louis, MO, 63110, USA; ³Neonatal-Perinatal Medicine, Washington University School of Medicine, St. Louis, MO, 63110, USA; ⁴Pediatrics and Division of Newborn Medicine, Washington University School of Medicine, St. Louis, MO, 63110, USA; ⁵Newborn Intensive Care Unit, St. Louis Children's Hospital, St. Louis, MO, 63110, USA; ⁶Nursing/Newborn Intensive Care Unit, St. Louis Children's Hospital, St. Louis, MO, 63110, USA; ⁷George Warren Brown School of Social Work, Washington University in St. Louis, St. Louis, MO, 63130, USA; ⁸Pediatrics and Newborn Medicine, Washington University School of Medicine, St. Louis, MO, 63110, USA

Correspondence: Joan Smith

Background

Neonatal brain injury due to hypoxic ischemic encephalopathy (HIE), or birth asphyxia, is the leading cause of all neonatal deaths worldwide. Therapeutic hypothermia (TH) is standard of care for moderate to severe HIE. However, significant practice variation and the potential for inappropriate treatment persist. Timely identification for appropriate treatment is critical to survival, outcome, and cost. Although knowledge on the use of TH is substantial, it is not always aptly applied. Using the Promoting Action on Research Implementation in Health Services (PARiHS) framework and human factors principles, our interdisciplinary team of providers, nurse/physician leaders, informatics analysts/specialists and family partners developed and deployed provider education and an evidence-based electronic clinical decision support (CDS) tool in the electronic health record (EHR) to reduce practice variation and improve teamwork, consistency and appropriateness of treatment.

Methods

A pre-post study design was used and an interdisciplinary team formed to test the CDS integration into provider workflow. Prior to implementation, we conducted usability testing with a standard ‘think aloud’ approach for iterative evaluation and provider education using a 10-item pre-post knowledge test. Following implementation, we measured provider satisfaction with the Computer System Usability Satisfaction Questionnaire (CSUQ; score range on each item: 1 = strongly disagree to 7 = strongly agree) and fidelity with an electronically abstracted adherence score. Appropriateness of treatment was determined by the percentage of infants treated with TH across the mild, moderate and severe encephalopathy categories.

Findings

Usability testing identified and corrected many issues in the functionality of the CDS prior to implementation. Provider knowledge increased from a median total score of 44.44 at baseline to 74.33 at follow-up (p < 0.001). Overall, providers were extremely satisfied with the tool and found it to be helpful (mean item score, 6.4). Fidelity with the CDS was high (>90%) with 98% of the infants being appropriately treated with TH.

Implications for D&I Research

Implementation science informed deployment of a provider-developed CDS tool integrated into clinical workflow and provider education decreased practice variation and improved interdisciplinary teamwork, consistency and appropriateness of treatment for hypoxic-ischemic encephalopathy. These findings have been sustained for 15 months. Further investigation of broader application is warranted.

Global Dissemination & Implementation

S39 Building capacity for implementation science in global health: four years of experience with the University of Washington’s PhD in implementation science

Kenneth Sherr, Emmanuela Gakidou, Stephen Gloyd

Global Health, University of Washington, Seattle, WA, 98105, USA

Correspondence: Kenneth Sherr

Background

Limited capacity for implementation science (IS) in low and middle-income countries (LMICs) contributes to poor coverage of evidence-based intervention. Though a range of IS training options are needed to meet the diverse needs for IS practitioners, there is an acute shortage of doctoral-level trainees with adequate skills to lead IS from in-country universities, Ministries of Health, and partner institutions. To respond to the need for advanced IS skills in LMICs, the University of Washington Department of Global Health developed a doctoral-level training program to prepare future IS leaders, emphasizing students based at leading institutions conducting research, developing policy and strategy, and driving implementation in LMICs.

Methods

Development of the PhD program in Global Health Metrics and Implementation Science began in 2011, enrolling the first cohort in the 2012-2013 academic year. The program has two areas of emphasis – Metrics and Implementation Science. This presentation shares the experience with student recruitment and retention, curriculum and skills-based training, and dissertation research within the Implementation Science area of emphasis.

Findings

The PhD program has continues to demonstrate high demand for IS training at the doctoral level. There are consistently over 60 applicants per year from over 30 countries for the implementation science area of emphasis, with 5-10% of applicants admitted to the program. A total of 13 IS students enrolled in the first four cohorts, 11 (85%) from LMICs (including Kenya ⁽⁴⁾, China ⁽³⁾, Rwanda ⁽¹⁾, Uganda ⁽¹⁾, Sudan ⁽¹⁾, Mozambique ⁽¹⁾). Support for trainees has been primarily from research and teaching assistantships, with dedicated training funds for students from Kenya and China. Students include junior faculty in LMIC universities or research institutions (5;38%), non-governmental organizations (3;23%), the Ministry of Health (1;8%), and a number who were not based at a specific institution (4;31%). To date, one student has graduated. This presentation will also describe the curriculum pathway and dissertation topics for implementation science students.

Implications for D&I Research

Doctoral-level IS training is needed to develop a global workforce with skills to design, evaluate, and disseminate effective strategies to implement and scale-up evidence-based interventions. Our novel PhD program provides a model for others designing novel advanced IS training programs.

S40 Adapting an adherence support workers intervention: engaging traditional healers as adherence partners for persons taking antiretroviral therapy in rural Mozambique

Carolyn Audet¹, Jose Salato², Sten Vermund³, Rivet Amico⁴

¹Health Policy, Vanderbilt University Medical Center, Nashville, TN, 37209, USA; ²Community Health, Friends in Global Health, Quelimane, 00000, Mozambique; ³Department of Pediatrics, Vanderbilt University Medical Center, Nashville, TN, 37203, USA; ⁴Health Behavior Health Education, University of Michigan, Ann Arbor, MI, 48109, USA

Correspondence: Carolyn Audet

Background

Systematic adaptation of evidence-informed interventions to increase retention in care and improve adherence to antiretroviral therapy (ART) will help disseminate innovation in rural sub-Saharan Africa. We selected and adapted an adherence support worker intervention employed in Malawi for use in rural Mozambique. Given the levels of trust and dependence previously expressed by patients for traditional medicine, we adapted the program to engage traditional healers within the allopathic health system.

Methods

Adaption followed a theoretically-driven approach to intervention adaption, the Assessment-Decision-Administration-Production-Topical Experts-Integration-Training-Testing model (ADAPT-ITT). Three rounds of performance-feedback based on theater presentations of the adapted intervention for stakeholders and idea generation were completed with 12 groups to develop the final model from March - July 2016. We offered healer support to 153 newly diagnosed HIV-infected patients.

Findings

Traditional healers, clinicians, and interested community members suggested novel strategies to tailor the Adherence Support Worker intervention, revealing a local culture of HIV-denialism, aversion to the health system and dislike of health care providers, as well as a preference for traditional treatments. Proposed changes to the intervention included modifications to the training language and topics, expanded community-based activities to support acceptability of an HIV diagnosis and to facilitate partner disclosure, and accompaniment to the health facility by healers to encourage delivery of respectful clinical care. Patients, healers, and clinicians deemed the intervention socially acceptable during focus groups. We subsequently recruited 153 newly diagnosed HIV-infected patients into the program; 146 (95%) accepted.

Implications for D&I Research

Systematic translation of interventions, even between regions with similar social and economic environments, is an important first step to successful program implementation. Using theater-based performances to demonstrate delivery of the intervention generated discussion about social norms, community concerns, and the merits of an acceptable strategy to improve retention and adherence to ART.

Primary Funding Source

National Institutes of Health - NIMH K01 award

S41 Evaluating partners in health's mental health integration program in Burera district, Rwanda

Stephanie Smith^1,2, Beatha Nyirandagijimana³, Hildegarde Mukasakindi³, Christian Rusangwa⁴, Molly Franke⁵, Giuseppe Raviola^1,5

¹Mental Health, Partners In Health, Boston, MA, 02199, USA; ²Psychiatry, Brigham and Women's Hospital, Boston, MA, 02115, USA; ³Mental Health, Inshuti Mu Buzima, Butaro, Rwanda; ⁴Health Systems Strengthening, Inshuti Mu Buzima, Butaro, Rwanda; ⁵Department of Global Health and Social Medicine, Harvard Medical School, Boston, MA, 02115, USA

Correspondence: Stephanie Smith

Background

Integrating mental health care into primary care could reduce the high global burden of mental disorders. In 2012, Partners In Health and the Rwandan Ministry of Health adapted an evidence-based program of supported supervision based on task shifting for HIV/AIDS care (Monitoring and Enhanced Supervision at Health Centers, MESH), to include care of severe mental disorders within primary care settings, specifically at rural satellite health centers. The aim of this study was to evaluate implementation processes and outcomes for patients seeking care at health centers participating in the MESH mental health program in one rural district of Rwanda.

Methods

All qualifying adults diagnosed with a neuropsychiatric disorder at four selected health centers were enrolled in the study over a nine-month period (n = 146). Assessments of clinical and functional status were conducted at baseline, two, and six month follow-up, using a single group cohort design. An evaluation of implementation efficacy was performed using service utilization data to assess mental health service uptake at eleven participating district health centers over a six month period, and using MESH mental health supervision checklists to determine whether primary care nurses adequately assessed and treated patients receiving care at select health centers.

Findings

Over six months, a total of 2293 mental health visits occurred at district health centers participating in the MESH MH program. The proportion of initial mental health evaluation questions performed correctly by primary care nurses at select health centers improved from 0.35 to 0.8. There were significant improvements in patient scores on the General Health Questionnaire (25.0 [95% CI: 23.6–26.3] to 11.8 [95% CI: 10.8-12.9], p < 0.0001) and the WHO-DAS Brief (25.5 [95%CI: 23.6-27.3] to 8.0 [95% CI: 6.7-9.4], p < 0.0001).

Implications for D&I Research

In this study, we demonstrate that primary care nurses in resource-limited settings can provide effective mental health care to patients living with severe mental disorders using an evidence based supported supervision program. We also show that effective implementation of the MESH Mental Health program results in significant clinical and functional improvement for patients.

S42 A multi-modal intervention to improve care of the severely ill in western Uganda: a prospective implementation study

Matthew Cummings¹, Elijah Goldberg², Savio Mwaka², Olive Kabajaasi², Adithya Cattamanchi³, Achilles Katamba⁴, Shevin Jacob⁵, Nathan Kenya-Mugisha², J. Lucian Davis⁶

¹Medicine, New York Presbyterian - Columbia University Medical Center, New York, NY, 10032, USA; ²Implementation, Walimu, Kampala, Uganda; ³Medicine, University of California San Francisco, San Francisco, CA, 94110, USA; ⁴Medicine, Makerere College of Health Sciences, Kampala, Uganda; ⁵Medicine, University of Washington, Seattle, WA, 98104, USA; ⁶Epidemiology (Microbial Diseases), Yale School of Public Health, New Haven, CT, 06520-8034, USA

Correspondence: J. Lucian Davis

Background

In low-income countries with a high burden of HIV-infection, the prevalence of severe illnesses such as hemodynamic shock and severe respiratory distress is high. In such settings, optimal management of severe illness remains challenged by infrequent vital sign monitoring and lack of standardized treatment practices. We sought to determine the impact of adding a structured implementation-support intervention to a standardized clinical training program on the identification and management of severely ill patients in western Uganda.

Methods

We conducted a prospective cohort study of clinician practices before and after introduction of a complex, multi-modal education and training intervention using a quasi-randomized, stepped-wedge, implementation design at 4 inpatient health facilities in Kabarole District, western Uganda. Training interventions were based on World Health Organization guidelines for management of severe illness in resource-limited settings.

Findings

From August 2014-May 2015, 6,028 patients were enrolled; 1,698 in the pre-intervention cohort and 4,330 in the intervention cohort. In-hospital mortality was significantly higher among HIV-infected vs. HIV-negative patients (14.6% vs. 3.1%; p < 0.001). Among all patients, rates of shock and severe respiratory distress were high, identified on 15.7% and 4.7% of all patient-days, respectively. Compared to the pre-intervention cohort, patients in the intervention cohort were more likely to have ≥3 vital signs collected (42.3% vs.4.3%; p < 0.001). For patients diagnosed with shock and severe respiratory distress, those in the intervention cohort were more likely to receive intravenous fluid resuscitation (53.6% vs. 36.6%; p < 0.001) and supplemental oxygen (61.1% vs. 16.4%; p < 0.001) within 24 hours, respectively.

Implications for D&I Research

Standardized clinical training programs plus post-training implementatuin support can improve identification and management of severe illness in resource-limited settings. Further prospective assessment of such interventions using validated implementation frameworks is needed.

Primary Funding Source

Other (please specify below) - An anonymous private foundation based in the Netherlands supporting work to improve care of patients with HIV/AIDS

S43 Context matters: adapting the Model for Understanding Success in Quality Improvement (MUSIQ) for low and middle income countries

Julie Reed¹, Rohit Ramaswamy², Gareth Parry³, Sylvia Sax⁴, Heather Kaplan⁵

¹Public Health and Primary Care, NIHR CLAHRC NWL, London, SW10 9NH, UK; ²Gillings School of Public Health, University of North Carolina, Chapel Hill, NC, 27599, USA; ³Institute for Healthcare Improvement, Boston, MA, 02138, USA; ⁴Independent Global Consultant, Heidelberg, 69118, Germany; ⁵Divisions of Neonatology, Cincinnati Children's Hospital Medical Center, Cincinnati, OH, 45229, USA

Correspondence: Rohit Ramaswamy

Background

The importance of incorporating local context in designing quality improvement (QI) initiatives has received increasing attention in implementation and improvement fields. The environment where change occurs has been highlighted as an explanatory factor for why some interventions work in some settings, but not in others. This observation led to the development and refinement of the Model for Understanding Success in Quality (MUSIQv2.0) to define what context is and how it interacts with implementation and improvement efforts. However, MUSIQ was developed and refined in high resource settings and the applicability of the results to low and middle income countries (LMICs) is unknown.

Methods

To explore the application of MUSIQ in LMICs, a dialogue collaborative process was utilised to draw on the authors’ individual experiences and understanding of context and to generate, through iterative development, a group understanding of context applied to improvement initiatives in LMICs. Authors of MUSIQv2.0, JR and HK have experience of studying context in high income countries and RR, SS and GP have experience of improvement in both settings. The process consisted of iterative discussions and refinement on how the MUSIQ findings applied to low resource settings.

Findings

Initial findings demonstrate that the three types of context, identified through MUSIQv2.0 for high income settings, were found to be relevant to LMIC: the care delivery where a QI intervention is introduced; the QI team conducting a QI project; and the wider context supporting general QI. The collaborative dialog revealed that while these categories are applicable for LMICs, their manifestation differed. For example, the QI team, often consisting of 6-8 personnel in HICs may have no more than one or two people in LMICs. This dialog resulted in recommendations for a simplified version of MUSIQ 2.0 applicable to LMICs which will be completed by November 2016.

Implications for D&I Research

This work articulates how interpretation of contextual factors affecting implementation can vary by setting, especially between HICs and LMICs. Insights from such comparisons can support design and modification of improvement initiatives and their evaluation as they are spread across a variety of countries.

S44 Integrating evidence-based pediatric behavioral health services into primary and community settings: pragmatic strategies and lessons learned from literature review and global implementation projects

Keng-yen Huang¹, Sabrina Cheng¹, Susan Yee¹, Kimberly Hoagwood², Mary McKay^3,4, Donna Shelley¹, Gbenga Ogedegbe⁵, Laurie Miller Brotman¹

¹Populaiton Health, Population Health at NYU Langone Medical Center, New York, NY, 10016, USA; ²Child & Adolescent Psychiatry, The Child Study Center at NYU Langone Medical Center, New York, NY, 10016, USA; ³Social Work, McSilver Institute for Poverty Policy and Research, New York, NY, 10013, USA ; ⁴Social Work, New York University, New York, NY, 10013, USA; ⁵Population Health and Medicine and Center for Healthful Behavior Change, New York University, New York, NY, 10016, USA

Correspondence: Keng-yen Huang

Background

Promoting pediatric health in low-resource and low-and-middle-income-country (LMIC) settings face numerous challenges in global health research. Children growing up in these settings tend to live in environments characterized by extreme poverty and violence. Toxic stressors combine to yield suboptimal child behavioral health. Although numerous evidence-based interventions (EBIs) are relevant for efforts to promote child behavioral health in low-resource settings, research on disseminating EBIs to these settings is extremely limited. Major challenges include: 1) lack of systematic review on pediatric behavioral health services from dissemination and implementation (D&I) perspectives; and 2) lack of general guidance for pediatric researchers on strategies to initiate D&I research. This paper addresses these gaps by focusing on two implementation objectives: 1) synthesizing pediatric D&I and behavioral health literature by identifying gaps and solutions (in multilevel contexts) that have been studied in primary and community service settings; and 2) generating a list of pragmatic guidance for pediatric researchers based on lessons learned from our global implementation projects (e.g., US, Uganda, Nepal).

Methods

The review of the study is guided by the Consolidated Framework for Implementation Research. A systematic review of US and global health pediatric health literature over the past 10 years has been initiated. Pragmatic strategies and research lessons have been generated based on co-authors' D&I project experiences in diverse community settings.

Findings

Preliminary results from the ongoing literature review identified several common barriers at the consumer, service provider, implementer, and system/policy levels (e.g., low health literacy, poor provider competency). Also, several D&I strategies have been found to be effective in addressing these implementation gaps (e.g., patient education models, policy stakeholder capacity building approaches). Guidance on conducting D&I research has been generated from lessons learned in multiple projects, such as applying effectiveness-implementation hybrid design in intervention study and characterizing inner and outer pediatric service contexts.

Implications for D&I Research

This study addresses important pediatric D&I research gaps by synthesizing literature. Considering the complexity of practice health research, this study also generates useful D&I guidance from multiple implementation projects to inform future pediatric behavioral health research in LMIC and US settings.

Primary Funding Source

National Institutes of Health - NIMH U19 MH110001-01; UBS Optimus Foundation Geneva

Health Policy Dissemination and Implementation

S45 Distortion of implementation techniques in health care: the case of "facilitation"

Roman Kislov¹, John Humphreys², Gill Harvey³, Paul Wilson¹

¹Alliance Manchester Business School, University of Manchester, Manchester, M15 6 PB, UK; ²NIHR CLAHRC Greater Manchester, Salford Royal NHS Foundation Trust, Salford, M6 8HD, UK; ³School of Nursing, The University of Adelaide, Adelaide, SA 5005, Australia

Correspondence: Paul Wilson

Background

When applied to solving real-world problems of health care, service improvement approaches are likely to evolve over time in response to the context of their implementation. The temporal dynamics of this evolution and its underlying processes, however, remain under-researched. To address this gap, we explore the evolution of facilitation, an implementation approach that can be broadly defined as enabling the processes of learning in group contexts and is often deployed to mobilize research knowledge into clinical practice.

Methods

The prospective longitudinal case study was conducted in a five-year UK-based collaborative research programme involving a university, a tertiary-care hospital and a number of local general practices. The programme aimed to increase the identification of chronic kidney disease (CKD) and improve the management of blood pressure in CKD patients by facilitating the mobilization of existing health research in day-to-day clinical practice. A purposive sampling strategy was used, with 40 research participants drawn both from the programme 'core' team and participating general practices. 45 semi-structured interviews (30-95 minutes in duration) served as the main method of data collection and were conducted (face-to-face or by phone) in three rounds (2010-2011, 2012-2013 and 2013-2014) to enable longitudinal analysis.

Findings

We argue that an uncritical and uncontrolled adaptation of implementation techniques may lead to their gradual distortion, undermining their promise to positively affect organizational learning processes and masking the unsustainable nature of the resulting improvement outcomes captured by conventional performance measurement. We describe the following three parallel and overlapping micro-processes underpinning the gradual distortion of facilitation over time: ⁽¹⁾ prioritization of (measurable) outcomes over the (interactive) process; ⁽²⁾ reduction of (multiprofessional) team engagement and ⁽³⁾ erosion of the facilitator role.

Implications for D&I Research

Our findings emphasize the lack of attention to the sustainability of change once the short-term outcomes of facilitated implementation projects have been attained, measured and reported. An exploration of new ways of maintaining context-sensitive adaptation of implementation techniques without losing their core elements could provide a useful direction for future empirical inquiry.

S46 Measuring the cost of patient-centered medical home implementation

Robert Lieberthal^1,2,3, Colleen Payton⁴, Mona Sarfaty⁴, George Valko⁴

¹Jefferson School of Population Health, Thomas Jefferson University, Philadelphia, PA, 19107, USA; ²Sidney Kimmel Cancer Center, Thomas Jefferson University, Philadelphia, PA, 19107, USA ³Department of Public Health, University of Tennessee, Knoxville, Knoxville, TN, 37996, USA; ⁴Department of Family and Community Medicine, Thomas Jefferson University, Philadelphia, PA, 19107, USA

Correspondence: Robert Lieberthal

Background

The 2010 Patient Protection and Affordable Care Act (ACA) and a number of state-based initiatives have significantly increased the incentives for primary care practices to become patient-centered medical homes (PCMHs). Currently, the cost for individual practices to become more patient-centered is not well recognized.

Methods

The objectives of this study are to develop a methodology to inventory the costly activities involved in implementing the PCMH model and apply that methodology to practices that had transformed to PCMHs. A convenience sample of eleven small and medium sized primary care practices in southeastern Pennsylvania that had previously attained NCQA PCMH recognition enrolled in our study. We assess the cost of PCMH transformation by categorizing the clinical activities required to attain PCMH recognition. We apply the economic principles of production to the primary care practice in order to develop an economic taxonomy for the costs of clinical activities.

Findings

In our taxonomy, the cost of each PCMH-related clinical activity can be classified as either 1) an NCQA activity category, 2) recognition as a PCMH, or 3) a change in practice culture. Cost offsets including financial incentives, pay for performance bonuses, and productivity improvements can be used to offset the cost of the PCMH model. We demonstrate the applicability of this method by applying it to medical and and nurse-led PCMH practices.

Implications for D&I Research

Small and medium sized practices may experience particular difficulty in separating PCMH and non-PCMH activities. Tools for planning the transition to PCMH could facilitate adoption of the model in a cost-effective manner.

Primary Funding Source

Agency for Healthcare Research and Quality - AHRQ grant R03-HS22630

S47 Successful strategies for implementing patient centered care in VA medical centers

Rendelle Bolton¹, Carol VanDeusen Lukas², Christine Hartmann^3,4, Nora Mueller⁵, Sally K Holmes^4,6, Barbara Bokhour^1,4

¹ENRM Veterans Hospital, VA HSR&D Center for Healthcare Organization and Implementation Research (CHOIR), Bedford, MA, 01730, USA; ²Center for Organization, Leadership & Mgmt. Rsch., Department of Veterans Affairs/Boston University-School of Public Health, Boston, MA, 02130, USA; ³Department of Veterans Affairs, Center for Healthcare Organization and Implementation Research (CHOIR), Bedford, MA, 01730, USA; ⁴Health Law, Policy and Management, Boston University School of Public Health, Boston, MA, 02118, USA; ⁵Center for Healthcare Organization and Implementation Research, VA HSR&D Center for Healthcare Organization and Implementation Research (CHOIR), Bedford, MA, 01730, USA; ⁶Research, VA HSR&D Center for Healthcare Organization and Implementation Research, Bedford/Boston, MA, 02130, USA

Correspondence: Rendelle Bolton

Background

The Veterans Health Administration (VHA) Office of Patient-Centered Care and Cultural Transformation (OPCC&CT) spearheads VHA’s effort to transform facility culture to embody patient-centered care (PCC). As part of this effort, OPCC&CT sought to identify optimal strategies used by facilities to successfully implement PCC. Drawing on prior work, this study examined the impact of key areas on PCC implementation at VA medical centers including the role of leadership, Veteran engagement, staff enculturation efforts, strategies to foster PCC innovation; staff roles and priorities; and organizational structures.

Methods

Semi-structured interviews were conducted with individuals leading PCC transformation at 31 geographically diverse VA medical centers, representing 18 of VA’s 21 official regions. These facilities were identified by regional PCC leaders, and were categorized by the regional leader as high, medium, or low performing with respect to perceived success with PCC implementation. Participants were asked about barriers, facilitators, and contextual factors that influenced implementation of PCC initiatives. Using a content analysis, transcripts were analyzed iteratively with both a priori and emergent codes grounded in interview data. We drew on the previously identified key areas critical for patient-centered transformation as sensitizing concepts for our analysis, and compared data across high, medium, and low performing facilities.

Findings

Key differences between high, medium, and low facilities were identified in the areas of leadership, Veteran engagement, staff enculturation, strategies to foster innovation, and organizational structures to support PCC. High performing facilities described receiving concrete leadership support, formally engaged Veterans in program development and implementation, and developed PCC programing from ideas generated by multiple stakeholder groups. They also developed infrastructure to support PCC programming and sustained staff enculturation efforts. Low performing facilities trained less staff in PCC principles, and had limited staff resources to champion PCC. Staff buy-in and competing priorities were challenges for all facilities.

Implications for D&I Research

As healthcare systems implement policies to promote patient-centered care in medical centers, attention should be given to actively engage leadership, patients, and staff at all stages of PCC implementation. Creating organizational infrastructure to support PCC is critical to the success of cultural transformation efforts.

Primary Funding Source

Department of Veterans Affairs - US Department of Veterans Affairs Office of Patient-Centered Care and Cultural Transformation, and QUERI program

S48 Mapping a method for rapid dissemination and implementation: the primary care extension program

Sarah Ono^1,2, Benjamin Crabtree³, Leah Gordon¹, William Miller⁴, Bijal Balasubramanian⁵, Leif Solberg⁶, Deborah Cohen¹

¹Department of Family Medicine, Oregon Health & Science University, Portland, OR, 97239, USA; ²VA Portland Health Care System, Department of Veterans Affairs, Portland, OR, 97239, USA; ³Department of Family Medicine and Community Health, Rutgers Robert Wood Johnson Medical School, New Brunswick, NJ, 08901, USA; ⁴Family Medicine, Lehigh Valley Hospital, Allentown, PA, 18105-7017, USA; ⁵Epidemiology, University of Texas Health Science Center at Houston, Houston, TX, 77030, USA; ⁶Institute for Education and Research, HealthPartners, Bloomington, MN, 55425, USA

Correspondence: Sarah Ono

Background

Primary Care Extension Programs (PCEP) were authorized, but not funded, through the Affordable Care Act in 2010. PCEPs are envisioned as a network of primary care practices, academic institutions, and state, federal, and community-based health extension agents that would institutionalize a resource, akin to the US Agricultural Cooperative Extension System, that would facilitate quality improvement of primary health care. In 2015, AHRQ funded EvidenceNOW so that seven regional Cooperatives would help move this extension-development forward. These Cooperatives were each tasked with covering a single state or multi-state contiguous region and each engaging 250 small primary care practices to rapidly disseminate and implement evidence related to cardiovascular disease prevention.

Methods

An EvidenceNOW National Evaluation was also funded by AHRQ to conduct a mixed methods evaluation of these interventions, using a range of tested and novel data collection approaches. Cross-case analyses of the infrastructure and resources across the diverse contexts of the seven Cooperatives were conducted to identify existing extension elements and variation in their design and function for regions at various stages of network development.

Findings

EvidenceNOW Cooperatives are using a combination of practice facilitation, expert consultation, HIT support, peer learning, and data feedback with benchmarking to support widespread dissemination and implementation of cardiovascular prevention services to practices in their regions. Cooperatives mobilized existing infrastructure (e.g., trained workforces, centralized data extraction capacity, and connections to local programs), capitalizing on partnerships to advance and explore PCEP potential. We identified variation in the PCEP models and show how these are influenced by regional resources, local politics, and unique visions for how to optimize and build networks. Our findings show that states considering developing a PCEP need a finite set of elements to support rapid dissemination and implementation.

Implications for D&I Research

Regional leaders who wish to develop a PCEP need to identify and engage the resources in their region in order to develop key partnerships and invest in building critical infrastructure. The goal of building PCEPs is to optimize health and health care by facilitating dissemination and implementation of evidence-based practices into primary care throughout the country, and support ongoing quality improvement on a national scale.

Primary Funding Source

Agency for Healthcare Research and Quality - R01: National Evaluation of EvidenceNOW Initiative

S49 Department of Defense (DoD) practice based implementation network: an optimized framework for provision of behavioral health interventions to the Military Health System (MHS)

Kate McGraw¹, Andrew Blatt², Demietrice Pittman³

¹DHCC, Defense Centers of Excellence for Psychological Health and Traumatic Brain Injury, Silver Spring, MD, 20910, USA; ²DHCC/PHCC, Defense Centers of Excellence for Psychological Health & TBI, Silver Spring, MD, 20901, USA; ³Implementation Science Team, Defense Centers of Excellence for Psychological Health and Traumatic Brain Injury (DCoE), Silver Spring, MD, 20910, USA

Correspondence: Kate McGraw

Background

In 2014, the Institute of Medicine reported that two decades may pass before psychological health research findings become part of routine clinical practice. The DoD in conjunction with the Department of Veterans Affairs (VA) developed a Practice Based Implementation (PBI) Network to rapidly translate psychological health research findings into clinical practice by facilitating practice change. In 2015, 14 military treatment facility sites were engaged to pilot the implementation of outcomes monitoring of PTSD treatment, and alcohol misuse screening and by 2016 the Defense Health Agency sustained the PBI Network.

Methods

Based upon the Promoting Action on Research Implementations in Health Services and the VA Quality Enhancement Research Initiative frameworks, the initial pilot effort presented Evidence Based Practices (EBPs) to clinicians through trainings that respect clinic culture and context while providing continuous support and facilitation to pilot sites to try to increase provider knowledge and accountability, promote coordination and information sharing, and potentially reduces cost by testing implementation initiatives prior to broader dissemination throughout the enterprise. Simultaneously a website was developed to serve as repository of shared resources and lessons learned.

Findings

Strong leadership engagement was associated with increased participation and an engaged clinic staff was associated with increased site commitment to develop and implement solutions. Feedback indicated increases in usage of outcome measures and positive PTSD Check List use to monitor treatment progress as well as consistent method to track care planning and delivery. Finally, increased leadership prioritization of outcome measure use as necessary for practice changes but access and ability to provide care tended to be ongoing barriers to comprehensive utilization of EBPs.

Implications for D&I Research

The PBI Network sustainment project is poised to provide ongoing training and clinician informed survey research feedback from an active Implementation Science informed knowledge dissemination network within one of the largest U.S. healthcare systems. This presentation will relate the processes and science that have served as foundation for more rapid adoption of behavioral health practice changes within the U.S. military.

Primary Funding Source

Joint Incentive Fund (JIF)

S50 Testing un-learning and substitution strategies to de-implement antipsychotics in nursing homes

Megan McCullough^1,2, Christine Hartmann^1,2, Helen Kales^3,4, Dan Berlowitz^1,2, Teresa Hudson^5,6, Chris Gillespie¹, Christian Helfrich^7,8

¹Center for Healthcare Organization and Implementation Research (CHOIR), VA Health Services Research & Development, Bedford, MA, 01730, USA; ²Department of Health Law, Policy and Management, Boston University School of Public Health, Boston, MA, 02118, USA; ³Deparment of Veterans Affairs, Center for Practice Management Outcomes Research, Ann Arbor, MI, 48105, USA; ⁴Department of Psychiatry, University of Michigan, Ann Arbor, MI, 48105, USA; ⁵Psychiatric Research Institute, University of Arkansas for Medical Sciences, Little Rock, AR, 72205, USA; ⁶Department of Veterans Affairs, VA Central Arkansas Veterans Healthcare System, North Little Rock, AR, 72114, USA; ⁷Health Services Research and Development, VA Puget Sound Health Care System, Seattle, WA, 98101, USA; ⁸Department of Health Services, University of Washington, School of Public Health, Seattle, WA, 98195, USA

Correspondence: Megan McCullough

Background

Medical overuse (i.e., treatment that provides no benefit and/or harm) represents 10%-46% of care depending on setting and practice. Use of antipsychotic medications to manage behavioral and psychological symptoms of dementia (BPSD) in nursing homes is an example of overuse. Antipsychotics in this setting are frequently used to manage dementia patients’ BPSD. Despite limited evidence of efficacy and significant evidence of risks including mortality, 1 in 4 dementia patients in the Veterans Health Administration (VHA) Community Living Centers (CLCs—i.e., nursing homes) is prescribed antipsychotics. We developed a planned action model founded on the utility of two distinct, synergistic processes: 1) unlearning; and 2) substitution. Building on prior work, our objective is to operationalize and test unlearning and substitution strategies in the de-implementation of antipsychotic use in 8 VHA CLCs.

Methods

This project tests unlearning and substitution via a stepped-wedge design. Academic detailing (i.e., unlearning) promotes change in prescribing habits through awareness of the limited effectiveness and significant adverse effects of antipsychotics. The WeCareAdvisor™ is an on-line tool for use by frontline CLC staff that contains the DICE (Describe, Investigate, Create, Evaluate) Approach for assessment and management of BPSD via an ecobiopsychosocial model (i.e., substitution). The tool guides them through assessing CLC residents’ symptoms and context and prompts them with individualized ways to address BPSD and underlying causes through behavioral and environmental interventions. This project includes a mixed-methods evaluation, including an interrupted time seriesanalysis of changes in prescribing and a quantitative and qualitative evaluation of the process of testing these strategies.

Findings

We will provide concrete guidance on operationalizing and measuring unlearning and substitution strategies in nursing home setting. We hope to help other researchers identify the necessary conditions to determine which strategy is more or less effective, or more or less easily deployed. We will also catalogue unintended consequences, particularly related to clinician morale, and outline potential mitigators.

Implications for D&I Research

We propose concrete ideas on operationalizing and testing unlearning and substitution strategies. Lessons about the unintended consequences of implementing these de-implementation strategies will also add to the practical and conceptual knowledge about de-implementation

Primary Funding Source

Department of Veterans Affairs

S51 A scoping review to evaluate the impact of prescription drug monitoring program implementation

Erin Finley^1,2, Ashley Garcia³, Kristen Rosen³, Claudina Tami³, Don McGeary³, Mary Jo Pugh², Jennifer Sharpe Potter³

¹School of Medicine, University of Texas Health Science Center at San Antonio, San Antonio, TX, 78229, USA; ²Veterans Evidence-based Research Dissemination and Implementation Center, South Texas Veterans Health Care System, San Antonio, TX, 78229, USA; ³Psychiatry, University of Texas Health Science Center at San Antonio, San Antonio, TX, 78229, USA

Correspondence: Erin Finley

Background

Prescription drug monitoring programs (PDMPs) have been implemented in 49 out of 50 states in an effort to mitigate opioid-related misuse, abuse, and mortality, yet the literature evaluating the impact of PDMP policy implementation remains limited. We conducted a scoping review to: ⁽¹⁾ describe available evidence regarding impact of PDMPs in the United States; and ⁽²⁾ propose a conceptual model to inform future PDMP implementation and evaluation efforts.

Methods

Scoping review following Arksey and O’Malley’s (2005)’s methodology. Of the 121 articles identified from the initial PubMed database search of English-language studies published between 1/1/2000-5/31/16, eleven articles were identified as relevant based on the inclusion criteria defined a priori, specifically: peer-reviewed; presents original research; provides direct assessment of outcomes related to impact or effectiveness of PDMP implementation. We extracted data from each article following a structured template, then conducted thematic analysis to synthesize results.

Findings

Thematic analysis revealed studies of opioid-related outcomes associated with PDMPs typically point to a shared logic for how PDMPs are expected to function: i.e., implementation of PDMPs will increase reporting and monitoring of controlled prescriptions, leading to reduced opioid prescribing, opportunities for opioid diversion and misuse, and opioid abuse and mortality. However, extant evidence for the impact of PDMPs as an opioid risk mitigation tool remains mixed, with studies reporting evidence that both supports and contradicts their efficacy. We identified four domains of opioid-related outcomes frequently examined in original studies evaluating PDMP implementation: ⁽¹⁾ opioid prescribing; ⁽²⁾ opioid diversion and supply; ⁽³⁾ opioid misuse; and ⁽⁴⁾ opioid-related morbidity and mortality. These domains inform a proposed evaluation framework that highlights significant gaps in empirical research across each of these domains.

Implications for D&I Research

There is currently no standard of best practices or guidelines regarding implementation or use of PDMPs for their intended purpose, and evidence for their impact remains mixed. We propose a conceptual model for evaluating the complexities of PDMP implementation with the goals of clarifying PDMP mechanisms of impact, identifying characteristics of PDMP implementation associated with best outcomes, and maximizing the utility of PDMP policy to reduce opioid-related public health burden.

Primary Funding Source

Air Force Research Laboratory FA8650-15-C-6588

S52 Baseline clinician interviews of two de-implementation projects: four factors that might inform de-implementation strategies

Christian Helfrich^1,2, Krysttel Stryczek³, David Au³, Steven Zeliadt^1,2,3, George Sayre^2,3, Chris Gillespie⁴

¹Health Services Research and Development, VA Puget Sound Health Care System, Seattle, WA, 98101, USA; ²Department of Health Services, University of Washington, School of Public Health, Seattle, WA, 98195, USA; ³Center of Innovation for Veteran-Centered and Value-Driven Care, Department of Veterans Affairs, Seattle, WA, 98108, USA; ⁴Center for Healthcare Organization and Implementation Research (CHOIR), Department of Veterans Affairs, Bedford, MA, 01730, USA

Correspondence: Christian Helfrich

Background

Medical overuse, or the provision of care that provides no benefit or where harms outweigh benefits, accounts for 10% to 46% of care, depending on the clinical practice and setting. In 2015, the VA funded a multisite program to develop evidence-based strategies to de-implement ineffective or harmful clinical practices. The goal of this program is to systematically study de-implementation in multiple clinical settings and for different clinical practices. As part of this effort, we assessed provider perceptions of two practices targeted for de-implementation: inappropriate follow-up of incidental lung nodules on computed tomography (CT) scans of the chest, and use of inhaled corticosteroids (ICS) for treating exacerbations among patients with mild-to-moderate chronic obstructive pulmonary disease (COPD).

Methods

We have conducted 26 baseline interviews (PCPs, Pulmonologists & Radiologists) at three VA Healthcare Systems using a semi-structured interviewer guide across two on-going quality improvement projects. Broad themes were identified based on representative interview responses and grouped under higher order headings to describe distinct aspects of participants’ experiences using content analysis.

Findings

We found four emergent themes across the two clinical practices. Perceived patient resistance. Even when providers say they explain risks to patients, e.g., from further CT scans, they report some patients resist doing less. Concern over patient resistance may be a significant factor in de-implementing some practices whether or not patients actually express resistance or it is merely assumed by providers. Limited primary care provider (PCP) capacity for staying abreast of current research. Given the scope of primary care practice there may be variation and clear gaps in PCPs awareness of evidence and current recommendations. Substitutions. Some providers had identified substitutes, e.g., putting a patient with COPD on tiotropium as a substitute that facilitated discontinuing ICS. Shared provider responsibility. Participants perceived multiple providers, e.g., PCPs, pulmonologists, and radiologists, to be responsible for the decision for further follow-up of incidental lung nodules. This kind diffusion of responsibility may be challenging for de-implementation of some practices.

Implications for D&I Research

Across two very different clinical practices and settings, we identified 4 themes that could be taken into account in designing de-implementation strategies.

Primary Funding Source

Department of Veterans Affairs - QUERI

S53 How do implementation strategies impact efforts to integrate evidence-based interventions into tobacco retail policy?

Jennifer Leeman¹, Allison Myers², Jennifer Grant², Mary Wangen³, Tara Queen⁴

¹School of Nursing, University of North Carolina at Chapel Hill, Chapel Hill, NC, 27599, USA; ²Counter Tools, Carrboro, NC, 27510, USA; ³Health Behavior, Gillings School of Global Public Health, Chapel Hill, NC, 27599, USA; ⁴Lineberger Comprehensive Cancer Center, University of North Carolina at Chapel Hill, Chapel Hill, NC, 27599, USA

Correspondence: Jennifer Leeman

Background

The US tobacco industry spends $8.7 billion annually on marketing at the point of sale (POS), a practice that promotes tobacco use. Evidence-based interventions (EBIs) are available to counter POS tobacco marketing (e.g., limiting retailer density). Integrating EBIs into policy is challenging, however, and involves leveraging and accommodating local resources, existing policy, and stakeholders (allies and adversaries) throughout an uncertain and often long policy change process. Little is known about how best to support POS EBI policy integration. The present study builds on a theory-based evaluation model. Kingdon’s multiple streams theory of policy change is applied to specify five core components of the EBI policy integration process: ⁽¹⁾ document local problems, ⁽²⁾ formulate evidence-informed solutions, ⁽³⁾ engage strategic partners, ⁽⁴⁾ raise public awareness, and ⁽⁵⁾ persuade decision makers to enact new policy. Building on Leeman’s theory of capacity-building, the model further posits that implementation strategies affect EBI policy integration through their effects on two intermediate outcomes: team-leader self-efficacy to facilitate and team performance of the five core components of the EBI policy integration process.

Methods

Implementation strategies (training, tools, and technical assistance) were delivered for one year to 30 community teams in one state. Surveys were conducted at baseline and 12 months to assess impact on team leaders’ self-efficacy and teams’ progress toward enacting new policies. In-depth interviews were conducted at 6 and 12 months to assess teams’ performance and barriers encountered across the five core components of EBI policy integration. Analysis included descriptive and bivariate statistics and content analysis.

Findings

Following one-year’s exposure to implementation strategies, team leaders’ self-efficacy increased significantly, and a greater number of teams were making progress towards enacting policies in four of six domains. Teams’ performance of 16 activities varied, with the greatest number of activities performed within the core component “document local problem.”

Implications for D&I Research

Additional research is need to assess implementation strategies’ impact beyond one year. Findings can inform delivery of implementation strategies and tests of their effectiveness at promoting EBI policy integration. The study’s conceptual model and measures may also contribute to research testing implementation strategies for other EBIs that require changes to local policy.

Primary Funding Source

Centers for Disease Control and Prevention - Centers for Disease Control and Prevention (CDC) and the National Cancer Institute/NIH, through Cooperative Agreement Number U48 DP005017-SIP to the Center for Health Promotion and Disease Prevention at the University of North Carolina at Chapel Hill.

S54 Framing and disseminating research information to legislators and advocates involved in cancer control policy change

Alexandra Morshed^1,2, Elizabeth Dodson^2,3, Rachel Tabak², Ross C Brownson^1,2

¹Brown School, Washington University in St. Louis, Saint Louis, MO, 63130, USA; ²Prevention Research Center, Washington University in St. Louis, St. Louis, MO, 63130, USA; ³Institute for Public Health, Washington University, St. Louis, MO, 63112, USA

Correspondence: Alexandra Morshed

Background

Evidence-based policy plays an important role in prevention of cancer and other chronic diseases. Dissemination strategies should be informed by the needs of actors involved in policy decision-making. This study examines the differences between state legislators and advocates in how they seek and use information and what their preferences are for how research information is framed and used.

Methods

We carried out a cross-sectional comparison of US advocates (n = 77) and state legislators (n = 265) working on issues related to cancer control.

Findings

The advocates differed significantly from the legislators on all demographic characteristics. Advocates reported seeking and using information overall more frequently than legislators, though legislators used legislative research bureaus more often (0.45 point difference, p = .004). Both legislators and advocates prioritized the presentation and timeliness of research information similarly, but reported different preferences for source of research information. Legislators emphasized information delivered to them by someone they trust (0.33 point difference, p = .004), while advocates rated objectivity (-0.26 point difference, p = .03) and relevance to constituents (-0.28 point difference, p = .004) more highly. Both groups put a similar priority on research information that supports the position they hold.

Implications for D&I Research

Our study provides leads for development of dissemination strategies to enhance evidence-based policymaking for cancer control that are tailored to state-level legislators and advocates. For example, as legislators prioritize having information delivered to them by someone they trust, dissemination strategies should include partnering with individuals and groups that have existing personal relationships with legislators and their staff. Also, as advocates put a higher priority on information that is unbiased and relevant, ensuring that the research evidence is generalizable to their constituencies and including local data and success stories in dissemination materials targeted at advocates are important. These and other strategies should be tested for effectiveness in future research. In addition, though a growing knowledge base exists on how to disseminate research to policymakers, few studies identify strategies for dissemination to advocates or examine the process of building stronger partnerships between research and advocacy groups. Future research efforts should examine how these key actors in the policymaking process can be more effectively engaged to promote evidence-based policymaking.

Primary Funding Source

National Institutes of Health - This research was funded in part by the National Cancer Institute, the National Institute of Diabetes and Digestive and Kidney Diseases, and Washington University Institute of Clinical and Translational Sciences.

S55 Using decision analysis to understand policy-makers’ use of research and local evidence for evidence-informed policies

R. Chris Sheldrick¹, Thomas Mackie², Justeen Hyde³, Laurel Leslie⁴

¹Department of Pediatrics, Tufts Medical Center, Boston, MA, 02111, USA; ²Institute for Health, Health Care Policy, and Aging Research, Rutgers University, New Brunswick, NJ, 08901, USA; ³Institute for Community Health, Malden, MA, 02148, USA; Center for Healthcare Organization and Implementation Research (CHOIR), US Department of Veterans Affairs, Bedford, MA, 01730, USA; ⁴Institute for Clinical Research and Health Policy Studies, Tufts Medical Center, Boston, MA, 02111, USA

Correspondence: R. Chris Sheldrick

Background

Initiatives to promote evidence use in policymaking increasingly force state-level policy-makers to make decisions about whether and how to prioritize, implement, sustain, and/or discontinue evidence-informed policies. To provide the best possible services, policymakers must consider not only the research evidence, but also the socioeconomic and politic context, absorptive capacity of relevant entities, and stakeholder-identified needs.

Methods

In medicine, decision analysis offers a range of methods to support rational decision making, informing everything from cost-effectiveness analyses to shared decision making with individual patients. Using a decision analysis model of physicians’ use of diagnostic tests as a guide, we constructed a decision-tree to depict policymakers’ use of research and local evidence to inform decisions regarding the implementation of EBPs. To illustrate this approach, we draw on a case example that investigates how state-level policymakers make decisions about psychotropic medication oversight policies for children in foster care. We conducted semi-structured qualitative interviews with 72 state child welfare policymakers, with at least on respondent from each of the 50 states and DC. For analysis, we employed the five systematic and visible steps of framework analysis.

Findings

Qualitative results suggest policymakers often rely on both research and local evidence to inform decisions about policy options. Serving as an interpretative frame, the decision-tree model highlights several aspects of such decisions that motivate local evidence use, including uncertainty regarding the effectiveness of specific policies to local population, uncertainty regarding policy costs, and expected comparative effectiveness with respect to existing policies. Our model also highlights how the collection and interpretation of local evidence—especially early in the decision making process when relevant data are often scarce—may bias decisions about potential policy options.

Implications for D&I Research

Decision analysis reveals how the influence of an array of factors makes even rational decision-making dynamic, complex, and fraught with uncertainty. To understand the application of research evidence across diverse contexts, implementation science should address the actual decisions policymakers face in negotiating the complex fit between available evidence and the variety of community settings. Our model implies that not just whether but how decision-makers employ local evidence may be critical to evidence-informed decisions and deserves closer examination.

S56 A longitudinal investigation of knowledge brokering as a mechanism for integrating research evidence into health policymaking

Itzhak Yanovitzky, Matthew Weber, Nicole Gesualdo, Teis Kristensen

Communication, Rutgers University, New Brunswick, NJ, 08901, USA

Correspondence: Itzhak Yanovitzky

Background

Evidence-based decision making is critical to the formulation of effective health policy and practice, but use of research evidence in the policymaking process continues to be infrequent, inconsistent, and often misinformed despite efforts to increase the availability and accessibility of research to policymakers. This study examined the potential utility of knowledge brokering as a mechanism for increasing policymakers’ use of research evidence in the context of U.S. federal policies to curb childhood obesity over the past 15 years.

Methods

A comprehensive set of Congressional and U.S. Government documents (transcripts of Congressional bills and hearings, floor debates, and Congressional reports) concerning federal policies to decrease childhood obesity from 2000-2014 (N = 1,041 documents) were retrieved and coded manually by the research team for variables measuring the scope, type, context, and timing of research evidence use. All excerpts containing explicit reference to research evidence were analyzed quantitatively and qualitatively using Dedoose. A two-mode network of suppliers and users of research evidence was created based on the actors (legislators, government officials, witnesses, etc.) referenced in each document and was used for tracking the flow of research evidence among actors at different phases of the policymaking process.

Findings

Analysis of the textual data demonstrates that conceptual use of research evidence was the most common at each phase of the policymaking process, but instrumental use (i.e., using evidence to choose among policy alternatives) was more likely when evidence was presented as authoritative and the policy proposed was relatively uncontroversial or unopposed. The social network analysis of these data show clustering of use of research evidence by policy and topic, with a distinct group of legislators serving knowledge brokering functions, including knowledge transfer, knowledge exchange, and knowledge advocacy (calling for investments in research).

Implications for D&I Research

There is a significant opportunity to increase use of evidence in health policymaking by identifying knowledge brokers regarding a particular issue and routinely supplying them with updated, high-quality, and relevant evidence.

Primary Funding Source

Use of Research Evidence Award (##185220)

Models, Measures, and Methods

S57 Defining and developing a pragmatic construct for implementation measurement

Cameo Stanick¹, Heather Halko², Caitlin Dorsey³, Byron Powell⁴, Bryan Weiner⁵, Cara Lewis⁶

¹Hathaway-Sycamores Child and Family Services, Pasadena, CA, 91105, USA; ²Department of Psychology, University of Montana, Missoula, MT, 59812, USA; ³Department of Psychological and Brain Sciences, Indiana University, Bloomington, IN, 47405, USA; ⁴Health Policy & Management, University of North Carolina at Chapel Hill, Chapel Hill, NC, 27599-7411, USA; ⁵Department of Global Health, University of Washington, Seattle, WA, 98195, USA; ⁶Group Health Research Institute, Seattle, WA, 98101, USA

Correspondence: Cameo Stanick

Background

Implementation measures largely remain a scientific phenomenon, rarely employed by stakeholders to make clinical program changes. Two likely reasons are: (1) stakeholders typically are not trained to use quantitative measures; (2) measures are typically not designed for use outside of the research context (e.g., often high participant burden, low clinical relevance). If measures are not made to be more pragmatic, stakeholders will remain limited in their ability to make implementation decisions and the gap between implementation science and practice will grow. Glasgow and Riley articulated a definition for the pragmatic construct, indicating that pragmatic measures are (a) important to stakeholders, (b) of low burden for respondents and staff, (c) ‘actionable,’ and (d) sensitive to change. This definition was not informed by stakeholders, nor did it systematically integrate previous literature. Although there is clear face validity to their dimensions, it is possible that key dimensions were overlooked and/or that stakeholders may prioritize dimensions differently than scientists. The present study aimed to conduct: (1) a systematic literature review and (2) stakeholder interviews to reveal pragmatic measures dimensions.

Methods

PsycINFO and PubMed were the databases used to identify dimensions of the pragmatic construct. Simultaneously, an international stakeholder panel was interviewed to obtain their perspectives of pragmatic measures.

Findings

Combined results from the literature review and stakeholder interviews revealed a final list of 47 short statements (e.g., low cost, brief), which will allow for the development of a rigorous, stakeholder-driven conceptualization of the pragmatic measures construct and will aid in measure development to support implementation practice.

Implications for D&I Research

Combined results from the literature review and stakeholder interviews revealed a final list of 47 short statements (e.g., low cost, brief), which will allow for the development of a rigorous, stakeholder-driven conceptualization of the pragmatic measures construct and will aid in measure development to support implementation practice.

Primary Funding Source

National Institutes of Health - Acknowledgements: Research reported in this publication was supported by the National Institute of Mental Health of the National Institutes of Health under Award Number R01MH106510

S58 Stakeholders’ perceptions of criteria for pragmatic measurement in implementation: a concept mapping approach

Byron Powell¹, Bryan Weiner², Cameo Stanick³, Heather Halko⁴, Caitlin Dorsey⁵, Cara Lewis⁶

¹Health Policy & Management, University of North Carolina at Chapel Hill, Chapel Hill, NC, 27599-7411, USA; ²Department of Global Health, University of Washington, Seattle, WA, 98195, USA; ³Hathaway-Sycamores Child and Family Services, Pasadena, CA, 91105, USA; ⁴Department of Psychology, University of Montana, Missoula, MT, 59812, USA; ⁵Department of Psychological and Brain Sciences, Indiana University, Bloomington, IN, 47405, USA; ⁶Group Health Research Institute, Seattle, WA, 98101, USA

Correspondence: Byron Powell

Background

There is a need for valid and reliable measures that can aid in prospectively assessing barriers and facilitators to implementing evidence-based practices; developing, selecting, and tailoring implementation strategies; and evaluating implementation outcomes. However, practitioners are unlikely to use these measures if they are not pragmatic (i.e., relevant and feasible for use in real-world settings); thus, there is a need to develop stakeholder-informed criteria by which to assess the extent to which measures are pragmatic. A previous study generated 47 criteria for pragmatic measures (e.g., easy to interpret, low cost, uses accessible language) through a structured literature review and semi-structured interviews with implementation stakeholders. The current study engaged stakeholders with expertise in implementation practice in order to 1) generate conceptually distinct clusters of criteria for pragmatic measures, and 2) to assess stakeholders’ perceptions of each criterion’s clarity and importance.

Methods

Twenty-four stakeholders with expertise in implementation practice were engaged in a concept mapping process, which involved structured sorting and rating tasks intended to organize the initial list of 47 criteria into conceptually distinct categories and to derive ratings of their clarity and importance. Data collection and analysis were completed using Concept Systems Global Max©, and involved the use of multidimensional scaling, hierarchical cluster analysis, and descriptive statistics

Findings

The 47 criteria were meaningfully grouped into four distinct categories: 1) useful (e.g., “able to inform decision making”), 2) compatible (e.g., “one that is the output of routine activities”), 3) easy (e.g., “brief”), and 4) acceptable (e.g., “offers relative advantage over existing methods”). Average ratings of clarity and importance at the category and individual criteria level will be presented.

Implications for D&I Research

This study illustrates how concept mapping can be leveraged to obtain stakeholder consensus, and advances implementation science by providing criteria that can be used to identify pragmatic measures. Next steps will include a Delphi process to develop consensus on the most important criteria, after which we will develop quantifiable pragmatic rating criteria that can be used to assess measures.

Primary Funding Source

National Institutes of Health - Acknowledgements: Research reported in this publication was supported by the National Institute of Mental Health of the National Institutes of Health under Award Number R01MH106510

S59 Psychometric assessment of three newly developed implementation outcome measures

Bryan Weiner¹, Caitlin Dorsey², Cameo Stanick³, Heather Halko⁴, Byron Powell⁵, Cara Lewis⁶

¹Department of Global Health, University of Washington, Seattle, WA, 98195, USA; ²Department of Psychological and Brain Sciences, Indiana University, Bloomington, IN, 47405, USA; ³Hathaway-Sycamores Child and Family Services, Pasadena, CA, 91105, USA; ⁴Department of Psychology, University of Montana, Missoula, MT, 59812, USA; ⁵Health Policy & Management, University of North Carolina at Chapel Hill, Chapel Hill, NC, 27599-7411, USA; ⁶Group Health Research Institute, Seattle, WA, 98101, USA

Correspondence: Bryan Weiner

Background

Implementation outcome measures are essential for monitoring and evaluating the success of implementation efforts and comparing the effectiveness of alternative implementation strategies. Yet available measures lack conceptual clarity and have questionable reliability and validity. We systematically developed and psychometrically assessed three new measures of implementation outcomes: acceptability, appropriateness, and feasibility.

Methods

Following domain delineation and item generation, we assessed substantive and discriminant content validity by asking 36 implementation scientists and 27 mental health professionals to assign 31 items to the 3 constructs and rate their confidence in their assignments. We used the Wilcoxon one-sample signed rank test to determine whether items represent their intended constructs more so than the other constructs. We used exploratory and confirmatory factor analysis (EFA and CFA) and Cronbach ɑ to assess the validity of our conceptual model. To assess structural validity, reliability, and known-groups validity, we asked 321 mental health counselors to read one of six randomly assigned vignettes depicting a therapist contemplating adopting an evidence-based practice (EBP). Participants used 15 items to rate the therapist’s perceptions of the acceptability, appropriateness, and feasibility of adopting the EBP. Structural validity was assessed using CFA. Reliability was assessed using Cronbach ɑ. Known-groups validity was assessed using analysis of variance (ANOVA).

Findings

Median weighted assignments for all but five items were significantly greater than zero, indicating that participants judged the items to reflect to a significantly greater degree the constructs they were intended to measure than the other constructs. A trimmed CFA with 5 items per construct exhibited good model fit, as evidenced by CFI = 0.98 and RMSEA = 0.08 [CI, 0.04-0.11], with factor loadings between .79 and .94. The ɑ’s for the trimmed scales were between .87-.89. Structural validity and reliability data has been collected and is presently being analyzed.

Implications for D&I Research

The three newly developed measures demonstrate promising psychometric properties. They are also brief. In planned work, we will assess their predictive validity.

Primary Funding Source

National Institutes of Health - Acknowledgements: Research reported in this publication was supported by the National Institute of Mental Health of the National Institutes of Health under Award Number R01MH106510

S60 Strategies for assessing fidelity to evidence-based interventions: a comparison of feasibility, accuracy, and associations with clinical outcomes

Shannon Wiltsey Stirman^1,2, Patricia Carreno², Kera Mallard², Tasoula Masina³, Candice Monson³

¹Stanford University, Menlo Park, CA, 94025, USA; ²Dissemination and Training Division, National Center for PTSD, Menlo Park, CA, 94025, USA; ³Department of Psychology, Ryerson University, Toronto, ON, M4C1B5, Canada

Correspondence: Shannon Wiltsey Stirman

Background

Fidelity is a key implementation outcome, but measurement is challenging due to time and costs associated with observation-based fidelity assessments. Potentially scalable forms of fidelity assessment, such as self-report or interview data, may be less accurate due to a variety of response biases, and clinical notes may be subject to similar biases. Work samples such as clinical notes or “products” generated during the intervention are promising alternatives for some interventions. However, the relative feasibility and predictive validity of these approaches is unknown. This study compared self-reported, interviewer, assessed, observer-rated and work-sample rated fidelity in a sample of clinicians delivering an evidence-based psychotherapy for PTSD.

Methods

Monthly fidelity assessment in the form of self-report and adherence checklists embedded in clinical notes, ratings of clinical worksheets, interview-reported fidelity, and observer-rated fidelity were assessed in a sample of 45 clinicians and 90 clients. Correlations between these forms of fidelity assessment were examined. Response rates, clinician burden (time) and time required for ratings were compared for the different modes of data collection were assessed and compared.

Findings

Preliminary data indicates that self-reported adherence was moderate, (m = 2.28/4; sd = .71; and observer rater adherence was high (m = .89/1; sd = .17). Observer and self-reported adherence were correlated (r = .48, p = .04). Compliance rates were highest for provision of audio recordings and self-report monthly surveys, and lower for clinical notes, perhaps because certification was contingent on provision of audio recordings and direct access to clinical records was not provided. Clinician burden for providing interviews, self-report and worksheet were perceived to be higher than audio recordings, although none were perceived to be significant burdens. Rater time was most intensive for observer ratings (60 minutes per session), followed by interview (10 minutes for relevant portions) and worksheets (5-7 minutes). Associations between fidelity scores and clinical outcomes will be examined (all data are collected).

Implications for D&I Research

Lower-burden forms of measurement strategies may be feasible and valid. Timing fidelity ratings to avoid temporal confounds with symptom change is necessary when examining associations between fidelity and outcomes. Certain types of fidelity assessment may be more feasible and appropriate for these analyses. Strategies for adequate data capture are discussed.

Primary Funding Source

National Institutes of Health - NIMH R01 MH 106506; Canadian Institute of Health Research

S61 An investigation of the impact of accuracy and frequency of self-report on observed fidelity

Taren Swindle¹, Geoffrey Curran², Zachary Patterson¹, Leanne Whiteside-Mansell¹

¹Family and Preventive Medicine, University of Arkansas for Medical Sciences, Little Rock, AR, 72205, USA; ²Department of Pharmacy Practice, University of Arkansas for Medical Sciences, Little Rock, AR, 72205, USA

Correspondence: Taren Swindle

Background

Fidelity