This article has Open Peer Review reports available.
Cluster randomized adaptive implementation trial comparing a standard versus enhanced implementation intervention to improve uptake of an effective re-engagement program for patients with serious mental illness
© Kilbourne et al.; licensee BioMed Central Ltd. 2013
Received: 17 July 2013
Accepted: 14 November 2013
Published: 20 November 2013
Persons with serious mental illness (SMI) are disproportionately burdened by premature mortality. This disparity is exacerbated by poor continuity of care with the health system. The Veterans Health Administration (VA) developed Re-Engage, an effective population-based outreach program to identify veterans with SMI lost to care and to reconnect them with VA services. However, such programs often encounter barriers getting implemented into routine care. Adaptive designs are needed when the implementation intervention requires augmentation within sites that do not initially respond to an initial implementation intervention. This protocol describes the methods used in an adaptive implementation design study that aims to compare the effectiveness of a standard implementation strategy (Replicating Effective Programs, or REP) with REP enhanced with External Facilitation (enhanced REP) to promote the uptake of Re-Engage.
This study employs a four-phase, two-arm, longitudinal, clustered randomized trial design. VA sites (n = 158) across the United States with a designated Re-Engage provider, at least one Veteran with SMI lost to care, and who received standard REP during a six-month run-in phase. Subsequently, 88 sites with inadequate uptake were stratified at the cluster level by geographic region (n = 4) and VA regional service network (n = 20) and randomized to REP (n = 49) vs. enhanced REP (n = 39) in phase two. The primary outcome was the percentage of veterans on each facility outreach list documented on an electronic web registry. The intervention was at the site and network level and consisted of standard REP versus REP enhanced by external phone facilitation consults. At 12 months, enhanced REP sites returned to standard REP and 36 sites with inadequate participation received enhanced REP for six months in phase three. Secondary implementation outcomes included the percentage of veterans contacted directly by site providers and the percentage re-engaged in VA health services.
Adaptive implementation designs consisting of a sequence of decision rules that are tailored based on a site’s uptake of an effective program may produce more relevant, rapid, and generalizable results by more quickly validating or rejecting new implementation strategies, thus enhancing the efficiency and sustainability of implementation research and potentially leading to the rollout of more cost-efficient implementation strategies.
Current Controlled TrialsISRCTN21059161.
Persons with serious mental illnesses (SMI), e.g., bipolar disorder or schizophrenia, experience a disproportionate burden in morbidity and premature mortality from common medical conditions including cardiovascular diseases and certain cancers[1–3]. These physical health disparities may be exacerbated by long gaps in care from the healthcare system due to psychiatric symptoms or access barriers such as lack of transportation, insurance, or relationship with a primary care provider[4, 5]. Many evidence-based practices help to mitigate these risks when this population remains engaged in care.
Continuity and coordination of care for vulnerable health populations with chronic conditions such as SMI are key components of the Chronic Care Model[7, 8]. The Chronic Care Model is a population- and measurement-based approach that calls for healthcare organizations to use electronic registries to monitor vulnerable populations and to adjust treatment according to patient response. Not only has this model of care been successful in managing mental health across various healthcare settings[9, 10], a number of large healthcare providers including the Veterans Health Administration (VA) have demonstrated that this model of care is effective for re-engaging persons with SMI who had been lost to care to prevent adverse health effects[11–13].
Despite the promise of the Chronic Care Model and similar population management programs, they are rarely routinely implemented in practice[14, 15]. Several reasons that contribute to this research-to-practice gap have been described elsewhere and include system and provider-level barriers to program uptake.
Identifying effective implementation interventions that address system and provider barriers can speed program uptake in routine practice. Implementation interventions are operationalized techniques based on an underlying framework or theory that are designed to enhance the uptake of effective programs across different healthcare settings. Implementation interventions address multilevel barriers to program adoption, such as organizational culture, leadership buy-in, and provider training and capacity[16, 18–24] to ultimately enhance program uptake.
Studies involving implementation intervention strategies have been referred to as type III hybrid-effectiveness implementation studies, where the intervention is the implementation strategy and the primary outcomes are focused on program uptake rather than testing the effectiveness of the program itself on patient outcomes.
A handful of type III hybrid effectiveness-implementation studies based on underlying implementation frameworks[26–30] have been recently conducted[17, 31–34]. These studies involved highly specified implementation intervention strategies such as Replicating Effective Programs, Facilitation, or Evidence-based Quality Improvement that address multiple organizational and provider barriers[17, 32, 34]. These studies typically randomized sites to receive a new implementation strategy or standard dissemination to enhance the uptake of an effective program. Most of these studies take place in highly organized sites or treatment settings such as the VA.
Applying traditional randomized trial designs to complex implementation interventions can be challenging because they require several sites to achieve adequate power, and involve monitoring of both program and implementation intervention fidelity. Hence, these designs may not accommodate lower resourced sites that are less willing to be randomized or participate in study assessments—the very sites implementation interventions are designed to assist. Moreover, not all sites may require the same level of implementation intervention, and some may require additional assistance due to underlying barriers to program adoption that are not apparent or measurable at baseline. This can lead to less cost-efficient use of implementation resources such as provider training, technical assistance, and the time require to build relationships with leaders and frontline providers across sites. In many situations, it is also unclear how long an implementation intervention is needed to improve program uptake[36, 37].
In response to these challenges, we describe a new approach to implementation interventions. Increasingly used in clinical research, adaptive interventions guide the decisions to augment (change or adapt) existing interventions given signs of non-response (or other intermediate outcomes) during treatment[38–41]. When applied to implementation intervention studies, adaptive interventions allow sites that are not responding to an initial implementation strategy to receive an augmented implementation intervention. In contrast to simply measuring correlates of implementation non-response across sites, studies of adaptive implementation interventions can help to determine the added value of a more intensive implementation intervention strategy and how long the more intensive implementation strategy should be continued to achieve improved program uptake at individual sites.
Aims and objectives
The aim of this study is to use an adaptive implementation design to compare the effectiveness of an enhanced versus standard version of an implementation strategy (Replicating Effective Programs, or REP) to promote the uptake of a population management program for patients with serious mental illness who have dropped out of care (‘Re-Engage’). REP is a previously operationalized implementation strategy that has been shown to improve the uptake of effective Chronic Care Model and related programs[28, 31, 42] and consists of program manual dissemination, training, and brief technical assistance. Enhanced REP includes standard REP with facilitation, which involves proactive coaching by a program expert that is focused on enhancing provider buy-in and uptake.
The primary implementation outcome is the uptake of the Re-Engage program, defined as the percentage of veterans’ with an updated documentation of their clinical status within 12 months, which is a central component of Re-Engage population management. The primary hypothesis is that among facilities not initially responding to standard REP, the addition of facilitation (enhanced REP) will be associated with increased percentages of documented updates to veterans’ clinical status. Secondary outcomes include facilities’ percentage of veterans who were provided brief care management, defined as percentage contacted or percentage returning to VA care. Additionally, we seek to explore whether among facilities that initially did not respond to standard REP the immediate addition of facilitation (enhanced REP) is associated with better outcomes than receiving Facilitation after a six-month delay.
Described previously[5, 11, 13, 43–48] Re-Engage is a VA nationally mandated brief care management program for veterans with serious mental illness. At the time of protocol submission, the trial intervention had already started and collection of outcomes had begun. This study was reviewed and approved by the local VA Institutional Review Boards and was registered as a clinical trial (Current Controlled TrialsISRCTN21059161).
The six-month run-in phase included standard REP components, and at the end of the run-in phase, sites not responding to standard REP were identified. Phase one involved randomization to two site-level implementation interventions: Enhanced REP (adding Facilitation) or continuation of standard REP technical assistance on an as-needed basis. Phase two involved offering sites who were randomized to standard REP in phase one the enhanced REP facilitation, and discontinuing facilitation among sites who received facilitation in phase one.
During the initial run-in phase (1 March 2012 to 31 August 2012), all eligible VA sites nationally received standard REP to implement the Re-Engage program. As in the original study, providers implementing Re-Engage were asked to identify and document their patients' current disposition based on a pre-generated list of those who had dropped out of care, as well to attempt to contact them and invite them back to VA health services. The initial run-in phase began when the requirement to implement the program was communicated to sites in March 2012 and continued until 31 August 2012. During this phase, the designated mental health provider was identified at each eligible facility, and he or she received a computerized list of patients who had dropped out of care and a website link to track their status, a package describing the Re-Engage program, training, and brief technical assistance.
Sites with inadequate implementation of Re-Engage (i.e., non-responding sites) as of 1 September 2012 were then identified based on a previously established eligibility criterion and randomized to receive enhanced REP or continued REP. Inadequate implementation of Re-Engage was defined as documenting and attempting to contact less than 80% of patients on the drop-out list, based on a review of the website registry from each site. This previously established measure is considered a core component of the Re-Engage program because it is an indicator of whether the provider reviewed the list and attempted to find the patient. This measure was used to benchmark implementation response because this measure is most likely to be impacted by individual providers. A cut-point of 80% was selected because it is a standard definition used to determine adequate adherence to practice guidelines based on the Agency for Healthcare Research and Quality.
Sites having successfully implemented Re-Engage by the end of the run-in phase (31 August 2012) continued to receive standard REP for phases one, two, and the follow-up phase of the trial (Figure 2).
Sites that had not adequately implemented Re-Engage as of the end of the run-in phase (i.e., non-responding sites) were stratified by geographic region and 1:1 randomized by the sites’ VA regional network (n = 20) to enhanced or continued standard REP. Because each VA regional network has mental health leaders that communicate across sites, randomization was conducted at the VA regional network level to minimize potential for contamination. Non-responding sites randomized to receive enhanced REP received six months of facilitation and those randomized to standard REP received technical assistance calls only if they requested it for six months during phase one.
After the end of phase one (28 February 2013), sites originally randomized to receive standard REP and who were still non-responsive (<80% documentation of patients’ clinical status) received enhanced REP facilitation. Sites that were initially randomized to receive standard REP and met the implementation benchmark as of the end of phase one continued to receive standard REP during phase two. Sites receiving enhanced REP during phase one received standard REP in phase two regardless of responsiveness.
After the end of phase two of the study (31 August, 2013), standard REP will continue and outcomes including VA use will be monitored through 31 August 2014 using previously established methods[51, 52].
This implementation trial is being conducted at the VA facility-level between January 2012 and August 2013. A VA facility was eligible for the current trial if it was included in the national VA Re-Engage program. VA facilities were included in the national Re-Engage program if they were within the 50 United States or Puerto Rico, were required, per VA policy, to have a mental health provider who filled the role of a Local Recovery Coordinator, and had at least one veteran with serious mental illness who was lost to care—i.e., had been seen at the facility in fiscal year (FY) 2008 or FY 2009, but had no subsequent outpatient visits or an inpatient stay of less than two days as of January 2012. There were a total of 158 facilities eligible for Re-Engage, of which 139 were medical centers (i.e., with hospital beds) and 19 were community-based outpatient clinics.
Setting and target population
Re-Engage is a national VA program which has three core components: panel management, brief care management, and proactive outreach services that are designed to re-engage in VA healthcare veterans with serious mental illness (i.e., schizophrenia or bipolar disorder) who previously received VA healthcare, but have not been seen in VA healthcare for at least one year. Re-Engage was initially developed by VA Office of Medical Inspector as a quality improvement program based on awareness that veterans with serious mental illness face high rates of medical comorbidities that require regular medical care[43–45], and that gaps in healthcare services among this population contribute to early mortality[5, 46]. The VA Office of Medical Inspector quality improvement program was completed in 2010 and found that veterans with SMI who returned to care had lower rates of mortality (0.3%) than veterans who were targeted for re-engagement, but did not return to care (3.9%)[11, 47]. As a result, VA mandated that Re-Engage be implemented as part of standard clinical care.
One provider at each VA facility, specifically the Local Recovery Coordinator,, is designated to implement Re-Engage components at his or her facility. Local Recovery Coordinators are typically social workers or psychologists who have both administrative and clinical duties. Re-Engage core components were designed to be part of Local Recovery Coordinator’s clinical duties, and include the following: panel management: receiving a list of veterans with SMI whose last VA healthcare visit was at their facility, reviewing the medical record and other informational sources to locate the veterans, updating their clinical status or disposition (e.g., vital status, whereabouts, etc.) in a web-based clinical registry; outreach: i.e., attempting to contact the veterans in person, via telephone, or mail; and brief care management: completing a semi-structured assessment of veterans current health status and healthcare needs and inviting veterans to return to VA care and assisting with the scheduling of any desired VA appointments.
As part of the Re-Engage program, the VA National Serious Mental Illness Treatment Resource and Evaluation Center (SMITREC) provides the Local Recovery Coordinator at each facility with lists containing the names, last known contact information, and last known recent clinical history for the veterans for the Re-Engage program. Based on previously described processes[11, 13], patients were eligible for Re-Engage if they had at least one diagnosis of schizophrenia (International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) codes 295.0–295.4; 295.6–295.9) or bipolar disorder (ICD-9-CM codes 296.0–296.8) recorded in an inpatient or outpatient visit in FY 2008 or FY 2009; had not been seen in VA care for at least one year (i.e., dropped out of care: defined as no recorded outpatient visits or an inpatient stay of less than two days in the FYs after their last year with any visit); and were still alive as of March 2012 based on currently available mortality information from the VA Beneficiary Identification and Records Locater Subsystem (BIRLS), a well-established resource for VA mortality data, the Social Security Administration Death Master File, and the National Death Index.
As of March 2012, a total of 5,240 veterans were identified through Re-Engage. The average number of identified veterans at each facility was 33.2 (standard dev = 22.5), with a range of 4 to 147. Of these, a subset of veterans were targeted as high-priority for the Re-Engage program if they had at a history of at least one inpatient hospitalization prior to drop out and were less than 65 years of age (i.e., less likely to be in a nursing home or covered by Medicare services). All high priority veterans were included on initial lists that were disseminated to facilities in March 2012. Additional veterans (up to a total of 42 per facility) were included on initial lists based on the dates they were last seen in VA healthcare. Each facility’s initial list contained no more than 42 veterans in order to provide Local Recovery Coordinators with a manageable number of veterans to contact. In March 2012, contact information for a total of 2,733 veterans was disseminated to facilities on the initial lists (n = 2,733, mean per facility = 17.3, std dev 6.3, range: 4 – 42). Contact information for the remaining 2,507 veterans (‘second list’) identified in March 2012 was disseminated in July 2012.
Although Re-Engage is an ongoing clinical program and over time additional veterans who have been lost-to-care will continue to be identified and their names disseminated to local Recovery Coordinators at each facility, this implementation study focused on the first list of veterans (n = 2,733) identified and disseminated in March 2012, and outcomes will be measured on the basis of this cohort.
In phase two of the trial, facilities with insufficient implementation of Re-Engage were stratified by geographic region and randomized at the VA integrated services network-level to receive enhanced REP or continue receiving standard REP. We stratified by geographic region because preliminary analyses indicated that uptake of Re-Engage at the end of phase one differed by geographic region. Randomization was conducted by the study program analyst and occurred at the veterans integrated service network-level as opposed to the facility level because in enhanced REP External facilitation involved communications with regional VA leadership and we sought to minimize the potential for contamination across facilities within the same integrated service network. Because the providers involved in the implementation knew that they were receiving enhanced versus standard REP, allocation concealment and blinded randomization were not applicable.
REP is based on the Centers for Disease Control and Prevention’s Research-to-Practice Framework[28, 29, 42]. Derived from Social Learning Theory and Rogers’ diffusion model, REP consists of three central operational components: program ‘packaging’ (i.e., translation of treatment materials into user-friendly language), provider training, and brief technical assistance for providers to address barriers to uptake. The combination of these three components compared to package dissemination alone resulted in improved uptake and fidelity to HIV prevention intervention programs in AIDS service organizations[42, 57].
Although standard REP employs key tactical strategies that can promote effective adoption of effective programs[42, 58], it was not designed address multilevel barriers to implementation, such as competing demands on providers and limited leadership support for new programs. Hence, REP was enhanced by including facilitation based on the Promoting Action on Research Implementation in Health Services (PARiHS) framework[58, 60–62]. Facilitation is a systematic and iterative process in which implementation experts promote program uptake by working with frontline providers to identify and mitigate barriers to program adoption[63, 64].
Implementation components of standard REP and enhanced REP
Implementation guide was disseminated to all providers at eligible sites, describing the Re-Engage program, a list of frequently asked questions, sample documents for program tasks, program policies, data security, and related research.
Three 1.5-hour national conference call trainings of mental health providers on how to conduct program. Program materials made available on mental health provider website. Research staff available to answer questions via email or telephone.
Ongoing assistance via 1-hour biweekly conference calls led by study staff for mental health providers to answer technical questions on Re-Engage program implementation and study staff were available on an ad-hoc basis to answer questions from individual providers. Monthly reports were generated to track Re-Engage uptake (% patients with updated clinical status documented).
Sites receiving standard REP technical assistance in phases one and two did not receive calls but study staff were available if they were contacted on an as –needed basis to address technical questions regarding Re-Engage implementation.
Facilitators gather information from various sources (monthly evaluation reports, VISN Mental Health Leadership, mental health providers, VA Mental Health Services Leadership) to identify potential facility-specific barriers and facilitators to implementation.
Ongoing partnership support
Weekly phone calls with Facilitators, Technical Assistance staff, and VA national leaders involved in national Re-Engage program and VA mental health services. Facilitators maintain open communication with VA leaders regarding implementation nationally and at specific sites through these phone calls and email communication. Facilitators also maintain ongoing contact with one another through separate weekly meetings.
Garner regional and local support
Facilitators initiate contact with regional mental health leadership affiliated with local sites, providing information regarding Re-Engage program implementation and added value. Maintain ongoing contact and request support from regional leadership as indicated.
Identify barriers and facilitators
Facilitators and mental health providers hold monthly calls for six months and collaboratively identify each facility’s specific challenges (e.g., time, resources) to program implementation as well as potential assets (e.g., consistency with other initiatives, support from local leadership) to program implementation.
Collaboratively develop action plans
Facilitators assist mental health providers in identifying what specific actions they can take to implement program.
Feedback and Link to available resources
Facilitators provide feedback to mental health providers regarding implementation and action plan progress. Facilitators refer mental health providers to existing resources, including the Technical Assistance available through standard REP, existing documents regarding the program intervention, facility-level, regional, or national leadership.
Primary implementation outcome
The primary measure of the uptake of Re-Engage was the percentage of veterans on each facility’s list whose clinical status was updated in the Re-Engage web-based clinical registry. The Veteran Clinical Status Updated measure was calculated as the number of veterans with an update on their clinical status or disposition in the web-based registry compared to the total number of veterans on each facility’s list. This outcome measure indicates whether facilities are actively attempting to locate and contact the veterans who have dropped out of care and is independent of whether the veterans are reachable, and thus is a good indicator of implementation. Consistent with recommendations from previous studies of cut-points used to establish adequate adherence to clinical processes or guidelines[66, 67], inadequate uptake of Re-Engage is defined as whether an updated clinical status was available for less than 80% of the veterans on a given facility list. Although this measure is tabulated monthly for the purposes of the VA Re-Engage program, for the purposes of this implementation study, this outcome measure was examined at the end of the run-in period (end of August 2012), the end of phase one (end of February 2013), the end of phase two (end of August 2013), and during the follow-up phase (end of February 2014, end of August 2014).
Secondary measures of implementation include the percentage of veterans that the Local Recovery Coordinators successfully contacted among those who were on their lists and still alive and able to be contacted (e.g., had available address or phone number, no documentation of institutionalization or incarceration), and the percentage of veterans contacted who re-engaged in VA healthcare services. These measures, percentage of veterans contacted and percentage of veterans Re-Engaged, reflect the brief care management strategies that are part of Re-Engage.
Additional secondary outcome measures include veteran-level variables. All-cause mortality and utilization of VA healthcare services will be compared as a function of whether or not veterans were able to be contacted, and if contacted, whether or not they indicated an interested in returning to VA healthcare. Healthcare utilization variables will include number and length of stay of inpatient medical and mental health hospitalizations, number of outpatient mental health and general medical visits, and number of emergency department visits. Utilization of mental healthcare will be further examined by identifying the number of visits to recovery-oriented mental health services (e.g., psychosocial rehabilitation and recovery centers, supported employment services), as these services are targeted to veterans with SMI. Moreover, because a large proportion of homeless veterans have a psychiatric diagnosis[51, 52], the number of visits to VA homeless program services will be examined.
Potential covariates of re-engage implementation
Covariates for all facilities
Construct from CFIR
Administrative Data- NPR
• Military service period
• Psychiatric Diagnoses
• Indication of a history of homelessness
• History of substance use disorder
Presence of PRRC at site
Administrative Data- MHS maintained
Inner Setting- Implementation Climate- Compatibility
Presence of LRC when Re-Engage was Rolled Out
Administrative Data- MHS maintained
Inner Setting- Readiness for Implementation- Available Resources
Inner Setting- Structural Characteristics
Number of Vets on List
Re-Engage Program Records
Intervention Characteristics- Complexity
Overall facility size
Inner Setting- Structural Characteristics
Number of SMI Vets associated with Facility
Administrative Data- NPR
Inner Setting- Structural Characteristics
Academic Affiliation of facility
Administrative Data- Ascertained via US News & World Report Med School Rankings
Inner Setting- Culture
Inner Setting- Structural Characteristics
Whether the site or VISN viewed Re-Engage as a research project
Technical Assistance Minutes
Either: Characteristics of Individuals- Knowledge & Beliefs about the Intervention OR Characteristics of the Intervention- Intervention Source
The performance of other VAs in VISN on SMI Re-Engage Implementation
Re-Engage Program Records
Outer Setting- Peer Pressure
Covariates for sites receiving enhanced REP
Construct from CFIR
Number of Facilitation Contacts with Site
Number of Facilitation Contacts with VISN Mental Health Leadership
Did the Facility adapt SMI Re-Engage and use a Team approach?
Intervention Characteristics- Adaptability
Was there evidence that SMI Re-Engage was a priority in VISN or at site?
Inner Setting- Implementation Culture- Relative Priority
Did the LRC perceive available time (or resources) in order to do the required aspects of the SMI Re-Engage program?
Inner Setting- Readiness for Implementation- Available Resources
Does the VISN Mental Health Lead/seem to have a positive view of SMI Re-Engage?
Characteristics of Individuals- Knowledge & Beliefs about the Intervention
Does the LRC at a site seem to have a positive view of SMI Re-Engage?
Characteristics of Individuals- Knowledge & Beliefs about the Intervention
Does the VISN Mental Health Lead accurately understand SMI Re-engage?
Characteristics of Individuals- Knowledge & Beliefs about the Intervention
Does the LRC accurately understand SMI Re-Engage?
Characteristics of Individuals- Knowledge & Beliefs about the Intervention
Does the LRC feel capable of executing the tasks associated with SMI Re-Engage?
Characteristics of Individuals- Self-efficacy
To examine the effects of contacting veterans and re-engaging veterans on patient-level outcomes of mortality and healthcare utilization, we will use generalized mixed effects models to account for VISN and facility-level covariates as well as patient-level characteristics. We will employ logistic regression models or Poisson models as appropriate, based on whether dependent measure is dichotomous (e.g., mortality) or a count variable (e.g., number of hospitalizations).
Our study included all eligible VA facilities (n = 158) within the 21 VA integrated service networks (VISNs) that had a provider to implement Re-Engage at the time of this study. All 158 VA facilities received standard REP in phase one of this trial. Among these, at the end of phase one, 88 facilities (55.7%) in 20 of the 21 VISNs had updated the clinical status of less than 80% of veterans’, indicating insufficient implementation of Re-Engage. These 20 VISNs (containing 88 facilities with insufficient implementation of Re-Engage) entered phase two of the trial and were thus randomized (as described above) to continue standard REP or receive enhanced REP. Through randomization, nine VISNs that included facilities were assigned to enhanced REP, and the remaining 11 VISNS that included 49 facilities were assigned to standard REP.
Statistical power considerations
The data analysis plan for the primary aim is a two-sample comparison of facilities within VISNs randomized to enhanced versus standard REP. Based on the sample sizes described above, we conducted analyses to determine whether we had adequate statistical power to detect a significant difference in our primary (percentage of veterans with updated clinical status) and secondary (percentage of veterans contacted, percentage veterans Re-Engaged) facility-level implementation outcomes between the two groups of facilities.
At the end of phase one, the average percentage of veterans with and updated clinical status (primary implementation outcome) among the 88 underperforming facilities was 22% (SD = 25%). To account for the between-VISN variation induced by the within-VISN correlation in the average rate of the percentage of veterans with updated clinical status, we inflate the variance term in the standard sample size formula by 1 + (n-1)*ICC where ICC is the VISN interclass correlation coefficient for the average percentage of veterans with updated clinical status. The ICC for the average percentage of veterans with updated clinical status among the 20 VISNs entering phase two was 0.177. Using a two-sided, two-sample t test based on the sample sizes given above, a Type-I error rate of 5%, an ICC = 0.177, we will have 80% power to detect an effect size of 0.72 (Cohen’s D). This effect size corresponds to a between-site difference of 21 percentage points in the percentage of veterans with an updated clinical status.
For the secondary outcome the percentage of veterans contacted, based on the initial values of 35% (SD = 29%), and ICC = 0.31, with 80% power we will can detect an effect size of 0.78 (Cohen’s D), which corresponds to an approximate difference of 22 percentage points. Similarly, for the secondary outcome, the percentage of veterans re-engaged in care, based on the initial values of 26% (SD = 38%), we will have 80% to detect an effect size of 0.88 (Cohen’s D), or a difference of 33 percentage points.
Covariates of implementation outcomes
Drawing on the Consolidated Framework for Implementation Science (CFIR) and the PARiHS Framework, we identified organizational and facility as well as patient-level variables that we will adjust for in examining implementation of Re-Engage. Covariates are listed in Table 2, along with the constructs they represent based on CFIR, and the data source(s) that will be used to ascertain the variable.
Data sources for patient-level data will include information ascertained from national VA administrative databases (e.g., National Psychosis Registry) as well as information gathered through Local Recovery Coordinators’ contacts with veterans. Data sources for organizational and contextual factors will include VA administrative records and the all-employee survey, minutes from standard REP technical assistance calls, and surveys completed by facility clinicians and administrators. Surveys were disseminated to facility Local Recovery Coordinators, Homelessness Coordinators, and Mental Health Service Line Leaders beginning in March 2012. The surveys sent to the Local Recovery Coordinators and Homeless Coordinators ascertained their job duties and the frequency of their interactions with clinicians in other roles. Mental Health Service Line Leaders were provided with the Mental Health Program Survey to assess structural and organizational characteristics of mental health services at the facility. For facilities that received enhanced REP, Facilitators’ notes on contacts with facilities and stakeholders will serve as additional data sources.
Qualitative data from Facilitators’ notes and technical assistance minutes will be coded to identify possible barriers and facilitators to program implementation. Although many of the potential covariates were identified a priori (e.g., structural and organizational characteristics), additional potential covariates have been identified through standard REP and enhanced REP processes.
Monitoring implementation intervention
Documentation of enhanced REP Facilitators’ Core Component Tasks and Time
Core facilitation task
Preparation for communication with facilities or regional leadership
• Review implementation progress
• Review barriers or facilitators to implementation described during last contact or ascertained through other sources as documented in Facilitator database
• Review stated actions planned from last contact (i.e., action plans) as documented in Facilitator database
Semi-structured communication with facilities and/or regional stakeholders (phone call)
• Discuss progress on action plans that were established in prior contact
• Discuss implementation progress based on monthly report
• Provide support, encouragement, reinforcement of progress made
• Collaboratively identify additional/existing barriers, changes to context that could affect implementation
• Problem-solve strategies, solutions to address barriers
• Collaboratively identify additional/existing facilitators and discuss how to use them to encourage implementation
• Provide suggestions for how to adapt intervention to local setting without compromising core components
• If needed, refer to technical assistance resources (available through standard REP)
• Provide information in response to questions, concerns, or promise to obtain needed information
• Collaboratively identify specific actions that can be taken to assist in implementation prior to next contact
• Number of contacts with facility mental health provider implementing Re-Engage
• Number of contacts with regional network leaders
• Number of barriers and solutions discussed with facility providers
• Follow-up emails and phone calls to link to existing resources (e.g., technical assistance, leadership, continue problem solving a specific issue, provide information in response to a question
• Schedule next contact (e.g., schedule conference call lines, email facilities that have not been responsive)
Facilitators weekly communication with leadership partners (phone call)
• Facilitators join the weekly calls between research staff and VA national mental health leadership
• Provide overview of facilitation progress
• Provide information to/seek information from VA Mental Health Services Leadership about facility-specific issues that may have arisen during the week’s facilitation communications, seek guidance as needed
• Obtain information regarding other initiatives affecting mental health providers
Facilitators weekly peer consultation meeting
• Review each facility receiving facilitation, identify implementation progress, barriers
• Discuss strategies being used to encourage implementation at each facility
• Provide support, encouragement, and accountability to one another
• Provide information to/seek information from Technical Assistance research staff, as needed
To date the run-in phase and phase one of the implementation trial have been completed, and phase two of the trial is ongoing at this time. All 39 sites within 9 VA regional networks that received enhanced REP during phase one have now returned to receiving standard REP. Of the 49 sites within 11 VA regional networks that were randomized to standard REP during phase one of the trial, 36 sites (73.5%) in 10 VA regional networks continued to have less than 80% of their patients on their list with updated clinical status at the end of phase 1 and will receive enhanced REP.
This paper describes to our knowledge one of the first adaptive implementation intervention trial designs. The study is testing the effectiveness of facilitation as an adjunct to standard REP among non-responding sites on the implementation of a national VA program, Re-Engage, which is designed to assist patients with serious mental illness who have dropped out of care return to VA services.
An adaptive implementation design was optimal for this particular study because enhanced REP included additional personnel time and effort that may, over time, be too expensive to implement. Not all sites may need a more intensive implementation intervention (enhanced REP facilitation) to promote the uptake of an effective program. Hence, the adaptive nature of this trial randomized sites that required additional assistance, which allowed for more efficient use of facilitation resources. In addition, comparing the timing of added facilitation (immediately after observed non-response in phase one or six months later during phase two) provided an opportunity to focus on the impact of facilitation among later adopters of effective programs at the site level.
In addition, this study was also to date one of the first implementation intervention trials that took advantage of a population-based, national rollout of an effective program within a US health system. Hence, the study sought to use the VA’s national mandate to implement Re-Engage as the foundation for a natural experiment to test different implementation intervention strategies. The VA was an ideal setting in which to conduct this adaptive implementation trial because of the availability of national patient data and provider networks to identify those who had dropped out of care and to monitor subsequent use and outcomes at the patient and site levels. By including all sites that were required to implement Re-Engage per the mandate, there was opportunity to test the implementation interventions among sites that were less likely to initially respond to a lower-cost implementation strategy (REP). Previous implementation intervention trials often had to seek permission first from sites to participate in a study, often leading to potential cherry-picking and selection effects that may skew implementation results.
Another advantage in conducting an adaptive implementation strategy is that measuring site contextual factors that might influence program uptake is not required. Instead, sites are included in the randomization portion of the study only if they are not responding to an initial implementation strategy, and the reasons for non-response may not be observable or measurable even with organizational assessments. Moreover, because the national rollout of Re-Engage occurred relatively quickly, the window of opportunity to ascertain organizational data across all of the sites to predict implementation non-response was limited. Several organizational assessments exist, but to date most have not been systematically used to identify predictors of program uptake, and those predictors may vary depending on the particular program.
There are limitations to this type of design that warrant consideration, especially when deciding to replicate the design in other settings. Notably, adaptive implementation intervention designs are potentially less feasible in settings without large numbers of sites that are either willing or mandated to provide the effective program, and have access to common data sources to gauge patient outcomes. Second, the large number of sites precluded more intensive monitoring of program fidelity beyond documentation by the frontline provider via the website. Third, cost considerations precluded having Facilitators as part of the enhanced REP intervention make site visits or involve local site leaders on a more regular basis. At least one prior study or enhanced REP included an Internal as well as External Facilitator who can provider more on-the-ground coaching and guidance to the frontline mental health provider, linking them to leaders and resources not available or known to an outside External Facilitator[31–34].
The results of this study will yield new information on how to conduct adaptive implementation intervention trials at the national level. These findings will have the potential to inform not only further implementation research, but also the actual implementation of effective programs in large healthcare settings. This study also sets the stage for determining the added value of more intensive implementation interventions within sites that need additional support to promote the uptake of effective programs. Ultimately, adaptive implementation designs may produce more relevant, rapid, and generalizable results by more quickly validating or rejecting new implementation strategies, thus enhancing the efficiency and sustainability of implementation research and potentially lead to the rollout of more cost-efficient implementation strategies.
This research was supported by the Department of Veterans Affairs, Veterans Health Administration, Health Service Research and Development Service (SDR 11–232). The views expressed in this article are those of the authors and do not necessarily represent the views of the VA. We would also like to acknowledge the support of the VA Office of the Medical Inspector, and the Office of Mental Health Services and Mental Health Operations of the Veterans Health Administration.
- Crump C, Winkleby MA, Sundquist K, Sundquist J: Comorbidities and Mortality in Persons With Schizophrenia: A Swedish National Cohort Study. Am J Psychiatr. 2013, 170: 324-333. 10.1176/appi.ajp.2012.12050599.View ArticlePubMedGoogle Scholar
- Kilbourne AM, Ignacio RV, Kim HM, Blow FC: Datapoints: are VA patients with serious mental illness dying younger?. Psychiatr Serv. 2009, 60: 589. 10.1176/appi.ps.60.5.589.View ArticlePubMedGoogle Scholar
- Saha S, Chant D, McGrath J: A systematic review of mortality in schizophrenia: is the differential mortality gap worsening over time?. Arch Gen Psychiatry. 2007, 64: 1123-1131. 10.1001/archpsyc.64.10.1123.View ArticlePubMedGoogle Scholar
- McCarthy JF, Blow FC, Valenstein M, Fischer EP, Owen RR, Barry KL, Hudson TJ, Ignacio RV: Veterans Affairs Health System and mental health treatment retention among patients with serious mental illness: evaluating accessibility and availability barriers. Health Serv Res. 2007, 42: 1042-1060. 10.1111/j.1475-6773.2006.00642.x.View ArticlePubMedPubMed CentralGoogle Scholar
- Copeland LA, Zeber JE, Rosenheck RA, Miller AL: Unforeseen inpatient mortality among veterans with schizophrenia. Med Care. 2006, 44: 110-116. 10.1097/01.mlr.0000196973.99080.fb.View ArticlePubMedGoogle Scholar
- Kreyenbuhl J, Buchanan RW, Dickerson FB, Dixon LB: Schizophrenia Patient Outcomes Research T: The Schizophrenia Patient Outcomes Research Team (PORT): updated treatment recommendations 2009. Schizophr Bull. 2010, 36: 94-103. 10.1093/schbul/sbp130.View ArticlePubMedGoogle Scholar
- Wagner EH, Austin BT, Von Korff M: Organizing care for patients with chronic illness. Milbank Q. 1996, 74: 511-544. 10.2307/3350391.View ArticlePubMedGoogle Scholar
- Goodrich DE, Kilbourne AM, Nord KM, Bauer MS: Mental health collaborative care and its role in primary care settings. Curr Psychiatry Rep. 2013, 15: 383.View ArticlePubMedPubMed CentralGoogle Scholar
- Woltmann E, Grogan-Kaylor A, Perron B, Georges H, Kilbourne AM, Bauer MS: Comparative effectiveness of collaborative chronic care models for mental health conditions across primary, specialty, and behavioral health care settings: systematic review and meta-analysis. Am J Psychiatr. 2012, 169: 790-804. 10.1176/appi.ajp.2012.11111616.View ArticlePubMedGoogle Scholar
- Thota AB, Sipe TA, Byard GJ, Zometa CS, Hahn RA, McKnight-Eily LR, Chapman DP, Abraido-Lanza AF, Pearson JL, Anderson CW: Collaborative care to improve the management of depressive disorders: a community guide systematic review and meta-analysis. Am J Prev Med. 2012, 42: 525-538. 10.1016/j.amepre.2012.01.019.View ArticlePubMedGoogle Scholar
- Davis CL, Kilbourne AM, Pierce JR, Langberg R, Blow FC, Winkel BM, Huycke E, Visnic S, Lyle D, Yancy P: Reduced mortality among VA patients with schizophrenia or bipolar disorder lost to follow-up and engaged in active outreach to return to care. Am J Public Health. 2012, 102: S74-S79. 10.2105/AJPH.2011.300502.View ArticlePubMedPubMed CentralGoogle Scholar
- Smith TE, Appel A, Donahue SA, Essock SM, Jackson CT, Karpati A, Marsik T, Myers RW, Tom L: Use of administrative data to identify potential service gaps for individuals with serious mental illness. Psychiatr Serv. 2011, 62: 1094-1097. 10.1176/appi.ps.62.9.1094.View ArticlePubMedGoogle Scholar
- Goodrich DE, Bowersox NW, Abraham KM, Burk JP, Visnic S, Lai Z, Kilbourne AM: Leading from the middle: replication of a re-engagement program for veterans with mental disorders lost to follow-up care. Depress Res Treat. 2012, 2012: 325249.PubMedPubMed CentralGoogle Scholar
- Riley WT, Glasgow RE, Etheredge L, Abernethy AP: Rapid, responsive, relevant (R3) research: a call for a rapid learning health research enterprise. Clin Transl Med. 2013, 2: 10. 10.1186/2001-1326-2-10.View ArticlePubMedPubMed CentralGoogle Scholar
- Kessler R, Glasgow RE: A proposal to speed translation of healthcare research into practice: dramatic change is needed. Am J Prev Med. 2011, 40: 637-644. 10.1016/j.amepre.2011.02.023.View ArticlePubMedGoogle Scholar
- Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC: Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci. 2009, 4: 50. 10.1186/1748-5908-4-50.View ArticlePubMedPubMed CentralGoogle Scholar
- Curran GM, Bauer M, Mittman B, Pyne JM, Stetler C: Effectiveness-implementation hybrid designs: combining elements of clinical effectiveness and implementation research to enhance public health impact. Med Care. 2012, 50: 217-226. 10.1097/MLR.0b013e3182408812.View ArticlePubMedPubMed CentralGoogle Scholar
- Shortell SM, Bennett CL, Byck GR: Assessing the impact of continuous quality improvement on clinical practice: what it will take to accelerate progress. Milbank Q. 1998, 76: 593-624. 10.1111/1468-0009.00107. 510View ArticlePubMedPubMed CentralGoogle Scholar
- VanDeusen LC, Engle RL, Holmes SK, Parker VA, Petzel RA, Nealon Seibert M, Shwartz M, Sullivan JL: Strengthening organizations to implement evidence-based clinical practices. Health Care Manage Rev. 2010, 35: 235-245. 10.1097/HMR.0b013e3181dde6a5.View ArticleGoogle Scholar
- Lukas CV, Holmes SK, Cohen AB, Restuccia J, Cramer IE, Shwartz M, Charns MP: Transformational change in health care systems: an organizational model. Health Care Manage Rev. 2007, 32: 309-320. 10.1097/01.HMR.0000296785.29718.5d.View ArticlePubMedGoogle Scholar
- Lukas CV, Mohr DC, Meterko M: Team effectiveness and organizational context in the implementation of a clinical innovation. Qual Manag Health Care. 2009, 18: 25-39. 10.1097/01.QMH.0000344591.56133.90.View ArticlePubMedGoogle Scholar
- Damschroder LJ, Lowery JC: Evaluation of a large-scale weight management program using the consolidated framework for implementation research (CFIR). Implement Sci. 2013, 8: 51. 10.1186/1748-5908-8-51.View ArticlePubMedPubMed CentralGoogle Scholar
- Rubenstein LV, Parker LE, Meredith LS, Altschuler A, DePillis E, Hernandez J, Gordon NP: Understanding team-based quality improvement for depression in primary care. Health Serv Res. 2002, 37: 1009-1029. 10.1034/j.1600-0560.2002.63.x.View ArticlePubMedPubMed CentralGoogle Scholar
- Lukas CV, Meterko MM, Mohr D, Seibert MN, Parlier R, Levesque O, Petzel RA: Implementation of a clinical innovation: the case of advanced clinic access in the Department of Veterans Affairs. J Ambul Care Manage. 2008, 31: 94-108. 10.1097/01.JAC.0000314699.04301.3e.View ArticlePubMedGoogle Scholar
- Shortell SM, O'Brien JL, Carman JM, Foster RW, Hughes EF, Boerstler H, O'Connor EJ: Assessing the impact of continuous quality improvement/total quality management: concept versus implementation. Health Serv Res. 1995, 30: 377-401.PubMedPubMed CentralGoogle Scholar
- Green LW, Kreuter MW, Deeds SG, Partridge KB: Health Education Planning: A Diagnostic Approach. 1980, Mountain View, CA: Mayfield Publishing Co., 1Google Scholar
- Greenhalgh T, Robert G, Macfarlane F, Bate P, Kyriakidou O: Diffusion of innovations in service organizations: systematic review and recommendations. Milbank Q. 2004, 82: 581-629. 10.1111/j.0887-378X.2004.00325.x.View ArticlePubMedPubMed CentralGoogle Scholar
- Kilbourne AM, Neumann MS, Pincus HA, Bauer MS, Stall R: Implementing evidence-based interventions in health care: application of the replicating effective programs framework. Implement Sci. 2007, 2: 42. 10.1186/1748-5908-2-42.View ArticlePubMedPubMed CentralGoogle Scholar
- Neumann MS, Sogolow ED: Replicating effective programs: HIV/AIDS prevention technology transfer. AIDS Educ Prev. 2000, 12: 35-48.PubMedGoogle Scholar
- Stetler CB: Role of the organization in translating research into evidence-based practice. Outcomes Manag. 2003, 7: 97-103. quiz 104–105PubMedGoogle Scholar
- Waxmonsky J, Kilbourne AM, Goodrich DE, Nord KM, Lai Z, Laird C, Clogston J, Kim HM, Miller C, Bauer MS: Enhanced fidelity to treatment for bipolar disorder: results from a randomized controlled implementation trial. Psychiatr Serv. 2013, 64: In pressGoogle Scholar
- Kirchner JE, Ritchie M, Dollar K, Gundlach P, Smith J: Using external and internal facilitation to improve care in the Veterans Healthcare Administration. 2012, Washington, DC: VA Mental Health Quality Enhancement Research Initiative (QUERI), Department of Veteran Affairs, 1Google Scholar
- Proctor EK, Landsverk J, Aarons G, Chambers D, Glisson C, Mittman B: Implementation research in mental health services: an emerging science with conceptual, methodological, and training challenges. Adm Policy Ment Health Ment Health Serv Res. 2009, 36: 24-34. 10.1007/s10488-008-0197-4.View ArticleGoogle Scholar
- Brown AH, Cohen AN, Chinman MJ, Kessler C, Young AS: EQUIP: implementing chronic care principles and applying formative evaluation methods to improve care for schizophrenia: QUERI Series. Implement Sci. 2008, 3: 9. 10.1186/1748-5908-3-9.View ArticlePubMedPubMed CentralGoogle Scholar
- Rothwell PM: External validity of randomised controlled trials: ‘to whom do the results of this trial apply?’. Lancet. 2005, 365: 82-93. 10.1016/S0140-6736(04)17670-8.View ArticlePubMedGoogle Scholar
- Mendel P, Meredith LS, Schoenbaum M, Sherbourne CD, Wells KB: Interventions in organizational and community context: a framework for building evidence on dissemination and implementation in health services research. Adm Policy Ment Health Ment Health Serv Res. 2008, 35: 21-37. 10.1007/s10488-007-0144-9.View ArticleGoogle Scholar
- Parker LE, Kirchner JE, Bonner LM, Fickel JJ, Ritchie MJ, Simons CE, Yano EM: Creating a quality-improvement dialogue: utilizing knowledge from frontline staff, managers, and experts to foster health care quality improvement. Qual Health Res. 2009, 19: 229-242.View ArticlePubMedGoogle Scholar
- Almirall D, Compton SN, Gunlicks-Stoessel M, Duan N, Murphy SA: Designing a pilot sequential multiple assignment randomized trial for developing an adaptive treatment strategy. Stat Med. 2012, 31: 1887-1902. 10.1002/sim.4512.View ArticlePubMedPubMed CentralGoogle Scholar
- Almirall D, McCaffrey DF, Ramchand R, Murphy SA: Subgroups Analysis when Treatment and Moderators are Time-varying. Prev Sci. 2013, 14: 169-179. 10.1007/s11121-011-0208-7.View ArticlePubMedPubMed CentralGoogle Scholar
- Almirall D, Ten Have T, Murphy SA: Structural nested mean models for assessing time-varying effect moderation. Biometrics. 2010, 66: 131-139. 10.1111/j.1541-0420.2009.01238.x.View ArticlePubMedGoogle Scholar
- Murphy SA, Collins LM, Rush AJ: Customizing treatment to the patient: adaptive treatment strategies. Drug Alcohol Depend. 2007, 88 (Suppl 2): S1-S3.View ArticlePubMedPubMed CentralGoogle Scholar
- Kelly JA, Somlai AM, DiFranceisco WJ, Otto-Salaj LL, McAuliffe TL, Hackl KL, Heckman TG, Holtgrave DR, Rompa D: Bridging the gap between the science and service of HIV prevention: transferring effective research-based HIV prevention interventions to community AIDS service providers. Am J Public Health. 2000, 90: 1082-1088.View ArticlePubMedPubMed CentralGoogle Scholar
- Kilbourne AM, Brar JS, Drayer RA, Xu X, Post EP: Cardiovascular disease and metabolic risk factors in male patients with schizophrenia, schizoaffective disorder, and bipolar disorder. Psychosomatics. 2007, 48: 412-417. 10.1176/appi.psy.48.5.412.View ArticlePubMedGoogle Scholar
- Kilbourne AM, Post EP, Bauer MS, Zeber JE, Copeland LA, Good CB, Pincus HA: Therapeutic drug and cardiovascular disease risk monitoring in patients with bipolar disorder. J Affect Disord. 2007, 102: 145-151. 10.1016/j.jad.2007.01.006.View ArticlePubMedGoogle Scholar
- Chwastiak L, Rosenheck R, Leslie D: Impact of medical comorbidity on the quality of schizophrenia pharmacotherapy in a national VA sample. Med Care. 2006, 44: 55-61. 10.1097/01.mlr.0000188993.25131.48.View ArticlePubMedGoogle Scholar
- Copeland LA, Zeber JE, Wang CP, Parchman ML, Lawrence VA, Valenstein M, Miller AL: Patterns of primary care and mortality among patients with schizophrenia or diabetes: a cluster analysis approach to the retrospective study of healthcare utilization. BMC Health Serv Res. 2009, 9: 127. 10.1186/1472-6963-9-127.View ArticlePubMedPubMed CentralGoogle Scholar
- Veterans Health Administration: Office of the Medical Inspector final report: Quality improvement assessment: outreach services to schizophrenic and bipolar patients lost to follow-up care (2010-D-252). 2010, Washington, DC: Department of Veteran AffairsGoogle Scholar
- Veterans Health Administration: VHA Directive 2012–002: Re-Engaging Veterans with Serious Mental Illness in Treatment. 2012, Washington, DC: Department of Veterans AffairsGoogle Scholar
- Methodological Considerations in Generating Provider Performance Score: A Guide for Community Quality Collaboratives: White Paper.http://www.ahrq.gov/professionals/quality-patient-safety/quality-resources/value/perfscoresmethods/index.html,
- Blow FC, Mccarthy JF, Valenstein M, Bowersox NW, Visnic S: Care for Veterans with Psychosis in the Veterans Health Administration, FY10: 12th Annual National Psychosis Registry Report. 2011, Ann Arbor, MI: VA Serious Mental Illness Treatment Resource and Evaluation Center (SMITREC)Google Scholar
- O'Toole TP, Conde-Martel A, Gibbon JL, Hanusa BH, Fine MJ: Health care of homeless veterans. J Gen Intern Med. 2003, 18: 929-933. 10.1046/j.1525-1497.2003.21209.x.View ArticlePubMedPubMed CentralGoogle Scholar
- Perl L: Veterans and homelessness. 2009, Washington, DC: Congressional Research ServiceGoogle Scholar
- Veterans Health Administration (VHA): VHA Handbook 1160.01: Uniform Mental Health Services in VA Medical Centers and Clinics. 2008, Washington, DC: Department of Veteran AffairsGoogle Scholar
- Arnold N, Sohn MW, Maynard C, Hynes DM: VIReC Technical Report 2: VA-NDI Mortality Data Merge Project. 2006, Hines, IL: VA Information Resource Center, Department of Veterans AffairsGoogle Scholar
- Bandura A: Social Learning Theory. 1977, Englewood Cliffs, NJ: Prentice HallGoogle Scholar
- Rogers E: Diffusion of Innovations. 2003, New York, NY: Free Press, 5Google Scholar
- Kelly JA, Heckman TG, Stevenson LY, Williams PN, Ertl T, Hays RB, Leonard NR, O'Donnell L, Terry MA, Sogolow ED, Neumann MS: Transfer of research-based HIV prevention interventions to community service providers: fidelity and adaptation. AIDS Educ Prev. 2000, 12: 87-98.PubMedGoogle Scholar
- Kitson A, Harvey G, McCormack B: Enabling the implementation of evidence based practice: a conceptual framework. Qual Health Care. 1998, 7: 149-158. 10.1136/qshc.7.3.149.View ArticlePubMedPubMed CentralGoogle Scholar
- Nutting PA, Crabtree BF, Stewart EE, Miller WL, Palmer RF, Stange KC, Jaen CR: Effect of facilitation on practice outcomes in the National Demonstration Project model of the patient-centered medical home. Ann Fam Med. 2010, 8 (Suppl 1): S33-S44. S92View ArticlePubMedPubMed CentralGoogle Scholar
- Kitson AL, Rycroft-Malone J, Harvey G, McCormack B, Seers K, Titchen A: Evaluating the successful implementation of evidence into practice using the PARiHS framework: theoretical and practical challenges. Implement Sci. 2008, 3: 1. 10.1186/1748-5908-3-1.View ArticlePubMedPubMed CentralGoogle Scholar
- Rycroft-Malone J: The PARIHS framework–a framework for guiding the implementation of evidence-based practice. J Nurs Care Qual. 2004, 19: 297-304. 10.1097/00001786-200410000-00002.View ArticlePubMedGoogle Scholar
- Stetler CB, Damschroder LJ, Helfrich CD, Hagedorn HJ: A Guide for applying a revised version of the PARIHS framework for implementation. Implement Sci. 2011, 6: 99. 10.1186/1748-5908-6-99.View ArticlePubMedPubMed CentralGoogle Scholar
- Kilbourne AM, Neumann MS, Waxmonsky J, Bauer MS, Kim HM, Pincus HA, Thomas M: Public-academic partnerships: evidence-based implementation: the role of sustained community-based practice and research partnerships. Psychiatr Serv. 2012, 63: 205-207. 10.1176/appi.ps.201200032.View ArticlePubMedGoogle Scholar
- Zmud RW: An examination of push-pull theory applied to process innovation in knowledge work. Manag Sci. 1984, 30: 727-738. 10.1287/mnsc.30.6.727.View ArticleGoogle Scholar
- Stetler CB, Legro MW, Rycroft-Malone J, Bowman C, Curran G, Guihan M, Hagedorn H, Pineros S, Wallace CM: Role of ‘external facilitation’ in implementation of research findings: a qualitative evaluation of facilitation experiences in the Veterans Health Administration. Implement Sci. 2006, 1: 23. 10.1186/1748-5908-1-23.View ArticlePubMedPubMed CentralGoogle Scholar
- McGlynn EA, Asch SM, Adams J, Keesey J, Hicks J, DeCristofaro A, Kerr EA: The quality of health care delivered to adults in the United States. N Engl J Med. 2003, 348: 2635-2645. 10.1056/NEJMsa022615.View ArticlePubMedGoogle Scholar
- Galvin RS, McGlynn EA: Using performance measurement to drive improvement: a road map for change. Med Care. 2003, 41: I48-I60.View ArticlePubMedGoogle Scholar
- Damschroder LJ, Goodrich DE, Robinson CH, Fletcher CE, Lowery JC: A systematic exploration of differences in contextual factors related to implementing the MOVE! weight management program in VA: a mixed methods study. BMC Health Serv Res. 2011, 11: 248. 10.1186/1472-6963-11-248.View ArticlePubMedGoogle Scholar
- Kilbourne AM, Greenwald DE, Hermann RC, Charns MP, McCarthy JF, Yano EM: Financial incentives and accountability for integrated medical care in Department of Veterans Affairs mental health programs. Psychiatr Serv. 2010, 61: 38-44. 10.1176/appi.ps.61.1.38.View ArticlePubMedGoogle Scholar
- Chaudoir SR, Dugan AG, Barr CH: Measuring factors affecting implementation of health innovations: a systematic review of structural, organizational, provider, patient, and innovation level measures. Implement Sci. 2013, 8: 22. 10.1186/1748-5908-8-22.View ArticlePubMedPubMed CentralGoogle Scholar
This article is published under license to BioMed Central Ltd. This is an open access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.