- Open Access
Using Getting To Outcomes to facilitate the use of an evidence-based practice in VA homeless programs: a cluster-randomized trial of an implementation support strategy
Implementation Science volume 12, Article number: 34 (2017)
Incorporating evidence-based integrated treatment for dual disorders into typical care settings has been challenging, especially among those serving Veterans who are homeless. This paper presents an evaluation of an effort to incorporate an evidence-based, dual disorder treatment called Maintaining Independence and Sobriety Through Systems Integration, Outreach, and Networking—Veterans Edition (MISSION-Vet) into case management teams serving Veterans who are homeless, using an implementation strategy called Getting To Outcomes (GTO).
This Hybrid Type III, cluster-randomized controlled trial assessed the impact of GTO over and above MISSION-Vet Implementation as Usual (IU). Both conditions received standard MISSION-Vet training and manuals. The GTO group received an implementation manual, training, technical assistance, and data feedback. The study occurred in teams at three large VA Medical Centers over 2 years. Within each team, existing sub-teams (case managers and Veterans they serve) were the clusters randomly assigned. The trial assessed MISSION-Vet services delivered and collected via administrative data and implementation barriers and facilitators, via semi-structured interview.
No case managers in the IU group initiated MISSION-Vet while 68% in the GTO group did. Seven percent of Veterans with case managers in the GTO group received at least one MISSION-Vet session. Most case managers appreciated the MISSION-Vet materials and felt the GTO planning meetings supported using MISSION-Vet. Case manager interviews also showed that MISSION-Vet could be confusing; there was little involvement from leadership after their initial agreement to participate; the data feedback system had a number of difficulties; and case managers did not have the resources to implement all aspects of MISSION-Vet.
This project shows that GTO-like support can help launch new practices but that multiple implementation facilitators are needed for successful execution of a complex evidence-based program like MISSION-Vet.
Widespread evidence supports integrating behavioral health and substance abuse treatments for those who are dually diagnosed with both mental illnesses and substance abuse disorders . However, several large-scale initiatives have demonstrated that it is challenging to provide integrated care in typical treatment settings for those who are homeless [2–5]. For example, a multi-site study that integrated dual disorders treatment and Housing First found that while overall scores for Housing First fidelity were high, scores of the item specifically addressing “Integrated, Stage-wise Substance Use Treatment” were rated 3 or lower on a 5-point scale . In the Department of Veterans Affairs (VA), up to 80% of the approximately 48,000 Veterans  who are homeless suffer from mental health and/or substance use disorders, threatening their housing stability and leading to higher rates of relapse, treatment dropout, poor community integration, and utilization of costly emergency and inpatient services . This paper presents a cluster-randomized trial of a specific implementation strategy aimed at supporting the use of a complex integrated dual disorders model for Veterans who have experienced homelessness.
Experts in dual diagnosis research have carried out several initiatives attempting to document the degree to which integrated treatments were adopted and implemented. For example, the National Evidence-Based Practice (EBP) Implementation Project was a non-experimental effort to use in-person consultant/trainers for 2 years to help 53 public-sector community mental health agencies adopt two of five EBPs for those with serious mental illnesses (SMI), one of which was an integrated dual disorder model. Although the sites did improve their fidelity, only 18% of the sites who chose the integrated treatment model had high fidelity [2, 3]. Other large-scale initiatives that employed similar implementation strategies found similar results [4, 5]. Building upon these contributions, more research is needed to specifically test implementation strategies in settings that serve those who are not only dually diagnosed but also homeless.
An ideal setting for such research is the VA program called HUD-VASH (Housing and Urban Development—Veteran Affairs Supportive Housing). Veterans at all VA medical centers who are homeless can receive subsidized housing from HUD and VA case management. In 2012, HUD-VASH adopted the Housing First philosophy, which states that individuals do not need to demonstrate complete sobriety in order to receive housing and case management services. In addition, HUD-VASH recently accepted a significant number of new Veterans to help support the goal of ending homelessness among Veterans by 2015 , straining available resources, especially staff. Both changes increased the number of Veterans in HUD-VASH with a need for integrated treatment for both substance abuse and mental health diagnoses.
Although many evidence-based integrated treatment models are available, one in particular—Maintaining Independence and Sobriety through Systems Integration, Outreach, and Networking—Veterans Edition (MISSION-Vet)—was developed specifically for homeless or formerly homeless Veterans [8, 9]. MISSION-Vet is a manualized, integrated, co-occurring disorder treatment model grounded in the Health Belief Model . In MISSION-Vet, a Veteran, case manager, and “peer specialist” work together for about 2.5 h a week. Peer specialists are individuals who have recovered from their own mental health and substance issues and are trained to provide support to others with the same difficulties . Wide-scale implementation of MISSION-Vet in HUD-VASH has not occurred, despite a strong evidence base, the fact that both share a Housing First treatment philosophy , and free, web-based training and manuals. While implementation models such as the Consolidated Framework for Implementation Research (CFIR)  show that research efficacy, concrete tools, and compatibility with host sites are important factors that facilitate implementation, these factors alone are usually not sufficient to promote adoption by settings like HUD-VASH . Because systems frequently do not adopt new practices even when they are known to improve outcomes , innovative strategies are needed at both individual and organizational levels to encourage adoption.
To facilitate the adoption of MISSION-Vet within HUD-VASH, we employed and then tested the Getting To Outcomes (GTO) approach , modified for use within the VA . GTO is both an implementation model—specifying steps practitioners should take when carrying out an EBP—and an implementation strategy, providing ongoing implementation training, technical assistance, and data feedback to improve practitioners’ capacity to complete those steps . GTO has been found to improve the capacity of individual drug and teen pregnancy prevention practitioners and the performance and fidelity of prevention programs in both quasi-experimental  and randomized controlled trials [19–22]. However, most of those studies involved non-evidence-based programs. This study is the first to empirically test a version of GTO in the VA with a rigorous design and also the first instance GTO was used to support an EBP in a clinical setting. We received funding from the VA Quality Enhancement Research Initiative to compare MISSION-Vet implementation with and without GTO support in three large VA HUD-VASH sites.
This study was a Hybrid Type III, cluster-randomized controlled trial assessing the impact of GTO over and above MISSION-Vet Implementation as Usual (IU). Both conditions received standard MISSION-Vet training and access to the MISSION-Vet treatment manuals, but only the GTO group received implementation support. The study was conducted in HUD-VASH teams at three large VA Medical Centers over a 2-year period (March 2013 to August 2015). Within each HUD-VASH team, existing sub-teams (case managers and Veterans they serve) were the clusters randomly assigned, which is particularly appropriate when desiring to lessen the risk of experimental contamination of the implementation strategy . With support from study staff and technical assistance, each of the GTO groups was able to fund two peer specialists. Necessitated by the scope of the study, this structure represents an adaptation to the MISSION-Vet model, where typically equal numbers of peer specialists and case managers are used to form treatment dyads.
Although Hybrid Type III trials do collect clinical outcome data, they primarily focus on the “utility of an implementation strategy” and measure “adoption of the clinical treatment and fidelity to it, as well as related factors (pg. 220)” . The primary focus of this paper was to evaluate how much the GTO implementation strategy facilitated HUD-VASH’s adoption and implementation of MISSION-Vet, operationalized as MISSION-Vet services delivered. These services were analyzed according to three domains of the implementation heuristic, RE-AIM  (Adoption, Reach, Implementation; see below). Blinding data collection was not possible; however, MISSION-Vet service data was from secondary sources, described below. We hypothesized that the GTO group would have greater Adoption, Reach, and Implementation compared to the IU group overall and at each site. To better understand the context for implementation, we interviewed HUD-VASH stakeholders about MISSION-Vet barriers and facilitators based on CFIR. As a Hybrid Type III study, a secondary focus was on the mental health and substance abuse outcomes of the participating Veterans. These were documented by case managers via the VA’s existing HUD-VASH data monitoring system and will be reported in a separate paper.
The VA Central IRB that oversees multi-site trials granted approval for this study in October of 2011. The study received a waiver of consent to use Veteran-level data from the HUD-VASH data management system. Harms of GTO were monitored by study staff during the time span GTO was active. None were reported.
Participation sites and recruitment
At the time of the study start, the size of the three HUD-VASH teams were: team 1 (450 Veterans receiving HUD-VASH housing vouchers and 18 case managers), team 2 (850 Veterans receiving HUD-VASH housing vouchers and 27 case managers), and team 3 (810 Veterans receiving HUD-VASH housing vouchers and 24 case managers), for a total of 2110 Veterans receiving HUD-VASH vouchers and 69 case managers. A housing voucher is a subsidy a Veteran receives from HUD that they can use to offset a portion of their rent. Each team is made of two sub-teams of case managers, which were randomized to IU or GTO by the team statistician using a random number generator. The PIs (DS, MC) informed each sub-team of their randomization status. Randomizing within team holds constant the variation due to team-level characteristics, funding streams, regulations, data collection activities, and political climates. The three HUD-VASH teams were selected based on their willingness to participate and their similarity. For example, 85–95% of each team’s vouchers were in active use (Veteran living in an apartment). Data compiled by the VA’s Northeast Program Evaluation Center shows that, on all three teams, all Veterans get at least one contact per month, the HUD-VASH minimum. Approximately 90% of the Veterans on all three HUD-VASH teams have substance use and/or mental health diagnoses according to reports of the VA Homeless Network Coordinators at those sites. Additionally, each of these teams report that approximately 75% of their Veterans receive other services at the VA or in the community beyond standard HUD-VASH case management services.
HUD-VASH case manager and Veteran recruitment
The research subjects were the HUD-VASH case managers and peer specialists, but the case managers also recorded data about the Veterans on their caseloads in the VA’s Computerized Patient Record System (CPRS). All case managers were invited to participate and verbally consented before randomization. Ten refused to participate (n = 3, 3, and 4 across teams 1–3, respectively); leaving 22 in the IU group and 37 in the GTO group (see Table 3 for team and study condition level sample sizes). A case manager and their total case load of Veterans were assigned to study condition based on the case managers’ team assignment to GTO or IU. All HUD-VASH case managers in both groups were invited to the webinar training on MISSION-Vet. After MISSION-Vet began, the case managers and peer specialists had the opportunity to deliver MISSION-Vet services for up to 1 year to any Veteran who met the following: (1) had a current substance abuse or dependence disorder and a co-occurring mental illness and (2) was willing to receive MISSION-Vet services.
The evidence-based program: MISSION-Vet
The core service components of MISSION-Vet are critical time intervention (CTI), weekly sessions delivered by a case manager using dual recovery therapy (DRT), and structured and unstructured peer specialist services. The MISSION-Vet Treatment Manual provides guidance to case managers/peer specialists, and the MISSION-Vet Consumer Workbook provides homework assignments, readings, and checklists for Veterans. In several studies, including a randomized matched attention control trial, MISSION-Vet has demonstrated efficacy in increasing treatment engagement, improving mental health and substance abuse outcomes, and reducing ER visits, re-hospitalizations, and recurring homelessness [26–29]. More details about MISSION-Vet are available in this project’s protocol paper published in Implementation Science .
GTO implementation model and strategy
Getting To Outcomes (GTO) is a strategy that builds capacity for implementing EBPs by strengthening the knowledge, attitudes, and skills needed to choose, plan, implement, evaluate, and sustain EBPs. GTO helps practitioners address 10 key steps (Table 1) needed to obtain positive results: steps 1–6 for planning EBPs, steps 7–8 for process and outcome evaluation, and steps 9–10 on the use of data to improve and sustain programming.
Implementation of these 10 steps is facilitated by three types of assistance: the GTO manual of text and tools published by the RAND Corporation  which was specifically tailored to VA homeless services , face-to-face training, and ongoing technical assistance (TA). The goal is to help practitioners and leadership integrate the practices GTO specifies into routine operations and work collaboratively to tailor the EBP to local conditions. Consistent with social cognitive theories of behavioral change [33, 34] and implementation science theories such as CFIR, the GTO supports enhance knowledge about GTO-related activities, which improves attitudes towards these activities, which in turn, improves execution of GTO-specified tasks, which supports the strong implementation of EBP needed for achieving outcomes . For details about how GTO operationalizes CFIR, see Acosta et al.  and Smelson et al. .
Use of GTO to facilitate MISSION-Vet in HUD-VASH
Each GTO sub-team created a “GTO Planning Team” of HUD-VASH staff (led by a designated point of contact) and the GTO TA staffperson (SM, in Pittsburgh) who lead the sub-team through the GTO process. The key GTO supports were as follows:
Manual of tools—each team received the manual Getting To Outcomes in services for homeless Veterans: 10 steps for achieving accountability , developed from a pilot project at the Pittsburgh VA Homelessness Center . Like all GTO manuals, it provides guidance about how to complete several “tools” or worksheets that prompt practitioners to make, and then record, decisions about various tasks prescribed by the GTO 10 step model.
Training—study staff (DS, MC, SM) conducted an in-person, 6-h training with each sub-team on how to use GTO to plan, implement, evaluate, and conduct quality improvement on MISSION-Vet. The MISSION-Vet webinar was conducted by DS a few weeks after the GTO training.
GTO technical assistance (TA)—GTO TA is similar to “facilitation” in the implementation science literature . Like facilitation, GTO TA emphasizes change in work practices through encouragement and action promotion via regular, ongoing meetings [37, 38]. Typically, GTO TA providers guide practitioners to use GTO-based tools to implement an EBP. After feedback from HUD-VASH staff indicated that completing the tools was overly time-consuming, an adaptation was made so that most of the activities that GTO prescribes (and that the tools address) were completed informally during team meetings with the GTO TA staffperson. However, all teams did complete the GTO Goals Tool, which facilitates setting service delivery benchmarks. The TA staffperson met by phone, bi-weekly for 18–23 months with each GTO Planning Team. HUD-VASH team leaders were invited, and those from teams 1 and 2 participated regularly; the team 3 leader participated rarely. As is typical in GTO projects [18, 21, 39], the TA staffperson received implementation supervision via weekly meetings with experts in the EBP (MISSION-Vet, DS) and GTO (MC), respectively. Initial meetings focused on setting goals and performance targets, tailoring MISSION-Vet for HUD-VASH teams, identifying any gaps in skills required by MISSION-Vet, and arranging additional training. Later meetings involved reviewing performance data and troubleshooting implementation. The TA staffperson (SM) visited team 1 twice, team 3 four times, and team 2 only once (for initial training) due to VA travel restrictions. The hours of TA for each team, as recorded in log by the TA staffperson, were 59, 34, and 35 h for teams 1, 2, and 3, respectively. While teams started and ended at slightly different times, they averaged about 1.5 h of technical assistance per month, or about two meetings of 45 min.
MISSION-Vet service tracking—MISSION-Vet service data was collected with a CPRS note template we developed for each team. Data from the notes were extracted to create feedback reports for each sub-team that were discussed in GTO Planning Team meetings approximately once a month.
Measures and data collection
To document GTO’s impact on MISSION-Vet’s Adoption, Reach, and Implementation, we extracted from the CPRS note templates completed by both groups the following MISSION-Vet services: which DRT sessions, peer specialist sessions, and Consumer Workbook exercises were completed; whether the MISSION-Vet Consumer workbook was provided; whether community activities were done with a Veteran (e.g., taken to appointment, NA/AA meetings, recreational events, meetings with landlords, or other activities); and referrals made to other services. This data was only collected about case managers (and their assigned Veterans) who were consented into the study.
In addition to MISSION-Vet service delivery, we invited all case managers and peer specialists from the GTO group to participate in semi-structured interviews. The response rates for teams 1, 2, and 3 were 83% (n = 5), 43% (n = 7), 55% (n = 10), respectively. Eighteen case managers, six supervisory staff, and one peer specialist were interviewed. The interview protocol was structured around four of the five CFIR domains: Intervention Characteristics, Implementation Process, Inner Setting, and Outer Setting; and the more granular implementation factors under each domain. Similar to Damschroder and Lowery’s use of CFIR to assess context of an implementation , we did not collect individual level information appropriate for the Individual Characteristics domain. Questions were drawn from the generic CFIR protocol and then tailored to this project. All interviews were conducted by a doctoral-level researcher who worked on the project (GH) but was not involved in the delivery of the GTO assistance. Interviews involved asking open-ended questions and follow-up probes about how each domain (and implementation factors under each domain) was expressed on the team during MISSION-Vet implementation. All interviews were digitally recorded and transcribed verbatim.
MISSION-Vet services by RE-AIM factors
RE-AIM states that impact of an intervention should be judged by the following factors (http://re-aim.org/): Reach—the proportion of individuals (in this case, Veterans) who participated in the intervention; Effectiveness—the impact of the program on important outcomes; Adoption—the absolute number or proportion of clinicians or teams that initiated the intervention; Implementation—the extent to which the interventions’ components were carried out with fidelity; and Maintenance—the extent to which a program becomes institutionalized or part of the routine organizational practices and policies. In this report, MISSION-Vet services of both groups were analyzed to assess how much GTO impacted Adoption, Reach, and Implementation—i.e., domains earlier in the cycle of EBP delivery. Table 2 shows how these domains were operationalized.
For each team, implementation data was collected from the day on which the first MISSION-Vet note was entered until the last day of the month in which the team was formally engaged in the study: team 1 (6/11/13–7/31/14), team 2 (12/31/13–8/31/15), and team 3 (4/19/13–12/31/14). These dates varied due to differences in how long each team participated in the study. For the variables of Adoption and Reach, Received any MISSION-Vet sessions were defined as at least one MISSION-Vet session. All the other implementation variables are computed on those Veterans who had at least two MISSION-Vet sessions (peer specialist or case manager). This was to exclude Veterans who received the MISSION-Vet orientation session but then chose not to receive MISSION-Vet. Calculations of Reach included all Veterans with a consented case manager. This is a conservative measure of Reach because not all of the Veterans with consented case managers had a dual diagnosis and, thus, were not appropriate candidates for MISSION-Vet. Although data on substance abuse and mental illness were available in CPRS, discussions with case managers suggested that this data was often inaccurate, so all Veterans were included in the denominator to calculate Reach. We used Fisher’s exact test to compare the GTO and Implementation as Usual groups with respect to the Adoption and Reach outcomes. We conducted separate comparisons within each team and across all teams.
Ratings of CFIR implementation factors
Based on methods used by Damschroder and Lowery , text data from the CFIR-based interviews were used to make ratings for the individual implementation factors under the four domains included. The inclusion and exclusion criteria from the CFIR Codebook Template  were used to associate text from the transcripts with CFIR factors. Text could be associated with multiple factors or no factor. One analyst (SM) coded the text initially and a second research associate (GH) reviewed the coding for accuracy. Any discrepancies in the coded text were discussed until consensus was reached. Two analysts then used the text associated with each CFIR factor to rate each factor on the extent to which it impeded or facilitated implementation of MISSION-Vet. Again, discrepancies in the ratings were discussed until consensus was reached. Factors for which there was insufficient text to make a rating were dropped (leaving 20 of 31 CFIR factors). Taking into account both the valence (i.e., either facilitating or hindering) and the strength of each implementation factor, the rating scale ranges from +2 (most facilitating) to −2 (most hindering). Again, discrepancies in ratings were discussed until consensus was reached.
MISSION-Vet services by RE-AIM domains
As shown in Table 3, no case managers in the Implementation as Usual (IU) group at any study site adopted MISSION-Vet while 68% (25 of 37) of case managers in the GTO group adopted MISSION-Vet, a difference that was statistically significant. Likewise, there was a significant difference between the GTO and IU group with respect to the adoption of MISSION-Vet within teams 1 and 2, but not team 3. Teams 1 and 2 had similar rates of case manager Adoption (100, 92%), much more than team 3 (39%). Team 1, which had the highest Adoption among the GTO group (100%), also had the smallest number of consented case managers in the GTO group (n = 6). Team 3, which had the lowest Adoption rate among GTO groups (39%), had the largest number of consented case managers (n = 18).
No Veterans with case managers in the IU group (n = 567) at any study site received any MISSION-Vet sessions. Seven percent of Veterans with case managers in the GTO group (n = 996) received at least one MISSION session. Only 1.7% of Veterans with case managers in the GTO group received at least half of the available 24 structured DRT or peer specialist MISSION-Vet sessions. The differences between the GTO and IU groups with respect to the Reach outcomes were all statistically significant when collapsing across all teams, although not all within-team comparisons were significant. Again, teams 1 and 2 had similar Reach rates, double or more the rates for team 3 across multiple levels of MISSION-Vet received (any, 10%, 25%). As one would expect, the teams with higher Adoption, also had higher Reach (see Table 3). For all teams, the percent of Reach declined as the RE-AIM criterion increased (any, 10%, 25%, 50%).
Across all teams, 73% of Veterans receiving MISSION-Vet were given the Consumer workbook. Veterans receiving MISSION-Vet received on average 4.2 DRT sessions, 3.3 peer specialist sessions, 1.5 self-guided exercises, and referrals to 1.5 of the categories of referral services. Teams 1 and 3 had similar patterns of case manager implementation (see Table 4). Team 2 provided less case manager services to MISSION-Vet Veterans but more peer specialist services. No team’s Veterans received more than half the sessions of any type called for by MISSION-Vet. Team 1’s Veterans received about a third of the DRT sessions, and a quarter of the self-guided exercises and peer specialist sessions. Team 2’s Veterans received a fifth of the DRT sessions, five percent of the self-guided exercises, and a third of the peer specialist sessions. Team 3’s Veterans received the most: nearly half of the DRT sessions, about a third of the self-guided exercises, and about a quarter of the peer specialist sessions.
Ratings of CFIR implementation factors
In this study, the domain of Intervention Characteristics addresses characteristics of the MISSION-Vet model (see Table 5). All teams perceived that the decision to use the MISSION-Vet model was reached externally to HUD-VASH (Intervention source), which is typically an implementation barrier. Teams 1 and 3 found MISSION-Vet to be complicated to implement while team 2 was neutral on Complexity. Across all teams, MISSION-Vet was perceived as highly adaptable, such that case managers could implement different aspects of MISSION-Vet in different ways for different Veterans, and this was seen as a major facilitator because of the flexibility it afforded. Further, all teams rated Relative Advantage as a facilitator, as case managers reported that MISSION-Vet offered concrete, useful ideas for how to treat those with dual diagnoses that were good additions to their current strategies.
The CFIR Inner Setting domain refers to the setting where the intervention took place, in this study, the HUD-VASH teams. At this level, the low relative priority given to implementing MISSION-Vet by the teams’ leadership, in comparison to the high pressure to house Veterans quickly, was seen as a major barrier to implementation across all three study teams. Further, many case managers found MISSION-Vet was incompatible with their work on the HUD-VASH team, as urgent pressure to house Veterans, and large caseload sizes left case managers with little time for the intensity of providing MISSION-Vet services. Case managers reported no positive influence from Incentives or Rewards by leadership for using MISSION-Vet (negative at two teams, neutral at another) or by any feedback provided about use of MISSION-Vet (neutral at two teams, negative at another). Team communications were also challenging at two of the three sites, especially so for the one site that hired peers specialists via contract (not as VA employees), which added an additional level of communication complexity.
There was only sufficient data available to provide ratings for two constructs in the Outer Setting domain. Understanding of patient needs and resources across all three teams were a positive factor in implementation. However, staff at team 2 made repeated statements indicating their belief that the MISSION-Vet intervention was inappropriate for the population they served because their Veterans were struggling to meet basic needs (housing, food, and clothing) and were not ready to address issues related to their dual diagnosis. This perception was seen as negatively affecting the implementation of MISSION-Vet on this team.
In the Process domain, planning for implementation—operationalized by the GTO implementation strategy—was seen as a strength—as GTO helped organize the teams, kept them on track, and provided a forum to troubleshoot implementation. All teams reported significant challenges engaging Veterans into MISSION-Vet treatment, which is reflected in the negative rating on the Engaging Veterans factor.
We presented data assessing the impact of an implementation strategy (GTO) on the use of a clinical treatment (MISSION-Vet) in three domains specified by the implementation heuristic, RE-AIM (Adoption, Reach, Implementation). To provide additional context, we also assessed various factors that could hinder or facilitate implementation according to CFIR. As hypothesized, the GTO group had greater rates of Adoption, Reach, and Implementation than the IU group. Regarding Adoption, most case managers assigned to GTO attempted MISSION-Vet, while no case managers in the IU group did. Given that case managers in the IU group had zero Adoption, they also had zero Reach and Implementation. These findings are consistent with a great deal of research that shows that passive approaches to implementation do not usually result in uptake of new practices . Although case managers in the GTO condition attempted to deliver MISSION-Vet more than IU case managers, they only implemented MISSION-Vet with a small number of Veterans (Reach) and delivered a small amount of MISSION-Vet services compared to what the EBP typically requires (Implementation). The findings regarding the low level of implementation are different from past GTO studies  but similar to past studies attempting to facilitate the delivery of integrated dual diagnosis treatment, as described above [2–5]. Compared to MISSION-Vet, however, the EBP in GTO studies have been much less complicated to implement—e.g., an eight-session teen pregnancy prevention program . As we show below, the CFIR ratings highlight the numerous challenges on the teams that made the use of MISSION-Vet more difficult and some factors which aided implementation.
Although most case managers appreciated the guidance and materials of MISSION-Vet, some case managers found the treatment complicated and confusing. In particular, some stated that it was challenging to know when and how to integrate the various components of MISSION-Vet into the treatment for each Veteran. MISSION-Vet was adapted quite often, which facilitated case managers trying it out, but appeared to have made it “acceptable” to not implement it in full.
There were several factors associated with the HUD-VASH setting that made MISSION-Vet implementation more challenging. First, although approval for introducing MISSION-Vet was secured from both medical center and HUD-VASH levels, there was little leadership involvement from either level once the project began. We primarily worked with frontline staff to implement MISSION-Vet. Without higher level leadership buy-in, there was little accountability when implementation lagged or reward for those who did implement. Through GTO, service delivery goals were established, but leadership did not track performance compared to those goals. The teams recognized that many Veterans needed more comprehensive dual diagnosis treatment; however, there were mixed feelings about whether it was reasonable for case managers to deliver this treatment given constant pressure to meet goals of housing Veterans within specified time frames. MISSION-Vet is a comprehensive treatment that requires 2.5 h a week of clinical time, which was incompatible with the increasingly large case load size of the HUD-VASH case managers.
Another significant challenge was that the data feedback system established through the GTO strategy experienced a number of difficulties which undercut its effectiveness and impact. First, case managers and peer specialists did not always use the note template, and it is likely that more services were delivered than recorded for many of those staff. Second, the VA data infrastructure made capturing the data extremely difficult and required searching, accessing, and merging records across multiple systems. Although the data was checked for errors, it is likely that some data errors remained. Third, because the VA system did not have an easy mechanism to capture the MISSION-Vet service data, there were often discrepancies between what case managers reported they were delivering and what the data reports stated. All together, these circumstances made the resulting feedback reports less accurate and thus the case managers were mixed on the utility of the feedback reports.
Serving patients was a priority for HUD-VASH case managers, and that facilitated trying MISSION-Vet according to the CFIR ratings. Although not officially rated, two other factors in the CFIR Outer Setting domain, Peer Pressure (competition between teams), and the presence of external policies and incentives may have been working against implementation. That is because there was not a large-scale effort within VA to use MISSION-Vet within HUD-VASH—i.e., there was little national momentum for its implementation. Thus, there were no external policies calling for MISSION-Vet across the VA nor were there any external incentives to deploy it. In the context of President Obama’s pledge to end Veteran homelessness by 2015, the incentives were tightly aligned to securing housing.
Ongoing planning meetings of the GTO teams at each site helped keep the HUD-VASH case managers focused on trying out MISSION-Vet. The GTO technical assistance staffperson, who ran those meetings, was the external change agent. However, case managers reported great difficulty in engaging Veterans to try MISSION-Vet. As stated, MISSION-Vet is intensive, and many Veterans in HUD-VASH who have substance abuse disorders may not have been ready to engage in such an intensive treatment.
Certain limitations should be noted. First, peer specialists were not available to teams in the IU condition in two of the three sites. The activation provided by GTO facilitated peer specialists being added to the intervention (GTO) teams, but the roll out of peer specialists varied: one team used contract peer specialists which involved greater complexity and coordination, one team had significant turn over in their peer specialists during the study, and one team experienced challenges in peer specialists recording notes into the templates. In addition, the shortage of peer specialists in this study meant that the case managers in all three GTO teams had to “share” peer specialists, which is not typical for MISSION-Vet implementation. Second, we were able to interview only one of the eight peer specialists who had been involved over time at the three teams. Third, the data from the electronic medical record were limited and relied upon case managers, not trained data collectors. It was not possible to completely confirm accuracy of the data, although we were able to make some data corrections by getting feedback from the case managers (via the data feedback process). Fourth, we know that the services documented underrepresented the actual amount of services delivered. Case managers and peer specialists in all GTO sites reported delivering some services not recorded on the CPRS note template. Lastly, using all Veterans on the HUD-VASH sub-teams as the denominator for calculation of Reach likely made Reach rates appear lower and could be considered a “lower bound” for Reach. That is because not all HUD-VASH Veterans have a dual diagnosis and would be appropriate for MISSION-Vet services.
This study suggests that while GTO implementation support was critical to the launch of MISSION-Vet, this alone does not equate to service delivery with high fidelity for a complex intervention like MISSION-Vet. Despite the support for the launching of MISSION-Vet services, a constellation of implementation factors on the ground (highlighted by the CFIR model) exerted tremendous influence on the amount of MISSION-Vet delivered and consequently reduced fidelity. This project was launched right as the VA was tasked to reduce the number of Veterans who were homeless to zero and HUD-VASH had been asked to adopt a Housing First philosophy (accepting Veterans into services much earlier in their recovery). As a result, case managers were under a great deal of pressure to house more Veterans who had more significant impairments than ever before, which left little room to take on new initiatives, regardless of their utility. It is possible that through the GTO approach, more could have been done to engage leadership levels beyond the team at each site; however, in the absence of any national momentum or external policies, it was unclear what incentives to leverage.
It could be concluded that MISSION-Vet was not an optimal fit for the HUD-VASH program at the time this study was conducted. Nonetheless, while choice in service engagement is a core philosophy for homeless clients in Housing First programs, it is also important to address co-occurring metal health and substance abuse to prevent housing instability and loss, an area with which the field is still struggling to optimally deliver support . Thus, given its evidence base and philosophical overlap with HUD-VASH, it is possible that under different circumstances, MISSION-Vet could be successful. Since the end of the trial, the national HUD-VASH program has funded additional training for Critical Time Intervention (a key component of MISSION-Vet), multidisciplinary teams, and shared caseloads as a way to improve access to a variety of services including integrated dual disorders treatment. As the study ended, a GTO team at one of the sites was in the process of creating a special intensive case management sub-team in which case managers with very low caseloads would use MISSION-Vet to treat high-acuity patients. This idea appears to be more favorable as it has high leadership support, has high priority, and would be more compatible with the new case load configuration—CFIR factors that were implementation barriers during the study. Overall, this project shows that GTO-like support can help launch new practices but that multiple implementation facilitators are needed for successful implementation of an evidence-based program like MISSION-Vet.
Computerized Patient Record System
Getting To Outcomes
Housing and Urban Development—Veteran Affairs Supportive Housing
Maintaining Independence and Sobriety through Systems Integration, Outreach, and Networking—Veterans Edition
Drake RE, Mueser KT, Brunette MF, McHugo GJ. A review of treatments for people with severe mental illnesses and co-occurring substance use disorders. Psychiatr Rehabil J. 2004;27(4):360–74. doi:10.2975/27.2004.360.374.
Rapp CA, Etzel-Wise D, Marty D, Coffman M, Carlson L, Asher D, et al. Barriers to evidence-based practice implementation: results of a qualitative study. Community Ment Health J. 2010;46(2):112–8. doi:10.1007/s10597-009-9238-z.
Rapp CA, Etzel-Wise D, Marty D, Coffman M, Carlson L, Asher D, et al. Evidence-based practice implementation strategies: results of a qualitative study. Community Ment Health J. 2008;44(3):213–24. doi:10.1007/s10597-007-9109-4. discussion 25-6.
Gustafson DH, Quanbeck AR, Robinson JM, Ford 2nd JH, Pulvermacher A, French MT, et al. Which elements of improvement collaboratives are most effective? A cluster-randomized trial. Addiction. 2013;108(6):1145–57. doi:10.1111/add.12117.
Nelson G, Stefancic A, Rae J, Townley G, Tsemberis S, Macnaughton E, et al. Early implementation evaluation of a multi-site housing first intervention for homeless people with mental illness: a mixed methods approach. Eval Program Plann. 2014;43:16–26. doi:10.1016/j.evalprogplan.2013.10.004.
Henry M, Shivji A, de Sousa T, Cohen R. The 2015 Annual Homeless Assessment Report (AHAR) to Congress. Washington: The U.S. Department of Housing and Urban Development; 2015.
Shinseki E. Secretary Shinseki details plan to end homelessness for Veterans. 2009.
Smelson D, Sawh L, Kuhn J, Kane V, Ziedonis D. The MISSION-VET treatment manual. Bedford: Department of Veterans Affairs; 2011.
Smelson D, Sawh L, Rodrigues S, Muñoz E, Marzilli A, Tripp J. In: Affairs DoV, editor. The MISSION-VET consumer workbook. Bedford: Department of Veterans Affairs; 2011.
Rosenstock IM. The health belief model and preventive health behavior. Health Educ Monogr. 1974;2(4):354–86.
Chinman M, George P, Dougherty RH, Daniels AS, Ghose SS, Swift A, et al. Peer support services for individuals with serious mental illnesses: assessing the evidence. Psychiatr Serv. 2014;65(4):429–41. doi:10.1176/appi.ps.201300244.
Kertesz SG, Crouch K, Milby JB, Cusimano RE, Schumacher JE. Housing first for homeless persons with active addiction: are we overreaching? Milbank Q. 2009;87(2):495–534. doi:10.1111/j.1468-0009.2009.00565.x.
Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci. 2009;4:50. doi:10.1186/1748-5908-4-50.
Schoenwald SK, Hoagwood K. Effectiveness, transportability, and dissemination of interventions: what matters when? Psychiatr Serv. 2001;52(9):1190–7. doi:10.1176/appi.ps.52.9.1190.
Ennett ST, Ringwalt CL, Thorne J, Rohrbach LA, Vincus A, Simons-Rudolph A, et al. A comparison of current practice in school-based substance use prevention programs with meta-analysis findings. Prev Sci. 2003;4(1):1–14.
Wandersman A, Imm P, Chinman M, Kaftarian S. Getting to outcomes: a results-based approach to accountability. Eval Program Plann. 2000;23(3):389–95. doi:10.1016/S0149-7189(00)00028-8.
Chinman M, Hannah G, McCarthy S. Lessons learned from a quality improvement intervention with homeless veteran services. J Health Care Poor Underserved. 2012;23(3 Suppl):210–24. doi:10.1353/hpu.2012.0125.
Chinman M, Hunter SB, Ebener P, Paddock SM, Stillman L, Imm P, et al. The getting to outcomes demonstration and evaluation: an illustration of the prevention support system. Am J Community Psychol. 2008;41(3-4):206–24. doi:10.1007/s10464-008-9163-2.
Acosta J, Chinman M, Ebener P, Malone PS, Paddock S, Phillips A, et al. An intervention to improve program implementation: findings from a two-year cluster randomized trial of Assets-Getting To Outcomes. Implement Sci. 2013;8:87. doi:10.1186/1748-5908-8-87.
Chinman M, Acosta J, Ebener P, Burkhart Q, Malone PS, Paddock SM, et al. Intervening with practitioners to improve the quality of prevention: one-year findings from a randomized trial of Assets-Getting To Outcomes. J Prim Prev. 2013;34(3):173–91. doi:10.1007/s10935-013-0302-7.
Chinman M, Tremain B, Imm P, Wandersman A. Strengthening prevention performance using technology: a formative evaluation of interactive Getting To Outcomes. Am J Orthopsychiatry. 2009;79(4):469–81. doi:10.1037/a0016705.
Chinman M, Acosta J, Ebener P, Malone PS, Slaughter ME. Can implementation support help community-based settings better deliver evidence-based sexual health promotion programs? A randomized trial of Getting To Outcomes(R). Implement Sci. 2016;11(1):78. doi:10.1186/s13012-016-0446-y.
Klar N, Donner A. Current and future challenges in the design and analysis of cluster randomization trials. Stat Med. 2001;20(24):3729–40. doi:10.1002/sim.1115.
Curran GM, Bauer M, Mittman B, Pyne JM, Stetler C. Effectiveness-implementation hybrid designs: combining elements of clinical effectiveness and implementation research to enhance public health impact. Med Care. 2012;50(3):217–26. doi:10.1097/MLR.0b013e3182408812.
Glasgow RE, Vogt TM, Boles SM. Evaluating the public health impact of health promotion interventions: the RE-AIM framework. Am J Public Health. 1999;89(9):1322–7. doi:10.2105/AJPH.89.9.1322.
Smelson D, Kalman D, Losonczy MF, Kline A, Sambamoorthi U, Hill LS, et al. A brief treatment engagement intervention for individuals with co-occurring mental illness and substance use disorders: results of a randomized clinical trial. Community Ment Health J. 2012;48(2):127–32. doi:10.1007/s10597-010-9346-9.
Smelson D, Losonczy M, Castles-Fonseca K, Stewart P, Kaune M, Ziedonis D. Preliminary outcomes from a booster case management program for individuals with a co-occurring substance abuse and a persistent psychiatric disorder. J Dual Diagn. 2005;3:47–59.
Smelson D, Williams J, Ziedonis D, Losonczy M, Williams J, Steinberg ML, et al. The efficacy of clonazepam for decreasing cue-elicited craving in individuals with schizophrenia and cocaine dependence: a preliminary report. J Clin Psychopharmacol. 2006;26(1):9–12.
Smelson DA, Kline A, Kuhn J, Rodrigues S, O’Connor K, Fisher W, et al. A wraparound treatment engagement intervention for homeless veterans with co-occurring disorders. Psychol Serv. 2013;10(2):161–7. doi:10.1037/a0030948.
Smelson DA, Chinman M, McCarthy S, Hannah G, Sawh L, Glickman M. A cluster randomized Hybrid Type III trial testing an implementation support strategy to facilitate the use of an evidence-based practice in VA homeless programs. Implement Sci. 2015;10:79. doi:10.1186/s13012-015-0267-4.
Chinman M, Imm P, Wandersman A. Getting to Outcomes 2004: promoting accountability through methods and tools for planning, implementation, and evaluation. Santa Monica: RAND Corporation; 2004. Contract No.: TR-TR101.
Hannah G, McCarthy S, Chinman M. Getting To Outcomes in services for homeless Veterans: 10 steps for achieving accountability. Philadelphia: National Center on Homelessness Among Veterans; 2011.
Bandura A. Health promotion by social cognitive means. Health Educ Behav. 2004;31(2):143–64. doi:10.1177/1090198104263660.
Fishbein M, Ajzen I. Attitudes towards objects as predictors of single and multiple behavioral criteria. Psychol Rev. 1974;81(1):59–74. doi:10.1037/h0035872.
Durlak JA, DuPre EP. Implementation matters: a review of research on the influence of implementation on program outcomes and the factors affecting implementation. Am J Community Psychol. 2008;41(3-4):327–50. doi:10.1007/s10464-008-9165-0.
Kirchner JE, Ritchie MJ, Pitcock JA, Parker LE, Curran GM, Fortney JC. Outcomes of a partnered facilitation strategy to implement primary care-mental health. J Gen Intern Med. 2014;29 Suppl 4:904–12. doi:10.1007/s11606-014-3027-2.
Kitson A, Harvey G, McCormack B. Enabling the implementation of evidence based practice: a conceptual framework. Qual Health Care. 1998;7(3):149–58.
Rycroft-Malone J, Harvey G, Seers K, Kitson A, McCormack B, Titchen A. An exploration of the factors that influence the implementation of evidence into practice. J Clin Nurs. 2004;13(8):913–24. doi:10.1111/j.1365-2702.2004.01007.x.
Chinman M, Acosta J, Ebener P, Burkhart Q, Clifford M, Corsello M, et al. Establishing and evaluating the key functions of an interactive systems framework using an assets-getting to outcomes intervention. Am J Community Psychol. 2012;50(3-4):295–310. doi:10.1007/s10464-012-9504-z.
Damschroder LJ, Lowery JC. Evaluation of a large-scale weight management program using the consolidated framework for implementation research (CFIR). Implement Sci. 2013;8:51. doi:10.1186/1748-5908-8-51.
Damschroder LJ. CFIR codebook template. Ann Arbor, MI. 2015. http://www.cfirguide.org/CFIRCodebookTemplate10.27.2014.docx. Accessed 1 Oct 2015.
Cividin TM, Ottoson JM. Linking reasons for continuing professional education participation with postprogram application. J Contin Educ Health Prof. 1997;17(1):46–55. doi:10.1002/chp.4750170107.
Smelson DA, Zaykowski H, Guevermont N, Siegfriedt J, Sawh L, Modzelewski D et al. Integrating permanent supportive housing and co-occurring disorders treatment for individuals who are homeless. J Dual Diagn. 2016:0. doi:10.1080/15504263.2016.1174010.
The contents of this paper are solely from the authors and do not represent the views of the US Department of Veterans Affairs or the US Government. We would like to acknowledge the contributions of Vince Kane, Jesse Vazzano, Julianne Siegfriedt, Brittany Walker, and Rachel Mullins for their assistance with this project.
All the authors are funded by a grant from the Health Services Research and Development Quality Enhancement Research Initiative, “MISSION-Vet HUD-VASH Implementation Study” (SDP 11-240).
Availability of data and supporting materials
Upon request, the authors will send interested parties an encrypted copy of the de-identified data from this study upon approval from the Department of Veterans Affairs.
DAS helped conceptualize the study and edited the manuscript, focusing on MISSION-Vet, the description of HUD-VASH, and the HOMES system. MC also helped conceptualize the study and wrote significant portions of the manuscript, focusing on Getting To Outcomes and RE-AIM. GH conceptualized the analyses and measures, conducted the analyses, and performed significant editing. THB helped conduct some of the analyses. SM developed the technical assistance component of Getting To Outcomes and performed significant editing. All authors read and approved the final manuscript.
The authors declare that they have no competing interests.
Consent for publication
Ethics approval and consent to participate
The VA Central IRB that oversees multi-site trials granted approval for this study in October of 2011. Case managers gave their consent to participate. This study did not involve human tissue, animals, or plants.
This project is registered at ClinicalTrials.gov with number NCT01430741 (URL: https://clinicaltrials.gov/ct2/show/study/NCT01430741?term=sdp+11-240&rank=1).
Central IRB approval for this study was granted in October of 2011. The three sites were trained on MISSION-Vet and GTO in the first half of 2013. The first GTO planning meetings began directly after training occurred, between January 2013 and November 2013 across the three sites. The data collection—via a fidelity measure embedded into the VA Electronic Medical Record System—began as each site initiated MISSION-Vet, between April 2013 and January 2014 and ended between July 2014 and August 2015.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Chinman, M., McCarthy, S., Hannah, G. et al. Using Getting To Outcomes to facilitate the use of an evidence-based practice in VA homeless programs: a cluster-randomized trial of an implementation support strategy. Implementation Sci 12, 34 (2017). https://doi.org/10.1186/s13012-017-0565-0
- Implementation support
- Co-occurring disorders
- Technical assistance