Skip to main content

Facilitators and barriers to effective scale-up of an evidence-based multilevel HIV prevention intervention

Abstract

Background

Since the scale-up of HIV/AIDS prevention evidence-based interventions (EBIs) has not been simple, it is important to examine processes that occur in the translation of the EBIs into practice that affect successful implementation. The goal of this paper is to examine facilitators and barriers to effective implementation that arose among 72 community-based organizations as they moved into practice a multilevel HIV prevention intervention EBI, the Mpowerment Project, for young gay and bisexual men.

Methods

CBOs that were implementing the Mpowerment Project participated in this study and were assessed at baseline, and 6-months, 1 year, and 2 years post-baseline. Semi-structured telephone interviews were conducted separately with individuals at each CBO. Study data came from 647 semi-structured interviews and extensive notes and commentaries from technical assistance providers. Framework Analysis guided the analytic process. Barriers and facilitators to implementation was the overarching thematic framework used across all the cases in our analysis.

Results

Thirteen themes emerged regarding factors that influence the successful implementation of the MP. These were organized into three overarching themes: HIV Prevention System Factors, Community Factors, and Intervention Factors. The entire HIV Prevention System, including coordinators, supervisors, executive directors, funders, and national HIV prevention policies, all influenced implementation success. Other Prevention System Factors that affected the effective translation of the EBI into practice include Knowledge About Intervention, Belief in the Efficacy of the Intervention, Desire to Change Existing Prevention Approach, Planning for Intervention Before Implementation, Accountability, Appropriateness of Individuals for Coordinator Positions, Evaluation of Intervention, and Organizational Stability. Community Factors included Geography and Sociopolitical Climate. Intervention Factors included Intervention Characteristics and Adaptation Issues.

Conclusions

The entire ecological system in which an EBI occurs affects implementation. It is imperative to focus capacity-building efforts on getting individuals at different levels of the HIV Prevention System into alignment regarding understanding and believing in the program’s goals and methods. For a Prevention Support System to be maximally useful, it must address facilitators or barriers to implementation, address the right people, and use modalities to convey information that are acceptable for users of the system.

Peer Review reports

Background

The field of Implementation Science is rapidly expanding as research into the creation of evidence-based interventions has yielded innovative approaches to ameliorate various problems, particularly adverse health issues. Considerable research has focused on the dissemination of information about evidence-based interventions (EBIs), since they cannot be adopted if potential implementing organizations are unaware of them. Other research has focused on various aspects of implementation, with the understanding that once organizations have decided to adopt an EBI, the program needs to be moved into effective practice. As the field advances, numerous researchers are developing dissemination and implementation (D&I) models. A recent paper analyzed 61 D&I models and found that most models emphasized dissemination rather than implementation of interventions: 27 models focused on dissemination, 17 models emphasized both equally, and only 17 focused on implementation issues. Of this latter group, only 12 models entirely focused on implementation [1].

The level of implementation achieved is an important determinant of effectiveness outcomes. Since poor implementation outcomes can impact effectiveness outcomes, it is critical to discriminate between implementation outcomes (Are they doing the program as intended?) and effectiveness outcomes (Is it resulting in good outcomes?) [2]. Implementation in this paper means, “the use of strategies to adopt and integrate evidence-based health interventions and change practice patterns within specific settings” ([3] p.424). When evidence-based HIV prevention interventions are implemented in the “real world,” they can achieve effects similar to those in the original research [4-7]. But how EBIs are implemented, including the intervention’s core elements (required project components that embody the intervention’s theory and internal logic and most likely produce the intervention’s main effects [8,9]), has a substantial impact on their effectiveness and sustainability over time [10]. Multiple implementation studies in the HIV prevention field have found that organizations often fail to implement all required core elements or adapt them significantly [11-14].

Although some evidence-based HIV prevention interventions have been widely adopted [15-19], including the intervention studied in this paper, the Mpowerment Project (MP), the scale-up of HIV/AIDS EBIs has not been simple and the “implementation gap,” the translation of scientific knowledge to effective and efficient program implementation, has received increasing attention [20,21]. The MP is a cost-effective HIV prevention intervention EBI for young gay/bisexual men (YGBM) that was tested in randomized, controlled trials and found to be effective in reducing rates of unprotected anal intercourse [22-24]. Widespread dissemination of the MP occurred because the CDC promoted it and other EBIs to health departments and community-based organizations (CBOs) as part of the “DEBI” (Diffusion of Effective Behavioral Interventions) project that sought to move HIV-prevention related EBIs into practice [25]. The present research focuses on implementation of the MP by CBOs that obtained funding to implement it. Examining implementation of the MP is important both because it focuses on YGBM, a population that remains at high risk for HIV transmission [26], and because it is a multilevel HIV prevention approach, which has been identified as critical for HIV prevention [27] in the US National HIV/AIDS Strategy [28] and the CDC’s High-Impact Prevention efforts [29].

The overall goal of the TRIP (Translating Research into Practice) Study was to determine if providing innovation-specific capacity building assistance improved the implementation of the MP by CBOs. Research to practice models are based on the paradigm that scientists conceptualize and establish an intervention’s efficacy, and then CBOs are expected to implement it [30,31]. This transfer of knowledge approach, from scientists to community, is in contrast with community-centered models, which are based on a paradigm that emphasizes improving practice within the context of the community while considering the resources and needs of the community. This means that capacity building assistance strives to help CBOs adapt interventions to the community context [30]. TRIP’s capacity-building approach was a blend of the two models. Our intent was for organizations to implement the MP as operationalized in the efficacy trials, including using community-based participatory methods, while also adapting the intervention to their communities in ways that preserve the core elements and conceptual underpinnings of the program (which we formulated into MP’s “guiding principles”; see Table 1). In this paper “successful” or “effective” implementation (used interchangeably) mean the extent to which intervention fidelity is maintained while adapting the MP for specific communities. The underlying assumption is that fidelity to the intervention is most likely to result in outcomes that approximate those achieved in the original trials.

Table 1 Overview of the Mpowerment Project

The TRIP study, a longitudinal project described previously [14,34], had two primary objectives: (1) to determine if an intervention for CBOs implementing the MP, called the Mpowerment Project Technology Exchange System (MPTES), would result in increases in MP fidelity across the CBOs over time, and (2) to gain an understanding of the issues CBOs experienced while running the EBI, specifically focusing on barriers and facilitators to effective implementation.

The MPTES is an integrated system of training, ongoing technical assistance (TA), implementation materials (manuals and videos), and web-based services, and collaboration with CBOs was used extensively in its development. The MPTES is based on social learning theory [35], diffusion of innovations [36], and theories and approaches to adult education [37]. Since there is a paucity of research or theory about the efficacy of different approaches to capacity building in order to improve intervention fidelity [1,3], the MPTES primarily focused on issues that had arisen in our previous work with CBOs that had contacted us for assistance [34]. The Interactive Systems Framework for Dissemination and Implementation has been developed as a heuristic to conceptualize the systems involved with moving an innovative program into practice [38]. A Prevention Support System is a key component that involves the provision of assistance to CBOs implementing an innovation. The MPTES is a Prevention Support System, and primarily targeted frontline staff who carry out the daily tasks of implementing the MP (called “coordinators”), although some information was targeted at supervisors.

As the primary organization that provided information and training to CBOs on the MP, we had access to all organizations seeking information on the MP. We provided the MPTES proactively to CBOs, and assessed each organization for two years. The goal of this paper is to examine the barriers and facilitators to effective implementation of the MP by CBOs that arose while the organizations were running the intervention during this study. See Table 1 for a brief overview of the MP.

Methods

All but 2 of the 74 CBOs implementing the MP agreed to participate. Organizations were recruited into the study through telephone calls and letters. We first obtained executive directors’ (EDs) or HIV prevention directors’ consent for the agency to participate in interviews. Then individuals we interviewed provided their own consent to participate. Interviews were conducted at baseline, when each CBO was recruited into the study and after which TA was provided; 6-months post-baseline, after which CBOs were provided with more TA; one year post-baseline, after which CBOs were offered more TA; and two years post-baseline. Since logistically the interviews at all participating CBOs could not be conducted simultaneously, we used rolling enrollment with new CBOs entering into the study continually over time until no other CBOs implementing the MP could be identified. Semi-structured telephone interviews were conducted separately with 2–4 individuals at each CBO, including the coordinators; their supervisors, who were typically the agencies’ HIV prevention directors; and 1–2 core group members (YGBM volunteers who serve as decision-makers to MP projects). Due to high job turnover, only a third of participants were interviewed multiple times over the two-year data collection period for each CBO. Participants were asked to locate an area at work where they could participate in the interviews while maintaining privacy. Participants were compensated $25 per interview, and the University of California San Francisco’s institutional review board approved the research protocol.

Data for this paper came from two sources: telephone interviews and extensive written TA fieldnotes (see Table 2). Barriers and facilitators to implementation was the overarching thematic framework used across the data analysis. Principles of Framework Analysis [39] guided the analytic process, allowing the research team to use the predetermined thematic framework, as well as to capture emergent codes within the data. The team met biweekly to discuss the data, note emergent themes, and refine the analytic process. After reviewing approximately 150 interview summaries and TA fieldnotes, the coding schema was operationalized by elaborating on apriori codes, emergent codes, and their subcodes, which were the most prominent organizing themes in the data. The first author completed extensive memo writing after each analysis session. The codebook was finalized after reviewing approximately 200 interview summaries and fieldnotes, and then applied across the dataset. Analysts continued to meet regularly, discussing any discrepancies that emerged during data collection or coding, and discussing particular illustrative cases in great detail to further understand the theoretical processes emergent in the data, and further refine how the codes related to one another.

Table 2 Data sources, data analysis and quality control methods used

Results

The CBOs were quite diverse (see Table 3). They served different ethnic/racial groups of YGBM, were located across the US, and the size of their budgets for the MP varied substantially. Thirteen themes emerged regarding influences on the successful implementation of the MP, which were organized into three overarching themes: HIV Prevention System Factors, Community Factors, and Intervention Factors (see Table 4). Examples of facilitators and barriers to successful implementation are included in the descriptions and verbatim quotes from the participants are presented in Table 4.

Table 3 Characteristics of the 72 community-based organizations in the study
Table 4 Barriers and facilitators to successful implementation of the empowerment project

HIV prevention system factors

The entire HIV prevention system affects intervention implementation

Coordinators, supervisors, executive directors (EDs), program funders, and national HIV prevention policies all greatly influenced implementation success, and program implementation suffered when there was not alignment in views about the intervention among these entities. For example, sometimes coordinators wanted to implement the MP with fidelity, whereas supervisors or EDs did not see the value of doing so as they distrusted a guiding principle (e.g., using social events to attract YGBM; see Table 4). Funders sometimes strongly positively or negatively affected implementation. In numerous situations, funders would not financially support a core element (see Table 4), but at other situations funders would push agencies to implement with fidelity by urging them to implement dropped core elements. Having coordinators, CBO management, and funders in alignment resulted in the most effective implementation. National HIV prevention policies that required CBOs to adopt interventions that had been shown to have evidence of effectiveness, such as MP, sometimes negatively impacted implementation when they were met with resistance by CBO staff (see Table 4).

Knowledge about the intervention

Staff at all levels of the CBOs who had a deeper understanding of the intervention were substantially more able to implement it with fidelity than were those who did not fully understand it. A lack of knowledge about the intervention affected analysis of the community, subsequent solutions, prioritization of tasks by front line staff and the abandonment of core elements and guiding principles originally designed to support the intervention (see Table 4).

It was also important for funders to understand the program’s core elements and guiding principles. Those who understood the intervention were more likely to develop contractual language that supported implementation with fidelity, whereas funders who lacked an understanding of the program often developed contracts that contained unrelated “deliverables” that the CBO was required to achieve (see Table 4).

Belief in the efficacy of the intervention

To run the intervention with fidelity, coordinators, supervisors, EDs, and funders all needed to believe that implementing the EBI would lead to sexual risk reduction with their target population. This was especially true for coordinators, as those who believed that the MP would be effective for their group were most enthusiastic about implementing the MP with fidelity (see Table 4).

Desire to change agency’s existing prevention approach

Frontline staff and management who felt that their previous HIV prevention strategies were ineffective were far more willing to implement the MP with fidelity, as were staff who saw the value of implementing EBIs. But institutional sources of resistance to change sometimes stymied the efficient adoption of the MP. Staff were sometimes slow to adopt the intervention out of inertia (see Table 4). While they did not necessarily believe what they had been doing was effective, they were not necessarily desirous of change. Other times staff believed they saw a similarity between the MP and their current strategy, and simply relabeled their previous program the MP without making changes. Some CBOs adopted the MP solely because funding to do so was available, but had little desire to work with YGBM. This did not bode well for implementation.

Planning for intervention before implementation

An important theme that emerged was that project implementation was more efficient and effective when CBO management, particularly EDs, planned ahead to implement the MP than when they did not have such preparation. When planning did not occur before CBOs received the funding, it resulted in management who did not recognize the need for a space or sufficient and appropriate staff, did not budget to fully implement all of the core elements, or did not make necessary policy changes at the CBO (e.g., allowing staff to work after business hours; see Table 4). Often organizations only learn a short time beforehand that they are funded, which does not facilitate careful planning for implementation.

Accountability for work

Coordinators need to accomplish many tasks for the program to function given its multiple core elements, and those who put effort into their jobs were substantially more effective at implementing the intervention than were men who put in little effort. CBOs varied considerably in the extent to which they held coordinators accountable for their work (see Table 4).

Capacity and appropriateness of individuals for coordinator positions

Coordinators could make or break the implementation of the MP. Successful coordinators attracted YGBM to the project, were the starting point of diffusion into YGBM communities, and helped core groups develop creative, appealing, and innovative activities. But CBO management were not always mindful about the characteristics and skills needed to effectively lead this project, and instead, hired men who were already working at the CBO who fit the demographics (male, gay, young), and assumed they could run it. Hence, CBOs sometimes hired coordinators who alienated others, were social isolates, had poor interpersonal skills, or lacked the capacity to run the program (see Table 4).

Evaluation of intervention’s functioning

Although ongoing reflection and evaluation of each core element’s implementation is necessary, some coordinators never critically assessed how their program was functioning, despite having ready-made, user-friendly, simple process evaluation materials in the implementation manual to use. They did not analyze if the program was achieving its objectives, if it was reaching new YGBM and social networks, and what, if anything, about their program needed to change. Frontline staff and CBO management who engaged in critical, ongoing analysis of program functioning were considerably more effective at implementing the intervention.

Organizational stability

When CBOs experience constant flux/financial crises it was difficult to implement the MP. It was clear that CBOs were unlikely to implement the intervention effectively when they were struggling to remain solvent or are going through substantial staff turnover.

Community factors

Geography

Where the MP was implemented affected implementation. Population size was important. The MP was originally developed for mid-sized cities (populations ranged from 100,000-1.5 million), where it attracted YGBM because it filled a gap: young men needed social opportunities separate from bars, clubs, and “cruising” places. The MP fills this gap in many locales and attracts young men by providing diverse social opportunities in a safe environment. Small communities, rural areas and communities within close range of a “gay magnet city” experienced difficulty attracting men (see Table 4).

Sociopolitical context

The degree to which the sociopolitical environment marginalized YGBM was another community issue that affected implementation. It can be challenging to implement some core elements with fidelity because of hostile responses in a conservative area, and staff often needed to spend considerable time dealing with such issues. For example, it can be challenging to find a project space because some landlords did not want to rent their space for a YGBM or HIV project.

Intervention factors

Intervention characteristics

Three intervention characteristics were barriers to implementation. The first barrier stemmed from MP being a multilevel, multicomponent, community-wide program and therefore complex, requiring significant staff time and effort. Many CBOs were underfunded, which resulted in insufficient staff to accomplish MP’s activities.

The second MP characteristic that was sometimes a barrier to implementation is that the intervention is not highly scripted, except for the M-groups, and the implementing agency has to decide precisely how to operationalize it. Implementing the MP effectively, which can include adapting it, requires understanding the purpose of each core element and guiding principle, and how they all relate to each other. However, many CBO coordinators and management never or rarely looked at any of the MPTES materials, did not attend a training on the intervention, and hence, did not understand the intervention well [34].

The third challenging MP characteristic is the need to use empowering, participatory methods. It can be difficult for the coordinators to learn how to draw out, listen, and incorporate the views of participants rather than simply directing the program themselves. Yet simply running the program without YGBM’s involvement has various adverse consequences, such as poor attendance at activities and little reach into diverse social networks.

Adaptation issues

Many organizations felt that adaptation of the intervention was essential for the MP to work in their community. Some coordinators understood how to adapt the MP using the guiding principles and retaining the core elements, and therefore created adaptations that were entirely in alignment with them. However, other adaptations did not contribute to successful implementation (see Table 4). Occasionally CBOs intentionally dropped core elements entirely as a part of adapting the intervention. Other times, however, core elements were dropped simply because they were overlooked. Other organizations wanted specific guidance about adapting the MP for their populations, and did not want to make adaptations themselves.

Importance of the themes for implementation success

The themes subsumed under HIV Prevention System factors and Intervention factors affected implementation far more frequently than did Community factors. The most frequent themes that arose concerned the coordinators’ capacity and appropriateness to serve as frontline staff for the MP; that the MP is a multicomponent intervention and thus requires adequate resources; planning for the intervention before implementation; and adaptation. Themes that occurred less often than others were staff members’ reluctance to change existing programming, and geography, meaning proximity to gay magnet areas.

It was difficult to determine if CBO characteristics were related to the effectiveness of implementation, other than sometimes being “insufficient” for implementation (e.g., very small organizations often could not implement the MP effectively because of insufficient resources). Yet a program associated with a university and which had few staff was implemented well, because their focus was on a limited group (students), whereas large organizations did not necessarily excel in their implementation since they sometimes had inertia in implementing a new approach or the management simply did not trust YGBM to make decisions, thus alienating volunteers and their staff. In general it seemed that AIDS organizations were able to implement the MP more effectively than non-AIDS organizations, although this varied as indicated above with the case of the university. Yet there was a considerable range of organizations that were the non-AIDS organizations and it does not necessarily make sense to lump them together in order to contrast them with AIDS organizations.

Discussion

This is one the first studies of an HIV prevention intervention that has looked at implementation issues across many CBOs, longitudinally, and in-depth; many implementation researchers have bemoaned the infrequency of this kind of research [3,40]. We found that the entire HIV Prevention System, including frontline staff, CBO management (both supervisors and EDs), funders, and HIV national policies strongly impacted the extent to which the program was moved into practice successfully. The most successful implementations occurred when the HIV Prevention System was in alignment with respect to the frontline staff, management, and funders. We had not anticipated that funders would be an important factor, such as when they pushed their CBOs to implement core elements with greater fidelity, or developed contracts to implement the MP that did not reflect the intervention. Similar to findings reported in other studies, national HIV prevention policies, social determinants, and the community in which the program was implemented also impacted implementation, as did intervention characteristics [41,42]. Additionally, many issues revealed in this study correspond with facilitators and barriers to implementation that have been identified in translating research into practice outside of HIV/AIDS [30,40,43].

The results obtained in this study are consistent with the Consolidated Framework for Implementation Research (CFIR) [44], which identifies five domains important to consider when studying implementation. These domains include the intervention itself, outer setting characteristics, inner setting characteristics, individuals, and process. Each of the five domains is comprised of a set of constructs that are believed to positively or negatively influence implementation. A multilevel ecological perspective, including CFIR, is necessary for understanding successful implementation, as others have proposed [31,45-48]. However, with an eye towards the development of effective Prevention Support Systems, we propose some changes. Figure 1 is a heuristic to capture these multiple levels, and is adapted from Durlak and DuPre [48].

Figure 1
figure1

Framework for understanding the forces impacting successful implementation.

The frontline staff, supervisors, and EDs should not be subsumed under general “provider characteristics” or “organizational factors”, since how they function vis-à-vis the intervention differs substantially, and each uniquely affects the implementation process. This is indicated through the dashes shown in the figure. Supervisors and EDs indeed play important roles in implementation, but frontline staff are of critical importance since they are the intervention “agents.” This finding is consistent with the CFIR, which highlights the importance of individuals’ knowledge and beliefs about the intervention as well as their competence, intellectual ability, tolerance of ambiguity, and learning style as a crucial component of the implementation system. Moreover, we have added a funder level, in addition to the community and the policy environment. The bidirectional arrows indicate considerable interaction among the ecological levels. We propose that these levels must be differentiated as indicated so that Prevention Support System addresses each level separately since the facilitators and barriers to implementation at each level are not necessarily the same.

We had theorized at the outset that greater knowledge about the intervention would be associated with greater fidelity, and this underlay our development of a thorough Prevention Support System. The manual was used extensively, with many CBO staff members calling the purple-colored manual their “Purple Bible” [34]. Yet frequently coordinators attempted to implement it without reading the manual, watching a video or attending a training, and failed to understand basic information about the MP. Knowing that many CBOs might have insufficient funds for in-person training, the manual covered all aspects of implementation and was low-cost or free if downloaded from our website. It was not only coordinators who sometimes did not understand the intervention, but many supervisors, EDs, and funders were also unaware of how the intervention worked. This resulted in poor planning, poor supervision, EDs who refused to implement some parts of the intervention, and funders who did not understand what contractual deliverables were needed. Supervisors and EDs who understood the EBI were more likely to implement it with fidelity. The variability in use of implementation materials was evident in TA sessions with coordinators. TA was provided proactively, not waiting for them to request it since our prior work had indicated that they often do not do so until they are in crisis [31,34,49]. Some TA involved explanations of basic concepts (e.g., the difference between the core group and M-groups), whereas other TA focused on more sophisticated issues, such as project adaptation or how to reach into different social networks. TA was not provided to supervisors, EDs, or funders, unless it was requested. These results suggest that it should be proactively provided to them as well.

Many of the issues that emerged correspond to concepts of organizational readiness (an inner setting component in the CFIR model), including planning for the intervention before implementation (a process domain in the CFIR), desire to change the agency’s existing prevention approach (an inner setting domain in the CFIR that is part of the implementation climate construct), belief in the efficacy of the intervention (an individual domain in the CFIR), knowledge about the intervention (an individual CFIR domain), and organizational stability (an inner setting domain in CFIR that is part of the structural characteristics construct). While these issues all relate to readiness to implement a new program, they refer to very different facets of the concept. Casteñada et al. [50] noted that readiness to implement has been a very broad conceptual category and their research review revealed four domains. While our findings could fit into these domains, recognizing the distinctiveness of the issues would likely make them easier to address in a Prevention Support System. For example, desire to change the existing prevention approach and belief in the efficacy of the EBI might be considered flip sides of the same issue, but this is not necessarily true. Many CBOs were ready to change from what they had been doing, but did not necessarily buy into the MP. Other CBOs were not dissatisfied with what they had been doing, but were happy to implement the MP. Achieving institutional buy-in and agency change are often necessary for successful implementation [42]. We found preparation to implement to be of considerable importance, and seems to be at the core of organizational readiness. But preparing front-line staff for change is quite different than convincing CBO management that change is necessary or helping a CBO determine if MP is the correct fit for its capacity and community. Addressing each issue would likely call for tailored discussions in implementation materials, TA, and training, and therefore, we have not found it useful to relegate these different issues to being “ready to implement.”

It was desired that CBOs adapt the MP for their locale and their population of YBGM after gaining an understanding of the core elements, guiding principles, and overall integrity of the intervention. Although Fixsen et al. [43,51] suggest that organizations first implement a program with fidelity and adapt a program thereafter, we have felt that that adaptation may be necessary to do from the start of implementation or it will not engender target population or agency buy-in. It has been suggested that programs should add but not delete or overly modify existing core elements [52]. This is what many CBOs did as they recognized that while the basic ideas and approaches should be preserved, it would not be problematic to add to the program, especially when done to further the social processes the program was seeking to propel (e.g., diffusion of messages or empowerment of YGBM). Indeed, many CBOs made intentional changes after a careful planning process that included analyzing if the modifications retained fidelity to MP’s guiding principles and core elements. However, there were also CBOs that drastically changed (or eliminated) core elements without considering how the intervention would work with such alterations. These changes were often unintentional, and occurred as the result of poor or no planning, or the lack of evaluation and critical analysis of the program functioning. Such changes often had deleterious effects.

The frequency with which a theme emerged was not necessarily associated with its impact on implementation success, as some issues had considerable impact even if they did not occur often. For example, funders’ refusal to pay for particular core elements was not a frequent occurrence, but when it happened, it had deleterious effects. In addition, the themes do not necessarily have independent effects, and instead can compensate for other barriers. For example, some programs had sufficient resources but poor leadership and challenging organizational issues to contend with (e.g., lack of interest in changing what they were doing) which adversely affected the program’s implementation. Other programs had highly effective staff who were able to motivate volunteers to take on important tasks and were creative in how they implemented core elements, while having low funding. Yet even the best of staff could not implement the intervention when the organization became unstable and went bankrupt.

The two most important factors affecting implementation with fidelity seemed to be resources/funding (outer setting CFIR domains), similar to others’ findings [15], and having effective frontline staff (individual CFIR domain) [43]. The MP was often underfunded, making it challenging to implement fully. Moreover, the policy change at the CDC that greatly increased HIV testing (an outer setting CFIR domain) [52] often resulted in funding for MP to be redirected to it, further reducing resources for implementation.

An important finding was the considerable variability in the extent to which CBO staff engage in reflection, analysis, hold themselves and others accountable for their work, and evaluate if they are reaching program objectives (all process CFIR process domains). These findings, in accordance with others’ recommendations [43,51,53], indicate that TA should include feedback on fidelity, especially if CBOs are not evaluating themselves. Empowerment evaluation [54] would seem particularly helpful, and might avoid a feeling of “top-down” evaluation that can put individuals into defensive modes of response.

These findings suggest that CBOs would benefit from capacity building assistance shortly after being funded, not just once they are implementing the intervention, and that it should be provided proactively instead of waiting for CBO staff to identify their needs. Although some researchers have suggested that CBOs go through specific processes when planning to implement an intervention, including a stage in which they examine their own capacities and consult with the community [9,55], we feel it would be helpful for EBI-specific capacity building to be provided to the CBO immediately upon being funded to help them make the best decisions from the start [38].

Organizational issues also need to be addressed in capacity building efforts, although the term “capacity” may suggest primarily focusing on knowledge, competencies, and skills building to the exclusion of other issues [56,57]. For example, the ED who did not trust YGBM to make decisions for the project did not necessarily lack the knowledge to run the intervention. Likewise, coordinators and supervisors who never looked at the manual or attended a training lacked knowledge of the program, but also needed to shift their attitudes regarding the importance of using such materials. Hence providing what is called “capacity building assistance” does not necessarily solely address knowledge or skills acquisition. Instead, capacity building may require attempting to change attitudes and motivations. This is in keeping with the definition of capacity that Flaspohler et al. [30] use: “the skills, motivations, knowledge, and attitudes necessary to implement innovations.” (p.183).

This research was not without its flaws. It would have been preferable to have recorded and transcribed the data for this study and to conduct site visits to the projects to observe challenges and facilitators to effective implementation firsthand. However, these were both prohibitively expensive, and so we relied on telephone interviews, extensive note-taking and participants’ description of their projects and the facilitators/challenges they experienced.

Conclusions

Capacity building is vitally important in helping organizations in their implementation of EBIs [41,57]. The MPTES was developed to build the capacity of CBOs to translate an HIV prevention EBI into practice. These results suggest that a Prevention Support System, such as the MPTES, should address the entire ecological system in which a program occurs, ranging from the frontline staff to the broader system of HIV prevention, including the impact of national policies. In addition, capacity building should focus on bringing individuals at different levels of the implementation system into alignment regarding understanding the program’s goals and methods, including CBOs’ frontline staff, supervisors, and EDs, as well as funders, and should target helping them all to gain an in-depth understanding of the program, buy into the new approach, and plan for implementing the program before attempting to move it into practice. Moreover, capacity building should address the importance of having expectations of accountability, and seek to increase front-line staff’s motivation and skills to reflect and analyze the program’s functioning. Importantly, the issues to address with individuals at different levels of the implementation system vary considerably. Finally, since a Prevention Support System is only effective if it is actually used, it is essential to focus on how to increase the willingness of CBO staff and funders to use such a system. It must be very user friendly, including being in an acceptable format. For a Prevention Support System to be maximally useful, it must address facilitators or barriers to implementation, address the right people, and use modalities to convey information that are acceptable for users of the system.

Abbreviations

AIDS:

Acquired immune deficiency syndrome

CDC:

Centers for disease control and prevention

CFIR:

Consolidated framework for implementation research

D&I research:

Dissemination and implementation research

DEBI project:

Diffusion of effective behavioral interventions, a program by the CDC to diffuse evidence-based interventions into practice

EBI:

Evidence-based intervention

ED:

Executive director

HIV:

Human immunodeficiency virus

M-groups:

M-groups, this is not an acronym but is the name for the small group component of the Mpowerment project

MP:

Mpowerment project, the intervention being studied in this research also referred to as a program

MPTES:

Mpowerment project technology exchange system, our integrated system of training, ongoing technical assistance (TA), implementation materials (manuals and videos), and web-based services.

TA:

Technical assistance

TRIP Study:

Translating research into practice study

YGBM:

Young gay or bisexual men

References

  1. 1.

    Tabak RG, Khoong EC, Chambers DA, Brownson RC. Bridging research and practice: models for dissemination and implementation research. Am J Prev Med. 2012;43:337–50.

    Article  PubMed  PubMed Central  Google Scholar 

  2. 2.

    Blase K, Fixsen D, Phillips E. Residential treatment for troubled children: developing service delivery systems. In: Paine S, Bellamy G, Wilcox B, editors. Human services that work: from innovation to standard practice. Baltimore: Paul H. Brookes Publishing; 1984. p. 149–65.

    Google Scholar 

  3. 3.

    Norton WE, Amico KR, Cornman DH, Fisher WA, Fisher JD. An agenda for advancing the science of implementation of evidence-based HIV prevention interventions. AIDS Behav. 2009;13:424–9.

    Article  PubMed  PubMed Central  Google Scholar 

  4. 4.

    Fisher HH, Patel-Larson A, Green K, Shapatava E, Uhl G, Kalayil EJ. Evaluation of an HIV prevention intervention for African Americans and Hispanics: findings from the VOICES/VOCES Community-based Organization Behavioral Outcomes Project. AIDS Behav. 2011;15:1691–706.

    Article  PubMed  Google Scholar 

  5. 5.

    Heitgerd JL, Kalayil EJ, Patel-Larson A, Uhl G, Williams WO, Griffin T, et al. Reduced sexual risk behaviors among people living with HIV: results from the Healthy Relationships Outcome Monitoring Project. AIDS Behav. 2011;15:1677–90.

    Article  PubMed  Google Scholar 

  6. 6.

    Jemmott 3rd JB, Jemmott LS, Fong GT, Morales KH. Effectiveness of an HIV/STD risk-reduction intervention for adolescents when implemented by community-based organizations: a cluster-randomized controlled trial. Am J Public Health. 2010;100:720–6.

    Article  PubMed  PubMed Central  Google Scholar 

  7. 7.

    Neumann MS, O'Donnell L, Doval AS, Schillinger J, Blank S, Ortiz-Rios E, et al. Effectiveness of the VOICES/VOCES sexually transmitted disease/human immunodeficiency virus prevention intervention when administered by health department staff: does it work in the “real world”? Sex Transm Dis. 2011;38:133–9.

    Article  PubMed  Google Scholar 

  8. 8.

    Kelly JA, Heckman TG, Stevenson LY, Williams PN, Ertl T, Hays RB, et al. Transfer of research-based HIV prevention interventions to community service providers: Fidelity and adaptation. AIDS Educ Prev. 2000;12:87–98.

    CAS  PubMed  Google Scholar 

  9. 9.

    McKleroy VS, Galbraith JS, Cummings B, Jones P, Harshbarger C, Collins C, et al. Adapting evidence-based behavioral interventions for new settings and target populations. AIDS Educ Prev. 2006;18:59–73.

    Article  PubMed  Google Scholar 

  10. 10.

    Feldman MB, Silapaswan A, Schaefer N, Schermele D: Is There Life After DEBI? Examining Health Behavior Maintenance in the Diffusion of Effective Behavioral Interventions Initiative. Am J Community Psychol 2014.

  11. 11.

    Dolcini MM, Canin L, Gandelman A, Skolnik H. Theoretical domains: a heuristic for teaching behavioral theory in HIV/STD prevention courses. Health Promot Pract. 2004;5:404–17.

    Article  PubMed  Google Scholar 

  12. 12.

    Galbraith JS, Stanton B, Boekeloo B, King W, Desmond S, Howard D, et al. Exploring implementation and fidelity of evidence-based behavioral interventions for HIV prevention: lessons learned from the focus on kids diffusion case study. Health Educ Behav. 2009;36:532–49.

    Article  PubMed  Google Scholar 

  13. 13.

    Miller RL, Forney JC, Hubbard P, Camacho LM. Reinventing Mpowerment for Black Men: Long-Term Community Implementation of an Evidence-Based Program. Am J Community Psychol. 2012;49:199–214.

    Article  PubMed  Google Scholar 

  14. 14.

    Rebchook GM, Kegeles SM, Huebner D, Trip Research Team. Translating research into practice: the dissemination and initial implementation of an evidence-based HIV prevention program. AIDS Educ Prev. 2006;18:119–36.

    Article  PubMed  Google Scholar 

  15. 15.

    Cunningham SD, Card JJ. Realities of replication: implementation of evidence-based interventions for HIV prevention in real-world settings. Implement Sci. 2014;9:5.

    Article  PubMed  PubMed Central  Google Scholar 

  16. 16.

    Hamdallah M, Vargo S, Herrera J. The VOICES/VOCES success story: effective strategies for training, technical assistance and community-based organization implementation. AIDS Educ Prev. 2006;18:171–83.

    Article  PubMed  Google Scholar 

  17. 17.

    Hitt JC, Robbins AS, Galbraith JS, Todd JD, Patel-Larson A, McFarlane JR, et al. Adaptation and implementation of an evidence-based prevention counseling intervention in Texas. AIDS Educ Prev. 2006;18:108–18.

    Article  PubMed  Google Scholar 

  18. 18.

    Prather C, Fuller TR, King W, Brown M, Moering M, Little S, et al. Diffusing an HIV prevention intervention for African American Women: integrating afrocentric components into the SISTA Diffusion Strategy. AIDS Educ Prev. 2006;18:149–60.

    Article  PubMed  Google Scholar 

  19. 19.

    Wingood GM, DiClemente RJ. Enhancing adoption of evidence-based HIV interventions: promotion of a suite of HIV prevention interventions for African American women. AIDS Educ Prev. 2006;18:161–70.

    Article  PubMed  Google Scholar 

  20. 20.

    Pangea Global AIDS Foundation. Report from the Expert Consultation on Implementation Science Research: A Requirement for Effective HIV/AIDS Prevention and Treatment Scale-Up. Cape Town, South Africa: Sponsored by Office of AIDS Research, National Institutes of Health, U.S. Department of Health and Human Services in collaboration with Pangaea Global AIDS Foundation; 2009.

    Google Scholar 

  21. 21.

    Schackman BR. Implementation science for the prevention and treatment of HIV/AIDS. J Acquir Immune Defic Syndr. 2010;55 Suppl 1:S27–31.

    Article  PubMed  PubMed Central  Google Scholar 

  22. 22.

    Herbst JH, Beeker C, Mathew A, McNally T, Passin WF, Kay LS, et al. The effectiveness of individual-, group-, and community-level HIV behavioral risk-reduction interventions for adult men who have sex with men: A systematic review. Am J Prev Med. 2007;32:S38–67.

    Article  PubMed  Google Scholar 

  23. 23.

    Kegeles SM, Hays RB, Coates TM. The Mpowerment project: a community-level HIV prevention intervention for young gay men. Am J Public Health. 1996;86:1129–36.

    CAS  Article  PubMed  PubMed Central  Google Scholar 

  24. 24.

    Kegeles SM, Hays RB, Pollack LM, Coates TJ. Mobilizing young gay and bisexual men for HIV prevention: A two-community study. AIDS. 1999;13:1753–62.

    CAS  Article  PubMed  Google Scholar 

  25. 25.

    Collins C, Harshbarger C, Sawyer R, Hamdallah M. The diffusion of effective behavioral interventions project: development, implementation, and lessons learned. AIDS Educ Prev. 2006;18:5–20.

    Article  PubMed  Google Scholar 

  26. 26.

    Smith A, Miles I, Le B, Finlayson T, Oster A, DiNenno E. Prevalence and Awareness of HIV Infection Among Men Who Have Sex With Men – 21 Cities, United States, 2008. MMWR. 2010;59:1201–7.

    Google Scholar 

  27. 27.

    Coates TJ, Richter L, Caceres C. Behavioural strategies to reduce HIV transmission: How to make them work better. Lancet. 2008;372:669–84.

    Article  PubMed  PubMed Central  Google Scholar 

  28. 28.

    Millett GA, Crowley JS, Koh H, Valdiserri RO, Frieden T, Dieffenbach CW, et al. A way forward: the national HIV/AIDS strategy and reducing HIV incidence in the United States. J Acquir Immune Defic Syndr. 2010;55 Suppl 1:S144–7.

    Article  PubMed  Google Scholar 

  29. 29.

    Centers for Disease Control and Prevention: High-Impact HIV Prevention: CDC’s Approach to Reducing HIV Infections in the United States. Atlanta, GA; 2011.

  30. 30.

    Flaspohler P, Duffy J, Wandersman A, Stillman L, Maras MA. Unpacking prevention capacity: an intersection of research-to-practice models and community-centered models. Am J Community Psychol. 2008;41:182–96.

    Article  PubMed  Google Scholar 

  31. 31.

    Wandersman A. Community science: bridging the gap between science and practice with community-centered models. Am J Community Psychol. 2003;31:227–42.

    Article  PubMed  Google Scholar 

  32. 32.

    Hays RB, Rebchook GM, Kegeles SM. The Mpowerment Project: community-building with young gay and bisexual men to prevent HIV. Am J Community Psychol. 2003;31:301–12.

    Article  PubMed  Google Scholar 

  33. 33.

    Mugavero MJ, Amico KR, Horn T, Thompson MA. The state of engagement in HIV care in the United States: from cascade to continuum to control. Clin Infect Dis. 2013;57:1164–71.

    Article  PubMed  Google Scholar 

  34. 34.

    Kegeles SM, Rebchook G, Pollack L, Huebner D, Tebbetts S, Hamiga J, et al. An intervention to help community-based organizations implement an evidence-based HIV prevention intervention: the Mpowerment Project technology exchange system. Am J Community Psychol. 2012;49:182–98.

    Article  PubMed  Google Scholar 

  35. 35.

    Bandura A. Social learning theory. Englewood Cliffs, New Jersey: Prentice-Hall; 1977.

    Google Scholar 

  36. 36.

    Rogers EM. Diffusion of innovations. 5th ed. New York: Free Press; 2003.

    Google Scholar 

  37. 37.

    Knowles MS, Holton EF, Swanson RA. The adult learner: The definitive classic in adult education and human resource development. 5th ed. Houston TX: Gulf Professional Publishing Co; 1998.

    Google Scholar 

  38. 38.

    Wandersman A, Duffy J, Flaspohler P, Noonan R, Lubell K, Stillman L, et al. Bridging the gap between prevention research and practice: the interactive systems framework for dissemination and implementation. Am J Community Psychol. 2008;41:171–81.

    Article  PubMed  Google Scholar 

  39. 39.

    Richie J, Spencer L. Qualitative data analysis for applied policy research. In: Huberman A, Miles M, editors. The Qualitative Researcher’s Companion. London: Sage; 2002. p. 173–94.

    Google Scholar 

  40. 40.

    Mihalic SF, Irwin K. Blueprints for violence prevention: From research to real-world settings–Factors influencing the successful replication of model programs. Youth Violence and Juvenile Justice. 2003;1:307–29.

    Article  Google Scholar 

  41. 41.

    Beyrer C, Baral S, Kerrigan D, El-Bassel N, Bekker LG, Celentano DD. Expanding the space: inclusion of most-at-risk populations in HIV prevention, treatment, and care services. J Acquir Immune Defic Syndr. 2011;57 Suppl 2:S96–9.

    Article  PubMed  PubMed Central  Google Scholar 

  42. 42.

    Gandelman A, Dolcini MM. The influence of social determinants on evidence-based behavioral interventions-considerations for implementation in community settings. Transl Behav Med. 2012;2:137–48.

    Article  PubMed  Google Scholar 

  43. 43.

    Fixsen DL, Naoom SF, Blase KA, Friedman RM, Wallace F: Implementation research: A synthesis of the literature. University of South Florida, Louis de la Parte Florida Mental Health Institute, the National Implementation Research Network; 2005.

  44. 44.

    Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci. 2009;4:50.

    Article  PubMed  PubMed Central  Google Scholar 

  45. 45.

    Altschuld JW, Kumar DD, Smith DW, Goodway JD. School-based educational innovations: Case illustrations of context-sensitive evaluations. Fam Community Health. 1999;22:66–79.

    Article  Google Scholar 

  46. 46.

    Riley BL, Taylor SM, Elliott SJ. Determinants of implementing heart health: promotion activities in Ontario public health units: a social ecological perspective. Health Educ Res. 2001;16:425–41.

    CAS  Article  PubMed  Google Scholar 

  47. 47.

    Shediac-Rizkallah MC, Bone LR. Planning for the sustainability of community-based health programs: conceptual frameworks and future directions for research, practice and policy. Health Educ Res. 1998;13:87–108.

    CAS  Article  PubMed  Google Scholar 

  48. 48.

    Durlak JA, DuPre EP. Implementation matters: a review of research on the influence of implementation on program outcomes and the factors affecting implementation. Am J Community Psychol. 2008;41:327–50.

    Article  PubMed  Google Scholar 

  49. 49.

    O'Donnell L, Scattergood P, Adler M, Doval AS, Barker M, Kelly JA, et al. The role of technical assistance in the replication of effective HIV interventions. AIDS Educ Prev. 2000;12:99–111.

    PubMed  Google Scholar 

  50. 50.

    Castaneda SF, Holscher J, Mumman MK, Salgado H, Keir KB, Foster-Fishman PG, et al. Dimensions of community and organizational readiness for change. Prog Community Health Partnersh. 2012;6:219–26.

    Article  PubMed  PubMed Central  Google Scholar 

  51. 51.

    Fixsen DL, Blase KA, Naoom SF, Wallace F. Core Implementation Components. Res Social Work Prac. 2009;19:531–40.

    Article  Google Scholar 

  52. 52.

    Janssen R, Onorato I, Valdiserri R, Durham T, Seiler E, Jaffe H. Advancing HIV prevention: new strategies for a changing epidemic–United States, 2003. MMWR. 2003;52:329–32.

    Google Scholar 

  53. 53.

    Fagan AA, Hanson K, Hawkins JD, Arthur MW. Bridging science to practice: achieving prevention program implementation fidelity in the community youth development study. Am J Community Psychol. 2008;41:235–49.

    Article  PubMed  Google Scholar 

  54. 54.

    Fetterman D, Kaftarian S, Wandersman A. Empowerment Evaluation: Knowledge and Tools for Self-Assessment and Accountability. Thousand Oaks, CA: Sage; 1996.

    Google Scholar 

  55. 55.

    Dworkin SL, Pinto R, Hunter J, Rapkin B, Remien RH. Keeping the spirit of community partnerships alive in the scale up of HIV/AIDS prevention: critical reflections on the roll out of DEBI (Diffusion of Effective Behavioral Interventions). Am J Community Psychol. 2010;42:51–9.

    Article  Google Scholar 

  56. 56.

    Goodman RM, Speers MA, McLeroy K, Fawcett S, Kegler M, Parker E, et al. Identifying and defining the dimensions of community capacity to provide a basis for measurement. Health Educ Behav. 1998;25:258–78.

    CAS  Article  PubMed  Google Scholar 

  57. 57.

    Hoge MA, Tondora J, Marrelli AF. The fundamentals of workforce competency: implications for behavioral health. Adm Policy Ment Health. 2005;32:509–31.

    Article  PubMed  Google Scholar 

Download references

Acknowledgements

We are most grateful for the partnerships we have had with the CBO staff around the United States who have participated in this research. We have learned a great deal from them, and they have provided a great deal of enormously helpful feedback about our materials, the TA we provide, and the Mpowerment Project itself. Research reported in this publication was supported by the National Institute of Mental Health for the National Institutes of Health under Award Number R01MH65196.

Author information

Affiliations

Authors

Consortia

Corresponding author

Correspondence to Susan M Kegeles.

Additional information

Competing interests

The authors declare that they have no competing interests.

Authors’ contributions

SK and GR conceived of the study. SK drafted the manuscript and participated in the analysis. GR co-directed all facets of the study and with EA directed the analysis. GR and EA assisted in drafting the manuscript. ST conducted all of the interviews and participated in the analysis. The TRIP team provided technical assistance to organizations, took copious field notes of issues that arose in technical assistance episodes, and engaged in ongoing analysis of the data throughout the project. All authors read and approved the final manuscript.

Authors’ information

The TRIP Team includes John Hamiga, Ben Zovod, David Sweeney, Robert Williams, and Andres Maiorana.

Rights and permissions

This article is published under license to BioMed Central Ltd. This is an open access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly credited. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Kegeles, S.M., Rebchook, G., Tebbetts, S. et al. Facilitators and barriers to effective scale-up of an evidence-based multilevel HIV prevention intervention. Implementation Sci 10, 50 (2015). https://doi.org/10.1186/s13012-015-0216-2

Download citation

Keywords

  • Translating research into practice
  • Translational science, HIV prevention interventions
  • Gay men
  • Implementation science
  • Young gay men
  • Young men who have sex with men
  • Structural interventions
  • Community level interventions
  • Community mobilization
  • Fidelity
  • Community-based participatory research