Intermediary/purveyor organizations for evidence-based interventions in the US child mental health: characteristics and implementation strategies

Abstract

Background

Many psychosocial interventions are disseminated and supported by organizations, termed “Intermediary/Purveyor Organizations” (IPOs). Because IPOs remain largely unstudied, we lack understanding of their scale and the strategies they utilize. The role and function of organizations that link resource systems with user systems, such as IPOs, have been identified as an important but understudied issue in implementation science. The objectives of this paper are to describe features of IPOs that disseminate evidence-based interventions (EBIs) for child behavioral health and identify the strategies they use to support their implementation.

Methods

The Substance Abuse and Mental Health Services (SAMHSA) National Registry of Evidence-based Programs and Practices (NREPP) listed 119 unique IPOs for the 127 child behavioral health EBIs listed on its website. Data characterizing each organization were drawn from NREPP and GuideStar profiles. From 119 unique IPOs identified, we found contact information for 108. We sent an electronic survey to capture additional organizational information and implementation strategies the IPOs employed in spreading the EBIs; response rate was 50%. Data are presented descriptively and analyzed using ordinary least squares (OLS) regression and Latent Class Analysis (LCA).

Results

Virtually all identified EBIs had an IPO. IPOs train individuals, organizations, and communities and provide supervision for the use of EBIs. About 20% of IPOs trained at large scale, some training 500–1000+ providers annually. IPOs reported using an average of 32 distinct strategies to implement their EBIs, with most using educational, planning, and quality improvement strategies. However, there was little convergence around strategy helpfulness. The only significant predictor of number of strategies used by an IPO was the NREPP-posted implementation readiness score of the intervention. LCA revealed that IPOs either used several implementation strategies or used very few.

Conclusions

Findings add significantly to knowledge about IPO structure, scale, and function. They use numerous and varying implementation strategies but report little consensus in what works. The study advances methods for measuring and characterizing real-world implementation by demonstrating the feasibility of using a common nomenclature, per a published compilation and of LCA for data reduction in characterizing profiles of implementation approaches.

Background

A recent Institute of Medicine of the National Academies (IOM) [1] report decries the gap between the availability and receipt of evidence-based psychosocial interventions (EBIs) for mental health disorders. Psychosocial EBIs have limited scale; although between 13 and 20% of children and youth experience a mental disorder each year [2], most do not receive EBIs. Bruns et al. report that uptake of mental health treatment for children and youth lags significantly behind adults, with EBIs reaching only 1–3% of youths with serious emotional disturbance [3]. Such data reflect the struggle to scale-up EBI adoption rates to the point where they substantially influence public health on a national scale.

EBI scale-up is limited because of weaknesses in the “distribution system” for bringing treatment discoveries to the attention of practitioners and into practice settings [4]. Unlike drugs and devices, psychosocial interventions have no Federal Drug Administration (FDA)-like delivery system. Instead, distribution is often dependent on the initiative of treatment developers, who lack the resources of an organization such as the FDA. Many treatment developers form organizations to spread information about and provide support for the adoption of their interventions. Such organizations have been called Intermediary/Purveyor Organizations (IPOs) [5, 6]. Well-known IPOs include such international entities as Trauma-Focused Cognitive Behavioral Therapy National Therapist Certification Program, Nurse-Family Partnerships, and Triple P International and their subsidiary, and Triple P America. IPOs have emerged as a major dissemination strategy over the past two decades.

IPO theory and functions

The term IPO combines two key constructs in the dissemination and implementation literature: purveyor and intermediary. Purveyor has been defined as “an individual or group of individuals representing a program or practice who actively work to implement that practice or program with fidelity and good effect” [5]. Intermediary organizations have been described as “…developing, implementing, and supporting multiple best practice programs or services, as well as building the capacity within an agency or system in order to implement and sustain such programs” [6,7,8]. Intermediary roles also include consultation, quality assurance, and continuous quality improvement; outcome evaluation; and training [6, 8]. The limited research on these organizations suggests that many organizations identify as both purveyors and intermediaries [6].

The IPO definition and role reflect several key concepts in the implementation science literature, concepts that are widely used but often with overlapping meaning, such as change agency, knowledge broker, or resource system. Many of these terms can be traced conceptually to diffusion of innovation theory [9], but the proliferation of terms may cloud understanding and thwart the harmonization of concepts and data for the field of implementation science.

In order to diffuse, disseminate, and help implement psychosocial interventions, most of these organizations probably engage in many of the core functions of distribution systems. These functions include “identify potential users, promote the product to them, provide them with easy access to the product through convenient (and usually local) channels, allow them to see and use the product before acquiring it, help them acquire it, and support the product after purchase if the buyer has questions or problems [4].”

Many IPOs function to spread particular EBIs, having been established by treatment developers after research to shape the intervention and test effectiveness. Such IPOs bring content expertise in one or two specific interventions to practitioners and agencies. Some such organizations may hold patents or copyrights to the name or training in the EBI. Other IPOs appear to be agnostic toward the specific EBIs they promote. Sometimes labeling themselves “intermediaries,” they often work to find the best intervention fit for a specific organization’s needs, link resources (including perhaps EBI-specific IPOs) to users, and provide coaching about implementation processes that are applicable to implementation of any innovation. In their efforts to find the best intervention fit for a specific organization’s needs, they may help users implement a variety of EBIs.

While specific EBIs have an extensive body of research, a more limited literature describes the role and function of IPOs. See for example literature describing such organizations as the National Center for Nurse-Family Partnerships and Healthy IDEAS [10, 11]. These organizations disseminate and train providers in EBIs for optimizing child development and treating geriatric frailty and depression, respectively. We found only one empirical study of IPOs. Franks and Bory presented survey data reflecting IPO organizational identity, development, activities, and functions [6]. Given the proliferation of terms and the paucity of research focused on IPOs, we lack understanding of their respective functions and the strategies they utilize [12,13,14,15]. The dissemination and implementation literature has identified understanding the role and function of organizations who link resource systems with user systems, such as IPOs, as an important but understudied issue [16]. Greater understanding of their structure, sustainability, and operations is needed to help clarify the nature and quality of their contributions to the dissemination and implementation of empirically supported interventions [12].

This paper seeks to shed light on IPOs and the specific strategies they use to implement EBIs in the area of child mental health. We sought to achieve the following objectives:

• 1. Identify the prevalence of IPOs for spreading child behavioral health EBIs in the US;

• 2. Identify IPO organizational characteristics;

• 3. Identify the implementation strategies used by IPOs;

• 4. Determine the strategies IPOs report as most effective;

• 5. Explore the association of organizational characteristics with the implementation strategies employed; and

• 6. Explore a data reduction approach, latent class analysis (LCA), to generate implementation strategy usage profiles.

Methods

Sample

The Substance Abuse and Mental Health Services (SAMHSA) National Registry of Evidence-based Programs and Practices (NREPP) served as the sampling frame for this study. NREPP is the federally managed and vetted registry of mental health EBIs and is designed to be a practical decision-making tool to assist clinicians in selecting efficacious treatments [17,18,19,20].

All EBIs housed in NREPP have been reviewed and scored by trained experts. Useful information for practitioners is extracted from and uniformly formatted in an easy-to-read report [21]. It is a highly utilized site with 30,052 unique visitors accessing it every month [22]. Though it does not purport to house all existing EBIs, its registry has grown considerably from its initial 25 interventions to a bank of 388 vetted EBIs at the time of our search, June 2015 [21, 22].

Our sample was generated using the advanced search function of NREPP. NREPP is searchable by intervention, rather than by IPO, so the inclusion criteria were based on intervention type and characteristics. Inclusion criteria for the sample: (1) EBI was designated by NREPP as “mental health treatment” or “mental health promotion,” (2) the EBI was designed for children up to age 17, and (3) the EBI’s primary outcomes were mental health related. Mental health-related outcomes included one of the following terms in their outcome description: depression, anxiety, trauma, behavioral disorder, disruptive behavior, substance use, social emotional disorder, aggression, or impulsivity. The resultant pool of EBIs was 127.

A member of the research team examined the EBI profiles to identify the intervention’s IPO. From the 127 EBIs, 119 unique IPOs were identified.

Data

The first research objective was addressed by reviewing NREPP’s profiles of the EBIs to see which reported having an IPO. We determined whether an EBI had an IPO based on whether or not the NREPP web site listed an organization for the EBI. To address the remaining research objectives, data were obtained from four sources: the IPO’s NREPP profile, the IPO’s GuideStar profile (if the IPO was non-profit), the IPO’s website, and IPO key informant responses to an electronic survey.

A member of the research team extracted data from the aforementioned web-based sources for each IPO. These data included contact information, website, date IPO was established, location, availability of training materials, how training was administered, and availability of provider certification, fidelity tools, and fidelity monitoring. For IPOs with GuideStar profiles, organization financial information was extracted.

We retrieved a Readiness for Dissemination (RFD) score for each EBI from the NREPP website. NREPP assigned each intervention with an RFD score. The scores ranged from 0 to 4, with high scores indicating that dissemination resources are available and are of high quality. RFD was assessed across three domains: availability of implementation materials, availability of training and support resources, and availability of quality assurance procedures. However, as of July 2015, the RFD score no longer included an NREPP review and the scale is no longer used for newly reviewed EBIs [21].

After initial data were gathered about the IPOs, the research team developed an electronic survey in Qualtrics. The research team contacted each IPO to determine who in the organization was most familiar with the IPOs’ implementation practice. Of the 119 IPOs previously identified, contact information was secured for 108. Respondents’ role within their respective IPOs varied. Our initial invitation to participate in the survey asked to speak with someone familiar with the organization’s implementation efforts. Most of the role titles suggested that the respondents were administrators within the IPO (directors, CEO, presidents, directors of training) or researchers involved in the treatment’s development.

The Qualtrics survey was emailed, with up to two reminders. The response rate for fully completed surveys was 50% (N = 54). The first section of the survey was designed to capture organizational characteristics not available from the existing web content (e.g., number of staff, specifics on training, cost, and funding).

The second portion of the survey was designed to capture the implementation strategies used by the IPOs. This section was based on the implementation strategy inventory organized by Powell, et al. [23]. The original inventory of 68 strategies was piloted on individuals with expertise in IPO operations within the sphere of child/youth mental health services. The language of certain implementation strategies was modified slightly to fit within the study’s context, and four strategies were excluded that did not apply to the types of strategies these organizations could employ, as determined by the content experts. The final inventory of 64 implementation strategies comprised the second section of the survey. For each item, respondents were asked whether they used or suggested the adopting agency/provider use the listed strategy. In addition, respondents were asked to select up to five strategies they felt were most successful. We did not provide definitions of success. We attempted to reduce social desirability through language stating that no one organization would be expected to use all strategies. Respondents were expected to interpret “success” from their own perspective. We did not ask respondents to identify situations or contexts in which strategies were used or deemed successful.

The responses to these items were used to estimate domain-specific strategy profiles using LCA. Both sections of the survey were tested for usability by individuals with experience working in or with IPOs.

In preparation for the LCA, an additional variable for implementation strategies was created. The individual implementation strategies identified were nested under one of five possible domains: planning, education, financial, restructure, and quality management [23]. To generate an overall implementation score for each of these domains, the number of strategies used in a domain was divided by the number of available strategies in that domain. The denominator for this score was based on the Expert Recommendations for Implementation Change (ERIC) compilation.

These domain-specific scores were then dichotomized into high use and low use, based on a cutoff of the 50th percentile. The resultant variable was a dichotomous variable for each IPO per implementation strategy domain, reflecting either high use or low use of strategies within that domain. These items are utilized in estimating the overall implementation strategy profile by LCA.

Analysis

Multiple data reduction strategies were used to address our research objectives. The number of EBIs with active contact information for the IPO or its representative was divided by the number of IPOs. We used descriptive statistics to report IPO characteristics. Scores were assigned to the implementation strategies based on the percentage of IPOs who endorsed using the strategy and rating it as most successful to their implementation efforts.

Ordinary least squares (OLS) regression was used as part of an exploratory analysis to study the association between IPO organizational characteristics and number of implementation strategies used. The dependent variable for the regression was the number of implementation strategies used by the IPO. The independent variables included number of IPO staff, the EBI’s NREPP dissemination score, how many years the IPO has been in operation, and the number of clinicians the IPO has trained. The data were assessed to ensure that none of the assumptions of OLS were violated. Regression was executed in Stata v.14.

We used LCA to identify implementation strategy profiles (or latent classes) in the sampled IPOs (N = 54). This allowed us to explore whether IPOs varied in meaningful ways in the combination of implementation strategies used. Prior to conducting this analysis, we generated two profile levels. The first was an overall implementation strategy profile based on the use of the domain-level strategies (planning, education, finance, restructure, and quality management). The second was a domain-specific profile based on the use of specific strategies within each domain.

An iterative process was used to determine the optimal number of latent classes based on three different model fit statistics: the likelihood-ratio G² statistic, Akaike’s Information Criterion (AIC), and Bayesian Information Criterion (BIC) [24]. A significant result of a likelihood-ratio test suggests that a more complicated model (i.e., “k” class model) has a significantly better model fit than a simpler model (i.e., “k-1” class model) [24]. A lower value on the AIC and BIC suggests a more optimal model in balance between model fit and parsimony [24].

Following the guidelines of methodological experts, selection of the final model took into account these model fit statistics, as well as an attentive examination on model interpretability (i.e., Is each class distinguishable in terms of the pattern of implementation strategies used? Is every class non-trivial in size? Can we define each latent class in a meaningful way?) [24, 25]. We used “PROC LCA” in SAS 9.4 for all LCA [26].

Results

Prevalence of IPOs

IPOs were widespread; virtually all of the EBIs appeared to have an IPO, and the majority of them have active contact information (N = 108, 90.7%). IPOs represented a substantial financial interest in both the public and private sectors. Those IPOs with GuideStar profiles outlining organizational finances (N = 11) showed a collective annual revenue of 1.8 billion.

IPO characteristics

By nature, IPOs reported that they were training organizations. They train at the individual (85%), organizational (84%), and community levels (33%). IPOs train at a variety of sites: onsite where the EBI will be delivered (89%), at IPO headquarters (58%), and online (47%). To get a sense of IPO training volume, a series of ranges were offered to IPOs for reporting the number of clinicians who completed training. In 2015, the modal range for completing EBI training was 26–50 clinicians. Approximately 11% of IPOs reported a range of 251–500 clinicians who completed training, and 9% of IPOs reported that over 1000 clinicians completed training.

Half of IPOs also offered individual-level certification in their EBI. Slightly over one-quarter offered organization-level certification, and one-tenth offered community-level certification. In order to receive the individual-level certification, 41% of IPOs required ongoing supervision which then provided the IPO with an additional source of revenue.

IPOs reported multiple streams of revenue. Nearly all IPOs (80%) received renumeration from their training efforts. Grants (64%), treatment manual sales (55%), and supervision fees (53%) were other frequently used sources of income.

Implementation strategies employed

IPOs used an average of 32 (SD = 13.74) distinct strategies in implementing EBIs, reflecting a wide range of approaches. IPOs endorsed a mean of 4.3 of five domains based on the Powell et al. typology [23].

Table 1 (third column) portrays IPOs’ endorsement of particular implementation strategies. By domain, IPOs heavily relied upon provider educational strategies, with virtually all distributing educational materials, such as intervention guidelines, manuals, and toolkits. Other frequently employed educational strategies included varying their training methods (90.7%) and materials (88.9%) to facilitate provider learning. Over 85% of IPOs followed training with consultation or booster sessions.

Table 1 Percentage of IPOs who use the strategy and rank it within their five most successful

Various planning strategies comprised the second most frequently endorsed domain. Over 75% of IPOs reported tailoring their work to address agency barriers and facilitators, engaging champions for the EBI, and assessing the readiness for change of the organizations in which providers work. The only other domain with strategies endorsed by 75% or more of IPOs was the quality domain. Around three fourths of IPOs developed and introduced quality monitoring tools specific to the EBI, and obtained client feedback on the implementation effort. Two thirds of IPOs endorsed one strategy within the finance domain: encouraging agencies to access new funding to facilitate the implementation process. Fewer than half of IPOs endorsed any implementation strategies in the restructure domain.

We also determined what percentage of the IPOs’ strategies pertained to each domain (Table 2). Organized by IPO, this table reiterates how the finance and restructure domains comprised a smaller percentage of each IPOs’ total strategies, with educational and planning strategies dominating.

Table 2 The percentage of the strategies used by domain for each IPO

Implementation strategies identified as most successful

The last column in Table 1 reflects the percent of IPOs who endorsed a strategy as among the five most successful of the employed strategies. The strategies most frequently endorsed as successful were following training with booster or advanced sessions or training providers to competency (46.3%). Over one third of IPOs endorsed as most successful three additional strategies, two within the educational domain—tailoring educational materials and providing ongoing implementation consultation to providers—and one in the planning domain—engaging EBI champions.

Figure 1 plots strategies by the percentage of IPOs who endorsed their usage against the percentage of IPOs who endorsed the same strategy as one of the most successful. The four previously mentioned strategies (#22, #26, #10, #4) comprise the small cluster of strategies that were endorsed by over 30% of respondents. The plot illustrates how there were various strategies endorsed as frequently used and were not endorsed as being one of the most successful.

Fig. 1

The numbers are the implementation strategies enumerated in Table 1. The numbers are to the right of the dot they are assigned to

IPO characteristics and implementation strategies used

For the exploratory analysis, the results of the multivariate regression indicated that the included covariates explain 27% of the of the variance (R² = .2706, F(4,46) = 4.27, p < .01). The NREPP dissemination readiness score significantly predicted the number of implementation strategies used. Holding all other variables constant, each one point increase on the NREPP score was associated with an IPOs use of approximately eight more implementation strategies.

Implementation strategy profiles

Taking into account both the model fit indicators (G², AIC, and BIC) and model interpretability, we consistently found that the data supported a two-class model in general. Model fit statistics are presented in Table 2. In all profiles (overall and domain-specific), the two-class model showed high entropy—a measure of certainty in latent class classification, a value closer to one suggesting higher certainty. Based on the pattern in class-based implementation strategy used (Fig. 2), we defined these two classes as the “high-use” group and the “low-use” group.

Fig. 2

Overall and domain-specific implementation strategy profiles. (ST strategy). Legend: Overall plot. The x-axis lists the names of the implementation strategy categories. Education, planning, quality, finance, and restructure plots. The x-axis lists the implementation strategies (ST implementation strategy) per the dimension identified by the plots’ title. And the item response probability for the two groups generated by the latent class analysis. A description of the strategies (identified by their number) can be found in Table 1. The groups are based on the two categories generated by the latent class analysis. They are differentiated by the number of strategies they use. ■ The high-use group. □ The low-use group

The class-based use of each implementation strategy (i.e., item-response probabilities) is presented in Fig. 2; fit statistics and other details are found in Table 3. The item response probability represents the likelihood that a strategy (or domain in the overall model) was endorsed by an IPO. For example, in the overall model, 100% of IPOs classified as high-use, endorsed using one or more planning strategies, while only 20% of those in the low-use class did so. For the overall implementation strategy profile, the patterns were quite distinct—high strategy use compared to low strategy use. IPOs either reported a high probability of using strategies within each domain or a low probability of using strategies within a given domain. This suggests that IPOs either used several implementation strategies—across domains—or used very few, with little variability in between.

Table 3 Comparison of latent class models for overall implementation strategy profile

While still maintaining high- vs low-strategy use profiles, some of the domain-specific profiles were more nuanced than the overall profile. In the education domain, for instance, both groups endorsed (1) developing educational materials to make it easier for stakeholders to learn to implement or learn about the EBI; (2) distributing educational materials (e.g., guidelines, manuals, and toolkits) in person, by mail, and/or electronically; ( 3) conducting training in the EBI that included follow-up training, advanced training, booster training, or training to competence; (4) varying training methods for different learning styles, or shape the training to be interactive; and (5) providing clinicians with ongoing consultation from an expert in the intervention. This was not surprising given nearly all IPOs reported using these strategies. Given that 96.34% of IPOs used educational materials (ST20), the actual variance between high- and low-use profiles was nearly non-existent. Conversely, in the finance domain, neither group had high probabilities of using three specific payment approaches (paying providers for delivery of a specific EBI, paying for outcomes, lowering costs of specific EBIs—ST35, ST37, or ST38). Additionally, no IPO in either finance profile reported using ST36, penalties for failing to use EBIs.

Limitations and strengths

While this paper adds to the limited body of evidence about organizations focused on implementing evidence-based child-focused behavioral health interventions, we acknowledge a number of limitations. First, our analysis focused only on EBIs for child behavioral health. Yet to be explored are the dissemination and implementation activities used by organizations focused on interventions for health, adult behavioral health, child development, and a variety of other issues such as corrections. We drew our IPO list and corresponding readiness scores from NREPP in July 2015; a few months later, NREPP removed the readiness score. NREPP has been critiqued about some of its criteria for intervention inclusion in an “evidence-based” list, with some researchers feeling like the standards are not sufficiently rigorous [27]. Nonetheless, the IOM has noted the importance of NREPP as a central hub to distribute information about mental health EBIs [1] and NREPP is widely accessed and unique in both its detail and scope. Our respondents, key informants of IPOs themselves, reflect the perspectives of only one implementation “actor”—the organizations that exist to roll out EBIs [28]. Results would likely differ if reported from other implementation actors, such as agency directors or policy makers. Furthermore, we acknowledge that our IPO prevalence rates are underestimated due to the difficulty of discerning the existence of certain IPOs from the available contact information on NREPP, coupled with the fact that IPO identification was conducted by one member of our team.

We also acknowledge limitations in measurement. As reflected in prior research [6], capturing scale and cost of IPO activities is challenging, requiring the provision of ordinal ranges (for budget, staff size, providers trained) rather than precise amounts, which we learned that most IPOs cannot retrieve. Measures of organizational characteristics, budgets, and EBI readiness scores were obtained from independent, publicly available sites (and have been used in prior research); however, we do not have data on their validity. We used Powell, an established nomenclature listing and labeling implementation strategies, but respondents’ subjectively determined and reported their strategy use and success with the strategies. While Franks & Bory asked (a convenience sample) of intermediary organization informants to identify their tasks [6], our measurement of dissemination and implementation strategies was based on a published compilation used in studies of implementation in the Veterans Administration [23] but which we adapted to the IPO “actor” perspective. The “number of strategies” employed is a blunt measure. In this work and in the field at large, measurement of implementation strategies is in early stages. However, the sheer number of strategies used seems to matter, given recent findings that use of more strategies is associated with ability to deliver EBIs [29]. Because implementation strategies are generally not available through medical records or archival data, self-report is the current state of art for capturing their use.

Finally, we note that the OLS analysis is exploratory and that our relatively small sample size limits the generalizability of the LCA results. Specifically, it is likely that class results (in our case, two classes) and item-response probabilities would differ using larger samples, particularly given the large number of strategies available. However, we performed the LCA as a proof of concept for a means to examine implementation strategies, and in that regards, we found it to be a useful tool.

Discussion

EBIs cannot be delivered unless they are adopted by service-providing organizations. How EBIs reach providers, and in turn consumers, remains poorly understood. Thus, our project addresses a troubling gap in the current literature. IPOs operate globally and across the span of healthcare and are particularly important for psychosocial interventions such as behavioral health. Unlike medications, whose adoption is fostered by strategic outreach by the pharmaceutical industry, psychosocial interventions have lacked a centralized distribution source of information or technical assistance.

IPOs are big business. Collectively, with nearly two billion dollars in revenue, these organizations train thousands of clinicians each year. Our own research experience tells us that agency directors considering adoption of new interventions often ask first, “how much will this cost?” Training, training materials, supervision, and certification are the key sources of revenue. Information about the cost of working with an IPO remains difficult to determine, as evidenced by our IPO respondents stating that “it depends.”

Information about implementation strategies, for use by distribution agents such as IPOs or by individual agencies and providers, has only recently been compiled [23, 30]. Strategy compilations inform both implementation research and practice [31]. In 2017, SAMHSA added a new resource to NREPP, the NREPP Learning Center. The Learning Center offers resources to support the selection, implementation, evaluation, and sustainment of evidence-based programs and practices, along with supporting materials in the form of case studies, stories, and videos. A “how to implement” page leads viewers to the Powell et al. compilation of 68 implementation strategies organized into 6 categories which also served as the framework for data collection and analysis in the present study. Regrettably, as of this writing, the future of NREPP and the availability of its resources is unsure [32]. NREPP is no longer maintained. This may hamper the opportunity for providers and organizations to contact IPOs for implementation assistance.

Our results suggest that there is no uniform approach, no magic bullet, to implementation. Nearly all IPOs report using a large number of strategies—nearly three dozens—in their efforts to roll-out EBIs, results aligned with those of Rogal et al. (2017). Moreover, they use strategies in most of the domains identified in the Powell et al. compilation [23]. While many strategies in the Powell compilation were rarely endorsed, we did find some convergence in strategies most frequently used. These include training, formal implementation planning, providing ongoing consultation, engaging local champions, and developing implementation materials.

In contrast to some convergence in strategies used, we found little such convergence around which strategies are most helpful to their work. No single strategy was identified as among the most helpful by more than half of IPOs. Training, which was used by nearly all IPOs, was rated as among the most helpful by only 46% of IPOs. The other most frequently employed strategies were reported as among the most helpful by fewer than 10% of IPOs. This may reflect the complexity of implementing new interventions, or it may reflect that in the absence of knowing what works, implementers use many different approaches. These findings align with those reported by Franks & Bory, indicating that IPOs are engaged in tasks that they do not think are effective [6].

Conclusions

These findings raise a number of questions for further research. Whether proprietary or not-for-profit, most IPOs employ similar dissemination, training, and support strategies. Research needs to examine the effectiveness of these strategies. Given the high number of strategies used by IPOs, is success a function of the number of strategies used, or do particular strategies contribute disproportionately to success? Recent evidence suggests that adoption of EBIs may indeed be associated with using a high number of strategies. Moreover, successful implementation likely depends on contextual factors and actions by a number of actors. What is the unique contribution of an IPO and what is required from the adopting organization, their staff, and their service recipients? What IPO implementation strategies are effective in which organizational and policy contexts [33]? And why do IPOs report widespread use of implementation strategies in which they have little confidence of success. A priority question for the field is: which strategies are effective, in what context, and for what types of EBIs? Given their preeminent role in implementing evidence-based behavioral health interventions, researchers need to address ways to enhance their effectiveness by advancing understanding of strategy use, measurement, and effectiveness.