The role of implementation organizations in scaling evidence-based psychosocial interventions
Implementation Science volume 18, Article number: 24 (2023)
To bring evidence-based interventions (EBIs) to individuals with behavioral health needs, psychosocial interventions must be delivered at scale. Despite an increasing effort to implement effective treatments in communities, most individuals with mental health and behavioral problems do not receive EBIs. We posit that organizations that commercialize EBIs play an important role in disseminating EBIs, particularly in the USA. The behavioral health and implementation industry is growing, bringing the implementation field to an important inflection point: how to scale interventions to improve access while maintaining EBI effectiveness and minimizing inequities in access to psychosocial intervention.
We offer a first-hand examination of five illustrative organizations specializing in EBI implementation: Beck Institute for Cognitive Behavioral Therapy; Incredible Years, Inc.; the PAXIS Institute; PracticeWise, LLC; and Triple P International. We use the Five Stages of Small Business Growth framework to organize themes. We discuss practical structures (e.g., corporate structures, intellectual property agreements, and business models) and considerations that arise when trying to scale EBIs including balancing fidelity and reach of the intervention. Business models consider who will pay for EBI implementation and allow organizations to scale EBIs.
We propose research questions to guide scaling: understanding the level of fidelity needed to maintain efficacy, optimizing training outcomes, and researching business models to enable organizations to scale EBIs.
An evidence-based intervention (EBI) must be implemented at scale to make a meaningful public health impact . Fewer than 50% of adults and 20% of youth with mental health disorders receive any treatment [2, 3], and far fewer receive EBIs . Three pillars support EBI impact: (1) research evidence supporting effectiveness, (2) demand for the intervention, and (3) capacity to implement the intervention at scale . Research has yielded an array of EBIs , the field has partially succeeded at increasing demand for services , and yet insufficient capacity exists for implementing EBIs in the community.
Behavioral health and implementation are growth industries . Corporations and entrepreneurs are entering the behavioral health industry, especially in the USA . Behavioral health companies received over US $4.82 billion in private capital in 2021, representing a fourfold increase in private funding since 2019 . Mental health teletherapy (e.g., Talkspace, BetterHelp, Lyra) and smartphone apps (e.g., Headspace and Calm) are increasing in popularity [10, 11]. As of 2021, seven mental health companies are now considered “unicorns” (companies valued at more than US $1 billion) , demonstrating potential to scale behavioral health practices. To achieve high-quality services for all, EBI implementation must keep pace with the scale of service growth. The implementation field is at an important juncture, prompting this discussion on scaling while promoting equitable access to EBIs.
Dissemination and implementation organizations
Organizations that specialize in EBI dissemination and implementation (D&I) include the following: intermediary and purveyor organizations (IPOs) , Centers for Excellence , and community-academic partnerships . IPOs, referred to as “bridging factors” in implementation efforts, bridge the gaps between societal contexts, organizations, and EBIs [15, 16]. There is considerable overlap in IPO activities ; both organizations implement EBIs in the community . Purveyor organizations work to effectively implement specific program(s) , while intermediary organizations implement many EBIs by building capacity within an agency or system, such as through workforce development and standard setting . Intermediary organizations often operate locally, while purveyor organizations operate nationally/internationally. IPOs work together or separately, as well as with local providers, to implement EBIs in a local context. The logic models of IPOs posit that through the delivery of training programs and ongoing implementation supports, providers gain knowledge in delivering a program, providers deliver the program (ideally with fidelity), and client outcomes improve [19, 20]. This focus differentiates IPOs from companies that employ or contract providers to provide services (e.g., Equip, Lyra).
IPOs have grown in number and revenue in the USA [12, 19]. More researchers and practitioners are starting or joining organizations with a mission to increase access to EBIs (e.g., Single-Session Support Solutions; Be Braver; Lyssn; the Center for Implementation; Equip). A 2019 study estimated the annual revenue across 11 nonprofit IPOs at almost US $2 billion, suggesting that IPOs play a significant role in the implementation industry . The implementation industry is in a period of both proliferation (i.e., new companies being created) and consolidation. Companies, such as Empower Community Care, have begun acquiring other purveyor organizations (e.g., Multisystemic Therapy Services, Incredible Years, Inc., Functional Family Therapy), suggesting that the implementation industry is entering a phase of consolidation of purveyor organizations. Understanding the strategies of successful IPOs can guide more efficient design of future implementation organizations.
A few studies have described the implementation strategies and frameworks IPOs use to implement EBIs [12, 19,20,21,22]. IPO primary functions are staff training and consultation support, quality assurance (i.e., fidelity monitoring), resource mobilization, facilitating connections among partners, providing implementation support, policy and systems development, advocacy and visibility, and best practice model development [12, 21,22,23]. McWilliam et al.  described Triple P International’s five implementation phases as engagement, commitment and contracting, implementation planning, training and accreditation, and implementation and maintenance. Proctor et al. found that IPOs use an average of 32 implementation strategies . Education, planning, and quality improvement were most frequently endorsed, while financial, restructuring, and quality management were less frequently endorsed . Finally, Franks and Bory  discussed factors that contribute to the success of developing an intermediary organization, including a favorable political climate, access to technical assistance, and stable funding (often government).
To advance analysis beyond identification of strategies, the present paper discusses the opportunities and threats that affect IPOs’ decision-making as they grow. We organize themes using the Five Stages of Small Business Growth framework ; stages include existence, survival, success, take-off, and maturity. Through each of these stages, businesses use various strategies to generate profit both to stabilize and grow their business, and to achieve their mission. This paper discusses practical decisions implementation organizations make while aiming to increase access to EBIs.
To elucidate practical issues that arise as IPO’s scale, we provide first-hand reflections of five implementation organizations (EBI that the company scales is in parenthesis): (1) Beck Institute (Beck model of cognitive behavioral therapy); (2) Incredible Years, Inc. (Incredible Years), purchased by Empower Community Care in 2022; (3) PAXIS Institute (PAX Good Behavior Game); (4) PracticeWise, LLC (Modular Approach to Treatment of Children with Anxiety, Depression, or Conduct Problems [MATCH], Managing and Adapting Practice [MAP], PracticeWise Evidence-Based Services, Core Elements of Family Therapy); and (5) Triple P International (Triple P-Positive Parenting Program). The five organizations were selected due to their wide reach: their operating budgets are in the top quartile of IPOs , and the number of clinicians trained is in the top 10% of IPOs . Table 1 provides an overview of these organizations; Table 2 summarizes their D&I activities. These organizations can be considered an IPO.Footnote 1 The organizations primarily disseminate EBIs for youth mental health; the Beck Institute also disseminates EBIs for adults. Four out of five organizations are based in the USA, and one is based in Australia. All five organizations operate internationally and have trained providers in over 120 countries. The considerations discussed in this paper may be most relevant for countries in which businesses – rather than universities, governments, or nongovernmental organizations – conduct the majority EBI trainings.
We discuss considerations implementation organizations address across stages of business growth related to organization structures and cost of services. A survey was administered to all five organizations to elicit key themes (see Appendix). Questions inquired about organizational structures, key collaborators, balancing financial and D&I priorities, and general feedback. Co-authors from each organization submitted responses to survey items in writing. The first author grouped responses thematically and drafted the results. Based on the themes that emerged, we presented the themes according to the Five Stages of Small Business Growth framework . Of note, many of the themes presented occur across each stage of growth; we discuss the theme in the phase of growth most relevant to that theme. Member checking was conducted by having all authors review the themes presented in the manuscript . Based on their feedback, the first author revised the manuscript. This process continued iteratively until all authors agreed on the themes presented in this manuscript.
Five Stages of Small Business Growth
The first stage of small business growth is existence, where a company is created to deliver a product to customers . All five illustrative implementation organizations began either to create a more sustainable mechanism for disseminating their program or to enable interstate transfer of trainings and resources without bureaucratic barriers of a university setting. The activities conducted by implementation organizations are demanding and complex and typically outside university researchers’ interests or skills . In most industries, there is a labor specialization between the person who invents, manufactures, distributes, markets, and provides technical assistance for a product . Some argue that the lack of widespread EBI availability is due to underemphasizing dissemination and greater need for specialization . The organizations reviewed reported that using professors and graduate students as trainers was insufficient to meet EBI training demands. Benefits of IPOs existing separately from universities include a lack of university bureaucracy, organizational independence, and the ability to operate more efficiently on a faster timeline and larger scale .
The creation of an implementation organization to disseminate EBIs is not the only model for wide-reaching D&I. EBI scale-ups can happen because of partnerships between governments, researchers, and local organizations . Some behavioral health professionals are uncomfortable with commercializing psychosocial interventions . Reasons against privatizing public health programs include challenges in ensuring effectiveness, equity, and trust in service quality . Nonetheless, more professors may start behavioral health or implementation organizations as innovation and entrepreneurship become increasingly recognized as a metric for tenure and promotions, as more universities train staff and students to commercialize intellectual property, and as more people move from academia to industry [30, 31].
In each of the organizations reviewed, at least one of the program developers was involved in establishing the organization. Intervention developers’ roles in organizations range from having no role, to consulting roles (e.g., focusing on scientific evaluations, training, and program fidelity), and to being the organization head. Developers’ university affiliations also vary (e.g., full time, part-time, to emeritus), both between organizations and over time. When researchers work with implementation organizations, roles must be delineated. The role program developers play in organizations affects their perspective and priorities as collaborators in the organization’s implementation work.
Program developers must decide whether they want to create a new organization or to have their EBI be disseminated by an existing organization. The changing behavioral health industry led some developers to sell their organizations to larger organizations. For example, since its merger with Empower Community Care in 2022, Incredible Years operates independently; the developer is in charge of fidelity, training mentors and coaches, updating programs, and consulting on research and has reduced her time on business matters, marketing, and inventory management. The merger aims to increase the reach of program by taking advantage of economies of scale to reduce fixed costs. Although consolidation could lead to monopolization of prices, communities can benefit from having fewer IPOs to collaborate with .
If a founder decides to form a new organization, an early decision is whether to be a for-profit and/or a nonprofit company. The primary difference is the capital source: for-profit businesses can raise investments from private investors (with the expectation of a financial gain to the investor/s), whereas nonprofit businesses raise money from foundations, individuals, corporations, and governments (with the expectation of a social return). The Beck Institute is nonprofit, while Empower Community Care (owner of Incredible Years), the PAXIS Institute, PracticeWise, and Triple P International are for-profit. Triple P International is a Certified B Corporation®, which recognizes its commitment to conducting business in a way that creates public benefit and sustainable value, beyond seeking profit. Regardless of profit status, the mission of the five organizations is to improve the lives of people and communities using EBIs supported by an economically sustainable dissemination model. The implementation organizations reviewed can be considered social enterprises given their social mission .
The degree to which a board of directors and advisory boards play a role in implementation organizations’ work is an important research question. Although both boards provide strategic advice, boards of directors have fiduciary responsibilities and a greater legal status. In the USA, a board of directors is required for nonprofit corporations and is common with for-profit organizations. The Beck Institute and PracticeWise have both a board of directors and an advisory board that include members with various societal interests who advise organizations on local and global needs.
IP ownership–such as trademark (e.g., brand name, logo) and copyright (e.g., materials distributed)–needs to be established to mitigate disputes. EBIs include copyrighted materials, videos, online programs, manuals, workbooks, and handouts. IP ownership affects licensing agreement(s) required for implementation organizations. The IP owner may assign ownership or establish a licensing agreement with the organization to allow them to exercise specific rights associated with the IP (e.g., reproduce and distribute copyrighted materials). In return, the IP owner negotiates consideration for IP exchange (e.g., a sale price; royalty rate). IP owners can reserve IP rights (e.g., the right to prepare derivative works) or negotiate nonexclusive agreements (e.g., asset licensing to ongoing research). Triple P International reported that their exclusive license agreement supports quality control by having common training standards globally. Organizations may develop proprietary IP (e.g., introducing new trademarks, creating new or derivative works) to which the program developer may or may not have negotiated rights. Similarly, consideration needs to be given to whether local community organizations share copyright IP or have a royalty consideration if they are involved in cultural adaptations of programs. Copyright sharing may vary depending on the type and extent of involvement in adaptation.
Creative Commons licenses are an alternative model that provides free access to materials. From a business perspective, Creative Commons effectively trade consideration in the form of citation for limited use and reproduction of copyright. This citation may generate value (e.g., brand awareness, reputation) that may be monetized (e.g., higher salaries, consulting). However, many of the same costs are still required (e.g., creating, maintaining, updating, translating, adapting, and printing materials as well as and training and implementation support). These are “hidden” costs that rely on professionals volunteering their time or ongoing funding from universities, foundations, or grants. Open-access materials shift costs away from consumers and end users and to program creators and other funders. This shift creates a risk that programs do not evolve, or there is program drift as infrastructure is not available to support ongoing implementation, research, and development.
In the survival stage of small business growth, companies focus on the relation between revenue and expenses . Revenue is necessary for a company to stay in business. Business models of implementation organizations are designed based on their goals and purpose, with considerations on who will pay for the EBI. Two models include (1) business to business (B2B)—one business selling to another (e.g., the five organizations reviewed) and (2) business to consumer (B2C)—a business selling directly to a consumer . Many companies (e.g., Talkspace, BetterHelp, Headspace; Triple P International) have both B2B and B2C models. Regardless of the model, implementation organizations have a business mindset . Income enables organizations to scale their business and disseminate EBIs to wider groups. The five organizations rely less on research grants and more on revenue from trainings, materials sales, and by providing consultations to agencies adopting the EBI. Potential funders include foundation grants, nonprofit organizations (e.g., schools, behavioral health organizations, hospitals), businesses (e.g., insurance companies), or the government (e.g., to train providers in a public system or schools). Some organizations, such as the Beck Institute, have an endowment. Behavioral health startups often rely on venture capital and other private funds to sustain operations until they are profitable. The five organizations described reinvest a substantial portion of profits in developing new programs, including updating and adapting materials, maintaining online programs, and creating mechanisms to increase program implementation.
Organizations use standard business practices (e.g., strategic planning, goal setting, and budgeting). From a unit economics perspective, many company costs are fixed. For example, there is a standard licensing fee, yet implementing a program in a new region often requires an initial investment to understand and adapt the program to local context. Global operations incur additional costs for legal advice, translations, transportation, and currency fluctuations. Subscription models to provide ongoing services can minimize working capital costs and increase financial stability. Multiple product types can generate resilient revenue streams to avoid unstable funding structures that can jeopardize organization sustainability and capacity to grow . Diverse revenue streams can help organizations adapt to emerging research evidence. One-time costs are often easier to arrange and manage than recurring costs. It is not uncommon for organizations to receive “last minute” requests (e.g., “We have unspent funds that need to be expended by X date”). However, last minute requests are not generally a recipe for long-term success. Short-term relationships can “plant a seed,” but long-term partnerships are preferred. Many of the organizations reviewed have long-term partnerships with public mental health, education, and child welfare systems.
Should EBIs be distributed freely? A more nuanced question is as follows: who should bear EBI costs? Government initiatives or nonprofits can pay both for clinicians to be trained in EBIs (thus decreasing organizational training costs) and for clients to receive EBIs, such as in the Improving Access to Psychological Therapies initiative in the UK . Absent such funding, freemium models can allow organizations to offer some services at no cost to establish a foundation for future transactions . For example, PracticeWise offers some materials for free, whereas other resources require a subscription. Conversely, Triple P International post selected program materials on their provider website, which are free after providers have received training.
The five organizations reviewed are aware of the question, “How are we generating real economic value?” Asking “Is this program low cost?” is a relative question; low cost is compared to what, including doing nothing? Quantification of return on investment of training or service for the individual or organization can be challenging . Training benefits can occur at the client, provider, and organizational levels . For example, consider the value proposition of the PracticeWise Evidence-Based Services database. This service aids clinical decision-making by allowing providers to search a regularly updated online database with youth-specific summaries from published intervention research. By outsourcing research analysis and reporting functions to a central agency, organizations share the cost of these functions.
The implementation organizations reviewed often operate within the underfunded human and social services sectors. They are aware that cost is a barrier for providers. Due to an implementation organization’s need to be sustainable, potential consumers of services (i.e., providers, organizations) may be negatively impacted by high up-front prices driven by the cost of materials and experts to provide training and consultation. Although consultation calls increase training efficacy [38, 39], they increase the cost and provider training burden . Many organizations offer flexible pricing for special needs or when services are purchased in bulk. For some programs, the contracted trainers and supervisors (i.e., not the organization) set their fee rate for training and supervision.
Cost–benefit analyses suggest that EBIs produce social benefits, decreased involvement with the criminal justice system, and increased work productivity and participation . Cost-effectiveness is an important aspect of scalability [5, 42]. One study found CBT training by the Beck Institute cost US $0.18 per consumer . An economic analysis of EBI cost-effectiveness for child behavioral health problems found the cost–benefit ratio per participant served was positive for Incredible Years (US $5.65), MATCH (US $7.64), the Triple P system (US $9.71), and the Good Behavior Game (US $62.73) . As the behavioral health industry grows, it will be important to consider what entities will hold these organizations accountable for offering a cost-effective service [7, 44]. Multiple available EBIs may help control prices through a competitive market.
In the success stage of small business growth, companies either create stable practices or focus on creating structures that will allow them to grow . During this stage, marketing is important. Trademark IP (i.e., brand names) can be leveraged to build EBI awareness. Program brand name awareness can help clients, agencies, and providers identify interventions that work. Program trademarks are shorthand for a package of EBIs that can be studied for their efficacy and cost-effectiveness . Ideally, program reputation is based on research evidence in addition to positive consumer experience and marketing. Marketing can include traditional advertising, as well as persuading influencers (e.g., policymakers, professional bodies) and potential referral sources . Brand loyalty can lead to repeat customers and can ease agencies into further EBI adoption [47, 48]. The implementation organizations reported that professional conferences are a strategy to maintain provider brand awareness, visibility, and loyalty. The organizations reviewed believed that name recognition, and related consumer demand and awareness, helped enable them to scale their EBIs nationally and internationally.
Credentialing is one way that implementation organizations can generate value from trademark IP. Accreditation (aka certification) requirements vary for each organization, ranging from attending a training course to multiple tape session reviews by the implementation organization (see Table 2). Some organizations have multiple level of accreditation levels, with higher costs associated with higher levels. Negotiations of standards between the payor and organization are part of contracting, as local standards differ . Regardless of specific requirements, when a funder pays for clinicians to be accredited in an EBI, they assume that client outcomes will improve . Quality assurance mechanisms (e.g., supervision, fidelity monitoring) are important to ensure accreditation translates to improved client outcomes. Accreditation effectively establishes a trademark/service mark (X-certified therapist) licensed to the provider to promote their services as evidence-based. Completing a training and accreditation process has been found to be related to practitioners’ program use .
Branding can be misleading. A consumer perception of a brand and the accuracy of the information conveyed by the brand are not necessarily isomorphic. An EBI’s benefit can be decreased when marketing forces are shaped by factors other than program efficacy. Consumers need to differentiate between evidence-based and pseudoscientific brands . EBI registries distill information from hundreds of trials for consumers, but the registries have varied standards for qualifying as an EBI . Some promote empirically supported principles; others promote “branded” therapies . There is a risk that the field and consumers may focus on accentuating brand differentiation when there is little. Even some brands (e.g., MAP) distill common elements of other branded protocols . Given that not all EBIs have overlapping functions or providers (e.g., teachers vs. clinicians), higher level coordination between programs would maximize the benefit of multiple EBIs to the service system (e.g., the International Congress on Evidence-Based Parenting Support hosted by the Parenting and Family Research Alliance).
Advertising and lobbying are not a primary focus of the five organizations reviewed due to their brand recognition. Unlike other industries (e.g., pharmaceuticals) and many behavioral health organizations funded by private capital (e.g., BetterHelp, Talkspace, Cerebral), the five organizations spend few resources on direct-to-consumer marketing. Triple P International developed several social marketing and communication strategies for the Stay Positive public health campaign, which is part of the universal, multitiered Triple P system . However, most contracts are initiated by an interested party (e.g., government agency, schools, hospitals, mental health organizations) who has heard of the program through referrals or word of mouth. Some organizations (e.g., Triple P International and the PAXIS Institute) proactively work with organizations and policymakers to inform them of services; structure systems of adoption, implementation, and maintenance of their programs; and request funding for implementation initiatives. Brand awareness, as well as knowledge of and personal connection to a program developer, may affect policymakers’ funding decisions. If EBI funding decisions and availability are partly based on word of mouth and connections, a challenge for the field is to ensure equitable access to EBIs of people who live in areas without such connections.
It should be noted that the five implementation organizations reviewed rarely work directly with EBI recipients. Triple P International disseminates an online version of Triple P, which can be delivered directly to the consumer. The organizations reported that they improve more equitable access to EBIs by training providers in a variety of settings and levels of care (e.g., schools, primary care, community mental health) that serve underserved communities. Some implementation organizations (e.g., PracticeWise) use evidence-based approaches to address barriers to treatment and to increase their capacity to promote access. However, these efforts do not address structural barriers to mental health care [56, 57], such as cost, language, and distance to treatment facilities. Additional strategies beyond the existence of implementation organizations are needed to improve equitable access to EBIs.
In the take-off stage of small business growth, the business’s reach increases . For implementation organizations, delegation of providing trainings is an important step to increase the number of providers trained in an EBI. Training and ongoing consultation help EBIs be implemented as effectively as in randomized control trials . Although pro bono trainings are admirable, pro bono work is difficult to sustain at scale without government/nonprofit funding (including research grant funding). There may be a trade-off between offering less intensive training to increase program reach, versus offering more intensive training to increase program fidelity. Online trainings can help increase reach, although online trainings are improved with consultation, which may limit reach . Simply granting access to EBIs does not guarantee increased EBI use [58, 59]. Therefore, it is unlikely that free materials with Creative Commons licenses will lead to a meaningful increase in proper EBI use. Thus, less costly implementation support may lead to a loss of fidelity and sustainability .
For quality control, IPOs typically require rigorous training. Train-the-trainer models can help increase the supply of trainers . Some implementation organizations limit trainer numbers to maintain quality control and fidelity, and research suggests expert-led trainings are superior to those using train-the-trainer . However, at what point does the focus on quality control lead organizations to become EBI training gatekeepers? Again, the trade-off is between training fidelity and reach. Some organizations set up checks and balances to manage the risk of lower fidelity (e.g., reviewing sessions, limiting train-the-trainer to one generation, requiring recertification), while others require the use of quality assurance/improvement targets.
Implementation consultants help tailor a program to fit local culture, context, and cost constraints while maintaining program fidelity . For example, tailoring the Incredible Years programs to family background, education, and culture as well as child developmental status is an integral part of the Incredible Years accreditation process. Te Whānau Pou Toru is a Māori adaptation of Triple P collaboratively created with Māori tribal elders and Triple P program creators . The PAXIS Institute has a process to approve field-initiated modifications to the program consistent with the evidence base. Indeed, a core part of EBI training is to teach providers to implement programs flexibly with fidelity .
In the maturity stage, the business focuses on redesigning structures to account for increased revenue while trying to maintain their entrepreneurial spirit and innovation . Research efforts are especially useful to help implementation organizations stay competitive and avoid becoming obsolete . The five organizations reviewed maintain ongoing research partnerships to foster research-informed developments and independent evaluations. End- users of EBIs and implementation organizations can suggest ideas for research [63, 64]. Researchers are encouraged to be more market facing (i.e., understanding clinicians’ and clients’ needs), rather than being market led (i.e., letting business override research) . The organizations reviewed reported a commitment to evidence and integrity from initial development to evaluation of implementation outcomes. For example, Triple P International experienced consumer demand for a version of Triple P for parents with babies but delayed its release until there was sufficient evidence supporting its efficacy. Researcher partnerships with implementation organizations provide a clear path-to-market for innovations and help EBIs be “designed for dissemination” [5, 64]. Does establishing a purveyor organization decrease EBI research? Using Triple P as an example (Fig. 1), the opposite can occur; increased research activity followed, particularly in growth of independent evaluations.
Collaborations between researchers and implementation organizations can provide students with industry experience. For example, PracticeWise has involved students in community events, secondary data analysis, literature reviews, building and testing assessment and performance projects, building and piloting training materials, attending training and conferences, and strategic planning. These opportunities are valuable given how few training programs provide such experience [66, 67].
Conducting research with commercialized programs is a source of potential conflict of interest (COI). In academia, potential COI may arise from income generated by books, lectures, professional fees, etc. . In the context of implementation organizations, COI potentially occurs because of revenues made from royalties, consultancy fees, and organizations ownership. Potential COI does not inherently corrupt research results, but transparency and disclosure are essential, including on peer-reviewed and invited scientific publications, reports, conferences, websites, grant applications, and trial registries [69, 70]. Researchers who work with our illustrative organizations reported having policies to disclose potential COI . Researchers at universities also complete annual financial disclosure forms. The organizations reviewed reportedly have research integrity policies and procedures to safeguard against potential COI, similar to research universities.
Does involvement with purveyors stifle negative research results? The organizations reviewed reported valuing independent evaluation and implementation efforts to improve practices. For example, in its research agreements, PracticeWise cedes the right of independent authorship, requiring only advanced publication notice to allow time to formulate a response if desired. Many implementation organizations have specific policies to ensure that all results (including null or negative findings) are published wherever possible. Systematic replication and independent evaluation are widely considered the key to becoming recognized as an EBI. However, as implementation organizations are usually involved in training trial clinicians, independent evaluations may still involve some developer or purveyor involvement.
We identified key themes that affect implementation organization decision-making across the Five Stages of Small Business Growth, including corporate structures, IP licensing agreements, pricing, and balancing fidelity and reach. Franks and Bory  found that IPOs are increasingly aware of empirical implementation evidence and understand the concept of implementation science, but many IPO employees are not explicitly trained in D&I. Because implementation organizations are one effective strategy to scale EBIs, evaluations of the facilitators, barriers, and motivators of change within these organizations and the adopting agencies are critical to mitigate a potential gap between implementation research and practice.
To increase EBI reach, research should examine how to make access to programs more affordable, scalable, and effective. There is a need to examine (a) how much and what type of training and ongoing consultation is needed to maximize client outcomes and (b) what type and level of fidelity/program integrity is needed to maximize client outcomes and justify training cost [71, 72]. Is there a differential benefit in training new practitioners relative to retraining practicing practitioners? It is unclear whether certifications/accreditation is an effective way to increase EBI fidelity in the short and long term. Additional research on mechanisms of change will help organizations refine and improve trainings. For example, online and hybrid trainings, including training using artificial intelligence, could increase EBI reach.
Future research can examine how to coordinate implementation initiatives of IPOs and other implementation organizations within service systems. Which implementation outcomes are the most important? In what ways do IPOs serve as facilitators, and perhaps barriers, to implementation? How do agencies select programs to implement (e.g., personal contact, peer advocacy, social media, or published research)? Knowledge management and decision support may be facilitated through external organizations (e.g., intermediary organizations) or within an organization. Finally, research can examine how to increase consumer’s knowledge of how to seek EBIs.
There are several limitations to this paper that warrant discussion. First, the themes gathered in the manuscript were developed through a co-creation process with the organizations, rather than an independent qualitative analysis (potential bias). Additional themes may have emerged from a different approach. We limited our review to five organizations, which mostly focus on disseminating EBIs for youth and operate primarily from the USA and Australia. Other implementation organizations, including organizations based in other countries, may have different perspectives.
The behavioral health industry is growing, and diverse approaches are used to disseminate EBIs. Future research can examine business models to enhance equitable EBI reach. In the meantime, we expect that in a time of unprecedented need for mental health services, more implementation organizations will be established to help scale existing EBIs.
Availability of data and materials
We refer to these organizations simply as implementation organization because some authors involved with these organizations either did not think the term “purveyor organization” or “intermediary organization” captured their work (e.g., because they disseminate many–rather than a single–EBIs at an international–not a local–scale); they felt that the term IPO was “too much academic jargon,” or they generally disliked the IPO term.
Dissemination and implementation
Intermediary and purveyor organization
Paltiel AD, Schwartz JL, Zheng A, Walensky RP. Clinical outcomes of a COVID-19 vaccine: implementation over efficacy. Health Affair. 2020;40:42–52. https://doi.org/10.1377/hlthaff.2020.02054.
Whitney DG, Peterson MD. National and state-level prevalence of mental health disorders and disparities of mental health care use in children. Jama Pediatr. 2019;173:389–91. https://doi.org/10.1001/jamapediatrics.2018.5399.
Substance Abuse and Mental Health Services Administration. Key substance use and mental health indicators in the United States: results from the 2020 National Survey on Drug Use and Health. Rockville: Center for Behavioral Health Statistics and Quality, Substance Abuse and Mental Health Services Administration; 2021.
Bruns EJ, Kerns SEU, Pullmann MD, Hensley SW, Lutterman T, Hoagwood KE. Research, data, and evidence-based treatment use in state behavioral health systems, 2001–2012. Psychiatr Serv. 2016;67:496–503. https://doi.org/10.1176/appi.ps.201500014.
Proctor EK, Toker E, Tabak R, McKay VR, Hooley C, Evanoff B. Market viability: a neglected concept in implementation science. Implement Sci. 2021;16:98. https://doi.org/10.1186/s13012-021-01168-2.
Weisz JR, Kuppens S, Ng MY, Eckshtain D, Ugueto AM, Vaughn-Coaxum R, et al. What five decades of research tells us about the effects of youth psychological therapy: A multilevel meta-analysis and implications for science and practice. Am Psychol. 2017;72:79–117. https://doi.org/10.1037/a0040360.
DeAngelis T. Mental health, meet venture capital. Monit Psychol. 2022;51:56.
Chorpita BF. Questions about industry. Behavior Therapist. 2019;42:1–3.
DeAngelis T. As funding cools, venture capitalists shift investments in mental health. Monitor Psychol. 2023;54.
Wasil AR, Venturo-Conerly KE, Shingleton RM, Weisz JR. A review of popular smartphone apps for depression and anxiety: assessing the inclusion of evidence-based content. Behav Res Ther. 2019;123:103498. https://doi.org/10.1016/j.brat.2019.103498.
Heilweil R. Feeling anxious about coronavirus? There’s an app for that. Vox; 2020.
Franks RP, Bory CT. Who supports the successful implementation and sustainability of evidence‐based practices? Defining and understanding the roles of intermediary and purveyor organizations. In: McCoy KP, Diana A, editors. The science, and art, of program dissemination: strategies, successes, and challenges new directions for child and adolescent development. 2015. p. 41–56. https://doi.org/10.1002/cad.20112
Mettrick J, Harburger DS, Kanary PJ, Lieman RB, Zabel M. Building cross-system implementation centers: a roadmap for state and local child- and family- serving agencies in developing centers of excellence. Baltimore: The Institute for Innovation & Implementation, University of Maryland; 2015.
Drahota A, Meza RD, Brikho B, Naaf M, Estabillo JA, Gomez ED, et al. Community-academic partnerships: a systematic review of the state of the literature and recommendations for future research. Milbank Q. 2016;94:163–214. https://doi.org/10.1111/1468-0009.12184.
Moullin JC, Dickson KS, Stadnick NA, Rabin B, Aarons GA. Systematic review of the exploration, preparation, implementation, sustainment (EPIS) framework. Implement Sci. 2019;14:1. https://doi.org/10.1186/s13012-018-0842-6.
Lengnick-Hall R, Stadnick NA, Dickson KS, Moullin JC, Aarons GA. Forms and functions of bridging factors: specifying the dynamic links between outer and inner contexts during implementation and sustainment. Implement Sci. 2021;16:34. https://doi.org/10.1186/s13012-021-01099-y.
Franks RP, Bory CT. Strategies for developing intermediary organizations: considerations for practice. Fam Soc. 2018;98:27–34. https://doi.org/10.1606/1044-3894.2017.6.
Fixsen DL, Naoom SF, Blase KA, Friedman RM. Implementation research: a synthesis of the literature. 2005. Report No.: FMHI Publication #231.
Proctor EK, Hooley C, Morse A, McCrary S, Kim H, Kohl PL. Intermediary/purveyor organizations for evidence-based interventions in the US child mental health: characteristics and implementation strategies. Implement Sci. 2019;14:1–14. https://doi.org/10.1186/s13012-018-0845-3.
McWilliam J, Brown J, Sanders MR, Jones L. The Triple P implementation framework: the role of purveyors in the implementation and sustainability of evidence-based programs. Prev Sci. 2016;17:636–45. https://doi.org/10.1007/s11121-016-0661-4.
Oosthuizen C, Louw J. Developing program theory for purveyor programs. Implement Sci. 2013;8:23. https://doi.org/10.1186/1748-5908-8-23.
Davis L, Wong L, Bromley E. Brokering system change: a logic model of an intermediary-purveyor organization for behavioral health care. Psychiatr Serv. 2022;73:933–6. https://doi.org/10.1176/appi.ps.202100425.
Franks RP. Role of the intermediary organization in promoting and disseminating mental health best practices for children and youth: the Connecticut Center for Effective Practice. Emotion Behav Disord Youth. 2010;10:87–93.
Churchill NC, Lewis VL. The Five Stages of Small Business Growth. Harvard Business Review. 1983;
Birt L, Scott S, Cavers D, Campbell C, Walter F. Member checking: a tool to enhance trustworthiness or merely a Nod to validation? Qual Health Res. 2016;26:1802–11. https://doi.org/10.1177/1049732316654870.
Kreuter MW, M. C C, Bernhardt JM. Enhancing dissemination through marketing and distribution systems: a vision for public health. In: Brownson RC, Colditz Graham A, Proctor EK, editors. Oxford University Press; 2012. p. 213–22.
Kreuter MW, Bernhardt JM. Reframing the dissemination challenge: a marketing and distribution perspective. Am J Public Health. 2011;99:2123–7. https://doi.org/10.2105/ajph.2008.155218.
Chamberlain P, Roberts R, Jones H, Marsenich L, Sosna T, Price JM. Three collaborative models for scaling up evidence-based practices. Adm Policy Ment Hlth. 2012;39:278–90. https://doi.org/10.1007/s10488-011-0349-9.
Gollust SE, Jacobson PD. Privatization of public services: organizational reform efforts in public education and public health. Am J Public Health. 2006;96:1733–9. https://doi.org/10.2105/ajph.2005.068007.
Promotion & Tenure Innovation & Entrepreneurship. PTIE findings: expanding promotion and tenure guidelines to inclusively recognize innovation and entrepreneurial impact. Oregon State University; 2020.
Flaherty C. Professors are leaving academe during the Great Resignation. Inside Higher Ed; 2022;
Peredo AM, McLean M. Social entrepreneurship: a critical review of the concept. J World Bus. 2006;41:56–65. https://doi.org/10.1016/j.jwb.2005.10.007.
Kotler P, Pfoertsch W. Being known or being one of many: the need for brand management for business-to-business (B2B) companies. J Bus Ind Mark. 2007;22:357–62. https://doi.org/10.1108/08858620710780118.
Clark DM. Realizing the mass public benefit of evidence-based psychological therapies: the IAPT program. Annu Rev Clin Psychol. 2018;14:159–83. https://doi.org/10.1146/annurev-clinpsy-050817-084833.
Rietveld J. Creating and capturing value from freemium business models: a demand-side perspective. Strateg Entrep J. 2018;12:171–93. https://doi.org/10.1002/sej.1279.
Phillips PP, Phillips JJ. Return on Investment (ROI) Basics. Association for Talent Development; 2006.
Aguinis H, Kraiger K. Benefits of training and development for individuals and teams, organizations, and society. Annu Rev Psychol. 2009;60:451–74. https://doi.org/10.1146/annurev.psych.60.110707.163505.
Beidas RS, Edmunds JM, Marcus SC, Kendall PC. Training and consultation to promote implementation of an empirically supported treatment: a randomized trial. Psychiatr Serv. 2012;63:660–5. https://doi.org/10.1176/appi.ps.201100401.
Webster-Stratton CH, Reid MJ, Marsenich L. Improving therapist fidelity during implementation of evidence-based practices: Incredible Years program. Psychiatr Serv. 2014;65:789–95. https://doi.org/10.1176/appi.ps.201200177.
Okamura KH, Wolk CLB, Kang-Yi CD, Stewart R, Rubin RM, Weaver S, et al. The price per prospective consumer of providing therapist training and consultation in seven evidence-based treatments within a large public behavioral health system: an example cost-analysis metric. Front Public Health. 2018;5:21. https://doi.org/10.3389/fpubh.2017.00356.
Laynard R, Clark DM, Knapp M, Mayraz G. Cost-benefit analysis of psychological therapy. Natl Inst Econ Rev. 2007;202.
Milat AJ, King L, Bauman AE, Redman S. The concept of scalability: increasing the scale and potential adoption of health promotion interventions into policy and practice. Health Promot Int. 2013;28:285–98. https://doi.org/10.1093/heapro/dar097.
Washington State Institute for Public Policy, University of Washington Evidence-Base Practice Institute. Updated inventory of evidence-based, research-based, and promising practices: for prevention and intervention services for children and juveniles in the child welfare, juvenile justice, and mental health systems. 2019. Report No.: E2SHB2536–10.
Beauchamp D. Public health, privatization, and market populism. Qual Manag Health Ca. 1997;5:73–9. https://doi.org/10.1097/00019514-199705020-00008.
Higa-McMillan CK, Francis SE, Rith-Najarian L, Chorpita BF. Evidence base update: 50 years of research on treatment for child and adolescent anxiety. J Clin Child Adolesc Psychol. 2015;45:91–113. https://doi.org/10.1080/15374416.2015.1046177.
Payne A, Ballantyne D, Christopher M. A stakeholder approach to relationship marketing strategy. Eur J Market. 2005;39:855–71. https://doi.org/10.1108/03090560510601806.
Chaudhuri A, Holbrook MB. The chain of effects from brand trust and brand affect to brand performance: the role of brand loyalty. J Market. 2001;65:81–93. https://doi.org/10.1509/jmkg.188.8.131.5255.
Christopher M, Payne A, Ballantyne D. Relationship marketing: bringing quality customer service and marketing together. London: Butterworth-Heinemann; 1991.
Lengnick-Hall R, Willging C, Hurlburt M, Fenwick K, Aarons GA. Contracting as a bridging factor linking outer and inner contexts during EBP implementation and sustainment: a prospective study across multiple U.S. public sector service systems. Implement Sci. 2020;15:43. https://doi.org/10.1186/s13012-020-00999-9.
Seng AC, Prinz RJ, Sanders MR. The role of training variables in effective dissemination of evidence-based parenting interventions. Int J Ment Health Pr. 2006;8:20–8. https://doi.org/10.1080/14623730.2006.9721748.
Lilienfeld SO. Distinguishing scientific from pseudoscientific psychotherapies: evaluating the role of theoretical plausibility, with a little help from Reverend Bayes. Clin Psychol-Sci Pr. 2011;18:105–12. https://doi.org/10.1111/j.1468-2850.2011.01241.x.
Burkhardt JT, Schröter DC, Magura S, Means SN, Coryn CLS. An overview of evidence-based program registers (EBPRs) for behavioral health. Eval Program Plann. 2015;48:92–9. https://doi.org/10.1016/j.evalprogplan.2014.09.006.
Hofmann SG. Imagine there are no therapy brands, it isn’t hard to do. Psychother Res. 2019;1–3. https://doi.org/10.1080/10503307.2019.1630781
Chorpita BF, Daleiden EL. Mapping evidence-based treatments for children and adolescents: application of the distillation and matching model to 615 treatments from 322 randomized trials. J Consult Clin Psychol. 2009;77:566.
Sanders MR. Development, evaluation, and multinational dissemination of the Triple P-Positive Parenting Program. Annu Rev Clin Psychol. 2012;8:345–79. https://doi.org/10.1146/annurev-clinpsy-032511-143104.
Green JG, McLaughlin KA, Fillbrunn M, Fukuda M, Jackson JS, Kessler RC, et al. Barriers to mental health service use and predictors of treatment drop out: racial/ethnic variation in a population-based study. Adm Policy Ment Hlth. 2020;47:606–16. https://doi.org/10.1007/s10488-020-01021-6.
Andrade LH, Alonso J, Mneimneh Z, Wells JE, Al-Hamzawi A, Borges G, et al. Barriers to mental health treatment: results from the WHO World Mental Health surveys. Psychol Med. 2014;44:1303–17. https://doi.org/10.1017/s0033291713001943.
Frank HE, Becker-Haimes EM, Kendall PC. Therapist training in evidence-based interventions for mental health: a systematic review of training approaches and outcomes. Clin Psychol Sci Pract. 2020;27:e12330. https://doi.org/10.1111/cpsp.12330.
Shafran R, Clark DM, Fairburn CG, Arntz A, Barlow DH, Ehlers A, et al. Mind the gap: improving the dissemination of CBT. Behav Res Ther. 2009;47:902–9. https://doi.org/10.1016/j.brat.2009.07.003.
Kendall PC, editor. Flexibility within fidelity: breathing life into a psychological treatment manual. New York: Oxford University Press; 2021.
Keown LJ, Sanders MR, Franke N, Shepherd M. Te Whānau Pou Toru: a randomized controlled trial (RCT) of a culturally adapted low-intensity variant of the Triple P-Positive Parenting Program for indigenous Māori families in New Zealand. Prev Sci. 2018;19:954–65. https://doi.org/10.1007/s11121-018-0886-5.
Kendall PC, Beidas RS. Smoothing the trail for dissemination of evidence-based practices for youth: flexibility within fidelity. Prof Psychol Res Pr. 2007;38:13–20. https://doi.org/10.1037/0735-7028.38.1.13.
Rotheram-Borus MJ, Swendeman D, Chorpita BF. Disruptive innovations for designing and diffusing evidence-based interventions. Am Psychol. 2012;67:463–76. https://doi.org/10.1037/a0028180.
Ramsey AT, Proctor EK, Chambers DA, Garbutt JM, Malone S, Powderly WG, et al. Designing for accelerated translation (DART) of emerging innovations in health. J Clin Transl Sci. 2019;3:53–8. https://doi.org/10.1017/cts.2019.386.
Dodgson M. Towards the fully engaged university: the particularly Australian challenge. 2015. p. 1–9.
Proctor EK, Chambers DA. Training in dissemination and implementation research: a field-wide perspective. Transl Behav Med. 2017;7:624–35. https://doi.org/10.1007/s13142-016-0406-8.
Davis R, D’Lima D. Building capacity in dissemination and implementation science: a systematic review of the academic literature on teaching and training initiatives. Implement Sci. 2020;15:97. https://doi.org/10.1186/s13012-020-01051-6.
Chivers T. Does psychology have a conflict-of-interest problem? Nature. 2019;571:20–3. https://doi.org/10.1038/d41586-019-02041-5.
Sanders MR, Kirby JN, Toumbourou JW, Carey TA, Havighurst SS. Innovation, research integrity, and change: a conflict of interest management framework for program developers. Aust Psychol. 2020;55:91–101. https://doi.org/10.1111/ap.12404.
Cristea I-A, Ioannidis JPA. Improving disclosure of financial conflicts of interest for research on psychosocial interventions. Am J Psychiatry. 2018;75:541–2. https://doi.org/10.1001/jamapsychiatry.2018.0382.
Kendall PC, Frank HE. Implementing evidence-based treatment protocols: flexibility within fidelity. Clin Psychol: Sci Pract. 2018;25:e12271. https://doi.org/10.1111/cpsp.12271.
Regan J, Daleiden EL, Chorpita BF. Integrity in mental health systems: an expanded framework for managing uncertainty in clinical care. Clin Psychol Sci Pract. 2013;20:78–98. https://doi.org/10.1111/cpsp.12024.
The authors are grateful for Jinny Hong, Elizabeth A. McGuier, Nicole B. Gumport, Elizabeth Baik, and the UC San Diego Dissemination and Implementation Science Journal Club and Works in Progress for their input on the conceptualization of this idea.
Manuscript preparation was supported by the National Institute of Mental Health grants awarded to M. E. C. (F31MH123038) and B. F. C. (5R01MH112516, 1R34MH117206), grants from the William T. Grant Foundation and Wellcome Trust awarded to B. F. C., and the Centre of Excellence for Children and Families over the Life Course, Australian Research Council, awarded to M. R. S. (CE140100027).
Ethics approval and consent to participate.
Consent for publication
All authors approve the use of the data in the manuscript.
M. E. C. was an employee in the Parenting and Family Support Centre from July 2014 to December 2016; she has no current conflict of interest to report. E. L. D. and B. F. C. are co‐owners of PracticeWise, LLC, a behavioral health consulting company. B. F. C.’s ownership arrangement is approved by the University of California. M. R. S., the Parenting and Family Support Centre, is partly funded by royalties stemming from published resources of the Triple P-Positive Parenting Program, which is developed and owned by the University of Queensland (UQ). Royalties are also distributed to the Faculty of Health and Behavioural Sciences at UQ and contributory authors of published Triple P resources. Triple P International (TPI) Pty Ltd. is a private company licensed by UniQuest Pty Ltd. on behalf of UQ, to publish and disseminate Triple P worldwide. The authors of this report have no share or ownership of TPI. M. R. S. receives royalties and consultancy fees from TPI. J. M. is employed by Triple P International. A. R. M. is employed by Beck Institute for Cognitive Behavioral Therapy. J. S. B. is employed by Beck Institute for Cognitive Behavioral Therapy. She also receives royalties from Guilford Press. C. W. S. is developer of Incredible Years programs, and she is now an employee of Empower and is still clinical supervisor of Incredible Years. She is involved in consulting, training, dissemination, and clinical supervision for research of Incredible Years programs worldwide. C. A. is a mentor who provides training, consultation, and supervision for the Incredible Years parenting programs of the Incredible Years Inc. in California. D. D. E. is the president/CEO of PAXIS Institute which owns the trademarks and copyrights associated with the Good Behavior Game®/PAX Good Behavior Game and conducts the training and supports in multiple countries; PAXIS Institute does not conduct or finance the research on GBG outcomes. The PAXIS Institute requires study results be published—especially if the results indicate potential harm. J. A. P. is the founder and CEO of Evidn, LLC. P. C. K. receives royalties from the sales of published materials related to the treatment of youth, and his spouse operates and receives income from the publisher. The other authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
What does your organization do?
How does your organization seek to achieve sustainable dissemination of evidence-based interventions?
How you’ve balanced potential conflicts of interest when doing research?
Any short comings you see of your model/having purveyor organizations?
What would be the one “next needed” study to maximize sustainable implementation of evidence-based interventions in the context of purveyor organizations?
How has your organization balanced financial realities of running an organization with making programs affordable to clients?
How your organization works with stakeholders, policy makers, community organizations?
How your organization provides dissemination and implementation training opportunities that students wouldn’t get from a university setting?
Do you see there being a value of the program brand to increasing pull demand and consumer awareness of their program? If so, please describe.
Other learnings (successes and failures)?
Other key points
About this article
Cite this article
Crane, M.E., Kendall, P.C., Chorpita, B.F. et al. The role of implementation organizations in scaling evidence-based psychosocial interventions. Implementation Sci 18, 24 (2023). https://doi.org/10.1186/s13012-023-01280-5
- Evidence-based intervention
- Stages of Small Business Growth