- Study protocol
- Open Access
Measuring sustainment of prevention programs and initiatives: a study protocol
Implementation Science volume 11, Article number: 95 (2015)
Sustaining prevention efforts directed at substance use and mental health problems is one of the greatest, yet least understood, challenges in the field of implementation science. A large knowledge gap exists regarding the meaning of the term “sustainment” and what factors predict or even measure sustainability of effective prevention programs and support systems.
The U.S. Substance Abuse and Mental Health Services Administration (SAMHSA) supports a diverse portfolio of prevention and treatment grant programs that aim to improve population and individual level behavioral health. This study focuses on four SAMHSA prevention grant programs, two of which target substance abuse prevention at the state or single community level, one targets suicide prevention, and one targets prevention of aggressive/disruptive behavior in elementary schools. An examination of all four grant programs simultaneously provides an opportunity to determine what is meant by the term sustainment and identify and support both the unique requirements for improving sustainability for each program as well as for developing a generalizable framework comprised of core components of sustainment across diverse prevention approaches. Based on an analysis of qualitative and quantitative data of 10 grantees supported by these four programs, we will develop a flexible measurement system, with both general and specific components, that can bring precision to monitoring sustainment of infrastructure, activities, and outcomes for each prevention approach. We will then transform this system for use in evaluating and improving the likelihood of achieving prevention effort sustainment. To achieve these goals, we will (1) identify core components of sustainment of prevention programs and their support infrastructures; (2) design a measurement system for monitoring and providing feedback regarding sustainment within the four SAMHSA’s prevention-related grant programs; and (3) pilot test the predictability of this multilevel measurement system across these programs and the feasibility and acceptability of a measurement system to evaluate and improve the likelihood of sustainment.
This project is intended to improve sustainment of the supporting prevention infrastructure, activities, and outcomes that are funded by federal, state, community, and foundation sources.
Despite notable declines in the past 10 years in the USA, substance abuse among adolescents remains high; a third of tenth-graders and half of 12th graders have ever used an illicit drug, one in 15 high school seniors use marijuana daily and a quarter of seniors have had five or more drinks at one time in the last 2 weeks, a 10 % increase from 2011 to 2012 . Adolescent drug abuse represents a substantial economic burden to society. Annually, drug abuse costs $600 billion . Adolescent drug abuse also increases STD and HIV sex risk behavior with youths accounting for 39 % of all infections . In terms of mental health, suicide among youths between the ages of 10 and 24 years is the second highest cause of death, and rates have been increasing . Six percent of adolescent females and 2 % of males also attempt suicide each year, with 90 % of these youths having a diagnosable mental disorder . Attempts are five to six times more common among those with an abuse/dependence disorder compared to those without such disorders [5, 6]. The high prevalence of psychiatric mood diagnoses and symptoms, combined with drug and alcohol use, place adolescents at a significantly higher risk of completing suicide [7, 8].
There are numerous evidence-based programs, practices, and initiatives designed to prevent substance abuse and suicide . Specific prevention programs such as the Good Behavior Game (GBG) are cost effective in preventing drug and alcohol abuse and dependence disorders , criminal behaviors and antisocial personality disorder , suicide ideation and behavior , and HIV risk behavior ; this intervention is also highly cost effective and available for implementation [14, 15]. In contrast, programs focused on suicide are generally less definitive about their overall preventive effects. Other than our own work with GBG , few prevention programs have yet to demonstrate impact on both ideation and attempts, and no universal prevention program has demonstrated a significant reduction in suicide deaths .
Unfortunately, very few of these programs are routinely used, much less sustained when government funding comes to an end [9, 17]. The major research challenge we now face is not the lack of scientific knowledge about what works, but about how to integrate and maintain effective prevention programs, practices, policies, and principles in the institutions and communities charged with preventing drug abuse, sex risk behaviors, mental disorders, violence, and related outcomes.
Several models of research translation have been proposed over the years (see Damschroeder et al.  for a review). Many of these models consider sustainment to be the final stage of the process of implementation [19, 20], but the factors that predict sustainment are not well understood [21, 22]. In part, this may be attributed to a lack of consensus as to what constitutes sustainment and how to measure it. There are no uniform or agreed upon criteria for determining whether something has been sustained or not . This may be due to the fact that what is to be sustained differs from one program to the next. For instance, with respect to the community coalitions supporting drug and suicide prevention activities, some definitions of sustainment focus on the coalition itself while others focus on the activities and impacts of the coalition . Furthermore, with few exceptions [24–26], most studies reporting positive results have focused on earlier stages of implementation progress (exploration, adoption, routine use) and not on sustainment.
In addition to uncertainty as to how to define sustainment, there is a lack of consensus as to how to measure it. Chamberlain, Brown, and Saldana  developed the Stages of Implementation Completion (SIC), an eight-stage assessment tool developed as part of a large-scale randomized implementation trial. The stages range from engagement with the developers to practitioner competency and map onto three well-accepted phases of implementation—Pre-Implementation, Implementation, and Sustainability—the latter stage is currently only measured by a single stage 8 certification step. While the SIC is a measure of sustainment for a particular program, the ability of this instrument to measure intervention sustainment across different interventions has not yet been validated.
Another measure of sustainment is the Program Sustainability Assessment Tool , containing 40 items across eight sustainability domains, with five items per domain. The instrument developers reported high internal consistency reliability and some evidence of validity; however, the instrument has been used largely with evaluating chronic disease prevention programs and appeared to perform poorly with public health impact domains. They concluded that future research and evaluation work needs to be done to ascertain the validity and reliability of the instrument with different fields and types of interventions.
Federal agencies responsible for wide-scale delivery of prevention programs, including the Substance Abuse and Mental Health Services Administration (SAMHSA), routinely collect information from their grantees to monitor progress toward completion of goals and objectives. As part of their initial proposal for funding, all SAMHSA grantees are required to submit a plan for sustainment of the grantee’s activities once the funding has come to an end. SAMHSA programs currently rely on electronic data collection systems including the Transformation Accountability (TRAC) data collection system for SAMHSA’s Center for Mental Health Services (CMHS) programs, and the Coalition Online Management and Evaluation Tool (COMET) and the Performance Management Reporting Tool (PMRT) used by SAMHSA’s Center for Substance Abuse Prevention (CSAP). This information is used to provide feedback to grantees when there is evidence of failure to achieve goals and objectives. Currently, there is no empirical evidence that such feedback leads to an improvement in performance or increases the likelihood of sustainment.
While monitoring and feedback are recognized as important for prevention , much of the relevant science on feedback in health has involved improvement in clinical performance [29–32]. This includes clinical supervision and use of technology like electronic dashboards that monitor patient behavior and clinician activity [33–35]. Such feedback offers the clinician a better understanding of whether they are on course to achieve a successful outcome or need to alter their treatment strategy in order to improve the likelihood of a successful outcome. Similar measurement-based quality improvement (MBQI) strategies hold great promise for facilitating implementation and sustainment of evidence-based practices .
Aims and objectives
SAMHSA supports a wide array of prevention grant programs targeting mental, emotional, and behavioral disorders including substance abuse, suicide, and antisocial behavior. Each of SAMHSA’s prevention initiatives has specific sets of goals and objectives, and each has different prevention approaches to be sustained once support from SAMHSA is no longer available. We will examine four SAMHSA prevention grant initiatives simultaneously to determine what is meant by the term sustainment in order to identify and support both the unique requirements for improving sustainment for each program as well as for developing a generalizable framework comprised of core components of sustainment across diverse prevention approaches. Based on an examination of grantees supported by these four programs, we will develop a flexible measurement system for sustainability, with both general and specific components that can bring precision to monitoring the structures and processes for sustaining each prevention approach. We will then transform this measurement system into a format that can be used to efficiently evaluate and improve the likelihood of achieving sustainment of any grantee’s prevention efforts, regardless of source of funding. To achieve these goals, we will:
Identify core components and their interrelationships across time for sustainment of prevention programs and their support infrastructures
Using ethnographic interviews, administrative data and network analysis of 10 grantees within four SAMHSA programs (Strategic Prevention Framework—State Initiative Grants, Sober Truth on Preventing Underage Drinking [STOP Act], Garrett Lee Smith Suicide Prevention Program, and Prevention Practices in Schools), we will identify relevant sustainment components, structures and functioning for agencies/organizations, community coalitions, and state prevention service systems that host prevention programs. This will enable us to determine whether or not a program has been sustained; what features of the intervention, grantee organization, external environment, implementation process, and individuals involved were critical to supporting that sustainment; and which of these features are common across all four programs.
Design a measurement system for monitoring and providing feedback regarding sustainment
Building on the results of aim 1, we will determine what additional information is necessary for SAMHSA to collect to determine the level and predictors of sustainment, the means for this data collection, and how this information can be organized into a brief scoring system for evaluating and improving the likelihood of sustainment.
Pilot test the predictability of the Sustainment Measurement System (SMS) and the feasibility and acceptability of this system to evaluate and improve sustainment likelihood
Using retrospective and prospective data from a larger sample of 100 grantee programs and analytic models based on advanced missing data procedures, we will examine the measurement system’s short-term predictability of sustainment in previous cohorts of these four SAMHSA programs. We will also evaluate the feasibility and acceptability for grantees of converting the information obtained from this measurement system into a format that can be used to provide feedback to grantees that will enable them to evaluate and improve their own progress toward sustainment.
Funded by SAMHSA’s Center for Substance Abuse Prevention (CSAP), the Strategic Prevention Framework—State Initiative Grant (SPF-SIG) Program has three goals: (1) prevent the onset and reduce the progression of substance abuse, (2) reduce substance abuse-related problems, and (3) build prevention capacity and infrastructure at the state, tribal, territory, and community levels through SAMHSA’s Strategic Prevention Framework (SPF) steps. These SPF steps require that grantees (a) assess their prevention needs based on epidemiological data; (b) build their prevention capacity; (c) develop a strategic plan; (d) implement effective community prevention programs, policies, and practices; and (e) evaluate their efforts for outcomes. Throughout all the five steps, grantees must address issues of sustainment and cultural competence . Sustainment issues include the process through which a prevention system becomes a norm and is integrated into ongoing operations, particularly the statewide drug prevention block grants and prevention efforts at the local community level in that state. This infrastructure sustainment is vital to ensuring that prevention values and processes are firmly established, that partnerships are strengthened, and that financial and other resources are secured over the long term .
The Sober Truth on Preventing Underage Drinking (STOP) Act is a collaborative funded by the White House Office of National Drug Control Policy (ONDCP) and administered by CSAP. Eligible applicants are community coalitions with representation from 12 required sectors (Drug Free Communities Act of 1997, Public Law 105-20). This program works to achieve two goals: (1) establish and strengthen collaboration among communities, public and private non-profit agencies, and federal, state, local, and tribal governments to support the efforts of community coalitions working to prevent and reduce substance use among youths; and (2) reduce substance use among youths and, over time, reduce substance abuse among adults by addressing the factors in a community that increase the risk of substance abuse and promoting the factors that minimize the risk of substance abuse. STOP-Act-funded sites are eligible to receive additional mentoring grants to support new communities applying for STOP Act funding. Prominent sustainment issues include outcome sustainment, i.e., a continued reduction in substance use/abuse.
The Prevention Practices in Schools (PPS) Program is SAMHSA’s sole prevention grant that requires grantees to implement the Good Behavior Game (GBG), a classroom management strategy that involves helping children to learn how to work together through group contingent activities. Funded through SAMHSA’s Center for Mental Health Services (CMHS), GBG outcomes encompass the prevention goals of both CSAP—reduce substance abuse and smoking—and CMHS—reduce conduct disorder and suicidal ideation in youths. Eligible applicants for this program are local education agencies (school districts and tribal organizations); to date, 21 school districts and one tribal organization have been PPS grantees. Sustainment for PPS means that GBG would continue to be used in schools after SAMHSA funding has ended.
The Garrett Lee Smith Suicide (GLS) Prevention Program provides grant funding to states, tribes, and territories, as well as universities, state colleges, minority-serving institutions of higher learning, and community colleges. This program is community based and requires that funds be used by grantees for program development that directly address substance abuse and other behavioral health problems (e.g., depression), risks which are directly linked to suicide [7, 8, 38, 39]. GLS has six goals: (1) increased development and implementation of community-based suicide prevention programs; (2) training for recognition of at-risk behaviors; (3) improvement in access to and linkages with substance abuse and mental health services; (4) improvement and expansion of surveillance of suicide-related outcomes; (5) increased awareness of suicide as a public health problem; and (6) development and implementation of strategies for reducing stigma associated with services for mental health and suicide prevention activities. The most common prevention approaches across grantee communities and organizations have been gatekeeper training and screening programs to identify youths at risk . SAMHSA has also emphasized the need for community collaborations and asks all sites to evaluate how well community coalitions have been developed through GLS. Sustainment issues emphasize continued coalition support and delivery of programs that positively affect risk and protective factors for suicide as well as monitoring of outcomes .
As illustrated in Table 1 below, the four programs provide shared as well as unique aspects of the problems of monitoring public support for the phases of implementation. The four programs were selected by SAMHSA for investigation because they represent the agency’s new strategic vision to align programs with the delivery of integrated behavioral health services as part of the Affordable Care Act of 2011. Preventive interventions range from a single evidence-based program (GBG) to combined strategies (SPF-SIG and STOP Act) to best available program where evidence is still being accumulated from trials (GLS). Further, all of these four SAMHSA grant programs share the problem of assessing and attending to the building and maintaining of public support for each stage of implementation including sustainment. They allow the development of a general method with specific as well as shared characteristics across different programs.
The Consolidated Framework for Implementation Research (CFIR)  offers an overarching typology for implementation research and comprises five major domains: the intervention, inner and outer setting in which it is implemented, the individuals involved in implementation, and the process by which implementation is accomplished. An illustration of this framework and the components of each domain is provided in Fig. 1. Elements of sustainment can be measured across all five dimensions, e.g., for intervention, whether a particular program continues after funding; for process, whether a strategic prevention framework is used; for inner setting, whether a school system maintains the infrastructure to support a program; for outer setting, whether a community coalition continues to commit to a prevention plan; and for individuals, whether a superintendent maintains support. By including a dynamic perspective where changes in each of these five settings occur, this framework also enables us to test what factors are predictors of sustainment.
Although many factors influence the implementation of evidence-based practices (EBPs), researchers have consistently found that interpersonal contacts within and between organizations and communities are important influences on the adoption of new behaviors. Based on Diffusion of Innovations Theory  and Social Learning Theory , studies and meta-analyses have shown that both the influence of trusted others in one’s personal network and having access and exposure to external information are important influences on rates of adoption and implementation of innovative practices [42–44]. Social networks have also been viewed as an important characteristic of community coalitions [45–47]. Feinberg and colleagues  found that network cohesion to be positively associated and network centralization to be negatively associated with community readiness to engage in the Communities That Care community-based prevention coalition. Bess and colleagues  found that initial coalition participation in a youth violence prevention program was associated with a pre-existing network of interorganizational relations. Hence, we will pay particular attention to the social networks of organizations implementing SAMHSA-funded programs.
Overview of study design
The project will proceed in three phases. Phase 1 involves ethnographic fieldwork, guided by the CFIR framework in 10 selected sites across the four programs and the analysis of archival administrative data routinely collected by SAMHSA to evaluate grantee performance. Our aim in this phase is to characterize what sustainment means across programs and extract shared and unique characteristics. Phase 2 involves the creation of the new measurement system incorporating data already being routinely collected along with additional data identified in Phase 1. Phase 3 involves the collection and analysis of data from SAMHSA grantees to validate the system as a means of monitoring progress toward sustainment of coalition process and products.
In collaboration with SAMHSA’s CMHS and CSAP associate directors and senior program staff, we identified two to three grantees within each of the four SAMHSA-funded programs and solicited their participation in the proposed project. These 10 sites include two PPS grantees currently implementing the Good Behavior Game, three SPF-SIG grantees, two STOP Act grantees, and three GLS grantees. These sites were purposefully sampled  to reflect diversity with respect to race/ethnicity, geography, quality of evidence supporting funded activities (i.e., the extent to which they are evidence-based or “evidence-informed”) and perceived level of success in achieving sustainment of program activities, infrastructure, or outcomes.
During a 2–3-day visit at each site, investigators will conduct individual semi-structured interviews with the grantee principal investigator, the project coordinator, and a minimum of four key informants representing coalition or community partners purposefully sampled on the basis of the site PI’s assessment of level of engagement in the project (two least engaged and two most engaged). In addition to these detailed interviews, all members of a coalition will also be invited to complete a brief questionnaire containing questions relating to types of interactions among coalition members and their social network relations.
Participating phase I SAMHSA grantees will provide project investigators with a copy of the original grant proposal, progress reports to SAMHSA, minutes of meetings with SAMHSA representatives, and any reports or publications disseminated outside SAMHSA. The hour-long interviews will be conducted with the use of an interview guide and comprised of three parts: (1) a series of semi-structured questions relating to experience with implementing and sustaining the program; (2) a free list exercise ; and (3) a template  of CFIR domains and components. In particular, we are interested in knowing the following: (1) what, if anything, they wanted to sustain; (2) how they perceive whether it was, in fact, sustained or not; and (3) their ranking of key determinants of sustainment.
All coalition members identified by the site PI, program coordinator, and key informants will be asked to complete a brief web-based survey that includes the full sampling frame of the site coalition thought to engage in the SAMHSA-funded initiative. They will then be asked how long they have known these individuals (in years) and to indicate whether (1) they had worked with each other member in the past year on any issue; (2) worked together in the past year on SAMHSA-funded initiative, program or practice issues; and, if yes, then (3) what types of collaboration each relationship involved (prompting for six areas, including advocacy and policy work, information sharing, program delivery, resource sharing, service delivery, or training/education; and (4) which of the defined members they considered a leader or innovator in the SAMHSA-funded initiative, program, or practice.
Data quality and management
All interviews will be digitally recorded and transcribed. Interviewers will then compare transcripts with digital records to insure accuracy of transcription. All field notes, interview transcripts, and interviewer notes summarizing interview and focus group experience will be entered into Dedoose . A data accounting and back-up system will be instituted to keep track of, and facilitate access to, all electronic and hard-copy data.
To insure credibility of findings and enhance the validity and reliability of data collected, all interviews will be reviewed by at least two members of the research team. As described in detail below, consensus on coding and coding procedures and modifications to coding books will occur through regular team meetings. When possible, study results will be presented to informants and other study participants, enabling them to provide comment of results and suggest modifications or additional avenues of investigation. An audit trail of data collected as well as memos and minutes of team meetings, indicating time, place, persons providing information, and persons collecting or analyzing information, will be kept throughout the study.
Qualitative data analysis
Three types of qualitative analysis will be used with the data collected during this phase. First, using a methodology of “Coding Consensus, Co-occurrence, and Comparison” developed for prevention research qualitative analysis , field notes and interview transcripts will be analyzed in the following manner. Each investigator will review this material and prepare short descriptive statements or “memos” to document initial impressions of topics and themes and their relationships and to define the boundaries of specific codes (i.e., the inclusion and exclusion criteria for assigning a specific code) . Segments of text in field notes and interview transcripts will be assigned codes based on a priori (i.e., from the interview guide) or emergent themes (also known as open coding ). Lists of codes developed by each investigator will be matched and integrated into a single codebook. Each text will be independently coded by at least two investigators. Disagreements in assignment or description of codes will be resolved through discussion between investigators and enhanced definition of codes. The final list of codes or codebook will consist of a numbered list of themes, issues, accounts of behaviors, and opinions that relate to program structure, function, development, and sustainment. With the final coding structure, two investigators will separately review transcripts to determine level of agreement in the codes applied. A level of agreement in the codes applied ranging from 66 to 97 % depending on level of coding (general, intermediate, specific) indicates good reliability in qualitative research . Based on these codes, Dedoose will be used to generate a series of categories arranged in a treelike structure connecting text segments grouped into separate categories of codes or “nodes.” These nodes and trees will be used to further the process of axial or pattern coding  to examine the association between different a priori and emergent categories. Through the process of constantly comparing these categories with each other, the different categories will be further condensed into broad themes using a format that places program formation, structure, and functioning within the framework of the site’s organizational and system context.
Second, the free lists of characteristics of and requirements for sustainment will be tabulated by counting the number of respondents who mentioned each item and then ordering in terms of frequency of responses. Multidimensional scaling analysis  will then be used to identify common and unique characteristics in each of the four SAMHSA-funded programs believed to be associated with sustainment.
Finally, a matrix of sustainment characteristics and requirements will be developed for each of the four SAMHSA-funded programs with a list of the CFIR domains and components on one dimension and the classification of sustainment (e.g., infrastructure, intervention, and outcomes) on the other dimension. Both the organization and content of subgroup matrices will be compared to identify features of coalitions that influence the likelihood of achieving sustainment on each dimension that are specific to particular SAMHSA programs and features common to all four programs . Comparison of these matrices will also enable project investigators to compare the project director, coordinator, and coalition member perspectives elicited during extended semi-structured interviews. The project investigators will then select a set of common and specific elements to be incorporated into the design of a heuristic model of sustainment.
Social network analysis
The matrix of ties used to analyze advice networks will be constructed from data collected from the web-based survey, supplemented by data collected during the qualitative interviews . The social network analysis will proceed in three stages: network visualization, structural analysis, and statistical analysis of outcomes. The network visualization will be accomplished using NetDraw 2.090. The spring embedder routine will be used to generate the network visualizations . Structural analyses will then be conducted on these network data using Ucinet for Windows, Version 6 . Several network level measures of structure will be assessed, including total number of ties, network size, density (the number of reported links divided by the maximum number of possible links), average distance between nodes, and the number of components (i.e., unique sub-networks). To assess status and interconnectivity within the network, we will calculate degree centrality for incoming ties (being nominated by alters) and outgoing ties (nominating alters). In-degree and out-degree centrality scores assess the relative status of a given node. We will also examine several other measures of network status, including between-ness, closeness, and eigenvector centrality. Eigenvector centrality also allows one to examine in- relative to out-ties, but in- and out-degree centrality correspond directly to counts of nominations by and toward an actor, and as such have a straight-forward substantive interpretation, which eigenvectors lack. Homophily (i.e., likeness between individuals in a network based on specified criteria) data will be assessed on two key variables of interest, the SAMHSA program funding the grantee and sustainment (sustained, not-sustained). Homophily scores can be regarded as the proportion of individuals in a person’s network who share a characteristic with that individual.
During this phase, we will identify data already being collected by SAMHSA corresponding to the relevant CFIR domains and components identified in phase 1. These data will be found in the Transformation Accountability (TRAC) data collection system for the GLS and PPS grantees, the Coalition Online Management and Evaluation Tool (COMET) for the STOP Act grantees, and the Performance Management Reporting Tool (PMRT) for the SPF-SIG grantees. This identification will include sustainment components that are collected across all four programs and components that are unique to each program.
Second, we will identify data relevant to sustainment not routinely collected by SAMHSA. This will be accomplished by comparing the matrix for relevant data elements identified in phase 1 with the list of common and unique sustainment data elements identified in the first activity of phase 2.
Third, we will design the Sustainment Measurement System (SMS), which integrates data currently being collected as part of existing SAMHSA data collection and reporting systems, and data that can be used to assess progress toward and likelihood of sustainment of project infrastructure, process (i.e., interventions and activities supported by the infrastructure), and outcomes. The SMS will be similar to the Stages of Implementation Completion (SIC) measure described above but involve ordinal scales with three or four categories so that sophisticated Item Response Theory (IRT) analyses can be conducted (see “Data analysis” section below). Like the SIC, it will also have separate sections pertaining to different roles in implementation, e.g., grantee administration, community coalition, and program home-site coordinator. In this instance, the tool will consist of all the elements identified in phase 1 as being relevant (i.e., a potential predictor or requirement) to sustainment of project infrastructure, process, and outcomes. The tool will also include the respondent’s assessment of whether or not sustainment has been achieved with respect to each of these categories, weighted on the basis of priority assigned by respondent to each category. Respondents will be asked to indicate what sustainment components are present and when key benchmarks of process were achieved.
A prototype measurement system will be developed that includes (1) data requesting (e.g., by whom, when); (2) data integration (e.g., with SAMHSA and additional data); (3) analytics (e.g., index to predict sustainment); and (4) visualization (e.g., profile to identify strengths and weaknesses). This will be developed in collaboration with SAMHSA as well as grantees to ensure utility and usability.
All site directors and collaborators or coalition representatives of all of the current grantees funded by the four SAMHSA programs will be invited to participate in this phase of the project. This will include all 21 grantees funded by Prevention Practices in Schools, 53 states and tribes funded by the Garrett Lee Smith Suicide Prevention Program, 35 SPF-SIG grantees, and 120 STOP Act grantees (estimated total site n = 230, see Table 1). Eligibility for study participation includes the following: (1) project is currently being funded or funded within the past 2 years by one of the four SAMHSA programs (PPS, SPF-SIG, STOP Act, or GLS), (2) project has submitted an annual progress report to SAMHSA for a minimum of 1 year, and (3) project is expected to end funding within this R34 grant period. There are 188 grantees close to the sustainability stage; we anticipate 140 responses in Years 02 and 100 at the 6-month and 12-month follow-up.
Each eligible SAMHSA-funded grantee will be invited to complete a web-based survey during Years 2 and 3. The PI or program coordinator will be asked to provide permission to access data already submitted to SAMHSA as part of its mandatory progress reporting system or procedures. These data will be limited to only that information relevant to the SMS. The survey will consist of a series of questions relating to SMS elements that are not currently or have not been previously collected by SAMHSA. These would include information relating to social networks of project coalitions or collaborators and other domains and components of the CFIR found in Phase 2 to be potentially relevant to the sustainment of program structure, process, and outcomes.
Using Mplus 7.11  and the R statistical package, a series of statistical analyses will be conducted to assess which factors appear to be strongest determinants of sustainment, taking account of the time to event (i.e., different lengths of follow-up since program termination). Since sustainment involves multiple subdimensions and shared as well as unique components, we will first examine one- and two-dimensional models of the latent structure of sustainment items, which are measures on three- to four-point scales in order to conduct item response theory (IRT) analyses. One-dimensional latent factor structure models of all items across infrastructure, process, interventions, and outcomes will be attempted first, but we anticipate that more complex models will be required. Specifically, we propose using two types of bifactor models for IRT analyses  since these decompose each item into a common factor (i.e., shared sustainment) and a second specific factor relevant to that particular subdimension or SAMHSA program. In bifactor models, the common factor score for an item can be used to assess the level of sustainment shared across all SAMHSA programs, while the specific factor characterizes that second dimension. For our first bifactor model, we would characterize how an item’s specific loading involves that item’s position in the CFIR framework (e.g., inner, outer, process, intervention, person). The second bifactor model would characterize each specific factor loading as pertaining to that particular SAMHSA program and therefore can be used to rank grantees within each grant program. A third analysis will integrate these into a comprehensive model that includes covariates and time.
We will then use the predictors from our Sustainability Measurement System in a latent variable model with the outcome being the latent common sustainment score from the bifactor model described above. Continuous measures, including the time that the grant was originally funded and categorical measures such as which of the four grant programs provided funding will be used to assess developmental as well as unique versus specific predictors in these analyses. Items that are specific to particular programs can be treated as informative indicators themselves or “missing at random” and thereby all measures across all grantees can be included in analyses.
The project is innovative in three specific respects. First, unlike other projects that focus on only one practice or program, we will be simultaneously examining sustainment of infrastructure, activities, and outcomes in four different sets of SAMHSA-funded programs. This will give us a rare opportunity to identify a set of common elements of sustainment that can be used to generate a model and set of testable hypotheses that apply to a broad array of drug abuse/mental disorder/suicide prevention programs, practices, and initiatives, regardless of objectives, outcomes, and infrastructure to achieve these outcomes. Second, we are developing a measure of sustainment that can be used validly and reliably across this broad array of programs, practices, and initiatives with varying levels of evidence to support their effectiveness. This will enable us to determine whether the extent to which a program or practice is evidence-based or evidence-informed determines whether it can be sustained [19–21]. Third, we will tailor this measure so that it can be used to monitor progress toward sustainment and provide feedback to stakeholders as to how to increase the likelihood of sustainment. This measurement system will thus have use as a tool for program management as well as research purposes. Although this project targets programs funded by SAMHSA, the work should have general applicability across diverse federal, statewide, and local prevention implementation initiatives.
CFIR, Consolidated Framework for Implementation Research; CMHS, Center for Mental Health Services; CSAP, Center for Substance Abuse Prevention; SAMHSA, Substance Abuse and Mental Health Services Administration; SIC, Stages of Implementation Completion
Johnston LD, O’Malley PM, Bachman JG, Schulenberg JE. Monitoring the future national results on drug use: 2012 overview, key findings on adolescent drug use. Ann Arbor: Institute for Social Research, University of Michigan; 2013.
National Institute on Drug Abuse. Principles of drug abuse treatment: a research based guide, Third Edition. USDHHS/NIH Publication No. 12-4180; 2012.
Patrick ME, O’Malley PM, Johnston LD, Terry-McElrath YM, Schulenberg JE. HIV/AIDS risk behaviors and substance use by young adults in the United States. Prev Sci. 2012;13(5):532–8.
Centers for Disease Control and Prevention [Internet]. Atlanta: National Center for Injury Prevention and Control; 2016. Web-based injury statistics query and reporting system (WISQARS). [Cited 2016 June 9] Available from: https://www.cdc.gov/injury/wisqars/.
Nock MK, Green JG, Hwang I, McLaughlin KA, Sampson NA, Zaslavsky AM, et al. Prevalence, correlates, and treatment of lifetime suicidal behavior among adolescents: results from the National Comorbidity Survey Replication Adolescent Supplement. JAMA Psychiatry. 2013;70(3):300–10.
Kessler RC, Borges G, Walters EE. Prevalence of and risk factors for lifetime suicide attempts in the National Comorbidity Survey. Arch Gen Psychiatry. 1999;56:617–26.
Goldston DB, Daniel SS, Erkanli A, Reboussin BA, Mayfield A, Frazier PH, et al. Psychiatric diagnoses as contemporaneous risk factors for suicide attempts among adolescents and young adults: developmental changes. J Consult Clin Psychol. 2009;77(2):281–90.
Goldston DB, Walrath CM, McKeon R, Puddy RW, Lubell KM, Potter LB, et al. The Garrett Lee Smith suicide prevention program. Suicide Life Threat Behav. 2010;40(3):245–56.
O’Connell ME, Boat T, Warner KE. Preventing mental, emotional and behavioral disorders among young people: progress and possibilities. Washington, DC: National Academies Press; 2009.
Kellam SG, Brown CH, Poduska JM, Ialongo NS, Wang W, Toyinbo P, et al. Effects of a universal classroom behavior management program in first and second grades on young adult behavioral, psychiatric, and social outcomes. Drug Alcohol Depend. 2008;95 Suppl 1:S5–S28.
Petras H, Kellam SG, Brown CH, Muthén BO, Ialongo NS, Poduska JM. Developmental epidemiological courses leading to antisocial personality disorder and violent and criminal behavior: effects by young adulthood of a universal preventive intervention in first- and second-grade classrooms. Drug Alcohol Depend. 2008;95 Suppl 1:S45–59.
Wilcox HC, Kellam SG, Brown CH, Poduska JM, Ialongo NS, Wang W, et al. The impact of two universal randomized first- and second-grade classroom interventions on young adult suicide ideation and attempts. Drug Alcohol Depend. 2008;95 Suppl 1:S60–73.
Kellam SG, Wang W, Mackenzie AC, Brown CH, Ompad DC, Or F, et al. The impact of the Good Behavior Game, a universal classroom-based preventive intervention in first and second grades, on high-risk sexual behaviors and drug abuse and dependence disorders into young adulthood. Prev Sci. 2014;15 Suppl 1:S6–18. doi:10.1007/s11121-012-0296-z.
Kellam SG, Mackenzie AC, Brown CH, Poduska JM, Wang W, Petras H, et al. The good behavior game and the future of prevention and treatment. Addict Sci Clin Pract. 2011;6(1):73–84.
Poduska J, Kellam S, Brown CH, Ford C, Windham A, Keegan N, et al. Study protocol for a group randomized controlled trial of a classroom-based intervention aimed at preventing early risk factors for drug abuse: integrating effectiveness and implementation research. Implement Sci. 2009;4:56.
Brown CH, Wyman PA, Brinales JM, Gibbons RD. The role of randomized trials in testing interventions for the prevention of youth suicide. Int Rev Psychiatry. 2007;19(6):617–31. PMID: 18092240.
Spoth R, Rohrbach LA, Greenberg M, Leaf P, Brown CH, Fagan A, et al. Addressing challenges for the next generation of type 2 translation research: the translation science to population impact (TSci2PI) framework. Prev Sci. 2013;14(4):319–51. doi:10.1007/s11121-012-0362-6.
Damschroeder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci. 2009;4:50.
Aarons GA, Hurlburt M, Horwitz SM. Advancing a conceptual model of evidence-based practice implementation in public service sectors. Adm Policy Ment Health. 2011;38:4–23.
Greenhalgh T, Robert G, Macfarlane F, Bate P, Kyriakidou O. Diffusion of innovations in service organizations: systematic review and recommendations. Milbank Q. 2004;82(4):581–629.
Chambers DA, Glasgow RE, Stange KC. The dynamic sustainability framework: addressing the paradox of sustainment amid ongoing change. Implement Sci. 2013;8:117. doi:10.1186/1748-5908-8-117.
Proctor EK, Landsverk J, Aarons G, Chambers D, Glisson C, Mittman B. Implementation research in mental health services: an emerging science with conceptual, methodological, and training challenges. Adm Policy Ment Health. 2009;36(1):24–34.
Benz J, Infante A, Oppenheimer C, Scherer H, Wilson W. Developing a conceptual framework to assess the sustainability of community coalitions post-federal funding. Chicago: National Opinion Research Center; 2011.
Feinberg ME, Bontempo DE, Greenberg MT. Predictors and level of sustainability of community prevention coalitions. Am J Prev Med. 2008;34(6):495–501.
Gloppen KM, Arthur MW, Hawkins JD, Shapiro VB. Sustainability of the communities that care prevention system by coalitions participating in the Community Youth Development Study. J Adolesc Health. 2012;51(3):259–64.
Rhew IC, Brown EC, Hawkins JD, Briney JS. Sustained effects of the communities that care system on prevention service system transformation. Am J Public Health. 2013;103(3):529–35.
Chamberlain P, Brown CH, Saldana L. Observational measures of implementation progression community based settings: the Stages of Implementation Completion (SIC). Implement Sci. 2011;6:116.
Luke D, Calhoun A, Robichaux CB, Elliott MB, Moreland-Russell S. The program sustainability assessment tool: a new instrument for public health programs. Prev Chron Dis. 2014;11:130184. doi:10.5888/pcd.11.130184.
Asch SM, McGlynn EA, Hogan MM, Hayward RA, Shekelle P, Rubenstein L, et al. Comparison of quality of care for patients in the Veterans Health Administration and patients in a national sample. Ann Intern Med. 2004;141(12):938–45.
Hysong SJ, Khan MM, Petersen LA. Passive monitoring versus active assessment of clinical performance: impact on measured quality of care. Med Care. 2011;49(10):883–90.
Institute of Medicine. Committee on Quality of Health Care in America: crossing the quality chasm: a new health care system for the 21st century. Washington: National Academies Press; 2001.
Institute of Medicine. Committee on Quality of Health Care in America: performance measurement: accelerating improvement. Washington DC: National Academies Press; 2006.
Bearman SK, Weisz JR, Chorpita BF, Hoagwood K, Ward A, Ugueto AM, et al. More practice, less preach? The role of supervision processes and therapist characteristics in EBP implementation. Adm Policy Ment Health. 2013;40(6):518–29. doi:10.1007/s10488-013-0485-5.
Persell SD, Kaiser D, Dolan NC, Andrews B, Levi S, Khandekar J, et al. Changes in performance after implementation of a multifaceted electronic-health-record-based quality improvement system. Med Care. 2011;49(2):117–25.
Seidman E, Chorpita BF, Reay WE, Stelk W, Garland AF, Kutash K, et al. A framework for measurement feedback to improve decision-making in mental health. Adm Policy Ment Health. 2010;37(1-2):128–31.
Hermann RC, Chan JA, Zazzali JL, Lerner D. Aligning measurement-based quality improvement with implementation of evidence-based practices. Adm Policy Ment Health. 2006;33(6):636–45.
SAMHSA [Internet]. Rockville, MD; 2013. Strategic prevention framework components [cited 2016 June 4] Available from http://www.samhsa.gov/capt/applying-strategic-prevention-framework.
Brent DA, Baugher M, Bridge J, Chen J, Beery L. Age and sex-related risk factors for adolescent suicide. J Am Acad Child Adolesc Psychiatry. 1999;38(12):1497–505.
Gould MS, Greenberg T, Velting DM, Shaffer D. Youth suicide risk and preventive interventions: a review of the past 10 years. J Am Acad Child Psychiatry. 2003;42(4):386–405.
Rogers EM. Diffusion of innovations. 5th ed. New York, NY: Free Press; 2003.
Bandura A. Social foundations of thought and action: a social cognitive theory. Prentice Hall: Englewood Cliffs, NJ; 1986.
Valente TW. Network interventions. Science. 2012;337:49–53.
Valente TW, Chou CP, Pentz MA. Community coalitions as a system: effects of network change and adoption of evidence-based substance abuse prevention. Am J Public Health. 2007;97(5):880–6.
Valente TW, Davis RL. Accelerating the diffusion of innovations using opinion leaders. Ann Am Acad Polit Soc Sci. 1999;566:55–67.
Bess KD, Speer PW, Perkins DD. Ecological contexts in the development of coalitions for youth violence prevention: an organizational network analysis. Health Educ Behav. 2012;39(5):526–37.
Kegler MC, Rigler J, Honeycutt S. How does community context influence coalitions in the formation stage? A multiple case study based on community coalition action theory. BMC Public Health. 2010;10:90.
Son J, Lin N. Social capital and civic action: a network-based approach. Soc Sci Res. 2008;37:330–49.
Feinberg ME, Riggs NR, Greenberg MT. Social networks and community prevention coalitions. J Prim Prev. 2005;26(4):279–98.
Palinkas LA, Horwitz SM, Green CA, Wisdom JP, Duan N, Hoagwood KE. Purposeful sampling for qualitative data collection and analysis in mixed method implementation research. Admin Policy Ment Health. 2015;42:533–44. doi:10.1007/s10488-013-0528-y.
Weller SC, Romney AK. Systematic data collection. Newbury Park, CA: Sage; 1988.
Crabtree BF, Miller WL. A template approach to text analysis: developing and using codebooks. In: Crabtree BF, Miller WL, editors. Doing qualitative research, vol. 3. Thousand Oaks, CA: Sage Press; 1992. p. 93–109.
Fraser D. QSR NVivo NUD*IST Vivo reference guide. Melbourne: QSR International; 2000.
Willms DG, Best JA, Taylor DW, Gilbert JR, Wilson DMC, Lindsay EA, et al. A systematic approach for using qualitative methods in primary prevention research. Med Anthropol Q. 1990;4(4):391–409.
Miles MB, Huberman AM. Qualitative data analysis: an expanded sourcebook. 2nd ed. Thousand Oaks, CA: Sage; 1994.
Strauss AL, Corbin J. Basics of qualitative research: techniques and procedures for developing grounded theory. Thousand Oaks, CA: Sage; 1998.
Boyatzis R. Transforming qualitative information: thematic analysis and code development. Thousand Oaks, CA: Sage; 1998.
Rice ER, Holloway IW, Barman-Adhikari A, Fuentes D, Brown CH, Palinkas LA. A mixed methods approach to network data collection. Field Methods. 2014;26(3):252–68.
Freeman LC. Visualizing social networks. J Social Structure. 2000; 1(1).
Borgatti SP, Everett MG, Freeman LC. Ucinet for windows: software for social network analysis. Cambridge, MA: Analytic Technologies; 2002.
Muthen LK, Muthen BO. Mplus user’s guide. 3rd ed. Los Angeles: Muthen & Muthen; 2009.
Gibbons RD, Bock D, Hedeker D, et al. Full information item bifactor analysis of graded response data. Appl Psych Meas. 2007;31:4–19. doi:10.1177/0146621606289485.
We would like to thank and acknowledge all the members of our SAMHSA-Ce-PIM Partnership, especially Frances Harding, Center Director of SAMHSA’s Center for Substance Abuse Prevention and appointed principal SAMHSA lead to our partnership. A special thanks to the following SAMHSA partners: Richard Lucey, Richard Moore, Charles Reynolds, Costella Green, Charlotte Olson, Tonia Gray, Susan Marsiguilia, Ingrid Donato, Richard McKeon, Gail Ritchie, Charity Goodman, Audrey Adade, James Wright, Angela Mark, and Minnjuan Flournoy Floyd. We also thank Gracelyn Cruden and Xiaoling Xiang for their contributions to our partnership activities.
This work is supported by the National Institute on Drug Abuse (1 R34 DA037516-01A1). The views presented here are solely the responsibility of the authors and do not necessarily represent the official views of the National Institutes of Health.
LAP and CHB are responsible for the study conception and design. LAP, SS, and JV performed the acquisition of data. LAP, SM, SS, CHB, and CPC performed the data analysis. LAP, SM, SS, TV, CHB, JL, and CPC performed the interpretation of data. LAP, SM, SS, TV, CPC, CHB, JL, SK, and JV performed the drafting and critical revision of the manuscript. All authors read and approved the final manuscript.
The authors declare that they have no competing interests.
Ethics approval and consent to participate
This study, including procedures for obtaining informed consent, was approved by the Institutional Review Board at the University of Southern California (UP-14-00398). Written informed consent will be obtained from study participants prior to data collection.
About this article
Cite this article
Palinkas, L.A., Spear, S.E., Mendon, S.J. et al. Measuring sustainment of prevention programs and initiatives: a study protocol. Implementation Sci 11, 95 (2015). https://doi.org/10.1186/s13012-016-0467-6
- Substance use