This article has Open Peer Review reports available.
Characterizing implementation strategies using a systems engineering survey and interview tool: a comparison across 10 prevention programs for drug abuse and HIV sexual risk behavior
© Czaja et al. 2016
Received: 7 October 2015
Accepted: 4 May 2016
Published: 17 May 2016
Although many behavioral interventions have proven to be efficacious, new methodologies are required beyond efficacy trials to understand how to adopt, implement with fidelity, and sustain behavioral interventions in community settings. In this paper, we present a new approach, based on systems engineering concepts and methods, for characterizing implementation strategies that are used to deliver evidence-based behavioral interventions in health and social service settings. We demonstrate the use of this approach with implementation strategies, used or being used for broader dissemination of 10 evidence-based prevention program projects focused on the prevention of drug or HIV sex risk behaviors.
The results indicate that there are wide variations in intervention approaches and that there are challenges in program implementation including maintaining program fidelity, serving community needs, and adequate resources. The results also indicate that implementation requires a committed partnership between the program developers, implementation researchers, and community partners. In addition, there is a need for adaptability within programs to meet community needs, resources, and priorities while maintaining program fidelity.
Our methodological approach enabled us to highlight challenges associated with the community implementation of health risk prevention interventions. We also demonstrate how comprehensive descriptions of interventions facilitate understanding of the requirements of program implementation and decisions about the feasibility of implementing a program in community settings.
Drug use and sexually transmitted infections remain significant public health problems in the USA. In 2011, an estimated 8.7 % of the population had used or abused an illicit drug or a psychotherapeutic medication; of these 24 % were 18–20 year olds . Drug and alcohol disorders and sexually risky behaviors, which increase the chance of sexually transmitted infections, are closely tied especially in adolescence [1–8].
In this regard, a large number of interventions have been shown to be effective in reducing the incidence of drug and alcohol abuse  and engagement in risky sexual behavior among young adults (e.g., [10, 11]). But there are challenges to widely delivering these programs , especially to minority populations [13, 14]. Despite the recent “policy push” to promote community-based adoption of evidence-based prevention programs, adoption is painstakingly slow and there is a tension between maintaining intervention fidelity while meeting community values, needs, and constraints. Overall, although there is a relatively robust body of research on effective interventions, the adoption of these interventions in clinical and community practice has lagged behind.
However, two major challenges remain; a limited shared language to describe interventions/implementation approaches and tools to characterize the goals and specific elements of intervention/implementation strategies. This paper describes one set of tools, based on a systems engineering approach, which can be used to elucidate the resources and other requirements necessary for successful implementation of an intervention in a consistent way. The tools are applied to characterize and compare implementation strategies that have been or will be tested through randomized implementation trials for broad dissemination of 10 evidence-based prevention programs focused on drug abuse and/or risky sexual behaviors. We begin with a brief overview of the 10 programs followed by an overview of systems engineering.
Overview of the 10 evidence-based programs
Among the 10 evidence-based prevention programs included in this study, some focused on drugs or sexual risk only, others focused on both, and still others, which targeted risk and protective factors at birth or in early elementary school, focused on important antecedent factors in these earlier stages of life before entering the period of high risk. Because of the high frequency of drug and sex risk behavior, as well as the overlap between these outcomes, programs that have demonstrated beneficial impact across these outcomes have great potential value if they can be implemented widely. The programs are all considered proven or promising evidence-based programs by either the Blueprints Project for Violence Prevention  or by the Centers for Disease Control and Prevention , and they constitute a network of leading prevention implementation researchers all affiliated with the National Institute on Drug Abuse’s funded Center for Prevention Implementation Methodology (Ce-PIM) for Drug Abuse and HIV Sexual Risk Behavior.
Summary of programs and implementation strategies
Prevention program name/Principal Investigator (references)
Program goals/significant impact
Stage of life
Implementation research and practice
Quit Using Drugs Intervention Trial (QUIT)
Screening, brief intervention and referral for treatment (SBIRT)/Gelberg 
Prevent escalation of drug use to abuse/dependence for adults/drug use.
The program has demonstrated effectiveness for reducing alcohol abuse and it is currently being tested on substance abuse 
Research: type 1 hybrida
Practice: exploration phaseb
Physician and health educator
Primary care clinics
Communities that Care: programs vary depending on community selection/Hawkins, Catalano 
Prevent youth externalizing behaviors and promote healthy development/violence, alcohol, drug, tobacco use
Research: randomized implementation triala
Practice: distributed widely through SAMHSA, sustainmentb
Community coalition leaders
Rural, small towns and cities, and portions of metropolitan areas, including Latin America
Familias Unidas/Pantin, Prado 
Use parent training to increase parent-child communication/drug use, sex risk, depressive symptoms.
Program reduced monthly substance use in Hispanics by 30 % and achieved higher frequency of condom use among adolescents sexually active 30 months after the intervention .
Research: pilot type 3 hybrida
Practice: exploration phaseb
Caregivers, supported by trained facilitator
Small parent groups and home visits with family
Family check-up/Dishion, Stormshak 
Family management practices designed to reduce problem behaviors, enhance parenting skills, reduce family conflict, and reduce substance use.
Elementary, middle, and high school
Research: randomized implementation triala
Practice: active implementationb
Caregivers, supported by trained counselors
School settings and mental health community agencies
Good Behavior Game (GBG)/Poduska, Kellam 
Use group-based contingencies to reduce child aggressive disruptive behavior/drug and alcohol abuse or dependence disorder, conduct disorder, antisocial personality disorder, suicide ideation and attempts, criminal arrests, sex risk behavior.
Results across 3 randomized control trials indicated a 50 % reduction in drug abuse/dependence disorders among males through age 21 [10, 64–69], a significant reduction in alcohol abuse/dependence disorders for males and females , and also demonstrated a reduction in unprotected and risky sexual behavior through young adulthood among the highest risk group .
First and second grade
Research: randomized implementation triala
Practice: wide scale implementation through SAMHSA (http://www.samhsa.gov/grants/2010/sm-10-017.aspx); active implementationb
First- and second-grade classrooms
Keeping Foster and Kin Parents Trained and Supported (KEEP)/Chamberlain 
Program promotes child well-being and prevents foster placement breakdowns through support and skill enhancement of foster and kinship parents.
The program has demonstrated positive outcomes for treatment and prevention of child and adolescent behavior problems in multiple randomized control trials [70–72] and significant reductions in marijuana, tobacco, and other drugs at 18 months  as well as sexual behavior [74, 75].
Research: randomized implementation triala
Practice: full scale implementation in New York city through child welfare system; active implementationb
Foster/kin parents supported by trained health workers
Foster/kin parents groups
Life Skills Training (LST)/Botvin, Griffin 
Prevention of substance abuse (alcohol, tobacco, drug use) and violence.
Elementary, middle, and high school
Research: randomized implementation triala
Practice: implementation in schools through the National Health Promotion Associates (http://www.lifeskillstraining.com/index.php); sustainmentb
Trained school personnel
Nurse-Family Partnership/Olds 
Increase pre-natal and early stage parenting skills.
The program resulted in significant reductions in youth alcohol use, fewer sexual partners, and fewer problems with alcohol or drugs in a 15-year follow-up .
First 2 years of life
Research: natural experiment, type 1 hybrid designa
Practice: implementation in communities through the NFP National Service Office; sustainmentb
Sisters Informing Sisters about Topics on AIDS (SiSTA)/Wingood, DiClemente 
Demonstrated high increases in condom use and when combined with a biological intervention of HPV vaccination, demonstrated significant reductions in incident high-risk HPV infection 
Young adulthood and adulthood
Research: type 1 hybrid designa
Practice: wide scale implementation through CDC’s Dissemination of Evidence-Based Intervention (DEBI) program (http://www.effectiveinterventions.org/en/HighImpactPrevention/Interventions/SISTA.aspx); active implementationb
Peer/near peer facilitator and health educators
Improve parent-child relationships.
Middle school, high school
Research: type 2 hybrid designa
Practice: exploration phaseb
Caregivers supported by trained facilitators
Community-based settings, community churches
Implementation of these programs currently ranges from some programs being completely unused outside of a research setting to other programs being rolled out at the national, state, and community level at a fast pace. For example, Familias Unidas , a parent-centered preventive intervention for Hispanic parents and their adolescents, is currently available only through the developers in Miami whereas Life Skills Training (LST), a primary prevention program for adolescent drug abuse, delivered in middle schools, has the broad distribution, being used in 10,000 US schools in every US state and in 32 counties . The Good Behavior Game (GBG), a universal program targeting substance abuse, for first and second grade students [10, 11], also has broad distribution and is currently being implemented in 29 school districts through funding from the Substance Abuse and Mental Health Services Administration (SAMHSA), and a randomized implementation trial of two training/supervision strategies is underway .
The Nurse-Family Partnership program, which starts prenatally and continues through the first 2 years of life, has been implemented in more than 440 counties in the USA and served more than 26,000 families, and its implementation strategy  is being updated through a non-experimental continuous quality improvement approach [30, 31].
The effectiveness of the Strong African American Families (SAAF) program, designed to prevent the initiation and escalation of alcohol use, among rural African American families with middle school children, when delivered by a community provider, was recently evaluated in a randomized trial with public school children across 8 counties of rural Georgia .
At the current time, most of the prevention programs have implementation strategies based on each program’s unique needs rather than being informed by studies of implementation. Our only exception to this is the implementation model of the Communities That Care (CTC) , where the community can choose to determine what prevention programs best suit their needs. Thus, CTC is a general implementation strategy not specific to any particular intervention.
To understand how the implementation strategies relate to one another, a myriad of implementation frameworks have been put forward for behavioral interventions . It is often difficult to map the sub-dimensions of these implementation frameworks into specific strategies, so in this paper, we have taken a different, fundamentally engineering approach to characterizing implementation systems. By characterizing implementation strategies across different levels of system influence, we can represent their different components and ultimately redesign the system to respond more efficiently to its environment, resources, and objectives.
Systems engineering applied to implementation research
Systems engineering is both a discipline and process to guide the development, implementation, and evaluation of complex systems in order for the success of the system to be realized . Systems engineering has a holistic focus and attempts to ensure all aspects of a system are considered and integrated into a whole. Thus, this approach is not only concerned with the design of the elements or components of a system but also with external factors that can constrain the design such as logistical support requirements, individual needs, and available resources .
System thinking and general systems theory dates back to Bertalanffy in the 1930s and the systems approach was developed initially in the biological sciences and refined in the 1940s in the communication and military industries . The systems approach has evolved throughout the years because of the increased complexity of systems largely due to continual developments in technology. The concepts and methods of systems engineering have been applied to enhance the performance of aviation, manufacturing, logistical, aerospace, and medical systems. In recent years, a thrust has been on human-machine systems. Within the discipline of human factors engineering, a human-machine system is composed of four main interdependent components; the human, the task/activity they are performing, the equipment/technology they are using to perform this task, and the context/environment (physical and organizational) where this transaction is occurring.
Recently, the National Academy of Engineering (NAE) and the Institute of Medicine (IOM) issued a report calling for the application of systems engineering methods to healthcare delivery systems and improved collaboration between medicine and engineering in healthcare delivery . To help clarify the complexity of the healthcare system, the report adopted a four-level model: (1) the individual patient; (2) the care team; (3) the organization; and (4) the social and political environment. The report underscored that optimizing the system as a whole requires of moving away from “silos” thinking and instead adopting “systems thinking”, which recognizes the interdependence among the levels of the model. Following this report, Tu and colleagues  examined the feasibility of applying systems engineering techniques used in traditional settings such as manufacturing, to implement an evidence-based intervention, a colorectal screening protocol, at a community health center. They found that the application of these techniques in healthcare settings is feasible but that there are unique challenges related to patient and organizational factors. They also stress that more research is needed to guide the application of these approaches in healthcare environments.
From a systems engineering perspective, optimization of successful implementation of a prevention program in a community setting requires that all of the components of an intervention system be considered in the implementation process. For example, implementation of a drug abuse prevention program requires an understanding of the school’s resources, policies, and readiness for change; the community/neighborhood of the children and the characteristics of those children; and their family situations. These factors all influence the degree to which intervention fidelity is likely and whether the goal of reduced substance abuse can be achieved. If the school has limited resources and the intervention is too burdensome with respect to the number and scheduling of sessions and reporting requirements for teachers, it is likely to be implemented poorly and less likely to be effective.
Thus, from a systems perspective, a fundamental aspect of successful implementation of an intervention within a community setting requires an understanding of the characteristics and associated implementation demands associated with the intervention (e.g., if the intervention requires Internet access; or 9 weekly 60-min sessions and 5 parent support group sessions) as well as the characteristics and constraints associated with each of the other components of the “intervention system” such as the target population and the organization, environment/setting where the intervention will be implemented.
Wandersman and colleagues  also advocate the use of a systems approach to help bridge the gap between prevention research and practice. They propose a framework, the Interactive Systems Framework (ISF) for Dissemination and Implementation. The framework focuses on how to move evidence-based prevention programs into practice and not on the development and testing of new interventions, which is a central component of our approach. However, the ISF does underscore the importance of considering the characteristics of the organization and the interdependencies among the components of the system. Greenhalgh and colleagues  also propose a systems model, based on a systematic review of health service innovations to guide the dissemination and implementation of innovations. The model takes into account various system components (innovation, organization, adopter) and some of the characteristics of these components such as the technical knowledge requirements of the innovation, organization readiness for innovation, and the adopter’s skills and motivation. However, as noted by the authors, the model is intended mainly as memory aid for considering the different aspect of a complex situation and their interactions and should not be viewed as a prescriptive formula. In addition, some of the component characteristics that are listed are rather general (e.g., “fuzzy boundaries” of the innovation) and may be difficult for an intervention researcher to operationalize.
Data collection protocol
The data collection protocol had two complimentary components, a survey that gathered information regarding the basic characteristics of the 10 evidence-based prevention programs and a guided telephone interview. The focus of the survey was on gathering information on the essential elements of the programs such as delivery characteristics, staffing, and degree of program adaptability. The telephone interview was intended to compliment the survey and allowed us to explore program implementation in greater depth and addressed issues related to the implementation context, training, intervention fidelity, community support and perceived implementation challenges. To insure that we had a basic understanding of the programs before conducting the interviews, the interviews were conducted after the survey was completed. To provide a context for the interview, the interviewees were provided with a hand out that briefly explained the systems engineering approach.
The implementation survey
The survey was adapted from the Intervention Taxonomy (ITAX), a survey instrument designed to help interventionists describe and understand the essential features of an intervention protocol . The survey contained 45 items which were grouped into six sections: (1) background information (e.g., number and type of intervention components such as initial component, maintenance component, home component, classroom component; identifying information for the person completing the survey), (2) program characteristics, (3) measurement of goals and processes, (4) staffing and training, (5) cost analysis, and (6) partnerships/foundations. The section on program characteristics included items that captured the following information: mode of intervention delivery (e.g., face-to-face, online), materials used (e.g., pamphlets, videos), setting of delivery (e.g., classroom, clinic, home), intervention strategies (e.g., problem solving, skill building), intervention schedule (e.g., duration of program, number of sessions), and adaptability (e.g., type of adaptability (language) and processes for adaptability). The section on measurement of goals and processes included three items related to the types of outcomes being assessed, the assessment schedule, and who conducted the assessments. The staffing and training section contained items related to characterizing the skills and qualification requirements of the people involved in the delivery of the intervention and type of and intensity of staff training. The section on cost analysis contained items related to variables that could be included in a cost analysis and the schedule for assessing these variables. The section on partnerships/foundations included items related to types and importance of community partnerships and the functions of the Community Advisory Board (if applicable).
For ease of administration, many of the item responses were multiple choice. The survey was completed online. In most cases, it was initially completed by a Project Manager/Coordinator and the Principal Investigator (PI) reviewed the responses and resolved any discrepancies.
The interview complemented the survey and enabled participants to describe their program in more detail. For example, more detailed information was gathered on the target population (e.g., skill level; SES status), the characteristics of the community/setting and all of the “players” involved in program implementation and so on. Issues of training and intervention fidelity were also probed in greater detail. In addition, information was gathered on perceptions of factors important to the success of the program and challenges to its implementation. A semi-structured format was used to guide the interview. It was conducted via teleconference and the duration was 60–90 min. The interviewees included the PIs of the programs. In some cases, it also included program staff such as a Project Coordinator.
The interviews were conducted by a senior researcher from a different field than drug/HIV prevention (SC), who was deliberately kept naïve to detailed scientific literature on each of these programs. Thus, the only information available prior to conducting these interviews consisted of the answers to the survey questions. The survey and interview protocol are available from the first author.
The data gathered in this study elucidate the characteristics of the 10 prevention programs that were in various states of implementation. The objective is to demonstrate how a systems engineering approach could be used to identify the requirements for implementing these programs in a community setting. The study was also designed to identify essential elements of and potential barriers to successful implementation of these programs. We first describe the program characteristics gathered from the survey. We then report on the findings from the interviews with the Principle Investigators. The interviews were transcribed and the topics that emerged were summarized into two main categories: (1) factors important to successful implementation and (2) challenges to program implementation.
Characteristics of the 10 programs
Summary of preventive intervention schedule characteristics
Number of sessions
Contact time (minutes/session)
Summary of program delivery characteristics and features of adaptability
Program delivery characteristics
Mode of intervention delivery
Technology (computer or telephone)
Type of intervention strategy
Provision of social support
Tracking and monitoring
Provision of information
Provision of behavioral incentives
Skill building techniques
Problem solving techniques
Stress management techniques
Intervention delivery location
Group residential facility
Clinical research space
Types of materials used
Program adaptability features
Intervention strategies are adapted for cultural sensitivity
The program is scripted
Degree of scripting
Goals of each session are specified but no further scripting
Goals and exercises/tasks of each session are specified
Specific language is provided, with room for elaboration
Exact scripts are provided for humans to speak
Intervention is delivered by machine
Aspect of intervention amenable to adaptation
Number of sessions
Schedule of sessions
Duration of sessions
Location of intervention delivery
Focus of sessions
Determination of modifications
Time of modifications
At intake/initial assessment
At scheduled increments
At any session/contact
At other point
The protocols for all of the programs are scripted but all programs also incorporate protocols to adapt some aspects of the intervention, such as the schedule, duration, or number of sessions or the session delivery location. Most programs (8) include specific guidance on how to adapt the program and to provide exact scripts for program adaptation. The impetus for the adaptation varies and includes clinical judgment and requests from participants. The programs can also be adapted to be responsive to the needs of different cultures, for example, involving members of the participants’ community in recruitment; or including individuals from a target organization (e.g. school) in the delivery of the intervention. A small number of programs (4) include interventionists whose racial/ethnic background is matched to that of the program participants. For most programs (9), the content of the intervention strategies was developed to match the target population’s literacy level.
The research teams use a variety of types of staff to deliver the programs including the following: assessors (n = 5), social workers or counselors (n = 4), nurses (n = 2), teachers (n = 2), and physicians (n = 1). This varied according to the needs of the program. For example, the Nurse-Family Partnership program involves home visits to mothers by public health nurses. For all programs, the staff receives specific training on the intervention. Interestingly, four of the programs require staff to speak a language other than English, such as Spanish, to accommodate the needs of the target population and the community.
About half of the programs gather data related to program goals (e.g., alcohol abuse, tobacco use) pre and post implementation and others gather this information at several times points (e.g., monthly). Common program goals include changes in behavior, enhanced behavioral and problem solving skills and increased knowledge. Data collection methods vary and generally include the use of questionnaires and rating scales, objective measurement, and interviews.
Finally, when asked about partnership building, surprisingly only 20 % of the programs have an advisory board. Those that do have boards indicated that the board functions to review program protocols, procedures, and measures and advises on implementation strategies. Most of the respondents (80 %) indicate that political support of their program was important to implementation.
Factors important to successful program implementation
Several main themes emerged during the interviews regarding factors important to facilitating successful program implementation. First is the importance of having a trusted, collaborative partnership with community groups/agencies and personnel within the targeted setting for program implementation. The general perception of the investigators is that implementation success is highly dependent on the extent to which the program team has a deep and lasting relationship with the community-based organizations (CBOs) or institutions that would ultimately be responsible for program implementation. Several investigators specifically mentioned the importance of a Community Advisory Board (CAB). For the programs included in this study, these partnerships were generally between academically based program teams composed of the program creators, research coordinators, and evaluators with community and agency leaders and staff. They also involved healthcare clinics, school districts schools, churches, coalitions, and law enforcement agencies. Furthermore, they were typically multi-level and involved numerous “players” including physicians, teachers, parents, and in some cases custodial and security personnel. As one interviewee said “A lot of how we [researcher and research team] did what we did was based on… these people [community partners] telling us what it is that we would need to do in order to make this [research] work, and we listened to them… you [researcher] have to be inclusive and you have to find ways to make people feel that you’re there not just to get from them but to provide as well.”
Second, partner agencies need to have a strong commitment to the program. Most of the interviewees felt strongly that if their partner agencies were not committed to the program and did not remain committed, implementation would fail. This commitment entails several elements. One essential element is that the agency has to have a complete understanding of the program’s characteristics and requirements. A second component is the agency’s readiness to implement the program and a third is having the staff, resources, and support necessary for a program. Another essential element is agency “buy-in”; the degree to which the agency is willing to expend the effort needed to take ownership of the program. This might include, for example, reducing the workload of case workers so that they have sufficient time to deliver the program or adapting the number of sessions to accommodate families. One interviewee discussed a reason for lack of receptivity among parents for a family-based program was the lengthy duration of the program. “Unable to commit to a 7-week program because of their time schedules or they had children in developmental situations … working swing shift jobs, those kinds of things.” Another key element of “buy-in” is having staff engaged in and able to deliver the program. The skill level of the staff is also critical to program success. In addition, if a program includes group activities the cohesiveness of the group is important. Many of the interviewees also mentioned the importance of flexibility. Successful implementation often depended on research staff being able to modify and adapt the program to local needs and exigencies. This might include adapting the language and exemplars used in a program or adapting a program to be delivered in a particular setting such as a church as opposed to a clinic or to the structure or personnel within an organization. For example, as one interviewee responded: “I think we are going to have to be more flexible in terms of fidelity because of skill level… let’s say there’s one principal for three middle schools and the person who has the most contact with the parents by default is the secretary in the front office … because of her skill set … we have to take out or redo some of the demands of our program.”
Common challenges to successful implementation
One major challenge identified by most of the interviewees is having strategies to assess and manage treatment fidelity. In some cases assessment of intervention fidelity is resource intensive and time-consuming. Not surprisingly it is also difficult to maintain fidelity over time. Most of the respondents also indicated the need for strategies for mechanisms to implement “corrective action” if required. As expected, there is tension between program fidelity and flexibility. As noted, flexibility or the ability to adapt a program to cultural or contextual nuances is often a key to successful implementation yet this adaption also raises concerns that core elements of the program might be omitted or changed in ways that reduce the effectiveness of an intervention.
An additional challenge to implementation for many programs is scheduling and logistics. Most of the interviewees stated that they faced challenges scheduling program sessions or lesson delivery. Collaborating agencies often have many demands that outweigh the desire to implement an intervention program. Scheduling and logistics challenges can be exacerbated by staff turnover and/or promotion both within the program development team and within partner CBOs is a critical challenge. Furthermore, the logistics associated with training staff when the program is being implemented at multiple locations—including international settings—is also a challenge and often resource demanding. Overall, the interviewees were conscious of the challenge of coordinating these activities and of attending to the many logistical challenges sustainable programs require.
In sum, adequate resources within CBOs and allocating these resources to program implementation are a major concern and represent a continued challenge to the implementation of intervention programs. Another major challenge is securing a commitment on the part of the organization to implement the program in a way that is consistent with program requirements.
The goal of this paper was to demonstrate how a systems engineering approach could be used to identify the requirements for implementing prevention programs, focused on the prevention of drug or HIV sex risk behaviors, in a community setting. The study was also designed to identify factors important to and potential barriers to implementation success. We provide data from a survey and interview with 10 evidence-based prevention programs that were in various states of program implementation.
In general, the programs were quite varied in terms of their characteristics and implementation requirements and strategies. Several important themes emerged that cut across all programs with respect to successful implementation, however. First, the tension between fidelity and adaptability is high. All of the programs were scripted and contained core elements, yet adaptation of a program is generally required to meet community needs and shifting resources and priorities. In addressing this tension, the community-research partnerships arrived at different decisions regarding adaptation, with respect to what aspects of the protocol can be adapted, to what extent and by whom. We recommend that during the design and evaluation of interventions researchers pay special attention to what aspects of an intervention can be adapted and provide general guidelines or protocols for implementing and documenting adaptations both during research trials and in practice settings. Adaptation issues should also be addressed in CONSORT-type checklists for randomized intervention and implementation trials.
Also, all of the study participants recognized the need to maintain program fidelity but struggled with ways to measure and maintain this fidelity given community constraints. From a systems perspective, simply monitoring for fidelity is not sufficient to maintain standards; mechanisms for feedback and accountability also need to be in place . It is important to measure the extent to which the intervention was implemented as intended, which, if low, could threaten internal validity and ultimately external validity. Thus, another important dimension to capture in intervention protocols is a description of monitoring and feedback subsystems, including who is ultimately responsible for overseeing that the program is delivered with fidelity. Coupled with this challenge is monitoring the implementation strategy itself. Having a written protocol of what stages and steps are needed for implementation, along with a mechanism to track protocol violations, is also important to include in an intervention Manual of Operations (MOP) and in CONSORT-type checklists for implementation trials much like the specialized one that now exists for cluster randomized trials .
All of the study participants also emphasized the need for a strong and committed partnership between the program developers, implementation researchers, and the community partners who will be involved in implementing the program. In this respect, many of the programs included in this study relied on a community partner such as a school, church, coalition, or other agency to help with access to clients and communities; and to help with program delivery. In all cases, the commitment of these partners is considered essential to program success. Also, having a strong partnership is critical to ensuring effective communication between researchers and organizational staff. Determining the optimal structures and functioning of community coalitions is an active area of research [44, 45], and information regarding these issues should also be provided in an intervention MOP.
The factors identified through the interview and survey presented above are also supported by the dissemination and implementation conceptual model put forth by Aarons and colleagues , which divides factors affecting implementation into two contexts, the outer context (i.e., community advisory board) and the inner context (i.e., agency “buy-in”, agency readiness, agency flexibility). Like Aarons and colleagues  suggest, our interviewees concluded that the importance of the inner and outer contexts varies depending on the implementation phase of the program.
This paper has identified a number of implementation challenges. However, these implementation challenges can also be considered opportunities for implementation science. For example, they imply that better tracking and coordination systems could help reduce the costs and barriers to successful implementation. The identified challenges also indicate that a process to help determine how to implement a program to achieve the optimal balance of maintaining program fidelity and adaptability to meet contextual demands would be a useful next step for implementation science. The use of technology systems to facilitate staff training especially staff in remote locations could also prove to be useful.
The systems perspective has enabled us to highlight these challenges and demonstrate how comprehensive descriptions of interventions facilitate understanding the requirements of program implementation and decisions about the feasibility of implementing a program in community settings. Intervention researchers in collaboration with community service providers can determine if they have adequate staff and resources to successfully replicate the essential elements of a program and if the program is sensitive to and meets the “culture” of the community. Indeed, a “cost calculator” tool already exists in child welfare that helps organizations and communities budget and plan for how a specific program can be implemented in their settings . This systems engineering approach could lead to a toolkit for communities and researchers to examine all required resources for an intervention, not just ones with monetary value. The tools used in the approach can also be used to evaluate different implementation strategies whether they take place in randomized implementation trials or adaptive  or hybrid trials  that combine effectiveness and implementation.
However, it should be noted that the usefulness of this approach depends on having a common language/taxonomy for describing programs and implementation strategies. This is a challenge within implementation science. Currently, despite some attempts, there is as yet no fully accepted standard language/taxonomy for describing the components and processes of prevention delivery systems across diverse fields [40, 48–50] or for implementation designs. Additional work is required to distinguish between programs in local investigations and implementation research , as well as to distinguish the current mélange of overlapping terms for trial designs in this field [18, 23, 52]. This makes it difficult to tease out the key elements of a program and to systematically conduct cross-study analyses. For example, in the development of the ITAX survey instrument that was applied to interventions that targeted healthy behaviors, a consensus process was used to identify intervention elements relevant to understanding outcomes and important to subsequent replication efforts. In addition, operational definitions of the terms used in the survey were provided and pilot tested with a sample of the investigators. This provided some assurance that the survey respondents were “speaking the same language.” This is an area that needs more work within implementation science as terms such as “programs,” “treatment,” and “implementation strategy” can take on different meanings.
Another challenge for implementation science is to identify strategies that can address implementation challenges so that program implementation proceeds more efficiently. This will almost certainly involve innovative uses of mixed methods  and technology (e.g., to assist with training across implementation sites) or systems science methods [13, 54, 55], as well as improved decision-making, especially with interventions that are rapidly evolving  or providing of a rich set of options at the individual and community level. Moreover, such strategies will be needed to create more effective and sustainable programs in the future especially as many of the programs described in this study are being implemented on a broad scale across diverse communities and cultures, including populations where health disparities are greatest [13, 20].
This survey and interview were tested together on 10 prevention programs for drug abuse and/or HIV sex risk behavior prevention, and thus we are not able at this time to generalize to their value in other settings, including treatment in contrast to prevention, where randomized implementation trials are being designed. However, the breadth of these 10 programs, and the deliberate choice of having the interviews conducted by a researcher who was unfamiliar with these particular programs and this particular scientific field, suggests optimism that this approach could be valuable for not only specifying differences in a wide variety of randomized implementation trials but also for purposes of training new investigators and providing useful feedback to the researchers and community leaders themselves. Other researchers conducting intervention trials can use this approach to help determine the essential requirements of their intervention, which will provide information about requirements for broad scale implementation.
However, this study has some limitations. As noted, it was restricted to trials concerned with prevention of substance abuse and sexual risky behaviors. Thus, it needs to be replicated with other types of intervention programs. In addition, we only examined limited components of the “intervention system” from the perspective of the researcher. To fully gain benefits from this approach, a more detailed analysis of the characteristics of the target population and other individuals involved in implementing the intervention should be conducted. If for example, the intervention is being delivered in a school setting and involves parents and students, an analysis of the characteristics of the students, parents, and teachers would be helpful to understand their preferences, skills, attitudes, and needs. Additionally, an analysis of the context should be conducted including the organization (e.g. school) as well as the neighborhood and larger social and political context. Finally, other aspects of the intervention should be included in the analysis such as the usability, perceived value, work flow, and communication patterns. The ultimate goal of the systems approach is to identify potential mismatches or degree of fit between the elements and requirements of an intervention; the needs, preferences, and characteristics of the target population; and the characteristics and resources/constraints of the implementation setting.
We also conducted a survey of the SUCCESS prevention program, aimed at reducing cervical cancer in Haitian women; results for this study are not included in this paper because the intervention target differs from the other 10.
We acknowledge support from the National Institute on Drug Abuse (NIDA) for this work through the Center for Prevention Implementation Methodology for Drug Abuse and Sexual Risk Behavior, P30DA027828. The content of this paper is solely the responsibility of the authors and does not necessarily represent the official views of the funding agencies. We thank the PIs and program staff of the model programs for their time and effort in being interviewed for this study. We also thank Shih Hua Fu for her assistance with the data analyses.
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.
- Substance Abuse and Mental Health Services Administration. Results from the 2012 national survey on drug use and health: mental health findings. (No. NSDUH Series H-47, HHS Publication No. (SMA) 13-4805). Rockville, MD: Substance Abuse and Mental Health Services Administration; 2013.Google Scholar
- Patrick ME, O'Malley PM, Johnston LD, Terry-McElrath YM, Schulenberg JE. HIV/AIDS risk behaviors and substance use by young adults in the United States. Prev Sci. 2012;13(5):532–8. doi:10.1007/s11121-012-0279-0. PMCID: PMC3586255.View ArticlePubMedPubMed CentralGoogle Scholar
- Capaldi DM. Prevention science supplemental issue commentary promoting healthy sexual practices: what we have learned from 100 years of work. Prev Sci. 2014;15(1):78–80. doi:10.1007/s11121-013-0434-2.View ArticlePubMed CentralGoogle Scholar
- Caruthers A, Ryzin M, Dishion T. Preventing high-risk sexual behavior in early adulthood with family interventions in adolescence: outcomes and developmental processes. Prev Sci. 2014;15(1):59–69. doi:10.1007/s11121-013-0383-9.View ArticlePubMed CentralGoogle Scholar
- Reider EE, Robertson EB, Sims BE. Does early intervention prevent health-risking sexual behaviors related to HIV/AIDS? Prev Sci. 2014;15(1):1–5. doi:10.1007/s11121-013-0455-x.View ArticleGoogle Scholar
- Skinner M, Fleming CB, Haggerty KP, Catalano RF. Sex risk behavior among adolescent and young adult children of opiate addicts: outcomes from the focus on families prevention trial and an examination of childhood and concurrent predictors of sex risk behavior. Prev Sci. 2014;15(1):70–7. doi:10.1007/s11121-012-0327-9.View ArticlePubMed CentralGoogle Scholar
- Spoth R, Clair S, Trudeau L. Universal family-focused intervention with young adolescents: effects on health-risking sexual behaviors and STDs among young adults. Prev Sci. 2014;15(1):47–58. doi:10.1007/s11121-012-0321-2.View ArticlePubMed CentralGoogle Scholar
- Centers for Disease Control and Prevention (CDC). HIV among youth. 2014. Retrieved November, 29, 2013, from http://www.cdc.gov/hiv/risk/age/youth/index.html. Accessed April 2016.
- National Research Council and Institute of Medicine. In committee on the prevention of mental disorders and substance abuse among children, youth, and young adults: research advances and promising interventions. In: O'Connell ME, Boat T, Warner KE, Board on Children, Youth, and Families, Division of Behavioral and Social Sciences and Education, editors. Preventing mental, emotional, and behavioral disorders among young people: Progress and possibilities. committee on prevention of mental disorders and substance abuse among children, youth, and young adults: Research advances and promising interventions. Washington, DC: The National Academies Press; 2009.Google Scholar
- Kellam SG, Brown CH, Poduska J, Ialongo N, Petras H, Wang W, Windham, A. Summary of cohort 2 analyses—supplement to “Effects of a universal classroom behavior management program in first and second grades on young adult behavioral, psychiatric, and social outcomes”. Drug Alcohol Depend. 2008. Only found online at: doi:10.1016/j.drugalcdep.2008.01.004.
- Kellam S, Wang W, Mackenzie AL, Brown CH, Ompad D, Or F, Windham A. The impact of the good behavior game, a universal classroom-based preventive intervention in first and second grades, on high-risk sexual behaviors and drug abuse and dependence disorders into young adulthood. Prev Sci. 2014;15(1):6–18. doi:10.1007/s11121-012-0296-z View ArticlePubMed CentralGoogle Scholar
- Hallfors D, Pankratz M, Hartman S. Does federal policy support the use of scientific evidence in school-based prevention programs? Prev Sci. 2007;8(1):75–81. doi:10.1007/s11121-006-0058-x.View ArticlePubMedGoogle Scholar
- Brown CH, Mohr DC, Gallo CG, Mader C, Palinkas L, Wingood G, Jacobs C. A computational future for preventing HIV in minority communities: how advanced technology can improve implementation of effective programs. J Acquir Immune Defic Syndr. 2013, 63(Supplement 1), S66-S71. PMCID: PMC3746769.Google Scholar
- Kellam SG, Mackenzie ACL, Brown CH, Poduska JM, Petras H, Wilcox HC. The good behavior game and the future of prevention and treatment. Addict Sci Clin Pract. 2011;6(1):73–84. PMCID: PMC3188824.PubMedPubMed CentralGoogle Scholar
- Aarons GA, Hurlburt M, Horwitz SM. Advancing a conceptual model of evidence-based practice implementation in public service sectors. Adm Policy Ment Health. 2011;38(1):4–23. doi:10.1007/s10488-010-0327-7. PMCID: PMC3025110.View ArticlePubMedPubMed CentralGoogle Scholar
- Tinkle M, Kimball R, Haozous EA, Shuster G, Meize-Grochowski R. Dissemination and implementation research funded by the US national institute of health, 2005–2012. Nurs Res Pract. 2013;2013:1. doi:10.1155/2013/909606.View ArticleGoogle Scholar
- Proctor E, Silmere H, Raghavan R, Hovmand P, Aarons G, Bunger A, Hensley M. Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Adm Policy Ment Health. 2011; 38(2):65–76. doi:10.1007/s10488-010-0319-7 PMCID: PMC3068522.
- Landsverk J, Brown CH, Chamberlain P, Palinkas L, Rolls Reutz J, Horwitz SM. Design and analysis in dissemination and implementation research. In: Brownson RC, Colditz GA, Proctor EK, editors. Dissemination and implementation research in health: Translating science to practice. London: Oxford University Press; 2012. p. 225–60.View ArticleGoogle Scholar
- Grimshaw J, Campbell M, Eccles M, Steen N. Experimental and quasi-experimental designs for evaluating guideline implementation strategies. Fam Pract. 2000;17 Suppl 1:S11–6.View ArticlePubMedGoogle Scholar
- Brown CH, Mason WA, Brown EC. Translating the intervention approach into an appropriate research design: the next-generation designs for effectiveness and implementation research. In: Sloboda Z, Petras H, editors. Advances in prevention science: Defining prevention science. 1st ed. NY: Springer; 2014. p. 363–88.View ArticleGoogle Scholar
- Chamberlain P, Brown CH, Saldana L. Observational measure of implementation progress in community based settings: the stages of implementation completion (SIC). Implementation Sci. 2011;6(1):1–8. doi:10.1186/1748-5908-6-116. PMCID: PMC3197550.View ArticleGoogle Scholar
- Saldana L, Chamberlain P, Wang W, Hendricks Brown C. Predicting program start-up using the stages of implementation measure. Adm Policy Ment Health. 2012;39(6):419–25. doi:10.1007/s10488-011-0363-y. PMCID: PMC3212640.View ArticlePubMedPubMed CentralGoogle Scholar
- Brown CH, Wang W, Kellam SG, Muthén BO, Petras H, Toyinbo P.The Prevention Science and Methodology Group. Methods for testing theory and evaluating impact in randomized field trials: intent-to-treat analyses for integrating the perspectives of person, place, and time. Drug Drug Alcohol Depend. 2008: 95(Suppl 1); S74-S104; Supplementary data associated with this article can be found, in the online version, at doi:10.1016/j.drugalcdep.2008.01.005. http://dx.doi.org/10.1016/j.drugalcdep.2007.11.013 PMCID: PMC2560173.
- University of Colorado, Boulder, Institute of Behavioral Science. Blueprints for healthy youth development. 2012. Retrieved September 23, 2013, from www.blueprintsprograms.com. Accessed April 2016.
- Centers for Disease Control and Prevention (CDC). Effective interventions: HIV prevention that works. 2012. Retrieved 12/1, 2013, from http://www.effectiveinterventions.org/en/HighImpactPrevention/Interventions.aspx. Accessed April 2016.
- Pantin H, Prado G, Lopez B, Huang S, Tapia MI, Schwartz SJ, Branchini J. A randomized controlled trial of Familias Unidas for Hispanic adolescents with behavior problems. Psychosom Med. 2009:71(9);987–995. doi:10.1097/PSY.0b013e3181bb2913 PMCID: PMC2805119.
- Griffin KBC. Distribution of life skills training in U.S. 2013.Google Scholar
- Poduska J, Kellam SG, Brown CH, Ford C, Windham A, Keegan N, Wang W. Study protocol for a group randomized controlled trial of a classroom-based intervention aimed at preventing early risk factors for drug abuse: Integrating effectiveness and implementation research. Implementation Sci. 2009;4:56. doi:10.1186/1748-5908-4-56. PMID: 19725979 . PMCID: PMC2753630.View ArticleGoogle Scholar
- Olds DL, Hill PL, O'Brien R, Racine D, Moritz P. Taking preventive intervention to scale: the nurse-family partnership. Cogn Behav Pract. 2003;10(4):278–90. http://dx.doi.org/10.1016/S1077-7229%2803%2980046-9.View ArticleGoogle Scholar
- Hill P, Olds D. Improving implementation of the nurse-family partnership in the process of going to scale. In: Halle T, Metz A, Martinez-Beck I, editors. Applying implementation science in early childhood programs. CO: Paul H. Brooks; 2013. p. 193.Google Scholar
- Olds D, Donelan-McCall N, O'Brien R, MacMillan H, Jack S, Jenkins T, Beeber L. Improving the nurse-family partnership in community practice. Pediatrics. 2013;132(Suppl 2):S110–7. doi:10.1542/peds.2013-1021I.
- Kogan SM, Lei MK, Brody GH, Ftris TG, Sperr M, Anderson T. Implementing family-centered prevention in rural African American communities: a randomized effectiveness trial of the strong African American families program. Prev Sci. 2016;17(2):248–58. doi:10.1007/s11121-015-0614-3.View ArticlePubMedGoogle Scholar
- Hawkins JD, Oesterle S, Brown EC, Abbott RD, Catalano RF. Youth problem behaviors 8 years after implementing the communities that care prevention system: a community-randomized trial. JAMA Pediatr. 2014;168(2):122–9. doi:10.1001/jamapediatrics.2013.4009. PMCID: PMC3946405.View ArticlePubMedPubMed CentralGoogle Scholar
- Greenhalgh T, Robert G, MacFarlane F, Bate P, Kyriakidou O. Diffusion of innovations in service organizations: systematic review and recommendations. Milbank Q. 2004;82(4):581–629.View ArticlePubMedPubMed CentralGoogle Scholar
- Martin J. Systems engineering guidebook: a process for developing systems and products. Boca Raton: CRC Press; 1997.Google Scholar
- Kossiakoff A, William N, Sweet WN, Sam Seymour S, Biemer SM. Systems engineering principles and practice. 2nd ed. New Jersey: John Wiley and Sons Inc; 2011. ISBN: 978-0-470-40548-2.View ArticleGoogle Scholar
- Banathy BH, Jenlink PM. Systems inquiry and its application in education. In: Jonassen DH, editor. Handbook of research on educational communications and technology. Bloomington, IN: Association for Educational Communications and Technology; 2004. p. 37–58.Google Scholar
- Reed PP, Compton WD, Grossman JH, Fanjiang G, editors. Building a better delivery system: a new engineering health partnership. Washington, D.C.: The National Academies Press; 2005.Google Scholar
- Tu S-P, Feng S, Storch R, Yip M-P, Shong H, Fu M, Chun A. Applying systems engineering to implement an evidenced-based intervention at a community health center. J Health Care Poor Underserved. 2012;23(4):1399–409.View ArticlePubMedGoogle Scholar
- Wandersman A, Duffy J, Flaspohler P, Noonan R, Lubell K, Stillman L, Blachman M, Dunville R, Saul J. Bridging the gap between prevention research and practice: the interactive systems framework for dissemination and implementation. Am J Community Psychol. 2008;41(3-4):171–81. doi:10.1007/s10464-008-9174-z.View ArticlePubMedGoogle Scholar
- Greenhalgh T, Glenn R, Macfarlane F, Bate P, Kyriakidou O. Diffusion of innovations in service organizations: systematic review and recommendations. Millbank Q. 2004;82(4):581–629.View ArticleGoogle Scholar
- Schulz R, Czaja SJ, McKay JR, Ory MG, Belle SH. Intervention taxonomy (ITAX): describing essential features of interventions. Am J Health Behav. 2010;34(6):811–21. PMCID: PMC2900778.View ArticlePubMedPubMed CentralGoogle Scholar
- Campbell MK, Piaggio G, Elbourne DR, Altman DG, CONSORT Group. Consort 2010 statement: extension to cluster randomised trials. BMJ. 2012;345:e5661. doi:10.1136/bmj.e5661.View ArticlePubMedGoogle Scholar
- Provan K, Kenis K. Modes of network governance: structure, management, and effectiveness. J Public Adm Res Theory. 2008;18:228–52. doi:10.1093/jopart/mum015.Google Scholar
- Valente TW, Coronges KA, Stevens GD, Cousineau MR. Collaboration and competition in a children’s health initiative coalition: a network analysis. Eval Program Plann. 2008;31(4):392–402.View ArticlePubMedGoogle Scholar
- Chamberlain P, Snowden LR, Padgett C, Saldana L, Roles Reutz J, Holmes L, Landsverk J. A strategy for assessing costs of implementing new practices in the child welfare system: adapting the English cost calculator in the United States. Adm Policy Ment Health Ment Health Serv. 2011: 38(1); 24-31. doi:10.1007/s10488-010-0318-8 PMID: 20976620.
- Curran GM, Bauer M, Mittman B, Pyne JM, Stetler C. Effectiveness-implementation hybrid designs: combining elements of clinical effectiveness and implementation research to enhance public health impact. Medical Care. 2012;50(3):217–26. doi:10.1097/MLR.0b013e3182408812. PMCID: PMC3731143.View ArticlePubMedPubMed CentralGoogle Scholar
- Atun R, Menabde N. Health systems and systems thinking. In: Coker R, Atuna R, McKee M, editors. Health systems and communicable disease control: Experiences from Europe and Latin America. Berkshire, United Kingdom: Open University Press, McGraw Hill Education; 2008. p. 121–40.Google Scholar
- Atun R, de Jongh T, Secci F, Ohiri K, Adeyi O. A systematic review of the evidence on integration of targeted health interventions into health systems. Health Policy Plan. 2010;25(1):1–14. doi:10.1093/heapol/czp053.View ArticlePubMedGoogle Scholar
- Atun R, de Jongh T, Secci F, Ohiri K, Adeyi O. Integration of targeted health interventions into health systems: a conceptual framework for analysis. Health Policy Plan. 2010;25(2):104–11. doi:10.1093/heapol/czp055.View ArticlePubMedGoogle Scholar
- Cheung K, & Duan N. Design of implementation studies for quality improvement programs: an effectiveness-cost-effectiveness framework. Am J Public Health. 2013; e1-e8. doi:10.2105/AJPH.2013.301579.
- Tunis SR, Stryer DB, Clancy CM. Practical clinical trials: increasing the value of clinical research for decision making in clinical and health policy. JAMA. 2003;290(12):1624–32. doi:10.1001/jama.290.12.1624.View ArticlePubMedGoogle Scholar
- Palinkas LA, Aarons GA, Horwitz SM, Chamberlain P, Hulburt M, Landsverk J. Mixed method designs in implementation research. Adm Policy Ment Health Ment Health Serv. 2011;38(1):44–53. PMCID: PMC3025112.View ArticleGoogle Scholar
- Palinkas LA, Holloway IW, Rice E, Brown CH, Valente T, Chamberlain P. Influence network linkages across treatment conditions in randomized controlled trial of two strategies for scaling up evidence-based practices in public youth-serving systems. Implementation Sci. 2013;8(1):133.View ArticleGoogle Scholar
- Valente T. Network interventions. Science. 2012;337(6090):49–53. doi:10.1126/science.1217330.View ArticlePubMedGoogle Scholar
- Mohr DC, Cheung K, Schueller SM, Hendricks Brown C, Duan N. Continuous evaluation of evolving behavioral intervention technologies. Am J Prev Med. 2013;45(4):517–23. http://dx.doi.org/10.1016/j.amepre.2013.06.006 PMCID: PMC3828034.View ArticlePubMedGoogle Scholar
- Padwa H, Ni YM, Barth-Rogers Y, Arangua L, Andersen R, Gelberg L. Barriers to drug use behavior change among primary care patients in urban United States community health centers. Subst Use Misuse. 2013. doi:10.3109/10826084.2013.866962.PubMedPubMed CentralGoogle Scholar
- Prado G, Cordova D, Huang S, Estrada Y, Rosen A, Bacio GA, McCollister K. The efficacy of Familias Unidas on drug and alcohol outcomes for Hispanic delinquent youth: main effects and interaction effects by parental stress and social support. Drug Alcohol Depend. 2012:125, Supplement 1(0); S18-S25. doi:http://dx.doi.org/10.1016/j.drugalcdep.2012.06.011.
- Stormshak EA, Fosco GM, Dishion TJ. Implementing interventions with families in school to increase youth school-engagement: the family check-up model. Sch Ment Health. 2010;2:82–92. doi:10.1007/s12310-009-9025-6.View ArticleGoogle Scholar
- Connell AM, Dishion TJ, Yasui M, Kavanagh K. An adaptive approach to family intervention: linking engagement in family-centered intervention to reductions in adolescent problem behavior. J Consult Clin Psychol. 2007;75(4):568–79. http://dx.doi.org/10.1037/0022-006X.75.4.568.View ArticlePubMedGoogle Scholar
- Dishion TJ, Kavanagh K, Schneiger A, Nelson S, Kaufman NK. Preventing early adolescent substance use: a family-centered strategy for the public middle school. Prev Sci. 2002;3(3):191–201.View ArticlePubMedGoogle Scholar
- Van Ryzin MJ, Stormshak EA, Dishion TJ. Engaging parents in the family check-up in middle school: longitudinal effects on family conflict and problem behavior through the high school transition. J Adolesc Health. 2012;50(6):627–33. doi:10.1016/j.jadohealth.2011.10.255. PMCID: PMC3360879.View ArticlePubMedPubMed CentralGoogle Scholar
- Connell AM, Dishion TJ. Reducing depression among at-risk early adolescents: three-year effects of a family-centered intervention embedded in schools. J Fam Psychol. 2008;22(4):574–85.View ArticlePubMedPubMed CentralGoogle Scholar
- Brown E, Hawkins J, Arthur M, Briney J, Abbott R. Effects of communities that care on prevention services systems: findings from the community youth development study at 1.5 years. Prev Sci. 2007;8(3):180–91. doi:10.1007/s11121-007-0068-3. PMID: 17602298.View ArticlePubMedGoogle Scholar
- Dolan LJ, Kellam SG, Brown CH, Werthamer-Larsson L, Rebok GW, Mayer LS, Wheeler L. The short-term impact of two classroom-based preventive interventions on aggressive and shy behaviors and poor achievement. J Appl Dev Psychol. 1993: 14(3); 317-345.Google Scholar
- Ialongo NS, Werthamer L, Kellam SG, Brown CH, Wang S, Lin Y. Proximal impact of two first-grade preventive interventions on the early risk behaviors for later substance abuse, depression, and antisocial behavior. Am J Community Psychol. 1999;27(5):599–641.View ArticlePubMedGoogle Scholar
- Petras H, Kellam SG, Brown CH, Muthén BO, Ialongo NS, Poduska JM. Developmental epidemiological courses leading to antisocial personality disorder and violent and criminal behavior: effects by young adulthood of a universal preventive intervention in first- and second-grade classrooms. Drug Alcohol Depend. 2008;95(Suppl1):S45–59. http://dx.doi.org/10.1016/j.drugalcdep.2007.10.015 PMCID: PMC2706504.View ArticlePubMedPubMed CentralGoogle Scholar
- Poduska JM, Kellam SG, Wang W, Brown CH, Ialongo NS, Toyinbo P. Impact of the good behavior game, a universal classroom-based behavior intervention, on young adult service use for problems with emotions, behavior, or drugs or alcohol. Drug Alcohol Depend. 2008;95(Suppl1):S29–44. Supplementary data associated with this article can be found in the online version at doi:http://dx.doi.org/10.1016/j.drugalcdep.2007.10.009 PMCID: PMC2757275.View ArticlePubMedPubMed CentralGoogle Scholar
- Wilcox HC, Kellam SG, Brown CH, Poduska JM, Ialongo NS, Wang W, Anthony JC. The impact of two universal randomized first- and second-grade classroom interventions on young adult suicide ideation and attempts. Drug Alcohol Depend. 2008;95 Suppl 1:S60–73. Supplementary data associated with this article can be found, in the online version, at doi:10.1016/j.drugalcdep.2008.01.005. doi:http://dx.doi.org/10.1016/j.drugalcdep.2008.01.005 PMCID: PMC2637412.View ArticlePubMedPubMed CentralGoogle Scholar
- Chamberlain P, Price J, Leve LD, Laurent H, Landsverk JA, Reid JB. Prevention of behavior problems for children in foster care: outcomes and mediation effects. Prev Sci. 2008;9(1):17–27.View ArticlePubMedPubMed CentralGoogle Scholar
- Chamberlain P, Price J, Reid J, Landsverk J. Cascading implementation of a foster and kinship parent intervention. Child Welfare. 2008;87(5):27–48. PMCID: PMC2676450.PubMedPubMed CentralGoogle Scholar
- Leve L, Chamberlain P. A randomized evaluation of multidimensional treatment foster care: effects on school attendance and homework completion in juvenile justice girls. Res Social Work Prac. 2007;17:657–63.View ArticleGoogle Scholar
- Smith DK, Chamberlain P, Eddy JM. Preliminary support for multidimensional treatment foster care in reducing substance use in delinquent boys. J Child Adolesc Subst Abuse. 2010;19(4):343–58.View ArticlePubMedPubMed CentralGoogle Scholar
- Kim HK, Leve LD. Substance use and delinquency among middle school girls in foster care: a three-year follow-up of a randomized controlled trial. J Consult Clin Psychol. 2011;79(6):740–50. doi:10.1037/a0025949. PMCID: PMC3226884.View ArticlePubMedPubMed CentralGoogle Scholar
- Kim HK, Pears KC, Leve LD, Chamberlain PC, Smith DK. Intervention effects on health-risking sexual behavior among girls in foster care: the role of placement disruption and tobacco and marijuana use. J Child Adolesc Subst Abuse. 2013;22(5):370–87. doi:10.1080/1067828X.2013.788880. PMCID: PMC3772734.View ArticlePubMedPubMed CentralGoogle Scholar
- Botvin GJ, Griffin k. Life skills training: empirical findings and future direction. J Prim Prev. 2004;25(2):211.View ArticleGoogle Scholar
- Botvin GJ, Griffin K. Advances in the science and practice of prevention: targeting individual-level etiologic factors and the challenge of going to scale. In: Scheier LM, editor. Handbook of drug use etiology: Theory, methods, and empirical findings. Washington, DC: American Psychological Association; 2010. p. 631–50.Google Scholar
- Botvin GJ. Preventing drug abuse in schools: social and competence enhancement approaches targeting individual-level etiologic factors. Addict Behav. 2000;25(6):887–97.View ArticlePubMedGoogle Scholar
- Griffin KW, Botvin GJ, Nichols TR. Effects of a school-based drug abuse prevention program for adolescents on HIV risk behavior in young adulthood. Prev Sci. 2006;7(1):103–12. doi:10.1007/s11121-006-0025-6.View ArticlePubMedGoogle Scholar
- Olds D, Henderson Jr CR, Cole R, Eckenrode J, Kitzman H, Luckey D, Pettitt L, Sidora K, Morris P, Powers J. Long-term effects of nurse home visitation on children’s criminal and antisocial behavior: 15-year follow-up of a randomized controlled trial. JAMA. 1998;280(14):1238–44. doi:joc80422 [pii].View ArticlePubMedGoogle Scholar
- Sapiano TN, Moore A, Kalayil EJ, Zhang X, Chen B, Uhl G, Williams W. Evaluation of an HIV prevention intervention designed for African American women: results from the SISTA community-based organization behavioral outcomes project. AIDS Behav. 2013: 17(3); 1052-1067. doi:10.1007/s10461-012-0292-0.
- Wingood G, DiClemente RJ, Robinson-Simpson L, Lang DL, Caliendo A, Hardin JW. Efficacy of a HIV prevention intervention in reducing incident high-risk HPV, nonviral STIs and sexual concurrency among African-American women: a randomized controlled trial. J Acquir Immune Defic Syndr. 2013;63 Suppl 1:S36–43. doi:10.1097/QAI.0b013e3182920031.View ArticlePubMedPubMed CentralGoogle Scholar
- Murray VM, Simons RL, Simons LG, Gibbons FX. Contributions of family environment and parenting processes to sexual risk and substance use of rural African American males: a 4-year longitudinal analysis. Am J Orthopsychiatry. 2013;83(2 Pt 3):299–309. doi:10.1111/ajop.12035.View ArticleGoogle Scholar
- Murray VM, Berkel C, Brody GH, Gibbons M, Gibbons FX. The strong African American families program: longitudinal pathways to sexual risk reduction. J Adolesc Health. 2007;41(4):333–42. doi:10.1016/j.jadohealth.2007.04.003.View ArticleGoogle Scholar
- Brody GH, Murry VM, Gerrard M, Gibbons FX, Molgaard V, McNair L, Neubaum-Carlan E. The strong African American families program: translating research into prevention programming. Child Dev. 2004:75(3);900–17.View ArticlePubMedGoogle Scholar
- Brody GH, Murry VM, Kogan SM, Gerrard M, Gibbons FX, Molgaard V, Wills TA. The strong African American families program: a cluster-randomized prevention trial of long-term effects and a mediational model. J Consult Clin Psychol. 2006:74(2);356–366.87.View ArticlePubMedGoogle Scholar
- Czaja SJ, Sharit SJ, Charness N, Fisk AD. The Center for Research and Education on Aging and Technology Enhancement (CREATE): A program to enhance technology for older adults. Gerontechnology. 2001;1:50–59.View ArticleGoogle Scholar