- Open Access
- Open Peer Review
An intervention to improve program implementation: findings from a two-year cluster randomized trial of Assets-Getting To Outcomes
Implementation Science volume 8, Article number: 87 (2013)
Studies have shown that communities have not always been able to implement evidence-based prevention programs with quality and achieve outcomes demonstrated by prevention science. Implementation support interventions are needed to bridge this gap between science and practice. The purpose of this article is to present two-year outcomes from an evaluation of the Assets Getting To Outcomes (AGTO) intervention in 12 Maine communities engaged in promoting Developmental Assets, a positive youth development approach to prevention. AGTO is an implementation support intervention that consists of: a manual of text and tools; face-to-face training, and onsite technical assistance, focused on activities shown to be associated with obtaining positive results across any prevention program.
This study uses a nested and cross-sectional, cluster randomized controlled design. Participants were coalition members and program staff from 12 communities in Maine. Each coalition nominated up to five prevention programs to participate. At random, six coalitions and their respective 30 programs received the two-year AGTO intervention and the other six maintained routine operations. The study assessed prevention practitioner capacity (efficacy and behaviors), practitioner exposure to and use of AGTO, practitioner perceptions of AGTO, and prevention program performance. Capacity of coalition members and performance of their programs were compared between the two groups across the baseline, one-, and two-year time points.
We found no significant differences between AGTO and control group’s prevention capacity. However, within the AGTO group, significant differences were found between those with greater exposure to and use of AGTO. Programs that received the highest number of technical assistance hours showed the most program improvement.
This study is the first of its kind to show that use of an implementation support intervention-AGTO -yielded improvements in practitioner capacity and consequently in program performance on a large sample of practitioners and programs using a randomized controlled design.
Prevention programming can improve youth outcomes and be cost effective . While programs need to be well implemented, they also require practices such as tracking outcomes and engaging in continuous improvement-to reap these benefits . Yet, many studies have shown that communities have not been able to implement evidence-based prevention programs with such quality and achieve outcomes demonstrated by prevention science [3–8]. This ‘ gap’ between science and practice (e.g., [9, 10]) can result when communities lack individuals with the capacity-defined as the self-efficacy and behaviors-needed to engage in critical prevention practices.
Getting To Outcomes® (GTO®)a was developed to bridge this gap between science and practice. GTO is an implementation model (Figure 1)-specifying ten steps (or sets of activities) prevention practitioners should take, shown to be associated with obtaining positive results across many different prevention programs . The first six involve planning activities (needs assessment, goal setting, choosing programs, ensuring appropriate capacity and fit, planning program details), the next two steps are process and outcome evaluation [7, 8], and the last two steps involve using data to improve programs and sustaining programs [9, 10]. GTO is also an implementation support intervention, which strengthens the efficacy and behaviors (collectively known as capacity) community practitioners need to perform these activities with quality. GTO has been found to improve the capacity of individual practitioners and the performance of drug prevention programs in both quasi-experimental  and randomized controlled trials . Congruent with social cognitive theories of behavioral change [14–17], the efficacy and behaviors related to the activities targeted by GTO’s 10 steps (i.e., prevention ‘ capacity’ at the individual level), are related to how well prevention is carried out at the program level , which, in turn, affects outcomes.
GTO has been adapted to a number of content areas, recently to the model of positive youth development developed by Search Institute. This model, called ‘ Developmental Assets,’ states that in order to thrive, young people need 40 key developmental supports and experiences that include but are broader than the notion of protective factors. From a caring school climate to participation in high-quality after-school programs, these factors have been shown to predict health and well-being outcomes consistently across sex, race-ethnicity, and family income [19–21]. This approach was blended into the 10 GTO steps, so that each step was enhanced to include a specific focus on building assets. For example, in step one (choose which problem(s) to focus on) individuals use both asset indicators, as well as risk information, to help select priorities. This blended approach is called Assets-Getting To Outcomes (AGTO) .
The AGTO model and corresponding intervention is grounded in implementation theory. For example, AGTO operationalizes the Consolidated Framework for Implementation Research (CFIR) to ensure that all the major domains that influence implementation are considered . Intervention characteristics (e.g., evidence strength and quality, relative advantage, complexity, adaptability), the first domain, can influence whether practitioners adopt an intervention. Without adaptation, many interventions come to settings as a poor fit. Even if these interventions have been known to improve outcomes, practitioners do not always adopt these interventions, underscoring the need for core intervention elements and what the CFIR terms an adaptable intervention periphery (i.e., elements that can be adapted to improve intervention fit . The AGTO intervention walks practitioners through a systematic process to assess, and then improve, how well interventions fit with their target population, organization, and broader community (step four: Fit).
The next three domains of the CFIR comprise the outer setting (e.g., broader social, political and economic context including policies, incentives, resources etc.), inner setting in which the intervention is implemented (e.g., context of the specific organization or group implementing the intervention including the structural characteristics, relationships, implementation readiness), and the characteristics of the individuals involved (e.g., knowledge, skills). Attention needs to be paid to factors both at the individual and organizational levels that impact the degree to which new practices are adopted and implemented. For example, a number of studies have shown that factors at both the individual level (e.g., training, skills, efficacy, involvement in decision making, and satisfaction of teachers) and at the organizational level (organization size, climate, and financial resources, and active support of principals and administrators) have been shown to predict successful implementation of prevention in public schools [3, 24–28]. In the AGTO study, the inner setting is the coalition, the outer setting is the broader community where coalitions are located, and the individuals involved are the prevention practitioners who are coalition members. To create both capacity and conditions for successful implementation of programs, the AGTO intervention targets both practitioners and coalitions. While AGTO helps the inner setting and individuals account for the outer setting, it is beyond its scope to specifically alter the broader social, political, and economic context.
Implementation process or the active change process aimed to achieve individual and organization use of the intervention as designed is the final domain in the CFIR. Research suggests that innovative multi-level strategies are needed to improve the implementation process. Passive approaches such as trainings by themselves do not lead to change, as attendees often experience barriers to incorporating newly learned information into their work [29, 30]. The AGTO intervention, which is proactive versus passive, specifically targets improvements to this implementation process at both the individual and program level so that implementation more closely aligns with empirically-based, high-quality prevention processes.
The purpose of this article is to present two-year outcomes from an evaluation of AGTO in 12 Maine communities engaged in promoting Developmental Assets (interim one year outcomes have been presented elsewhere ). At random, six coalitions and their respective programs received the two-year AGTO intervention and the other six maintained routine operations. Capacity of coalition members and performance of their programs were compared between the two groups across the baseline, one-, and two-year timepoints. The study had the following hypotheses:
Hypothesis one: Individual prevention practitioners in the AGTO group would demonstrate greater gains in capacity than those in the control group.
Hypothesis two: Individual prevention practitioners with greater exposure to and use of AGTO would demonstrate greater gains in capacity than those who were assigned to use AGTO but used less or none.
Hypothesis three: Whole prevention programs in the AGTO group would demonstrate significant gains in prevention performance (i.e., performing various prevention tasks with high quality) overall and in proportion with their use of AGTO.
Building upon previous GTO studies, the current study is the first to evaluate GTO on both individual capacity and program performance in a randomized trial with a larger number of programs in a way that uses a traditional approach to conducting a randomized trial, while being flexible enough to account for the ‘ complex, interacting, multi-level, and transients states of constructs in the real world’. Consistent with CFIR, the study attempts to evaluate AGTO in terms of its impact on three of the CFIR domains: inner setting, individual characteristics, and implementation process; while using measures of the outer setting as covariates. Table 1 provides a summary of how the study design aligns with and measures the CFIR domains.
The study was a nested cross-sectional matched pairs group randomized trial . Twelve coalitions were randomized to either receive the two-year AGTO intervention or to serve as comparison coalitions. Prior to randomization, coalitions were matched into pairs based on the total population and demographic characteristics of the community served by each coalition (from the 2000 Census) and a rating of each coalition’s functioning at baseline. Two technical assistance providers and a local coalition expert came to consensus to derive the pairings based on each participating coalition’s overall stage of development, as well as their stage of development in four functional areas: coalition development and management, coordination of prevention programs and services, conducting environmental strategies, and acting as intermediary or support organization to other community organizations and coalitions. Coalitions were rated on their mastery of each functional area on a Likert scale. The matched pairs were sent to the project statistician, who allocated one coalition per matched pair to the AGTO intervention by using the random number generator in Microsoft Excel. Programs and practitioners followed their coalition assignment. The rating and resulting four-stage categorization were developed by Office of National Drug Control Policy’s evaluation of the Drug-Free Communities program after reviewing the literature on coalition functioning and receiving input from experts and coalition leaders . Analysis of these stages within the Drug-Free Communities program showed that the more advanced coalitions had stronger prevention practices and better outcomes . As will be seen below, it was not possible to blind research staff or participants to their study condition.
Setting: 12 community-based coalitions
Participants were all the coalition members and program staff from 12 coalitions in Maine. Community-based coalitions are comprised of prevention practitioners from non-profit organizations from multiple levels (individual, organizational, policy) and sectors (parents, youth, criminal justice, education) that collaborate to improve community health . These 12 coalitions are similarly structured (a small group of paid staff working with a larger group of volunteers), work in similar geographic and demographics settings,b and have similar rates of reported youth risk behaviors (e.g., alcohol and drug use),c missions (to promote healthy youth development), annual budgets, and numbers of distinct programs. However, it is important to note that membership and leadership in coalitions are not static, but change over time. These coalitions represent the inner setting defined by the CFIR.
Coalition members, characterized by the CFIR as the individuals involved, completed a coalition survey at baseline (just prior to the AGTO intervention, April 2009), mid (about one year later, June 2010) and post (about two years later, July 2011). After randomization, but prior to completing the survey, we received approval from RAND’s Human Subjects Protection Committee for the study and written consent was obtained from all participants. The overall response rate was 82% across all sites at baseline, 79% at mid, and 89% at post. Community coalitions often experience high turnover and the same was true here, as 41% percent of those surveyed at baseline or mid were also surveyed at post. New participants joined the coalition after baseline as well. There were 376 survey participants at baseline, 303 at mid, and 315 at post (152 of whom had participated in the baseline survey and 163 who had not), yielding a total of 591 unique participants. The size of the clusters used in the analyses (in this case, coalitions) ranged from n = 8 to n = 50 across the 12 coalitions and three time points. In lieu of a CONSORT diagram (Additional file 1), Table 2 presents additional response rate and sample size information is, which better presents this information for this study. At baseline and post, we compared the distributions of several participant characteristics (see covariates below) between the intervention and control groups using ordinal, logistic, or linear regression, with the AGTO group indicator as a predictor variable (see Table 3). We found no statistically significant differences in characteristics between the groups at either baseline or post, except for satisfaction with involvement in the coalition. The AGTO group reported significantly higher satisfaction at post than the comparison group (p = 0.003). Analyses were adjusted to account for this statistically significant difference, and given that it was one of a large number of t-tests we do not anticipate this difference influenced the findings presented in this article.
Each coalition nominated up to five prevention programs to participate in the study (one-half of the programs were assigned to intervention group, one-half to the control group). Despite some variability, all programs aimed to promote healthy development with middle and high school youth. Although some were evidence-based programs including Project ALERT , Lifeskills Training , and Making Proud Choices [37–39], most were designed by coalitions, including mentoring, social norms campaigns, programs to divert first offenders from the juvenile justice system, and leadership training. Most programs were ongoing at the outset of the study.
The AGTO intervention includes three types of assistance which are adapted to fit the needs and priorities of the individuals involved, as well as the inner and outer setting: a manual of text and tools; face-to-face training, and onsite technical assistance (TA). These three types of assistance aim to improve the implementation process for each program. Two full-time, Maine-based staff, one with a master’s and one with a bachelor’s degree, provided AGTO tools, training, and TA to the intervention coalitions and programs during the two-year intervention period. The tools are in the Search Institute-published manual, Getting To Outcomes with Developmental Assets: Ten steps to measuring success in youth programs and communities, which all intervention participants received. The training was delivered separately to each coalition over a full day after baseline, and covered the AGTO model, tools in the manual, and an introduction to the TA process. Based on TA literature [40–43], the AGTO-based TA involves three structured steps, including an initial diagnosis of program functioning, development of a logic model, and development of a plan for how the TA and program staff were to make improvements, carried out during and in between bi-weekly TA visits. TA staff provided consultation and feedback to practitioners on conducting AGTO tasks as applied to their program [44–46]. An expanded discussion about the AGTO intervention is available elsewhere .
The coalition survey included demographic information about prevention practitioners (i.e., age, gender, race/ethnicity, education, and employment status), consistent with CFIR’s individual characteristics domain. Measures related to coalition functioning, leadership, and the incorporation of new practices were also included in the coalition survey, and relates to the inner setting domain of CFIR. The leadership score  is the mean of ten items rating the effectiveness of coalition leadership from five response options (1 = ‘ poor’ to 5 = ‘ excellent’), αBL = 0.96, αmid = 0.95, αpost = 0.96. From the staff survey of organizational readiness for change measure , Cohesion (αBL = 0.86, αmid = 0.82, αpost = 0.86) and receptivity to change scores (αBL = 0.73, αmid = 0.72, αpost = 0.75) were adapted, averaging across five response options (1 = ‘ strongly disagree’ to 5 = ‘ strongly agree’). Stand-alone coalition survey items used in the analysis were overall satisfaction with involvement in the coalition (1 = ‘very dissatisfied’ to 7 = ‘very satisfied’), years of involvement in the coalition, and type of involvement (paid staff, volunteer individual, or volunteer from a partner organization).
Practitioner prevention capacity
Assessed in the coalition survey, prevention capacity was defined as efficacy and behaviors of practitioners and relates to CFIR’s individual characteristics domain. There are two scales that assess perceived efficacy (i.e., completing various tasks without assistance) using a three-point scale (1 = ‘ would need a great deal of help to carry out this task,’ 2 = ‘ could carry out this task, but would need some help,’ 3 = ‘ could carry out this task without any help’). The asset efficacy scale includes five items asking about helping adults connect with youth; involving multiple sectors of the community; engaging youth in asset-building; incorporating assets into existing programs; and helping community leaders understand how their decisions affect the development of youth (αBL = 0.81, αmid = 0.86, αpost = 0.82). The GTO efficacy scale includes items asking about activities from the GTO steps -e.g., evaluation, planning, quality improvement (αBL = 0.84, αmid = 0.89, αpost = 0.88).
There are three behavior scales that are means of items with seven-point scales (1 = ‘ never’ to 7 = ‘ very often’) assessing the frequency with which respondents engaged in these activities during the previous 12 months. The Asset Behaviors scale includes five items assessing whether individuals are motivating both adults and youth to become asset-builders; incorporating asset building into existing youth programs; influencing community leaders to implement asset-aligned policies (e.g., employ funding criteria that rewards asset-building); and engaging various sectors of the community to support asset building (αBL = 0.91, αmid = 0.93, αpost = 0.93). GTO behaviors include 11 items that align with the 10 steps (two items focus on needs and resources assessment) (αBL = 0.92, αmid = 0.94, αpost = 0.95). AGTO behaviors includes 11 items that tailor the GTO 10 steps to Assets (e.g., needs assessment on youth assets) (αBL = 0.94, αmid = 0.96, αpost = 0.95). The GTO scales are shortened versions of scales used in previous studies (e.g., ). All Assets scales were developed for the current study, based on the structure used for the GTO scales. Although the goal of the intervention was build practitioners’ capacity related to the blended AGTO model, GTO and assets efficacy and behaviors were asked about separately to identify which aspects of their capacity improved and to avoid masking potential effects of the intervention (e.g., practitioners that apply GTO to other areas of their work life, may have disproportionately greater increases in capacity related to GTO).
Use of and exposure to AGTO
To assess AGTO’s potential impact on each program’s implementation process, we first documented the degree to which each program engaged in the AGTO intervention through the AGTO participation index, which is the sum of six true/false items added to the mid and post coalition survey. Based on the Hall et al. model of categorizing the degree to which individuals ‘ use’ an innovation [49–51], these items assess key markers of use including participation in training, reading the materials, planning, discussing the model with colleagues, securing resources, and receiving TA. Exposure to AGTO was also documented by TA providers recording hours of TA they delivered to each program, by AGTO step. The Participation Index and hours of TA have been shown to be related to prevention capacity and performance in a previous study of GTO [12, 52]. A dichotomous measure was created if a user participated (AGTO participation index ≥1) at either mid or post. Most users who participated at mid also participated at post; however, there were a few cases in which respondents were documented to have used AGTO at mid but not at post (n = 4), or used at post, but not at mid (n = 19). Users at mid but not at post were assumed to hold residual knowledge or exposure at post. Preliminary analysis showed no significant differences between outcome measures of these groups and thus we felt confident characterizing participation as a single dichotomous measure of ever using (AGTO participation index ≥1) versus never using (AGTO participation index = 0).
A structured interview was used to assess the impact of AGTO on the implementation process, administered on the same timeline as the coalition survey. Prevention practitioners performance of tasks associated with high-quality prevention targeted by AGTO were captured through the interview. Whole programs are rated, not individuals, because programs operate as a unit. Using the interview responses, a set of ratings were made assessing performance of activities of seven key domains: goals and objectives, best practices, planning, process evaluation, outcome evaluation, continuous quality improvement, and sustainability. Because programs were mid-implementation the other three AGTO domains (needs assessment, fit, and capacity), all of which are pre-program implementation activities, were not assessed. The ratings come from 10 items (or ‘ components’) that assess how well each of the abovementioned activities are performed over the last year. Each component has seven response choices, described with specific behaviors, that range from ‘ highly faithful = 7’ to ‘ highly divergent = 1’ from ideal performance. This measure has been shown to be sensitive to change and reliable in previous GTO studies [12, 53]. We calculated a score for each of the 10 components and a total score (i.e., average across all 10 components) for both the intervention and control-assigned programs at baseline and post. We then calculated the percent change for each component and the total score between baseline and post for all programs. In addition to the ratings, it was simply noted for each program whether they had conducted any process and outcome evaluation at each time point (yes/no).
Trained research staff conducted and audiotaped the structured interviews by telephone with the directors of 51 programs at baseline. The 32d programs that were still operating two years later were interviewed again at post (AGTO = 17, control = 15). A second coder double-rated 10% of the programs using the audio recordings. The two raters discussed the scoring to reach a consensus when there were discrepancies. We calculated percent agreement and estimated a prevalence-adjusted bias-adjusted kappa (PABAK)  because of the potential impact on kappa from the relatively high prevalence of positive ratings. At baseline and post, respectively, the percent agreement between raters was 79% and 99% and the PABAK was 0.59 and 0.99.
Perceptions of AGTO
After the intervention concluded, we conducted semi-structured interviews with staff from 13 programs (March to May, 2012) to gather practitioners’ perceptions of AGTO, focusing on ease of use and satisfaction with AGTO as well as the benefits and challenges they experienced with AGTO. These are all characteristics that influence implementation success and are consistent with the intervention characteristics identified by the CFIR. A convenience sample of staff that were consistently engaged in the project (i.e., no turnover) and significant users of AGTO were selected for these interviews because they were determined to be those most likely to have the knowledge and experience needed for the interview.
Assessing changes in prevention capacity
We used two approaches to assess changes in capacity: an intent-to-treat analysis and because AGTO is a voluntary intervention, an analysis comparing users of AGTO to non-users. Our data was clustered by individuals within matched coalition, therefore, for the intent-to-treat analysis we fit a model with two sets of random effects, one for matched coalitions and one with random intercepts and linear functions of time to account for trajectories of repeated observations within respondent. The model included three fixed effect terms: group (AGTO versus control), time (baseline = 0 years, mid = 1 year, post = 2 years), and an interaction between group and time. The interaction term (i.e., ‘ difference of differences’) was tested to determine whether the slope of the best-fit line across all three time points varied by group. The data were analyzed in accordance with intervention assignment. For the AGTO user versus non-user analysis, we repeated the intent-to-treat models, but instead of comparing the AGTO to non-AGTO groups we compared AGTO users (AGTO participation index ≥1) to AGTO non-users (Index = 0). To ensure these groups were equivalent we conducted preliminary analyses to determine if there were significant differences in baseline capacity measures. Only one significant difference was detected in GTO behaviors (p = 0.047, SE = 3.629), and given that it was one of a large number of t-tests we do not anticipate this difference influenced the findings presented in this article.
Assessing use of and exposure to AGTO
Means and percentages were calculated for the participation index, and its individual components, for both study groups.
Program performance and its relationship to AGTO exposure
Given the small number of programs in this analysis, we were limited to primarily descriptive analyses. To assess the relationship between the exposure to AGTO (i.e., TA hours, by performance domain) and change in program performance, we aggregated across all AGTO programs and treated each AGTO domain as a case (i.e., n = 7), and calculated the relationship of TA hours spent on each domain with the amount of change exhibited in that domain’s Performance rating.
Assessing perceptions of AGTO
We used constant comparative analysis to analyze structured interview data . A member of the research team read each interview transcript and coded text to identify categories representing program staff perceptions of AGTO. Thirty-eight percent of transcripts were read and coded by two different team members. Inter-rater reliability was 0.86. After coding, the codes were sorted and categorized into themes. Thirty-one themes (e.g., It is easier to implement AGTO when it is integrated into routine activities) were identified and grouped into categories (e.g., factors that facilitated the implementation of AGTO).
Prevention capacity of AGTO versus control
Model-based mean estimates at each of the time points are shown in Table 4 along with tests of the change over time within group and between groups. In the intent-to-treat analysis, assets efficacy within the control group significantly increased over time (a 2.6 point increase, SE = 1.23) between each time point, making a 5.2 increase from baseline to post; p = 0.035. The control group showed a decline in GTO behaviors (adjusted averaged decline of 4.3 points baseline to post) but the change was not significant (p = 0.055). The amount of change between the intervention and control groups did not significantly differ across the three time points for assets efficacy, GTO behaviors, or for any of the other prevention capacity scales in tests of condition by time interaction.
The clustering among the five outcomes was low. It was 0.00 for assets efficacy (95% CI, 00.0, 0.09) and GTO behaviors (95% CI, 00.0, 0.07), 0.01 for GTO efficacy (95% CI, 00.0, 0.09) and assets behaviors (95% CI, 00.0, 0.07), and 0.02 for AGTO behaviors (95% CI, 00.0, 0.07). All clusters were represented in the analyses.
Use of and exposure to AGTO
The participation index showed that 47% of those in the AGTO group (N = 281) had used some portion of the intervention (participation index ≥1, M = 1.94, SD = 2.37) by reading AGTO materials (37%), participating in training (36%), talking to others about AGTO (33%) or making plans to use it (32%), receiving TA (30%), or securing resources for AGTO (24%). For the non-AGTO group, participation was at 20.6% (N = 64). Participation from reading most of the materials was 12.6%, 11.3% made plans to use, 7.7% talked to others, 7.1% participated in training, 5.5% secured or tried to secure resources, and 2.9% received TA. Mean use among the non-AGTO group was 0.48 (SD 1.09) and ranged from 1 to 5. Use of AGTO varied across each coalition (see Table 2).
Prevention capacity of users and non-users in the AGTO group
Four prevention capacity scales in the ‘ use/non-use’ analyses (Table 5) had significant difference of difference tests: assets efficacy (p = 0.020), GTO efficacy (p = 0.005), assets behaviors (p = 0.003), and AGTO behaviors (p = 0.0003). The average difference across two time points between the AGTO use and AGTO non-use groups ranged from 6.1 to 9.3 points, resulting in a range of 12.2 to 18.6 from baseline to post. Figure 2 shows the model-based means at each time point for the four scales with significant difference of differences.
Program performance and its relationship to AGTO exposure
Noteworthy improvements for the AGTO programs were made in the domains of goals and process and outcome evaluation (Table 6). 64% of participating programs began a new process evaluation and 29% began a new outcome evaluation since receiving AGTO. Among the control group, only outcome evaluation had any improvement. The correlation between improvement in each domain and the TA hours spent on that domain was not statistically significant, possibly due to a small sample size [r(7) = 0.66, p = 0.102]. Best practices, sustainability, planning, and continuous quality improvement did not improve or declined, despite having some TA hours spent on them. However, with greater hours, more improvement was noted in goals and process and outcome evaluation.
Perceptions of AGTO
The interviews with staff who had used AGTO suggested some contextual factors that may both facilitate (e.g., key leader support) and hinder (e.g., staff turnover) the implementation of AGTO, as well as characteristics of the AGTO intervention that may have supported (e.g., access to a qualified TA provider) or challenged (e.g., complexity of the model) its implementation (Table 7). Factors that may facilitate implementation include whether AGTO was integrated into routine activities (31%), the number of people trained (23%), and key leader support (15%); whereas, barriers included diminishing funding (31%) and staff turnover (15%). Interviewees reported the AGTO process was beneficial because it offered access to a qualified TA provider (100%), a wide range of written support materials (92%), and peer-to-peer networking opportunities (85%), among other benefits. However, the process was also challenging for users who viewed it as cumbersome (54%), at times, complex (46%), and in competition with regular job duties (46%). Despite these challenges interviewees described a series of accomplishments as a result of engaging in AGTO, some of which sustained past the end of TA. Among the most notable accomplishments were that programs used more systematic processes for their work (100%), started collecting outcome evaluation data (100%), established stronger partnerships (92%), and integrated more evidence-based practices into their work (92%). Evaluation efforts were sustained by 77% of interviewees.
Using the implementation research typology outlined in CFIR, this study evaluated the AGTO intervention’s impact on the capacity of individual prevention practitioners (i.e., CFIR’s individual characteristics) and the performance of whole programs (CFIR’s implementation process), while accounting for several factors in CFIR’s inner setting domain. Contrary to study hypothesis one, the intent-to treat analyses showed prevention practitioners in the control group demonstrated significant gains in assets efficacy, although the control group’s GTO behaviors declined. There were no other significant changes over time for either group on any of the other measures in these analyses. The variation in AGTO use within the treatment group and use of AGTO in the control group potentially contributed to these non-significant findings. There are a few possible explanations for why individuals in the non-AGTO group reported use of AGTO. Since AGTO is a set of well-known practices for the implementation of high-quality prevention, it is possible that individuals would have been exposed to or using some of these practices. In addition, the AGTO book is freely available for purchase by anyone and in some instances control coalition members attended AGTO trainings.
While intent-to-treat analyses are more traditional, they may be overly conservative given that GTO is a voluntary intervention. To help address these potential analytic limitations, the ‘ use/no-use’ analyses were conducted and showed, consistent with study hypothesis two, significant differences between those with greater exposure to and use of AGTO. Another advantage of these analyses is that they avoid the potential biases of contamination (i.e., control groups using AGTO) and differential response rates between intervention and control groups common in implementation research. We also found that whole prevention programs in the AGTO group demonstrated significant gains in certain domains of prevention performance (i.e., performing various prevention tasks with high quality) overall and related to their use of TA (hypothesis three).
In the interviews, practitioners stated that they generally valued AGTO, especially the TA component, and found it helpful to their work. However, practitioners also experienced challenges of not having enough time and resources to sufficiently carry out AGTO, further highlighting the need for TA. Prior research has indicated that these types of costs (i.e., time and resources) can be negatively associated with implementation . The importance and contribution of TA to improving the implementation process is consistent with past GTO  and other studies that involve TA [58, 59].
Although replicating earlier findings on GTO’s impact on capacity [12, 52] and program performance [12, 52, 53], this study is the first of its kind to show that an implementation support intervention-AGTO-yielded improvements in practitioner capacity and consequently in program performance on a larger sample of practitioners and programs using a randomized controlled design. These findings support the theory behind implementation support interventions generally and AGTO specifically, which posit that improving practitioner capacity to engage in the activities of the 10 steps of AGTO will consequently improve program performance. The final link in the GTO theoretical chain (implementation support→capacity→program performance→youth outcomes) is currently being tested in an ongoing multi-site, randomized trial (1R01HD069427-01, Chinman, PI), with results expected in 2014.
Findings from the current study show that using AGTO helps improve the quality of real world implementation. There are other interventions that aim to build capacity-e.g., in the area of alcohol and drug prevention (e.g., Communities That Care  and PROmoting School-community-university Partnerships to Enhance Resilience ). However, unlike these interventions that provide support and resources to help community practitioners select and begin implementing evidence-based prevention programs, AGTO engages existing programs (whether evidence-based or home-grown) and the leadership of organizations that house these programs to improve their quality through support with no additional financial resources for implementation. As such, the goal of the AGTO intervention is to help the leadership of community organizations to integrate the practices AGTO targets into routine operations.
The methods used in the current study also have implications for broader implementation research using the CFIR. The study used measures to assess the quality of the implementation process as well as the relevant capacity of practitioners (i.e., individuals involved) that could be adapted to many other interventions, and exist already for interventions in the areas of substance abuse prevention, positive youth development, prevention of teen pregnancy and sexually transmitted infections, and homelessness prevention. These measures were used in an randomized controlled trial and our analytic approach addressed certain challenges of conducting research on real-world implementation (e.g., the turnover of membership at participating coalitions, differential response rates between intervention and control groups). More innovative methods and more acceptance of nontraditional research methods will be needed to capture the complex, multi-level and transient status of constructs in the real-world as the study of implementation evolves.
The interviews to gather perceptions of AGTO were done with a convenience sample, which could have introduced bias into those findings. This was in part a result of diminishing budgets for prevention in Maine, which caused a sizable dropout of programs between baseline and post (although equivalent across groups). A sample of 32 programs with complete program performance ratings is modest. Therefore the percent change analyses should be interpreted with caution. Future studies that recruited even larger numbers of schools, communities, or coalitions would provide the opportunity to use inferential statistics to better test the relationships between TA and various prevention tasks. Further, TA hours, even by domain, is not the only important measure of TA; quality of the TA, timing about when it was given (e.g., planning before evaluation), and type of program are other factors to be explored in future research [62, 63]. Also, because the use versus non-use groups were not randomly assigned, there may have been selection biases within use versus non-use analyses-e.g., practitioners who engaged in the intervention may have been more likely to improve than those who did not. However, this bias may be mitigated somewhat because the capacity scores with significant group (user versus non-user) X time (pre-mid-post) interactions were not significantly different at baseline among the use versus non-use groups. It is important to note, capacity-building interventions like AGTO are voluntary and thus this analysis represents a real-world test.
The types of programs and practitioners AGTO targeted in this study are typical of prevention efforts in many communities. Despite challenges, AGTO helped programs that made use of it make critical improvements in the quality of their work. Implementation support interventions like AGTO can help ensure investments in real world prevention programming are conducted with high quality and consequently yield positive outcomes.
aGetting To Outcomes and GTO are trademarks registered by University of South Carolina and RAND.
bUS Census 2000.
cAccording to the 2006 Maine Youth Drug and Alcohol Use Survey (http://www.maine.gov/dhhs/osa/data/mydaus/mydaus2006.htm).
dNine programs ended between the time the study was planned and begun, leaving only 51 programs at baseline. 19 more programs ended between baseline and post.
Joie Acosta is a Behavioral Scientist at the RAND Corporation and specializes in research and evaluation to improve the implementation of evidence-based positive youth development, substance abuse and mental health prevention and treatment programs. From this work, she co-authored reports on building capacity for outcomes measurement among human service organizations and improving the implementation of social services and behavioral health care.
Matthew Chinman is a Senior Behavioral Scientist at the RAND Corporation, whose recent focus has been to develop strategies to enhance the capacity of community-based practitioners. He co-developed Getting To Outcomes® (GTO), a model designed to help community organizations to better plan, implement, and self-evaluate programs across a number of domains. He has led the successful testing of the several Getting To Outcomes-based guides in the areas of substance abuse prevention, underage drinking prevention, and youth development.
Patricia Ebener is a senior researcher with RAND's Drug Policy Research Center and a senior survey director with RAND's Survey Research Group. She has collaborated on community based participatory research and quality of prevention and treatment services projects for over twenty years.
Patrick Malone, a quantitative and social psychologist, is an Associate Professor of Psychology at the University of South Carolina. He specializes in methods for latent variable modeling and longitudinal analysis for adolescent health risk behaviors.
Susan Paddock is a Senior Statistician at the RAND Corporation. Dr. Paddock’s research interests include statistical methods development and statistical applications to substance abuse treatment studies.
Andrea Phillips is a Project Associate at the RAND Corporation.
Peter C. Scales is Senior Fellow at Search Institute. For nearly four decades, he has been widely recognized nationally, and more recently, internationally, for his research, direct service work, and comprehensive writings on promoting positive child and youth development. He is the world's leading researcher on the Developmental Assets approach to positive development, has led more than 15 large-scale or nationally representative studies, and international studies in a dozen countries on five continents, and authored or co-authored more than 250 publications, including 10 books, and 75 peer-reviewed journal articles and book chapters.
Mary Ellen Slaughter is a statistical/quantitative analyst at RAND with statistical expertise in regression analysis, categorical data analysis, survival analysis, and multivariate analysis. She also has experience with cost-effectiveness analysis, large health care databases, geographic information systems, and is proficient in SAS and R statistical programming.
Assets getting to outcomes
Comprehensive framework for implementation research
Getting to outcomes
Prevalence-adjusted bias-adjusted kappa
Miller T, Hendrie D: Substance abuse prevention dollars and cents: A cost-benefit analysis (DHHS Pub. No. (SMA) 07–4298). 2009, Rockville, MD: Center for Substance Abuse Prevention, Substance Abuse and Mental Health Services Administration
Backer TE: Finding the balance: program fidelity and adaptation in substance abuse prevention: CSAP. 2001
Ennett ST, Ringwalt CL, Thorne J, Rohrbach LA, Vincus A, Simons-Rudolph A: A comparison of current practice in school-based substance abuse prevention programs with meta-analysis findings. Prev Sci. 2003, 4: 1-14. 10.1023/A:1021777109369.
Cantor D, Crosse S, Hagen CA, Mason MJ, Siler AJ, Von Glatz A: Wide Scope, Questionable Quality: Drug and Violence Prevention Efforts in American Schools. Report on the Study on School Violence and Prevention. In. 2001, Washington, DC: U.S. Department of Education, Planning and Evaluation Service
Crosse S, Burr M, Cantor D, Hagen CA, Hantman I: Wide Scope, Questionable Quality: Drug and Violence Prevention Efforts in American Schools. Report on the Study on School Violence and Prevention. In. 2001, Washington, DC: U.S. Department of Education, Planning and Evaluation Service
Reuter P, Timpane JM: Assessing options for the safe and drug free schools and communities act. 2001, Santa Monica, CA: RAND, No. MR-1328
Silvia ES, Thorne J: School-based drug prevention programs: A longitudinal study in selected school districts. In. 1997, North Carolina: Research Triangle Institute
U.S. Department of Education OoP, Evaluation and Policy Development, Policy and Program Studies Service: Prevalence and Implementation Fidelity of Research-Based Prevention Programs in Public Schools: Final Report. 2011, Washington, D.C:
Wandersman A, Florin P: Community interventions and effective prevention: bringing researchers/evaluators, funders and practitioners together for accountability. Am Psychol. 2003, 58 (6/7): 441-448.
Green L: From research to 'Best Practices' in other settings and populations. Am J Health Behav. 2001, 25 (3): 165-178. 10.5993/AJHB.25.3.2.
Livet M, Wandersman A: Organizational functioning: facilitating effective interventions and increasing the odds of programming success. Empowerment evaluation in practice. Edited by: Fetterman DM, Wandersman A. 2005, New York, NY: Guilford, 123-154.
Chinman M, Hunter S, Ebener P, Paddock S, Stillman L, Imm P, Wandersman A: The Getting To Outcomes demonstration and evaluation: an illustration of the prevention support system. Am J Community Psychol. 2008, 41: 206-224. 10.1007/s10464-008-9163-2.
Chinman M, Tremain B, Imm P, Wandersman A: Strengthening prevention performance using technology: a formative evaluation of interactive Getting To OutcomesTM. Am J Orthopsychiatry. 2009, 79: 469-481.
Fishbein M, Ajzen A: Beliefs, attitudes, intentions, and behavior: An introduction to theory and research. 1975, Reading, MA: Addison-Wesley
Ajzen I, Fishbein M: Attitude-behavior relations: a theoretical analysis and review of empirical research. Psychol Bull. 1977, 84: 888-918.
Bandura A: Health promotion by social cognitive means. Health Educ Behav. 2004, 31 (2): 143-164. 10.1177/1090198104263660.
Fishbein M, Ajzen I: Atitudes toward objects as predictive of single and multiple behavioral criteria. Psychol Rev. 1974, 81: 59-74.
Chinman M, Acosta J, Ebener P, Burkhart Q, Clifford M, Corsello M, Duffy T, Hunter S, Jones M, Lahti M: Establishing and evaluating the key functions of an interactive systems framework based on Assets-Getting To Outcomes. Am J Commun Psychol. 2012,http://rd.springer.com/article/10.1007%2Fs10464-012-9504-z,
Benson PL, Scales PC, Leffert N, Roehlkepartain ER: A fragile foundation: The state of developmental assets among American youth. 1999, Minneapolis, MN: Search Institute
Leffert N, Benson PL, Scales PC, Sharma AR, Drake DR, Blyth DA: Developmental assets: measurement and prediction of risk behaviors among adolescents. Appl Dev Sci. 1998, 2: 209-230. 10.1207/s1532480xads0204_4.
Scales PC, Benson PL, Leffert N, Blyth DA: Contribution of developmental assets to the prediction of thriving among adolescents. Appl Dev Sci. 2000, 4: 27-46. 10.1207/S1532480XADS0401_3.
Fisher D, Imm PS, Chinman M, Wandersman A: Getting To Outcomes with Developmental Assets: Ten steps to measuring success in youth programs and communities. 2006, Minneapolis, MN: Search Institute
Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC: Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci. 2009, 4: 50-65. 10.1186/1748-5908-4-50.
Rohrbach LA, D’Onofrio CN, Backer TE, Montgomery SB: Diffusion of school-based substance abuse prevention programs. Am Behav Sci. 1996, 39: 919-934. 10.1177/0002764296039007012.
Rohrbach LA, Graham JW, Hansen WB: Diffusion of a school-based substance abuse prevention program: predictors of program implementation. Prev Med. 1993, 22 (2): 237-260. 10.1006/pmed.1993.1020.
McCormick LK, Steckler AB, McLeroy KR: Diffusion of innovations in schools: a study of adoption and implementation of school-based tobacco prevention curricula. Am J Health Promot. 1995, 9: 210-219. 10.4278/0890-1171-9.3.210.
Steckler AB, McLeroy KR, Goodman RM, McCormick L, Bird ST: Toward integrating qualitative and quantitative methods: an introduction. Health Educ Q. 1992, 19: 1-8. 10.1177/109019819201900101.
Parcel GS, Erikson MP, Lovato CY, Gottlieb NH, Brink SG, Green LW: The diffusion of school-based tobacco-use prevention program: project description and baseline data. Health Educ Res. 1989, 4: 111-124. 10.1093/her/4.1.111.
Cividin TM, Ottoson JM: Linking reasons for continuing professional education participation with post-program application. J Contin Educ Health Professions. 1997, 17: 46-55. 10.1002/chp.4750170107.
Ottoson JM: After the applause: exploring multiple influences on application following an adult education program. Adult Educ Quart. 1997, 47: 92-107. 10.1177/074171369704700203.
Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC: Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci. 2009, 4: 1-10.1186/1748-5908-4-1.
Murray DM: Design and analysis of group-randomized trials. 1998, Oxford, UK: Press OU
Battelle: Annual findings report: Drug-free communities support program national evaluation. 2008, Washington, DC: Executive Office of the President Office of Administration Office of National Drug Control Policy
Butterfoss FD, Goodman RM, Wandersman A: Community coalitions for prevention and health promotion. Health Educ Res. 1993, 8: 315-330. 10.1093/her/8.3.315.
Ellickson PL, Bell RM: Drug prevention in junior high: a multi-site longitudinal test. Science. 1990, 247: 1265-1372.
Botvin GJ, Baker E, Dusenbury L, Botvin EM, Diaz T: Long-term follow-up results of a randomized drug abuse prevention trial in a white middle-class population. JAMA. 1995, 273 (14): 1106-1112. 10.1001/jama.1995.03520380042033.
Jemmott JB, Jemmott L, Spears H, Hewitt N, Cruz-Collins M: Self-efficacy, hedonistic expectancies, and condom-use intentions among inner-city Black adolescent women: a social cognitive approach to AIDS risk behavior. J Adolescent Health. 1992, 13: 512-519. 10.1016/1054-139X(92)90016-5.
Jemmott JB, Jemmott LS, Fong GT: Reductions in HIV risk-associated sexual behaviors among Black male adolescents: effects of an AIDS prevention intervention. Am J Public Health. 1992, 82: 372-377. 10.2105/AJPH.82.3.372.
Jemmott JB, Jemmott LS, Fong GT, McCaffree K: Reducing HIV risk-associated sexual behavior among African American adolescents: Testing the generality of intervention effects. Am J Community Psychol. 1999, 27: 161-187. 10.1007/BF02503158.
Chinman M, Hannah G, Wandersman A, Ebener P, Hunter S, Imm P, Sheldon J: Developing a community science research agenda for building community capacity for effective preventive interventions. Am J Community Psychol. 2005, 3/4: 143-157.
Mitchell R, Stone-Wiggins B, Stevenson JF, Florin P: Cultivating capacity: outcomes of a statewide support system for prevention coalitions. J Prev Interv Community. 2004, 27: 67-87. 10.1300/J005v27n02_05.
O’Donnell LO, Scattergood P, Adler M, San Doval A, Barker M, Kelly JA: The role of technical assistance in the replication of effective HIV interventions. AIDS Educ Prev. 2000, 12: 99-111.
Stevenson JF, Florin P, Mills DS, Andrade M: Building evaluation capacity in human service organizations: a case study. Eval Program Plann. 2002, 25 (3): 233-243. 10.1016/S0149-7189(02)00018-6.
Kitson A, Harvey G, McCormack B: Enabling the implementation of evidence based practice: a conceptual framework. Qual Health Care. 1998, 7: 149-158. 10.1136/qshc.7.3.149.
Rycroft-Malone J, Harvey G, Seers K, Kitson A, McCormack B, Titchen A: An exploration of the factors that influence the implementation of evidence into practice. J Clin Nurs. 2002, 13: 913-924.
Stetler CB, Legro MW, Rycroft-Malone J, Bowman C, Curran G, Guihan M: Role of 'external facilitation' in implementation of research findings: a qualitative evaluation of facilitation experiences in the Veterans Health Administration. Implement Sci. 2006, 23:
Weiss E, Miller-Anderson R, Lasker R: Making the most of collaboration: exploring the relationship between partnership synergy and partnership functioning. Health Educ Behav. 2002, 29: 683-698. 10.1177/109019802237938.
Lehman WEK, Greener JM, Simpson DD: Assessing organizational readiness for change. J Subst Abuse Treat. 2002, 22 (4): 197-209. 10.1016/S0740-5472(02)00233-7.
Hall GE, Hord SM: Implementing change. 2001, Boston, MA: Allyn and Bacon
Hall GE, Loucks SF, Rutherford WL, Newlove BW: Levels of use of the innovation: a framework for analyzing innovation adoption. J Teach Educ. 1975, 26: 52-10.1177/002248717502600114.
Hall GE, Loucks SF: A developmental model for determining whether the treatment is actually implemented. Am Educ Res J. 1977, 14 (3): 263-276. 10.3102/00028312014003263.
Chinman M, Acosta J, Ebener P, Burkhart Q, Clifford M, Corsello M, Duffy T, Hunter S, Jones M, Lahti M: Intervening with practitioners to improve the quality of prevention: One year findings from a randomized trial of Assets-Getting To Outcomes. J Prim Prev. In press
Chinman M, Tremain B, Imm P, Wandersman A: Strengthening prevention performance using technology: a formative evaluation of interactive Getting To Outcomes. Am J Orthopsychiatry. 2009, 79: 469-481. PMC2859836
Bryt T, Bishop J, Carlin JB: Bias, prevalence and kappa. J Clin Epidemiol. 1993, 46 (5): 423-429. 10.1016/0895-4356(93)90018-V.
Lincoln YS, Guba EG: Naturalistic inquiry. In. 1985, Beverly Hills, CA: Sage
Graham ID, Logan J: Innovations in knowledge transfer and continuity of care. Can J Nurs Res. 2004, 36 (2): 89-103.
Hunter SB, Chinman M, Ebener P, Imm P, Wandersman A, Ryan G: Technical assistance as a prevention capacity-building tool: a demonstration using the Getting To Outcomes framework. Health Educ Behav. 2009, 36: 810-828. 10.1177/1090198108329999.
Keener DC: Toward a science of capacity building: An examination of technical assistance following a training program for prevention professionals. In. 2007, Columbia, SC, USA: University of South Carolina
Feinberg ME, Ridenour TA, Greenberg MT: The longitudinal effect of technical assistance dosage on the functioning of communities that care prevention boards in Pennsylvania. J Prim Prev. 2008, 29 (2): 145-165. 10.1007/s10935-008-0130-3.
Hawkins JD, Oesterle S, Brown EC, Arthur MW, Abbott RD, Fagan AA, Catalano RF: Results of a type 2 translational research trial to prevent adolescent drug use and delinquency: a test of communities that care. Arch Pediatr Adolesc Med. 2009 September, 163 (9): 789-798. 10.1001/archpediatrics.2009.141.
Spoth R, Redmond C, Shin C, Greenberg M, Clair S, Feinberg M: Substance-use outcomes at 18 months past baseline: the PROSPER Community-University partnership trial. Am J Prev Med. 2007 May, 32 (5): 395-402. 10.1016/j.amepre.2007.01.014.
Wandersman A: Four keys to success (theory, implementation, evaluation, and resource/system support): high hopes and challenges in participation. Am J Community Psychol. 2009, 43: 3-21. 10.1007/s10464-008-9212-x.
Wandersman A, Chien VH, Katz J: Toward an evidence-based system for innovation support for implementing innovations with quality: tools, training, technical assistance, and quality assurance/quality improvement. Cultivating communities of practice. Edited by: Wenger E, McDermott R, Snyder WM. 2002, Boston, MA: Harvard Business School Press, In press
Enhancing Prevention Capacity with Developmental Assets and Getting to Outcomes, 1R01DA023277-01A2, National Institute on Drug Abuse (NIH).
The authors declare that they have no competing interests.
AP carried out the qualitative interviews, qualitative analysis of interviews and drafted the manuscript. MS, SP and PM designed and performed the statistical analyses. PS participated in the drafting of the manuscript. JA, MC, and PE conceived of the study, and participated in its design and coordination and helped to draft the manuscript. All authors read and approved the final manuscript.
Electronic supplementary material
About this article
Cite this article
Acosta, J., Chinman, M., Ebener, P. et al. An intervention to improve program implementation: findings from a two-year cluster randomized trial of Assets-Getting To Outcomes. Implementation Sci 8, 87 (2013). https://doi.org/10.1186/1748-5908-8-87
- Implementation support
- Technical assistance
- Capacity building