We’re sorry, something doesn't seem to be working properly.
Please try refreshing the page. If that doesn't work, please contact support so we can address the problem.
Training scholars in dissemination and implementation research for cancer prevention and control: a mentored approach
Implementation Sciencevolume 13, Article number: 18 (2018)
As the field of D&I (dissemination and implementation) science grows to meet the need for more effective and timely applications of research findings in routine practice, the demand for formalized training programs has increased concurrently. The Mentored Training for Dissemination and Implementation Research in Cancer (MT-DIRC) Program aims to build capacity in the cancer control D&I research workforce, especially among early career researchers. This paper outlines the various components of the program and reports results of systematic evaluations to ascertain its effectiveness.
Essential features of the program include selection of early career fellows or more experienced investigators with a focus relevant to cancer control transitioning to a D&I research focus, a 5-day intensive training institute, ongoing peer and senior mentoring, mentored planning and work on a D&I research proposal or project, limited pilot funding, and training and ongoing improvement activities for mentors. The core faculty and staff members of the MT-DIRC program gathered baseline and ongoing evaluation data regarding D&I skill acquisition and mentoring competency through participant surveys and analyzed it by iterative collective reflection.
A majority (79%) of fellows are female, assistant professors (55%); 59% are in allied health disciplines, and 48% focus on cancer prevention research. Forty-three D&I research competencies were assessed; all improved from baseline to 6 and 18 months. These effects were apparent across beginner, intermediate, and advanced initial D&I competency levels and across the competency domains. Mentoring competency was rated very highly by the fellows––higher than rated by the mentors themselves. The importance of different mentoring activities, as rated by the fellows, was generally congruent with their satisfaction with the activities, with the exception of relatively greater satisfaction with the degree of emotional support and relatively lower satisfaction for skill building and opportunity initially.
These first years of MT-DIRC demonstrated the program’s ability to attract, engage, and improve fellows’ competencies and skills and implement a multicomponent mentoring program that was well received. This account of the program can serve as a basis for potential replication and evolution of this model in training future D&I science researchers.
Awareness of the need for scientific approaches to dissemination and implementation (D&I), to provide for more effective and timely ways to translate research in practical applications, has increased over the past two decades [1, 2]. With much of the cancer burden being preventable , the need to close the estimated 17-year gap between publication of new research findings and their application to prevention in practice has become urgent [4,5,6]. This increased awareness, and the development of D&I science has initiated programs training researchers in this domain [1, 7,8,9]. Funding agencies are designating funds for D&I specific proposals [10, 11], and there are increasing numbers of faculty positions focusing on D&I research ; however, the availability of training programs has lagged behind the demand from a growing D&I research workforce [13, 14]. Moreover, beyond efforts to identify requisite competencies, curricula, and desired outcomes of trainings, competencies must be put into action and tested [4,5,6, 15, 16].
The Mentored Training for Dissemination and Implementation Research in Cancer (MT-DIRC) program is perhaps the first to incorporate systematic mentored training for D&I research with a focus on cancer control. It builds on experiences with D&I science training programs of varying depth and formats offered by several organizations in the USA and globally [1, 17], with a particular focus on providing extended mentoring. Programs specific to mentoring in D&I science include the Implementation Research Institute (IRI)  sponsored by the National Institute of Mental Health (NIMH), and the Training Institute for Dissemination and Implementation Research in Health (TIDIRH), sponsored by the National Institutes of Health [7, 18]. IRI, which pioneered the approach of providing one-on-one 2-year mentoring, focuses on mental health researchers. The IRI recently added a component of training fellows to become mentors. TIDIRH, initially started as a 5-day training institute , has evolved into an online-based program with a 2-day in-person session. Both IRI and TIDIRH have completed 7 years of summer trainings. The TIDIRH program has an informal mentoring component, but there is no established expectation for those mentoring relationships to last beyond the few in-person sessions. Outside the USA, Knowledge Translation Canada (or more informally known as KT Canada) has also offered summer training institutes that rotate cities and hosts throughout the years [14, 19], and the Cochrane Collaboration, in conjunction with Public Health Insight at the University of Melbourne, offers a 1-day Knowledge Translation Training workshops . Overall, relatively few of these programs provide significant opportunities for interaction or offer a sustained mentoring component .
Several universities offer ongoing support and training through their respective Clinical and Translational Science Award Programs [14, 21] as well as concentrations in implementation science as a component of their masters, doctoral, or post-doctoral programs [14, 16]. Online webinars and resources are also provided through institutions such as National Cancer Institute’s Division of Cancer Control and Population Science , the Veterans Affairs’ Quality Enhancement Research Initiative & Health Services Research & Development group through the Center for Implementation Practice and Research Support . The variety in formats can be helpful to provide access for researchers who otherwise would not have the time or resources to be trained.
Various groups have undertaken work to develop competency and curriculum lists and expectations [12,13,14,15,16]. A few programs have conducted evaluations of their trainings pertaining to knowledge acquisition [13, 24], although few have assessed skill gains of their trainees over a longer period of time. An evaluation of the IRI using social network analysis found that mentoring was significantly related to outcomes, 2 years later, of collaborations focused on new research, grant submission, and scholarly publications . The Implementation Science Editors’ recently issued call to action to build capacity for researchers to be able to conduct D&I research . In this context, we set out to explore the effectiveness of the MT-DIRC D&I training program.
Purpose of the MT-DIRC training program
Funded by the National Cancer Institute, MT-DIRC is an R25 grant-funded, post-doctoral education program, aimed at building capacity among early to mid-career, cancer control researchers in D&I science through supplemental training and mentorship in D&I research. During each year of the 2-year MT-DIRC program, fellows attended a 5-day summer institute at Washington University in St. Louis to receive didactic, group, and individual instruction on their research area of interest as it pertains to D&I science. Ongoing mentoring relationships occurred over the 2 years of the program, complemented by intermittent webinar sessions with topics chosen by the fellows themselves. This paper describes the various components of the MT-DIRC program and shares preliminary results from the first two trainee cohorts. The purpose of this paper is to describe the development of the MT-DIRC program and share the experience to date to inform the development of other D&I training programs with the aim of advancing the field of D&I research within cancer control and more broadly.
The program was led by Program Director, Ross Brownson, a D&I scholar who has been a contributor to other similar D&I training programs (e.g., IRI and TIDIRH). Dr. Brownson was joined by other renowned D&I scholars (Table 1) who together have a combined 340-year experience with grant application writing, reviewing, and conducting D&I research  and have been central in setting the research agenda and priority for D&I science in the USA and Canada. This core faculty’s expertise covered all areas of the cancer control continuum as well as the various areas of D&I science. The core faculty members contributed to all areas of the program such as mentoring fellows throughout the year, shaping the agenda for the summer institute, and presenting at least one session. Core faculty are designated as co-investigators or consultants on this project and are compensated for their time accordingly. Three to four guest faculty members are invited to lecture on topics which enhance the breadth of expertise already available through the team of faculty. The guest faculty attended the summer institute anywhere from 1 day to the entire week depending on their schedules and provided fellows with special consultation in their particular subject area. The guest faculty members were given an honorarium for their time at the institute.
The MT-DIRC program sought a range of fellows from across the USA and internationally to apply for the program. In order to be considered for the fellowship, applicants must have completed a doctoral-level degree and have a full-time appointment in a research setting prior to applying. All researchers whose work focused along the cancer control continuum were welcome to apply . The program focused on training early-career researchers; however, mid-to-late career researchers who were looking to shift their work to focus on D&I science were also encouraged to apply. Applicants were required to fill out an informational cover page, submit a two-page concept paper outlining a D&I pilot study, include an NIH-formatted biosketch, and two letters of references.
To recruit fellows, emails were sent out through various listservs and networks related to D&I science and cancer control. Flyers were made available at the annual Conference on the Science of Dissemination and Implementation each December in Washington D.C. Faculty and fellows were also encouraged to mention the program when giving presentations on their D&I work at related conferences.
In each year, every core faculty member reviewed a subset of the submitted applications, with each application reviewed by three faculty members. Faculty scored each application based on eight questions regarding overall quality, demonstrated commitment to D&I science, demonstration of experience working in trans-disciplinary networks, evidence of research support and potential, likelihood for career development, appropriate methods in concept paper, appropriate topic in concept paper, and potential impact of the work proposed. The scores were compiled and averaged with additional commentary included. All core faculty met by conference call to review scores and select fellows for that year’s cohort.
Each year of the grant, MT-DIRC sponsored 12 fellows covering all costs for travel to St. Louis, MO, during the 5-day summer institute, hotel accommodations, and meals for the week as well as an additional $1000 for pilot funds to be used during the first year of the program. An additional collaboration with the Veterans Administration provided funding for two additional fellows who are VA-affiliated researchers. The Cancer Research Network also sponsored one additional fellow in the 2014 and 2015 cohorts.
Figure 1 provides a visual depiction of the various components of the MT-DIRC program and their timing during the 2-year enrollment, including the points at which the baseline, 6- and 18-month D&I and Mentoring Competency Assessments were administered. A description of each component follows.
Competencies for MT-DIRC program curriculum
The foundation for the MT-DIRC program curriculum lies within a set of comprehensive D&I competencies. Building on prior work to define competencies for D&I science training [16, 19, 27], MT-DIRC faculty and staff engaged in a further effort to develop a comprehensive set of competencies that could be adapted to various types of curricula. This was done before the first summer institute in 2014. As described in detail elsewhere , a card-sorting activity was utilized and a list of 43 unique competencies was developed, sorted into learning levels (beginner, intermediate, and advanced), and used to guide the content that would be covered during the week-long summer institute.
To identify any remaining gaps within this curriculum, the MT-DIRC team conducted a concept mapping activity . Concept mapping looked at competency development not only from the perspective of the researchers but also the practitioners, who are end users of the research. Nine clusters of competency themes were identified and mapped to the previous list of competencies to see where our previous competencies to assess alignment. See Tabak et al.  for further details.
Starting in 2014, 5-day MT-DIRC summer institutes have been held each year at Washington University in St. Louis. The institutes consisted of didactic sessions, one-on-one mentoring, and small group breakout sessions. The focus of the week was to address each of the identified competencies with the goals of improving D&I research skills and provide hands-on support to each fellow’s project development. Each day had a specific focus (e.g., aims, study design, and measurement) for project development. First-year fellows were asked to bring an updated version of the two-page D&I concept paper they had submitted in their application to the program. Returning fellows were asked to bring one or more “works in progress” to focus on during the week, such as in-progress manuscripts, grants in-development, and data sets. During breakout sessions, fellows took turns presenting their work and receiving feedback from mentors and peers. Breakout sessions were also used to allow work with their assigned MT-DIRC mentors, see Additional file 1 for a sample agenda of the summer institute.
All MT-DIRC fellows were matched with a core faculty mentor who was committed to working with 2–4 fellows across a 2-year time frame. Fellows were matched to their assigned mentor about 1 month prior to their arrival at the summer institute. Mentoring matches were done by the Program Coordinator with input from the faculty on preference, based on shared research interests or experiences.
In preparation, all faculty who served as mentors engaged in mentor training which was offered in three-stages by Dr. Christine Pfund (CP), Mentor Consultant, from the University of Wisconsin-Madison. The training included an asynchronous, self-paced component , a synchronous online component [30,31,32,33], and a face-to-face training at the summer institute. All mentors were required to participate in all three parts of the training. Table 2 provides an outline of the components of the mentor training as well as description of each.
The Mentor Consultant provided continual support on the mentoring relationships throughout the duration of the program. Mentors were expected to have at least an hour of monthly contact with each of their assigned fellows (group mentor sessions could count toward that 1-hour requirement). If this expectation was not met, fellows or mentors were encouraged to contact the Program Coordinator and Principal Investigator so the situation could be addressed appropriately. On monthly calls with the core faculty, the group was invited to share facilitators or barriers to their current mentoring relationships. The core faculty tried to troubleshoot current issues as they arose as well as to provide suggestions on how to better facilitate these relationships.
Training of fellows
On the first day of the MT-DIRC institute, the Mentor Consultant presented a 1-hour session on “Evidence-Informed Mentoring for D&I Research” which included presentations of the key elements of research mentor training, background on the mentoring competencies, and overall expectations for the mentoring relationships, which are a critical element of the program. During this session, fellows were given time to identify their expectations for their relationships with MT-DIRC program mentors. Mentors and fellows were then given an opportunity to meet for the first time and lay out expectations for the week and beyond. Mentors and fellows had face-to face contact every day during the institute, and on the final day were given the opportunity to plan how the mentoring relationship would be handled during the monthly calls and the 2 years of the fellowship.
Ongoing interactions with fellows through calls and webinars
Since most of the mentoring interaction between the fellows and faculty mentors was conducted at a distance, the team brainstormed ways for the entire cohort to connect between their attendance at the two summer institutes. While there were informal gatherings at the Annual Conference on the Science of Dissemination and Implementation in Health  held in each December in Washington D.C., fellows indicated early on in the program a preference for continuing education throughout the year. The team at first addressed this with quarterly cohort calls for the fellows. Beginning in 2015, quarterly webinars were instituted. These webinars were generally led by one or two faculty members on a topic of interest to the group. Webinar attendance was limited to MT-DIRC fellows past and present. Webinars were 1 h in length and consisted of a formal presentation portion with time for questions and answers at the end. As of January 2018, the program had hosted seven webinars on a variety of topics that included how to craft an aims page, demystifying the grant review process, funding mechanisms, career awards, and mixed methods evaluation and building D&I research programs and choosing models and frameworks. All webinars are archived and available to all fellows and faculty. These webinars complemented other ongoing distance education training activities such as those offered by the NCI and the VA [35, 36].
To assist fellows in securing grants, the program offered $1000 in pilot funds to each fellow during their first year in the program. These funds were intended to support pilot research with the goal of helping fellows obtain preliminary results which could be leveraged for larger D&I grant applications. Fellows also used these funds to attend and present at various conferences, attend relevant meetings pertaining to D&I science, and support the effort of research assistants or a data analyst, or for participant incentives in pilot work.
To assess the on-going mentoring relationship, D&I skill level, as well as the general satisfaction with the institute and the MT-DIRC program overall, a variety of surveys were conducted throughout the year via the Qualtrics online survey platform. In addition to the various surveys, fellows’ academic outputs (grant applications submitted and awarded, publications, and presentations) were tracked over the years. The project coordinator utilized these outputs for annual NCI training grant reporting purposes and the long-term assessment of fellows’ productivity.
Pre-institute skills survey
A pre-institute survey was administered to fellows 1 month before they attended their first summer institute. This 20-min survey was used to collect pertinent demographic information, intended for grant reporting purposes as well as some logistical questions in preparation of the training institute. The survey was also used to collect baseline data on each fellow’s level of D&I skills according to the 43 competencies (a full list of competencies is included in Additional file 1). Fellows were also asked to indicate their general mentoring needs. This survey was administered only once at the beginning of each fellow’s time in the program.
Post-institute satisfaction survey
At the completion of the 5-day institute, fellows were immediately sent an online survey asking them to rate the various components of the institute. Fellows rated their satisfaction with information communicated to them before the institute, travel and accommodation arrangements, quality of individual faculty presentations, structure of the institute’s agenda, and any additional feedback regarding the institute.
Six and 18-month post-institute skills and mentoring survey
Longitudinal data were collected via survey from fellows at 6 and 18 months after attending their first institute. Fellows were asked to reassess their D&I skill levels for the same 43 competencies that were assessed in the pre-institute survey. Consistent with other literature on skill assessments, we describe these outcomes as skills, although there are some overlaps with self-efficacy [37,38,39]. In addition, theoretical concepts around self-efficacy posit that researchers can have the same knowledge or skills rating but differ in their self-efficacy, thus differ in their implementation of that knowledge and those skills [37, 38]. The 6- and 18-month post-institute surveys also assessed fellows’ mentoring relationships. Fellows were asked to rate the skills of their mentors utilizing the Mentoring Competency Assessment (MCA). The MCA is a validated  26-item skill inventory evaluating six areas of mentoring competencies: maintaining effective communication, aligning expectations, assessing understanding, addressing diversity, fostering independence, and promoting professional development .
To gain additional evaluation of the program’s performance, fellows were asked to rank their mentoring priorities and their satisfaction with those priorities being met. They were asked to rate (on a five-point Likert scale) the importance of eight mentoring priority areas (skill building, sharing resources and infrastructure, performance feedback, providing opportunity, career planning, professional networking, professional socialization, providing emotional support) and their level of satisfaction with each of the aforementioned priorities being met.
As of this writing, all four cohorts had completed their pre-institute survey; three cohorts had completed the 6-month post-institute survey, and two had completed the 18-month post-institute evaluation. We are reporting these preliminary data at this time to make them accessible to others attempting to replicate a similar program.
Using descriptive statistics, we examined the demographic composition of the four cohorts. D&I skills and mentoring outcomes were analyzed for the two cohorts (2014 and 2015 cohorts) with complete data (pre-institute, 6- and 18-month post-institute surveys completed). D&I skill gains were tested using a Repeated Measures ANOVA between the three time points (pre-institute, 6- and 18-month post-institute) with Greehouse-Geisser’s correction utilized where assumption of sphericity was violated . Cohen’s D between the 18-month post-institute mean and the pre-institute mean was used to determine overall effect size. Cohen suggests the following effect ranges: small 0.2, medium 0.4, and large 0.8 . In keeping with the work that was done with the card-sorting to develop these competencies , a summary variable was created to look at the competency scores by each domain and each competency skill level. For mentoring competencies, independent t tests were used to compare mentors’ self-ratings versus fellows’ ratings of mentors’ competencies across the MCA. The analysis looked at the fellows and mentor’s initial 6-month assessment of mentoring skills. Additionally, differences in mean ratings of importance and satisfaction among the eight mentoring priority areas were explored. These measurements were taken at fellows’ 6- and 18-month post-institute survey.
From 2014 to 2017, 56 fellows participated in the MT-DIRC program. Participants were recruited nationally with the most successful recruitment tactic being word of mouth from colleagues in a previous cohort. Demographics for each of the four participating cohorts are in Table 3. A majority (79%) of the fellows accepted into the program were female, focused their research on Cancer Prevention (48%) and were Assistant Professors (55%). A majority (59%) came from an allied health discipline. Fellows represented all geographic regions of the USA, two fellows were from Canada, and three were from Australia. An additional three fellows indicated “other” as they were foreign-nationals currently employed in the USA. All fellows attended the institute and on average, 15–20 fellows from all of the cohorts attended each webinar.
A primary goal of the MT-DIRC program is to increase the skills of D&I research across a range of competencies identified by this program. Results indicate that fellows have had a statistically significant improvement across all 43 targeted competencies and that these gains are sustained at the 18-month time point. Mean skills gain score at the baseline, 6- and 18-month time points, grouped by competency domains are presented in Table 4. The skill that saw the largest change between pre- and 18 months was “Identify common D&I measures and analytic strategies for your research questions” (MD = − 1.69, Cohen’s D = 2.151). The skill that saw the least change between pre- and 18 months was “Identify sites to participate in D&I studied and negotiate or proved incentives to secure their involvement” (MD = − 0.92, D = 0.88). There were improvements in all summary domains. The domain that saw the largest increase in skills between pre- and 18 months was “Definitions, Background and Rationale” (MD = − 1.22, D = 1.97). The domain that saw the least increase in skill between pre- and 18 months was “Design and analysis” (MD = − 1.27, D = 1.37).
Changes in skill level across the competency domains were also analyzed using the skill levels that were predetermined in the card sort activity . Among the three competency categories (beginner, intermediate, advanced), the largest increase in skill was shown for beginner skills (at 18 months, MD = − 1.26, D = 2.10) (Table 5).
Mentoring is a critical element of the MT-DIRC program, and therefore, the mentoring skills of the core faculty as well as the mentoring relationships were evaluated on several levels. Table 6 shows comparisons at 6 months between the ratings the 2014, 2015, and 2016 fellows gave their mentors and those the mentors gave themselves during the same period across the 26 items of the MCA . Fellows and mentors have similar rating in six of the identified 26 competencies having shown no significant difference between the mean scores of mentors and fellows. The remaining 20 competences showed a significant difference in the way fellows rated their mentors and the way that mentors rated themselves.
Alignment of fellows’ priorities and their satisfaction of having those priorities met was assessed using five-point Likert scale about the mentoring priorities and satisfaction. This alignment was assessed at both the 6- and 18-month marks to explore how importance and satisfaction changed over time. As shown in Fig. 2, while there was general alignment between the fellows rating of importance for a given mentoring domain (priority) and their satisfaction with that needs being met; priorities and satisfaction did shift over their time in the program. Skill building was consistently seen as very important (mean importance, μ = 4.50), but fellows initially indicated a mismatch in rating their satisfaction with this need being met lower than other domains (mean satisfaction, μ = 3.81). However, by the 18-month point, there was alignment in ranking of fellows’ importance and satisfaction place on skill building. By the 18-month point, most of the factors had improved their ratings in terms of satisfaction to fellows and were more closely aligned to the initial fellows’ perceived importance in that mentoring area. Across the board, mentors and fellows indicated that dedicated time for each other was their biggest barrier to these relationships.
Overall, early results from the MT-DIRC program show fellows are improving in their D&I skills and mentoring needs of the fellows are being met. Findings, discussed below, highlight the impact of the program and provide guidance for similar model programs. The emphasis on mentoring and skill development throughout the course of the program we believe has been integral to its success. However, the team has aimed to keep the program flexible to meet the needs of each cohort and the individual fellows and do not believe that all programs must contain every component in order to be successful. Rather, those interested in creating similar programs can utilize these results to inform their decision-making and help shape the content and format for their programs.
All D&I skills showed significant improvement from the time of entering into the program to the 6-month post-institute, with further improvements 18-month post-institute. While fellows showed improvement from their time in the program just within the first 6 months, there was continued improvement in skill levels over the subsequent 12 months suggesting that participants were using the skills they learned at the institute over time which led to continued growth in these areas.
Since the program was aimed at early-career researchers or mid-career researchers looking to switch their research focus into D&I science, it is not surprising that the largest area of improvement for skill levels were previously identified beginner skills (see Table 5 and Additional file 2). And since most of those beginner level skills were located within the “Definitions, Background, and Rationale” domain, we also saw that domain as having the largest overall skill increase. With the field of D&I research growing and its visibility expanding, more fellows are entering the program with a D&I conceptual framework to build upon when they enter the program and are able to master the beginner level concepts more quickly. These results may also suggest that the program does a reasonable job in training beginner level D&I skills, and more focus is needed on addressing advanced-level skills, whether through additional training throughout the year or perhaps even adding a training program in the future that focuses on advanced level D&I skills. However, the level of advancement of the D&I skills over the various time points also speaks well to the results of the original mapping of these competency levels when originally developed .
Our findings show that, in general, faculty mentors rated themselves lower on mentoring skills (MCA) than their fellows rated them, consistent with previous research . This finding may indicate that the standard mentors hold themselves to may be higher, in comparison to the expectations fellows have for a mentoring relationship in an academic research context. It may also show that the mentor training, while building mentor’s confidence in mentoring skills, also enabled mentors to more critically evaluate themselves . Fellows, only having received a 1-h training related to mentoring, were perhaps less inclined to evaluate their mentors to the same degree due the narrower scope of training received regarding mentorship competencies. This comparison of mentor self-assessment and fellow assessment allows for identification of future areas to focus on within our program, either by providing feedback for changes to the program structure or greater support for mentors.
While this is a formal mentoring program, mentoring contracts with fellows were not required (though they were encouraged and a template was provided) nor were meeting duration or structures stipulated. The only requirement was that mentors connected with each of their fellows at least once per month, for a recommended time of 1 h. Mentors were given the choice to hold phone calls or skype calls (very few could arrange face to face meetings) and whether they wanted to conduct group calls with all their fellows and/or individual calls. These elements of flexibility may suggest that perceived capabilities of the mentor may vary based on the expectations set out from the initial contact between the mentor and fellow pair. As noted in other research, codifying the mentoring relationship roles, in a way in which calls upon fellows to also be active participants in articulating their needs, may enhance the quality of the relationship and improve perceived success of the relationship .
The questions in the Mentoring Priorities and Satisfaction section of the survey showed mismatches among the specific priority areas of skill building and sharing resources (see Fig. 2 for a visual depiction). This mismatch reflects that fellows were highly satisfied with these priority areas but considered them less important than other priority areas. Another mismatch at 6 months was the high importance of “skill building” (μ = 4.50) coupled with the lower satisfaction among fellows (μ = 3.81). This alignment levels out by the 18-month mark, suggesting that some skills and priorities may take more time to develop and conceptualize than others. However, during the 18-month assessment of satisfaction, the mean score of each priority area shifted upward, indicating a general positive shift in all areas of the mentor priority areas. So, even if there was no exact alignment of mentoring areas between priority and satisfaction, overall, the fellows indicated higher levels of satisfaction with their mentoring toward the end of their time in the program.
Other research has found that fellows desire greater emphasis within the mentoring relationship on broader topics such as personal-professional life balance, whereas our findings show that fellows typically rate these psychosocial topics as less important while being highly satisfied with the support in these topics . In particular, “emotional support” is listed as the least important priority for fellows but is rated highly in terms of satisfaction. It may be that expectations in these areas are lower than that of others. Since the mentoring received here is of a formal nature and geared specifically toward D&I capacity-building in cancer research, it may be that the fellows are already receiving the psychosocial support they need from previously established mentoring relationships at their home institutions.
There are some limitations in the way that skill assessments were conducted. First, the assessments of fellows’ skills were self-rated. Fellows were asked about “How skilled do they feel” for a specific competency at that particular point in time. The project team has suggested administering an objective skill test to fellows at the end of their time in the program. However, there is currently no known validated, objective D&I research skill test.
The relationships between the fellows and their home mentors were also not explored in depth. All applicants to the program were asked to identify home mentors in their applications, but there was no formalized process to connect home mentors to the assigned MT-DIRC mentors. This would have been an interesting relationship to follow to see if it produced higher satisfaction with mentoring or increased D&I skills. There is also the issue of power dynamic when assessing mentor’s skills by the fellows. While individual mentoring data are never shared, fellows may be unlikely to rate their mentors too critically given that these mentors serve in this role outside their normal research duties.
While additional data will be gathered that may strengthen or modify these results, we see the need to present these early findings as soon as possible to help inform the next set of D&I research training programs in their development. Final data collection for all participating cohorts will be completed with the 18-month assessment in January 2019. At that time, our team will assess the overall impact the program has had on all 56 fellows who have been enrolled. Additional case studies and qualitative analysis will help provide more robust feedback on the overall impact the program has had on the individual fellows and their career trajectory. Following methods used in the Luke et al.  evaluation of the IRI, the team plans to continue to collect data regarding the fellow’s academic output as well as conduct a Social Network Analysis of fellows’ professional relationships. There are plans for future work to connect the outcomes between a fellow’s social connectedness and their skill level.
Additional work can also be done to compare the effectiveness between the different D&I program formats. While this program currently uses its own evaluation tools, these tools can be formalized to be useful across disciplines and formats. This additional comparison could provide support for the range in the types of D&I training formats and assess which ones are suited to particular populations (e.g., students, post-docs, assistant professors, tenured associate, or full professors) or learning styles.
The results from this program are valuable in demonstrating the effect and need for such training programs. The experience with MT-DIRC over the past few years provide a sound understanding of the landscape and a solid infrastructure that can be carried forward to fit with the new training models supported by sponsoring agencies. Results like these demonstrate promising evidence of the effectiveness of such mentored training programs and support the funding of such programs in the future. It is the hope that those who intend to replicate D&I trainings can apply many of the principles outlined to develop and evaluate related training programs.
In the first several years of the Mentored Training in Dissemination and Implementation Research in Cancer program, we were able to demonstrate the effects of such a mentored approach to training program as evidenced by the increase in skill level over time and the satisfaction that fellows have demonstrated with their mentoring and the program overall. We hope that by disseminating our findings and lessons learned, other groups can replicate and improve upon this model in order to continue to train the next generation of D&I science researchers.
Dissemination and Implementation
Implementation Research Institute
Mentoring Competency Assessment
Mentored Training for Dissemination and Implementation Research in Cancer
National Institute for Mental Health
Training Institute in Dissemination and Implementation Research in Health
Chambers DA, Proctor EK, Brownson RC, Straus SE. Mapping training needs for dissemination and implementation research: lessons from a synthesis of existing D&I research training programs. Transl Behav Med. 2017;7(3):593–601.
Norton WE, Lungeanu A, Chambers DA, Contractor N. Mapping the growing discipline of dissemination and implementation science in health. Scientometrics. 2017;112(3):1367–90.
Colditz GA, Wolin KY, Gehlert S. Applying what we know to accelerate cancer prevention. Sci Transl Med. 2012;4(127):127rv124.
Shea CM, Young TL, Powell BJ, Rohweder C, Enga ZK, Scott JE, Carter-Edwards L, Corbie-Smith G. Researcher readiness for participating in community-engaged dissemination and implementation research: a conceptual framework of core competencies. Transl Behav Med. 2017;7(3):393–404.
Brownson RC, Colditz GA, Dobbins M, Emmons KM, Kerner JF, Padek M, Proctor EK, Stange KC. Concocting that magic elixir: successful grant application writing in dissemination and implementation research. Clin Transl Sci. 2015;8(6):710–6.
Tabak RG, Padek MM, Kerner JF, Stange KC, Proctor EK, Dobbins MJ, Colditz GA, Chambers DA, Brownson RC. Dissemination and implementation science training needs: insights from practitioners and researchers. Am J Prev Med. 2017;52(3s3):S322–s329.
Meissner HI, Glasgow RE, Vinson CA, Chambers D, Brownson RC, Green LW, Ammerman AS, Weiner BJ, Mittman B. The U.S. training institute for dissemination and implementation research in health. Implement Sci. 2013;8:12.
Proctor EK, Landsverk J, Baumann AA, Mittman BS, Aarons GA, Brownson RC, Glisson C, Chambers D. The implementation research institute: training mental health implementation researchers in the United States. Implement Sci. 2013;8:105.
Kho ME, Estey EA, DeForge RT, Mak L, Bell BL. Riding the knowledge translation roundabout: lessons learned from the Canadian Institutes of Health Research Summer Institute in knowledge translation. Implement Sci. 2009;4:33.
National Institutes of Health: Dissemination and implementation research in health (R01). In., vol. PAR-16-238. Bethesda, MD: National Institutes of Health; 2016: 24.
Dissemination and implementation of PCORI-Funded Patient-Centered Outcomes Research Results. https://www.pcori.org/funding-opportunities/announcement/limited-pcori-funding-announcement-dissemination-and-4. Accessed 1 Dec 2017.
Job Opportunities: Implementation Science News. http://news.consortiumforis.org/category/jobs/. Accessed 1 Dec 2017.
Morrato EH, Rabin B, Proctor J, Cicutto LC, Battaglia CT, Lambert-Kerzner A, Leeman-Castillo B, Prahl-Wretling M, Nuechterlein B, Glasgow RE, et al. Bringing it home: expanding the local reach of dissemination and implementation training via a university-based workshop. Implement Sci. 2015;10:94.
Proctor EK, Chambers DA. Training in dissemination and implementation research: a field-wide perspective. Transl Behav Med. 2017;7(3):624–35.
Padek M, Colditz G, Dobbins M, Koscielniak N, Proctor EK, Sales AE, Brownson RC. Developing educational competencies for dissemination and implementation research training programs: an exploratory analysis using card sorts. Implement Sci. 2015;10:114.
Gonzales R, Handley MA, Ackerman S, O'Sullivan PS. A framework for training health professionals in implementation and dissemination science. Acad Med. 2012;87(3):271–8.
Darnell D, Dorsey CN, Melvin A, Chi J, Lyon AR, Lewis CC. A content analysis of dissemination and implementation science resource initiatives: what types of resources do they offer to advance the field? Implement Sci. 2017;12(1):137.
Stamatakis KA, Norton WE, Stirman SW, Melvin C, Brownson RC. Developing the next generation of dissemination and implementation researchers: insights from initial trainees. Implement Sci. 2013;8(1):29.
Straus SE, Brouwers M, Johnson D, Lavis JN, Legare F, Majumdar SR, McKibbon KA, Sales AE, Stacey D, Klein G, et al. Core competencies in the science and practice of knowledge translation: description of a Canadian strategic training initiative. Implement Sci. 2011;6:127.
Jones K, Armstrong R, Pettman T, Waters E. Knowledge translation for researchers: developing training to support public health researchers KTE efforts. J Public Health (Oxford, England). 2015;37(2):364–6.
Brownson RC, Proctor EK, Luke DA, Baumann AA, Staub M, Brown MT, Johnson M. Building capacity for dissemination and implementation research: one university’s experience. Implement Sci. 2017;12(1):104.
Webinars. https://cyberseminar.cancercontrolplanet.org/implementationscience/. Accessed 27 Nov 2017.
Kilbourne AM, Elwy AR, Sales AE, Atkins D. Accelerating research impact in a learning health care system: VA’s quality enhancement research initiative in the Choice Act Era. Med Care. 2017;55(7 Suppl 2):S4–S12.
Carlfjord S, Roback K, Nilsen P. Five years’ experience of an annual course on implementation science: an evaluation among course participants. Implement Sci. 2017;12(1):101.
Luke DA, Baumann AA, Carothers BJ, Landsverk J, Proctor EK. Forging a link between mentoring and collaboration: a new training model for implementation science. Implement Sci. 2016;11(1):137.
Cancer Control Continuum. http://cancercontrol.cancer.gov/od/continuum.html. Accessed 5 May 2017.
Brownson R, Colditz G, Proctor E, editors. Dissemination and implementation research in health: translating science to practice. New York: Oxford University Press; 2012.
Trochim W, Kane M. Concept mapping: an introduction to structured conceptualization in health care. Int J Qual Health Care. 2005;17(3):187–91.
Mentor training. http://www.ctsi.umn.edu/education-and-training/mentoring/mentor-training. Accessed 7 Apr 2017.
Pfund C, Branchaw J, Handelsman J. Entering mentoring. 2nd ed. New York: W.H. Freeman & Co.; 2015.
Pfund C, House S, Spencer K, Asquith P, Carney P, Masters KS, McGee R, Shanedling J, Vecchiarelli S, Fleming M. A research mentor training curriculum for clinical and translational researchers. Clin Transl Sci. 2013;6(1):26–33.
Pfund C, House SC, Asquith P, Fleming MF, Buhr KA, Burnham EL, Eichenberger Gilmore JM, Huskins WC, McGee R, Schurr K, et al. Training mentors of clinical and translational research scholars: a randomized controlled trial. Acad Med. 2014;89(5):774–82.
Pfund C, Maidl Pribbenow C, Branchaw J, Miller Lauffer S, Handelsman J. Professional skills. The merits of training mentors. Science. 2006;311(5760):473–4.
10th Annual Conference on the Science of Dissemination and Implementation. http://www.academyhealth.org/events/site/10th-annual-conference-science-dissemination-and-implementation-health. Accessed 10 Dec 2017.
VA HSR&D Cyberseminars. https://www.hsrd.research.va.gov/cyberseminars/. Accessed 10 July 2017.
Webinars-Implementation Science. https://cyberseminar.cancercontrolplanet.org/implementationscience/. Accessed 27 Aug 2017.
Bandura A. Self-efficacy: toward a unifying theory of behavioral change. Psychol Rev. 1977;84(2):191–215.
Artino AR Jr. Academic self-efficacy: from educational theory to instructional practice. Perspectives Med Educ. 2012;1(2):76–85.
Fleming M, House S, Hanson VS, Yu L, Garbutt J, McGee R, Kroenke K, Abedin Z, Rubio DM. The Mentoring Competency Assessment: validation of a new instrument to evaluate skills of research mentors. Acad Med. 2013;88(7):1002–8.
Field A. Discovering statistics using IBM SPSS statistics. 4th ed. Los Angeles: Sage Publications; 2013.
Feldman MD, Huang L, Guglielmo BJ, Jordan R, Kahn J, Creasman JM, Wiener-Kronish JP, Lee KA, Tehrani A, Yaffe K, et al. Training the next generation of research mentors: the University of California, San Francisco, Clinical & Translational Science Institute Mentor Development Program. Clinical Transl Sci. 2009;2(3):216–21.
Blixen CE, Papp KK, Hull AL, Rudick RA, Bramstedt KA. Developing a mentorship program for clinical researchers. J Contin Educ Heal Prof. 2007;27(2):86–93.
Feldman MD, Arean PA, Marshall SJ, Lovett M, O'Sullivan P. Does mentoring matter: results from a survey of faculty mentees at a large health sciences university. Med Educ Online. 2010;15
We thank Matthew Kreuter, PhD; Graham Colditz, MD, DrPH; Anne Sales RN, PhD; and Debra Haire-Joshu, PhD, for their service as the core faculty members to the MT-DIRC program. We are also grateful to Melissa Franco, BS, for her assistance in supporting MT DIRC.
Support for this project came from the National Cancer Institute at the National Institutes of Health Mentored Training for Dissemination and Implementation Research in Cancer Program (MT-DIRC) (5R25CA171994-02) and the US Department of Veterans Affairs and the Cancer Research Network. This work was also funded in part by the National Institute of Diabetes and Digestive and Kidney Diseases (1P30DK092950) and Washington University Institute of Clinical and Translational Sciences grant UL1 TR000448 and KL2 TR000450 from the National Center for Advancing Translational Sciences. The findings and conclusions in this article are those of the authors and do not necessarily represent the official positions of the National Institutes of Health.
Availability of data and materials
The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request.
This program and its subsequent evaluations were approved by Washington University’s Institutional Review Board with the “Exempt” designation (IRB #201303122). All fellows were provided informational flyers regarding the evaluations prior to their entry in the program.
Consent for publication
No identifying information on any individual’s data is presented in this paper. All fellows and faculty were given relevant informational flyers prior to the administration of each survey.
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.