Phase 1: initial competency list

The MT-DIRC principal investigator (PI) and core faculty established an initial list of 33 D&I competencies for the program’s grant application developed during the fall of 2012. These competencies were based on the core faculty’s previous experience with planning and coordinating prior D&I training programs (the Implementation Research Institute (IRI) and the Training Institute for Dissemination and Implementation Research in Health (TIDIRH)) [4, 5]; recommendations from the Institute of Medicine [20]; and the faculty members’ previous endeavors with their textbook, Dissemination and Implementation Research in Health: Translating Science to Practice [10]. These were the primary sources that helped inform this initial list of competencies. This group consisted of seven researchers from across the USA and Canada who not only have been among national and international leaders in the fields of D&I science but also come from various disciplines such as social work, nursing, behavioral science, health communication, medicine, and epidemiology. All core members have had prior experience as instructors in D&I training programs.

To refine and amend the initial competency list, we engaged additional experts in the field of dissemination and implementation science. This additional group fits with the recommendation from the Council of the National Postsecondary Education Cooperative’s report that stakeholders in educational programs should have a role in the identification of such competencies [21]. For this particular program, our stakeholders are additional researchers who would be recipients or providers of such training. The core members each recommended names of colleagues who were also considered experts in the field of D&I science. This list consisted of 26 individuals who were from various institutions in the USA, Canada, and Australia. These additional experts all received an email from the PI with this initial list. In these emails, recipients were asked to review the attached list (initial 33 competencies) and “provide input: additions to the list, deletions, or rewording/questions about clarity.” In all, 16 different reviewers (62 %) gave unique comments or suggestions about the initial list.

Individual feedback was compiled into an Excel document and coded according to the suggested wording and action (deletion, addition, clarification). Many of the responses from the reviewer were similar in context or repetitious. The cumulative feedback from colleagues generated a list of over 100 different statements. Based on redundancy of competencies and to express clarity of meaning, the list was edited down by the PI, core faculty, and program coordinator to a final list that consisted of 43 D&I competencies. Some statements were also combined based on the similarity of concepts to provide a more feasible list that could be incorporated into a curriculum.

Based on prior experience within previous training programs, these initial competencies were categorized into four domains [4, 5]: Definition, Background, and Rationale; Theory and Approaches; Design & Analysis; and Practice-Based Considerations. A fifth domain was created which addressed Grant Development-specific competencies. While the Grant Development-specific competencies were believed to be pertinent to a training curriculum, the analysis hereafter does not focus on these competencies, as they are not content-specific to D&I research.

Phase 2

We next categorized the newly identified 43 statements into hierarchical training levels. To do this, the team employed the use of a card sort approach. Card sort participants should be those who would be the most likely users of a particular program and allow for the categorization of knowledge within a training program [22]. In this case, we utilized previous participants in D&I training programs as they would have familiarity with the content area as well as the type of program curriculum being proposed. Because D&I science is not a general knowledge area for all health researchers, the research team engaged participants who would at least have a base knowledge of the field in order to garner more accurate results from the card sort. The use of attendance lists from previous training programs was the most complete and feasible method for obtaining a broad list of card sort participants.

Card sort participants

Participants in the card sort were recruited through contact lists from the 2010–2013 IRI (R25MH080916) training program (n = 55), as well as from the 2013 TIDIRH (n = 65). Participants were also added from an internal Washington University D&I network listserv (n = 87). The research team identified the remaining participants from their previous collaborations in D&I work (n = 93), particularly those individuals located outside the USA. After a participant’s contact information was compiled and identified for the D&I competency card sort, potential participants were sent an email explaining the project and inviting them to participate in this activity. This activity received exempt status from Washington University’s Institutional Review Board.

Card sorting

We used a card sort to help organize the identified D&I competencies by skill level [20]. Card sorts are widely used today by technology companies for website development and organization in user testing. However, social science researchers have been using this type of activity within participatory-based research for many years [20] as a means to organize concepts in a way that the community served by the intervention is best able to understand [21, 22]. The MT-DIRC research team took this same approach, having those who have experienced D&I training programs organize competencies into learning levels that seemed the most logical to the individual.

Qualtrics©, a web-based survey technology provider, was used to execute the virtual card sort activity. The previously identified competency list from phase 1 was used as statements within Qualtrics© in their “Question & Sort” feature. Participants were asked to place each competency statement into the column that best expressed the skill level needed to address that particular competency. Columns were marked “Beginner,” “Intermediate,” and “Advanced.” Definitions were not given for these labels to participants. The researchers intentionally chose not to define the categories to allow the participants to self-define, which gives insight into how these competencies are conceptualized into learning levels [22]. Lack of term definitions also helps to exclude any unintended bias from the research team (who are themselves considered D&I experts) on how they would define learning levels [22].

Participants were given 3 weeks to complete the card sorting. Reminder emails, a week before the close date, cued the participants to either start or finish the activity. At the end of the card sort activity, participants were asked to denote their own level of expertise in the field of D&I research (beginner, intermediate, advanced), as well as indicate their country of origin. These were the only two demographic questions asked of participants in phase 2. Participants also had the opportunity to leave comments and feedback about the list. The software tracked the amount of time it took to complete the activity with the average completion time being approximately 15 min.

Data analysis

This activity used an exploratory analysis, employing qualitative feedback to set up the content of the activity and quantitative methods to assess the overall categorization process. While cluster and hierarchical analysis can be used during the card sort process, the group used descriptive analyses and analysis of variance (ANOVA) for this particular analysis [23, 24]. This statistical analysis was possible because of the closed nature of the card sort (i.e., the groups to which statements were sorted were predetermined) [24, 25]. After participants sorted the competency statements and data compiled, mean scores were calculated for these groupings. Competency statements placed within the beginner, intermediate, or advanced groups were coded as “1, 2, or 3,” respectively. These numbers represented categorical coding rather than ordinal coding in order to create cutoffs between the different experience level groups. The research team intended to see which competencies on average fell into which learning levels, not to determine an ordinal rank of their difficulty. The mean score was then calculated for each competency statement.

Competencies were sorted by their mean score in ascending order and then divided into tertiles based on the distribution of scores. The first tertile included competencies that on average were sorted within the beginner level. The second tertile corresponded to competencies that on average were sorted within the intermediate level. The last tertile included competencies that on average were placed within the advanced level. Once tertiles were determined, minor adjustments between groups were made based on the frequency distribution of the responses. Despite the tertile cutoffs, some distributions of competencies had a higher response rate in one category that ultimately did not result in their tertile ranking within that category. The core team examined the frequency of scores for that particular competency, and if there was a higher distribution of scores in a different experience level than what the mean score placed the competency, then adjustments were made to better reflect the majority of responses within that particular competency.