- Open Access
- Open Peer Review
A content analysis of dissemination and implementation science resource initiatives: what types of resources do they offer to advance the field?
Implementation Sciencevolume 12, Article number: 137 (2017)
The recent growth in organized efforts to advance dissemination and implementation (D & I) science suggests a rapidly expanding community focused on the adoption and sustainment of evidence-based practices (EBPs). Although promising for the D & I of EBPs, the proliferation of initiatives is difficult for any one individual to navigate and summarize. Such proliferation may also result in redundant efforts or missed opportunities for participation and advancement. A review of existing D & I science resource initiatives and their unique merits would be a significant step for the field. The present study aimed to describe the global landscape of these organized efforts to advance D & I science.
We conducted a content analysis between October 2015 and March 2016 to examine resources and characteristics of D & I science resource initiatives using public, web-based information. Included resource initiatives must have engaged in multiple efforts to advance D & I science beyond conferences, offered D & I science resources, and provided content in English. The sampling method included an Internet search using D & I terms and inquiry among internationally representative D & I science experts. Using a coding scheme based on a priori and grounded approaches, two authors consensus coded website information including interactive and non-interactive resources and information regarding accessibility (membership, cost, competitive application, and location).
The vast majority (83%) of resource initiatives offered at least one of seven interactive resources (consultation/technical assistance, mentorship, workshops, workgroups, networking, conferences, and social media) and one of six non-interactive resources (resource library, news and updates from the field, archived talks or slides, links pages, grant writing resources, and funding opportunities). Non-interactive resources were most common, with some appearing frequently across resource initiatives (e.g., news and updates from the field).
Findings generated by this study offer insight into what types of D & I science resources exist and what new resources may have the greatest potential to make a unique and needed contribution to the field. Additional interactive resources may benefit the field, particularly mentorship opportunities and resources that can be accessed virtually. Moving forward, it may be useful to consider strategic attention to the core tenets of D & I science put forth by Glasgow and colleagues to most efficiently and effectively advance the field.
In the last decade, numerous efforts to advance the science of dissemination and implementation (D & I) have emerged globally. Conferences to disseminate scientific innovations occur with regularity around the world, D & I science training is available through specialized institutes, and D & I science is seen as global public health priority , receiving support from funding agencies worldwide . The proliferation of such efforts is a testament to the widespread recognition that knowledge of what is effective for patients is frequently not translated into real world practice [3,4,5] and that there appears to be a lack of empirical scrutiny applied to methods for disseminating and implementing evidence-based practices (EBPs) [6,7,8].
Anecdotally, we have observed a similar proliferation of organized efforts to disseminate knowledge and skills for D & I science, which we refer to as “D & I science resource initiatives.” The motives for developing D & I science-focused resource initiatives likely vary greatly, but may include reasons such as expanding professional networks to identify collaborators or to facilitate institutional support and interest in D & I (e.g., the US Clinical and Translational Science Awards; ). These resource initiatives provide useful resources to the D & I science community, which ultimately bodes well for improvements in human health. However, the impact of these resource initiatives is dependent on how well they can reach their target audiences and whether potential participants fully understand their relative merits. Further, the diffusion of efforts to advance D & I science without attention to how these resources complement each other could unwittingly stymie scientific progress . For example, concerns about inefficient use of resources prompted researchers at National institutes of health (NIH) to offer guidance to D & I scientists on how future efforts can harness resources to most effectively advance the field . For example, one suggested means of improving relevance and rigor of research is for scientists to engage in collaborations with community stakeholders. Such guidance may apply equally well to how D & I science resource initiatives approach the selection of their activities and resources offered.
It is not clear how D & I science resource initiatives prioritize activities to pursue and resources to offer to the public or members, and we are not aware of any effort to summarize the activities and the resources they offer to the field. Therefore, the main objectives of the present study were to survey the global landscape of current D & I science resource initiatives, identify types of resources they offer, and characterize their accessibility to the D & I science community. Given the value of practice and policy collaborators to support contextually relevant research questions and methods , we aimed to identify whether resource initiatives incorporated non-researcher stakeholders and whether resources varied between these types of resource initiatives. Additionally, we aimed to specify whether resource initiatives focused on D & I as they pertain to advancing EBPs for physical versus behavioral health (i.e., both mental and substance use) problems. This is a meaningful distinction given the preponderance of EBPs for behavioral health is complex psychosocial interventions that require unique ways of measuring fidelity and the involvement of purveyors, for example, that often does not occur in other disciplines . We then explored whether behavioral health versus health focused resource initiatives varied in the types of resources they offer.
We conducted a web-based and expert-informed search and content analysis to examine characteristics and available resources of D & I science resource initiatives using public website information and both a priori and an emergent, grounded approach . Data represents publicly available website information accessed between October 2015 and March 2016.
To obtain the sample of resource initiatives, we utilized two search methods: (1) searching the Internet for websites and (2) inquiring about known resource initiatives from internationally representative D & I science experts and their colleagues (i.e., snowball sampling). For the first method, we used Google Advanced Search to find publicly available websites. Two members of the research team independently inputted search strings comprised of common D & I science-related terms using phrases taken from Proctor and colleagues’ review of D & I concepts (see Table 1) . Search string examples include “implementation science” and “knowledge translation,” the latter of which highlights our efforts to include international resource initiatives given that is the term more frequently used in Canada . These terms can be found in Table 1. This search process continued until both members perceived saturation (i.e., search terms no longer produced novel resource initiatives). This search method resulted in a total of 97 resource initiatives for potential study inclusion.
For the second method, we employed a snowball-sampling approach. Specifically, emails detailing the purpose of the study, as well as our resource initiative inclusion criteria (see below), were sent out to over 200 members of the Society for Implementation Research Collaboration’s (SIRC) Network of Expertise (NoE; ). SIRC’s NoE is comprised of implementation science researchers at various career stages, including established implementation scientists, new investigators, and student members. To supplement the NoE emails, the same content was posted publicly on SIRC’s website. Finally, this same request for resource initiatives was sent to an international advisory board of nine implementation scientists representing the following countries: Australia, Canada, England, Germany, and Sweden. This second search method resulted in a total of 33 additional resource initiatives for potential study inclusion.
We defined resource initiatives as organized efforts to disseminate knowledge and foster skills for advancing D & I science (not translational science broadly) via provision of resources to the research community broadly (e.g., excluding academic centers that only provide resources to the local institution). We were particularly interested in efforts that engaged in multiple strategies for advancing D & I science and therefore excluded efforts that reflect only a conference or conference series, a training institute, or funding mechanism. Resource initiatives could advance D & I science through conducting their own research, promoting D & I science, or promoting use of science-informed D & I practice. Additionally, the content website had to be available in English.
All initial 130 potential resource initiatives were rated for inclusion by two independent raters; any discrepancies were resolved by the research team. The final sample included 42 resource initiatives that met inclusion criteria. The names of resource initiatives, their corresponding website, and countries of origin are noted in Table 2. The majority of exclusions were due to not advancing D & I science beyond the organization of a conference or beyond the scope of a single institution.
Units of analysis
Units of analysis for this study included D & I science resource initiative websites, including information from the main website and any related conference websites if applicable. Each page that contained the same base URL was eligible for coding purposes (described below under the “Coding Process” section).
We developed codes for resource initiative characteristics based on a priori theoretical interests (see Table 3 for code definitions). Codes for resources were also identified by an emergent process, whereby we observed types of resources offered by resource initiatives during initial review of the websites. Resources that warranted coding emerged based on an anticipated interest of the resource to the field of D & I science or commonality among the resource initiatives. Additionally, we qualified the degree to which resources may be more or less available to the larger D & I science community through the use of sub-codes. Where relevant, we gathered qualitative data to describe the nature of coded resources.
Codes for characteristics of resource initiatives included whether the resource initiative was focused on implementing EBPs targeting physical health, behavioral health (i.e., mental health and/or substance use disorder) EBPs, or a combination of both. Additionally, we captured whether the resource initiative included participation of a non-researcher stakeholder and whether there was an option to become a formal member of the resource initiative (either for a fee or free).
The types of resources provided by resource initiatives were classified into interactive (i.e., require participation of two parties in some way) and non-interactive (i.e., accessible by one party). Interactive resources included (1) consultation/technical assistance, (2) mentorship, (3) workshops, (4) workgroups, (5) networking, (6) conferences, and (7) social media. Non-interactive resources included (1) a resource library, (2) news and updates from the field, (3) archived talks or slides (includes webinars), (4) links pages, (5) grant writing resources, and (6) funding opportunities.
As applicable, sub-codes identified whether a resource was accessible to only members or the general public, associated with a fee, required a competitive application, or was available in-person, virtually, or both.
Coder training and pilot coding
The research team completed three rounds of pilot coding, which resulted in iterative modifications to the coding scheme to enhance reliability of its application and inductively inform the coding scheme. Two coders were trained in the final coding scheme as well as how to navigate the resource initiative’s website to obtain information relevant to the coding scheme. This included instruction to (1) open all of the links on a page as soon as a new page is opened, (2) go through each link on a page before progressing to the next page, and (3) work from top-to-bottom and left-to-right when opening links and reading through a webpage. Coders only included information present on the resource initiative’s main URL. The only exception to this was if the resource initiative’s conference had a unique URL, in which case the information from the conference website was also included. Two coders independently coded each of the 42 resource initiatives and then met to review and determine final codes based on consensus when coding was discrepant. Each coder coded the website within 3 months of the other coder.
We estimated inter-rater reliability for each main code (i.e., resources and characteristics). Because sub-codes were dependent on the main codes, and some main codes occurred infrequently, we did not calculate estimates of inter-rater reliability for sub-codes. After independent coding had been completed for approximately half of the resource initiatives, the rates of agreement calculated for each main code were reasonable, ranging from 67 to 90%.
Plan of analysis
We observed frequencies of resource initiative characteristics and resources, as well as the frequencies of sub-codes, when applicable. We explored bivariate relations between resource initiative characteristics and available resources, as well as the relations between resource initiative characteristics, using Chi-square tests of independence, with p < .05 indicating statistical significance. As these analyses were exploratory in nature, we did not use a corrected significance level for the large number of statistical tests.
Resource initiative characteristics
Table 4 presents the results of the coding process described above. Per websites, the majority of resource initiatives (n = 31, 74%) incorporated non-researcher stakeholder involvement, and a little more than half focused solely on D & I science as it relates to physical health EBPs (n = 25, 60%). Half of the resource initiatives offered formal membership (n = 21, 50%), and when offered, membership was associated with a fee half of the time (n = 10, 24% of all resource initiatives). We did not observe bivariate relationships between the resource initiative characteristics.
The vast majority of resource initiatives offered both interactive and non-interactive resources (n = 35, 83%). Of the seven types of interactive resources, resource initiatives had an average of 2.57 (SD = 1.81). The most commonly occurring was social media (n = 27, 64%). Qualitative data collected on the types of social media indicated that resource initiatives frequently utilized more than one type, with Twitter being the most frequently utilized, (n = 25, 60%), followed by Facebook (n = 17, 40%) and LinkedIn (n = 9, 21%). The presence of workshops (n = 19) and conferences (n = 18) were also common. Workshops often had a cost associated (n = 10) and the majority (n = 16) were open to the public. Conferences predominately occurred annually (n = 13) were open to the public (n = 15) and did not require a competitive application to attend (n = 16). The majority of the conferences did have an associated cost (n = 12). Workgroups (n = 18, 43%) predominately did not have a cost associated with them (n = 17). Roughly half of the workgroups did not require resource initiative membership (n = 9). The least commonly occurring interactive resources were mentorship (n = 6, 14%), followed by technical assistance (n = 9, 21%) and networking (n = 12, 29%). The locations of interactive resources varied: those most commonly only available in-person were mentorship, workshops, and networking.
Of the six types of non-interactive resources, resource initiatives had an average of 3.67 (SD = 1.52). The most common non-interactive resource was the presence of a links page (n = 35, 83%), followed by archived talks and slides (which includes webinars, n = 30, 71%) and the organization of news and updates from the field (n = 34, 81%). These resources rarely cost anything to the user, required membership, or required a competitive application to access. The less frequent non-interactive resources were grant writing resources (n = 16, 38%) and funding opportunities (n = 14, 33%). It was rare for grant writing resources to be associated with a cost, be limited to members, or contingent on a competitive application. For most resource initiatives that offered funding opportunities, they made these available to the public (n = 11). Additionally, 25 (60%) resource initiatives were found to include a resource library, none of which had an associated cost or application and were largely (n = 23) open to the public. Common types of resources included in the library were tools/toolkits, guides/guidelines, articles, reports, and systematic reviews.
Bivariate relations between characteristics and resources
Resource initiatives with a formal membership option were more likely to have mentorship, workshops, and workgroups and to use social media (see Table 5). There were no differences in resources available based on whether resource initiatives had non-researcher stakeholders involved or whether they had a physical health only focus versus a behavioral health only or combined physical health focus.
We identified 42 resource initiatives globally that provide resources to the English-speaking D & I science community and imagined that this number will continue to increase along with growing enthusiasm for the field. The 42 resource initiatives identified represent a diverse array of organized efforts to advance D & I science that provide a variety of interactive and non-interactive resources to not only researchers but also non-researcher stakeholders.
Availability of interactive and non-interactive resources
Resource initiatives commonly provide both interactive (i.e., require participation of two or more parties) and non-interactive (i.e., accessible by one party) resources (n = 35, 83%). Overall, however, non-interactive resources (e.g., resource libraries, archives, grant-writing resources, news/updates) were more common. This is relatively unsurprising given that these resources are often less expensive and easier to maintain than more interactive resources, which often carry a substantial time commitment for those involved (e.g., mentorship). Other benefits to non-interactive resources are that they are typically available “on demand,” more easily and flexibly accessed, and therefore are potentially more scalable than interactive resources. As a result, non-interactive resources have the potential for greater reach  to those in the field of D & I, particularly with regard to increased knowledge about D & I science and practice . Findings from the field, however, have shown us that less interactive approaches to information provision are unlikely to result in the uptake or application of new skills , which may limit the impact  of non-interactive resources. This makes the availability of interactive resources imperative. Additionally, efforts to innovate on scalable web-based didactic material to target skill development may be warranted to build capacity. Moreover, it may be that certain content is best suited for non-interactive resources (i.e., research findings), whereas other content (i.e., new methodologies) might demand interactive resources such as a workshop.
Interactive resources (e.g., conferences, workshops, mentorship, etc.), which provide opportunities for networking and collaboration, are essential to the promotion of the multidisciplinary, team approach that is often needed in the field of D & I science . Additionally, interactive resources may be more impactful than non-interactive resources, both in terms of skill acquisition as previously mentioned and with regard to professional development. For instance, mentorship is known to be integral to professional development , and a recent evaluation of the Implementation Research Institute, a training program for D & I scientists, indicates that mentorship plays an essential role in academic successes of its fellows . However, mentorship was infrequent among the resources offered by resource initiatives (n = 6, 14%). This is likely due to the ongoing time commitment required for mentors and mentees and the fact that, to be most effective, D & I science mentors are likely to be drawn from a pool of established experts that remains relatively small. A recent study of D & I scientists indicates many pursue research collaborators to obtain mentorship, but few pursue collaborators to provide mentorship . To match the growing demand for D & I scientists, it is prudent for resource initiatives to expand mentoring opportunities.
Social media was commonly, but not uniformly, used by resource initiatives. Social media is particularly useful for publicizing a resource initiative, promoting the larger D & I science field, and sharing information with and between members or stakeholders who use social media platforms. There is some indication that social media can spark collaborations among researchers, particularly through sites like ResearchGate (https://www.researchgate.net/), developed specifically to connect researchers ; however, the incremental value of social media for collaborative purposes remains to be seen. Review of the types of social media resource initiatives commonly used indicates most rely on sites designed for primarily social (i.e., Twitter) versus research networking (i.e. LinkedIn) activities. It may be that integrated use of both types of sites, as well as creative integration with other types of resources, may hold the most promise for the impact of social media on D & I science .
Interactive workshops were offered by just under half of the resource initiatives and have the potential to develop knowledge and skills related to D & I science, as well as offer enhanced networking opportunities. Although these were predominantly offered in-person, the use of virtual workshops may greatly enhance the reach of these resources, as they can be as effective and well-received by participants as in-person workshops . Conferences were also offered by a substantial minority of resource initiatives. Conferences offer researchers the quintessential opportunity for networking combined with sharing novel research ideas and disseminating research findings. They are highly resource intensive for the resource initiative as well as for attendees, requiring time, finances, and preparation of research presentations but valued by funding agencies (e.g., ) as well as the researchers.
Recently, there have been growing concerns about the potential for a widening divide between D & I science and practice in which findings from D & I science are not routinely applied when integrating new innovations into service systems, and concern that D & I science is likely to produce tools and strategies that may be impractical in many service settings [7, 25]. Such a divide runs the risk of replicating the well-documented gap between research and practice in intervention science, which was one of the primary catalysts for the original emergence of the D & I field [26,27,28]. To address this divide, interdisciplinary conferences that bring together researchers and non-researchers, such as policy makers and practice-based administrators and clinicians who champion EBPs, may be essential. However, non-researchers may be less able than researchers to attend conferences in-person, as their organizations may not have financial resources available for travel and attendance or the resources to allot staff time away from clinical or administrative duties. Conferences that aim to reduce the D & I science-practice divide may benefit from incorporating virtual participation to accommodate attendees who are valued but unable to otherwise attend in-person and ensuring that continuing education credit offerings are abundant.
A sizable majority of resource initiatives incorporated both stakeholders who operate in a research or non-research role (n = 31, 74%), which may be essential to addressing the aforementioned D & I science-practice divide. Collaborations between D & I scientists and practitioners (i.e., EBP purveyors, intermediaries, or administrators implementing EBPs in the community), decision-makers, and patient stakeholders are particularly important to advance rigor and relevance of the research  and policy to support implementation . However, it remains unclear the extent of non-researcher stakeholder involvement in these resource initiatives and the degree to which interactive resources would include both groups for optimal impact and bidirectional learning.
Although a minority of resource initiatives provided opportunities for formal membership (n = 21, 50%), those that had this characteristic were more likely to offer interactive resources including mentorship, workshops, workgroups, and social media. However, having a formal membership option was not related to non-interactive resources. It may be that interactive resources are more intensive in nature and are best supported by resource initiatives that have the infrastructure that goes along with being able to offer a formal membership and, potentially, receive financial support through that mechanism. Additionally, membership, even when not associated with a cost, as was the case for half of resource initiatives, may be a means of restricting access to what may be more scarce interactive resources.
The specific healthcare domain on which resource initiatives focused (i.e., having a physical health only focus versus a focus on behavioral health only or in combination with physical health) was not related to the type of resources offered. This result aligns well with the common conceptualization that D & I science transcends any particular domain of healthcare or social service . Indeed, most of the existing theories, frameworks, and models ; critical predictors of successful D & I ; and strategies to support adoption and sustainment are intended to have applicability across a broad range of EBP and practice domains. Increasingly, these commonalities are giving rise to the use of standard language that is readily interpretable and which serves to promote information sharing and collaboration regardless of the backgrounds of professionals or contexts in which they work. Nevertheless, we anticipate that efforts to integrate different healthcare domains within implementation science may continue to face some barriers due to variable terminology, diverse publication outlets, distinct funding streams, and independent professional networks. Although these barriers are not insurmountable, they may require diligence when conducting literature searches, tracking innovations, and identifying collaborators. It may also be the case that elements not coded for in our analysis differ across these domains such as the types of measures included in repositories (e.g., observational coding schemes for fidelity rating in behavioral health) or the kinds of non-researcher stakeholders (e.g., purveyors in behavioral health).
Future considerations for advancing D & I science
In addition to those already noted, we offer considerations for advancing the field of D & I science based on study findings and the larger literature. First, the present study relied on the resource of a publicly available website. Websites provide essential communication about, and often access to, the resource initiative’s resources. Additionally, websites provide at least one point of access to the resource initiative for interested stakeholders. Therefore, up-to-date websites with rich content are themselves a resource to the D & I science community. Anecdotally, the reviewed websites varied widely in the extent to which they could be navigated and information easily found. Website usability issues are well-documented across a wide variety of sectors and impact the effectiveness and satisfaction with which users can engage with online materials [32,33,34,35]. More explicit attention to potential usability issues may further the goal of making websites accessible to the broadest range of D & I professionals.
The field of D & I science is rapidly expanding, and a large body of knowledge has been accumulated. Through the resources they offer, D & I science resource initiatives provide considerable opportunities to help researchers and practitioners sift through and apply this knowledge. Additionally, although promising for EBP D & I, this rapid expansion risks redundancy of efforts or potentially overlooked areas of need . Our content analysis revealed that some resources are more common than others among the resource initiatives (e.g., news and updates from the field). Although we cannot say whether these resources are redundant without more information on the nature of the resource and who accesses them, we have observed a degree of resource clustering within these categories. Initiatives, particularly those with limited funding or workforce resources, may choose to put more effort into providing resources that do not already appear to saturate the field of D & I science. Our analysis also identified areas where more resources may be warranted (e.g., mentorship).
We propose that D & I science resource initiatives may also benefit from considering how their resources and characteristics align with the tenets of D & I science delineated by Glasgow and colleagues . The tenets include collaboration, efficiency and speed, rigor and relevance, improved capacity, and cumulative knowledge. For instance, the interactive resources that clearly promote collaboration (e.g., workgroups, conferences) and improved capacity (e.g., mentorship) were less common than the non-interactive resources that likely promote cumulative knowledge (e.g., resource library, archived talks/slides), indicating a potential need for a shift in resource allocation. Unfortunately, without knowing the exact nature of the resources offered and stakeholder involvement in resource initiatives, it is unclear the extent to which the tenets of efficiency and rigor and relevance are well promoted. Even without such clarity, however, resource initiatives may still benefit from considering how their own resources and characteristics align with the tenets and incorporating them into strategic planning. For instance, the Society for Implementation Research Collaboration underwent a self-study with this exact purpose and determined the theme of their 2015 conference to fill a perceived gap in targeted efforts at advancing efficiency and speed .
There are several limitations to the current study in addition to those previously noted. Our content analysis was based on information gathered on publicly available websites. Websites are not static sources of information, making it possible that the results reported are not reflective of the current state of resource initiative characteristics and offerings. Rather, the data presented reflects the information available on each resource initiative’s website from October 21, 2015 to March 21, 2016. Additionally, we were not able to access all the information about some resource initiatives when membership was required to view certain parts of the website. Accordingly, this may have resulted in an underreporting of offerings for such resource initiatives. A more accurate depiction of resource initiatives, including characteristics such as non-researcher stakeholder role, and the resources they offer may require interviews with resource initiative members or leadership. However, the latter approach was beyond the scope of the modestly funded present study. Finally, although we attempted to sample from the global population of D & I science resource initiatives, it is also possible that our sampling strategies, particularly the requirement that materials be provided in English, resulted in under-sampling resource initiatives outside of the USA and Canada. In fact, the majority of resource initiatives identified were based in the USA. Future research might explicitly target non-English language websites to ensure a more comprehensive review of global resource initiatives. Additionally, given that the field uses myriad terms worldwide to refer to D & I science, we may have missed relevant search terms and unintentionally excluded resource initiatives that use those terms to refer to their efforts.
The recent growth in D & I science resource initiatives suggests a rapidly expanding community focused on the adoption and sustainment of health-related EBPs. We are unaware of any previous attempt to systematically collect knowledge about these resource initiatives and examine their unique merits. Despite the limitations above, the present study reflects a considerable step forward in understanding current resources available for D & I science and the resource initiatives that promote the field. Although spanning many disciplines and health domains, D & I resource initiatives appear to reflect a single field of D & I science devoted to supporting information sharing and collaboration to improve the D & I of EBPs. Nevertheless, there exists room for improvement. Mentorship opportunities, in particular, seem to be ripe for expansion, although they may require the most intensive resources to be successful. An increase in options for virtual access to interactive resources may improve the reach of these vital resources. Additionally, as resource initiatives continue with and expand on their resources, it may be useful to consider strategic attention to the core tenets of D & I science put forth by Glasgow and colleagues  to most efficiently and effectively advance the field.
- D & I:
Dissemination and implementation
National institutes of health
Network of expertise
Society for implementation research collaboration
World Health Organization. The WHO strategy on research for health. Geneva: Switzerland: WHO, 2012.
Smits P, Denis J. How research funding agencies support science integration into policy and practice: an international overview. Implementation Sci. 2014;9:28.
Glasgow RE, Lichtenstein E, Marcus AC. Why don’t we see more translation of health promotion research to practice? Rethinking the efficacy-to-effectiveness transition. Am J Public Health. 2003;93:1261–7.
Shen B. Toward cross-sectoral team science. Am J Prev Med. 2008;35:S240–2.
Esparza J, Yamada T. The discovery value of “Big Science”. J Exp Med. 2007;204:701–4.
Borner K, Contractor N, Falk-Krzesinski HJ, Fiore SM, Hall KL, Keyton J, Spring B, Stokols D, Trochim W, Uzzi B. A multi-level systems perspective for the science of team science. Sci Transl Med. 2010;2:49cm24.
Weisz JR, Ng MY, Bearman SK. Odd couple? Reenvisioning the relation between science and practice in the dissemination-implementation era. Clin Psychol Sci. 2014;2:58–74.
Pearl JB III, Rubino A. The power of cluster evaluation networking conferences. TX: In Annual Meeting of the American Evaluation Association Dallas; 1993.
Clinical & Translational Science Awards. https://ctsacentral.org/. Accessed 25 Oct 17.
Glasgow RE, Vinson C, Chambers D, Khoury MJ, Kaplan RM, Hunter C. National Institutes of Health approaches to dissemination and implementation science: current and future directions. Am J Public Health. 2012;102:1274–81.
Neuendorf KA. The content analysis guidebook. Thousand Oaks: Sage; 2002.
Rabin BA, Brownson RC, Haire-Joshu D, Kreuter MW, Weaver NL. A glossary for dissemination and implementation research in health. J Public Health Manag Pract. 2008;14:117–23.
Graham ID, Logan J, Harrison MB, Straus SE, Tetroe J, Caswell W, Robinson N. Lost in knowledge translation: time for a map? J Contin Educ Heal Prof. 2006;26:13–24.
Network of Expertise. https://societyforimplementationresearchcollaboration.org/network-of-expertise/. Accessed 25 Oct 17.
Glasgow R, Vogt T, Boles S. Evaluating the public health impact of health promotion interventions: the RE-AIM framework. Am J Public Health. 1999;89:1322–7.
Rabin BA, Nehl E, Elliott T, Deshpande AD, Brownson RC, Glanz K. Individual and setting level predictors of the implementation of a skin cancer prevention program: a multilevel analysis. Implement Sci. 2010;5:40.
Chambers DA, Azrin ST. Research and services partnerships: partnership: a fundamental component of dissemination and implementation research. Psychiatr Serv. 2013;64:509–11.
Allen TD, Eby LT, Poteet ML, Lentz E, Lima L. Career benefits associated with mentoring for protegee: a meta-analysis. J Appl Psychol. 2004;89:127–36.
Luke DA, Baumann AA, Carothers BJ, Landsverk J, Proctor EK. Forging a link between mentoring and collaboration: a new training model for implementation science. Implement Sci. 2016;11:137.
Norton W, Lungeanu A, Chambers D, Contractor N. Mapping the growing discipline of dissemination and implementation science in health. Scientometrics. 2017:1–24.
Van Noorden R. Online collaboration: scientists and the social network. Nature. 2014;512:126–9.
Hanna R, Rohm A, Crittenden VL. We’re all connected: the power of the social media ecosystem. Business Horizons. 2011;54:265–73.
Marriott BR, Rodriguez AL, Landes SJ, Lewis CC, Comtois KA. A methodology for enhancing implementation science proposals: comparison of face-to-face versus virtual workshops. Implement Sci. 2016;11:62.
NIH Support for Conferences and Scientific Meetings. (Parent R13) https://grants.nih.gov/grants/guide/pa-files/PA-16-294.html. Accessed 25 Oct 17.
Lyon AR, Comtois KA, Kerns SEU, Landes SJ, Lewis CC. Closing the science-practice gap in implementation before it widens. In: Shlonsky A, Mildon R, Albers B, editors. The science of implementation. New York: Springer. in press.
Balas EA, Boren SA. Managing clinical knowledge for health care improvement. Yearb Med Inform. 2000:65–70.
Eccles MP, Mittman BS. Welcome to implementation science. Implementation Sci. 2006;1:1.
Proctor E, Landsverk J, Aarons G, Chambers D, Glisson C, Mittman B. Implementation research in mental health services: an emerging science with conceptual, methodological, and training challenges. Admin Pol Ment Health. 2009;36:24–34.
Gagliardi A, Berta W, Kothari A, Boyko J, Urquhart R. Integrated knowledge translation (IKT) in health care: a scoping review. Implement Sci. 2016;11:38.
Tabak RG, Khoong EC, Chambers DA, Brownson RC. Bridging research and practice: models for dissemination and implementation research. Am J Prev Med. 2012;43:337–50.
Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci. 2009;4:50.
Jeffcoat King H, Jannik CM. Redesigning for usability: information architecture and usability testing for Georgia Tech Library’s website. In: OCLC systems & services. 2005;21:235–43. 235–243.
Kuan HH, Bock GW, Vathanophas V. Comparing the effects of usability on customer conversion and retention at e-commerce websites. InSystem Sciences, HICSS'05. Proc 38th Annu Hawaii Int Conf Syst Sci. 2005:174a–74a.
Law R, Qi S, Buhalis D. Progress in tourism management: a review of website evaluation in tourism research. Tour Manag. 2010;31:297–313.
Sutcliffe A: Heuristic evaluation of website attractiveness and usability. In Interactive systems: design, specification, and verification: 8th international workshop, DSV-IS 2001 Glasgow, Scotland, UK, June 13–15, 2001 revised papers. Edited by Johnson C. Berlin, Heidelberg: Springer Berlin Heidelberg; 2001: 183-198.
Lewis C, Darnell D, Kerns S, Monroe-DeVita M, Landes SJ, Lyon AR, Stanick C, Dorsey S, Locke J, Marriott B, et al. Proceedings of the 3rd Biennial Conference of the Society for Implementation Research Collaboration (SIRC) 2015: advancing efficient methodologies through community partnerships and team science. Implementation Sci. 2016;11:85.
The preparation of this manuscript was also supported, in kind, through the National Institutes of Health R13 award entitled, “Development and Dissemination of Rigorous Methods for Training and Implementation of Evidence-Based Behavioral Health Treatments” granted to PI: KA Comtois from 2010 to 2015.
Investigator time to conduct this research, including data collection, analysis, and interpretation and manuscript preparation, was funded by the National Institute of Mental Health of the National Institutes of Health award numbers R01MH106510, K08MH095939, UH3MH106338-03S1, and T32MH082709.
Availability of data and materials
The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request.
Ethics approval and consent to participate
Consent for publication
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
- Implementation science
- Translational research
- Evidence-based practice
- Behavioral health
- Mental health
- Physical health