Research | Open | Open Peer Review | Published:
Assessing citation networks for dissemination and implementation research frameworks
Implementation Sciencevolume 12, Article number: 97 (2017)
A recent review of frameworks used in dissemination and implementation (D&I) science described 61 judged to be related either to dissemination, implementation, or both. The current use of these frameworks and their contributions to D&I science more broadly has yet to be reviewed. For these reasons, our objective was to determine the role of these frameworks in the development of D&I science.
We used the Web of Science™ Core Collection and Google Scholar™ to conduct a citation network analysis for the key frameworks described in a recent systematic review of D&I frameworks (Am J Prev Med 43(3):337–350, 2012). From January to August 2016, we collected framework data including title, reference, publication year, and citations per year and conducted descriptive and main path network analyses to identify those most important in holding the current citation network for D&I frameworks together.
The source article contained 119 cited references, with 50 published articles and 11 documents identified as a primary framework reference. The average citations per year for the 61 frameworks reviewed ranged from 0.7 to 103.3 among articles published from 1985 to 2012. Citation rates from all frameworks are reported with citation network analyses for the framework review article and ten highly cited framework seed articles. The main path for the D&I framework citation network is presented.
We examined citation rates and the main paths through the citation network to delineate the current landscape of D&I framework research, and opportunities for advancing framework development and use. Dissemination and implementation researchers and practitioners may consider frequency of framework citation and our network findings when planning implementation efforts to build upon this foundation and promote systematic advances in D&I science.
The field of dissemination and implementation (D&I) science continues to evolve with contributions from a variety of disciplines, researchers, and institutions across the globe . Significant advances in our understanding of how to conceptualize D&I research and practice were facilitated by a recent comprehensive review of relevant models, theories, and frameworks . The review identified 61 frameworks to guide D&I researchers and practitioners in their research-to-practice activities at different socio-ecologic levels within the health care system (individual, organization, community, healthcare system, policy). The goal was to develop a D&I framework inventory to inform selection efforts for researchers and practitioners based on a given framework’s construct flexibility, its predilection for dissemination and/or implementation activities, as well as its socio-ecologic level targeting.
However, better understanding the most frequently cited D&I frameworks and the citation networks surrounding these frameworks can also provide useful information for selection, conceptualization, and resources for operationalization. For example, in cases where several different frameworks might be applicable to a given implementation intervention, identifying the most prominent and commonly applied frameworks in the field could have several advantages. First, it could provide researchers and practitioners with the most supporting literature to inform their effort. Second, accessing this information may increase the chances of intervention success and therefore help the best frameworks emerge. Third, as the framework literature evolves, there will be increasing opportunities to advance D&I science with respect to fidelity of framework use, core framework components, standardized measurement, advantages and disadvantages of a given framework, and ultimately implementation outcomes . More broadly, mapping D&I framework networks can build upon this foundation to promote systematic advances in D&I science through identifying the common set of assumptions and knowledge that constitutes consensus in the field.
Bibliometric (or citation) analysis is one method to investigate the scholarly landscape surrounding D&I frameworks from the review. This quantitative technique is increasingly applied to measure the impact of academic research and examine relationships using tools such as citation network analysis [4,5,6]. In general, citation network analysis provides a map of the most highly cited publications within a given research domain, much like the way Google™ uses page rank to identify the most relevant websites . This approach to understanding the state of scientific advancement has been used across a range of fields, including public administration, public health service systems, physical activity environments, and analytic method development, to discern the degree to which information flows through a scholarly network and identify opportunities for transdisciplinary collaboration and crosstalk [8,9,10,11,12,13,14]. Using citation analysis to examine the rapidly evolving D&I field could not only indicate the most frequently cited D&I frameworks but also determine their relationships across time and discipline, and map the emerging knowledge network constituting the D&I framework field.
For these reasons, we conducted a citation network analysis of D&I research frameworks. We created a snapshot of the scientific development of D&I framework research based on carefully selected framework articles followed forward in time as they integrated into the growing body of D&I knowledge. We examined citation rates and the main paths through the citation network to delineate the current landscape of D&I framework research, and opportunities for advancing framework development and use.
Citation network analysis
We used a citation data network collection tool, the Citation Network Analyzer (CNA), to generate the data and conduct our study [15, 16]. This tool uses a constrained snowball sampling approach to identify a network of documents (i.e., journal and conference papers, theses and dissertations, academic books, pre-prints, abstracts, technical reports) in Google Scholar™ that can be used for descriptive, main path, and other network analyses via an R software package. In general, a constrained snowball sample of academic publications is created by identifying seed articles, determining the levels of data (articles that cite the seeds, articles that cite those, and so on), and selecting the sampling rate at each level. This vetted, efficient and inclusive networking approach to following citations forward in time is uniquely suited to advance our current understanding of the literature surrounding D&I framework development and use. In addition, the output from the CNA tool can be used to graphically represent the citation network and assign weights to the articles based on their importance in maintaining the network architecture as described below.
Our approach of using citation network analysis to conduct structured literature reviews was based on prior work using the CNA tool [9, 10, 13, 15, 17]. This approach can lead to a less biased assessment of the academic literature than traditional narrative reviews for at least two reasons. First, a citation analysis approach can avoid the cognitive bias associated with traditional literature searches using keyword searches which may be limited by the researcher expertise, training, and preferences. Second, the use of Google Scholar™ and a snowball sampling technique based on selected seed articles, rather than Web of Science™ citation tools based on keywords for instance, is able to survey a broader scope of publications that may be relevant to D&I frameworks especially given their expansive roots in fields ranging from agriculture, business, and political science to public health and medicine [18, 19]. In addition, the CNA tool allows for a constrained approach to snowball sampling, rather than traditional snowball sampling where the sample grows exponentially, in order to limit the articles at each level from the seed article to arrive at empirical findings using a fraction of the data .
As detailed in Additional file 1 , we conducted two analyses using this novel approach. First, we synthesized the literature covered in the framework review article by Tabak et al.  with respect to recent citations and performed a structured literature review of the article itself. Next, we applied a structured literature review to a snowball sample of ten framework articles identified as the most important by the study team, largely based on the Tabak review. Overall, this work allowed us to understand the relevance of the framework review article as a D&I resource and to identify those frameworks forming the current backbone of the D&I framework field (i.e., framework articles in the network’s main path).
Characterizing the Tabak et al. framework review article and its citation network
The Tabak systematic review contained 119 references, with 50 published articles and 11 documents (reports/chapters/books) identified as a primary D&I framework reference (n = 61) . These D&I frameworks were identified first through selecting commonly cited frameworks, then through snowball sampling and expert consultation including with U.S. National Institute of Health offıcials who process and review D&I grants. Frameworks were excluded from the review according to the following criteria: (1) focused on practitioner rather than D&I researcher; (2) applied to individual behavior change only (i.e., without ties to local, organizational or community dissemination); (3) intended only for national level use versus local, community, or organizational level; (4) frameworks focused only on dissemination after research study completion; and (5) articles not written in the English language. The frameworks were then judged by the authors to be related either more to dissemination, implementation, or both equally. Each framework’s construct flexibility was rated as broad and flexible versus operational and defined for a given context and activity. Last, the socio-ecologic level (individual, organization, community, healthcare system, policy) targeted by the framework was categorized, with most operating at more than one level.
We extracted the primary citation for each framework. In cases where more than one primary reference was used (n = 21), we selected the most relevant reference, usually the oldest, as the primary reference. The primary references for 11 frameworks were reports, chapters, or books. Because peer-reviewed articles were the most common documents cited in this study, we use the term article to denote all documents throughout the remainder of the manuscript.
To better understand the framework articles discussed in the Tabak review, we conducted descriptive analyses to identify the most common journals, authors, and countries of origin for the 61 models. We also examined the citation rates for each framework. We defined a citation rate as the number of citations/year(s) since publication. We used the Web of Science™ Core Collection in January 2016 to conduct these descriptive citation analyses and inform our subsequent network analysis described in the Additional file 1 .
Citation network analysis of selected D&I frameworks
Next, we conducted a citation network analysis of ten carefully selected D&I framework articles we felt reflected the current state of the field. Eight of these were based on citation rates and the Tabak review. However, we also included two additional frameworks given their relevance to implementation science and relatively high citation rates: (1) Theoretical Domains Framework (TDF) [20, 21] and the (2) Knowledge to Action Framework (KTA) , for a total of ten seed articles for our next citation network analysis. Both of these models were developed by researchers outside the USA and were not included in the Tabak review. The details of the D&I framework citation analysis are included in the Additional file 1 .
Last, we performed a main path analysis to identify the connectedness and links among the articles considered to be the backbone of the D&I framework citation network. This approach identifies the key articles influencing D&I models based on the selected seed articles. We determined the traversal weights indicating the proportion of network paths that included a given article node in the network . For instance, a traversal weight of 0.25 for framework X indicates that its article exists in 25% of the citation paths in the network. This traversal weight indicates the importance of any particular node (i.e., article) in the network. We constructed the main path by removing all ties in the network scoring below the 95% percentile for traversal weight value. We normalized the traversal weights according to flow using the Search Path Count method . All computations were accomplished with Pajek .
All analyses were conducted between January 2016 and August 2016. This study was deemed not regulated by the Institutional Review Board at the University of Michigan.
Tabak framework review article and its citation network
As illustrated in Fig. 1, the Tabak framework review article is an increasingly cited resource. As of January 2016, it had been cumulatively cited 456 times across 388 articles and other source items indexed within Web of Science™ Core Collection. As shown in Table 1, there was a broad distribution of citation numbers and annual citation rates across the 61 framework articles within the Tabak review and our two selected framework articles (KTA and TDF). The average number of citations per year ranged from 0 to 1949 among articles published from 1962 to 2012. The outlier with the highest citation rate was a book reference for Rogers’ Diffusion of Innovations.
Based on the structured literature review of the Tabak article using the CNA tool, we identified 239 articles across the network and its three levels of ‘distance.’ This included 17 level-one articles directly referencing the Tabak article, with the remainder of articles residing two and three levels from the Tabak source article. The majority of the documents were journal articles (84%), followed by books (16%). The articles in the Tabak network were published between 2002 and 2016, with 51 articles published prior to the source article year of 2012. The majority (86%) of these were three levels from the Tabak seed article and (35%) were book references. We identified 202 unique first authors contributing to this network. Each author contributed 1.18 articles (standard deviation (SD) = 0.58), on average. Most first authors contributed only one article to the network (one = 177; two = 19, three = 3, four = 2, six = 1). We identified 123 unique journals (books excluded) contributing to the Tabak network, each providing an average of 1.62 articles (SD = 2.63). Most journals contributed one article (n = 95). The top three journals producing the most articles were: Implementation Science (n = 29), Annual Review of Public Health (n = 6), and BMC Public Health (n = 5). All other journals had four or fewer articles each. The articles in the Tabak network were cited between 0 and 4410 times. The top ten cited articles in the Tabak network are shown in Table 2, and none of which served as a primary framework reference. As illustrated in Fig. 2, there were prominent ties in the Tabak network to social care and the law by Aveyard; normalization process and general implementation theory by May; implementation work by Glasgow, Proctor, Neta, and Chambers; a gateway to broader literature via a movement science article by Peters; a Karlin article which ties in psychotherapy; and a 2013 contribution by Straus that was an introduction to knowledge translation in healthcare.
Citation network analysis of selected D&I frameworks
The citation network for our seed articles highlighted in Table 1 included 355 unique documents published between 1996 and 2014. There were 302,472 citation links connecting the articles in this network. The majority of citations was from 323 journal articles (91%), followed by 29 books (8%), and 3 in-proceedings (1%). We identified 274 unique first authors, each contributing 1.30 articles (SD = 0.84), on average. The majority of first authors provided one article to the network with only six authors contributing greater than three. We also identified 128 unique journals contributing to this network, each providing an average of 2.52 articles (SD = 4.04). While many journals contributed one article (n = 29), the top five journals producing the most articles were: Strategic Management Journal (n = 29), Academy of Management Journal (n = 25), Implementation Science (n = 20), Organization Science (n = 15), and Management Science (n = 10). All other journals contributed less than ten articles each. The top ten cited articles are shown in Table 3, with Szulanski’s Sticky Knowledge as the only primary framework reference from the Tabak review. The remainder of articles tended to focus on business practices and knowledge sharing, collaboration networks, and social and/or intellectual capital. The articles for the D&I framework network contributed between 64 and 12,680 citations, with a median of 489.
As illustrated in Fig. 3, the D&I framework citation network appears centered around the 2004 Greenhalgh et al. article with prominent ties to the Theoretical Domains Framework, the Knowledge to Action Framework, the Promoting Action on Research Implementation in Health Services Framework (PARiHS), the Consolidated Framework for Implementation Research (CFIR), and an article conceptualizing implementation outcomes, among others. A more complete picture of the network’s primary core is offered with the main path analysis, which consists of those ties above the 95% percentile score for traversal weight (0.0106). The main path, illustrated in Fig. 4, is comprised of the 15 articles listed in Table 4. A simple interpretation of the main path is that these articles are most important in holding the entire D&I framework citation network together. In this case, seven of the ten D&I framework seed articles are part of the main path, along with eight non-seed articles. Visually, one can inspect the main path and observe the chronological flow of influence from earlier to more recent publications. Kitson  and Klein  act as the primary originating sources of influence in the main path, which serve to influence Greenhalgh , Damschroder , and Proctor . These five articles, along with Glasgow , all converge in Aarons , which acts as a major hub for the remainder of the more recent works on the periphery of the main path.
Using citation analysis, we identified the most frequently cited D&I frameworks and their relationships across time and discipline and mapped the knowledge network constituting the D&I framework field. We discovered that the Tabak framework review has been increasingly cited and that it was included in the periphery of the main D&I framework network path indicating its value as a recognized resource for D&I researchers and practitioners. We identified the leading journals and authors contributing to the D&I framework literature using methods that limit cognitive biases associated with traditional literature searches using keywords. Using the CNA tool to conduct our structured literature review, we were able to identify the main path articles that signify those most important in holding the entire D&I framework citation network together. Overall, D&I researchers and practitioners may consider frequency of citation and this network structure when planning implementation efforts to build upon this foundation and promote systematic advances in D&I science. Further work is necessary to delineate how these frameworks are being used in the literature, framework selection criteria for planning D&I research efforts, the core components of these frameworks, and how framework use relates to improved implementation outcomes .
This study provides insight into at least two aspects of the evolving D&I scientific field. First, it confirms that D&I research has witnessed a surge of frameworks with most developed in the last two decades . However, we found that the majority of articles were rarely cited, leaving only a few highly cited frameworks. It is difficult to know whether more recent frameworks will be used or not based on this analysis though several recent articles, including the Tabak review, were highly cited. Nonetheless, there does appear to be framework saturation creating an increasing need to delve further into better understanding the current cadre rather than creating new D&I frameworks. Second, taking into consideration citation rates and this network structure may be a key factor to consider when choosing a framework, in addition to the socioecological level, construct flexibility, and location on the D&I spectrum. For example, increasing citations and centrality in the network indicates more literature is available to highlight the advantages and disadvantages of using a given framework. In addition, there may be more operational and measurement resources with increasing centrality. Taking these additional aspects into consideration creates opportunities to scrutinize frameworks, starting with those in the main path, and advance D&I science by examining issues of fidelity, core and adaptive components, measurement, and relationships to implementation outcomes .
We found a broad range of scientific fields contributing to the D&I citation network given our use of Google Scholar™ and its extensive search capabilities [7, 19]. This reinforces the need to scan literature outside of health-related fields to discover new guidance for D&I sciences. For example, other than the specialized journal Implementation Science, which focuses specifically on the field, most citations of the Tabak framework review article were from public health journals due in part to it being a narrative review that used snowball sampling methods and focused on health. In addition, the journals other than Implementation Science, which published the highest number of citations in the broader D&I framework network, were all in the management and business fields. This is consistent with a prior review of leading management journals that found a significant degree of knowledge translation and organizational change literature relevant to D&I in healthcare . While there is some current cross-over among these fields, they are often quite distinct and separate from each other when it comes to research and practice. Taken together, our findings suggest that greater efforts to scan across these journals and fields could provide unique transdisciplinary collaborations and innovation opportunities to hasten D&I research and practice. For that matter, D&I advances could also serve to improve management and business practices.
However, citing a framework does not imply use or specify what its application entails. How to operationalize determinants of practice across frameworks also needs to be better understood to advance D&I science. A recent study examined use of the KTA framework using citation analysis and systematic review to see if the framework was used in practice and how . The authors found that it was used with varying degrees of completeness from a simple reference to integration into the design, delivery, and evaluation of the implementation activities. The latter contributing most to advancing D&I science and generalizability of outcomes. Similarly, another recent systematic review examined use of the CFIR among empirical studies in the peer-reviewed literature . Twenty-six articles met inclusion criteria across a breadth of settings and units of analysis. Justification for which CFIR constructs were selected, integration throughout the research study, and relation to outcomes remained poorly articulated, again limiting contributions to D&I research more broadly. Furthermore, systematic efforts to reconcile determinants of healthcare professional practice across 12 different frameworks have generated practical checklists and implementation strategy recommendations to support implementation and quality improvement efforts . Better understanding framework use, consolidation and operationalization of framework determinants, not just citations, could yield more to consider when selecting and using D&I frameworks for research and practice.
There are several limitations to our study approach. First, framework citation rates are influenced by a multitude of factors including journal impact factor, the authors’ fame and publication rate, the degree of research in a given field, whether citation is perceived as positive or negative, and do not necessarily indicate the quality of a given publication or framework [5,6,7, 19]. Nonetheless, citation rates do serve as an approximation of the impact of a scholarly work. We also used an expert-led review article for seed article identification and a robust network analysis tool, coupled with citation rate data, to provide our snapshot of the scientific development of the D&I framework field with substantial face validity. Second, there could be issues with respect to language and the definition of D&I research leading to ascertainment bias. Using our comprehensive CNA approach in Google Scholar™, rather than keyword searches for example, actually created a broader scope for our study. Last, whether the use of highly cited documents (e.g., textbooks) as seed articles, rather than the journal articles selected as seeds in our study, would dramatically change our findings is unclear. Our network tool was inclusive of such documents although they were the minority of articles in both network analyses. Indeed, publishing frameworks outside of journal articles creates challenges, both in terms of physically obtaining the material and being able to grasp the conceptual and operational components dispersed throughout a given textbook. Perhaps corresponding peer-reviewed articles serving as a book review, preferably in open-access formats to improve dissemination, could help mitigate access and citation issues .
In conclusion, bibliometric analysis is one way to understand how D&I frameworks are used in the development of D&I science. We used a bibliometric citation analysis tool to help identify the most prevalent models influencing D&I. D&I researchers and practitioners may consider frequency of citation and this network structure when planning implementation efforts to build upon this foundation and promote systematic advances in D&I science.
Consolidated Framework for Implementation Research
Citation Network Analyzer
Dissemination and implementation
Knowledge to Action Framework
Promoting Action on Research Implementation in Health Services Framework
Theoretical Domains Framework
Brownson R, Colditz G, Proctor E, editors. Dissemination and implementation research in health: translating science to practice. New York: Oxford University Press; 2012.
Tabak RG, Khoong EC, Chambers DA, Brownson RC. Bridging research and practice: models for dissemination and implementation research. Am J Prev Med. 2012;43(3):337–50.
Chaudoir SR, Dugan AG, Barr CH. Measuring factors affecting implementation of health innovations: a systematic review of structural, organizational, provider, patient, and innovation level measures. Implement Sci. 2013;8:22.
Marx W, Bornmann L. Change of perspective: bibliometrics from the point of view of cited references-a literature overview on approaches to the evaluation of cited references in bibliometrics. Scientometrics. 2016;109(2):1397–415.
Bornmann L, Marx W. How good is research really? Measuring the citation impact of publications with percentiles increases correct assessments and fair comparisons. EMBO Rep. 2013;14(3):226–30.
Garfield E. The history and meaning of the journal impact factor. JAMA. 2006;295(1):90–3.
Bakkalbasi N, Bauer K, Glover J, Wang L. Three options for citation tracking: Google Scholar, Scopus and Web of Science. Biomed Digit Libr. 2006;3:7.
Harris JK. Connecting discovery and delivery: the need for more evidence on effective smoking cessation strategies for people living with HIV/AIDS. Am J Public Health. 2010;100(7):1245–9.
Harris JK, Beatty KE, Lecy JD, Cyr JM, Shapiro RM 2nd. Mapping the multidisciplinary field of public health services and systems research. Am J Prev Med. 2011;41(1):105–11.
Harris JK, Lecy J, Hipp JA, Brownson RC, Parra DC. Mapping the development of research on physical activity and the built environment. Prev Med. 2013;57(5):533–40.
Harris JK, Luke DA, Zuckerman RB, Shelton SC. Forty years of secondhand smoke research: the gap between discovery and delivery. Am J Prev Med. 2009;36(6):538–48.
Lecy J, Schmitz HP, Swedlund H. NGO and NPO effectiveness: a modern synthesis. Voluntas. 2012;23(2):434–57.
Shadish WR, Lecy JD. The meta-analytic big bang. Res Synth Methods. 2015;6(3):246–64.
Harper G, Peattie K. Tracking the influence of the first special journal issue on ‘Green Marketing’: a citation network analysis. Soc Bus. 2011;1(3):239–61.
Jesse D. Lecy and Kate E. Beatty, Representative literature reviews using constrained snowball sampling and citation network analysis 2012. Available at SSRN: https://ssrn.com/abstract=1992601 or http://dx.doi.org/10.2139/ssrn.1992601.
Jesse Lecy and Diego Moreda (2013). “cna: citation network analyzer.” R package version 0.3-4.
Lecy JD, Mergel IA, Schmitz HP. Networks in public administration: current scholarship in review. Public Manag Rev. 2014;16(5):643–65.
Noruzi A. Google scholar: the new generation of citation indexes. Libri:Int J Libr Inform Serv. 2005;55(4):170–80.
Maslov S, Redner S. Promise and pitfalls of extending Google’s PageRank algorithm to citation networks. J Neurosci. 2008;28(44):11103–5.
Cane J, O'Connor D, Michie S. Validation of the theoretical domains framework for use in behaviour change and implementation research. Implement Sci. 2012;7:37.
Francis JJ1, O'Connor D, Curran J. Theories of behaviour change synthesised into a set of theoretical groupings: introducing a thematic series on the theoretical domains framework. Implement Sci. 2012;7:35.
Graham ID, Logan J, Harrison MB, Straus SE, Tetroe J, Caswell W, Robinson N. Lost in knowledge translation: time for a map? J Contin Educ Heal Prof. 2006;26(1):13–24.
Wd N, Mrvar A, Batagelj V. Exploratory social network analysis with Pajek. New York: Cambridge University Press; 2005.
Batagelj V, Doreian P, Ferligoj A, Kejzar N. Understanding Large Temporal Networks and Spatial Networks: Exploration, Pattern Searching, Visualization and Network Evolution. West Sussex: Wiley; 2014.
Kitson A, Harvey G, McCormack B. Enabling the implementation of evidence based practice: a conceptual framework. Qual Health Care. 1998;7(3):149–58.
Klein KJ, Sorra JS. The challenge of innovation implementation. Acad Manag Rev. 1996;21(4):1055–80.
Greenhalgh T, Robert G, Macfarlane F, Bate P, Kyriakidou O. Diffusion of innovations in service organizations: systematic review and recommendations. Milbank Q. 2004;82(4):581–629.
Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci. 2009;4
Proctor EK, Landsverk J, Aarons G, Chambers D, Glisson C, Mittman B. Implementation research in mental health services: an emerging science with conceptual, methodological, and training challenges. Adm Policy Ment Health. 2009;36(1):24–34.
Glasgow RE, Vogt TM, Boles SM. Evaluating the public health impact of health promotion interventions: the RE-AIM framework. Am J Public Health. 1999;89(9):1322–7.
Aarons GA, Hurlburt M, Horwitz SM. Advancing a conceptual model of evidence-based practice implementation in public service sectors. Adm Policy Ment Health. 2011;38(1):4–23.
Harlos K, Tetroe J, Graham ID, Bird M, Robinson N. Mining the management literature for insights into implementing evidence-based change in healthcare. Health Policy. 2012;8(1):33–48.
Field B, Booth A, Ilott I, Gerrish K. Using the knowledge to action framework in practice: a citation analysis and systematic review. Implement Sci. 2014;9:172.
Kirk MA, Kelley C, Yankey N, Birken SA, Abadie B, Damschroder L. A systematic review of the use of the consolidated framework for implementation research. Implement Sci. 2016;11:72.
Flottorp SA, Oxman AD, Krause J, Musila NR, Wensing M, Godycki-Cwirko M, Baker R, Eccles MP. A checklist for identifying determinants of practice: a systematic review and synthesis of frameworks and taxonomies of factors that prevent or enable improvements in healthcare professional practice. Implement Sci. 2013;8:35.
Baetens F, Prislan V. The dissemination of international scholarship: the future of books and book reviews. Leiden J Int Law. 2014;27(3):559–69.
Szulanski G. Exploring internal stickiness: impediments to the transfer of best practice within the firm. Strateg Manag J. 1996;17:27–43.
Michie S, Johnston M, Abraham C, Lawton R, Parker D, Walker A, Grp PT. Making psychological theory useful for implementing evidence based practice: a consensus approach. Qual Saf Health Care. 2005;14(1):26–33.
Lavis JN, Robertson D, Woodside JM, McLeod CB, Abelson J, Grp KTS. How can research organizations more effectively transfer research knowledge to decision makers? Milbank Q. 2003;81(2):221-+.
Wandersman A, Duffy J, Flaspohler P, Noonan R, Lubell K, Stillman L, Blachman M, Dunville R, Saul J. Bridging the gap between prevention research and practice: the interactive systems framework for dissemination and implementation. Am J Community Psychol. 2008;41(3–4):171–81.
Green LW, Ottoson JM, Garcia C, Hiatt RA. Diffusion theory and knowledge dissemination, utilization, and integration in public health. Annu Rev Public Health. 2009;30:151–74.
May C, Finch T. Implementing, embedding, and integrating practices: an outline of normalization process theory. Sociology. 2009;43(3):535–54.
Frambach RT, Schillewaert N. Organizational innovation adoption—a multi-level framework of determinants and opportunities for future research. J Bus Res. 2002;55(2):163–76.
Pathman DE, Konrad TR, Freed GL, Freeman VA, Koch GG. The awareness-to-adherence model of the steps to clinical guideline compliance—the case of pediatric vaccine recommendations. Med Care. 1996;34(9):873–89.
Pronovost P, Berenholtz S, Needham D. Translating evidence into practice: a model for large scale knowledge translation. BMJ. 2008;337:a1714
Ward V, House A, Hamer S. Developing a framework for transferring knowledge into action: a thematic analysis of the literature. J Health Serv Res Policy. 2009;14(3):156–64.
Mendel P, Meredith LS, Schoenbaum M, Sherbourne CD, Wells KB. Interventions in organizational and community context: a framework for building evidence on dissemination and implementation in health services research. Adm Policy Ment Health. 2008;35(1–2):21–37.
Atun R, de Jongh T, Secci F, Ohiri K, Adeyi O. Integration of targeted health interventions into health systems: a conceptual framework for analysis. Health Policy Plan. 2010;25(2):104–11.
Bowen S, Zwi AB. Pathways to “evidence-informed” policy and practice: a framework for action. PLoS Med. 2005;2(7):600–5.
Glisson C, Schoenwald SK. The ARC organizational and community intervention strategy for implementing evidence-based children’s mental health treatments. Ment Health Serv Res. 2005;7(4):243–59.
Feldstein AC, Glasgow RE. A practical, robust implementation and sustainability model (PRISM) for integrating research findings into practice. Jt Comm J Qual Patient Saf. 2008;34(4):228–43.
Weiner BJ, Lewis MA, Linnan LA. Using organization theory to understand the determinants of effective implementation of worksite health promotion programs. Health Educ Res. 2009;24(2):292–305.
Logan J, Graham ID. Toward a comprehensive interdisciplinary model of health care research use. Sci Commun. 1998;20(2):227–46.
Owen N, Glanz K, Sallis JF, Kelder SH. Evidence-based approaches to dissemination and diffusion of physical activity interventions. Am J Prev Med. 2006;31(4):S35–44.
Kilbourne AM, Neumann MS, Pincus HA, Bauer MS, Stall R. Implementing evidence-based interventions in health care: application of the replicating effective programs framework. Implement Sci. 2007;2
Jacobson N, Butterill D, Goering P. Development of a framework for knowledge translation: understanding user context. J Health Serv Res Policy. 2003;8(2):94–9.
Baumbusch JL, Kirkham SR, Khan KB, McDonald H, Semeniuk P, Tan E, Anderson JM. Pursuing common agendas: a collaborative model for knowledge translation between research and practice in clinical settings. Res Nurs Health. 2008;31(2):130–40.
Dearing JW, Maibach EW, Buller DB. A convergent diffusion and social marketing approach for disseminating proven approaches to physical activity promotion. Am J Prev Med. 2006;31(4):S11–23.
Dobbins M, Ciliska D, Cockerill R, Burnsley J, DiCenso A. A framework for the dissemination and utilization of research for health-care policy and practice. Online J Knowl Synth Nurs. 2002;9(7).
Green LW, Orleans CT, Ottoson JM, Cameron R, Pierce JP, Bettinghaus EP. Inferring strategies for disseminating physical activity policies, programs, and practices from the successes of tobacco control. Am J Prev Med. 2006;31(4 Suppl):S66–81.
Kontos PC, Poland BD. Mapping new theoretical and methodological terrain for knowledge translation: contributions from critical realism and the arts. Implement Sci. 2009;4:1.
Lomas J. Retailing research: increasing the role of evidence in clinical services for childbirth. Milbank Q. 1993;71(3):439–75.
Majdzadeh R, Sadighi J, Nejat S, Mahani AS, Gholami J. Knowledge translation for research utilization: design of a knowledge translation model at Tehran University of Medical Sciences. J Contin Educ Heal Prof. 2008;28(4):270–7.
Dreisinger ML, Boland EM, Filler CD, Baker EA, Hessel AS, Brownson RC. Contextual factors influencing readiness for dissemination of obesity prevention programs and policies. Health Educ Res. 2012;27(2):292–306.
Martin GW, Herie MA, Turner BJ, Cunningham JA. A social marketing model for disseminating research-based treatments to addictions treatment providers. Addiction. 1998;93(11):1703–15.
Elliott SJ, O'Loughlin J, Robinson K, Eyles J, Cameron R, Harvey D, Raine K, Gelskey D, Canadian Heart Health Dissemination Project S, Research Advisory G. Conceptualizing dissemination research and activity: the case of the Canadian heart health initiative. Health Educ Behav. 2003;30(3):267–82. discussion 283-266
Kramer DM, Cole DC. Sustained, intensive engagement to promote health and safety knowledge transfer to and utilization by workplaces. Sci Commun. 2003;25(1):56–82.
Farkas M, Jette AM, Tennstedt S, Haley SM, Quinn V. Knowledge dissemination and utilization in gerontology: an organizing framework. Gerontologist. 2003;43:47–56.
Robinson K, Elliott SJ, Driedger SM, Eyles J, O'Loughlin J, Riley B, Cameron R, Harvey D, Grp CSRA. Using linking systems to build capacity and enhance dissemination in heart health promotion: a Canadian multiple-case study. Health Educ Res. 2005;20(5):499–513.
Yuan CT, Nembhard IM, Stern AF, Brush JE Jr, Krumholz HM, Bradley EH. Blueprint for the dissemination of evidence-based practices in health care. Issue Brief (Commonw Fund). 2010;86:1–16.
Harris JR, Cheadle A, Hannon PA, Forehand M, Lichiello P, Mahoney E, Snyder S, Yarrow J. A framework for disseminating evidence-based health promotion practices. Prev Chronic Dis. 2012;9:E22.
Nolan K, Schall MW, Erb F, Nolan T. Using a framework for spread: the case of patient access in the veterans health administration. Jt Comm J Qual Patient Saf. 2005;31(6):339–47.
Anderson M, Cosby J, Swan B, Moore H, Broekhoven M. The use of research in local health service agencies. Soc Sci Med. 1999;49(8):1007–19.
Bauman AE, Nelson DE, Pratt M, Matsudo V, Schoeppe S. Dissemination of physical activity evidence, programs, policies, and surveillance in the international public health arena. Am J Prev Med. 2006;31(4):S57–65.
Sogolow ED, Kay LS, Doll LS, Neumann MS, Mezoff JS, Eke AN, Semaan S, Anderson JR. Strengthening HIV prevention: application of a research-to-practice framework. AIDS Educ Prev. 2000;12(5):21–32.
Orlandi MA. Health promotion technology transfer: organizational perspectives. Can J Public Health. 1996;87:S28–33.
Winkler JD, Lohr KN, Brook RH. Persuasive communication and medical technology-assessment. Arch Intern Med. 1985;145(2):314–7.
Lester JP. The utilization of policy analysis by state agency officials. Knowl. 1993;14(3):267–90.
Funk SG, Tornquist EM, Champagne MT. A model for improving the dissemination of nursing research. West J Nurs Res. 1989;11(3):361–72.
Scullion PA. Effective dissemination strategies. Nurs Res. 2002;10(1):65–77.
Rogers EM. Diffusion of innovations. New York: Free Press of Glencoe; 1962.
Kingdon JW. Agendas, alternatives, and public policies. Boston: Little, Brown; 1984.
Fixsen DL, Naoom SF, Blasé KA, Friedman RM, Wallace F. Implementation research: a synthesis of the literature. Tampa: University of South Florida, Louis de la Parte Florida Mental Health Institute, The National Implementation Research Network; 2005.
Green LW, Kreuter MW, Green LW. Health program planning: an educational and ecological approach. 4th ed. New York: McGraw-Hill; 2005.
Havelock RG. Planning for innovation through dissemination and utilization of knowledge. Centre for Research on utilization of Scientifıc knowledge, Institute for Social Research, University of Michigan, 1969.
Pettigrew AM, Ferlie E, McKee L. Shaping strategic change: making change in large organizations: the case of the National Health Service. Thousand Oaks: Sage Publications; 1992.
Nieva VF, Murphy R, Ridley N, et al. From science to service: a framework for the transfer of patient safety. In: Henriksen K, Battles JB, Marks ES, Lewin DI, editors. Advances in patient safety: from research to implementation (Vol. 2: concepts and methodology). Agency for Healthcare Research and Quality: Rockville; 2005.
Dodson EA, Brownson RC, Weiss SW. Policy dissemination research. In: Brownson R, Colditz G, Proctor EK, editors. Dissemination and implementation research in health: translating science to practice. Oxford: Oxford University Press; 2012.
Kreuter MW, Casey CM, Bernhardt JM. Enhancing dissemination though marketing and distribution systems: a vision for public health. In: Brownson RC, Colditz G, Proctor EK, editors. Dissemination and implementation research in health: translating science to practice. New York: Oxford University Press; 2012.
Damush T, Bravata DM, Plue L, Woodward-Hagg H, Williams LS. Facilitation of best practices (FAB) framework. Stroke QUERI Center annual report. 2008.
TIDIRH Working Group. Interacting elements of integrating science, policy, and practice. In: Training institute for dissemination and implementation research in health. Chapel Hill: Conference proceedings; 2011.
Bell J, Waters S. Doing Your Research Project: A Guide for First-Time Researchers. England: McGraw-Hill Education; 2014.
Gratton C, Jones I. Research Methods for Sports Studies. London: Routledge; 2010.
Hanney SR, Gonzalez-Block MA, Buxton MJ, Kogan M. The utilisation of health research in policy-making: concepts, examples and methods of assessment. Health Res Policy Syst. 2003;1(1):2.
Hersh W. Information retrieval: a health and biomedical perspective. New York: Springer; 2008.
Dalrymple J, Burke B. Anti-oppressive practice: social care and the law. England: McGraw-Hill Companies, Incorporated; 2006.
Aveyard H. Doing a literature review in health and social care: a practical guide. England: McGraw-Hill Education; 2014.
Trevithick P. Social work skills: a practice handbook. England: Open University Press; 2005.
Thompson N. Understanding social work: preparing for practice. England: Palgrave Macmillan; 2015.
D'Cruz H, Gillingham P, Melendez S. Reflexivity, its meanings and relevance for social work: a critical review of the literature. Brit J Soc Work. 2007;37(1):73–90.
Dagenais S, Tricco AC, Haldeman S. Synthesis of recommendations for the assessment and management of low back pain from recent clinical practice guidelines. Spine J. 2010;10(6):514–29.
Nahapiet J, Ghoshal S. Social capital, intellectual capital, and the organizational advantage. Acad Manag Rev. 1998;23(2):242–66.
Eisenhardt KM, Martin JA. Dynamic capabilities: what are they? Strateg Manag J. 2000;21(10–11):1105–21.
Dyer JH, Singh H. The relational view: cooperative strategy and sources of interorganizational competitive advantage. Acad Manag Rev. 1998;23(4):660–79.
Wenger E, McDermott RA, Snyder W. Cultivating communities of practice: a guide to managing knowledge. Boston: Harvard Business School Press; 2002.
Alavi M, Leidner DE. Review: knowledge management and knowledge management systems: conceptual foundations and research issues. Mis Quart. 2001;25(1):107–36.
Zahra SA, George G. Absorptive capacity: a review, reconceptualization, and extension. Acad Manag Rev. 2002;27(2):185–203.
Hansen MT. The search-transfer problem: the role of weak ties in sharing knowledge across organization subunits. Admin Sci Quart. 1999;44(1):82–111.
Ahuja G. Collaboration networks, structural holes, and innovation: a longitudinal study. Admin Sci Quart. 2000;45(3):425–55.
Dyer JH, Nobeoka K. Creating and managing a high-performance knowledge-sharing network: the Toyota case. Strateg Manag J. 2000;21(3):345–67.
Stirman SW, Kimberly J, Cook N, Calloway A, Castro F, Charns M. The sustainability of new programs and innovations: a review of the empirical literature and recommendations for future research. Implement Sci. 2012;7
Meyers DC, Durlak JA, Wandersman A. The quality implementation framework: a synthesis of critical steps in the implementation process. Am J Community Psychol. 2012;50(3–4):462–80.
Schoenwald SK, Garland AF, Chapman JE, Frazier SL, Sheidow AJ, Southam-Gerow MA. Toward the effective and efficient measurement of implementation fidelity. Admin Pol Ment Health. 2011;38(1):32–43.
Palinkas LA, Aarons GA, Horwitz S, Chamberlain P, Hurlburt M, Landsverk J. Mixed method designs in implementation research. Adm Policy Ment Health. 2011;38(1):44–53.
Sanders MR. Development, evaluation, and multinational dissemination of the triple P-positive parenting program. Annu Rev Clin Psychol. 2012;8:345–79.
Aarons GA, Glisson C, Green PD, Hoagwood K, Kelleher KJ, Landsverk JA, Hlth RNYM. The organizational social context of mental health services and clinician attitudes toward evidence-based practice: a United States national study. Implement Sci. 2012;7
Ryan Blake, BS, for administrative and data collection support.
Dr. Skolarus was supported by a VA HSR&D Career Development Award-2 (CDA 12–171) and the Mentored Training for Dissemination and Implementation Research in Cancer (MT-DIRC) Program, National Cancer Institute, 1 R25 CA171994-01A1 during this study. This study did not receive any dedicated funding.
Availability of data and materials
Data for this project is stored on secure servers. Data can be made publicly available upon request.
Ethics approval and consent to participate
This study was deemed not regulated by the Institutional Review Board at the University of Michigan.
Consent for publication
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Citation Network Analysis Methods. (DOCX 14 kb)
About this article
- Network analysis
- Knowledge translation
- Management science
- Implementation science
- Quality improvement
- Behavioral theory