Skip to main content

Assessing citation networks for dissemination and implementation research frameworks

Abstract

Background

A recent review of frameworks used in dissemination and implementation (D&I) science described 61 judged to be related either to dissemination, implementation, or both. The current use of these frameworks and their contributions to D&I science more broadly has yet to be reviewed. For these reasons, our objective was to determine the role of these frameworks in the development of D&I science.

Methods

We used the Web of Science™ Core Collection and Google Scholar™ to conduct a citation network analysis for the key frameworks described in a recent systematic review of D&I frameworks (Am J Prev Med 43(3):337–350, 2012). From January to August 2016, we collected framework data including title, reference, publication year, and citations per year and conducted descriptive and main path network analyses to identify those most important in holding the current citation network for D&I frameworks together.

Results

The source article contained 119 cited references, with 50 published articles and 11 documents identified as a primary framework reference. The average citations per year for the 61 frameworks reviewed ranged from 0.7 to 103.3 among articles published from 1985 to 2012. Citation rates from all frameworks are reported with citation network analyses for the framework review article and ten highly cited framework seed articles. The main path for the D&I framework citation network is presented.

Conclusions

We examined citation rates and the main paths through the citation network to delineate the current landscape of D&I framework research, and opportunities for advancing framework development and use. Dissemination and implementation researchers and practitioners may consider frequency of framework citation and our network findings when planning implementation efforts to build upon this foundation and promote systematic advances in D&I science.

Peer Review reports

Background

The field of dissemination and implementation (D&I) science continues to evolve with contributions from a variety of disciplines, researchers, and institutions across the globe [1]. Significant advances in our understanding of how to conceptualize D&I research and practice were facilitated by a recent comprehensive review of relevant models, theories, and frameworks [2]. The review identified 61 frameworks to guide D&I researchers and practitioners in their research-to-practice activities at different socio-ecologic levels within the health care system (individual, organization, community, healthcare system, policy). The goal was to develop a D&I framework inventory to inform selection efforts for researchers and practitioners based on a given framework’s construct flexibility, its predilection for dissemination and/or implementation activities, as well as its socio-ecologic level targeting.

However, better understanding the most frequently cited D&I frameworks and the citation networks surrounding these frameworks can also provide useful information for selection, conceptualization, and resources for operationalization. For example, in cases where several different frameworks might be applicable to a given implementation intervention, identifying the most prominent and commonly applied frameworks in the field could have several advantages. First, it could provide researchers and practitioners with the most supporting literature to inform their effort. Second, accessing this information may increase the chances of intervention success and therefore help the best frameworks emerge. Third, as the framework literature evolves, there will be increasing opportunities to advance D&I science with respect to fidelity of framework use, core framework components, standardized measurement, advantages and disadvantages of a given framework, and ultimately implementation outcomes [3]. More broadly, mapping D&I framework networks can build upon this foundation to promote systematic advances in D&I science through identifying the common set of assumptions and knowledge that constitutes consensus in the field.

Bibliometric (or citation) analysis is one method to investigate the scholarly landscape surrounding D&I frameworks from the review. This quantitative technique is increasingly applied to measure the impact of academic research and examine relationships using tools such as citation network analysis [4,5,6]. In general, citation network analysis provides a map of the most highly cited publications within a given research domain, much like the way Google™ uses page rank to identify the most relevant websites [7]. This approach to understanding the state of scientific advancement has been used across a range of fields, including public administration, public health service systems, physical activity environments, and analytic method development, to discern the degree to which information flows through a scholarly network and identify opportunities for transdisciplinary collaboration and crosstalk [8,9,10,11,12,13,14]. Using citation analysis to examine the rapidly evolving D&I field could not only indicate the most frequently cited D&I frameworks but also determine their relationships across time and discipline, and map the emerging knowledge network constituting the D&I framework field.

For these reasons, we conducted a citation network analysis of D&I research frameworks. We created a snapshot of the scientific development of D&I framework research based on carefully selected framework articles followed forward in time as they integrated into the growing body of D&I knowledge. We examined citation rates and the main paths through the citation network to delineate the current landscape of D&I framework research, and opportunities for advancing framework development and use.

Methods

Citation network analysis

We used a citation data network collection tool, the Citation Network Analyzer (CNA), to generate the data and conduct our study [15, 16]. This tool uses a constrained snowball sampling approach to identify a network of documents (i.e., journal and conference papers, theses and dissertations, academic books, pre-prints, abstracts, technical reports) in Google Scholar™ that can be used for descriptive, main path, and other network analyses via an R software package. In general, a constrained snowball sample of academic publications is created by identifying seed articles, determining the levels of data (articles that cite the seeds, articles that cite those, and so on), and selecting the sampling rate at each level. This vetted, efficient and inclusive networking approach to following citations forward in time is uniquely suited to advance our current understanding of the literature surrounding D&I framework development and use. In addition, the output from the CNA tool can be used to graphically represent the citation network and assign weights to the articles based on their importance in maintaining the network architecture as described below.

Our approach of using citation network analysis to conduct structured literature reviews was based on prior work using the CNA tool [9, 10, 13, 15, 17]. This approach can lead to a less biased assessment of the academic literature than traditional narrative reviews for at least two reasons. First, a citation analysis approach can avoid the cognitive bias associated with traditional literature searches using keyword searches which may be limited by the researcher expertise, training, and preferences. Second, the use of Google Scholar™ and a snowball sampling technique based on selected seed articles, rather than Web of Science™ citation tools based on keywords for instance, is able to survey a broader scope of publications that may be relevant to D&I frameworks especially given their expansive roots in fields ranging from agriculture, business, and political science to public health and medicine [18, 19]. In addition, the CNA tool allows for a constrained approach to snowball sampling, rather than traditional snowball sampling where the sample grows exponentially, in order to limit the articles at each level from the seed article to arrive at empirical findings using a fraction of the data [15].

As detailed in Additional file 1 , we conducted two analyses using this novel approach. First, we synthesized the literature covered in the framework review article by Tabak et al. [2] with respect to recent citations and performed a structured literature review of the article itself. Next, we applied a structured literature review to a snowball sample of ten framework articles identified as the most important by the study team, largely based on the Tabak review. Overall, this work allowed us to understand the relevance of the framework review article as a D&I resource and to identify those frameworks forming the current backbone of the D&I framework field (i.e., framework articles in the network’s main path).

Characterizing the Tabak et al. framework review article and its citation network

The Tabak systematic review contained 119 references, with 50 published articles and 11 documents (reports/chapters/books) identified as a primary D&I framework reference (n = 61) [2]. These D&I frameworks were identified first through selecting commonly cited frameworks, then through snowball sampling and expert consultation including with U.S. National Institute of Health offıcials who process and review D&I grants. Frameworks were excluded from the review according to the following criteria: (1) focused on practitioner rather than D&I researcher; (2) applied to individual behavior change only (i.e., without ties to local, organizational or community dissemination); (3) intended only for national level use versus local, community, or organizational level; (4) frameworks focused only on dissemination after research study completion; and (5) articles not written in the English language. The frameworks were then judged by the authors to be related either more to dissemination, implementation, or both equally. Each framework’s construct flexibility was rated as broad and flexible versus operational and defined for a given context and activity. Last, the socio-ecologic level (individual, organization, community, healthcare system, policy) targeted by the framework was categorized, with most operating at more than one level.

We extracted the primary citation for each framework. In cases where more than one primary reference was used (n = 21), we selected the most relevant reference, usually the oldest, as the primary reference. The primary references for 11 frameworks were reports, chapters, or books. Because peer-reviewed articles were the most common documents cited in this study, we use the term article to denote all documents throughout the remainder of the manuscript.

To better understand the framework articles discussed in the Tabak review, we conducted descriptive analyses to identify the most common journals, authors, and countries of origin for the 61 models. We also examined the citation rates for each framework. We defined a citation rate as the number of citations/year(s) since publication. We used the Web of Science™ Core Collection in January 2016 to conduct these descriptive citation analyses and inform our subsequent network analysis described in the Additional file 1 .

Citation network analysis of selected D&I frameworks

Next, we conducted a citation network analysis of ten carefully selected D&I framework articles we felt reflected the current state of the field. Eight of these were based on citation rates and the Tabak review. However, we also included two additional frameworks given their relevance to implementation science and relatively high citation rates: (1) Theoretical Domains Framework (TDF) [20, 21] and the (2) Knowledge to Action Framework (KTA) [22], for a total of ten seed articles for our next citation network analysis. Both of these models were developed by researchers outside the USA and were not included in the Tabak review. The details of the D&I framework citation analysis are included in the Additional file 1 .

Last, we performed a main path analysis to identify the connectedness and links among the articles considered to be the backbone of the D&I framework citation network. This approach identifies the key articles influencing D&I models based on the selected seed articles. We determined the traversal weights indicating the proportion of network paths that included a given article node in the network [23]. For instance, a traversal weight of 0.25 for framework X indicates that its article exists in 25% of the citation paths in the network. This traversal weight indicates the importance of any particular node (i.e., article) in the network. We constructed the main path by removing all ties in the network scoring below the 95% percentile for traversal weight value. We normalized the traversal weights according to flow using the Search Path Count method [24]. All computations were accomplished with Pajek [23].

All analyses were conducted between January 2016 and August 2016. This study was deemed not regulated by the Institutional Review Board at the University of Michigan.

Results

Tabak framework review article and its citation network

As illustrated in Fig. 1, the Tabak framework review article is an increasingly cited resource. As of January 2016, it had been cumulatively cited 456 times across 388 articles and other source items indexed within Web of Science™ Core Collection. As shown in Table 1, there was a broad distribution of citation numbers and annual citation rates across the 61 framework articles within the Tabak review and our two selected framework articles (KTA and TDF). The average number of citations per year ranged from 0 to 1949 among articles published from 1962 to 2012. The outlier with the highest citation rate was a book reference for Rogers’ Diffusion of Innovations.

Fig. 1
figure 1

Citation report through 2015 for ‘Bridging Research and Practice Models for Dissemination and Implementation Research’ by Tabak et al. [2]

Table 1 Citations for D&I frameworks in published articles as of January 2016

Based on the structured literature review of the Tabak article using the CNA tool, we identified 239 articles across the network and its three levels of ‘distance.’ This included 17 level-one articles directly referencing the Tabak article, with the remainder of articles residing two and three levels from the Tabak source article. The majority of the documents were journal articles (84%), followed by books (16%). The articles in the Tabak network were published between 2002 and 2016, with 51 articles published prior to the source article year of 2012. The majority (86%) of these were three levels from the Tabak seed article and (35%) were book references. We identified 202 unique first authors contributing to this network. Each author contributed 1.18 articles (standard deviation (SD) = 0.58), on average. Most first authors contributed only one article to the network (one = 177; two = 19, three = 3, four = 2, six = 1). We identified 123 unique journals (books excluded) contributing to the Tabak network, each providing an average of 1.62 articles (SD = 2.63). Most journals contributed one article (n = 95). The top three journals producing the most articles were: Implementation Science (n = 29), Annual Review of Public Health (n = 6), and BMC Public Health (n = 5). All other journals had four or fewer articles each. The articles in the Tabak network were cited between 0 and 4410 times. The top ten cited articles in the Tabak network are shown in Table 2, and none of which served as a primary framework reference. As illustrated in Fig. 2, there were prominent ties in the Tabak network to social care and the law by Aveyard; normalization process and general implementation theory by May; implementation work by Glasgow, Proctor, Neta, and Chambers; a gateway to broader literature via a movement science article by Peters; a Karlin article which ties in psychotherapy; and a 2013 contribution by Straus that was an introduction to knowledge translation in healthcare.

Table 2 Ten most cited articles within the Tabak framework review citation network
Fig. 2
figure 2

Citation network for ‘Bridging Research and Practice Models for Dissemination and Implementation Research’ by Tabak et al. [2]. Most first authors contributed only one article (one = 177). Those authors with two articles—Aarons, G; Archambault, P; Bjurlin, M; Blease, CR; Brownson, R; Chambers, D; Chor, K; Davidoff, F; Edwards, N; Gagliardi, A; Kozica, S; May, C; Naci, H; Neta, G; Page, A; Partridge, SR; Rhoades, E; Trevithick, P; Trockel, M; three articles—Aveyard, H; O’Brien, J; Proctor, E; four articles—Glasgow, R and Powell, B; and six articles—Thompson, N

Citation network analysis of selected D&I frameworks

The citation network for our seed articles highlighted in Table 1 included 355 unique documents published between 1996 and 2014. There were 302,472 citation links connecting the articles in this network. The majority of citations was from 323 journal articles (91%), followed by 29 books (8%), and 3 in-proceedings (1%). We identified 274 unique first authors, each contributing 1.30 articles (SD = 0.84), on average. The majority of first authors provided one article to the network with only six authors contributing greater than three. We also identified 128 unique journals contributing to this network, each providing an average of 2.52 articles (SD = 4.04). While many journals contributed one article (n = 29), the top five journals producing the most articles were: Strategic Management Journal (n = 29), Academy of Management Journal (n = 25), Implementation Science (n = 20), Organization Science (n = 15), and Management Science (n = 10). All other journals contributed less than ten articles each. The top ten cited articles are shown in Table 3, with Szulanski’s Sticky Knowledge as the only primary framework reference from the Tabak review. The remainder of articles tended to focus on business practices and knowledge sharing, collaboration networks, and social and/or intellectual capital. The articles for the D&I framework network contributed between 64 and 12,680 citations, with a median of 489.

Table 3 Ten most cited articles within the D&I framework citation network

As illustrated in Fig. 3, the D&I framework citation network appears centered around the 2004 Greenhalgh et al. article with prominent ties to the Theoretical Domains Framework, the Knowledge to Action Framework, the Promoting Action on Research Implementation in Health Services Framework (PARiHS), the Consolidated Framework for Implementation Research (CFIR), and an article conceptualizing implementation outcomes, among others. A more complete picture of the network’s primary core is offered with the main path analysis, which consists of those ties above the 95% percentile score for traversal weight (0.0106). The main path, illustrated in Fig. 4, is comprised of the 15 articles listed in Table 4. A simple interpretation of the main path is that these articles are most important in holding the entire D&I framework citation network together. In this case, seven of the ten D&I framework seed articles are part of the main path, along with eight non-seed articles. Visually, one can inspect the main path and observe the chronological flow of influence from earlier to more recent publications. Kitson [25] and Klein [26] act as the primary originating sources of influence in the main path, which serve to influence Greenhalgh [27], Damschroder [28], and Proctor [29]. These five articles, along with Glasgow [30], all converge in Aarons [31], which acts as a major hub for the remainder of the more recent works on the periphery of the main path.

Fig. 3
figure 3

D&I framework citation network. The majority of first authors provided only one article to the network with only six authors contributing greater than three including Hansen, M and Pronovost, P—four articles; Michie, S and Rycroft-Malone, J—five articles; Greenhalgh, T—seven articles; and Glasgow, R—eight articles

Fig. 4
figure 4

The main path for a D&I framework citation network. A simple interpretation of the citation network main path is that these articles are the most important in holding the entire D&I framework citation network together. In this case, seven of the ten D&I framework seed articles were part of the main path, along with eight non-seed articles

Table 4 Main path articles for leading D&I research frameworks

Discussion

Using citation analysis, we identified the most frequently cited D&I frameworks and their relationships across time and discipline and mapped the knowledge network constituting the D&I framework field. We discovered that the Tabak framework review has been increasingly cited and that it was included in the periphery of the main D&I framework network path indicating its value as a recognized resource for D&I researchers and practitioners. We identified the leading journals and authors contributing to the D&I framework literature using methods that limit cognitive biases associated with traditional literature searches using keywords. Using the CNA tool to conduct our structured literature review, we were able to identify the main path articles that signify those most important in holding the entire D&I framework citation network together. Overall, D&I researchers and practitioners may consider frequency of citation and this network structure when planning implementation efforts to build upon this foundation and promote systematic advances in D&I science. Further work is necessary to delineate how these frameworks are being used in the literature, framework selection criteria for planning D&I research efforts, the core components of these frameworks, and how framework use relates to improved implementation outcomes [3].

This study provides insight into at least two aspects of the evolving D&I scientific field. First, it confirms that D&I research has witnessed a surge of frameworks with most developed in the last two decades [2]. However, we found that the majority of articles were rarely cited, leaving only a few highly cited frameworks. It is difficult to know whether more recent frameworks will be used or not based on this analysis though several recent articles, including the Tabak review, were highly cited. Nonetheless, there does appear to be framework saturation creating an increasing need to delve further into better understanding the current cadre rather than creating new D&I frameworks. Second, taking into consideration citation rates and this network structure may be a key factor to consider when choosing a framework, in addition to the socioecological level, construct flexibility, and location on the D&I spectrum. For example, increasing citations and centrality in the network indicates more literature is available to highlight the advantages and disadvantages of using a given framework. In addition, there may be more operational and measurement resources with increasing centrality. Taking these additional aspects into consideration creates opportunities to scrutinize frameworks, starting with those in the main path, and advance D&I science by examining issues of fidelity, core and adaptive components, measurement, and relationships to implementation outcomes [1].

We found a broad range of scientific fields contributing to the D&I citation network given our use of Google Scholar™ and its extensive search capabilities [7, 19]. This reinforces the need to scan literature outside of health-related fields to discover new guidance for D&I sciences. For example, other than the specialized journal Implementation Science, which focuses specifically on the field, most citations of the Tabak framework review article were from public health journals due in part to it being a narrative review that used snowball sampling methods and focused on health. In addition, the journals other than Implementation Science, which published the highest number of citations in the broader D&I framework network, were all in the management and business fields. This is consistent with a prior review of leading management journals that found a significant degree of knowledge translation and organizational change literature relevant to D&I in healthcare [32]. While there is some current cross-over among these fields, they are often quite distinct and separate from each other when it comes to research and practice. Taken together, our findings suggest that greater efforts to scan across these journals and fields could provide unique transdisciplinary collaborations and innovation opportunities to hasten D&I research and practice. For that matter, D&I advances could also serve to improve management and business practices.

However, citing a framework does not imply use or specify what its application entails. How to operationalize determinants of practice across frameworks also needs to be better understood to advance D&I science. A recent study examined use of the KTA framework using citation analysis and systematic review to see if the framework was used in practice and how [33]. The authors found that it was used with varying degrees of completeness from a simple reference to integration into the design, delivery, and evaluation of the implementation activities. The latter contributing most to advancing D&I science and generalizability of outcomes. Similarly, another recent systematic review examined use of the CFIR among empirical studies in the peer-reviewed literature [34]. Twenty-six articles met inclusion criteria across a breadth of settings and units of analysis. Justification for which CFIR constructs were selected, integration throughout the research study, and relation to outcomes remained poorly articulated, again limiting contributions to D&I research more broadly. Furthermore, systematic efforts to reconcile determinants of healthcare professional practice across 12 different frameworks have generated practical checklists and implementation strategy recommendations to support implementation and quality improvement efforts [35]. Better understanding framework use, consolidation and operationalization of framework determinants, not just citations, could yield more to consider when selecting and using D&I frameworks for research and practice.

There are several limitations to our study approach. First, framework citation rates are influenced by a multitude of factors including journal impact factor, the authors’ fame and publication rate, the degree of research in a given field, whether citation is perceived as positive or negative, and do not necessarily indicate the quality of a given publication or framework [5,6,7, 19]. Nonetheless, citation rates do serve as an approximation of the impact of a scholarly work. We also used an expert-led review article for seed article identification and a robust network analysis tool, coupled with citation rate data, to provide our snapshot of the scientific development of the D&I framework field with substantial face validity. Second, there could be issues with respect to language and the definition of D&I research leading to ascertainment bias. Using our comprehensive CNA approach in Google Scholar™, rather than keyword searches for example, actually created a broader scope for our study. Last, whether the use of highly cited documents (e.g., textbooks) as seed articles, rather than the journal articles selected as seeds in our study, would dramatically change our findings is unclear. Our network tool was inclusive of such documents although they were the minority of articles in both network analyses. Indeed, publishing frameworks outside of journal articles creates challenges, both in terms of physically obtaining the material and being able to grasp the conceptual and operational components dispersed throughout a given textbook. Perhaps corresponding peer-reviewed articles serving as a book review, preferably in open-access formats to improve dissemination, could help mitigate access and citation issues [36].

Conclusion

In conclusion, bibliometric analysis is one way to understand how D&I frameworks are used in the development of D&I science. We used a bibliometric citation analysis tool to help identify the most prevalent models influencing D&I. D&I researchers and practitioners may consider frequency of citation and this network structure when planning implementation efforts to build upon this foundation and promote systematic advances in D&I science.

Abbreviations

CFIR:

Consolidated Framework for Implementation Research

CNA:

Citation Network Analyzer

D&I:

Dissemination and implementation

KTA:

Knowledge to Action Framework

PARiHS:

Promoting Action on Research Implementation in Health Services Framework

TDF:

Theoretical Domains Framework

References

  1. Brownson R, Colditz G, Proctor E, editors. Dissemination and implementation research in health: translating science to practice. New York: Oxford University Press; 2012.

    Google Scholar 

  2. Tabak RG, Khoong EC, Chambers DA, Brownson RC. Bridging research and practice: models for dissemination and implementation research. Am J Prev Med. 2012;43(3):337–50.

    Article  PubMed  PubMed Central  Google Scholar 

  3. Chaudoir SR, Dugan AG, Barr CH. Measuring factors affecting implementation of health innovations: a systematic review of structural, organizational, provider, patient, and innovation level measures. Implement Sci. 2013;8:22.

    Article  PubMed  PubMed Central  Google Scholar 

  4. Marx W, Bornmann L. Change of perspective: bibliometrics from the point of view of cited references-a literature overview on approaches to the evaluation of cited references in bibliometrics. Scientometrics. 2016;109(2):1397–415.

    Article  PubMed  PubMed Central  Google Scholar 

  5. Bornmann L, Marx W. How good is research really? Measuring the citation impact of publications with percentiles increases correct assessments and fair comparisons. EMBO Rep. 2013;14(3):226–30.

    Article  CAS  PubMed  Google Scholar 

  6. Garfield E. The history and meaning of the journal impact factor. JAMA. 2006;295(1):90–3.

    Article  CAS  PubMed  Google Scholar 

  7. Bakkalbasi N, Bauer K, Glover J, Wang L. Three options for citation tracking: Google Scholar, Scopus and Web of Science. Biomed Digit Libr. 2006;3:7.

    Article  PubMed  PubMed Central  Google Scholar 

  8. Harris JK. Connecting discovery and delivery: the need for more evidence on effective smoking cessation strategies for people living with HIV/AIDS. Am J Public Health. 2010;100(7):1245–9.

    Article  PubMed  PubMed Central  Google Scholar 

  9. Harris JK, Beatty KE, Lecy JD, Cyr JM, Shapiro RM 2nd. Mapping the multidisciplinary field of public health services and systems research. Am J Prev Med. 2011;41(1):105–11.

    Article  PubMed  Google Scholar 

  10. Harris JK, Lecy J, Hipp JA, Brownson RC, Parra DC. Mapping the development of research on physical activity and the built environment. Prev Med. 2013;57(5):533–40.

    Article  PubMed  PubMed Central  Google Scholar 

  11. Harris JK, Luke DA, Zuckerman RB, Shelton SC. Forty years of secondhand smoke research: the gap between discovery and delivery. Am J Prev Med. 2009;36(6):538–48.

    Article  PubMed  Google Scholar 

  12. Lecy J, Schmitz HP, Swedlund H. NGO and NPO effectiveness: a modern synthesis. Voluntas. 2012;23(2):434–57.

    Article  Google Scholar 

  13. Shadish WR, Lecy JD. The meta-analytic big bang. Res Synth Methods. 2015;6(3):246–64.

    Article  PubMed  Google Scholar 

  14. Harper G, Peattie K. Tracking the influence of the first special journal issue on ‘Green Marketing’: a citation network analysis. Soc Bus. 2011;1(3):239–61.

    Article  Google Scholar 

  15. Jesse D. Lecy and Kate E. Beatty, Representative literature reviews using constrained snowball sampling and citation network analysis 2012. Available at SSRN: https://ssrn.com/abstract=1992601 or http://dx.doi.org/10.2139/ssrn.1992601.

    Google Scholar 

  16. Jesse Lecy and Diego Moreda (2013). “cna: citation network analyzer.” R package version 0.3-4.

    Google Scholar 

  17. Lecy JD, Mergel IA, Schmitz HP. Networks in public administration: current scholarship in review. Public Manag Rev. 2014;16(5):643–65.

    Article  Google Scholar 

  18. Noruzi A. Google scholar: the new generation of citation indexes. Libri:Int J Libr Inform Serv. 2005;55(4):170–80.

    Article  Google Scholar 

  19. Maslov S, Redner S. Promise and pitfalls of extending Google’s PageRank algorithm to citation networks. J Neurosci. 2008;28(44):11103–5.

    Article  CAS  PubMed  Google Scholar 

  20. Cane J, O'Connor D, Michie S. Validation of the theoretical domains framework for use in behaviour change and implementation research. Implement Sci. 2012;7:37.

    Article  PubMed  PubMed Central  Google Scholar 

  21. Francis JJ1, O'Connor D, Curran J. Theories of behaviour change synthesised into a set of theoretical groupings: introducing a thematic series on the theoretical domains framework. Implement Sci. 2012;7:35.

  22. Graham ID, Logan J, Harrison MB, Straus SE, Tetroe J, Caswell W, Robinson N. Lost in knowledge translation: time for a map? J Contin Educ Heal Prof. 2006;26(1):13–24.

    Article  Google Scholar 

  23. Wd N, Mrvar A, Batagelj V. Exploratory social network analysis with Pajek. New York: Cambridge University Press; 2005.

    Google Scholar 

  24. Batagelj V, Doreian P, Ferligoj A, Kejzar N. Understanding Large Temporal Networks and Spatial Networks: Exploration, Pattern Searching, Visualization and Network Evolution. West Sussex: Wiley; 2014.

  25. Kitson A, Harvey G, McCormack B. Enabling the implementation of evidence based practice: a conceptual framework. Qual Health Care. 1998;7(3):149–58.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  26. Klein KJ, Sorra JS. The challenge of innovation implementation. Acad Manag Rev. 1996;21(4):1055–80.

    Google Scholar 

  27. Greenhalgh T, Robert G, Macfarlane F, Bate P, Kyriakidou O. Diffusion of innovations in service organizations: systematic review and recommendations. Milbank Q. 2004;82(4):581–629.

    Article  PubMed  PubMed Central  Google Scholar 

  28. Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci. 2009;4

  29. Proctor EK, Landsverk J, Aarons G, Chambers D, Glisson C, Mittman B. Implementation research in mental health services: an emerging science with conceptual, methodological, and training challenges. Adm Policy Ment Health. 2009;36(1):24–34.

    Article  PubMed  Google Scholar 

  30. Glasgow RE, Vogt TM, Boles SM. Evaluating the public health impact of health promotion interventions: the RE-AIM framework. Am J Public Health. 1999;89(9):1322–7.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  31. Aarons GA, Hurlburt M, Horwitz SM. Advancing a conceptual model of evidence-based practice implementation in public service sectors. Adm Policy Ment Health. 2011;38(1):4–23.

    Article  PubMed  Google Scholar 

  32. Harlos K, Tetroe J, Graham ID, Bird M, Robinson N. Mining the management literature for insights into implementing evidence-based change in healthcare. Health Policy. 2012;8(1):33–48.

    Google Scholar 

  33. Field B, Booth A, Ilott I, Gerrish K. Using the knowledge to action framework in practice: a citation analysis and systematic review. Implement Sci. 2014;9:172.

    Article  PubMed  PubMed Central  Google Scholar 

  34. Kirk MA, Kelley C, Yankey N, Birken SA, Abadie B, Damschroder L. A systematic review of the use of the consolidated framework for implementation research. Implement Sci. 2016;11:72.

    Article  PubMed  PubMed Central  Google Scholar 

  35. Flottorp SA, Oxman AD, Krause J, Musila NR, Wensing M, Godycki-Cwirko M, Baker R, Eccles MP. A checklist for identifying determinants of practice: a systematic review and synthesis of frameworks and taxonomies of factors that prevent or enable improvements in healthcare professional practice. Implement Sci. 2013;8:35.

    Article  PubMed  PubMed Central  Google Scholar 

  36. Baetens F, Prislan V. The dissemination of international scholarship: the future of books and book reviews. Leiden J Int Law. 2014;27(3):559–69.

    Article  Google Scholar 

  37. Szulanski G. Exploring internal stickiness: impediments to the transfer of best practice within the firm. Strateg Manag J. 1996;17:27–43.

    Article  Google Scholar 

  38. Michie S, Johnston M, Abraham C, Lawton R, Parker D, Walker A, Grp PT. Making psychological theory useful for implementing evidence based practice: a consensus approach. Qual Saf Health Care. 2005;14(1):26–33.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  39. Lavis JN, Robertson D, Woodside JM, McLeod CB, Abelson J, Grp KTS. How can research organizations more effectively transfer research knowledge to decision makers? Milbank Q. 2003;81(2):221-+.

    Article  PubMed  PubMed Central  Google Scholar 

  40. Wandersman A, Duffy J, Flaspohler P, Noonan R, Lubell K, Stillman L, Blachman M, Dunville R, Saul J. Bridging the gap between prevention research and practice: the interactive systems framework for dissemination and implementation. Am J Community Psychol. 2008;41(3–4):171–81.

    Article  PubMed  Google Scholar 

  41. Green LW, Ottoson JM, Garcia C, Hiatt RA. Diffusion theory and knowledge dissemination, utilization, and integration in public health. Annu Rev Public Health. 2009;30:151–74.

    Article  PubMed  Google Scholar 

  42. May C, Finch T. Implementing, embedding, and integrating practices: an outline of normalization process theory. Sociology. 2009;43(3):535–54.

    Article  Google Scholar 

  43. Frambach RT, Schillewaert N. Organizational innovation adoption—a multi-level framework of determinants and opportunities for future research. J Bus Res. 2002;55(2):163–76.

    Article  Google Scholar 

  44. Pathman DE, Konrad TR, Freed GL, Freeman VA, Koch GG. The awareness-to-adherence model of the steps to clinical guideline compliance—the case of pediatric vaccine recommendations. Med Care. 1996;34(9):873–89.

    Article  CAS  PubMed  Google Scholar 

  45. Pronovost P, Berenholtz S, Needham D. Translating evidence into practice: a model for large scale knowledge translation. BMJ. 2008;337:a1714

  46. Ward V, House A, Hamer S. Developing a framework for transferring knowledge into action: a thematic analysis of the literature. J Health Serv Res Policy. 2009;14(3):156–64.

    Article  PubMed  PubMed Central  Google Scholar 

  47. Mendel P, Meredith LS, Schoenbaum M, Sherbourne CD, Wells KB. Interventions in organizational and community context: a framework for building evidence on dissemination and implementation in health services research. Adm Policy Ment Health. 2008;35(1–2):21–37.

    Article  PubMed  Google Scholar 

  48. Atun R, de Jongh T, Secci F, Ohiri K, Adeyi O. Integration of targeted health interventions into health systems: a conceptual framework for analysis. Health Policy Plan. 2010;25(2):104–11.

    Article  PubMed  Google Scholar 

  49. Bowen S, Zwi AB. Pathways to “evidence-informed” policy and practice: a framework for action. PLoS Med. 2005;2(7):600–5.

    Article  Google Scholar 

  50. Glisson C, Schoenwald SK. The ARC organizational and community intervention strategy for implementing evidence-based children’s mental health treatments. Ment Health Serv Res. 2005;7(4):243–59.

    Article  PubMed  Google Scholar 

  51. Feldstein AC, Glasgow RE. A practical, robust implementation and sustainability model (PRISM) for integrating research findings into practice. Jt Comm J Qual Patient Saf. 2008;34(4):228–43.

    Article  PubMed  Google Scholar 

  52. Weiner BJ, Lewis MA, Linnan LA. Using organization theory to understand the determinants of effective implementation of worksite health promotion programs. Health Educ Res. 2009;24(2):292–305.

    Article  PubMed  Google Scholar 

  53. Logan J, Graham ID. Toward a comprehensive interdisciplinary model of health care research use. Sci Commun. 1998;20(2):227–46.

    Article  Google Scholar 

  54. Owen N, Glanz K, Sallis JF, Kelder SH. Evidence-based approaches to dissemination and diffusion of physical activity interventions. Am J Prev Med. 2006;31(4):S35–44.

    Article  PubMed  Google Scholar 

  55. Kilbourne AM, Neumann MS, Pincus HA, Bauer MS, Stall R. Implementing evidence-based interventions in health care: application of the replicating effective programs framework. Implement Sci. 2007;2

  56. Jacobson N, Butterill D, Goering P. Development of a framework for knowledge translation: understanding user context. J Health Serv Res Policy. 2003;8(2):94–9.

    Article  PubMed  Google Scholar 

  57. Baumbusch JL, Kirkham SR, Khan KB, McDonald H, Semeniuk P, Tan E, Anderson JM. Pursuing common agendas: a collaborative model for knowledge translation between research and practice in clinical settings. Res Nurs Health. 2008;31(2):130–40.

    Article  PubMed  Google Scholar 

  58. Dearing JW, Maibach EW, Buller DB. A convergent diffusion and social marketing approach for disseminating proven approaches to physical activity promotion. Am J Prev Med. 2006;31(4):S11–23.

    Article  PubMed  Google Scholar 

  59. Dobbins M, Ciliska D, Cockerill R, Burnsley J, DiCenso A. A framework for the dissemination and utilization of research for health-care policy and practice. Online J Knowl Synth Nurs. 2002;9(7).

  60. Green LW, Orleans CT, Ottoson JM, Cameron R, Pierce JP, Bettinghaus EP. Inferring strategies for disseminating physical activity policies, programs, and practices from the successes of tobacco control. Am J Prev Med. 2006;31(4 Suppl):S66–81.

    Article  PubMed  Google Scholar 

  61. Kontos PC, Poland BD. Mapping new theoretical and methodological terrain for knowledge translation: contributions from critical realism and the arts. Implement Sci. 2009;4:1.

    Article  PubMed  PubMed Central  Google Scholar 

  62. Lomas J. Retailing research: increasing the role of evidence in clinical services for childbirth. Milbank Q. 1993;71(3):439–75.

    Article  CAS  PubMed  Google Scholar 

  63. Majdzadeh R, Sadighi J, Nejat S, Mahani AS, Gholami J. Knowledge translation for research utilization: design of a knowledge translation model at Tehran University of Medical Sciences. J Contin Educ Heal Prof. 2008;28(4):270–7.

    Article  Google Scholar 

  64. Dreisinger ML, Boland EM, Filler CD, Baker EA, Hessel AS, Brownson RC. Contextual factors influencing readiness for dissemination of obesity prevention programs and policies. Health Educ Res. 2012;27(2):292–306.

    Article  PubMed  Google Scholar 

  65. Martin GW, Herie MA, Turner BJ, Cunningham JA. A social marketing model for disseminating research-based treatments to addictions treatment providers. Addiction. 1998;93(11):1703–15.

    Article  CAS  PubMed  Google Scholar 

  66. Elliott SJ, O'Loughlin J, Robinson K, Eyles J, Cameron R, Harvey D, Raine K, Gelskey D, Canadian Heart Health Dissemination Project S, Research Advisory G. Conceptualizing dissemination research and activity: the case of the Canadian heart health initiative. Health Educ Behav. 2003;30(3):267–82. discussion 283-266

    Article  PubMed  Google Scholar 

  67. Kramer DM, Cole DC. Sustained, intensive engagement to promote health and safety knowledge transfer to and utilization by workplaces. Sci Commun. 2003;25(1):56–82.

    Article  Google Scholar 

  68. Farkas M, Jette AM, Tennstedt S, Haley SM, Quinn V. Knowledge dissemination and utilization in gerontology: an organizing framework. Gerontologist. 2003;43:47–56.

    Article  PubMed  Google Scholar 

  69. Robinson K, Elliott SJ, Driedger SM, Eyles J, O'Loughlin J, Riley B, Cameron R, Harvey D, Grp CSRA. Using linking systems to build capacity and enhance dissemination in heart health promotion: a Canadian multiple-case study. Health Educ Res. 2005;20(5):499–513.

    Article  PubMed  Google Scholar 

  70. Yuan CT, Nembhard IM, Stern AF, Brush JE Jr, Krumholz HM, Bradley EH. Blueprint for the dissemination of evidence-based practices in health care. Issue Brief (Commonw Fund). 2010;86:1–16.

    Google Scholar 

  71. Harris JR, Cheadle A, Hannon PA, Forehand M, Lichiello P, Mahoney E, Snyder S, Yarrow J. A framework for disseminating evidence-based health promotion practices. Prev Chronic Dis. 2012;9:E22.

    Article  PubMed  Google Scholar 

  72. Nolan K, Schall MW, Erb F, Nolan T. Using a framework for spread: the case of patient access in the veterans health administration. Jt Comm J Qual Patient Saf. 2005;31(6):339–47.

    Article  PubMed  Google Scholar 

  73. Anderson M, Cosby J, Swan B, Moore H, Broekhoven M. The use of research in local health service agencies. Soc Sci Med. 1999;49(8):1007–19.

    Article  CAS  PubMed  Google Scholar 

  74. Bauman AE, Nelson DE, Pratt M, Matsudo V, Schoeppe S. Dissemination of physical activity evidence, programs, policies, and surveillance in the international public health arena. Am J Prev Med. 2006;31(4):S57–65.

    Article  PubMed  Google Scholar 

  75. Sogolow ED, Kay LS, Doll LS, Neumann MS, Mezoff JS, Eke AN, Semaan S, Anderson JR. Strengthening HIV prevention: application of a research-to-practice framework. AIDS Educ Prev. 2000;12(5):21–32.

    CAS  PubMed  Google Scholar 

  76. Orlandi MA. Health promotion technology transfer: organizational perspectives. Can J Public Health. 1996;87:S28–33.

    PubMed  Google Scholar 

  77. Winkler JD, Lohr KN, Brook RH. Persuasive communication and medical technology-assessment. Arch Intern Med. 1985;145(2):314–7.

    Article  CAS  PubMed  Google Scholar 

  78. Lester JP. The utilization of policy analysis by state agency officials. Knowl. 1993;14(3):267–90.

    Article  Google Scholar 

  79. Funk SG, Tornquist EM, Champagne MT. A model for improving the dissemination of nursing research. West J Nurs Res. 1989;11(3):361–72.

    Article  CAS  PubMed  Google Scholar 

  80. Scullion PA. Effective dissemination strategies. Nurs Res. 2002;10(1):65–77.

    Article  Google Scholar 

  81. Rogers EM. Diffusion of innovations. New York: Free Press of Glencoe; 1962.

    Google Scholar 

  82. Kingdon JW. Agendas, alternatives, and public policies. Boston: Little, Brown; 1984.

    Google Scholar 

  83. Fixsen DL, Naoom SF, Blasé KA, Friedman RM, Wallace F. Implementation research: a synthesis of the literature. Tampa: University of South Florida, Louis de la Parte Florida Mental Health Institute, The National Implementation Research Network; 2005.

    Google Scholar 

  84. Green LW, Kreuter MW, Green LW. Health program planning: an educational and ecological approach. 4th ed. New York: McGraw-Hill; 2005.

    Google Scholar 

  85. Havelock RG. Planning for innovation through dissemination and utilization of knowledge. Centre for Research on utilization of Scientifıc knowledge, Institute for Social Research, University of Michigan, 1969.

    Google Scholar 

  86. Pettigrew AM, Ferlie E, McKee L. Shaping strategic change: making change in large organizations: the case of the National Health Service. Thousand Oaks: Sage Publications; 1992.

    Google Scholar 

  87. Nieva VF, Murphy R, Ridley N, et al. From science to service: a framework for the transfer of patient safety. In: Henriksen K, Battles JB, Marks ES, Lewin DI, editors. Advances in patient safety: from research to implementation (Vol. 2: concepts and methodology). Agency for Healthcare Research and Quality: Rockville; 2005.

    Google Scholar 

  88. Dodson EA, Brownson RC, Weiss SW. Policy dissemination research. In: Brownson R, Colditz G, Proctor EK, editors. Dissemination and implementation research in health: translating science to practice. Oxford: Oxford University Press; 2012.

    Google Scholar 

  89. Kreuter MW, Casey CM, Bernhardt JM. Enhancing dissemination though marketing and distribution systems: a vision for public health. In: Brownson RC, Colditz G, Proctor EK, editors. Dissemination and implementation research in health: translating science to practice. New York: Oxford University Press; 2012.

    Google Scholar 

  90. Damush T, Bravata DM, Plue L, Woodward-Hagg H, Williams LS. Facilitation of best practices (FAB) framework. Stroke QUERI Center annual report. 2008.

    Google Scholar 

  91. TIDIRH Working Group. Interacting elements of integrating science, policy, and practice. In: Training institute for dissemination and implementation research in health. Chapel Hill: Conference proceedings; 2011.

    Google Scholar 

  92. Bell J, Waters S. Doing Your Research Project: A Guide for First-Time Researchers. England: McGraw-Hill Education; 2014.

  93. Gratton C, Jones I. Research Methods for Sports Studies. London: Routledge; 2010.

  94. Hanney SR, Gonzalez-Block MA, Buxton MJ, Kogan M. The utilisation of health research in policy-making: concepts, examples and methods of assessment. Health Res Policy Syst. 2003;1(1):2.

    Article  PubMed  PubMed Central  Google Scholar 

  95. Hersh W. Information retrieval: a health and biomedical perspective. New York: Springer; 2008.

    Google Scholar 

  96. Dalrymple J, Burke B. Anti-oppressive practice: social care and the law. England: McGraw-Hill Companies, Incorporated; 2006.

  97. Aveyard H. Doing a literature review in health and social care: a practical guide. England: McGraw-Hill Education; 2014.

  98. Trevithick P. Social work skills: a practice handbook. England: Open University Press; 2005.

  99. Thompson N. Understanding social work: preparing for practice. England: Palgrave Macmillan; 2015.

  100. D'Cruz H, Gillingham P, Melendez S. Reflexivity, its meanings and relevance for social work: a critical review of the literature. Brit J Soc Work. 2007;37(1):73–90.

    Article  Google Scholar 

  101. Dagenais S, Tricco AC, Haldeman S. Synthesis of recommendations for the assessment and management of low back pain from recent clinical practice guidelines. Spine J. 2010;10(6):514–29.

    Article  PubMed  Google Scholar 

  102. Nahapiet J, Ghoshal S. Social capital, intellectual capital, and the organizational advantage. Acad Manag Rev. 1998;23(2):242–66.

    Google Scholar 

  103. Eisenhardt KM, Martin JA. Dynamic capabilities: what are they? Strateg Manag J. 2000;21(10–11):1105–21.

    Article  Google Scholar 

  104. Dyer JH, Singh H. The relational view: cooperative strategy and sources of interorganizational competitive advantage. Acad Manag Rev. 1998;23(4):660–79.

    Google Scholar 

  105. Wenger E, McDermott RA, Snyder W. Cultivating communities of practice: a guide to managing knowledge. Boston: Harvard Business School Press; 2002.

  106. Alavi M, Leidner DE. Review: knowledge management and knowledge management systems: conceptual foundations and research issues. Mis Quart. 2001;25(1):107–36.

    Article  Google Scholar 

  107. Zahra SA, George G. Absorptive capacity: a review, reconceptualization, and extension. Acad Manag Rev. 2002;27(2):185–203.

    Google Scholar 

  108. Hansen MT. The search-transfer problem: the role of weak ties in sharing knowledge across organization subunits. Admin Sci Quart. 1999;44(1):82–111.

    Article  Google Scholar 

  109. Ahuja G. Collaboration networks, structural holes, and innovation: a longitudinal study. Admin Sci Quart. 2000;45(3):425–55.

    Article  Google Scholar 

  110. Dyer JH, Nobeoka K. Creating and managing a high-performance knowledge-sharing network: the Toyota case. Strateg Manag J. 2000;21(3):345–67.

    Article  Google Scholar 

  111. Stirman SW, Kimberly J, Cook N, Calloway A, Castro F, Charns M. The sustainability of new programs and innovations: a review of the empirical literature and recommendations for future research. Implement Sci. 2012;7

  112. Meyers DC, Durlak JA, Wandersman A. The quality implementation framework: a synthesis of critical steps in the implementation process. Am J Community Psychol. 2012;50(3–4):462–80.

    Article  PubMed  Google Scholar 

  113. Schoenwald SK, Garland AF, Chapman JE, Frazier SL, Sheidow AJ, Southam-Gerow MA. Toward the effective and efficient measurement of implementation fidelity. Admin Pol Ment Health. 2011;38(1):32–43.

    Article  Google Scholar 

  114. Palinkas LA, Aarons GA, Horwitz S, Chamberlain P, Hurlburt M, Landsverk J. Mixed method designs in implementation research. Adm Policy Ment Health. 2011;38(1):44–53.

    Article  PubMed  Google Scholar 

  115. Sanders MR. Development, evaluation, and multinational dissemination of the triple P-positive parenting program. Annu Rev Clin Psychol. 2012;8:345–79.

    Article  PubMed  Google Scholar 

  116. Aarons GA, Glisson C, Green PD, Hoagwood K, Kelleher KJ, Landsverk JA, Hlth RNYM. The organizational social context of mental health services and clinician attitudes toward evidence-based practice: a United States national study. Implement Sci. 2012;7

Download references

Acknowledgements

Ryan Blake, BS, for administrative and data collection support.

Funding

Dr. Skolarus was supported by a VA HSR&D Career Development Award-2 (CDA 12–171) and the Mentored Training for Dissemination and Implementation Research in Cancer (MT-DIRC) Program, National Cancer Institute, 1 R25 CA171994-01A1 during this study. This study did not receive any dedicated funding.

Availability of data and materials

Data for this project is stored on secure servers. Data can be made publicly available upon request.

Author information

Authors and Affiliations

Authors

Contributions

The individual contributions of the authors are as follows: TS, TL, RT, JH, JL, and AS contributed to the study conception and design. TS, TL, and JL contributed to the acquisition of data. TS, TL, JL, JH, RT, and AS contributed to the analysis and interpretation of data. TS, TL, and AS drafted the manuscript. TS, TL, RT, JH, JL, and AS made critical revisions. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Ted A. Skolarus.

Ethics declarations

Ethics approval and consent to participate

This study was deemed not regulated by the Institutional Review Board at the University of Michigan.

Consent for publication

N/a.

Competing interests

The authors declare that they have no competing interests.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Additional file

Additional file 1:

Citation Network Analysis Methods. (DOCX 14 kb)

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Skolarus, T.A., Lehmann, T., Tabak, R.G. et al. Assessing citation networks for dissemination and implementation research frameworks. Implementation Sci 12, 97 (2017). https://doi.org/10.1186/s13012-017-0628-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s13012-017-0628-2

Keywords