Usage of an online tool to help policymakers better engage with research: Web CIPHER
© Makkar et al.; licensee Biomed Central. 2015
Received: 6 November 2014
Accepted: 1 April 2015
Published: 23 April 2015
There is a need to develop innovations that help policymakers better engage with research in order to increase its use in policymaking. As part of the Centre for Informing Policy in Health with Evidence from Research (CIPHER), we established Web CIPHER, an online tool with dynamic interactive elements such as hot topics, research summaries, blogs from trusted figures in health policy and research, a community bulletin board, multimedia section and research portal. The aim of this study was to examine policymakers’ use of the website, and determine which sections were key drivers of use.
Google Analytics (GA) was used to gather usage data during a 16-month period. Analysis was restricted to Web CIPHER members from policy agencies. We examined descriptive statistics including mean viewing times, number of page visits and bounce rates for each section and performed analyses of variance to compare usage between sections. Repeated measures analyses were undertaken to examine whether a weekly reminder email improved usage of Web CIPHER, particularly for research-related content.
During the measurement period, 223 policymakers from more than 32 organisations joined Web CIPHER. Users viewed eight posts on average per visit and stayed on the site for approximately 4 min. The bounce rate was less than 6%. The Blogs and Community sections received more unique views than all other sections. Blogs relating to improving policymakers’ skills in applying research to policy were particularly popular. The email reminder had a positive effect on improving usage, particularly for research-related posts.
The data indicated a relatively small number of users. However, this sample may not be representative of policymakers since membership to the site and usage was completely voluntarily. Nonetheless, those who used the site appeared to engage well with it. The findings suggest that providing blog-type content written by trusted experts in health policy and research as well as regular email reminders may provide an effective means of disseminating the latest research to policymakers through an online web portal.
KeywordsHealth policy Research Websites Knowledge Platforms Portals Innovations Policymakers
The process of using the best available research evidence to inform decision making in health is known as Evidence-Informed Health Policymaking (EIHP) [1,2]. EIHP is purported to lead to better health policies, more effective implementation and more efficient use of resources, with the ultimate goal of improving health outcomes for the wider community [3-5]. Research has informed policies that have improved health in areas such as smoking, alcohol use, immunisation, falls prevention, cardiovascular health, neural development and mental health [6-12]. Evidence indicates, however, that many opportunities to use research to inform health decision making are currently missed [4,6,13-23]. This is, in part, due to various barriers to EIHP such as poor dissemination of research to policymakers [24,25]; deficits in policymakers’ skills in accessing, appraising and applying research to policy [25-28]; a lack of relevant, timely, actionable and locally applicable research [25,28,29]; negative perceptions of the value, quality and usefulness of research held by policymakers [18,30-32]; and an absence of organisational tools and processes to support research access, generation, dissemination and use in policymaking [25,31,33].
There is a need for innovative tools to help policymakers overcome these barriers to accessing research and improve their engagement with research in policymaking. Numerous organisations are now utilising IT-enabled (often web based) tools such as Knowledge Platforms in order to improve the ways in which information and knowledge is accessed, stored, retrieved and distributed across organisations [34-38]. It is likely that web-based Knowledge Platforms could be used to overcome some barriers to EIHP, notably difficulties in accessing and disseminating up-to-date, credible and relevant research information to policymakers [31,36].
The Centre for Informing Policy in Health with Evidence from Research (CIPHER) is a Centre for Research Excellence funded by the Australian National Health and Medical Research Council. One of its chief aims is to develop and test new strategies to increase research use in policy. In keeping with this, Web CIPHER (http://www.cipher.org.au) was established by the centre as an online resource to help health decision makers better use, access and engage with research in their work, based on evidence that suggests internet channels can play a role in this [39,40]. Web CIPHER, like other Knowledge Platforms, uses web/email technology to provide users with an efficient means of accessing and delivering comprehensive and up-to-date information regarding health research and policy. Web CIPHER also provides a forum where policymakers and researchers can interact to share knowledge and build on each other’s capacities .
Details of key sections in Web CIPHER
Selection criteria for content
Short blurbs linking to news stories relevant to health policy
Media monitoring with keywords of interest set up via Google Alerts. Keywords include the following: “health quality and safety”, “aboriginal health”, “evidence in policy”, “research evidence”, “health and medical research”, “e-health”, “linked health data”, “ageing”, “public health”.
Daily scan of these alerts for relevant stories of interest to the network, with an emphasis on intervention research, program reviews, locally-relevant news, or examples of evidence-based policy.
Summaries of systematic reviews, research papers and reports, with links to full articles
Fortnightly scan of relevant journals and sources for systematic reviews, research articles and reports.
Journals include the “Medical Journal of Australia”, “Milbank Quarterly”, “Australian and New Zealand Journal of Public Health”, “Journal of Health Services Research”.
Other sources include email alerts from the Australian Healthcare and Hospitals Association, the Australian Commission on Safety and Quality in Health Care, Australian Policy Online, the NSW Agency for Clinical Innovation, the National Health and Medical Research Council, the Australian Research Council, the Primary Health Care Research Information Service, the Bureau of Health Information and the Australian Institute of Health and Welfare and the Cochrane Collaboration.
Sources are reviewed for research, reviews and reports of interest, with an emphasis on public health intervention research, program evaluation, locally-relevant research and research translation.
Summaries of upcoming events of interest to health policymakers
Monthly scan of organisations of interest for relevant events, including the Australian Healthcare and Hospitals Association, the Australian Commission on Safety and Quality in Health Care, Australian Policy Online, the NSW Agency for Clinical Innovation, the National Health and Medical Research Council, the Australian Research Council, the Primary Health Care Research Information Service, the Bureau of Health Information, the Australian Institute of Health and Welfare and the Cochrane Collaboration.
Also reviewed are organisations represented by members, in particular those listed on this page: https://cipher.org.au/the-web-cipher-community/.
Videos, audio recordings and presentations of interest to health policymakers
Monthly scan of email alerts from organisations: the Australian Healthcare and Hospitals Association, the Australian Commission on Safety and Quality in Health Care, Australian Policy Online, the NSW Agency for Clinical Innovation, the National Health and Medical Research Council, the Australian Research Council, the Primary Health Care Research Information Service, the Bureau of Health Information, the Australian Institute of Health and Welfare and the Cochrane Collaboration.
Also scanned are organisations represented by members, in particular those listed on this page: https://cipher.org.au/the-web-cipher-community/.
500-word conversational-style articles written by health leaders, focusing on their area of expertise, with lessons for policymakers
Quarterly meetings with management team to agree on potential topics and contributors. Pitch sent to contributor, and article written by contributor in consultation with senior staff member at the Sax Institute to ensure tone and style matches Web CIPHER requirements.
Bulletin board where users can post information for other community members, such as new research, jobs or opportunities
Links to sources of high-quality research and data, such as the Cochrane Library, the Campbell Collaboration, health-evidence.ca, Eppi-CENTRE, NIHR Centre for Reviews and Dissemination, the Australian Institute of Health and Welfare and the Australian Bureau of Statistics.
Updated as required only. Sources are those identified as high quality by CIPHER investigators and Web CIPHER users.
This section contains short blurbs, posted on the site daily, which link to news stories relevant to health policy and the application of research in health policy. Hot topics aim to provide users with digestible and accessible summaries of the latest developments in health policy and research [21,25,27,30,32,41-43]. The content aims to be engaging, appealing and highly relevant to the Australian policy context [18,26,32,44]. For example, numerous articles focus on health issues related to key subgroups (such as the elderly, children or Indigenous Australians) of interest to the policy agencies in the Web CIPHER community. Many articles also highlight the importance and value of evidence-informed decision making.
This section contains clear, concise, jargon-free summaries, posted fortnightly, of new systematic reviews, research papers and reports, with links to full articles. The summaries are presented in the style of a newspaper report to help users easily comprehend the key findings of each research paper and hopefully encourage them to access the full article. The research articles in this section include primary research studies and systematic reviews covering topics from EIHP to evidence-informed interventions and patterns of diseases and other health conditions of relevance to the agencies in the Web CIPHER community. Primary research articles are included in addition to reviews to provide users with up-to-date, locally and topically relevant information for which systematic reviews have yet to be produced. Also included here are articles on things like evaluations of specific policies, which do not necessarily lend themselves to systematic review.
This section summarises upcoming events of interest to health policymakers, such as conferences, workshops, seminars or symposia. Policymakers and leaders can use this section to find out about programs to improve their research skills as well as conferences and seminars to learn more about the value of evidence-informed decision making [20,45-47]. This section also alerts policymakers to conferences and symposia where they can gain direct access to the latest research findings in their particular field and establish connections (or perhaps, ongoing partnerships) with researchers [14,48,49].
Blogs contain 500-word conversational-style articles written by health leaders, focusing on their area of expertise. The blogs provide users with a conversational, engaging, reliable and concise presentation of research findings and the application of research to health policy [21,25,27,30,32,41-43]. Many of the blog posts are geared towards improving policymakers’ perceptions of the value of EIHP and providing policymakers with first-hand guidance and advice on strategies to improve research use in decision making [20,27,45].
This section contains videos, audio recordings and presentations of interest to health policymakers. In many of the videos and audio files, internationally recognised and trusted health researchers present research findings, discuss innovations and strategies to integrate research into policy, as well as the value of EIHP. The use of a variety of media formats facilitates dissemination of research findings and presents research-policy information in an accessible and engaging format to users [28,44].
This section contains a Bulletin Board where users can post information for other community members, such as new research, research-policy events like conferences or symposia, new job opportunities or other relevant updates. The Bulletin Board can thus be used by policymakers and researchers to access research, disseminate relevant research studies to other policymakers and discuss its applicability to current policy decisions [14,25,49].
The research portal is a static section that provides users with web links to sources of high-quality research and data, such as the Cochrane Library. There are links to websites that offer one-stop shopping for systematic reviews, evidence briefs, high-quality research journals and databases and reliable sources of data and statistics (e.g. Australian Institute of Health and Welfare). The Research portal is especially helpful to policymakers if their organisations do not provide tools and systems to assist them in searching for and accessing research findings [45,48].
Users who become members of Web CIPHER are sent a weekly summary email that provides links to the latest Hot topics, Research updates, Blogs and Multimedia added to the site during the week. This email provides an effective means of disseminating relevant research to policymakers and improving their access to such research [45,48].
Finally, Web CIPHER also includes a search bar and tags (such as “Obesity”, “Nutrition”, “Aboriginal and Torres Strait Islander Health”) that allow users to easily search for and access research articles and summaries, hot topics and other posts on topics of interest.
Web CIPHER aims to help policymakers access and engage with research more effectively, but these benefits are dependent on users accessing and using the website. Consequently, we conducted a pilot study analysing preliminary usage data to identify which sections were most appealing to users and what features of Web CIPHER were particularly effective at driving usage. We intend to use this information to guide future strategies to improve policymakers’ use and engagement with the website.
A large number of state and federal-level health agencies were formally invited by the CIPHER chief investigators to attend the official launch of CIPHER. These agencies were invited on the basis that they routinely developed health policies, programs and practises and were based at the state or federal level. Following the launch, executives from these agencies were formally invited by email to join the CIPHER network. Employees from those agencies interested in becoming CIPHER members were then invited by email to join Web CIPHER.
As part of applying to join the site, applicants were required to report their place of work. Users were accepted as members if they reported working for one of the CIPHER network agencies or any other Australian organisation engaged in health policy or program work. For this pilot study, the target sample thus consisted of health decision makers, practitioners or researchers (primarily heads of research institutes or schools) from organisations within the CIPHER network.
Mean usage and standard deviations for Web CIPHER sections and other select pages
Total number of posts
Total number of unique post views
Minimum views for a single post
Maximum views for a single post
Unique page views
Average viewing time (s)
Research Value posts
Posts accessed directly from email link
Posts accessed indirectly from an email link
Accessed from an alternative route
We conducted a series of statistical analyses using the GA data. First we calculated basic usage statistics for the site overall and individual sections (e.g. Research updates, Hot topics, Blogs). We then conducted multivariate analysis of variance (MANOVA) with follow-up univariate tests and Scheffe comparisons to compare usage between key sections. Usage was defined by three indicators: (i) number of unique visits to each page, (ii) average time on page in seconds, and (iii) page bounce rate (i.e. percentage of visits where users left the site immediately after viewing that particular page).
Statistical analyses were then conducted to examine usage for pages with similar topics across all sections. We examined usage for the following: (i) posts relating to improving research skills and/or the use of research in policy (Skills pages) and (ii) posts focused on promoting the value of incorporating research into policy (Value pages). These pages were identified and coded by the first author (SM). MANOVA was conducted to compare usage between Skills/Value pages and other post types (Hot topics, Blogs, Research updates, Events or Multimedia).
Finally, we conducted repeated measures analyses to examine whether the weekly Web CIPHER email improved the ability of users to access the posts on the website. Specifically, we compared the level of usage between the same page accessed (i) directly via an email link, (ii) indirectly (the page was visited from another page accessed by an email link) or (iii) an alternative route (e.g. the Sax Institute website, a search engine or typing in the address directly). In total, there were 303 posts advertised in the weekly Web CIPHER email throughout the study period. To conduct this analysis, each post was examined separately in GA and manually coded to evaluate the number of times individuals accessed that post through each of the above-mentioned access routes. These analyses are described in detail below. Means and standard deviations for the usage data are displayed in Table 2.
A total of 223 policymakers from 27 organisations joined Web CIPHER during the 16-month study period. Six of the organisations accounted for more than 60% of the total target sample. Users held a diverse range of positions within their organisations, although the most commonly reported were directors or deputy/assistant directors (13%), managers (24%), project officers/coordinators (17%), policy officers, advisors, or analysts (12%) and research officers/fellows (7.6%). Other demographic information such as age and sex was not collected from participants.
General usage data revealed that during the 16-month time period analysed, users viewed 8 posts on an average visit, staying on the site for approximately 4 min. The bounce rate was just over 6%.
Usage of Web CIPHER sections
Data revealed that for the 272 Research updates posts on Web CIPHER, each received an average of five unique post views, an average viewing time of 33 s and a bounce rate of less than 1%. Events posts had about four unique post views, an average viewing time of about 31 s and a bounce rate of less than 0.1%. Hot topics and Multimedia posts each had close to three unique post views and very low bounce rates (each less than 0.1%). Each Hot topics post was viewed for about 43 s, and Multimedia posts were viewed for about 23 s. Posts within the Blog section of the website were the most popular, receiving on average almost 13 unique views per post, a bounce rate of less than 0.1% and an average viewing time of 43 s per post. Community posts were viewed an average of 12 times and for 27 s each. The bounce rate again was less than 0.1%. It should be noted that there were only five posts from non-Sax Institute members on the Bulletin Board, indicating that the Community section was virtually unused by members. The Research portal page received 124 total unique post views, with an average viewing time of 46 s and a 0% bounce rate.
The Research portal section was not included because it was only a single static page. The analyses revealed a significant multivariate main effect of post type. Scheffe follow-up comparisons showed that posts in the Blog section received a higher average number of unique views than all other types of posts (p < .05 for each section comparison) except Community posts (p > .05). There were no significant differences between Research updates and Multimedia, Events or Hot topics posts in terms of post views, viewing time or bounce rate; between Events posts and Hot topics or Multimedia; or between Hot topics and Multimedia (all ps > .05).
Despite the lack of posts on the Bulletin Board, the Community section had a significantly greater number of unique views relative to posts in the Hot topics, Research updates, Events, and Multimedia sections (p < .05 for each comparison). Therefore, Blogs and Community posts may provide the most effective means of disseminating research to users.
Usage of pages on a specific topic
Pages relating to improving research skills
Fifty three Skills pages were identified. Descriptive statistics revealed that each Skills post had, on average, 7.57 unique post views, with a bounce rate of less than 1% and an average post viewing time of 41.08 s. Multivariate analyses did not reveal any significant differences between Skills posts and other post types.
Since Blogs was the most popular section, we conducted an exploratory two-way MANOVA to see whether Blogs focused on Skills were more popular than either Skills or Blogs posts on their own. The two-way MANOVA revealed a non-significant main effect for Skills posts, F(1, 1264) = 1.68, p = .14, ηp2 = .01, and a significant main effect for Blogs posts, F(5, 1264) = 5.40, p < .01, ηp2 = .02. Interestingly, however, the Skills by Blog interaction effect approached significance, F(5, 1264) = 1.98, p = .08, ηp2 = .01. Examining univariate tests revealed that this interaction effect was marginally significant for average viewing time per post, F(5, 1268) = 3.09, p < .08, ηp2 = .002. This indicates that although Blogs received longer average viewing times relative to other posts, this effect was greater for Skills posts than non-Skills posts. In other words, viewing times were greater for posts that combined Skills with Blogs (M = 79.12 s, SD = 23.97), compared to Skills (M = 29.93 s, SD = 12.97) or Blog posts alone (M = 34.53 s, SD = 11.99). These findings suggest that Blogs may provide an effective means of conveying advice and building policymakers’ research skills.
Pages related to the value of research in health policy
Eighty-eight Value pages were identified. Descriptive statistics revealed that each Value-type post had, on average, six unique post views, with a bounce rate of less than 1% (M < .001, SD = .002) and an average post viewing time of 41.08 s (SD = 67.27). Multivariate analyses comparing Value pages to other posts revealed a multivariate main effect, F(5, 1266) = 3.102, p < .01, ηp2 = .01. Univariate tests revealed viewing time was significantly greater for Value posts relative to other posts. The findings suggest that policy maker members of Web CIPHER are quite engaged with posts that support the use of research in policymaking.
Impact of the weekly Web CIPHER email on usage
Repeated measures analysis of all Web CIPHER posts that were delivered via the weekly newsletter email, revealed a significant main effect of access route, F(2, 301) = 5.34, p = .005, ηp2 = .03. Scheffé comparisons showed that posts received significantly more unique views via a direct email link or indirect email link relative to an alternative route (all ps < .01; these access routes are described in the ‘Method’ section). However, the number of unique views did not differ between direct and indirect email links (p = .16). A similar result emerged when we just focused on research-related posts (i.e. posts from within the Research updates and Hot topics section), although this time, posts received significantly more unique views via a direct link, relative to an indirect link (p < .01).
An exploratory analysis was undertaken to determine if the effect was greater for Research posts (Research updates and Hot topics posts) relative to Other posts (Blogs, Community, Multimedia, Events). A 2 (type of post: Research versus Other) × 3 (access route: direct email link, indirect, alternative) MANOVA revealed a significant main effect of post type.
Our findings suggest that Web CIPHER has the potential to increase the extent to which policymakers engage with research. Usage statistics revealed that for the 223 site members, average viewing time was greater than 35 s, the bounce rate was well below 5% in all sections and users typically viewed more than one page per visit. These statistics are positive given that the average bounce rate for most websites falls within 40%–60% and that website users, on average, leave web pages within 10–20 s .
Online Knowledge Platforms are increasingly being used by organisations to improve performance and best practise by overcoming barriers relating to information access, storage, retrieval and exchange [35-37]. Evidence indicates that effective Knowledge Platforms (i) are easy to use and (ii) provide comprehensive, up-to-date, relevant and easily accessible information [34,38]. Web CIPHER aims to apply these features in order to improve the extent to which health policymakers engage with research. The presentation of relevant research in multiple formats (e.g. summaries, blogs, multimedia, full-text articles) through an easy-to-use website and a weekly email was designed to ensure research findings were easily accessible, appropriately disseminated, digestible, engaging, timely and relevant to the local context. Because membership to Web CIPHER is free, there were no financial and only a few organisational barriers to accessing and using the site over the study period.
Of greatest interest was the result demonstrating that Blogs and Community posts were the most popular sections on the site. This indicates users were keen to read the points of view of experts in health policy and research. Blog posts were helpful in increasing viewing time for research skill-related posts (relative to non-skill-related posts). This indicates that blog posts may provide a medium for conveying advice to policymakers about ways to integrate research into policy, which may to some extent address a key barrier to EIHP: policymakers not having the skills to access, appraise and apply research to policy [25,31,32,51].
There are numerous possible explanations as to why the Blogs were particularly popular. They were written by trusted figures in health policy and research. This increased sense of trust and credibility may have motivated users to not only read the Blogs, but to access the other key sections of the website. Research suggests Knowledge Platforms are more likely to be utilised by staff if perceived as credible and championed by trusted and senior individuals [34,35]. Having blog posts written by trusted figures may also have contributed to a sense of community, trust, and social presence on the site, all of which are key predictors of website use [52-56]. Further, the fact that Blogs were written by familiar and trusted figures could potentially overcome negative attitudes reportedly held by some policymakers towards the quality and credibility of research and researchers [24,27].
The Blogs in Web CIPHER were also written in a way that addresses some of the research needs of policymakers . The authors strove to write Blogs in a non-intimidating and conversational tone, with minimal use of jargon. They were presented in a clear, concise and easy-to-use format, akin to the style of newspaper reports or editorials. Importantly, they often summarised relevant research findings and provided clear and concrete recommendations to policymakers. Numerous studies indicate that these features are key facilitators of evidence-informed health policymaking [18,26-28,57]. Furthermore, research in website usage has shown that the perceived ease of use of a website and the usefulness of the information it provides are significant predictors of its use [58-60]. In sum, Web CIPHER Blogs were written in a way that addressed many of the aforementioned barriers to research use such as poor formatting and presentation of research findings, excessive use of jargon, limitations in policymakers’ research skills and a lack of clarity, timeliness and policy relevance of research [25,31,51].
The fact that Community Bulletin Board posts also received a relatively high number of unique views compared to other sections indicates that members are keen to read about the points of view of their colleagues and contemporaries. Unfortunately, though, few members posted on the Bulletin Board. Nonetheless, our findings indicate that blogs and bulletin board posts may provide an effective means of conveying or disseminating the latest research, providing advice regarding evidence-informed health policy, building policymakers’ research skills and improving their perceptions towards research use in policy.
In light of these findings, we intend to test strategies to improve Web CIPHER’s capacity to improve policymakers’ engagement with and use of research in policy. Specifically, we aim to capitalise on the most popular feature of the site by increasing the number of blog posts. Further, we will test the impact of strategies such as having leaders from Web CIPHER’s member agencies contribute to these blogs on use of the site amongst other agency staff. Since blogs may provide an effective conduit for engaging users with research, we also intend to link relevant research posts directly to these blogs to determine the impact of this on research post views.
Another key finding was that posts, particularly research-related, were most likely to be accessed from a link provided in the weekly summary email. This reinforces the importance of improving access as a way to facilitate research engagement [21,25,27,30,32,41-43,48,61]. An effective means of building on this strategy may be to invite agency leaders to occasionally guest author the weekly emails sent to Web CIPHER members alerting them to the latest Web CIPHER research and information updates. It is hoped that these leaders and nominated staff will then be seen as champions of research use by the Web CIPHER community. Indeed, studies indicate that use of Knowledge Platforms is related to the extent to which they are championed by organisations’ senior management [34,38]. Such championing is likely to improve users’ perceptions of trust and informational credibility, relevance, social presence and sense of community [52-56] and may ultimately impact on policymakers’ attitudes towards EIHP.
In order to examine these proposed mechanisms more closely, it will be essential in future researches to document policymakers’ views on whether and how these strategies impacted on their use of Web CIPHER. Furthermore, it will be valuable to investigate whether policymakers’ usage of Web CIPHER has to some extent changed the way in which they perceive and engage with research in policy.
Strengths and limitations
The current paper has a number of strengths. In particular, it describes the innovative use of Google Analytics to observe the usage of an online knowledge exchange portal to help policymakers engage with research in policymaking and empirically examine which aspects of the portal were particularly appealing to this audience. We incorporated a number of usage metrics (unique views, average time on page, bounce rate), analysed and compared usage of key relevant sections of the website, identified the most appealing content on the site and evaluated the positive impacts of delivering research content directly to users via email. These analyses have shed light on promising strategies to improve user engagement with Web CIPHER and policymakers’ perceptions towards EIHP. We hope this will be helpful to others as we are not aware of any other research that describes effective ways for organisations to set up similar knowledge exchange portals. We believe, therefore, that our paper makes a significant contribution to understanding the potential benefits of such online portals and potential key strategies that can be implemented to enhance their potential for improving policymaker engagement with research.
Limitations primarily concern the sample on which the analyses were based. Firstly, the number of registered users was relatively small (n = 223). These small numbers were likely impacted by the site’s requirement for users to apply, obtain approval to join the site and to be logged in with a username and password to view content. This was a requirement for analysis, although it is certain to have posed a barrier to site use. Further, as Web CIPHER was still in its pilot phase, access had only been offered to a limited number of Australian policy agencies.
Selection bias may also be an issue for our study sample. Specifically, because membership (and usage) was completely voluntary, our sample was essentially self selected and may have primarily consisted of policymakers with pre-existing interests in research or knowledge exchange. Therefore, our conclusions regarding usage and suggested improvements are necessarily preliminary. Randomised controlled trials would allow us to make more definitive conclusions about what components of web portals are particularly effective at driving usage and improving how policymakers engage with research.
It is possible that the findings observed in the present study are specific to the time frame in which observations were made (i.e. the 16-month period from when the site was first launched in January 2012 until April 2014). Google Analytics data indicates there has been gradual growth in the number of Web CIPHER users after this period. It is possible that the distribution of users will change leading to variations in site usage. It will be important to utilise a longer observation period in future research in order to verify the findings observed in the present study.
There are also limitations to the use of Google Analytics metrics to assess usage, in particular unique page views. A unique page view occurs when a user visits a page in one viewing session. If the user visits the page more than once in a session, it is still only counted as a single unique page view. However, unique page views may be the result of multiple users viewing a page, or a single user visiting the same page across multiple sessions. Therefore, unique views alone cannot accurately indicate the popularity of particular pages or the extent to which they are used by multiple users. It is therefore essential that other metrics are incorporated, which is why we also reported average page viewing time and bounce rate (i.e. the percentage of visitors that view only a single page before exiting the site). These two indices can provide additional indications of the extent to which users are interested with the site . Nonetheless, it is important to note that such online metrics only provide, at best, an estimate of the extent of usage of a particular website.
Like other Knowledge Platforms, Web CIPHER has the potential to improve information access, dissemination and exchange in order to improve policymakers’ engagement with research in policymaking. The current study makes a useful contribution to the literature by providing insights into some of the most effective features of such sites in a health policy environment. Our findings revealed that users were generally engaged with Web CIPHER content, and were particularly drawn to the Blogs and Community sections of the site. The Blogs may have been particularly appealing to users because they addressed topical issues presented by trusted figures in health policy and research and were written in a conversational and engaging tone. A weekly reminder email increased the likelihood that users accessed site content, particularly those posts relating to research. The findings suggest that online knowledge exchange portals may improve policymakers’ engagement with research by incorporating topically relevant blogs written by trusted figures in health policy and research and utilising a weekly reminder email to facilitate access to this content.
We would like to thank Sally Redman and Louisa Jorm for providing valuable feedback on an earlier version of this manuscript. We wish to thank all the policymakers and researchers that directly contributed content on Web CIPHER, particularly those who provided blogs and multimedia. We would also like to acknowledge the chief investigators of the CIPHER Centre for Research Excellence for conceptualising the development of Web CIPHER and other innovations designed to increase the use of research in policymaking. CIPHER is a joint project of the Sax Institute; Australasian Cochrane Centre, Monash University; the University of Newcastle; the University of New South Wales; the University of Technology, Sydney; the Research Unit for Research Utilisation, University of St Andrews and University of Edinburgh; and the University of Western Sydney. Finally, we would like to thank the reviewers for providing valuable comments and suggestions that have strengthened this manuscript.
- Buse K, Mays N, Walt G. Making health policy. Glasgow: Open University Press; 2005.Google Scholar
- LaRocca R, Yost J, Dobbins M, Ciliska D, Butt M. The effectiveness of knowledge translation strategies used in public health: a systematic review. BMC Public Health. 2012;12:751.View ArticlePubMedPubMed CentralGoogle Scholar
- Brownson RC, Chriqui JF, Stamatakis KA. Understanding evidence-based public health policy. Am J Public Health. 2009;99:1576–83.View ArticlePubMedPubMed CentralGoogle Scholar
- Brownson RC, Fielding JE, Maylan CM. Evidence-based public health: a fundamental concept for public health practice. Annu Rev Public Health. 2009;30:175–201.View ArticlePubMedGoogle Scholar
- Hanney SR, Gonzalez-Block MA, Buxton MJ, Kogan M. The utilisation of health research in policy-making: concepts, examples and methods of assessment. Health Res Policy Syst. 2003;1:2.View ArticlePubMedPubMed CentralGoogle Scholar
- Buchan H. Gaps between best evidence and practice: causes for concern. MJA. 2004;180:S48–9.PubMedGoogle Scholar
- Bowen S, Erickson T, Martens PJ, Crockett S. More than “using research”: the real challenges in promoting evidence-informed decision-making. Health Policy. 2009;4:87–102.Google Scholar
- Fielding JE, Briss PA. Promoting evidence-based public health policy: can we have better evidence and more action? Health Aff. 2006;4:969–78.View ArticleGoogle Scholar
- Andre FE, Booy R, Bock HL, Clemens J, Datta SK, John TJ, et al. Vaccination greatly reduces disease, disability, death and inequity worldwide. Bull World Health Organ. 2008;86:140–6.View ArticlePubMedGoogle Scholar
- Hanna JN, Hills SL, Humphreys JL. Impact of hepatitis A vaccination of indigenous children on notifications of hepatitis A in north Queensland. Med J Aust. 2004;181:482–5.PubMedGoogle Scholar
- Morrato EH, Elias M, Gericke CA. Using population-based routine data for evidence-based health policy decisions: lessons from three examples of setting and evaluating national health policy in Australia, the UK and the USA. J Public Health. 2007;19:463–71.View ArticleGoogle Scholar
- Milat A, Laws R, King L, Newson R, Rychetnik L, Rissel C, et al. Policy and practice impacts of applied research: a case study analysis of the New South Wales health promotion demonstration research grants scheme 2000-2006. Health Res Policy Syst. 2013;11:5.View ArticlePubMedPubMed CentralGoogle Scholar
- Ham C, Hunter DJ, Robinson R. Evidence based policymaking: research must inform health policy as well as medical care. BMJ. 1995;310:71–2.View ArticlePubMedPubMed CentralGoogle Scholar
- Orton L, Lloyd-Williams F, Taylor-Robinson D, O'Flaherty M, Capewell S. The use of research evidence in public health decision making processes: systematic review. PLoS ONE. 2011;6:e21704.View ArticlePubMedPubMed CentralGoogle Scholar
- Weiss C. Knowledge creep and decision accretion. Sci Commun. 1980;1:381–404.View ArticleGoogle Scholar
- Elshaug AG, Hiller JE, Tunis SR, Moss JR. Challenged in Australian policy processes for disinvestment from existing, ineffective health care practices. Aust New Zealand Health Policy. 2007;4:23.View ArticlePubMedPubMed CentralGoogle Scholar
- Amara N, Ouimet M, Landry R. New evidence on instrumental, conceptual, and symbolic utilization of university research in government agencies. Sci Commun. 2004;26:75–106.View ArticleGoogle Scholar
- Campbell DM, Redman S, Jorm L, Cooke M, Zwi AB, Rychetnik L. Increasing the use of evidence in health policy: practice and views of policy makers and researchers. Aust New Zealand Health Policy. 2009;6:21.View ArticlePubMedPubMed CentralGoogle Scholar
- Chagnon F, Poullot L, Malo C, Gervais MJ, Pigeon ME. Comparison of determinants of research knowledge utilization by practitioners and administrators in the field of child and family social services. Implement Sci. 2010;5:41.View ArticlePubMedPubMed CentralGoogle Scholar
- El-Jardali F, Lavis JN, Ataya N, Jamal D. Use of health systems and policy research evidence in the health policymaking in eastern Mediterranean countries: views and practices of researchers. Implement Sci. 2012;7:2.View ArticlePubMedPubMed CentralGoogle Scholar
- Ritter A. How do drug policy makers access research evidence? Int J Drug Policy. 2009;20:70–5.View ArticlePubMedGoogle Scholar
- Trostle J, Bronfman M, Langer A. How do researchers influence decision makers? Case studies of Mexican policies. Health Policy Plan. 1999;14:103–14.View ArticlePubMedGoogle Scholar
- Runciman WB, Hunt TD, Hannaford NA, Hibbert PD, Westbrook JI, Coiera EW, et al. CareTrack: assessing the appropriateness of health care delivery in Australia. Med J Aust. 2012;197:100–5.View ArticlePubMedGoogle Scholar
- Contandriopoulos D, Lemire M, Denis JL, Tremblay E. Knowledge exchange processes in organizations and policy arenas: a narrative systematic review of the literature. Milbank Q. 2010;88:444–83.View ArticlePubMedPubMed CentralGoogle Scholar
- Hennink M, Stephenson R. Using research to inform health policy: barriers and strategies in developing countries. J Health Commun. 2006;10:163–80.View ArticleGoogle Scholar
- Petticrew M, Whitehead M, Macintyre SJ, Graham H, Egan M. Evidence for public health policy on inequalities: 1: the reality according to policymakers. J Epidemiol Community Health. 2004;58:811–6.View ArticlePubMedPubMed CentralGoogle Scholar
- Helmsey-Brown J. Facilitating research utilisation: a cross-sector review of research evidence. Int J Public Sect Manage. 2004;17:534–52.View ArticleGoogle Scholar
- Lavis J, Davies H, Oxman A, Denis JL, Golden-Biddle K, Ferlie E. Towards systematic reviews that inform health care management and policy-making. J Health Serv Res Policy. 2005;10(S1):35–48.View ArticlePubMedGoogle Scholar
- Ssengooba F, Atuyambe L, Kiwanuka SN, Puvanachandra P, Glass N, Hyder AA. Research translation to inform national health policies: learning from multiple perspectives in Uganda. BMC Int Health Hum Rights. 2011;11:S13.View ArticlePubMedPubMed CentralGoogle Scholar
- Innvaer S, Vist G, Trommald M, Oxman A. Health policy-makers’ perceptions of their use of evidence: a systematic review. J Health Serv Res Policy. 2002;7:239–44.View ArticlePubMedGoogle Scholar
- Oliver K, Innvar S, Lorenc T, Woodman J, Thomas J. A systematic review of barriers to and facilitators of the use of evidence by policymakers. BMC Health Serv Res. 2014;14:2.View ArticlePubMedPubMed CentralGoogle Scholar
- de Goede J, Putters P, van ger Grinten T, van Oers HAM. Knowledge in process? Exploring barriers between epidemiological research and local health policy development. Health Res Policy Syst. 2010;8:26.View ArticlePubMedPubMed CentralGoogle Scholar
- Ellen ME, Lavis JN, Ouimet M, Grimshaw J, Bedard PO. Determining research knowledge infrastructure for healthcare systems: a qualitative study. Implement Sci. 2011;6:60.View ArticlePubMedPubMed CentralGoogle Scholar
- Purvis RL, Sambamurthy V, Zmud RW. The assimilation of knowledge platforms in organizations: an empirical investigation. Organ Sci. 2001;12:117–35.View ArticleGoogle Scholar
- Lazaric N, Longhi C, Thomas C. Gatekeepers of knowledge versus platforms of knowledge: from potential to realized absorptive capacity. Reg Stud. 2008;42:837–52.View ArticleGoogle Scholar
- Mitton C, Adair CE, McKenzie E, Patten SB, Waye PB. Knowledge transfer and exchange: review and synthesis of the literature. Milbank Q. 2007;85:729–68.View ArticlePubMedPubMed CentralGoogle Scholar
- Alavi M, Leidner DE. Knowledge management and knowledge management systems: conceptual foundations and research issues. MIS Q. 2001;25:107–36.View ArticleGoogle Scholar
- O'Donnell S, Cranney A, Jacobson MJ, Graham ID, O'Connor AM, Tugwell P. Understanding and overcoming the barriers of implementing patient decision aids in clinical practice. J Eval Clin Pract. 2006;23:174–81.View ArticleGoogle Scholar
- Forsetlund L, Bradley P, Forsen L, Nordheim L, Jamtvedt G, Bjorndal A. Randomised controlled trial of a theoretically grounded tailored intervention to diffuse evidence-based public health practice [ISRCTN23257060]. BMC Med Educ. 2003;3:2.View ArticlePubMedPubMed CentralGoogle Scholar
- Di Noia J, Schwinn TM, Dastur ZA, Schinke SP. The relative efficacy of pamphlets, CD-ROM, and the Internet for disseminating adolescent drug abuse prevention programs: an exploratory study. Prev Med. 2003;37:646–53.View ArticlePubMedPubMed CentralGoogle Scholar
- Dobbins M, Cockerill R, Barnsley J. Factors affecting the utilization of systematic reviews: a study of public health decision makers. Int J Technol Assess Health Care. 2001;17:203–14.View ArticlePubMedGoogle Scholar
- Dobbins M, Rosenbaum P, Plews N, Law M, Fysh A. Information transfer: what do decision makers want and need from researchers? Implement Sci. 2007;2:20.View ArticlePubMedPubMed CentralGoogle Scholar
- Gold M. Pathways to the use of health services research in policy. Health Res Educ Trust. 2009;44:1111–36.Google Scholar
- Whitehead M, Petticrew M, Graham H, Macintyre S, Bambra C, Egan M. Evidence for public health policy on inequalities 2: assembling the evidence jigsaw. J Epidemiol Community Health. 2004;58:817–21.View ArticlePubMedPubMed CentralGoogle Scholar
- Dobbins M, DeCorby K, Twiddy RN. A knowledge transfer strategy for public health decision makers. Worldviews Evid Based Nurs. 2004;1(2):120–8.View ArticlePubMedGoogle Scholar
- Denis JL, Lomas J, Stipich N. Creating receptor capacity for research in the health system: the Executive Training for Research Application (EXTRA) program in Canada. J Health Serv Res Policy. 2008;13 Suppl 1:1–7.View ArticlePubMedGoogle Scholar
- Taylor RS, Reeves BC, Ewings PE, Taylor RJ. Critical appraisal skills training for health care professionals: a randomized controlled trial [ISRCTN46272378]. BMC Med Educ. 2004;4:30.View ArticlePubMedPubMed CentralGoogle Scholar
- Ellen ME, Leon G, Bouchard G, Lavis JN, Ouimet M, Grimshaw JM. What supports do health system organizations have in place to facilitate evidence-informed decision-making? A qualitative study Implement Sci. 2013;8:84.View ArticlePubMedGoogle Scholar
- Hyder AA, Corluka A, Winch PJ, El-Shinnawy A, Ghassany H, Malekafzali H, et al. National policymakers speak out: are researchers giving them what they need? Health Policy Plan. 2011;26:73–82.View ArticlePubMedGoogle Scholar
- Cutroni J. Google Analytics. Sebastopol: O'Reilly Media; 2010.Google Scholar
- McCaughan D, Thompson C, Cullum N, Sheldon TA, Thompson DR. Acute care nurses’ perceptions of barriers to using research information in clinical decision-making. Issues Innov Nurs Pract. 2002;39:46–60.Google Scholar
- Braun MT. Obstacles to social networking website use among older adults. Comput Hum Behav. 2013;29:673–80.View ArticleGoogle Scholar
- Chen R. Member use of social networking sites - an empirical examination. Decis Support Syst. 2013;54:1219–27.View ArticleGoogle Scholar
- Li D, Browne GJ, Chau PYK. An empirical investigation of web site use using a commitment-based model. Decis Sci. 2006;37:427–44.View ArticleGoogle Scholar
- Mantymaki M, Merikivi J. Investigating the drivers of the continuous use of social virtual worlds. In: Proceedings of the 43rd Hawaii international conference on system sciences: 5-8 June 2010 2010. Koloa Kauai, Hawaii: IEEE Computer Society; 2010. p. 1–10.View ArticleGoogle Scholar
- Sledgianowski D, Kuliviwat S. Using social network sites: the effects of playfulness, critical mass and trust in a hedonic context. J Comput Inf Syst. 2009;49:74–83.Google Scholar
- Ettelt S, Mays N. Health services research in Europe and its use for informing policy. J Health Serv Res Policy. 2011;16:48–60.View ArticlePubMedGoogle Scholar
- Chang PV. The validity of an extended Technology Acceptance Model (TAM) for predicting intranet/portal usage. Chapel Hill, North Carolina: University of North Carolina; 2004.Google Scholar
- Shih HP. Extended technology acceptance model of Internet utilization behavior. Inf Manag. 2004;41:719–29.View ArticleGoogle Scholar
- van der Heijden H. Factors influencing the usage of websites: the case of a generic portal in the Netherlands. Inf Manage. 2003;40(6):541–9.View ArticleGoogle Scholar
- Patera N, Wild C. Linking public health research with policy and practice in three European countries. J Public Health. 2013;21:473–9.View ArticleGoogle Scholar
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly credited. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.