- Systematic review
- Open Access
What is the effectiveness of printed educational materials on primary care physician knowledge, behaviour, and patient outcomes: a systematic review and meta-analyses
Implementation Science volume 10, Article number: 164 (2015)
Printed educational materials (PEMs) are commonly used simple interventions that can be used alone or with other interventions to disseminate clinical evidence. They have been shown to have a small effect on health professional behaviour. However, we do not know whether they are effective in primary care. We investigated whether PEMs improve primary care physician (PCP) knowledge, behaviour, and patient outcomes.
We conducted a systematic review of PEMs developed for PCPs. Electronic databases were searched for randomized controlled trials, quasi randomized controlled trials, controlled before and after studies, and interrupted time series. We combined studies using meta-analyses when possible. Statistical heterogeneity was examined, and meta-analysis was performed using a random effects model when significant statistical heterogeneity was present and a fixed effects model otherwise. The template for intervention description and replication (TIDieR) checklist was used to assess the quality of intervention description.
Our search identified 12,439 studies and 40 studies met our inclusion criteria. We combined outcomes from 26 studies in eight meta-analyses. No significant effect was found on clinically important patient outcomes, physician behaviour, or physician cognition when PEMs were compared to usual care. In the 14 studies that could not be included in the meta-analyses, 14 of 71 outcomes were significantly improved following receipt of PEMs compared to usual care. Most studies lacked details needed to replicate the intervention.
PEMs were not effective at improving patient outcomes, knowledge, or behaviour of PCPs. Further trials should explore ways to optimize the intervention and provide detailed information on the design of the materials.
Printed educational materials (PEMs) are a simple, relatively inexpensive knowledge translation (KT) intervention for the dissemination of clinical information (such as clinical practice guidelines, journal articles, or evidence-based PDF or email summaries), aimed at improving the provision of care. A recent Cochrane review found that PEMs may have a small (0.02–0.13 standardized mean difference) beneficial effect on health professional practice outcomes . However, despite continued publication of randomized controlled trials (RCTs) utilizing this intervention, we still know little about which behaviours can be influenced by PEMs, within which settings, and how to optimize the effect of these interventions for various health professionals.
Primary care physicians (PCPs) are required to have a vast and comprehensive knowledge base to treat different patient groups and diseases. On average, they have been observed to have 3.2 questions for every 10 patients they see  but these questions often go unanswered. PEMs are a potential strategy for meeting these needs. Non-interactive PEMs are easy to implement and scale across various primary care clinics. Reviews of the literature have found that printed resources (including books) remain a common source of information for physicians [3–5], with one systematic review finding that 50–80 % of physicians used printed materials for information . However, if there is no demonstrated effectiveness of these interventions on knowledge, behaviour, or patient outcomes when targeted at PCPs, they should not be implemented as behaviour change techniques. To our knowledge, this is the first review to examine the effect of PEMs on PCPs.
Though interactive computer-based KT interventions such as those integrated within electronic health records have been shown to be effective in changing behaviour and are increasingly more popular than non-interactive paper-based interventions, they are expensive and require technological infrastructure and training, obstacles to implementation given limited budgets and overworked clinicians . Surveys show that only 64 % of Canadian PCPs  and 41.5 % of American physicians  use electronic medical records, limiting the reach of complex interventions that are integrated into electronic records and possibly unintentionally leaving PCPs out of these interventions. With many different software vendors being used across practices (for example, there are 14 certified electronic medical record products to date in Canada alone ), creating a one-size-fits-all solution is challenging. As such, PEMs, a non-interactive and low-tech intervention, will likely continue to be used to disseminate new evidence and important clinical information or as a part of multi-component KT interventions.
The objective of this review was to examine what effect PEMs have on PCP knowledge, behaviour, and patient outcomes, in comparison to no intervention or to other single- or multi-component educational interventions. This review contributes to existing literature by examining the effect of interventions specifically designed for PCPs. The primary care setting is considerably different from other health care settings, and PCPs are likely to experience barriers unique to their setting and their scope of practice. Physicians are the population of interest to limit participant heterogeneity as we anticipated that differences in training and role among diverse primary care clinicians may influence behaviour change. PEMs for PCPs may have different content and may target different behaviour than PEMs for other professionals. More importantly, PCPs may respond differently than other clinicians to PEMs, and we anticipate PEMs have a different effect size when targeting behaviour change in different providers. We also examined the quality of reporting of PEM interventions in included studies.
A systematic review protocol was written for this review and registered with PROSPERO, the international prospective register of systematic reviews (registration no. CRD42013004356). We based the methods for this review on those described in the Cochrane Handbook for Systematic Reviews of Interventions and the Cochrane Effective Practice and Organisation of Care Group (EPOC) . It is reported using the PRISMA Statement for Reporting Systematic Reviews .
We included studies reporting the effectiveness of PEMs for PCPs (family physicians as well as specialists practising primary care such as pediatricians), defined for the purpose of the review as guidelines, summaries of guidelines, the dissemination of published or non-published information, recommendations, or evidence presented in print or electronic form. Electronic materials include PDFs, other document files (e.g. Microsoft Word documents), and non-interactive web pages. Studies examining interactive online educational materials (such as online courses) or patient-specific materials were excluded. We included any method of delivery of the intervention (e.g. email, mail, fax) or level of intensity (i.e. how often the intervention was delivered). We limited study designs to RCTs, quasi randomized trials, controlled before and after studies, and interrupted time series (ITS) analyses. We included reported patient outcomes, physician cognition (skills and knowledge), and physician behaviour outcomes. Physician attitudes were not included in the review. No restrictions were placed on publication status or date of publication. Studies not published in English were excluded. Lastly, comparisons had to allow for the isolation of the effect of PEMs on outcomes, meaning that studies comparing two PEMs with no control group or studies comparing two interventions where both groups included PEMs were excluded. Researchers independently and in duplicate screened each title and abstract (AG, SR, RK, DR) and reviewed the full text of selected studies for eligibility using the criteria listed above (AG, SR, RK).
Information sources and search
Studies were identified by a search of electronic databases developed by an information specialist and independently reviewed by a second information specialist. Medline (Ovid MEDLINE(R)), EMBASE (Embase Classic + Embase), ERIC (ProQuest), and the Cochrane Central Register of Controlled Trials were searched on November 25, 2014. The following search terms were used: print, message, book, monograph, pamphlet, journal, educational materials, online, email, web-based, general practitioner, and family physician (see Additional file 1 for Medline search). Appropriate wildcards were used in the search to account for plurals and variations in spelling. The search was supplemented by searching the reference lists of included articles and of other systematic reviews. Authors of studies published within the last 10 years were contacted to collect missing data needed for meta-analyses.
We extracted data independently and in duplicate (AG, RK) using a modified version of the Cochrane EPOC standardized data collection checklist . A calibration exercise was done before screening and data extraction with each of the researchers to ensure consistency. Information was extracted from each study on study design, the intervention, controls, type of targeted behaviour, professional and patient participants, setting, methods, outcomes, costs of the intervention, changes in healthcare costs, and results. All disagreements for screening and extraction were resolved by discussion.
Risk of bias
Risk of bias was assessed independently and in duplicate using the Cochrane EPOC risk of bias assessment tool .
Clinical (e.g. type of population, topic of intervention), methodological (e.g. study duration), and statistical heterogeneity were assessed. For studies with similar clinical and methodological characteristics that reported similar outcomes, data were pooled statistically using meta-analysis, with separate meta-analyses carried out for patient outcomes and for physician outcomes. Physician outcomes were grouped post hoc according to Miller’s framework for clinical assessment into physician behaviour (“does” and “shows how”) and physician cognition (“knows” and “knows how”). Miller’s framework presents four categories, with knowledge (“knows”) at the base to represent that a physician knows what is required to effectively carry out professional functions. Following knowledge is competence, or “knows how”, which captures the skills needed to acquire information, analyze and interpret data, and translate findings into their practice. Performance (“shows how”) follows, where physicians must demonstrate their knowledge within an examination setting, and finally is action (“does”) which is the independent action of the physician in clinical practice . We then grouped the studies within these categories into either binary or continuous outcomes. Only one outcome per study was used in any single meta-analysis to avoid double counting. The study’s primary outcome was chosen when possible. When more than one primary or eligible outcome existed, the outcome included in the meta-analysis was chosen at random. A meta-analysis was conducted when outcomes from two studies could be pooled using R software and the Metafor Package [13, 14].
Statistical heterogeneity was examined using the I 2 statistic. A random-effects meta-analysis was conducted when heterogeneity was statistically significant; otherwise, a fixed-effects model was used. When I 2 was significant, we conducted post hoc sub-analyses to explore the source of heterogeneity. For continuous outcomes, a standard mean difference (SMD) was calculated because outcomes were measured using different scales. For binary outcomes, a relative risk (RR) was used as an effect measure. Studies were adjusted for clustering when applicable using study-reported intracluster coefficients (ICCs). When not provided in the study, an ICC was selected from the University of Aberdeen Database of ICCs . When a range of possible ICCs was provided in the literature, sensitivity analysis was performed using multiple adjustments to determine sensitivity of the pooled estimates with respect to ICC values. For meta-analyses of continuous outcomes, when a standard deviation (SD) or standard error was not provided, it was imputed from similar outcomes within the study. Sensitivity analysis was also performed with respect to the imputed standard errors. Studies that did not provide sufficient data for the calculation of a summary statistic were not included in the meta-analysis. A narrative description was used to present the data that could not be pooled.
Studies were assessed against the template for intervention description and replication (TIDieR checklist) to determine the completeness of reporting and replicability of interventions in the included studies . The checklist was applied from the perspective of the intervention of interest for our review, which may have been one of several interventions tested or used as control.
From the 16,735 articles retrieved, the literature search resulted in 12,439 citations (duplicates removed) and 146 potentially relevant citations. Forty studies and two companion reports met eligibility criteria. See Fig. 1 for a flow diagram representing identification of eligible studies .
Thirty-seven of the included studies were RCTs, and three were studies with an ITS design. Studies were conducted between 1983 and 2014 in the USA [17–27], UK [28–36], Canada [37–44], Australia [45–47], Germany [48–50], the Netherlands , Denmark , Brazil , Switzerland , Norway , and Italy . A wide range of intervention topics was observed in our systematic review. Study characteristics are outlined in Table 1.
Very few studies provided details on the study population such as age and time since graduation. Fourteen studies provided information on age of the participants, and nine studies provided information on time since graduation from medical school. The number of physician participants in each study ranged from 8 to 5048. All studies were composed of a majority of family physicians, with some studies including pediatricians , general internists [17, 20, 26, 40, 48], and a small number of other specialists [17, 20, 48]. Three [19, 28, 47] of the studies included in this review examined the cost of the intervention, with two studies [31, 47] calculating the direct impact of the interventions on health care costs. Avorn and Soumerai  found a cost reduction of $105 (US dollars) per physician over 9 months for their academic detailing intervention but did not calculate any savings for the PEM (used as control). Watson et al.  calculated the costs of the guideline intervention and found no significant reduction in prescribing costs. French et al.  conducted a cost-effectiveness analysis that found that a more active intervention (including a workshop) was less expensive than the existing standard PEM dissemination strategy. However, savings in health gains and service reductions were not sufficient to make the active strategy cost-effective. No studies looked at changes in non-health care costs.
Of the RCTs, 21 studies compared PEMs to usual care [17, 18, 21, 22, 24, 29, 30, 35, 38, 40–42, 45, 48–55], 12 to both usual care and another intervention [19, 20, 23, 25, 28, 33, 34, 39, 43, 44, 46, 56], and four to another intervention [26, 27, 37, 47].
Several types of interventions were included such as guidelines [32, 37, 44, 47, 49, 56], guideline summaries or sections of guidelines [29, 46], practice recommendations [19, 21, 22, 24, 27, 28, 31, 36, 39, 43, 48, 55], educational or information packages [20, 34, 35, 40, 42, 45, 53], manuals [23, 25, 50], bulletins [38, 51], clinical vignettes , a journal summary , a checklist with recommendations , an electronic case analysis , an algorithm , a poster , and a journal article . One study was not clear whether the intervention was a complete guideline or summary .
Reporting of interventions (TIDieR checklist)
All studies defined or provided a name for their intervention and the majority (80 %, N = 32) of the studies provided a rationale, theory, or goal of the intervention. All but three studies described the materials in the intervention; 32.5 % (N = 13) of the studies made the materials available in the paper or in supplementary materials or websites. Two studies provided web links or references to the PEMs that did not work or were not available; these were counted as materials not provided [18, 26]. Thirty-three studies described the intervention procedures, 35 described the location of the intervention, and 26 described the timeframe and number of times the intervention was delivered. Fifteen percent (N = 6) of the studies addressed planned measurement of adherence and 22.5 % (N = 9) studies measured adherence to the intervention (e.g. receipt of materials, reading of materials). Details on reporting of interventions are provided in Additional file 2.
Outcomes measured and behaviour targeted
Eight studies measured one or more patient outcomes as a result of the PCP-targeted intervention [18, 21, 25, 37, 40, 42, 46, 53]. The majority of included studies (77.5 %, N = 31) measured physician behaviour outcomes [17, 19, 20, 22–25, 27–33, 35–42, 44, 46–48, 50, 51, 53, 56, 58], and 25 % (N = 10) of the studies measured physician cognition outcomes [26, 34, 45–47, 49, 51, 52, 54, 55]. Additional File 3.
Risk of bias of included studies
Using the Cochrane EPOC risk of bias assessment tool, 33 RCTs had unclear or high risk of bias for at least two criteria. Two RCTs were appraised as low risk of bias on eight of nine criteria [38, 43], and only two RCTs were appraised as having low risk of bias on all nine criteria [41, 42] (See Additional file 4). The ITS studies had at least two unclear risk of bias out of seven criteria. Figure 2 illustrates the percentage of studies at low, high, and uncertain risk of bias.
A meta-analysis of dichotomous patient outcomes was conducted with five studies and 935,252 patients [18, 25, 40, 42, 53], combined using a fixed-effects model. The summary statistic demonstrated no improvement with use of a PEM, RR = 1.00 (95 % CI = 1.00,1.00, I 2 = 0). One study  was allocated 100 % of the weight in the analysis; therefore, we completed a sensitivity analysis without the study for a total of 1463 patients, which resulted in a small but statistically insignificant improvement, RR = 1.09 (95 % CI = 0.91,1.29, I 2 = 0).
Physician behaviour outcomes
Five meta-analyses of studies reporting impact on physician behaviour outcomes were conducted. A meta-analysis of continuous physician outcomes was conducted with four studies and 531 physicians [24, 25, 28, 35]. The results were not statistically significant, SMD = 0.35 (95 % CI = −0.06, 0.76, I 2 = 67.88 %). Sub-analyses grouping studies by similar topic, type of PEM, length of intervention, type of participants, and the behaviour targeted found that the PEM topic was the largest source of heterogeneity (see Additional File 6). Three studies assessing impact on continuous physician outcomes per patient including 1106 patients were combined [22, 33, 41]. No statistically significant result was found, SMD = 0.27 (95 % CI = −0.03, 0.57, I 2 = 68.74 %). Sub-analyses grouping studies by similar type of intervention, risk of bias, and behaviour targeted found that the behaviour targeted was the largest source of heterogeneity.
A meta-analysis of dichotomous physician behaviour outcomes was conducted with results from three studies including 999 physicians [20, 46, 50]. No statistically significant improvement was observed with the use of PEM, RR = 1.01 (95 % CI = 0.96, 1.07, I 2 = 0.00 %). Nine studies assessing impact of PEMs on dichotomous physician behaviour outcomes per patient including 3,273,788 patients were combined [22, 38, 40, 42, 44, 48, 53, 56, 58]. No statistically significant improvement was observed with PEM, RR = 0.99 (95 % CI = 0.97, 1.01, I 2 = 0.00 %). Two studies assessing dichotomous physician behaviour outcomes of PEMs compared to workshops were combined including 153,089 patients [37, 47]. No statistically significant difference was found between the two interventions, RR = 0.57 (95 % CI = 0.12, 2.73, I 2 = 64.48 %).
Physician cognition outcomes
Three studies (438 physicians) were combined in a meta-analysis of studies reporting continuous physician cognition outcomes [45–47]. No statistically significant effect was found, SMD = 0.65 (95 % CI = −0.21, 1.51, I 2 = 89.75 %). Three studies (806 physicians) were combined in a meta-analysis of dichotomous physician cognition outcomes [34, 51, 52], and no significant effect was found, RR = 1.51 (95 % CI = 0.90, 2.52, I 2 = 90.27 %). The high statistical heterogeneity for both cognitive meta-analyses could not be examined. Subgroup analyses could not be performed as all studies varied in topic, method of knowledge measurement, type of intervention, and length of intervention.
Fourteen studies examined 71 outcomes that were not included in the meta-analyses due to heterogeneity in study outcomes (i.e. outcomes reported could not be pooled with any other study), study design (i.e. ITS studies were not pooled with RCTs), or missing data (i.e. studies did not provide sufficient data and authors could not be reached) (see Additional file 3 for all study outcomes). The results are presented below by type of outcome.
One study  examined the effect of PEMs on four clinical patient outcomes compared to usual care. A significant effect was found for only one outcome (physical functioning of patients with multisomatoform disorder).
Nine studies [17, 19, 23, 29–32, 36, 39] examined the effect of PEMs on 55 physician behaviour outcomes compared to usual care. Bearcroft et al.  found a significant effect of PEMs on all four outcomes studied in chest radiography referral, including concordance with guidelines and documentation of history, clinical diagnosis, and smoking history. Guthrie et al.  found a significant effect of PEMs on six out of 12 outcomes examining prescriptions of antipsychotics. Oakeshott et al.  found a significant improvement with PEMs on total radiology requests but not when examined by X-ray type (limbs, chest, or spine). Worrall et al.  found a significant improvement with PEMs on two of five prescribing outcomes. No effect was found on outcomes in five other studies [17, 19, 23, 32, 39].
One study  examined the effect of PEMs compared to a web course on eight physician behaviour outcomes. There was a significant difference between the two interventions for three of the eight outcomes (overall shared decision-making, guiding in decision-making, stating that a PSA test would be ordered).
We included 40 studies and conducted eight meta-analyses with data from 26 studies. The reported quality of evidence ranged from low to high and many studies were unclear on important methodological factors such as allocation sequence generation (15 studies unclear), baseline outcome measurement (15 studies unclear), and baseline characteristic measurement (nine studies unclear). No significant effect was seen across the eight meta-analyses. However, six of the meta-analyses had three or fewer studies included.
Statistical heterogeneity was high for both physician cognition meta-analyses, possibly due to different ways of measuring knowledge outcomes, the different topic areas, duration of the intervention, risk of bias, or the type of PEM intervention. Clinical heterogeneity was visible across studies in all meta-analyses as the topic of the interventions varied greatly. Behaviour change of PCPs may vary by clinical area addressed by the intervention. Though heterogeneity in the population was limited by the review’s focus on PCPs, it was difficult to determine how the population varied across studies due to limited reporting on participant age, time since graduation, and academic affiliation. Methodological heterogeneity was also a concern as study durations varied widely and many factors could not be assessed such as how the intervention was delivered, the source of the evidence, setting of the participant’s clinical practice, and participant demographic details.
The narrative synthesis examined 71 study-level outcomes across three outcome categories. When compared to usual care, PEMs resulted in significant improvement in outcomes for one of four clinical patient outcomes, 13 physician behaviour outcomes, and no physician cognition outcomes. Though it is difficult to draw conclusions from the narrative syntheses of these outcomes, it is evident that a large majority of the measured outcomes did not statistically improve as a result of PEMs. These results differ from the review by Giguère and colleagues , which included all health professionals, and found a median improvement in SMD for categorical practice outcomes of 0.02 and a median improvement in SMD for continuous professional outcomes of 0.13 when compared to usual care. Only 11 of our 40 studies overlapped with the Giguère review. Our results suggest that the effect found in the review by Giguère and colleagues may not be applicable to PCPs and that the effect was a result of interventions targeted at specialist physicians. Most of the studies included in the Cochrane review (42 of 45) included physicians, though the clinical speciality was not clear in many of the included studies and no subgroup analyses were conducted by type of health professional. The difference in effect size may be a result of differences in PEM content between PCP and other clinician PEMs, in the evidence needs of PCPs, in the practice context, or in the behaviours targeted by the PEM. The methods used to synthesize data in the Cochrane review included the use of more than one outcome per study and may have resulted in an inflated effect size. The authors of the Cochrane review also used a different approach when more than one measure was provided in a study by abstracting the median measure whereas we used a random selection process. Furthermore, our review builds on the existing literature as it examines the intervention’s effect on knowledge, an outcome that was excluded from the Cochrane review.
Though it is assumed that the cost of these interventions is small relative to interactive educational tools or active evidence-dissemination strategies, the costs need to be systematically assessed. Three of the studies included in this review provided an economic analysis, but two of the three studies provided costs for a more active intervention and not for the control PEM. Before widely disseminating any behaviour change intervention, information on cost effectiveness of the intervention is critical.
Intervention reporting TIDieR checklist
One study  met all criteria of the TIDieR checklist, and three studies met all but one of the 12 criteria [41, 43, 49]. This is a concerning finding given the relative simplicity of PEM interventions. However, it may be this perceived simplicity that has discouraged study authors from describing the intervention and its development. Improvement in intervention reporting is needed, such as providing access to the intervention materials through appendices or permanent web links. Most study authors noted that they did not know if PEMs were received and read, a significant limitation of the included studies. Other key areas for improvement in reporting of interventions are detailing who provided the intervention, describing how the intervention was delivered, and specifying the date the intervention was delivered.
Differences in outcome measurement and limited reporting resulted in the inability to pool the results for a number of studies. A small number of trials were eligible for each meta-analysis, with one of the eight meta-analyses only pooling two studies, limiting our ability to conduct a meta-regression to explore effect modifiers. It is possible that the meta-analyses had too few included outcomes to detect the small differences that may be expected from PEMs. The physician cognition meta-analyses may be limited in their usefulness due to high statistical heterogeneity. The review may also have been limited by only including English-language studies.
This systematic review has found that PEMs do not improve patient, PCP behaviour, or PCP knowledge outcomes. A previous Cochrane systematic review of 45 studies (14 RCTs and 31 ITSs) noted that similar, passive dissemination strategies have small to moderate effects (1). However, the Cochrane review included all health care professionals, excluded knowledge outcomes, and combined more than one outcome per study (possibly inflating the result) in its meta-analyses.
These findings are of particular importance to those interested in knowledge translation (guideline developers, quality agencies, government agencies, pharmaceutical companies, etc.) and in developing, disseminating, and evaluating PEM interventions. Individuals and groups interested in developing PEMs should note that in their current state, there is no evidence that PEMs are effective at improving PCP outcomes. However, it is difficult to ascertain whether the PEMs tested have been optimized and whether adequate reach of the intervention has been achieved. There are too few details in published studies on how these materials were developed (use of theory, use of evidence-based design, involvement of the end-user) and too limited descriptions of the materials. The results also have implications for research funding bodies. Continuing to fund studies on PEMs with little description of the intervention, inadequate power, or a lack of optimization of the intervention is a poor use of resources. Researchers interested in conducting future studies of PEMs for primary care should invest resources in better design of the tools and should provide detailed descriptions of the intervention to determine if PEM optimization can result in improved outcomes. Trials should be sufficiently powered to detect the small effect of these interventions. Head to head studies of different designs of PEMs may help us understand how certain design elements may contribute to effectiveness, though these studies should continue to include usual care comparisons. Otherwise, we should accept the role of PEMs as simple tools for the dissemination of information or as a part of other more interactive interventions, rather than as tools to influence knowledge and behaviour of PCPs.
primary care physician
printed educational materials
Randomized controlled trials
Interrupted time series
Standard mean difference
Intracluster correlation coefficient
Giguère A, Légaré F, Grimshaw J, Turcotte S, Fiander M, Grudniewicz A, et al. Printed educational materials: effects on professional practice and healthcare outcomes (review). Cochrane Database Syst Rev. 2012;10:CD004398.
Ely JW, Osheroff JA, Ebell MH, Bergus GR, Levy BT, Chambliss ML, et al. Analysis of questions asked by family doctors regarding patient care. BMJ. 1999;319:358–61.
Dawes M, Sampson U. Knowledge management in clinical practice: a systematic review of information seeking behavior in physicians. Int J Med Inform. 2003;71:9–15.
Coumou HC, Meijman FJ. How do primary care physicians seek answers to clinical questions? A literature review. J Med Libr Assoc. 2006;94:55–60.
Davies K. The information-seeking behaviour of doctors: a review of the evidence. Health Info Libr J. 2007;24:78–94.
Grimshaw JM, Shirran L, Thomas R, Mowatt G, Fraser C, Bero L, et al. Changing provider behavior an overview of systematic reviews of interventions. Med Care. 2001;39(8, SII):II2–45.
Canadian Medical Association. The Royal College of Physicians and Surgeons: National Physician Survey; Mississauga, Ontario (Canada). 2013.
Hsiao C-J, Beatty PC, Hing ES, Woodwell DA, Rechtsteiner EA, Sisk JE. Electronic medical record/electronic health record use by office-based physicians: United States, 2008 and preliminary 2009: National Center for Health Statistics; Hyattsville, MD (United States). 2009.
Infoway certified products. [https://www.infoway-inforoute.ca/index.php/programs-services/certification-services/infoway-certified-products]
Cochrane effective practice and organization of care group. [http://epoc.cochrane.org/].
Liberati A, Altman DG, Tetzlaff J, Mulrow C, Ioannidis JPA, Clarke M, et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. Ann Intern Med. 2009;151.
Miller GE. The assessment of clinical skills/competence/performance. Acad Med. 1990;65(9 Suppl):S63–7.
R Core Team. R: a language and environment for statistical computing. R Foundation for Statistical Computing. Vienna. 2012. [http://www.r-project.org/].
Viechtbauer W. Conducting meta-analyses in R with the metafor package. J Stat Softw. 2010;36:1–48.
Database of ICCs. [http://www.abdn.ac.uk/hsru/research/delivery/behaviour/methodological-research/].
Hoffmann TC, Glasziou PP, Boutron I, Milne R, Perera R, Moher D, et al. Better reporting of interventions: template for intervention description and replication (TIDieR) checklist and guide. BMJ. 2014;348:g1687.
Bjornson DC, Rector TS, Daniels CE, Wertheimer AI, Snowdon DA, Litman TJ. Impact of drug-use review program intervention on prescribing after publication of a randomized clinical trial. Am J Hosp Pharm. 1990;47:1541–6.
Hazard RG, Haugh LD, Reid S, McFarlane G, MacDonald L. Early physician notification of patient disability risk and clinical guidelines after low back injury: a randomized, controlled trial. Spine (Phila Pa 1976). 1997;22:2951–8.
Avorn J, Soumerai S. Improving drug-therapy decisions through educational outreach. N Engl J Med. 1983;308:1457–63.
Rabin DL, Boekeloo BO, Marx ES, Bowman MA, Russell NK, Willis AG. Improving office-based physicians’ prevention practices for sexually transmitted diseases. Ann Intern Med. 1994;121:513–9.
Dickinson WP, Dickinson LM, de Gruy F, Main DS, Candib LM, Rost K. A randomized clinical trial of a care recommendation letter intervention for somatization in primary care. Ann Fam Med. 2003;1:228–35.
Guadagnoli E, Normand S-LT, DiSalvo TG, Palmer RH, McNeil BJ. Effects of treatment recommendations and specialist intervention on care provided by primary care physicians to patients with myocardial infarction or heart failure. Am J Med. 2004;117:371–9.
Tziraki C, Graubard BI, Manley M, Kosary C, Moler JE, Edwards BK. Effect of training on adoption of cancer prevention nutrition-related activities by primary care practices: results of a randomized, controlled study. J Gen Intern Med. 2000;15:155–62.
Nicholas J, Dennison BA, De Long R, Prokorym M, Brissette I. Randomized controlled trial of a mailed toolkit to increase use of body mass index percentiles to screen for childhood obesity. Prev Chronic Dis Public Heal Res Pract Policy. 2009;6(4):A122.
Kottke TE, Brekke ML, Solberg LI, Hughes JR. A randomized trial to increase smoking intervention by physicians, doctors helping smokers, round 1. JAMA. 1989;14:2101–6.
Mukohara K, Schwartz MD. Electronic delivery of research summaries for academic generalist doctors: a randomised trial of an educational intervention. Med Educ. 2005;39:402–9.
Feng B, Srinvisan M, Hoffman JR, Rainwater JA, Griffin E, Dragojevic M, et al. Prostate cancer screening: analysis of unannounced standardized patient visits. Ann Fam Med. 2013;11:315–23.
Watson M, Gunnell D, Peters T, Brookes S, Sharp D. Guidelines and educational outreach visits from community pharmacists to improve prescribing in general practice: a randomised controlled trial. J Health Serv Res Policy. 2001;6:207–13.
Oakeshott P, Kerry SM, Williams JE. Randomized controlled trial of the effect of the Royal College of Radiologists’ guidelines on general practitioners’ referrals for radiographic examination. Br J Gen Pract. 1994;44:197–200.
Bearcroft P, Small J, Flower C. Chest radiography guidelines for general practitioners: a practical approach. Clin Radiol. 1994;49:56–8.
Wright NMJ, Roberts AJ, Allgar VL, Tompkins CNE, Greenwood DC, Laurence G. Impact of the CSM advice on thioridazine on general practitioner prescribing behaviour in Leeds: time series analysis. Br J Gen Pract. 2004;54:370–3.
Matowe L, Ramsay CR, Grimshaw JM, Gilbert FJ, Macleod M-J, Needham G. Effects of mailed dissemination of the Royal College of Radiologists’ guidelines on general practitioner referrals for radiography: a time series analysis. Clin Radiol. 2002;57:575–8.
Downs M, Turner S, Bryans M, Wilcock J, Keady J, Levin E, et al. Effectiveness of educational interventions in improving detection and management of dementia in primary care: a cluster randomized controlled study. BMJ. 2006;332.
Watson E, Clements A, Yudkin P, Rose P, Bukach C, Mackay J, et al. Evaluation of the impact of two educational interventions on GP management of familial breast/ovarian cancer cases: a cluster randomised controlled trial. Br J Gen Pract. 2001;51:817–21.
Mcewen A, Preston A, West R. Effect of a GP desktop resource on smoking cessation activities of general practitioners. Addiction. 2001;97:595–7.
Guthrie B, Clark SA, Reynish EL, McCowan C, Morales DR. Differential impact of two risk communications on antipsychotic prescribing to people with dementia in Scotland: segmented regression time series analysis 2001–2011. PLoS ONE. 2013;8(7):e68976.
Worrall G, Angel J, Chaulk P, Clarke C, Robbins M. Effectiveness of an educational strategy to improve family physicians’ detection and management of depression: a randomized controlled trial. CMAJ. 1999;161:37–40.
Dormuth CR, Maclure M, Bassett K, Jauca C, Whiteside C, Wright JM. Effect of periodic letters on evidence-based drug therapy on prescribing behaviour: a randomized trial. CMAJ. 2004;171:1057–61.
Rahme E, Choquette D, Beaulieu M, Bessette L, Joseph L, Toubouti Y, et al. Impact of a general practitioner educational intervention on osteoarthritis treatment in an elderly population. Am J Med. 2005;118:1262–70.
Evans CE, Haynes RB, Birkett NJ, Gilbert RJ, Taylor DW, Sackett DL, et al. Does a mailed continuing education program improve physician performance. JAMA. 1986;255:501–4.
Dubey V, Mathew R, Iglar K, Moineddin R, Glazier R. Improving preventive service delivery at adult complete health check-ups: the preventive health evidence-based recommendation form (PERFORM) cluster randomized controlled trial. BMC Fam Pract. 2006;7:44.
Shah BR, Bhattacharyya O, Yu CHY, Mamdani MM, Parsons JA, Straus SE, et al. Effect of an educational toolkit on quality of care: a pragmatic cluster randomized trial. PLoS Med. 2014;11.
Zwarenstein M, Shiller SK, Croxford R, Grimshaw JM, Kelsall D, Paterson JM, et al. Printed educational messages aimed at family practitioners fail to increase retinal screening among their patients with diabetes: a pragmatic cluster randomized controlled trial [ISRCTN72772651]. Implement Sci. 2014;9:87.
Bishop PB, Wing PC. Knowledge transfer in family physicians managing patients with acute low back pain: a prospective randomized control trial. Spine J. 2006;6:282–8.
Szonyi G, Millard R. Controlled trial evaluation of a general practitioner education package on incontinence: use of a mailed questionnaire. Br J Urol. 1994;73:615–20.
Liaw S-T, Sulaiman ND, Barton CA, Chondros P, Harris CA, Sawyer S, et al. An interactive workshop plus locally adapted guidelines can improve general practitioners asthma management and knowledge: a cluster randomised trial in the Australian setting. BMC Fam Pract. 2008;9:22.
French SD, McKenzie JE, O’Connor DA, Grimshaw JM, Mortimer D, Francis JJ, et al. Evaluation of a theory-informed implementation intervention for the management of acute low back pain in general medical practice: the IMPLEMENT cluster randomised trial. PLoS One. 2013;8.
Kunz R, Wegscheider K, Guyatt G, Zielinski W, Rakowsky N, Donner-Banzhoff N, et al. Impact of short evidence summaries in discharge letters on adherence of practitioners to discharge medication. A cluster-randomised controlled trial. Qual Saf Health Care. 2007;16:456–61.
Butzlaff M, Vollmar H, Floer B, Koneczny N, Isfort J, Lange S. Learning with computerized guidelines in general practice? A randomized controlled trial. Fam Pract. 2004;21:183–8.
Ulbricht S, Gross B, Kunstmann W, John U, Meyer C. Efficacy of disseminating educational materials in general practices. Sucht. 2014;60:85–91.
Denig P, Haaijer-Ruskamp FM, Djurre ZH. Impact of a drug bulletin on the knowledge, perception of drug utility, and prescribing behaviour of physicians. DICP, Ann Pharmacother. 1990;24:87–93.
Secher N, Mikkelsen MM, Adelborg K, Mikkelsen R, Grove EL, Rubak JM, et al. Direct mail improves knowledge of basic life support guidelines in general practice: a randomised study. Scand J Trauma Resusc Emerg Med. 2012;20:72.
Tsuji SR, Atallah AN, Aranha FC, Tonhom AP, Siqueira AC, Matos D. Cluster randomized clinical trial (ISRCTN23732000) to evaluate the effectiveness of a diagnosis recognition and treatment guide for depressive disorders in primary care. J Eval Clin Pract. 2009;15:222–5.
Simon AE, Jegerlehner S, Müller T, Cattapan-Ludewig K, Frey P, Grossenbacher M, et al. Prodromal schizophrenia in primary care: a randomised sensitisation study. Br J Gen Pract. 2010;60:e353–9.
Hunskaar S, Hannestad YS, Backe B, Matheson I. Direct mailing of consensus recommendations did not alter GPs’ knowledge and prescription of oestrogen in the menopause. Scand J Prim Health Care. 1996;14:203–8.
Perria C, Mandolini D, Guerrera C, Jefferson T, Billi P, Calzini V, et al. Implementing a guideline for the treatment of type 2 diabetics: results of a cluster-randomized controlled trial (C-RCT). BMC Health Serv Res. 2007;7:79.
Katon W, Von Korff M, Lin E, Simon G. Rethinking practitioner roles in chronic illness: the specialist, primary care physician, and the practice nurse. Gen Hosp Psychiatry. 2001;23:138–44.
Zwarenstein M, Goldman J, Reeves S. Interprofessional collaboration: effects of practice-based interventions on professional practice and healthcare outcomes (review). Cochrane Database Syst Rev. 2009;3:CD000072.
AG is funded by the Canadian Institutes of Health Research Doctoral Research Award and by the Knowledge Translation Canada Strategic Initiative in Health Research Fellowship.
SES is a Tier 1 Canada Research Chair.
We would like to acknowledge Alissa Epworth for her help with the literature search.
Dr. Sharon Straus is an associate editor for Implementation Science but was not involved in the decision around this manuscript.
AG designed and coordinated the study, participated in data collection, data analysis, data interpretation, and drafted the manuscript. RK, RR, and DR participated in data collection and revised the manuscript. JH contributed to the data analysis and helped draft the manuscript. SS designed and coordinated the study, participated in data interpretation and helped to draft the manuscript. All authors read and approved the final manuscript.
Medline Search. Search strategy for the Medline database. (PDF 142 kb)
TIDieR Checklist. Details on reporting of interventions. (PDF 204 kb)
Study Outcomes. All study outcomes for included studies. (PDF 543 kb)
Risk of Bias. Appraisal of risk of bias of included RCT studies using Cochrane EPOC risk of bias assessment tool. (PDF 141 kb)
Meta-Analysis Results. Forest Plots for Meta-Analyses. (PDF 395 kb)
Sub-analyses for Heterogeneity. Results of sub-analyses for heterogeneity. (PDF 140 kb)
About this article
Cite this article
Grudniewicz, A., Kealy, R., Rodseth, R.N. et al. What is the effectiveness of printed educational materials on primary care physician knowledge, behaviour, and patient outcomes: a systematic review and meta-analyses. Implementation Sci 10, 164 (2015). https://doi.org/10.1186/s13012-015-0347-5
- Primary care
- Evidence-based medicine
- Printed educational materials