Open Access
Open Peer Review

This article has Open Peer Review reports available.

How does Open Peer Review work?

What is the effectiveness of printed educational materials on primary care physician knowledge, behaviour, and patient outcomes: a systematic review and meta-analyses

  • Agnes Grudniewicz1, 2,
  • Ryan Kealy3,
  • Reitze N. Rodseth4, 5,
  • Jemila Hamid2, 6,
  • David Rudoler1 and
  • Sharon E. Straus2Email author
Implementation Science201510:164

https://doi.org/10.1186/s13012-015-0347-5

Received: 8 September 2015

Accepted: 29 October 2015

Published: 1 December 2015

Abstract

Background

Printed educational materials (PEMs) are commonly used simple interventions that can be used alone or with other interventions to disseminate clinical evidence. They have been shown to have a small effect on health professional behaviour. However, we do not know whether they are effective in primary care. We investigated whether PEMs improve primary care physician (PCP) knowledge, behaviour, and patient outcomes.

Methods

We conducted a systematic review of PEMs developed for PCPs. Electronic databases were searched for randomized controlled trials, quasi randomized controlled trials, controlled before and after studies, and interrupted time series. We combined studies using meta-analyses when possible. Statistical heterogeneity was examined, and meta-analysis was performed using a random effects model when significant statistical heterogeneity was present and a fixed effects model otherwise. The template for intervention description and replication (TIDieR) checklist was used to assess the quality of intervention description.

Results

Our search identified 12,439 studies and 40 studies met our inclusion criteria. We combined outcomes from 26 studies in eight meta-analyses. No significant effect was found on clinically important patient outcomes, physician behaviour, or physician cognition when PEMs were compared to usual care. In the 14 studies that could not be included in the meta-analyses, 14 of 71 outcomes were significantly improved following receipt of PEMs compared to usual care. Most studies lacked details needed to replicate the intervention.

Conclusions

PEMs were not effective at improving patient outcomes, knowledge, or behaviour of PCPs. Further trials should explore ways to optimize the intervention and provide detailed information on the design of the materials.

Protocol registration

PROSPERO, CRD42013004356

Keywords

Primary careEvidence-based medicinePrinted educational materials

Background

Printed educational materials (PEMs) are a simple, relatively inexpensive knowledge translation (KT) intervention for the dissemination of clinical information (such as clinical practice guidelines, journal articles, or evidence-based PDF or email summaries), aimed at improving the provision of care. A recent Cochrane review found that PEMs may have a small (0.02–0.13 standardized mean difference) beneficial effect on health professional practice outcomes [1]. However, despite continued publication of randomized controlled trials (RCTs) utilizing this intervention, we still know little about which behaviours can be influenced by PEMs, within which settings, and how to optimize the effect of these interventions for various health professionals.

Primary care physicians (PCPs) are required to have a vast and comprehensive knowledge base to treat different patient groups and diseases. On average, they have been observed to have 3.2 questions for every 10 patients they see [2] but these questions often go unanswered. PEMs are a potential strategy for meeting these needs. Non-interactive PEMs are easy to implement and scale across various primary care clinics. Reviews of the literature have found that printed resources (including books) remain a common source of information for physicians [35], with one systematic review finding that 50–80 % of physicians used printed materials for information [3]. However, if there is no demonstrated effectiveness of these interventions on knowledge, behaviour, or patient outcomes when targeted at PCPs, they should not be implemented as behaviour change techniques. To our knowledge, this is the first review to examine the effect of PEMs on PCPs.

Though interactive computer-based KT interventions such as those integrated within electronic health records have been shown to be effective in changing behaviour and are increasingly more popular than non-interactive paper-based interventions, they are expensive and require technological infrastructure and training, obstacles to implementation given limited budgets and overworked clinicians [6]. Surveys show that only 64 % of Canadian PCPs [7] and 41.5 % of American physicians [8] use electronic medical records, limiting the reach of complex interventions that are integrated into electronic records and possibly unintentionally leaving PCPs out of these interventions. With many different software vendors being used across practices (for example, there are 14 certified electronic medical record products to date in Canada alone [9]), creating a one-size-fits-all solution is challenging. As such, PEMs, a non-interactive and low-tech intervention, will likely continue to be used to disseminate new evidence and important clinical information or as a part of multi-component KT interventions.

The objective of this review was to examine what effect PEMs have on PCP knowledge, behaviour, and patient outcomes, in comparison to no intervention or to other single- or multi-component educational interventions. This review contributes to existing literature by examining the effect of interventions specifically designed for PCPs. The primary care setting is considerably different from other health care settings, and PCPs are likely to experience barriers unique to their setting and their scope of practice. Physicians are the population of interest to limit participant heterogeneity as we anticipated that differences in training and role among diverse primary care clinicians may influence behaviour change. PEMs for PCPs may have different content and may target different behaviour than PEMs for other professionals. More importantly, PCPs may respond differently than other clinicians to PEMs, and we anticipate PEMs have a different effect size when targeting behaviour change in different providers. We also examined the quality of reporting of PEM interventions in included studies.

Methods

A systematic review protocol was written for this review and registered with PROSPERO, the international prospective register of systematic reviews (registration no. CRD42013004356). We based the methods for this review on those described in the Cochrane Handbook for Systematic Reviews of Interventions and the Cochrane Effective Practice and Organisation of Care Group (EPOC) [10]. It is reported using the PRISMA Statement for Reporting Systematic Reviews [11].

Eligibility criteria

We included studies reporting the effectiveness of PEMs for PCPs (family physicians as well as specialists practising primary care such as pediatricians), defined for the purpose of the review as guidelines, summaries of guidelines, the dissemination of published or non-published information, recommendations, or evidence presented in print or electronic form. Electronic materials include PDFs, other document files (e.g. Microsoft Word documents), and non-interactive web pages. Studies examining interactive online educational materials (such as online courses) or patient-specific materials were excluded. We included any method of delivery of the intervention (e.g. email, mail, fax) or level of intensity (i.e. how often the intervention was delivered). We limited study designs to RCTs, quasi randomized trials, controlled before and after studies, and interrupted time series (ITS) analyses. We included reported patient outcomes, physician cognition (skills and knowledge), and physician behaviour outcomes. Physician attitudes were not included in the review. No restrictions were placed on publication status or date of publication. Studies not published in English were excluded. Lastly, comparisons had to allow for the isolation of the effect of PEMs on outcomes, meaning that studies comparing two PEMs with no control group or studies comparing two interventions where both groups included PEMs were excluded. Researchers independently and in duplicate screened each title and abstract (AG, SR, RK, DR) and reviewed the full text of selected studies for eligibility using the criteria listed above (AG, SR, RK).

Information sources and search

Studies were identified by a search of electronic databases developed by an information specialist and independently reviewed by a second information specialist. Medline (Ovid MEDLINE(R)), EMBASE (Embase Classic + Embase), ERIC (ProQuest), and the Cochrane Central Register of Controlled Trials were searched on November 25, 2014. The following search terms were used: print, message, book, monograph, pamphlet, journal, educational materials, online, email, web-based, general practitioner, and family physician (see Additional file 1 for Medline search). Appropriate wildcards were used in the search to account for plurals and variations in spelling. The search was supplemented by searching the reference lists of included articles and of other systematic reviews. Authors of studies published within the last 10 years were contacted to collect missing data needed for meta-analyses.

Data extraction

We extracted data independently and in duplicate (AG, RK) using a modified version of the Cochrane EPOC standardized data collection checklist [10]. A calibration exercise was done before screening and data extraction with each of the researchers to ensure consistency. Information was extracted from each study on study design, the intervention, controls, type of targeted behaviour, professional and patient participants, setting, methods, outcomes, costs of the intervention, changes in healthcare costs, and results. All disagreements for screening and extraction were resolved by discussion.

Risk of bias

Risk of bias was assessed independently and in duplicate using the Cochrane EPOC risk of bias assessment tool [10].

Data synthesis

Clinical (e.g. type of population, topic of intervention), methodological (e.g. study duration), and statistical heterogeneity were assessed. For studies with similar clinical and methodological characteristics that reported similar outcomes, data were pooled statistically using meta-analysis, with separate meta-analyses carried out for patient outcomes and for physician outcomes. Physician outcomes were grouped post hoc according to Miller’s framework for clinical assessment into physician behaviour (“does” and “shows how”) and physician cognition (“knows” and “knows how”). Miller’s framework presents four categories, with knowledge (“knows”) at the base to represent that a physician knows what is required to effectively carry out professional functions. Following knowledge is competence, or “knows how”, which captures the skills needed to acquire information, analyze and interpret data, and translate findings into their practice. Performance (“shows how”) follows, where physicians must demonstrate their knowledge within an examination setting, and finally is action (“does”) which is the independent action of the physician in clinical practice [12]. We then grouped the studies within these categories into either binary or continuous outcomes. Only one outcome per study was used in any single meta-analysis to avoid double counting. The study’s primary outcome was chosen when possible. When more than one primary or eligible outcome existed, the outcome included in the meta-analysis was chosen at random. A meta-analysis was conducted when outcomes from two studies could be pooled using R software and the Metafor Package [13, 14].

Statistical heterogeneity was examined using the I 2 statistic. A random-effects meta-analysis was conducted when heterogeneity was statistically significant; otherwise, a fixed-effects model was used. When I 2 was significant, we conducted post hoc sub-analyses to explore the source of heterogeneity. For continuous outcomes, a standard mean difference (SMD) was calculated because outcomes were measured using different scales. For binary outcomes, a relative risk (RR) was used as an effect measure. Studies were adjusted for clustering when applicable using study-reported intracluster coefficients (ICCs). When not provided in the study, an ICC was selected from the University of Aberdeen Database of ICCs [15]. When a range of possible ICCs was provided in the literature, sensitivity analysis was performed using multiple adjustments to determine sensitivity of the pooled estimates with respect to ICC values. For meta-analyses of continuous outcomes, when a standard deviation (SD) or standard error was not provided, it was imputed from similar outcomes within the study. Sensitivity analysis was also performed with respect to the imputed standard errors. Studies that did not provide sufficient data for the calculation of a summary statistic were not included in the meta-analysis. A narrative description was used to present the data that could not be pooled.

Intervention reporting

Studies were assessed against the template for intervention description and replication (TIDieR checklist) to determine the completeness of reporting and replicability of interventions in the included studies [16]. The checklist was applied from the perspective of the intervention of interest for our review, which may have been one of several interventions tested or used as control.

Results

Literature search

From the 16,735 articles retrieved, the literature search resulted in 12,439 citations (duplicates removed) and 146 potentially relevant citations. Forty studies and two companion reports met eligibility criteria. See Fig. 1 for a flow diagram representing identification of eligible studies [11].
Fig. 1

Identification of eligible studies.

Study characteristics

Thirty-seven of the included studies were RCTs, and three were studies with an ITS design. Studies were conducted between 1983 and 2014 in the USA [1727], UK [2836], Canada [3744], Australia [4547], Germany [4850], the Netherlands [51], Denmark [52], Brazil [53], Switzerland [54], Norway [55], and Italy [56]. A wide range of intervention topics was observed in our systematic review. Study characteristics are outlined in Table 1.
Table 1

Characteristics of included studies

Author (year)

Study type

Country

Study topic

Intervention

Comparison

# of physicians

Behaviours targeted

Avorn, J. (1983)

RCT

USA

Drug prescribing

Recommendations

Another intervention, usual care

435

Physician behaviour

Bearcroft, P.W.P. (1994)

RCT

England

Medical imaging

Full guideline or summary

Usual care

210

Physician behaviour

Bishop, P. (2006)

RCT

Canada

Low back pain

Whole guideline

Another intervention, usual care

462

Physician behaviour

Bjornson, D.C. (1990)

RCT

USA

Hypertension/cardiovascular illness

Journal article

Usual care

576

Physician behaviour

Butzlaff, M. (2003)

RCT

Germany

General

Whole guideline

Usual care

72

Physician cognition

Denig, P. (1990)

RCT

Netherlands

Irritable bowel syndrome and renal colic

Bulletin

Usual care

209

Physician behaviour, physician cognition

Dickinson, W.P. (2003)

RCT

USA

Mental illness

Recommendations

Usual care

Not clear

Patient

Dormuth, C.R. (2004)

RCT

Canada

Drug prescribing

Bulletin

Usual care

499

Physician behaviour

Downs, M. (2006)

RCT

UK

Dementia

Electronic case analysis

Another intervention, usual care

Not clear

Physician behaviour

Dubey, V. (2006)

RCT

Canada

Prevention

Checklist with recommendations

Usual care

38

Physician behaviour

Evans, C.E. (1986)

RCT

Canada

Hypertension/cardiovascular illness

Educational/Informational Package

Usual care

76

Patient, physician behaviour

Feng, B. (2013)

RCT

USA

Prostate cancer screening

Recommendations

Another intervention

118

Physician behaviour

French, S. (2013)

RCT

Australia

Low back pain

Whole guideline

Another intervention

92

Physician behaviour, physician cognition

Guadagnoli, E. (2004)

RCT

USA

Hypertension/cardiovascular illness

Recommendations

Usual care

247

Physician behaviour

Guthrie, B. (2013)

ITS

Scotland

Dementia

Recommendations

N/A

Not clear

Physician behaviour

Hazard, R.G. (1997)

RCT

USA

Low back pain

Algorithm

Usual care

30

Patient

Hunskaar, S. (1996)

RCT

Norway

Drug prescribing

Recommendations

Usual care

374

Physician cognition

Kottke, T.E. (1989)

RCT

USA

Smoking cessation

Manual

Another intervention, usual care

66

Patient, physician behaviour

Kunz, R. (2007)

RCT

Germany

Drug prescribing

Recommendations

Usual care

132

Physician behaviour

Liaw, S.T. (2008)

RCT

Australia

Asthma

Guideline summary

Another intervention, usual care

51 total

Physician behaviour, physician cognition

Matowe, L. (2002)

ITS

Scotland

Medical imaging

Whole guideline

N/A

376

Physician behaviour

McEwan, A. (2002)

RCT

England

Smoking cessation

Educational/informational package

Usual care

107

Physician behaviour

Mukohara, K. (2005)

RCT

USA

General

Journal summary

Another intervention

107

Physician cognition

Nicholas, J. (2009)

RCT

USA

Obesity

Recommendations

Usual care

1000

Physician behaviour

Oakeshott, P. (1994)

RCT

England

Medical imaging

Sections of a guideline

Usual care

170

Physician behaviour

Perria, C. (2007)

RCT

Italy

Diabetes

Whole guideline

Another intervention, usual care

252

Physician behaviour

Rabin, D. (1994)

RCT

USA

Sexually transmitted diseases

Educational/informational package

Another intervention, usual care

961

Physician behaviour

Rahme, E. (2005)

RCT

Canada

Osteoarthritis

Recommendations

Another intervention, usual care

249

Physician behaviour

Secher, N. (2012)

RCT

Denmark

Life support

Poster

Usual care

830

Physician cognition

Shah, B. (2014)

RCT

Canada

Diabetes and cardiovascular disease

Educational/informational package

Usual care

Not clear

Patient, physician behaviour

Simon, A.E. (2010)

RCT

Switzerland

Mental illness

Clinical vignette

Usual care

1138

Physician cognition

Szonyi, G. (1994)

RCT

Australia

Incontinence

Educational/informational package

Usual care

124

Physician cognition

Tsuji, S.R. (2007)

RCT

Brazil

Mental illness

Educational/informational package

Usual care

8

Patient, physician behaviour

Tziraki, C. (2000)

RCT

USA

Prevention

Manual

Another intervention, usual care

810

Physician behaviour

Ulbricht, S. (2014)

RCT

Germany

Psychotropic drug use

Manual

Usual care

852

Physician behaviour

Watson, E. (2001)

RCT

England

Genetic services for breast cancer

Educational/informational package

Another intervention, usual care

688

Physician cognition

Watson, M. (2001)

RCT

England

Drug prescribing

Recommendations

Another intervention, usual care

107

Physician behaviour

Worrall, G. (1999)

RCT

Canada

Mental illness

Whole guideline

Another intervention

42

Patient, physician behaviour

Wright, N.M.J. (2004)

ITS

England

Drug prescribing

Recommendations

N/A

444

Physician behaviour

Zwarenstein, M. (2014)

RCT

Canada

Diabetes and retinal screening

Recommendations

Usual care, variations in intervention

5048

Physician behaviour

Very few studies provided details on the study population such as age and time since graduation. Fourteen studies provided information on age of the participants, and nine studies provided information on time since graduation from medical school. The number of physician participants in each study ranged from 8 to 5048. All studies were composed of a majority of family physicians, with some studies including pediatricians [24], general internists [17, 20, 26, 40, 48], and a small number of other specialists [17, 20, 48]. Three [19, 28, 47] of the studies included in this review examined the cost of the intervention, with two studies [31, 47] calculating the direct impact of the interventions on health care costs. Avorn and Soumerai [19] found a cost reduction of $105 (US dollars) per physician over 9 months for their academic detailing intervention but did not calculate any savings for the PEM (used as control). Watson et al. [28] calculated the costs of the guideline intervention and found no significant reduction in prescribing costs. French et al. [47] conducted a cost-effectiveness analysis that found that a more active intervention (including a workshop) was less expensive than the existing standard PEM dissemination strategy. However, savings in health gains and service reductions were not sufficient to make the active strategy cost-effective. No studies looked at changes in non-health care costs.

Comparisons

Of the RCTs, 21 studies compared PEMs to usual care [17, 18, 21, 22, 24, 29, 30, 35, 38, 4042, 45, 4855], 12 to both usual care and another intervention [19, 20, 23, 25, 28, 33, 34, 39, 43, 44, 46, 56], and four to another intervention [26, 27, 37, 47].

Intervention details

Several types of interventions were included such as guidelines [32, 37, 44, 47, 49, 56], guideline summaries or sections of guidelines [29, 46], practice recommendations [19, 21, 22, 24, 27, 28, 31, 36, 39, 43, 48, 55], educational or information packages [20, 34, 35, 40, 42, 45, 53], manuals [23, 25, 50], bulletins [38, 51], clinical vignettes [57], a journal summary [26], a checklist with recommendations [41], an electronic case analysis [33], an algorithm [18], a poster [52], and a journal article [17]. One study was not clear whether the intervention was a complete guideline or summary [30].

Reporting of interventions (TIDieR checklist)

All studies defined or provided a name for their intervention and the majority (80 %, N = 32) of the studies provided a rationale, theory, or goal of the intervention. All but three studies described the materials in the intervention; 32.5 % (N = 13) of the studies made the materials available in the paper or in supplementary materials or websites. Two studies provided web links or references to the PEMs that did not work or were not available; these were counted as materials not provided [18, 26]. Thirty-three studies described the intervention procedures, 35 described the location of the intervention, and 26 described the timeframe and number of times the intervention was delivered. Fifteen percent (N = 6) of the studies addressed planned measurement of adherence and 22.5 % (N = 9) studies measured adherence to the intervention (e.g. receipt of materials, reading of materials). Details on reporting of interventions are provided in Additional file 2.

Outcomes measured and behaviour targeted

Eight studies measured one or more patient outcomes as a result of the PCP-targeted intervention [18, 21, 25, 37, 40, 42, 46, 53]. The majority of included studies (77.5 %, N = 31) measured physician behaviour outcomes [17, 19, 20, 2225, 2733, 3542, 44, 4648, 50, 51, 53, 56, 58], and 25 % (N = 10) of the studies measured physician cognition outcomes [26, 34, 4547, 49, 51, 52, 54, 55]. Additional File 3.

Risk of bias of included studies

Using the Cochrane EPOC risk of bias assessment tool, 33 RCTs had unclear or high risk of bias for at least two criteria. Two RCTs were appraised as low risk of bias on eight of nine criteria [38, 43], and only two RCTs were appraised as having low risk of bias on all nine criteria [41, 42] (See Additional file 4). The ITS studies had at least two unclear risk of bias out of seven criteria. Figure 2 illustrates the percentage of studies at low, high, and uncertain risk of bias.
Fig. 2

Risk of bias— Legend: green: low, yellow: unclear, red: high

Meta-analyses

Eight meta-analyses were conducted. Results are presented in Table 2, and forest plots are available in Additional File 5.
Table 2

Results of meta-analyses

Outcome

Number of studies

Summary statistic

Confidence interval

I 2 statistic (%)

RR = relative risk

SMD = standard mean difference

Patient outcomes

 Dichotomous patient outcomes

5

RR = 1.00

1.00–1.00

0

 Dichotomous patient outcomes (sensitivity analysis with Shah 2014 removed)

4

RR = 1.09

0.91–1.29

0

Physician behaviour outcomes

 Continuous physician behaviour outcomes (N = physicians)

4

SMD = 0.35

−0.06–0.76

67.88

 Continuous physician behaviour outcomes (N = patients)

3

SMD = 0.27

−0.03–0.57

68.74

 Dichotomous physician behaviour outcomes (N = physicians)

3

RR = 1.01

0.96–1.07

0

 Dichotomous physician behaviour outcomes (N = patients)

9

RR = 0.99

0.97–1.01

0

 Dichotomous physician behaviour outcomes (PEM vs. workshop)

2

RR = 0.57

0.12–2.73

64.48

Physician cognition outcomes

 Continuous physician cognition outcomes

3

SMD = 0.65

−0.21–1.51

89.75

 Dichotomous cognition outcomes

3

RR = 1.51

0.90–2.52

90.27

Patient outcomes

A meta-analysis of dichotomous patient outcomes was conducted with five studies and 935,252 patients [18, 25, 40, 42, 53], combined using a fixed-effects model. The summary statistic demonstrated no improvement with use of a PEM, RR = 1.00 (95 % CI = 1.00,1.00, I 2 = 0). One study [42] was allocated 100 % of the weight in the analysis; therefore, we completed a sensitivity analysis without the study for a total of 1463 patients, which resulted in a small but statistically insignificant improvement, RR = 1.09 (95 % CI = 0.91,1.29, I 2 = 0).

Physician behaviour outcomes

Five meta-analyses of studies reporting impact on physician behaviour outcomes were conducted. A meta-analysis of continuous physician outcomes was conducted with four studies and 531 physicians [24, 25, 28, 35]. The results were not statistically significant, SMD = 0.35 (95 % CI = −0.06, 0.76, I 2 = 67.88 %). Sub-analyses grouping studies by similar topic, type of PEM, length of intervention, type of participants, and the behaviour targeted found that the PEM topic was the largest source of heterogeneity (see Additional File 6). Three studies assessing impact on continuous physician outcomes per patient including 1106 patients were combined [22, 33, 41]. No statistically significant result was found, SMD = 0.27 (95 % CI = −0.03, 0.57, I 2 = 68.74 %). Sub-analyses grouping studies by similar type of intervention, risk of bias, and behaviour targeted found that the behaviour targeted was the largest source of heterogeneity.

A meta-analysis of dichotomous physician behaviour outcomes was conducted with results from three studies including 999 physicians [20, 46, 50]. No statistically significant improvement was observed with the use of PEM, RR = 1.01 (95 % CI = 0.96, 1.07, I 2 = 0.00 %). Nine studies assessing impact of PEMs on dichotomous physician behaviour outcomes per patient including 3,273,788 patients were combined [22, 38, 40, 42, 44, 48, 53, 56, 58]. No statistically significant improvement was observed with PEM, RR = 0.99 (95 % CI = 0.97, 1.01, I 2 = 0.00 %). Two studies assessing dichotomous physician behaviour outcomes of PEMs compared to workshops were combined including 153,089 patients [37, 47]. No statistically significant difference was found between the two interventions, RR = 0.57 (95 % CI = 0.12, 2.73, I 2 = 64.48 %).

Physician cognition outcomes

Three studies (438 physicians) were combined in a meta-analysis of studies reporting continuous physician cognition outcomes [4547]. No statistically significant effect was found, SMD = 0.65 (95 % CI = −0.21, 1.51, I 2 = 89.75 %). Three studies (806 physicians) were combined in a meta-analysis of dichotomous physician cognition outcomes [34, 51, 52], and no significant effect was found, RR = 1.51 (95 % CI = 0.90, 2.52, I 2 = 90.27 %). The high statistical heterogeneity for both cognitive meta-analyses could not be examined. Subgroup analyses could not be performed as all studies varied in topic, method of knowledge measurement, type of intervention, and length of intervention.

Narrative synthesis

Fourteen studies examined 71 outcomes that were not included in the meta-analyses due to heterogeneity in study outcomes (i.e. outcomes reported could not be pooled with any other study), study design (i.e. ITS studies were not pooled with RCTs), or missing data (i.e. studies did not provide sufficient data and authors could not be reached) (see Additional file 3 for all study outcomes). The results are presented below by type of outcome.

Patient outcomes

One study [21] examined the effect of PEMs on four clinical patient outcomes compared to usual care. A significant effect was found for only one outcome (physical functioning of patients with multisomatoform disorder).

Physician behaviour

Nine studies [17, 19, 23, 2932, 36, 39] examined the effect of PEMs on 55 physician behaviour outcomes compared to usual care. Bearcroft et al. [30] found a significant effect of PEMs on all four outcomes studied in chest radiography referral, including concordance with guidelines and documentation of history, clinical diagnosis, and smoking history. Guthrie et al. [36] found a significant effect of PEMs on six out of 12 outcomes examining prescriptions of antipsychotics. Oakeshott et al. [29] found a significant improvement with PEMs on total radiology requests but not when examined by X-ray type (limbs, chest, or spine). Worrall et al. [37] found a significant improvement with PEMs on two of five prescribing outcomes. No effect was found on outcomes in five other studies [17, 19, 23, 32, 39].

One study [27] examined the effect of PEMs compared to a web course on eight physician behaviour outcomes. There was a significant difference between the two interventions for three of the eight outcomes (overall shared decision-making, guiding in decision-making, stating that a PSA test would be ordered).

Physician cognition

Three studies [26, 49, 55] examined four physician cognition outcomes. None of the studies found an improvement in cognition after a PEM intervention.

Discussion

We included 40 studies and conducted eight meta-analyses with data from 26 studies. The reported quality of evidence ranged from low to high and many studies were unclear on important methodological factors such as allocation sequence generation (15 studies unclear), baseline outcome measurement (15 studies unclear), and baseline characteristic measurement (nine studies unclear). No significant effect was seen across the eight meta-analyses. However, six of the meta-analyses had three or fewer studies included.

Statistical heterogeneity was high for both physician cognition meta-analyses, possibly due to different ways of measuring knowledge outcomes, the different topic areas, duration of the intervention, risk of bias, or the type of PEM intervention. Clinical heterogeneity was visible across studies in all meta-analyses as the topic of the interventions varied greatly. Behaviour change of PCPs may vary by clinical area addressed by the intervention. Though heterogeneity in the population was limited by the review’s focus on PCPs, it was difficult to determine how the population varied across studies due to limited reporting on participant age, time since graduation, and academic affiliation. Methodological heterogeneity was also a concern as study durations varied widely and many factors could not be assessed such as how the intervention was delivered, the source of the evidence, setting of the participant’s clinical practice, and participant demographic details.

The narrative synthesis examined 71 study-level outcomes across three outcome categories. When compared to usual care, PEMs resulted in significant improvement in outcomes for one of four clinical patient outcomes, 13 physician behaviour outcomes, and no physician cognition outcomes. Though it is difficult to draw conclusions from the narrative syntheses of these outcomes, it is evident that a large majority of the measured outcomes did not statistically improve as a result of PEMs. These results differ from the review by Giguère and colleagues [1], which included all health professionals, and found a median improvement in SMD for categorical practice outcomes of 0.02 and a median improvement in SMD for continuous professional outcomes of 0.13 when compared to usual care. Only 11 of our 40 studies overlapped with the Giguère review. Our results suggest that the effect found in the review by Giguère and colleagues may not be applicable to PCPs and that the effect was a result of interventions targeted at specialist physicians. Most of the studies included in the Cochrane review (42 of 45) included physicians, though the clinical speciality was not clear in many of the included studies and no subgroup analyses were conducted by type of health professional. The difference in effect size may be a result of differences in PEM content between PCP and other clinician PEMs, in the evidence needs of PCPs, in the practice context, or in the behaviours targeted by the PEM. The methods used to synthesize data in the Cochrane review included the use of more than one outcome per study and may have resulted in an inflated effect size. The authors of the Cochrane review also used a different approach when more than one measure was provided in a study by abstracting the median measure whereas we used a random selection process. Furthermore, our review builds on the existing literature as it examines the intervention’s effect on knowledge, an outcome that was excluded from the Cochrane review.

Though it is assumed that the cost of these interventions is small relative to interactive educational tools or active evidence-dissemination strategies, the costs need to be systematically assessed. Three of the studies included in this review provided an economic analysis, but two of the three studies provided costs for a more active intervention and not for the control PEM. Before widely disseminating any behaviour change intervention, information on cost effectiveness of the intervention is critical.

Intervention reporting TIDieR checklist

One study [48] met all criteria of the TIDieR checklist, and three studies met all but one of the 12 criteria [41, 43, 49]. This is a concerning finding given the relative simplicity of PEM interventions. However, it may be this perceived simplicity that has discouraged study authors from describing the intervention and its development. Improvement in intervention reporting is needed, such as providing access to the intervention materials through appendices or permanent web links. Most study authors noted that they did not know if PEMs were received and read, a significant limitation of the included studies. Other key areas for improvement in reporting of interventions are detailing who provided the intervention, describing how the intervention was delivered, and specifying the date the intervention was delivered.

Limitations

Differences in outcome measurement and limited reporting resulted in the inability to pool the results for a number of studies. A small number of trials were eligible for each meta-analysis, with one of the eight meta-analyses only pooling two studies, limiting our ability to conduct a meta-regression to explore effect modifiers. It is possible that the meta-analyses had too few included outcomes to detect the small differences that may be expected from PEMs. The physician cognition meta-analyses may be limited in their usefulness due to high statistical heterogeneity. The review may also have been limited by only including English-language studies.

Conclusions

This systematic review has found that PEMs do not improve patient, PCP behaviour, or PCP knowledge outcomes. A previous Cochrane systematic review of 45 studies (14 RCTs and 31 ITSs) noted that similar, passive dissemination strategies have small to moderate effects (1). However, the Cochrane review included all health care professionals, excluded knowledge outcomes, and combined more than one outcome per study (possibly inflating the result) in its meta-analyses.

These findings are of particular importance to those interested in knowledge translation (guideline developers, quality agencies, government agencies, pharmaceutical companies, etc.) and in developing, disseminating, and evaluating PEM interventions. Individuals and groups interested in developing PEMs should note that in their current state, there is no evidence that PEMs are effective at improving PCP outcomes. However, it is difficult to ascertain whether the PEMs tested have been optimized and whether adequate reach of the intervention has been achieved. There are too few details in published studies on how these materials were developed (use of theory, use of evidence-based design, involvement of the end-user) and too limited descriptions of the materials. The results also have implications for research funding bodies. Continuing to fund studies on PEMs with little description of the intervention, inadequate power, or a lack of optimization of the intervention is a poor use of resources. Researchers interested in conducting future studies of PEMs for primary care should invest resources in better design of the tools and should provide detailed descriptions of the intervention to determine if PEM optimization can result in improved outcomes. Trials should be sufficiently powered to detect the small effect of these interventions. Head to head studies of different designs of PEMs may help us understand how certain design elements may contribute to effectiveness, though these studies should continue to include usual care comparisons. Otherwise, we should accept the role of PEMs as simple tools for the dissemination of information or as a part of other more interactive interventions, rather than as tools to influence knowledge and behaviour of PCPs.

Abbreviations

PCP: 

primary care physician

PEMs: 

printed educational materials

RCT: 

Randomized controlled trials

ITS: 

Interrupted time series

SMD: 

Standard mean difference

RR: 

Relative risk

ICC: 

Intracluster correlation coefficient

SD: 

Standard deviation

Declarations

Acknowledgements

AG is funded by the Canadian Institutes of Health Research Doctoral Research Award and by the Knowledge Translation Canada Strategic Initiative in Health Research Fellowship.

SES is a Tier 1 Canada Research Chair.

We would like to acknowledge Alissa Epworth for her help with the literature search.

Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Authors’ Affiliations

(1)
Institute of Health Policy, Management and Evaluation, University of Toronto
(2)
Li Ka Shing Knowledge Institute, St. Michael’s Hospital
(3)
Interactive Media Lab, Department of Mechanical and Industrial Engineering, University of Toronto
(4)
Perioperative Research Group, Department of Anaesthetics, Grey’s Hospital, Nelson R. Mandela School of Medicine, University of KwaZulu-Nata
(5)
Department of Outcomes Research, Cleveland Clinic
(6)
Clinical Epidemiology and Biostatistics, McMaster University

References

  1. Giguère A, Légaré F, Grimshaw J, Turcotte S, Fiander M, Grudniewicz A, et al. Printed educational materials: effects on professional practice and healthcare outcomes (review). Cochrane Database Syst Rev. 2012;10:CD004398.PubMedGoogle Scholar
  2. Ely JW, Osheroff JA, Ebell MH, Bergus GR, Levy BT, Chambliss ML, et al. Analysis of questions asked by family doctors regarding patient care. BMJ. 1999;319:358–61.PubMed CentralView ArticlePubMedGoogle Scholar
  3. Dawes M, Sampson U. Knowledge management in clinical practice: a systematic review of information seeking behavior in physicians. Int J Med Inform. 2003;71:9–15.View ArticlePubMedGoogle Scholar
  4. Coumou HC, Meijman FJ. How do primary care physicians seek answers to clinical questions? A literature review. J Med Libr Assoc. 2006;94:55–60.PubMed CentralPubMedGoogle Scholar
  5. Davies K. The information-seeking behaviour of doctors: a review of the evidence. Health Info Libr J. 2007;24:78–94.View ArticlePubMedGoogle Scholar
  6. Grimshaw JM, Shirran L, Thomas R, Mowatt G, Fraser C, Bero L, et al. Changing provider behavior an overview of systematic reviews of interventions. Med Care. 2001;39(8, SII):II2–45.PubMedGoogle Scholar
  7. Canadian Medical Association. The Royal College of Physicians and Surgeons: National Physician Survey; Mississauga, Ontario (Canada). 2013.Google Scholar
  8. Hsiao C-J, Beatty PC, Hing ES, Woodwell DA, Rechtsteiner EA, Sisk JE. Electronic medical record/electronic health record use by office-based physicians: United States, 2008 and preliminary 2009: National Center for Health Statistics; Hyattsville, MD (United States). 2009.Google Scholar
  9. Infoway certified products. [https://www.infoway-inforoute.ca/index.php/programs-services/certification-services/infoway-certified-products]
  10. Cochrane effective practice and organization of care group. [http://epoc.cochrane.org/].
  11. Liberati A, Altman DG, Tetzlaff J, Mulrow C, Ioannidis JPA, Clarke M, et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. Ann Intern Med. 2009;151.Google Scholar
  12. Miller GE. The assessment of clinical skills/competence/performance. Acad Med. 1990;65(9 Suppl):S63–7.View ArticlePubMedGoogle Scholar
  13. R Core Team. R: a language and environment for statistical computing. R Foundation for Statistical Computing. Vienna. 2012. [http://www.r-project.org/].
  14. Viechtbauer W. Conducting meta-analyses in R with the metafor package. J Stat Softw. 2010;36:1–48.View ArticleGoogle Scholar
  15. Database of ICCs. [http://www.abdn.ac.uk/hsru/research/delivery/behaviour/methodological-research/].
  16. Hoffmann TC, Glasziou PP, Boutron I, Milne R, Perera R, Moher D, et al. Better reporting of interventions: template for intervention description and replication (TIDieR) checklist and guide. BMJ. 2014;348:g1687.View ArticlePubMedGoogle Scholar
  17. Bjornson DC, Rector TS, Daniels CE, Wertheimer AI, Snowdon DA, Litman TJ. Impact of drug-use review program intervention on prescribing after publication of a randomized clinical trial. Am J Hosp Pharm. 1990;47:1541–6.PubMedGoogle Scholar
  18. Hazard RG, Haugh LD, Reid S, McFarlane G, MacDonald L. Early physician notification of patient disability risk and clinical guidelines after low back injury: a randomized, controlled trial. Spine (Phila Pa 1976). 1997;22:2951–8.View ArticleGoogle Scholar
  19. Avorn J, Soumerai S. Improving drug-therapy decisions through educational outreach. N Engl J Med. 1983;308:1457–63.View ArticlePubMedGoogle Scholar
  20. Rabin DL, Boekeloo BO, Marx ES, Bowman MA, Russell NK, Willis AG. Improving office-based physicians’ prevention practices for sexually transmitted diseases. Ann Intern Med. 1994;121:513–9.View ArticlePubMedGoogle Scholar
  21. Dickinson WP, Dickinson LM, de Gruy F, Main DS, Candib LM, Rost K. A randomized clinical trial of a care recommendation letter intervention for somatization in primary care. Ann Fam Med. 2003;1:228–35.PubMed CentralView ArticlePubMedGoogle Scholar
  22. Guadagnoli E, Normand S-LT, DiSalvo TG, Palmer RH, McNeil BJ. Effects of treatment recommendations and specialist intervention on care provided by primary care physicians to patients with myocardial infarction or heart failure. Am J Med. 2004;117:371–9.View ArticlePubMedGoogle Scholar
  23. Tziraki C, Graubard BI, Manley M, Kosary C, Moler JE, Edwards BK. Effect of training on adoption of cancer prevention nutrition-related activities by primary care practices: results of a randomized, controlled study. J Gen Intern Med. 2000;15:155–62.PubMed CentralView ArticlePubMedGoogle Scholar
  24. Nicholas J, Dennison BA, De Long R, Prokorym M, Brissette I. Randomized controlled trial of a mailed toolkit to increase use of body mass index percentiles to screen for childhood obesity. Prev Chronic Dis Public Heal Res Pract Policy. 2009;6(4):A122.Google Scholar
  25. Kottke TE, Brekke ML, Solberg LI, Hughes JR. A randomized trial to increase smoking intervention by physicians, doctors helping smokers, round 1. JAMA. 1989;14:2101–6.View ArticleGoogle Scholar
  26. Mukohara K, Schwartz MD. Electronic delivery of research summaries for academic generalist doctors: a randomised trial of an educational intervention. Med Educ. 2005;39:402–9.View ArticlePubMedGoogle Scholar
  27. Feng B, Srinvisan M, Hoffman JR, Rainwater JA, Griffin E, Dragojevic M, et al. Prostate cancer screening: analysis of unannounced standardized patient visits. Ann Fam Med. 2013;11:315–23.PubMed CentralView ArticlePubMedGoogle Scholar
  28. Watson M, Gunnell D, Peters T, Brookes S, Sharp D. Guidelines and educational outreach visits from community pharmacists to improve prescribing in general practice: a randomised controlled trial. J Health Serv Res Policy. 2001;6:207–13.View ArticlePubMedGoogle Scholar
  29. Oakeshott P, Kerry SM, Williams JE. Randomized controlled trial of the effect of the Royal College of Radiologists’ guidelines on general practitioners’ referrals for radiographic examination. Br J Gen Pract. 1994;44:197–200.PubMed CentralPubMedGoogle Scholar
  30. Bearcroft P, Small J, Flower C. Chest radiography guidelines for general practitioners: a practical approach. Clin Radiol. 1994;49:56–8.View ArticlePubMedGoogle Scholar
  31. Wright NMJ, Roberts AJ, Allgar VL, Tompkins CNE, Greenwood DC, Laurence G. Impact of the CSM advice on thioridazine on general practitioner prescribing behaviour in Leeds: time series analysis. Br J Gen Pract. 2004;54:370–3.PubMed CentralPubMedGoogle Scholar
  32. Matowe L, Ramsay CR, Grimshaw JM, Gilbert FJ, Macleod M-J, Needham G. Effects of mailed dissemination of the Royal College of Radiologists’ guidelines on general practitioner referrals for radiography: a time series analysis. Clin Radiol. 2002;57:575–8.View ArticlePubMedGoogle Scholar
  33. Downs M, Turner S, Bryans M, Wilcock J, Keady J, Levin E, et al. Effectiveness of educational interventions in improving detection and management of dementia in primary care: a cluster randomized controlled study. BMJ. 2006;332.Google Scholar
  34. Watson E, Clements A, Yudkin P, Rose P, Bukach C, Mackay J, et al. Evaluation of the impact of two educational interventions on GP management of familial breast/ovarian cancer cases: a cluster randomised controlled trial. Br J Gen Pract. 2001;51:817–21.PubMed CentralPubMedGoogle Scholar
  35. Mcewen A, Preston A, West R. Effect of a GP desktop resource on smoking cessation activities of general practitioners. Addiction. 2001;97:595–7.View ArticleGoogle Scholar
  36. Guthrie B, Clark SA, Reynish EL, McCowan C, Morales DR. Differential impact of two risk communications on antipsychotic prescribing to people with dementia in Scotland: segmented regression time series analysis 2001–2011. PLoS ONE. 2013;8(7):e68976.PubMed CentralView ArticlePubMedGoogle Scholar
  37. Worrall G, Angel J, Chaulk P, Clarke C, Robbins M. Effectiveness of an educational strategy to improve family physicians’ detection and management of depression: a randomized controlled trial. CMAJ. 1999;161:37–40.PubMed CentralPubMedGoogle Scholar
  38. Dormuth CR, Maclure M, Bassett K, Jauca C, Whiteside C, Wright JM. Effect of periodic letters on evidence-based drug therapy on prescribing behaviour: a randomized trial. CMAJ. 2004;171:1057–61.PubMed CentralView ArticlePubMedGoogle Scholar
  39. Rahme E, Choquette D, Beaulieu M, Bessette L, Joseph L, Toubouti Y, et al. Impact of a general practitioner educational intervention on osteoarthritis treatment in an elderly population. Am J Med. 2005;118:1262–70.View ArticlePubMedGoogle Scholar
  40. Evans CE, Haynes RB, Birkett NJ, Gilbert RJ, Taylor DW, Sackett DL, et al. Does a mailed continuing education program improve physician performance. JAMA. 1986;255:501–4.View ArticlePubMedGoogle Scholar
  41. Dubey V, Mathew R, Iglar K, Moineddin R, Glazier R. Improving preventive service delivery at adult complete health check-ups: the preventive health evidence-based recommendation form (PERFORM) cluster randomized controlled trial. BMC Fam Pract. 2006;7:44.PubMed CentralView ArticlePubMedGoogle Scholar
  42. Shah BR, Bhattacharyya O, Yu CHY, Mamdani MM, Parsons JA, Straus SE, et al. Effect of an educational toolkit on quality of care: a pragmatic cluster randomized trial. PLoS Med. 2014;11.Google Scholar
  43. Zwarenstein M, Shiller SK, Croxford R, Grimshaw JM, Kelsall D, Paterson JM, et al. Printed educational messages aimed at family practitioners fail to increase retinal screening among their patients with diabetes: a pragmatic cluster randomized controlled trial [ISRCTN72772651]. Implement Sci. 2014;9:87.PubMed CentralView ArticlePubMedGoogle Scholar
  44. Bishop PB, Wing PC. Knowledge transfer in family physicians managing patients with acute low back pain: a prospective randomized control trial. Spine J. 2006;6:282–8.View ArticlePubMedGoogle Scholar
  45. Szonyi G, Millard R. Controlled trial evaluation of a general practitioner education package on incontinence: use of a mailed questionnaire. Br J Urol. 1994;73:615–20.View ArticlePubMedGoogle Scholar
  46. Liaw S-T, Sulaiman ND, Barton CA, Chondros P, Harris CA, Sawyer S, et al. An interactive workshop plus locally adapted guidelines can improve general practitioners asthma management and knowledge: a cluster randomised trial in the Australian setting. BMC Fam Pract. 2008;9:22.PubMed CentralView ArticlePubMedGoogle Scholar
  47. French SD, McKenzie JE, O’Connor DA, Grimshaw JM, Mortimer D, Francis JJ, et al. Evaluation of a theory-informed implementation intervention for the management of acute low back pain in general medical practice: the IMPLEMENT cluster randomised trial. PLoS One. 2013;8.Google Scholar
  48. Kunz R, Wegscheider K, Guyatt G, Zielinski W, Rakowsky N, Donner-Banzhoff N, et al. Impact of short evidence summaries in discharge letters on adherence of practitioners to discharge medication. A cluster-randomised controlled trial. Qual Saf Health Care. 2007;16:456–61.PubMed CentralView ArticlePubMedGoogle Scholar
  49. Butzlaff M, Vollmar H, Floer B, Koneczny N, Isfort J, Lange S. Learning with computerized guidelines in general practice? A randomized controlled trial. Fam Pract. 2004;21:183–8.View ArticlePubMedGoogle Scholar
  50. Ulbricht S, Gross B, Kunstmann W, John U, Meyer C. Efficacy of disseminating educational materials in general practices. Sucht. 2014;60:85–91.View ArticleGoogle Scholar
  51. Denig P, Haaijer-Ruskamp FM, Djurre ZH. Impact of a drug bulletin on the knowledge, perception of drug utility, and prescribing behaviour of physicians. DICP, Ann Pharmacother. 1990;24:87–93.Google Scholar
  52. Secher N, Mikkelsen MM, Adelborg K, Mikkelsen R, Grove EL, Rubak JM, et al. Direct mail improves knowledge of basic life support guidelines in general practice: a randomised study. Scand J Trauma Resusc Emerg Med. 2012;20:72.PubMed CentralView ArticlePubMedGoogle Scholar
  53. Tsuji SR, Atallah AN, Aranha FC, Tonhom AP, Siqueira AC, Matos D. Cluster randomized clinical trial (ISRCTN23732000) to evaluate the effectiveness of a diagnosis recognition and treatment guide for depressive disorders in primary care. J Eval Clin Pract. 2009;15:222–5.View ArticlePubMedGoogle Scholar
  54. Simon AE, Jegerlehner S, Müller T, Cattapan-Ludewig K, Frey P, Grossenbacher M, et al. Prodromal schizophrenia in primary care: a randomised sensitisation study. Br J Gen Pract. 2010;60:e353–9.PubMed CentralView ArticlePubMedGoogle Scholar
  55. Hunskaar S, Hannestad YS, Backe B, Matheson I. Direct mailing of consensus recommendations did not alter GPs’ knowledge and prescription of oestrogen in the menopause. Scand J Prim Health Care. 1996;14:203–8.View ArticlePubMedGoogle Scholar
  56. Perria C, Mandolini D, Guerrera C, Jefferson T, Billi P, Calzini V, et al. Implementing a guideline for the treatment of type 2 diabetics: results of a cluster-randomized controlled trial (C-RCT). BMC Health Serv Res. 2007;7:79.PubMed CentralView ArticlePubMedGoogle Scholar
  57. Katon W, Von Korff M, Lin E, Simon G. Rethinking practitioner roles in chronic illness: the specialist, primary care physician, and the practice nurse. Gen Hosp Psychiatry. 2001;23:138–44.View ArticlePubMedGoogle Scholar
  58. Zwarenstein M, Goldman J, Reeves S. Interprofessional collaboration: effects of practice-based interventions on professional practice and healthcare outcomes (review). Cochrane Database Syst Rev. 2009;3:CD000072.PubMedGoogle Scholar

Copyright

© Grudniewicz et al. 2015

Advertisement