Printed educational messages aimed at family practitioners fail to increase retinal screening among their patients with diabetes: a pragmatic cluster randomized controlled trial [ISRCTN72772651]
Implementation Science volume 9, Article number: 87 (2014)
Evidence of the effectiveness of printed educational messages in narrowing the gap between guideline recommendations and practice is contradictory. Failure to screen for retinopathy exposes primary care patients with diabetes to risk of eye complications. Screening is initiated by referral from family practitioners but adherence to guidelines is suboptimal. We aimed to evaluate the ability of printed educational messages aimed at family doctors to increase retinal screening of primary care patients with diabetes.
Design: Pragmatic 2×3 factorial cluster trial randomized by physician practice, involving 5,048 general practitioners (with 179,833 patients with diabetes). Setting: Ontario family practitioners. Interventions: Reminders (that retinal screening helps prevent diabetes-related vision loss and is covered by provincial health insurance for patients with diabetes) with prompts to encourage screening were mailed to each physician in conjunction with a widely-read professional newsletter. Alternative printed materials formats were an ‘outsert’ (short, directive message stapled to the outside of the newsletter), and/or a two-page, evidence-based article (‘insert’) and a pre-printed sticky note reminder for patients. Main outcome measure: A successful outcome was an eye examination (which includes retinal screening) provided to a patient with diabetes, not screened in the previous 12 months, within 90 days after visiting a family practitioner. Analysis accounted for clustering of doctors within practice groups.
No intervention effect was detected (eye exam rates were 31.6% for patients of control physicians, 31.3% for the insert, 32.8% for the outsert, 32.3% for those who received both, and 31.2% for those who received both plus the patient reminder with the largest 95% confidence interval around any effect extending from −1.3% to 1.1%).
This large trial conclusively failed to demonstrate any impact of printed educational messages on screening uptake. Despite their low cost, printed educational messages should not be routinely used in attempting to close evidence-practice gaps relating to diabetic retinopathy screening.
Gaps between evidence-based ideal clinical practice and what doctors actually do can be responsible for substantial harm [1, 2]. For example, beta-blockers, angiotensin-converting enzyme inhibitors, and lipid-lowering drugs have been shown to reduce re-hospitalizations and mortality following an acute myocardial infarction (AMI), yet these drugs are underprescribed . Printed educational messages (PEMs) directed to doctors may be one way to address such gaps. PEMs have the considerable advantage of low cost and easy dissemination, so even small improvements may be efficiently obtained . However, the evidence on their effectiveness is contradictory. Freemantle et al. conducted a systematic review of the effects of PEMs . None of the small number of included trials found statistically significant improvements in practice, leading to the conclusion that PEMs were ineffective. When Farmer et al. updated this review  using a superior analytic approach (summarising observed effect sizes instead of vote counting studies with statistically significant effects), they observed a median absolute improvement of care of 4.9% (range −8.0% to +9.6%), on par with other much more expensive interventions like audit and feedback, or academic outreach . A further updated review by Giguère et al. observed smaller improvements in care (median absolute improvement of care of 2.0%, range 0 to +11.0%), and argued for the need for additional primary research and further investigation of different characteristics of PEMS in head-to-head comparisons. These contradictory findings leave policy makers uncertain about the role of PEMs, at a time when closing evidence-to-practice gaps may offer a more cost-effective investment of health system resources than developing new interventions .
Reviews cannot overcome limitations of the primary evidence. There are few trials of PEMs, often small in size with methodological weaknesses (for example unit of analysis errors in cluster randomized trials). To clarify the usefulness of PEMs, large, pragmatic [7, 10], and well-designed randomized controlled trials on the effect of PEMs on guideline adherence, conducted in real world settings, amongst typical practitioners are needed. The Ontario Printed Education Materials (OPEM) trial on retinopathy screening was such a trial: a large pragmatic cluster randomised trial powered to be able to detect small effects.
In Ontario, diabetes mellitus (diabetes) affects over 15% of women and 19% of men aged 65 years and older. Diabetic retinopathy is a common complication, affecting about 70% of persons with type 1 diabetes and 40% of persons with type 2 diabetes, and is the leading cause of preventable blindness in Canadians aged 30 to 69 . Most vision loss related to diabetic retinopathy can be prevented through retinal screening examinations and appropriate treatment. Early treatment has been shown to decrease the risk of severe vision loss from proliferative diabetic retinopathy by 90% and the risk of vision loss from macular oedema by 50% . Evidence-based practice guidelines in Canada call for screening for retinopathy at the time of diagnosis and every two years thereafter for everyone with type 2 diabetes .
Under the Ontario Health Insurance Plan (OHIP) Ontario residents have universal public health insurance, including free at point of service access to family practitioner (FP) care. Eye examinations (which include retinal screening), performed by an optometrist, FP, or ophthalmologist, are also an insured service, and thus are free at point of service to patients without any out-of-pocket payment, for all adults aged 65 years and older and also for those under age 65 who have been diagnosed with diabetes.
Yet in Ontario, the observed screening rates within the first year of diabetes diagnosis falls far below recommended levels (43% and 50% for diabetes patients aged 30 to 49 and 50 to 64 respectively).
If the impact on screening uptake of PEMs was indeed 5% improvement, then at the trivial cost of a letter to each FP, Ontario’s rate of screening and treatment, especially among younger patients, could be substantially increased, averting blindness in more than 100 Ontarians per annum .
We report here a trial of PEMs, evaluating their impact on screening rates for diabetic retinopathy. The objective of the study was to determine whether sending a PEM to FPs (and the format of the PEM) would affect the likelihood that a patient visiting their doctor would receive an eye exam within 90 days of their visit.
These are presented in detail in our published protocol .
Study design and randomisation
The study was a pragmatic, factorial, cluster-randomized controlled trial (Table 1). The study intervention was mailed on 1 April 2005 and the follow up period extended from 1 July 2005 to 30 June 2006. To prevent contamination (sharing of information among doctors in group practice) we randomised at the level of the practice. FPs were placed into practices on the basis of a shared street address. Practices were randomly assigned to an intervention group by the study statistician, using computer-generated random numbers.
The study participants were all FPs with an active practice in Ontario in 2003/2004. ‘Active’ practice was defined as a total billing volume for the year of at least $50,000 and writing prescriptions for at least 100 different patients, with at least one prescription in at least 10 of the 12 months.
The Ontario Diabetes Database was used to identify Ontario residents diagnosed with diabetes on or before 31 March 2005. Individuals were included if they were at least 30 years old on 31 March 2005, were alive at the end of the follow-up period (30 June 2006), and visited one of the target FPs within one year of the intervention mail-out.
Patients who had an eye exam in the preceding nine months were excluded from the study.
The printed educational materials interventions
Informed was a free, peer-reviewed, evidence-based primary care practice synopsis, written and produced by clinical and research staff from the Institute for Clinical Evaluative Sciences (ICES, http://www.ices.on.ca), and mailed to nearly 15,000 health care providers in Ontario from 1994 to 2007.
We designed two types of PEM to address the identified evidence-practice gap: a short, directive, evidence-based PEM on a postcard-sized card stapled to the front page of informed (the ‘outsert’); and a two-page insert, indistinguishable from the rest of informed in size and style (the ‘insert’). The insert excluded the directive statements of the outsert but included more background, a summarised evidence-based guideline, and references. Both were developed over three months of meetings with input from five ICES staff with knowledge translation experience (an internist, two family physicians, and two knowledge translation researchers), the writing team of the informed newsletter and a communications consultant.
We also issued a pad of take-home reminders (aimed at patients, to remind them to make an appointment for an eye exam), to be given to patients by their FP. Because it was not clear whether the reminder would be any more effective than the doctor’s verbal advice, doctors randomised to receive the outsert were also randomized to receive or not receive a pad of patient-aimed reminder slips.
Insert, outsert, and patient reminder were included with the 1 April 2005 edition of informed.
Practices were randomly assigned to one of four intervention groups. The two intervention groups selected to receive an outsert were further randomly divided into two subgroups, one of which received the patient reminder notepad.
The following administrative data sources were used . These data sets were held securely in a linked, de-identified form and analyzed at the Institute for Clinical Evaluative Sciences.
The OHIP Claim History Database details payments to health care professionals, including an encrypted provider number unique to each health care professional, an anonymous, encrypted patient identifier unique to each patient, the service provided, and the service date.
The Registered Persons Database contains basic demographic information for each insured person.
The Corporate Provider Database contains limited demographic and practice information for each practitioner.
Finally, we used the Ontario Diabetes Database, a validated registry of Ontario residents diagnosed with diabetes .
Records from these databases were linked, using the patient and provider identifiers, to determine which individuals with diabetes visited one of the target physicians in the year following the intervention, and whether that individual subsequently received an eye examination.
The primary outcome measure was whether or not an eligible trial patient received an eye exam within 90 days of their first FP visit during the one-year period following the mail-out. Because the PEMs specifically addressed the issue of insured eye exams for patients with diabetes younger than 65 years, a secondary analysis examined the impact of patient age on the uptake of eye exams.
Based on Monte Carlo simulations, assuming an intra-cluster correlation coefficient of 0.1 , three patients with diabetes per physician, and a baseline rate of screening of 36%, a trial with 1,250 practices per arm would provide over 97% power to detect an absolute increase of 5% in the screening rate, and over 98% power to distinguish between the effects of the combined intervention and either alone, assuming the effect of each to be 5% and the combined effect to be additive.
Analysis was on an intention-to-treat basis. Logistic regression models, estimated using generalized estimating equations (GEE) methods to account for the clustering of patients within physician practices, were used to estimate the effect of the intervention on the likelihood of patient screening. Two logistic regression models were fitted: one including only the interventions (insert and/or outsert and the patient reminders), and a second also adjusting for patient- and physician-level covariates. All analyses were performed using SAS version 9.1 (SAS Institute, Cary, North Carolina). Two-tailed p-values less than or equal to 0.05 were considered to be significant.
This study was approved by the Research Ethics Board at Sunnybrook Health Sciences Centre.
Physician and patient selection
Figure 2 shows the number of physicians and patients included in the study. Three-quarters of patients with diabetes who met the inclusion criteria visited one of the physicians targeted by this study during the follow up period. However, two-thirds of these individuals had already had an eye examination in the nine months immediately prior to the office visit and were excluded (Figure 2).
There were small, clinically unimportant, differences between the demographics of patients with diabetes who paid a visit to a study physician and those who did not, and between those who were and were not included in the analysis (Table 2). Patients aged 65 years and older were significantly more likely to visit a primary care doctor, and also more likely to have received a recent eye exam (and hence to be excluded from the analysis).
There were no meaningful physician differences among the intervention groups (Table 3).
Analysis of intervention effects
Intervention effects are shown in Table 4. Neither the unadjusted nor the adjusted results show any evidence that the interventions (alone or in combination), were effective in increasing rates of eye examination among eligible trial patients. The widest confidence interval reported in the table, an odds ratio between 0.94 and 1.04 for the insert + outsert + reminder notepad, corresponds to a true absolute effect of the intervention lying between a decrease in retinal screening of 1.3% and an increase of 1.1%. Thus, the estimate of effect for the presumably strongest ‘dose’ of intervention is of trivial clinical value, is not statistically significantly superior compared no intervention, and the confidence interval does not contain values of clinical importance.
While the probability of having an eye examination depended on patient age and rises sharply at age 65, there was no indication that intervention effectiveness varied with patient age.
The intracluster correlation coefficient was 0.024 (95% confidence interval 0.022 to 0.025).
In the face of the previous uncertainty about the effectiveness of printed educational materials in changing physician behaviour, this study, with sufficient power to detect even a small effect, failed to change practitioner behaviour, as measured by retinal screening rates among patients with diabetes. The intervention was a faithful operationalisation of the sort of printed educational materials that are routinely used and explored both long and short formats; this can be regarded as an evaluation of a realistic intervention.
The effectiveness of printed educational materials depends on the message being received by the physicians at whom the PEMs were aimed, read, judged credible, transmitted to (and acted on by) patients who might benefit from screening. The first three steps appear likely to have occurred: The reliability of the postal services in Canada are such that we can be confident that the materials were successfully delivered. They are also likely to have been read and judged credible: In 1997, The Strategic Council Inc., a market research organisation, contacted 500 Ontario physicians by phone to determine readership and recall of informed. They found that 71% of the respondents recalled receiving informed and that of these, 89% found it useful or very useful and 53% read most or every issue (internal report, personal communication, Dianne Kelsall, editor of informed).
It is possible that the physicians did recommend screening but did not make a referral or that patients did not keep their ophthalmology appointments. However, given that eye examinations are free for Ontario patients with diabetes and for all people aged 65 and over (in which group eye examination among patients with diabetes rises to 89% coverage within one year of diagnosis), it seems unlikely that patients with diabetes would fail to take up their physician’s recommendation simply because they are not yet 65. Thus patient failure to respond to the physician recommendation seems to be an unlikely explanation for the lack of impact of our intervention. We believe that the message was likely received, read, and judged credible by physicians, and would not have been ignored by patients. Yet PEMs were not successful at improving patient screening rates.
This suggests that PEMs simply failed to improve physician knowledge regarding the availability of free tests for patients with diabetes who were aged under 65; alternatively the PEMs failed to either influence the transmission of this message to patients or to lead to referral.
The content of our PEMs was not based on a prior assessment of barriers to screening and so it is possible that these are not the relevant barriers. Perhaps the actual barriers to improved screening lie elsewhere, with possibilities including the organisation of screening or patient factors such as motivation or geographical access. These potential barriers are indicated by the process study conducted in parallel with this RCT on a subsample of practitioners, and reported alongside .
This study is reliable and was conducted under realistic conditions to mimic typical programme delivery. The use of administrative data as the source of our outcome measure allowed us to include nearly the complete population of family practitioners and all of their eligible patients, to be able to provide a definitive result. This trial is one of the largest published randomised trials of knowledge translation or implementation to date, and given its power a false negative result is unlikely.
This real-world study supports the conclusion that PEMs, whether long and discursive or short and directive, and with or without patient reminder notes, do not change physician behaviour as measured by adherence to evidence based screening recommendations for diabetic retinopathy. These results strongly suggest that the use of PEMs alone is not a useful strategy for closing this evidence-practice gap, and underline the importance of continuing to explore complementary or alternative behaviour change interventions. We have replicated these results for two prescribing behaviours (in preparation) and further replication for diagnostic and other therapeutic behaviours would be informative.
Retired: Pamela M. Slaughter.
Hollaway R, Benesch C, Rush S: Stroke prevention: narrowing the evidence practice gap. Neurol. 2000, 54: 1899-1906. 10.1212/WNL.54.10.1899.
McGlynn E, Asch S, Adams J, Keesey J, Hicks J, DeCristofaro A, Kerr E: The quality of health care delivered to adults in the United States. N Engl J Med. 2003, 348 (26): 2635-2645. 10.1056/NEJMsa022615.
Beck C, Lauzon C, Eisenberg M, Huynh T, Dion D, Roux R, Racine N, Carignan S, Diodati J, Charbonneau F, Lévesque C, Pouliot JJ, Pilote LL: Discharge prescriptions following admission for acute myocardial infarction at tertiary care and community hospitals in Quebec. Can J Cardiol. 2001, 17 (1): 33-40.
Lomas J: Words without action? The production, dissemination and impact of consensus recommendations. Ann Rev Pub Health. 1991, 12: 41-65. 10.1146/annurev.pu.12.050191.000353.
Freemantle N, Harvey E, Wolf F, Grimshaw J, Grilli R, Bero L: Printed educational materials: effects on professional practice and health care outcomes. Cochrane Database Syst Rev. 2000, 2: CD000172-
Farmer AP, Legare F, Turcot L, Grimshaw JM, Harvey E, McGowan JL, Wolf F: Printed educational materials: effects on professional practice and health care outcomes. Cochrane Database Syst Rev. 2008, 3: 004398-
Grimshaw JM, Thomas RE, MacLennan G, Fraser C, Ramsay CR, Vale L, Whitty P, Eccles MP, Matowe L, Shirran L, Wensing M, Dijkstra R, Donaldson C: Effectiveness and efficiency of guideline dissemination and implementation strategies. Health Technol Assess. 2004, 8: 1-72.
Giguère A, Légaré F, Grimshaw J, Turcotte S, Fiander M, Grudniewicz A, Makosso-Kallyth S, Wolf FM, Farmer AP, Gagnon M-P: Printed educational materials: effects on professional practice and healthcare outcomes. Cochrane Database Syst Rev. 2012, 10: CD004398-
Woolf S, Johnson R: The break-even point: when medical advances are less important than improving the fidelity with which they are delivered. Ann Fam Med. 2005, 3: 545-552. 10.1370/afm.406.
Schwartz D, Lellouch J: Explanatory and pragmatic attitudes in therapeutic trials. J Chron Dis. 1967, 20: 637-648. 10.1016/0021-9681(67)90041-0.
Buhrmann R, Assaud D, Hux J, Tang M, Sykora K: Diabetes and the eye. Diabetes in Ontario: An ICES Practice Atlas. Edited by: Hux J, Booth G, Slaughter P, Laupacis A. 2003, Toronto, Canada: Institute for Clinical Evaluative Sciences
Klein R, Klein B: Diabetic eye disease. Lancet. 1997, 350: 197-204. 10.1016/S0140-6736(97)04195-0.
Meltzer S, Leiter L, Daneman D, Gerstein H, Lau D, Ludwig S, Yale J, Zinman B, Lillie D: 1998 clinical practice guidelines for the management of diabetes in Canada. CMAJ. 1998, 159 (Suppl 8): S1-S29.
Preliminary report on effects of photocoagulation therapy: The Diabetic Retinopathy Study Research Group. Am J Ophthalmol. 1976, 81 (4): 383-396.
Zwarenstein M, Hux JE, Kelsall D, Paterson M, Grimshaw JM, Davis D, Laupacis A, Evans M, Austin PC, Slaughter PM, Shiller SK, Croxford R, Tu K: The Ontario printed educational message (OPEM) trial to narrow the evidence-practice gap with respect to prescribing practices of general and family physicians: a cluster randomized controlled trial, targeting the care of individuals with diabetes and hypertension in Ontario Canada. Implement Sci. 2007, 2: 37-10.1186/1748-5908-2-37.
Ontario Ministry of Health and Long-Term Care, Ontario Health Planning Data Guide, Release 3.0. [http://www.health.gov.on.ca/transformation/providers/information/im_resources.html#analyst]
Hux J, Ivis F, Flintoft V, Bica A: Diabetes in Ontario: determination of prevalence and incidence using a validated administrative data algorithm. Diabetes Care. 2002, 25: 512-516. 10.2337/diacare.25.3.512.
Campbell M, Fayers P, Grimshaw J: Determinants of the intracluster correlation coefficient in cluster randomized trials: the case of implementation research. Clin Trials. 2005, 2 (2): 99-107. 10.1191/1740774505cn071oa.
Grimshaw JM, Presseau J, Tetroe J, Eccles MP, Francis JJ, Godin G, Graham ID, Hux JE, Johnston M, Légaré F, Lemyre L, Robinson N, Zwarenstein M: Looking inside the black box: Results of a theory-based process evaluation exploring the results of a randomised controlled trial of printed educational messages to increase primary care physicians’ diabetic retinopathy referrals. Implement Sci.
This study was funded by the Canadian Institutes for Health Research, under grant 724180703. The Canadian Institutes of Health Research had no role in the design and conduct of the study; collection, management, analysis and interpretation of the data; or in the preparation, review or approval of the manuscript. This study was also supported by the Institute for Clinical Evaluative Sciences (ICES), which is funded by an annual grant from the Ontario Ministry of Health and Long-Term Care (MOHLTC). The opinions, results and conclusions reported in this paper are those of the authors and are independent from the funding sources. No endorsement by ICES or the Ontario MOHLTC is intended or should be inferred.
All authors declare no financial relationships with any organisations that might have an interest in the submitted work in the previous three years, no other relationships or activities that could appear to have influenced the submitted work.
MZ conceived the study, wrote the protocol and led the analysis and writeup of results. He takes responsibility for the integrity of the data and the accuracy of the data analysis. JH, JG, PA, SS, and AL contributed to protocol development and design. SS, PS, KT, JH, DK, and MZ contributed to design of the insert, outsert and patient reminders. RC, JG and LY contributed to the analysis and writeup. All authors contributed to editing and approved the final version for submission.
Electronic supplementary material
About this article
Cite this article
Zwarenstein, M., Shiller, S.K., Croxford, R. et al. Printed educational messages aimed at family practitioners fail to increase retinal screening among their patients with diabetes: a pragmatic cluster randomized controlled trial [ISRCTN72772651]. Implementation Sci 9, 87 (2014). https://doi.org/10.1186/1748-5908-9-87