This article has Open Peer Review reports available.
Looking inside the black box: results of a theory-based process evaluation exploring the results of a randomized controlled trial of printed educational messages to increase primary care physicians’ diabetic retinopathy referrals [Trial registration number ISRCTN72772651]
© Grimshaw et al.; licensee BioMed Central Ltd. 2014
Received: 14 September 2013
Accepted: 21 April 2014
Published: 6 August 2014
Theory-based process evaluations conducted alongside randomized controlled trials provide the opportunity to investigate hypothesized mechanisms of action of interventions, helping to build a cumulative knowledge base and to inform the interpretation of individual trial outcomes. Our objective was to identify the underlying causal mechanisms in a cluster randomized trial of the effectiveness of printed educational materials (PEMs) to increase referral for diabetic retinopathy screening. We hypothesized that the PEMs would increase physicians’ intention to refer patients for retinal screening by strengthening their attitude and subjective norm, but not their perceived behavioral control.
Design: A theory based process evaluation alongside the Ontario Printed Educational Material (OPEM) cluster randomized trial. Postal surveys based on the Theory of Planned Behavior were sent to a random sample of trial participants two months before and six months after they received the intervention. Setting: Family physicians in Ontario, Canada. Participants: 1,512 family physicians (252 per intervention group) from the OPEM trial were invited to participate, and 31.3% (473/1512) responded at time one and time two. The final sample comprised 437 family physicians fully completing questionnaires at both time points. Main outcome measures: Primary: behavioral intention related to referring patient for retinopathy screening; secondary: attitude, subjective norm, perceived behavioral control.
At baseline, family physicians reported positive intention, attitude, subjective norm, and perceived behavioral control to advise patients about retinopathy screening suggesting limited opportunities for improvement in these constructs. There were no significant differences on intention, attitude, subjective norm, and perceived behavioral control following the intervention. Respondents also reported additional physician- and patient-related factors perceived to influence whether patients received retinopathy screening.
Lack of change in the primary and secondary theory-based outcomes provides an explanation for the lack of observed effect of the main OPEM trial. High baseline levels of intention to advise patients to attend retinopathy screening suggest that post-intentional and other factors may explain gaps in care. Process evaluations based on behavioral theory can provide replicable and generalizable insights to aid interpretation of randomized controlled trials of complex interventions to change health professional behavior.
Printed educational materials (PEMs) are a commonly used mode for delivering professional behavior change interventions. A recent Cochrane review of printed educational materials identified seven randomized controlled trials (RCTs) with categorical outcomes and observed a median absolute risk difference of +2% (range 0% (deterioration of care) to +11%), and identified three RCTs with continuous outcomes, observing a median standardised mean difference of 0.13 (range from -0.16 to +0.36) . However the majority of included studies were small and had methodological problems. The Ontario Printed Educational Materials (OPEM) retinopathy trial, published alongside this process evaluation, is a large trial of PEMs and demonstrated no benefit of any form or combination of PEMs (‘short’ or ‘long’ educational messages with or without patient reminder notes) in improving attendance of family physicians’ patients with diabetes for retinopathy screening—only about 30% of patients in the study received regular diabetic retinopathy screening . The rigorous methods and precision of the estimated effect provided compelling evidence of the lack of any effect of the intervention within the trial, but provided no information about why the intervention was unsuccessful.
Process evaluations collect data alongside randomized trials of complex interventions to explore possible causal mechanisms and effect modifiers [3, 4]. This is akin to measuring intermediate endpoints in clinical trials to further understand the biological basis of any observed effects (for example, measuring lipid profiles alongside trials of lipid-lowering drugs where a primary endpoint could be reduction in vascular related deaths). Theory-based process evaluations offer the added advantage of using validated constructs (and measures) to explore hypothesized mechanisms of action of interventions that can help to build a cumulative knowledge base while also informing the interpretation of individual trial outcomes . The OPEM trial aimed to change physicians’ behavior, and we hypothesized that the interventions would achieve this through changes in physicians’ intention, as a consequence of enhancing their positive attitude to retinal screening, and their awareness of retinal screening as normative in this situation, but the interventions would not alter physicians’ perceptions of their abilities to carry out the behavior (refer for retinopathy screening). We conducted a prospective theory-based process evaluation alongside the OPEM trial to test the hypothesis that the intervention would operate through increasing physicians’ intention to refer for retinopathy screening  using the Theory of Planned Behavior (TPB), a well validated social cognition model [6–8].
The methods of both the OPEM trial and the OPEM theory-based process evaluation have been described in detail [5, 9]. The OPEM trial intervention was replicated across three different behaviors; in this paper, we report the results of the process evaluation of the OPEM retinopathy trial. Briefly, the OPEM retinopathy trial used a 2 × 3 factorial design to test the effectiveness of three forms of printed educational materials (short or long educational messages and patient reminder notes) to enhance physicians’ referrals of patients with diabetes for retinopathy screening. Retinopathy is a common complication in diabetes and when left untreated can lead to loss of vision. Prevention can be achieved through regular retinal screening and treatment when retinopathy is detected. Screening rates in Ontario are low, with less than half of people aged between 30 and 64 years newly diagnosed with diabetes being screened .
The main trial report provides a detailed description of the PEM interventions . Briefly, the PEMs were embedded within an existing newsletter mailed to 15,000 healthcare providers in Ontario. Physicians were randomized to receive a factorial combination of either a short bullet-point based outsert stapled to the outside of the newsletter; and/or a longer two-page insert within the newsletter. Those randomized to receive an outsert were also randomized to receive a pad of reminders that could be given to patients to remind them to make an eye exam appointment.
Theory of planned behavior and outcome measures
Using standard methods [10, 11] we developed a TPB questionnaire that had 18 items, each scored with a seven point Likert scale (see Additional file 1). TPB items were reverse-scored if necessary so that high scores represented agreement (or positive attitude) and low scores, disagreement (or negative attitude). We used the mean of the items measuring each theory-based construct to create a composite score that ranged from one to seven for each construct: Intention (three items; pre- and post-test Cronbach’s α = 0.93), Attitude (five items, pre-test α = 0.92, post-test α = 0.93), Subjective Norm (five items, pre-test α = 0.92, post-test α = 0.94), PBC (four items, pre- and post-test α = 0.79).
Study participants and sample size
Study participants were randomly chosen from the family practitioner participants of the OPEM trial by the OPEM trial research team. We required 378 participants to have 80% power of detecting an effect size of 0.5 standard deviations using a significance level of 5%, assuming a 50% response rate for each survey (pre- and post-intervention) we surveyed 1,512 family physicians.
In addition, we asked an open-ended question about physicians’ perceptions of factors that might influence whether patients were screened for diabetic retinopathy. We surveyed the same participants two months before and six months after the OPEM retinopathy intervention. We used Dillman’s total design method to maximize response rates . In addition, we provided $20 (CDN) to every physician who returned a completed survey [13, 14].
Independent data entry verification was conducted for a 10% sample of responses and a data entry error rate of less than 1% was found. To compare responders and non-responders, we extracted personal and demographic data from a random sample of 20% of each group from the MD Select physician database .
Using analyses of covariance in SPSS, we compared groups factorially on their post-intervention TPB construct scores while adjusting for baseline differences to determine whether there had been changes in the predicted constructs across the study groups as hypothesized. We used a p-value of 0.01 to determine statistical significance to adjust for multiple testing.
The comments from the open ended questions were coded thematically by three independent coders and grouped into categories involving physicians’ beliefs about advising their patients to attend for retinal screening, and physicians’ perceptions about patients’ beliefs and actions following their advice. Discrepancies were resolved through majority decision and discussion with a fourth member of the team when required.
This study received approval from the Research Ethics Boards at The Ottawa Hospital and Sunnybrook Health Sciences Centre.
Response rates, representativeness of respondents and reliability of the composite measures
Participant flow by group
Retinopathy screening (N=437)
Patient Reminder Note
n=252 Baseline questionnaires sent n=144 Baseline non-response
n=252 Baseline questionnaires sent n=136 Baseline non-response
n=36 Follow-up attrition
n=34 Follow-up attrition
No Patient Reminder Note
n=252 Baseline questionnaires sent n=150 Baseline non-response
n=252 Baseline questionnaires sent n=146 Baseline non-response
n=20 Follow-up attrition
n=29 Follow-up attrition
No Patient Reminder Note
n=252 Baseline questionnaires sent n=141 Baseline non-response
n=252 Baseline questionnaires sent n=146 Baseline non-response
n=33 Follow-up attrition
n=24 Follow-up attrition
Assessing change in cognitions towards referring for diabetic retinopathy screening
Across all 649 respondents at baseline 580 (89.4%) physicians reported that they thought that their patients with diabetes should be screened annually or every one or two years.
Pre- and post-intervention descriptive statistics of primary and secondary outcomes by intervention group
Mean ( SD)
Mean ( SD)
Mean ( SD)
Mean ( SD)
No patient reminder
No patient reminder
No patient reminder
No patient reminder
Analysis of covariance for primary outcome of change in intention
Physicians’ perceptions about factors influencing diabetic retinopathy screening
Over 80% of physicians in the sample identified at least one barrier or facilitator in response to the open-ended question. Respondents identified a range of physician-level factors that influenced whether they refer patients about diabetic retinopathy screening including: their clinical assessment of the patient (‘Degree of diabetic control,’ ‘Any change in her vision’); their beliefs around when screening was required (‘No perceived need in asymptomatic patient’ , ‘I strongly recommend this for all my diabetic patients annually’); time constraints (‘Main obstacle is busy-ness of practice’); difficulty remembering to advise patients (‘Forgetting to tell patient to do so’); patient loss to follow up in general practice (‘Non-attendance at my clinic for regular diabetic tests/care’); and administrative burden associated with referral (‘I need to write referral letter to ophthalmologists every year’). Respondents also identified patient-level factors affecting screening including: patients’ perceptions of likely benefits from screening (‘Patient not convinced of importance of screening’); patients’ awareness of health insurance coverage issues (‘Sometimes people don't go because eye exams were delisted by Ontario Health Insurance Plan even though diabetics are covered’); distance to specialist services (‘Geographic location of specialists and travel requirements are the largest problems’); and waiting times for specialists appointment (‘Wait time to see specialist,’ ‘Difficulty of getting appointment with ophthalmologist’).
Summary of key findings
In the face of suboptimal performance, we identified very positive cognitions of family physicians to refer their patients with diabetes for retinopathy screening during the baseline period of the OPEM retinopathy trial. There were no statistically significant increases in these cognitions following the OPEM interventions. Physicians reported both physician and patient factors that they felt might influence referral of patients for retinopathy screening.
Interpretation and implications of results
The interventions in the OPEM trial were chosen based on empirical evidence of their potential effectiveness across a number of conditions and settings , rather than any specific, or theoretically driven, assessment of the determinants of retinopathy screening. The high levels of intention, and positive attitude, social norm, and PBC scores at baseline suggest that the majority of physicians already intended to perform this evidence-based behavior before the distribution of the printed educational materials. Further there was no increase in intention following the intervention. This suggests that factors other than intention and PBC at the physician level may account for the low rates of the desired behavior of patients. The physicians in the study were able to identify post-intentional factors at both the physician and patient level that may influence whether patients receive retinopathy screening. This could explain why educational materials or physician distributed patient reminders may not be effective for changing behaviors for which physicians’ intentions, perceived control, and underlying cognitions are already consistent with the evidence.
Strengths and limitations
The strengths of this study include the use of a well-validated theory to explore proposed causal mechanisms alongside a large rigorous pragmatic trial of a professional behavior change intervention. There is increasing recognition of the value of process evaluations alongside trials of complex interventions such as professional behavior change interventions [3, 4]. Commonly process evaluations have used qualitative methods to explore participants’ attitudes towards and experiences of study interventions that provide valuable context specific information that can help interpret the results of an individual trial, but may be less helpful in predicting the likely generalizability of findings due to the lack of standardized constructs and measurements. In contrast, our theory-based process evaluation used a previously validated behavioral theory concerning determinants of behavior that enhances the generalizability and replicability of our methods. Using a behavioral theory strengthened our ability to explore the reasons underlying the failure of a complex intervention to change health professional behavior, allowing future interventions to cumulatively build on these generalizable insights. The addition of an open-ended question provided some insight to the individual and contextual variables contributing to the sub-optimal retinopathy screening rate observed in the trial.
Potential study limitations relate to the survey response rates and choice of theory. The response rates to the surveys were similar to those in other recent surveys of physicians, though may indicate some selection bias favoring more motivated respondents; however, our non-response analyses suggest no major differences in key characteristics between responders and non-responders. Whilst we were confident that the TPB included the key constructs that we believed would be influenced by the interventions, it mainly focuses on a limited number of motivational factors and does not represent all of the potential determinants of family physicians’ behavior. Future theory-based process evaluations could take a broader theoretical approach.
Implications for research
The study has a number of implications for future research. First, this study highlights the importance of considering a broad range of potential determinants of health care professional behavior including post intentional factors. Second, it suggests that when planning intervention studies, researchers should prospectively identify factors likely to mediate the proposed mechanism of action for the behavior change and use this information to develop their intervention (in order to avoid evaluating interventions likely to be ineffective) and to identify variables which reflect the mediating processes (to enhance the informativeness of the trial). Third, theory-based process evaluations can be undertaken alongside evaluations of interventions, even where those interventions have not been developed based on an explicit behavioral theory, provided that the researchers conducting the process evaluation are able to make explicit hypotheses about the likely mediating mechanisms of the interventions and identify theories that correspond to these mechanisms. Finally, it reinforces the need to make explicit the hypothesized mechanism(s) of action for proposed professional behavior change interventions, which can be informed by the explicit use of validated theory.
Lack of change in the primary and secondary theory-based outcomes provides an explanation for the lack of observed effect of the main OPEM trial. High baseline levels of intention to advise patients to attend retinopathy screening suggest that post-intentional and other factors may explain gaps in care. Process evaluations based on behavioral theory can provide replicable and generalizable insights to aid interpretation of RCTs of complex interventions to change health professional behavior.
We would like to thank Keith O‘Rourke for his contribution to the analysis strategy for the study and Ruth Croxford and Susan Shiller for their contribution to data management and retrievals at ICES.
The OPEM trial and the OPEM process evaluation were funded by the Canadian Institutes of Health Research (CIHR) Funding Reference Number KTS-73423. The preliminary work and student funding for JP was provided by the KT ICEBERG CIHR funded team grant. JMG, GG, FL hold Canada Research Chairs. JJF is funded by the Chief Scientist Office of the Scottish Health Directorates.
- Giguère A, Légaré F, Grimshaw J, Turcotte S, Fiander M, Grudniewicz A, Makosso-Kallyth S, Wolf FM, Farmer AP, Gagnon M-P: Printed educational materials: effects on professional practice and healthcare outcomes. Cochrane Database Syst Rev. 2012, 10: CD004398-PubMedGoogle Scholar
- Zwarenstein M, Shiller SK, Croxford R, Grimshaw JM, Kelsall D, Paterson JM, Laupacis A, Austin PC, Tu K, Yun L, Hux JE: Printed educational messages aimed at family practitioners fail to increase retinal screening among their patients with diabetes: a pragmatic cluster randomized controlled trial [ISRCTN72772651]. Implement Sci. 2014, 9: 87-View ArticlePubMedPubMed CentralGoogle Scholar
- Oakley A, Strange V, Bonell C, Allen E, Stephenson J, Ripple Study T: Process evaluation in randomised controlled trials of complex interventions. BMJ. 2006, 332 (7538): 413-416.View ArticlePubMedPubMed CentralGoogle Scholar
- Craig P, Dieppe P, Macintyre S, Michie S, Nazareth I, Petticrew M, Medical Research Council G: Developing and evaluating complex interventions: the new Medical Research Council guidance. BMJ. 2008, 337: a1655-View ArticlePubMedPubMed CentralGoogle Scholar
- Grimshaw JM, Zwarenstein M, Tetroe J, Godin G, Graham I, Lemyre L, Eccles MP, Johnston M, Francis JJ, Hux J, O’Rourke K, Legare F, Presseau J: Looking inside the black box: a theory-based process evaluation alongside a randomised controlled trial of printed educational materials (the Ontario printed educational message, OPEM) to improve referral and prescribing practices in primary care in Ontario, Canada. Implement Sci. 2007, 2: 38-View ArticlePubMedPubMed CentralGoogle Scholar
- Ajzen I: The theory of planned behavior. Organ Behav Hum Decis Process. 1991, 50: 179-211.View ArticleGoogle Scholar
- Armitage CJ, Conner M: Efficacy of the theory of planned behaviour: a meta-analytic review. Br J Soc Psychol. 2001, 40: 471-499.View ArticlePubMedGoogle Scholar
- Godin G, Bélanger-Gravel A, Eccles MP, Grimshaw JM: Healthcare professionals’ intentions and behaviours: a systematic review of studies based on social cognitive theories. Implement Sci. 2008, 3: 36-View ArticlePubMedPubMed CentralGoogle Scholar
- Zwarenstein M, Hux JE, Kelsall D, Paterson M, Grimshaw JM, Davis D, Laupacis A, Evans M, Austin PC, Slaughter PM, Shiller SK, Croxford R, Tu K: The Ontario printed educational message (OPEM) trial to narrow the evidence-practice gap with respect to prescribing practices of general and family physicians: a cluster randomized controlled trial, targeting the care of individuals with diabetes and hypertension in Ontario, Canada. Implement Sci. 2007, 2: 37-View ArticlePubMedPubMed CentralGoogle Scholar
- Francis JJ, Eccles MP, Johnston M, Walker A, Grimshaw JM, Foy R, Kaner EFS, Smith L, Bonetti D: Constructing questionnaires based on the theory of planned behaviour: A manual for health services researchers. 2004, Newcastle upon Tyne: University of Newcastle,http://pages.bangor.ac.uk/~pes004/exercise_psych/downloads/tpb_manual.pdf,Google Scholar
- Godin G, Kok G: The theory of planned behavior: a review of its applications to health-related behaviors. Am J Health Promot. 1996, 11: 87-98.View ArticlePubMedGoogle Scholar
- Dillman DA: The design and administration of mail surveys. Annu Rev Sociol. 1991, 17: 225-248.View ArticleGoogle Scholar
- Asch DA, Christakis NA, Ubel PA: Conducting physician mail surveys on a limited budget: a randomized trial comparing $2 bill versus $5 bill incentives. Med Care. 1998, 36 (1): 95-99.View ArticlePubMedGoogle Scholar
- Halpern SD, Ubel PA, Berlin JA, Asch DA: Randomized trial of 5 dollars versus 10 dollars monetary incentives, envelope size, and candy to increase physician response rates to mailed questionnaires. Med Care. 2002, 40 (9): 834-839.View ArticlePubMedGoogle Scholar
- MD select - Canadian medical directory (on CD-ROM). Bus Inf Group. 2004Google Scholar
- Grimshaw JM, Thomas RE, MacLennan G, Fraser C, Ramsay CR, Vale L, Whitty P, Eccles MP, Matowe L, Shirran L, Wensing M, Dijkstra R, Donaldson C: Effectiveness and efficiency of guideline dissemination and implementation strategies. Health Technol Assess. 2004, 8: 1-72.View ArticleGoogle Scholar
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly credited. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.