- Open Access
- Open Peer Review
Looking inside the black box: results of a theory-based process evaluation exploring the results of a randomized controlled trial of printed educational messages to increase primary care physicians’ diabetic retinopathy referrals [Trial registration number ISRCTN72772651]
Implementation Sciencevolume 9, Article number: 86 (2014)
Theory-based process evaluations conducted alongside randomized controlled trials provide the opportunity to investigate hypothesized mechanisms of action of interventions, helping to build a cumulative knowledge base and to inform the interpretation of individual trial outcomes. Our objective was to identify the underlying causal mechanisms in a cluster randomized trial of the effectiveness of printed educational materials (PEMs) to increase referral for diabetic retinopathy screening. We hypothesized that the PEMs would increase physicians’ intention to refer patients for retinal screening by strengthening their attitude and subjective norm, but not their perceived behavioral control.
Design: A theory based process evaluation alongside the Ontario Printed Educational Material (OPEM) cluster randomized trial. Postal surveys based on the Theory of Planned Behavior were sent to a random sample of trial participants two months before and six months after they received the intervention. Setting: Family physicians in Ontario, Canada. Participants: 1,512 family physicians (252 per intervention group) from the OPEM trial were invited to participate, and 31.3% (473/1512) responded at time one and time two. The final sample comprised 437 family physicians fully completing questionnaires at both time points. Main outcome measures: Primary: behavioral intention related to referring patient for retinopathy screening; secondary: attitude, subjective norm, perceived behavioral control.
At baseline, family physicians reported positive intention, attitude, subjective norm, and perceived behavioral control to advise patients about retinopathy screening suggesting limited opportunities for improvement in these constructs. There were no significant differences on intention, attitude, subjective norm, and perceived behavioral control following the intervention. Respondents also reported additional physician- and patient-related factors perceived to influence whether patients received retinopathy screening.
Lack of change in the primary and secondary theory-based outcomes provides an explanation for the lack of observed effect of the main OPEM trial. High baseline levels of intention to advise patients to attend retinopathy screening suggest that post-intentional and other factors may explain gaps in care. Process evaluations based on behavioral theory can provide replicable and generalizable insights to aid interpretation of randomized controlled trials of complex interventions to change health professional behavior.
Printed educational materials (PEMs) are a commonly used mode for delivering professional behavior change interventions. A recent Cochrane review of printed educational materials identified seven randomized controlled trials (RCTs) with categorical outcomes and observed a median absolute risk difference of +2% (range 0% (deterioration of care) to +11%), and identified three RCTs with continuous outcomes, observing a median standardised mean difference of 0.13 (range from -0.16 to +0.36) . However the majority of included studies were small and had methodological problems. The Ontario Printed Educational Materials (OPEM) retinopathy trial, published alongside this process evaluation, is a large trial of PEMs and demonstrated no benefit of any form or combination of PEMs (‘short’ or ‘long’ educational messages with or without patient reminder notes) in improving attendance of family physicians’ patients with diabetes for retinopathy screening—only about 30% of patients in the study received regular diabetic retinopathy screening . The rigorous methods and precision of the estimated effect provided compelling evidence of the lack of any effect of the intervention within the trial, but provided no information about why the intervention was unsuccessful.
Process evaluations collect data alongside randomized trials of complex interventions to explore possible causal mechanisms and effect modifiers [3, 4]. This is akin to measuring intermediate endpoints in clinical trials to further understand the biological basis of any observed effects (for example, measuring lipid profiles alongside trials of lipid-lowering drugs where a primary endpoint could be reduction in vascular related deaths). Theory-based process evaluations offer the added advantage of using validated constructs (and measures) to explore hypothesized mechanisms of action of interventions that can help to build a cumulative knowledge base while also informing the interpretation of individual trial outcomes . The OPEM trial aimed to change physicians’ behavior, and we hypothesized that the interventions would achieve this through changes in physicians’ intention, as a consequence of enhancing their positive attitude to retinal screening, and their awareness of retinal screening as normative in this situation, but the interventions would not alter physicians’ perceptions of their abilities to carry out the behavior (refer for retinopathy screening). We conducted a prospective theory-based process evaluation alongside the OPEM trial to test the hypothesis that the intervention would operate through increasing physicians’ intention to refer for retinopathy screening  using the Theory of Planned Behavior (TPB), a well validated social cognition model [6–8].
The methods of both the OPEM trial and the OPEM theory-based process evaluation have been described in detail [5, 9]. The OPEM trial intervention was replicated across three different behaviors; in this paper, we report the results of the process evaluation of the OPEM retinopathy trial. Briefly, the OPEM retinopathy trial used a 2 × 3 factorial design to test the effectiveness of three forms of printed educational materials (short or long educational messages and patient reminder notes) to enhance physicians’ referrals of patients with diabetes for retinopathy screening. Retinopathy is a common complication in diabetes and when left untreated can lead to loss of vision. Prevention can be achieved through regular retinal screening and treatment when retinopathy is detected. Screening rates in Ontario are low, with less than half of people aged between 30 and 64 years newly diagnosed with diabetes being screened .
The main trial report provides a detailed description of the PEM interventions . Briefly, the PEMs were embedded within an existing newsletter mailed to 15,000 healthcare providers in Ontario. Physicians were randomized to receive a factorial combination of either a short bullet-point based outsert stapled to the outside of the newsletter; and/or a longer two-page insert within the newsletter. Those randomized to receive an outsert were also randomized to receive a pad of reminders that could be given to patients to remind them to make an eye exam appointment.
Theory of planned behavior and outcome measures
The TPB  states that the proximal antecedents of behavior are intention to perform the behavior and perceived behavioral control (PBC) over the behavior. Intention is predicted by three underlying social cognition-based mechanisms: attitude (i.e., in favor or not) towards the behavior in question, the influence of subjective norm (i.e., perceptions of others’ views, in this instance most likely peers and experts) concerning the behavior, and their perceived control over the behavior (PBC; i.e., the ease with which they could execute a referral should they wish to do so; see Figure 1). Therefore, intention was the primary outcome and attitude, subjective norm, and PBC were secondary outcomes and were believed to be prior in the causal pathway for referral.
Using standard methods [10, 11] we developed a TPB questionnaire that had 18 items, each scored with a seven point Likert scale (see Additional file 1). TPB items were reverse-scored if necessary so that high scores represented agreement (or positive attitude) and low scores, disagreement (or negative attitude). We used the mean of the items measuring each theory-based construct to create a composite score that ranged from one to seven for each construct: Intention (three items; pre- and post-test Cronbach’s α = 0.93), Attitude (five items, pre-test α = 0.92, post-test α = 0.93), Subjective Norm (five items, pre-test α = 0.92, post-test α = 0.94), PBC (four items, pre- and post-test α = 0.79).
Study participants and sample size
Study participants were randomly chosen from the family practitioner participants of the OPEM trial by the OPEM trial research team. We required 378 participants to have 80% power of detecting an effect size of 0.5 standard deviations using a significance level of 5%, assuming a 50% response rate for each survey (pre- and post-intervention) we surveyed 1,512 family physicians.
In addition, we asked an open-ended question about physicians’ perceptions of factors that might influence whether patients were screened for diabetic retinopathy. We surveyed the same participants two months before and six months after the OPEM retinopathy intervention. We used Dillman’s total design method to maximize response rates . In addition, we provided $20 (CDN) to every physician who returned a completed survey [13, 14].
Independent data entry verification was conducted for a 10% sample of responses and a data entry error rate of less than 1% was found. To compare responders and non-responders, we extracted personal and demographic data from a random sample of 20% of each group from the MD Select physician database .
Using analyses of covariance in SPSS, we compared groups factorially on their post-intervention TPB construct scores while adjusting for baseline differences to determine whether there had been changes in the predicted constructs across the study groups as hypothesized. We used a p-value of 0.01 to determine statistical significance to adjust for multiple testing.
The comments from the open ended questions were coded thematically by three independent coders and grouped into categories involving physicians’ beliefs about advising their patients to attend for retinal screening, and physicians’ perceptions about patients’ beliefs and actions following their advice. Discrepancies were resolved through majority decision and discussion with a fourth member of the team when required.
This study received approval from the Research Ethics Boards at The Ottawa Hospital and Sunnybrook Health Sciences Centre.
Response rates, representativeness of respondents and reliability of the composite measures
Six hundred and forty-nine participants (of 1,512; 42.9%) responded at time one and, of these, 473 (72.9%) responded at time two (giving a cumulative response rate of 31.3%). Table 1 details the response rates. There were no significant differences between pre-intervention respondents and non-respondents (Additional file 2). Respondents who completed both the pre- and post-intervention surveys were compared to those who completed only the pre-intervention questionnaire to explore the impact of response attrition on the results. Using each group’s pre-intervention TPB scores, analysis of variance found no significant differences between groups. Thirty-six cases (36/473 = 7.6%) were excluded from analyses (listwise deletion) due to missing data. The final sample was 437 Ontario family practitioners who responded to both pre- and post-intervention surveys and had no missing data on any of the TPB variables.
Assessing change in cognitions towards referring for diabetic retinopathy screening
Across all 649 respondents at baseline 580 (89.4%) physicians reported that they thought that their patients with diabetes should be screened annually or every one or two years.
Of the 437 participants providing pre- and post-intervention data, at pre-intervention, physicians reported strong intention (mean = 6.29), positive attitude (mean = 6.18), strong subjective norm (mean = 6.07) and strong perceived behavioral control (mean = 6.04) to refer the patient with diabetes for retinopathy screening (Table 2). No improvements in intention, attitude, subjective norm and perceived behavioral control were observed post intervention for any form of printed educational message (controlling for baseline scores; Table 3 for primary outcome ANCOVA results and Additional file 3 for secondary results).
Physicians’ perceptions about factors influencing diabetic retinopathy screening
Over 80% of physicians in the sample identified at least one barrier or facilitator in response to the open-ended question. Respondents identified a range of physician-level factors that influenced whether they refer patients about diabetic retinopathy screening including: their clinical assessment of the patient (‘Degree of diabetic control,’ ‘Any change in her vision’); their beliefs around when screening was required (‘No perceived need in asymptomatic patient’ , ‘I strongly recommend this for all my diabetic patients annually’); time constraints (‘Main obstacle is busy-ness of practice’); difficulty remembering to advise patients (‘Forgetting to tell patient to do so’); patient loss to follow up in general practice (‘Non-attendance at my clinic for regular diabetic tests/care’); and administrative burden associated with referral (‘I need to write referral letter to ophthalmologists every year’). Respondents also identified patient-level factors affecting screening including: patients’ perceptions of likely benefits from screening (‘Patient not convinced of importance of screening’); patients’ awareness of health insurance coverage issues (‘Sometimes people don't go because eye exams were delisted by Ontario Health Insurance Plan even though diabetics are covered’); distance to specialist services (‘Geographic location of specialists and travel requirements are the largest problems’); and waiting times for specialists appointment (‘Wait time to see specialist,’ ‘Difficulty of getting appointment with ophthalmologist’).
Summary of key findings
In the face of suboptimal performance, we identified very positive cognitions of family physicians to refer their patients with diabetes for retinopathy screening during the baseline period of the OPEM retinopathy trial. There were no statistically significant increases in these cognitions following the OPEM interventions. Physicians reported both physician and patient factors that they felt might influence referral of patients for retinopathy screening.
Interpretation and implications of results
The interventions in the OPEM trial were chosen based on empirical evidence of their potential effectiveness across a number of conditions and settings , rather than any specific, or theoretically driven, assessment of the determinants of retinopathy screening. The high levels of intention, and positive attitude, social norm, and PBC scores at baseline suggest that the majority of physicians already intended to perform this evidence-based behavior before the distribution of the printed educational materials. Further there was no increase in intention following the intervention. This suggests that factors other than intention and PBC at the physician level may account for the low rates of the desired behavior of patients. The physicians in the study were able to identify post-intentional factors at both the physician and patient level that may influence whether patients receive retinopathy screening. This could explain why educational materials or physician distributed patient reminders may not be effective for changing behaviors for which physicians’ intentions, perceived control, and underlying cognitions are already consistent with the evidence.
Strengths and limitations
The strengths of this study include the use of a well-validated theory to explore proposed causal mechanisms alongside a large rigorous pragmatic trial of a professional behavior change intervention. There is increasing recognition of the value of process evaluations alongside trials of complex interventions such as professional behavior change interventions [3, 4]. Commonly process evaluations have used qualitative methods to explore participants’ attitudes towards and experiences of study interventions that provide valuable context specific information that can help interpret the results of an individual trial, but may be less helpful in predicting the likely generalizability of findings due to the lack of standardized constructs and measurements. In contrast, our theory-based process evaluation used a previously validated behavioral theory concerning determinants of behavior that enhances the generalizability and replicability of our methods. Using a behavioral theory strengthened our ability to explore the reasons underlying the failure of a complex intervention to change health professional behavior, allowing future interventions to cumulatively build on these generalizable insights. The addition of an open-ended question provided some insight to the individual and contextual variables contributing to the sub-optimal retinopathy screening rate observed in the trial.
Potential study limitations relate to the survey response rates and choice of theory. The response rates to the surveys were similar to those in other recent surveys of physicians, though may indicate some selection bias favoring more motivated respondents; however, our non-response analyses suggest no major differences in key characteristics between responders and non-responders. Whilst we were confident that the TPB included the key constructs that we believed would be influenced by the interventions, it mainly focuses on a limited number of motivational factors and does not represent all of the potential determinants of family physicians’ behavior. Future theory-based process evaluations could take a broader theoretical approach.
Implications for research
The study has a number of implications for future research. First, this study highlights the importance of considering a broad range of potential determinants of health care professional behavior including post intentional factors. Second, it suggests that when planning intervention studies, researchers should prospectively identify factors likely to mediate the proposed mechanism of action for the behavior change and use this information to develop their intervention (in order to avoid evaluating interventions likely to be ineffective) and to identify variables which reflect the mediating processes (to enhance the informativeness of the trial). Third, theory-based process evaluations can be undertaken alongside evaluations of interventions, even where those interventions have not been developed based on an explicit behavioral theory, provided that the researchers conducting the process evaluation are able to make explicit hypotheses about the likely mediating mechanisms of the interventions and identify theories that correspond to these mechanisms. Finally, it reinforces the need to make explicit the hypothesized mechanism(s) of action for proposed professional behavior change interventions, which can be informed by the explicit use of validated theory.
Lack of change in the primary and secondary theory-based outcomes provides an explanation for the lack of observed effect of the main OPEM trial. High baseline levels of intention to advise patients to attend retinopathy screening suggest that post-intentional and other factors may explain gaps in care. Process evaluations based on behavioral theory can provide replicable and generalizable insights to aid interpretation of RCTs of complex interventions to change health professional behavior.
Giguère A, Légaré F, Grimshaw J, Turcotte S, Fiander M, Grudniewicz A, Makosso-Kallyth S, Wolf FM, Farmer AP, Gagnon M-P: Printed educational materials: effects on professional practice and healthcare outcomes. Cochrane Database Syst Rev. 2012, 10: CD004398-
Zwarenstein M, Shiller SK, Croxford R, Grimshaw JM, Kelsall D, Paterson JM, Laupacis A, Austin PC, Tu K, Yun L, Hux JE: Printed educational messages aimed at family practitioners fail to increase retinal screening among their patients with diabetes: a pragmatic cluster randomized controlled trial [ISRCTN72772651]. Implement Sci. 2014, 9: 87-
Oakley A, Strange V, Bonell C, Allen E, Stephenson J, Ripple Study T: Process evaluation in randomised controlled trials of complex interventions. BMJ. 2006, 332 (7538): 413-416.
Craig P, Dieppe P, Macintyre S, Michie S, Nazareth I, Petticrew M, Medical Research Council G: Developing and evaluating complex interventions: the new Medical Research Council guidance. BMJ. 2008, 337: a1655-
Grimshaw JM, Zwarenstein M, Tetroe J, Godin G, Graham I, Lemyre L, Eccles MP, Johnston M, Francis JJ, Hux J, O’Rourke K, Legare F, Presseau J: Looking inside the black box: a theory-based process evaluation alongside a randomised controlled trial of printed educational materials (the Ontario printed educational message, OPEM) to improve referral and prescribing practices in primary care in Ontario, Canada. Implement Sci. 2007, 2: 38-
Ajzen I: The theory of planned behavior. Organ Behav Hum Decis Process. 1991, 50: 179-211.
Armitage CJ, Conner M: Efficacy of the theory of planned behaviour: a meta-analytic review. Br J Soc Psychol. 2001, 40: 471-499.
Godin G, Bélanger-Gravel A, Eccles MP, Grimshaw JM: Healthcare professionals’ intentions and behaviours: a systematic review of studies based on social cognitive theories. Implement Sci. 2008, 3: 36-
Zwarenstein M, Hux JE, Kelsall D, Paterson M, Grimshaw JM, Davis D, Laupacis A, Evans M, Austin PC, Slaughter PM, Shiller SK, Croxford R, Tu K: The Ontario printed educational message (OPEM) trial to narrow the evidence-practice gap with respect to prescribing practices of general and family physicians: a cluster randomized controlled trial, targeting the care of individuals with diabetes and hypertension in Ontario, Canada. Implement Sci. 2007, 2: 37-
Francis JJ, Eccles MP, Johnston M, Walker A, Grimshaw JM, Foy R, Kaner EFS, Smith L, Bonetti D: Constructing questionnaires based on the theory of planned behaviour: A manual for health services researchers. 2004, Newcastle upon Tyne: University of Newcastle,http://pages.bangor.ac.uk/~pes004/exercise_psych/downloads/tpb_manual.pdf,
Godin G, Kok G: The theory of planned behavior: a review of its applications to health-related behaviors. Am J Health Promot. 1996, 11: 87-98.
Dillman DA: The design and administration of mail surveys. Annu Rev Sociol. 1991, 17: 225-248.
Asch DA, Christakis NA, Ubel PA: Conducting physician mail surveys on a limited budget: a randomized trial comparing $2 bill versus $5 bill incentives. Med Care. 1998, 36 (1): 95-99.
Halpern SD, Ubel PA, Berlin JA, Asch DA: Randomized trial of 5 dollars versus 10 dollars monetary incentives, envelope size, and candy to increase physician response rates to mailed questionnaires. Med Care. 2002, 40 (9): 834-839.
MD select - Canadian medical directory (on CD-ROM). Bus Inf Group. 2004
Grimshaw JM, Thomas RE, MacLennan G, Fraser C, Ramsay CR, Vale L, Whitty P, Eccles MP, Matowe L, Shirran L, Wensing M, Dijkstra R, Donaldson C: Effectiveness and efficiency of guideline dissemination and implementation strategies. Health Technol Assess. 2004, 8: 1-72.
We would like to thank Keith O‘Rourke for his contribution to the analysis strategy for the study and Ruth Croxford and Susan Shiller for their contribution to data management and retrievals at ICES.
The OPEM trial and the OPEM process evaluation were funded by the Canadian Institutes of Health Research (CIHR) Funding Reference Number KTS-73423. The preliminary work and student funding for JP was provided by the KT ICEBERG CIHR funded team grant. JMG, GG, FL hold Canada Research Chairs. JJF is funded by the Chief Scientist Office of the Scottish Health Directorates.
JMG, JP, IDG, FL, MPE are on the editorial board of Implementation Science, and were not involved in any decisions concerning this manuscript. The authors declare that they have no competing interests.
JMG and MZ conceived of the study. All authors contributed to the design of the study. JMG, JP, JT, JJF, NR, and MZ contributed to the analysis and interpretation of the data. All authors commented on drafts of the paper and have approved the submitted version. JMG is the guarantor for the paper. All authors read and approved the final manuscript.