Use of routine healthcare data in randomised implementation trials: a methodological mixed-methods systematic review

Background Routine data are increasingly used in randomised controlled trials evaluating healthcare interventions. They can aid participant identification, outcome assessment, and intervention delivery. Randomised implementation trials evaluate the effect of implementation strategies on implementation outcomes. Implementation strategies, such as reminders, are used to increase the uptake of evidence-based interventions into practice, while implementation outcomes, such as adoption, are key measures of the implementation process. The use of routine data in effectiveness trials has been explored; however, there are no reviews on implementation trials. We therefore aimed to describe how routine data have been used in randomised implementation trials and the design characteristics of these trials. Methods We searched MEDLINE (Ovid) and Cochrane Central Register of Controlled Trials from Jan 2000 to Dec 2021 and manually searched protocols from trial registers. We included implementation trials and type II and type III hybrid effectiveness-implementation trials conducted using routine data. We extracted quantitative and qualitative data and narratively synthesised findings. Results From 4206 titles, we included 80 trials, of which 22.5% targeted implementation of evidence-based clinical guidelines. Multicomponent implementation strategies were more commonly evaluated (70.0%) than single strategies. Most trials assessed adoption as the primary outcome (65.0%). The majority of trials extracted data from electronic health records (EHRs) (62.5%), and 91.3% used routine data for outcome ascertainment. Reported reasons for using routine data were increasing efficiency, assessing outcomes, reducing research burden, improving quality of care, identifying study samples, confirming findings, and assessing representativeness. Data quality, the EHR system, research governance, and external factors such as government policy could act either as facilitators or barriers. Conclusions Adherence to guidance on designing and reporting implementation studies, and specifically to harmonise the language used in describing implementation strategies and implementation outcomes, would aid identification of studies and data extraction. Routine healthcare data are widely used for participant identification, outcome assessment and intervention delivery. Researchers should familiarise themselves with the barriers and facilitators to using routine data, and efforts could be made to improve data quality to overcome some of the barriers. Registration PROSPERO CRD42022292321. Supplementary Information The online version contains supplementary material available at 10.1186/s13012-023-01300-4.


Data items
10a List and define all outcomes for which data were sought.Specify whether all results that were compatible with each outcome domain in each study were sought (e.g. for all measures, time points, analyses), and if not, the methods used to decide which results to collect.

Data extraction and analysis
10b List and define all other variables for which data were sought (e.g.participant and intervention characteristics, funding sources).Describe any assumptions made about any missing or unclear information.

Data extraction and analysis, table 1
Study risk of bias assessment 11 Specify the methods used to assess risk of bias in the included studies, including details of the tool(s) used, how many reviewers assessed each study and whether they worked independently, and if applicable, details of automation tools used in the process.

N/A
Effect measures 12 Specify for each outcome the effect measure(s) (e.g.risk ratio, mean difference) used in the synthesis or presentation of results.N/A,

Item # Checklist item
Location where item is reported findings 13c Describe any methods used to tabulate or visually display results of individual studies and syntheses.

Descriptive tables/figures
13d Describe any methods used to synthesize results and provide a rationale for the choice(s).If meta-analysis was performed, describe the model(s), method(s) to identify the presence and extent of statistical heterogeneity, and software package(s) used.

Data extraction and analysis
13e Describe any methods used to explore possible causes of heterogeneity among study results (e.g.subgroup analysis, meta-regression).N/A,

Narrative synthesis
13f Describe any sensitivity analyses conducted to assess robustness of the synthesized results.N/A

Reporting bias assessment
14 Describe any methods used to assess risk of bias due to missing results in a synthesis (arising from reporting biases).N/A Certainty assessment 15 Describe any methods used to assess certainty (or confidence) in the body of evidence for an outcome.N/A

Study selection 16a
Describe the results of the search and selection process, from the number of records identified in the search to the number of studies included in the review, ideally using a flow diagram.

Results section; PRISMA flow diagram
16b Cite studies that might appear to meet the inclusion criteria, but which were excluded, and explain why they were excluded.The degree to which an intervention was implemented as it was prescribed in the original protocol or as it was intended by the program developers

Cost-effectiveness; Cost-benefit
The cost impact of an implementation effort.

Spread
The integration of a practice within a service setting and its subsystems.Confirming findings "The administrative database analysis offers partial validation for the assessment of adherence to medication." [s61]

SUSTAINABILITY
"We evaluated the reliability of the trial results against routinely collected data from the health check management information system."[s39]

Increasing efficiency
"The conduct of trials using electronic health records as a means for increasing the speed and efficiency with which trials can be completed is a topic of growing importance."[s39] "Registries can track influenza immunisation rates in recalled children and can increase the efficiency of reminder/recall by restricting subsequent mailings only to those patients who are not immunised after an initial recall letter."[s7]

Reducing research burden
"We were able to embed the intervention within the workflow of a primary care clinic, which resulted in high rates of participation by providers and a protocol that did not increase provider burden." [s68] "It minimised the burden of research for practices and encouraged recruitment and it reduced the cost compared to trial-specific data collection."[s37] "To maximise efficiency, the approach involved automated enrolment from an existing provincial registry, centralised delivery of the intervention, and use of routinely collected administrative data." [s61] "Embedding multi-arm, pragmatic trials of interventions aiming at implementing improved care or outcomes within routine operations of health systems can help to reduce research waste in implementation science by producing generalisable scientific results, and findings that directly improve patient care" [s61] "CHSP data enabled us to assess maternal age and deprivation at population level."[s16] "Our trial was a cost-effective approach to reach a population of over 100 000 at-risk patients, costing less than £1 per patient."[s37] "An EHR-based tool is available that can facilitate meaningful use documentation" [s43] "The modern electronic health record (EHR) has previously been used to increase the likelihood of CHB screening via message prompts to providers 24 hours prior to appointments involving at-risk patients.

" [s48]
"There was no reliable and practical biochemical method of verifying that an infant is breastfed, and strong concerns were voiced that seeking direct proof of breastfeeding (eg, through observation of a feed) would have a negative effect on the relationship between clinicians and women.We therefore used the method by which infant feeding status is recorded for the purposes of routine data collection in the UK's National Health Service.[54] "Incorporating alerts into electronic medical records (EMRs) has proved effective for improving the practice of different preventive care" [s35] Additional file 6. Reported rationale themes with full examples

Reported facilitators and barriers of using RHD in RITs
s21.Amemori M, Virtanen J, Korhonen T, et al.Impact of educational intervention on implementation of tobacco counselling among oral health professionals: a cluster-randomized community trial.Community Dent Oral Epidemiol 2013;41(2):120-9.s22.Cohen AN, Chinman MJ, Hamilton AB, et al.Using patient-facing kiosks to support quality improvement at mental health clinics.Med Care 2013;51(3 Suppl 1):S13-20.s23.French SD, McKenzie JE, O'Connor DA, et al.Evaluation of a theory-informed implementation intervention for the management of acute low back pain in general medical practice: the IMPLEMENT cluster randomised trial.PLoS One 2013;8(6):e65471.s24.Green BB, Wang CY, Anderson ML, et al.An automated intervention with stepped increases in support to increase uptake of colorectal cancer screening: a randomized trial.Ann Intern Med 2013;158(5 Pt 1):301-11.s25.Kilgore ML, Outman R, Locher JL, et al.Multimodal intervention to improve osteoporosis care in home health settings: results from a cluster randomized trial.Osteoporos Int 2013;24(10):2555-60. s26.Persell SD, Lloyd-Jones DM, Friesema EM, et al.Electronic health record-based patient identification and individualized mailed outreach for primary cardiovascular disease prevention: a cluster randomized trial.J Gen Intern Med 2013;28(4):554-60.s27.Smidth M, Christensen MB, Fenger-Gron M, et al.The effect of an active implementation of a disease management programme for chronic obstructive pulmonary disease on healthcare utilization--a cluster-randomised controlled trial.BMC Health Serv Res 2013;13:385.s28.Dombkowski KJ, Costello LE, Harrington LB, et al.Age-specific strategies for immunization reminders and recalls: a registry-based randomized trial.Am J Prev Med 2014;47(1):1-8.s29.Radde K, Gottschalk A, Bussas U, et al.Invitation to cervical cancer screening does increase participation in Germany: Results from the MARZY study.Int J Cancer 2016;139(5):1018-30. s30.Taveras EM, Marshall R, Horan CM, et al.Improving children's obesity-related health care quality: process outcomes of a cluster-randomized controlled trial.Obesity (Silver Spring) 2014;22(1):27-31.s31.Mertens JR, Chi FW, Weisner CM, et al.Physician versus non-physician delivery of alcohol screening, brief intervention and referral to treatment in adult primary care: the ADVISe cluster randomized controlled implementation trial.Addict Sci Clin Pract 2015;10:26.s32.Palacio AM, Uribe C, Hazel-Fernandez L, et al.Can phone-based motivational interviewing improve medication adherence to antiplatelet medications after a coronary stent among racial minorities?A randomized trial.J Gen Intern Med 2015;30(4):469-75.s33.Sinnema H, Majo MC, Volker D, et al.Effectiveness of a tailored implementation programme to improve recognition, diagnosis and treatment of anxiety and depression in general practice: a cluster randomised controlled trial.Implement Sci 2015;10:33.

and Topic Item # Checklist item Location where item is reported 20b
Present results of all statistical syntheses conducted.If meta-analysis was done, present for each the summary estimate and its precision (e.g.confidence/credible interval) and measures of statistical heterogeneity.If comparing groups, describe the direction of the effect.
20a For each synthesis, briefly summarise the characteristics and risk of bias among contributing studies.Study characteristics sections Section N/A 20c Present results of all investigations of possible causes of heterogeneity among study results.N/A 20d Present results of all sensitivity analyses conducted to assess the robustness of the synthesized results.
Franks AL, Lee NC, et al.A randomized intervention to improve ongoing participation in mammography.Am J Manag Care 2001;7(9):887-94.s2.Wilcox SA, Koepke CP, Levenson R, et al.Registry-driven, community-based immunization outreach: a randomized controlled trial.Scheitel SM, Fitz JS, et al.A Randomized Trial of Three Diabetes Registry Implementation Strategies in a Community Internal Medicine Practice.The Joint Commission Journal on Quality Improvement 2002;28(8):441-50.s5.Tierney WM, Overhage JM, Murray MD, et al.Effects of computerized guidelines for managing heart disease in primary care.J Gen Intern Med 2003;18(12):967-76.s6.Bahrami M, Deery C, Clarkson JE, et al.Effectiveness of strategies to disseminate and implement clinical guidelines for the management of impacted and unerupted third molars in primary dental care, a cluster randomised controlled trial.Br Dent J 2004;197(11):691-6; discussion 88.s7.Daley MF, Barrow J, Pearson K, et al.Identification and recall of children with chronic medical conditions for influenza vaccination.Pediatrics 2004;113(1 Pt 1):e26-33.Academic detailing has no effect on prescribing of asthma medication in Danish general practice: a 3-year randomized controlled trial with 12-monthly follow-ups.Fam Pract 2004;21(3):248-53.s10.Kempe A, Daley MF, Barrow J, et al.Implementation of universal influenza immunization recommendations for healthy young children: results of a randomized, controlled trial with registry-based recall.Pediatrics 2005;115(1):146-54.s11.Stein K, Lewendon G, Jenkins R, et al.Improving uptake of cervical cancer screening in women with prolonged history of non-attendance for screening: a randomized trial of enhanced invitation methods.J Med Screen 2005;12(4):185-9.s12.Cleland JA, Hall S, Price D, et al.An exploratory, pragmatic, cluster randomised trial of practice nurse training in the use of asthma action plans.Rigotti NA, Schneider LI, et al.An electronic health record-based intervention to improve tobacco treatment in primary care: a cluster-randomized controlled trial.Arch Intern Med 2009;169(8):781-7.s18.Peremans L, Rethans JJ, Verhoeven V, et al.Empowering patients or general practitioners?A randomised clinical trial to improve quality in reproductive health care in Belgium.Eur J Contracept Reprod Health Care 2010;15(4):280-9.s19.Chaney EF, Rubenstein LV, Liu CF, et al.Implementing collaborative care for depression treatment in primary care: a cluster randomized evaluation of a quality improvement practice redesign.Implement Sci 2011;6:121.s20.Krist AH, Woolf SH, Rothemich SF, et al.Interactive preventive health record to enhance delivery of recommended care: a randomized trial.Ann Fam Med 2012;10(4):312-9.
s34.Forberg U, Unbeck M, Wallin L, et al.Effects of computer reminders on complications of peripheral venous catheters and nurses' adherence to a guideline in paediatric care--a cluster randomised study.Implement Sci 2016;11:10.s35.Guiriguet C, Munoz-Ortiz L, Buron A, et al.Alerts in electronic medical records to promote a colorectal cancer screening programme: a cluster randomised controlled trial in primary care.Br J Gen Pract 2016;66(648):e483-90. s36.Hemming K, Ryan R, Gill P, et al.Targeted case finding in the prevention of cardiovascular disease: a stepped wedge cluster randomised controlled trial.Wright AJ, Cornelius V, et al.Enhanced invitation methods and uptake of health checks in primary care: randomised controlled trial and cohort study using electronic health records.Health Technol Assess 2016;20(84):1-92.s40.Nowalk MP, Lin CJ, Pavlik VN, et al.Using the 4 Pillars Practice Transformation Program to increase adult Tdap immunization in a randomized controlled cluster trial.Vaccine 2016;34(41):5026-33. s41.Arts DL, Abu-Hanna A, Medlock SK, et al.Effectiveness and usage of a decision support system to improve stroke prevention in general practice: A cluster randomized controlled trial.PLoS One 2017;12(2):e0170974. s42.Chinman M, McCarthy S, Hannah G, et al.Using Getting To Outcomes to facilitate the use of an evidence-based practice in VA homeless programs: a cluster-randomized trial of an implementation support strategy.Meade C, Hull SK, et al.Integration of Personalized Health Planning and Shared Medical Appointments for Patients with Type 2 Diabetes Mellitus.South Med J 2018;111(11):674-82. s51.McClure JB, Anderson ML.Evaluation of a population-level strategy to promote tobacco treatment use among insured smokers: a pragmatic, randomized trial.BMC Public Health 2018;18(1):228.s52.Nielson CM, Rivelli JS, Fuoco MJ, et al.Effectiveness of automated and live phone reminders after mailed-FIT outreach in a pilot randomized trial.Prev Med Rep 2018;12:210-13.s53.Pallan M, Hurley KL, Griffin T, et al.A cluster-randomised feasibility trial of a children's weight management programme: the Child weigHt mANaGement for Ethnically diverse communities (CHANGE) study.Pilot Feasibility Stud 2018;4:175.s54.Relton C, Strong M, Thomas KJ, et al.Effect of Financial Incentives on Breastfeeding: A Cluster Randomized Clinical Trial.JAMA Pediatr 2018;172(2):e174523. s55.Svane JK, Chiou ST, Groene O, et al.A WHO-HPH operational program versus usual routines for implementing clinical health promotion: an RCT in health promoting hospitals (HPH).Implement Sci 2018;13(1):153.s56.Bauer MS, Miller CJ, Kim B, et al.Effectiveness of Implementing a Collaborative Chronic Care Model for Clinician Teams on Patient Outcomes and Health Status in Mental Health: A Randomized Clinical Trial.JAMA Netw Open 2019;2(3):e190230.s57.Gold R, Bunce A, Cowburn S, et al.Does increased implementation support improve community clinics' guideline-concordant care?Results of a mixed methods, pragmatic comparative effectiveness trial.Implement Sci 2019;14(1):100.s58.King C, Llewellyn C, Shahmanesh M, et al.Sexual risk reduction interventions for patients attending sexual health clinics: a mixed-methods feasibility study.Health Technol Assess 2019;23(12):1-122.s59.Wang SV, Rogers JR, Jin Y, et al.Stepped-wedge randomised trial to evaluate population health intervention designed to increase appropriate anticoagulation in patients with atrial fibrillation.BMJ Qual Saf 2019;28(10):835-42.s60.Cai Y, Gong W, He H, et al.Mobile Texting and Lay Health Supporters to Improve Schizophrenia Care in a Resource-Poor Community in Rural China (LEAN Trial): Amroze A, Vakil F, et al.SUPPORT-AF II: Supporting Use of Anticoagulants Through Provider Profiling of Oral Anticoagulant Therapy for Atrial Fibrillation: A Cluster-Randomized Study of Electronic Profiling and Messaging Combined With Academic Detailing for Providers Making Decisions About Anticoagulation in Patients With Atrial Fibrillation.Circ Cardiovasc Qual Outcomes 2020;13(2):e005871.s64.Lipscomb J, Escoffery C, Gillespie TW, et al.Improving Screening Uptake among Breast Cancer Survivors and Their First-Degree Relatives at Elevated Risk to Breast Cancer: Results and Implications of a Randomized Study in the State of Georgia.Int J Environ Res Public Health 2020;17(3).s65.Lum HD, Dukes J, Daddato AE, et al.Effectiveness of Advance Care Planning Group Visits Among Older Adults in Primary Care.J Am Geriatr Soc 2020;68(10):2382-89. s66.Lowery JT, Horick N, Kinney AY, et al.A randomized trial to increase colonoscopy screening in members of high-risk families in the colorectal cancer family registry and cancer genetics network.Cancer Epidemiol Biomarkers Prev 2014;23(4):601-10.
Additional file 5. Summary of single implementation strategies used in included trials The use of electronic health records (EHRs) offers potential for improving the quality of care for obese children and for accelerating the adoption of research evidence regarding childhood obesity screening and management.It also holds promise for establishing treatment benchmarks and for supporting patients and their clinical teams in care improvement" [s30]"Collection of routine data could allow care quality to be monitored, and improved, by identifying patients who would benefit from a specific evidence-based treatment, fostering access to and monitoring retention in that treatment."[s22]EHRs have the potential to increase adherence to guidelines, and it could be used to remind clinicians to document smoking status and deliver brief advice, prompt clinicians to prescribe cessation medications, and facilitate referrals to counseling." [s17] "For vaccination outreach, data recorded in the electronic health record through routine care delivery can inform real-time identification of unvaccinated populations."[s72]"Dataavailable in the EHR can be utilised to identify the appropriate target population for a smoking cessation intervention."[s74]"Wecomparedroutinelycollected child data on study participants and programme participants not taking part in the study to assess representativeness of the study sample."[s53]"Use of EHR allowed for studying the natural process of care, studying a large number of patients, and having a population-based sample of patients and providers, rather than limiting our sample to those willing to be recruited into a study."[s31] "