The Minimum Quality Criteria Set (QI-MQCS) for critical appraisal: advancing the science of quality improvement
© Rubenstein et al. 2015
Published: 14 August 2015
Effective learning across related scientific investigations through evidence synthesis is critical to promoting evidence-based approaches to healthcare. Synthesis of findings from quality improvement intervention (QII) publications, however, poses challenges. We aimed to develop a critical appraisal instrument (the Minimum Quality Criteria Set or QI-MQCS) to promote identification, dissemination and implementation of findings from high quality QII evaluations.
We convened a 9 person expert panel to guide QII evidence synthesis methods development through a one year iterative telephone, survey and in-person panel process. We developed and empirically tested electronic search and screening methods for identifying QII publications, and a critical appraisal instrument. Finally, we iteratively tested and improved QI-MQCS psychometric properties based on review of 54 electronically searched and systematically screened QII articles.
Panelists agreed QI-MQCS should focus on QII specific domains, not evaluation design criteria. The 16 QI-MQCS domains address Organizational Motivation, Intervention Rationale, Intervention Description, Organizational Characteristics, Implementation, Study Design, Comparator Description, Data Sources, Timing, Adherence / Fidelity, Health Outcomes, Organizational Readiness, Penetration / Reach, Sustainability, Spread, and Limitations. The median inter-rater agreement for QI-MQCS items was kappa 0.57 (83% agreement). Items discriminated between studies in terms of quality (median criteria met 67%). Internal consistency measures indicated coherence without excessive conceptual overlap (Cronbach's alpha = 0.60, absolute mean inter-item correlation = 0.19). The critical appraisal instrument is accompanied by a user manual detailing What to consider, Where to look, and How to rate.
The QI-MQCS had acceptable psychometric properties for critical appraisal, and can support systematic review of diverse QII evaluations. It is a ready-to-use critical appraisal tool accompanied by a user manual and empirically tested forms and methods.
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.