- Open Access
Applying GRADE-CERQual to qualitative evidence synthesis findings—paper 3: how to assess methodological limitations
Implementation Sciencevolume 13, Article number: 9 (2018)
The GRADE-CERQual (Confidence in Evidence from Reviews of Qualitative research) approach has been developed by the GRADE (Grading of Recommendations Assessment, Development and Evaluation) Working Group. The approach has been developed to support the use of findings from qualitative evidence syntheses in decision-making, including guideline development and policy formulation.
CERQual includes four components for assessing how much confidence to place in findings from reviews of qualitative research (also referred to as qualitative evidence syntheses): (1) methodological limitations, (2) coherence, (3) adequacy of data and (4) relevance. This paper is part of a series providing guidance on how to apply CERQual and focuses on CERQual’s methodological limitations component.
We developed the methodological limitations component by searching the literature for definitions, gathering feedback from relevant research communities and developing consensus through project group meetings. We tested the CERQual methodological limitations component within several qualitative evidence syntheses before agreeing on the current definition and principles for application.
When applying CERQual, we define methodological limitations as the extent to which there are concerns about the design or conduct of the primary studies that contributed evidence to an individual review finding. In this paper, we describe the methodological limitations component and its rationale and offer guidance on how to assess methodological limitations of a review finding as part of the CERQual approach. This guidance outlines the information required to assess methodological limitations component, the steps that need to be taken to assess methodological limitations of data contributing to a review finding and examples of methodological limitation assessments.
This paper provides guidance for review authors and others on undertaking an assessment of methodological limitations in the context of the CERQual approach. More work is needed to determine which criteria critical appraisal tools should include when assessing methodological limitations. We currently recommend that whichever tool is used, review authors provide a transparent description of their assessments of methodological limitations in a review finding. We expect the CERQual approach and its individual components to develop further as our experiences with the practical implementation of the approach increase.
The GRADE-CERQual (Confidence in Evidence from Reviews of Qualitative research) approach has been developed by the GRADE (Grading of Recommendations Assessment, Development and Evaluation) Working Group. The approach has been developed to support the use of findings from qualitative evidence syntheses in decision-making, including guideline development and policy formulation. GRADE-CERQual (hereafter referred to as CERQual) includes four components for assessing how much confidence to place in findings from reviews of qualitative research (also referred to as qualitative evidence syntheses): (1) methodological limitations, (2) coherence, (3) adequacy of data and (4) relevance. This paper focuses on one of these four components: methodological limitations.
When carrying out a CERQual assessment, we define methodological limitations as the extent to which there are concerns about the design or conduct of the primary studies that contributed evidence to an individual review finding. Where the primary studies underlying a review finding are assessed as having methodological limitations, and these are considered to have a clear/direct impact on the review finding, we may be less confident that the review finding reflects the phenomenon of interest .
When assessing methodological limitations, our goal is not to judge whether some absolute standard of methodological quality has been achieved, but rather to indicate concerns where any methodological limitations have been identified as serious enough to lower our confidence in the review finding. The methodological limitations component is analogous to the risk of bias domain used in the GRADE approach for findings from systematic reviews of effectiveness [2, 3].
The aim of this paper, part of a series (Fig. 1), is to describe what we mean by methodological limitations of the body of data (data from included primary studies) contributing to a review finding in the context of a qualitative evidence synthesis and to give guidance on how to operationalise this component in the context of a review finding as part of the CERQual approach. This paper should be read in conjunction with the papers describing the other three CERQual components [4,5,6] and the paper describing how to make an overall CERQual assessment of confidence and create a summary of qualitative findings table . Key definitions for the series are provided in Additional file 1.
How CERQual was developed
The initial stages of the process for developing CERQual, which started in 2010, are outlined elsewhere . Since then, we have further refined the current definitions of each component and the principles for application of the overall approach using a number of methods. When developing CERQual’s methodological limitations component, we undertook informal searches of the literature, including Google and Google Scholar for relevant critical appraisal tools, and for definitions and discussion papers related to the concept of methodological quality in the context of qualitative research. We carried out similar searches for the other three components. We presented an early version of the CERQual approach in 2015 to a group of methodologists, researchers and end users with experience in qualitative research, GRADE or guideline development. We further refined the approach through training workshops, seminars and presentations during which we actively sought, collated and shared feedback, by facilitating discussions of individual CERQual components within relevant organisations, through applying the approach within diverse qualitative evidence syntheses [8,9,10,11,12,13,14,15,16,17,18] and through supporting other teams in using CERQual [19, 20]. As far as possible, we used a consensus approach in these processes. We also gathered feedback from CERQual users through an online feedback form and through short individual discussions with members of the review teams. The methods used to develop CERQual are described in more detail in the first paper in this series .
Assessing methodological limitations
Methodological limitations in the context of findings from qualitative evidence syntheses
The methodological approaches used in a primary study may have consequences for how much we can trust the findings from that study. Where there are concerns regarding the appropriateness of these approaches (e.g. data collection or analysis methods), or how the studies were conducted, study findings may be produced that are not an adequate representation of the phenomenon of interest. For example, we may have less trust in findings from a study where participants were recruited in a manner that did not fully address the aims of the research or where the data analysis methods were not appropriate for the study design.
One or more studies contribute data to each review finding in a qualitative evidence synthesis, and these data make up the body of data for a review finding. The methodological limitations of the body of data supporting a review finding are assessed as a whole to identify whether or not any methodological weaknesses within individual studies impact our confidence in a review finding. The methodological limitations for each review finding must be assessed separately since different studies contribute varying amounts of data to each review finding, and methodological quality issues may have varying impacts on different review findings. For example, the same set of studies may contribute data to many review findings. However, individual features of study design may have implications for some of those review findings, but not necessarily other review findings. Methodological limitations of the body of data may weaken our overall assessment of confidence in the review findings to which these studies contribute. See Table 1 for the examples of review findings with concerns regarding methodological limitations.
Critical appraisal of qualitative research
The extent to which it is possible or appropriate to critically appraise the methodological quality of qualitative research is contested among researchers in the field [22,23,24,25]. However, the starting point for the CERQual approach is that there is a need for ‘clear evaluative criteria that are responsive to the unique nature of qualitative inquiry’ ( p. 113).
Despite the existence of more than 50 guidelines for assessing the quality of qualitative research , there is no agreement on the best approach for assessing the methodological quality of primary qualitative studies. Even where essential criteria for assessing methodological quality have been agreed upon, there are challenges related to the definitions underlying these criteria, and how much importance should be given to them within a critical appraisal tool  (p. 151).
The methodological limitations component of the CERQual approach requires some systematic and transparent approach for identifying methodological weaknesses of individual studies. Critical appraisal tools can help us to identify such weaknesses. We are, however, dependent on the quality and completeness of the individual study reports. The Cochrane Qualitative and Implementation Methods Group recommends that review authors consider a number of issues when choosing a tool to assess methodological strengths and limitations of qualitative studies, including the type of designs or methods selected to address the review question and the included primary studies . In addition, the group advises using tools that privilege assessment of methodological strengths and limitations over the quality of reporting . In the context of CERQual, we take a similar approach, and currently recommend that review authors use an approach that fits their review question and synthesis methods, with which they are familiar and comfortable using, and that focuses on methodological strengths and limitations. Regardless of the approach chosen, the review authors should provide a detailed and transparent assessment for each element of the tool. In line with Cochrane Qualitative and Implementation Methods Group guidance, we do not recommend using reporting guidelines as proxies for quality appraisal tools (see Table 2). To date, review authors using the CERQual approach have primarily used the Critical Appraisal Skills Programme (CASP) checklist or an adapted version of it . Research is underway to examine which elements of critical appraisal are key for assessing the quality of research in the context of qualitative evidence synthesis and for use in the CERQual approach. See Table 2 for an outline of areas where further work is needed with respect to critical appraisal tools for qualitative research.
Guidance on how to assess methodological limitations in the context of a review finding
The steps taken when assessing methodological limitations are shown in Fig. 2 and detailed below.
Step 1: collect and consider the necessary information related to methodological limitations
To assess methodological limitations of the body of data contributing to a review finding, you first need to choose an appropriate critical appraisal tool to assess the methodological strengths and limitations of the primary studies contributing data to the review finding. Regardless of the chosen tool, you will need to collect detailed information regarding the methods of data collection and analysis used in each study, as well as other aspects covered by the critical appraisal tool that you have chosen. The level of detail reported on the conduct of the included studies may vary greatly depending on the study design, the topic/field, type of publication or journal specifications.
When applying CERQual to a review that you have conducted, you will normally have gathered this information during data extraction, as this is a standard part of the review process. When doing so, you should present and explain in detail the assessments of each criterion within the critical appraisal tool for each primary study. Some review teams choose to present this as a matrix of methodological limitations of included studies. However, if you are applying CERQual to findings from somebody else’s review, you will need to have access to their critical appraisal assessments of the included studies, which are often published as part of the review. Where these assessments are not available, you may need to go directly to the included primary studies and assess the methodological strengths and limitations for each included study yourself. For more information on applying CERQual to findings from someone else’s review, see .
Issues to consider in this step
Consider the specific study design and the research question when choosing and applying a critical appraisal tool to assess the methodological strengths and limitations of the included studies.
Remember that as part of the CERQual assessment, the methodological limitations for each review finding will be examined separately. Therefore, you will need to provide a detailed explanation for your assessment of each component of the critical appraisal tool, rather than ticking ‘yes’ or ‘no’ on a checklist.
Step 2: assess the body of data that contributes to each review finding and decide whether you have concerns about methodological limitations
Once you have carried out your critical appraisal of the included studies, you can start to assess whether you have concerns regarding any methodological limitations of the body of data supporting each review finding.
Issues to consider in this step
Some methodological strengths and weaknesses may be important for some review findings but not others. For instance, many critical appraisal tools ask you to assess whether the method of data collection was appropriate, but while methods such as focus groups may be inappropriate as a method of collecting data for some sensitive topics, they would not be considered inappropriate for other less sensitive topics. You should reflect on whether the review finding is particularly affected by any methodological limitations identified in contributing studies. You may find it helpful for the review team to meet before starting an assessment of methodological limitations to discuss and agree upon any specific issues likely to affect review findings, such as (but not limited to) privacy/sensitivity of issues, risk to participants, social desirability, the presence of observation that might affect ‘authentic’ behaviour (e.g. Hawthorne effect) and researcher effects.
Consider each study’s relative contribution to the review finding. For instance, if one study with serious methodological limitations contributes most of the data to a review finding, you may consider indicating serious concerns with methodological limitations, regardless of the methodological limitations of the other contributing studies.
Recognise that not all methodological limitations raise the same level of concerns. Consider the types of methodological limitations identified, and to what degree your concerns regarding those limitations may affect your overall confidence in the review finding.
Consider if the assessment of methodological limitations is impacted by absent information regarding how a study was conducted and if the lacking information is important to the review finding. Consider contacting the authors of the primary studies if essential reporting information is missing.
Be aware of whether the critical appraisal tool addresses issues related to one of the other three components of CERQual. For example, the critical appraisal tool may prompt you to examine the richness of the data presented (which overlaps with CERQual’s adequacy component) or whether the study findings would be applicable to the review context (which overlaps with CERQual’s relevance component). You should indicate in your assessment of methodological limitations that these issues have been included, and consider this when assessing the other components. Alternatively, you may choose to leave out questions in a critical appraisal tool that are covered by other CERQual components.
Step 3: make a judgement about the seriousness of your concerns and justify this judgement
Once you have assessed methodological limitations for each review finding, you should categorise any concerns that you have identified as either of the following:
No or very minor concerns
You should begin with the assumption that there are no concerns regarding methodological limitations for the body of data contributing to each review finding. In practice, minor concerns will not lower our confidence in the review finding, while serious concerns will lower our confidence. Moderate concerns may lead us to consider lowering our confidence in our final assessment of all four CERQual components.
Where you have concerns about methodological limitations, describe these concerns in the CERQual Evidence Profile in sufficient detail to allow users of the review findings to understand the reasons for the assessments made. The Evidence Profile presents each review finding along with the assessments for each CERQual component, the overall CERQual assessment for that finding and an explanation of this overall assessment. For more information, see the second paper in this series .
Examples of assessing methodological limitations
In Table 1, we give examples of how methodological limitations can be assessed for a selection of review findings. These examples illustrate how different, and differing degrees of, methodological weaknesses can affect the overall assessment of methodological limitations for a review finding.
Implications when methodological limitations are identified
Concerns about methodological limitations may not only have implications for our confidence in a review finding but can also point to ways of improving future research. Firstly, where serious methodological limitations have been identified, this may indicate the need for future primary researchers to use more appropriate methods or to report their methods more clearly. You should also consider updating the review once this research is available.
Concerns regarding methodological limitations may lower our confidence in review findings and are therefore part of the CERQual assessment. However, it is also important to remember that this is just one component of the CERQual approach. Having concerns about methodological limitations may not necessarily lead to a downgrading of overall confidence in a review finding, as this will be assessed alongside the other three CERQual components.
In this paper, we have described how the methodological limitations component have been used so far and have provided guidance to review authors and others on how to assess this component. However, more work is needed to determine which criteria critical appraisal tools should include and to explore how different methodological weaknesses might impact upon an overall assessment of confidence. We currently recommend that whichever tool is used, you provide a transparent description of their assessments of methodological limitations and how this impacts our confidence in a review finding. We expect the methodological limitations component, as well as the CERQual approach more generally, to develop as we gain experience and feedback from increased practical application of the approach.
Open peer review
Peer review reports for this article are available in Additional file 2.
Lewin S, Glenton C, Munthe-Kaas H, Carlsen B, Colvin CJ, Gulmezoglu M, Noyes J, Booth A, Garside R, Rashidian A. Using qualitative evidence in decision making for health and social interventions: an approach to assess confidence in findings from qualitative evidence syntheses (GRADE-CERQual). PLoS Med. 2015;12(10):e1001895.
Guyatt GH, Oxman AD, Vist G, Kunz R, Brozek J, Alonso-Coello P, Montori V, Akl EA, Djulbegovic B, Falck-Ytter Y, et al. GRADE guidelines: 4. Rating the quality of evidence—study limitations (risk of bias). J Clin Epidemiol. 2011;64(4):407–15.
Higgins JP, Altman DG, Gotzsche PC, Juni P, Moher D, Oxman AD, Savovic J, Schulz KF, Weeks L, Sterne JA, et al. The Cochrane Collaboration’s tool for assessing risk of bias in randomised trials. BMJ. 2011;343:d5928.
Colvin CJ, Garside R, Wainwright M, Lewin S, Bohren M, Glenton C, Munthe-Kaas HM, Carlsen B, Tuncalp Ö, Noyes J, et al. Applying GRADE-CERQual to qualitative evidence synthesis findings—paper 4: how to assess coherence. Implement Sci. 2018;13(Suppl 1): https://doi.org/10.1186/s13012-017-0691-8.
Glenton C, Carlsen B, Lewin S, Munthe-Kaas HM, Colvin CJ, Tuncalp Ö, Bohren M, Noyes J, Booth A, Garside R, et al. Applying GRADE-CERQual to qualitative evidence synthesis findings—paper 5: how to assess adequacy of data. Implement Sci. 2018;13(Suppl 1): https://doi.org/10.1186/s13012-017-0692-7.
Noyes J, Booth A, Lewin S, Carlsen B, Glenton C, Munthe-Kaas HM, Colvin CJ, Garside R, Bohren M, Rashidian A, et al. Applying GRADE-CERQual to qualitative evidence synthesis findings—paper 6: how to assess relevance of the data. Implement Sci. 2018;13(Suppl 1): https://doi.org/10.1186/s13012-017-0693-6.
Lewin S, Bohren M, Rashidian A, Glenton C, Munthe-Kaas HM, Carlsen B, Colvin CJ, Tuncalp Ö, Noyes J, Booth A, et al. Applying GRADE-CERQual to qualitative evidence synthesis findings—paper 2: how to make an overall CERQual assessment of confidence and create a Summary of Qualitative Findings Table. Implement Sci. 2018;13(Suppl 1): https://doi.org/10.1186/s13012-017-0689-2.
Ames HMR, Glenton C, Lewin S. Parents’ and informal caregivers’ views and experiences of communication about routine childhood vaccination: a synthesis of qualitative evidence. Cochrane Database Syst Rev. 2017;(2) Art. No.: CD011787.
Aslam RW, Hendry M, Carter B, Noyes J, Rycroft Malone J, Booth A, Pasterfield D, Charles JM, Craine N, Tudor Edwards R, et al. Interventions for preventing unintended repeat pregnancies among adolescents (protocol). Cochrane Database Syst Rev. 2015;(1) Art. No.: CD011477.
Bohren MA, Hunter EC, Munthe-Kaas HM, Souza JP, Vogel JP, Gulmezoglu AM. Facilitators and barriers to facility-based delivery in low- and middle-income countries: a qualitative evidence synthesis. Reprod Health. 2014;11(1):71.
Bohren MA, Vogel JP, Hunter EC, Lutsiv O, Makh SK, Souza JP, Aguiar C, Saraiva Coneglian F, Diniz AL, Tuncalp O, et al. The mistreatment of women during childbirth in health facilities globally: a mixed-methods systematic review. PLoS Med. 2015;12(6):e1001847. discussion e1001847
Colvin CJ, de Heer J, Winterton L, Mellenkamp M, Glenton C, Noyes J, Lewin S, Rashidian A. A systematic review of qualitative evidence on barriers and facilitators to the implementation of task-shifting in midwifery services. Midwifery. 2013;29(10):1211–21.
Glenton C, Colvin CJ, Carlsen B, Swartz A, Lewin S, Noyes J, Rashidian A. Barriers and facilitators to the implementation of lay health worker programmes to improve access to maternal and child health: qualitative evidence synthesis. Cochrane Database Syst Rev. 2013;10:CD010414.
Munabi-Babigumira S, Glenton C, Lewin S, Fretheim A, Nabudere H. Factors that influence the provision of intrapartum and postnatal care by skilled birth attendants in low- and middle-income countries: a qualitative evidence synthesis. Cochrane Database Syst Rev. 2017;(11) Art. No.: CD011558.
Munthe-Kaas HM, Hammerstrøm KT, et al. Effekt av og erfaringer med kontinuitetsfremmende tiltak i barnevernsinstitusjoner. Oslo: Norwegian Knowledge Centre for the Health Services; 2013.
O’Brien TD, Noyes J, Spencer LH, Kubis HP, Hastings RP, Edwards RT, Bray N, Whitaker R. ‘Keep fit’ exercise interventions to improve health, fitness and well-being of children and young people who use wheelchairs: mixed-method systematic review protocol. J Adv Nurs. 2014;70(12):2942–51.
Rashidian A, Shakibazadeh E, Karimi-Shahanjarini A, Glenton C, Noyes J, Lewin S, Colvin C, Laurant M. Barriers and facilitators to the implementation of doctor-nurse substitution strategies in primary care: qualitative evidence synthesis (protocol). Cochrane Database Syst Rev. 2013;2:CD010412.
Whitaker R, Hendry M, Booth A, Carter B, Charles J, Craine N, Edwards RT, Lyons M, Noyes J, Pasterfield D, et al. Intervention now to eliminate repeat unintended pregnancy in teenagers (INTERUPT): a systematic review of intervention effectiveness and cost-effectiveness, qualitative and realist synthesis of implementation factors and user engagement. BMJ Open. 2014;4(4):e004733.
Downe S, Finlayson K, Tuncalp, Metin Gulmezoglu A. What matters to women: a systematic scoping review to identify the processes and outcomes of antenatal care provision that are important to healthy pregnant women. BJOG. 2016;123(4):529–39.
Odendaal WA, Goudge J, Griffiths F, Tomlinson M, Leon N, Daniels K. Healthcare workers’ perceptions and experience on using mHealth technologies to deliver primary healthcare services: qualitative evidence synthesis (protocol). Cochrane Database Syst Rev. 2015;(11) Art. No.: CD011942.
Lewin S, Booth A, Glenton C, Munthe-Kaas HM, Rashidian A, Wainwright M, Bohren MA, Tuncalp Ö, Colvin CJ, Garside R, et al. Applying GRADE-CERQual to qualitative evidence synthesis findings: introduction to the series. Implement Sci. 2018;13(Suppl 1): https://doi.org/10.1186/s13012-017-0688-3.
Denzin N. Qualitative inquiry under fire: toward a new paradigm dialogue. USA: Left Coast Press; 2009.
Hammersley M. The issue of quality in qualitative research. Int J Res Method Educ. 2007;30(3):287–305.
Smith J. The problem of criteria for judging interpretive inquiry. Educ Eval Policy Anal. 1984;6(4):379–91.
Smith J, Deemer D. The problem of criteria in the age of relativism. In: Denzin N, Lincoln Y, editors. Handbook of Qualitative Research. London: Sage; 2000.
Saini M, Shlonsky A. Systematic synthesis of qualitative research. Oxford: Oxford University Press; 2012.
Santiago-Delefosse M, Gavin A, Bruchez C, Roux P, Stephen SL. Quality of qualitative research in the health sciences: analysis of the common criteria present in 58 assessment guidelines by expert users. Soc Sci Med. 2016;148:142–51.
Hannes K. Chapter 4: critical appraisal of qualitative research. In: Noyes J, Booth A, Hannes K, Harden A, Lewin S, Lockwood C, editors. Supplementary Guidance for Inclusion of Qualitative Research in Cochrane Systematic Reviews of Interventions Version 1 (Updated August 2011). London: Cochrane Qualitative and Implementation Methods Group. http://cqrmg.cochrane.org/supplemental-handbook-guidance; 2011.
Noyes J, Booth A, Flemming K, Garside R, Harden A, Lewin S, Pantoja T, Hannes K, Harris J, Cargo M, et al. Cochrane Qualitative and Implementation Methods Group Guidance paper 2: Methods for assessing methodological limitations, data extraction and synthesis, and confidence in synthesized qualitative findings. J Clin Epidemiol. 2017. In press. https://doi.org/10.1016/j.jclinepi.2017.06.020.
CASP. Qualitative appraisal checklist for qualitative research. Available at: http://www.casp-uk.net/#!casp-tools-checklists/c18f8: Critical Appraisal Skills Programme; 2011.
Walsh D, Downe S. Appraising the quality of qualitative research. Midwifery. 2006;22(2):108–19.
Our thanks for their feedback to those who participated in the GRADE-CERQual Project Group meetings in January 2014 or June 2015 or gave comments to the paper: Elie Akl, Heather Ames, Zhenggang Bai, Rigmor Berg, Jackie Chandler, Karen Daniels, Hans de Beer, Kenny Finlayson, Bela Ganatra, Stephen Giles, Chan Kam-wa, Manasee Mishra, Susan Munabi-Babigumira, Andy Oxman, Tomas Pantoja, Vicky Pileggi, Kent Ranson, Rebecca Rees, Holger Schünemann, Elham Shakibazadeh, Anna Selva, Birte Snilstveit, James Thomas, Hilary Thomson, Judith Thornton, Joseph D. Tucker and Joshua Vogel. Thanks also to Sarah Rosenbaum for developing the figures used in this series of papers and to members of the GRADE Working Group for their input. The guidance in this paper has been developed in collaboration and agreement with the GRADE Working Group (http://www.gradeworkinggroup.org).
This work, including the publication charge for this article, was supported by funding from the Alliance for Health Policy and Systems Research, WHO (http://www.who.int/alliance-hpsr/en/). Additional funding was provided by the Department of Reproductive Health and Research, WHO (www.who.int/reproductivehealth); Norad (Norwegian Agency for Development Cooperation: www.norad.no), the Research Council of Norway (www.forskningsradet.no) and the Cochrane Methods Innovation Fund. SL is supported by funding from the South African Medical Research Council (www.mrc.ac.za). The funders had no role in the study design, data collection and analysis, preparation of the manuscript or the decision to publish.
Availability of data and materials
Additional materials are available on the GRADE-CERQual website (www.cerqual.org)
To join the CERQual Project Group and our mailing list, please visit our website: http://www.cerqual.org/contact/. Developments in CERQual are also made available via our Twitter feed: @CERQualNet.
About this supplement
This article has been published as part of Implementation Science Volume 13 Supplement 1, 2018: Applying GRADE-CERQual to Qualitative Evidence Synthesis Findings. The full contents of the supplement are available online at https://implementationscience.biomedcentral.com/articles/supplements/volume-13-supplement-1.
Ethics approval and consent to participate
Not applicable. This study did not undertake any formal data collection involving humans or animals.
Consent for publication
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
- Qualitative research
- Qualitative evidence synthesis
- Systematic review methodology
- Research design
- Evidence-based practice
- Methodological limitations, GRADE
- Critical appraisal