Skip to main content
Download PDF

Applying GRADE-CERQual to qualitative evidence synthesis findings—paper 3: how to assess methodological limitations

Implementation Science volume 13, Article number: 9 (2018) Cite this article

Abstract

Background

The GRADE-CERQual (Confidence in Evidence from Reviews of Qualitative research) approach has been developed by the GRADE (Grading of Recommendations Assessment, Development and Evaluation) Working Group. The approach has been developed to support the use of findings from qualitative evidence syntheses in decision-making, including guideline development and policy formulation.

CERQual includes four components for assessing how much confidence to place in findings from reviews of qualitative research (also referred to as qualitative evidence syntheses): (1) methodological limitations, (2) coherence, (3) adequacy of data and (4) relevance. This paper is part of a series providing guidance on how to apply CERQual and focuses on CERQual’s methodological limitations component.

Methods

We developed the methodological limitations component by searching the literature for definitions, gathering feedback from relevant research communities and developing consensus through project group meetings. We tested the CERQual methodological limitations component within several qualitative evidence syntheses before agreeing on the current definition and principles for application.

Results

When applying CERQual, we define methodological limitations as the extent to which there are concerns about the design or conduct of the primary studies that contributed evidence to an individual review finding. In this paper, we describe the methodological limitations component and its rationale and offer guidance on how to assess methodological limitations of a review finding as part of the CERQual approach. This guidance outlines the information required to assess methodological limitations component, the steps that need to be taken to assess methodological limitations of data contributing to a review finding and examples of methodological limitation assessments.

Conclusions

This paper provides guidance for review authors and others on undertaking an assessment of methodological limitations in the context of the CERQual approach. More work is needed to determine which criteria critical appraisal tools should include when assessing methodological limitations. We currently recommend that whichever tool is used, review authors provide a transparent description of their assessments of methodological limitations in a review finding. We expect the CERQual approach and its individual components to develop further as our experiences with the practical implementation of the approach increase.

Background

The GRADE-CERQual (Confidence in Evidence from Reviews of Qualitative research) approach has been developed by the GRADE (Grading of Recommendations Assessment, Development and Evaluation) Working Group. The approach has been developed to support the use of findings from qualitative evidence syntheses in decision-making, including guideline development and policy formulation. GRADE-CERQual (hereafter referred to as CERQual) includes four components for assessing how much confidence to place in findings from reviews of qualitative research (also referred to as qualitative evidence syntheses): (1) methodological limitations, (2) coherence, (3) adequacy of data and (4) relevance. This paper focuses on one of these four components: methodological limitations.

When carrying out a CERQual assessment, we define methodological limitations as the extent to which there are concerns about the design or conduct of the primary studies that contributed evidence to an individual review finding. Where the primary studies underlying a review finding are assessed as having methodological limitations, and these are considered to have a clear/direct impact on the review finding, we may be less confident that the review finding reflects the phenomenon of interest [1].

When assessing methodological limitations, our goal is not to judge whether some absolute standard of methodological quality has been achieved, but rather to indicate concerns where any methodological limitations have been identified as serious enough to lower our confidence in the review finding. The methodological limitations component is analogous to the risk of bias domain used in the GRADE approach for findings from systematic reviews of effectiveness [2, 3].

Aim

The aim of this paper, part of a series (Fig. 1), is to describe what we mean by methodological limitations of the body of data (data from included primary studies) contributing to a review finding in the context of a qualitative evidence synthesis and to give guidance on how to operationalise this component in the context of a review finding as part of the CERQual approach. This paper should be read in conjunction with the papers describing the other three CERQual components [4,5,6] and the paper describing how to make an overall CERQual assessment of confidence and create a summary of qualitative findings table [7]. Key definitions for the series are provided in Additional file 1.

Fig. 1
figure 1

Overview of the GRADE-CERQual series of papers

Full size image

How CERQual was developed

The initial stages of the process for developing CERQual, which started in 2010, are outlined elsewhere [1]. Since then, we have further refined the current definitions of each component and the principles for application of the overall approach using a number of methods. When developing CERQual’s methodological limitations component, we undertook informal searches of the literature, including Google and Google Scholar for relevant critical appraisal tools, and for definitions and discussion papers related to the concept of methodological quality in the context of qualitative research. We carried out similar searches for the other three components. We presented an early version of the CERQual approach in 2015 to a group of methodologists, researchers and end users with experience in qualitative research, GRADE or guideline development. We further refined the approach through training workshops, seminars and presentations during which we actively sought, collated and shared feedback, by facilitating discussions of individual CERQual components within relevant organisations, through applying the approach within diverse qualitative evidence syntheses [8,9,10,11,12,13,14,15,16,17,18] and through supporting other teams in using CERQual [19, 20]. As far as possible, we used a consensus approach in these processes. We also gathered feedback from CERQual users through an online feedback form and through short individual discussions with members of the review teams. The methods used to develop CERQual are described in more detail in the first paper in this series [21].

Assessing methodological limitations

Methodological limitations in the context of findings from qualitative evidence syntheses

The methodological approaches used in a primary study may have consequences for how much we can trust the findings from that study. Where there are concerns regarding the appropriateness of these approaches (e.g. data collection or analysis methods), or how the studies were conducted, study findings may be produced that are not an adequate representation of the phenomenon of interest. For example, we may have less trust in findings from a study where participants were recruited in a manner that did not fully address the aims of the research or where the data analysis methods were not appropriate for the study design.

One or more studies contribute data to each review finding in a qualitative evidence synthesis, and these data make up the body of data for a review finding. The methodological limitations of the body of data supporting a review finding are assessed as a whole to identify whether or not any methodological weaknesses within individual studies impact our confidence in a review finding. The methodological limitations for each review finding must be assessed separately since different studies contribute varying amounts of data to each review finding, and methodological quality issues may have varying impacts on different review findings. For example, the same set of studies may contribute data to many review findings. However, individual features of study design may have implications for some of those review findings, but not necessarily other review findings. Methodological limitations of the body of data may weaken our overall assessment of confidence in the review findings to which these studies contribute. See Table 1 for the examples of review findings with concerns regarding methodological limitations.

Table 1 CERQual assessments of methodological limitations in the context of a review finding – Examples
Full size table

Critical appraisal of qualitative research

The extent to which it is possible or appropriate to critically appraise the methodological quality of qualitative research is contested among researchers in the field [22,23,24,25]. However, the starting point for the CERQual approach is that there is a need for ‘clear evaluative criteria that are responsive to the unique nature of qualitative inquiry’ ([26] p. 113).

Despite the existence of more than 50 guidelines for assessing the quality of qualitative research [27], there is no agreement on the best approach for assessing the methodological quality of primary qualitative studies. Even where essential criteria for assessing methodological quality have been agreed upon, there are challenges related to the definitions underlying these criteria, and how much importance should be given to them within a critical appraisal tool [27] (p. 151).

The methodological limitations component of the CERQual approach requires some systematic and transparent approach for identifying methodological weaknesses of individual studies. Critical appraisal tools can help us to identify such weaknesses. We are, however, dependent on the quality and completeness of the individual study reports. The Cochrane Qualitative and Implementation Methods Group recommends that review authors consider a number of issues when choosing a tool to assess methodological strengths and limitations of qualitative studies, including the type of designs or methods selected to address the review question and the included primary studies [28]. In addition, the group advises using tools that privilege assessment of methodological strengths and limitations over the quality of reporting [29]. In the context of CERQual, we take a similar approach, and currently recommend that review authors use an approach that fits their review question and synthesis methods, with which they are familiar and comfortable using, and that focuses on methodological strengths and limitations. Regardless of the approach chosen, the review authors should provide a detailed and transparent assessment for each element of the tool. In line with Cochrane Qualitative and Implementation Methods Group guidance, we do not recommend using reporting guidelines as proxies for quality appraisal tools (see Table 2). To date, review authors using the CERQual approach have primarily used the Critical Appraisal Skills Programme (CASP) checklist or an adapted version of it [30]. Research is underway to examine which elements of critical appraisal are key for assessing the quality of research in the context of qualitative evidence synthesis and for use in the CERQual approach. See Table 2 for an outline of areas where further work is needed with respect to critical appraisal tools for qualitative research.

Table 2 Areas where further work is needed – Critical appraisal tools
Full size table

Guidance on how to assess methodological limitations in the context of a review finding

The steps taken when assessing methodological limitations are shown in Fig. 2 and detailed below.

Step 1: collect and consider the necessary information related to methodological limitations

To assess methodological limitations of the body of data contributing to a review finding, you first need to choose an appropriate critical appraisal tool to assess the methodological strengths and limitations of the primary studies contributing data to the review finding. Regardless of the chosen tool, you will need to collect detailed information regarding the methods of data collection and analysis used in each study, as well as other aspects covered by the critical appraisal tool that you have chosen. The level of detail reported on the conduct of the included studies may vary greatly depending on the study design, the topic/field, type of publication or journal specifications.

Fig. 2
figure 2

Steps when assessing methodological limitations

Full size image

When applying CERQual to a review that you have conducted, you will normally have gathered this information during data extraction, as this is a standard part of the review process. When doing so, you should present and explain in detail the assessments of each criterion within the critical appraisal tool for each primary study. Some review teams choose to present this as a matrix of methodological limitations of included studies. However, if you are applying CERQual to findings from somebody else’s review, you will need to have access to their critical appraisal assessments of the included studies, which are often published as part of the review. Where these assessments are not available, you may need to go directly to the included primary studies and assess the methodological strengths and limitations for each included study yourself. For more information on applying CERQual to findings from someone else’s review, see [7].

Issues to consider in this step

Consider the specific study design and the research question when choosing and applying a critical appraisal tool to assess the methodological strengths and limitations of the included studies.

Remember that as part of the CERQual assessment, the methodological limitations for each review finding will be examined separately. Therefore, you will need to provide a detailed explanation for your assessment of each component of the critical appraisal tool, rather than ticking ‘yes’ or ‘no’ on a checklist.

Step 2: assess the body of data that contributes to each review finding and decide whether you have concerns about methodological limitations

Once you have carried out your critical appraisal of the included studies, you can start to assess whether you have concerns regarding any methodological limitations of the body of data supporting each review finding.

Issues to consider in this step

Some methodological strengths and weaknesses may be important for some review findings but not others. For instance, many critical appraisal tools ask you to assess whether the method of data collection was appropriate, but while methods such as focus groups may be inappropriate as a method of collecting data for some sensitive topics, they would not be considered inappropriate for other less sensitive topics. You should reflect on whether the review finding is particularly affected by any methodological limitations identified in contributing studies. You may find it helpful for the review team to meet before starting an assessment of methodological limitations to discuss and agree upon any specific issues likely to affect review findings, such as (but not limited to) privacy/sensitivity of issues, risk to participants, social desirability, the presence of observation that might affect ‘authentic’ behaviour (e.g. Hawthorne effect) and researcher effects.

Consider each study’s relative contribution to the review finding. For instance, if one study with serious methodological limitations contributes most of the data to a review finding, you may consider indicating serious concerns with methodological limitations, regardless of the methodological limitations of the other contributing studies.

Recognise that not all methodological limitations raise the same level of concerns. Consider the types of methodological limitations identified, and to what degree your concerns regarding those limitations may affect your overall confidence in the review finding.

Consider if the assessment of methodological limitations is impacted by absent information regarding how a study was conducted and if the lacking information is important to the review finding. Consider contacting the authors of the primary studies if essential reporting information is missing.

Be aware of whether the critical appraisal tool addresses issues related to one of the other three components of CERQual. For example, the critical appraisal tool may prompt you to examine the richness of the data presented (which overlaps with CERQual’s adequacy component) or whether the study findings would be applicable to the review context (which overlaps with CERQual’s relevance component). You should indicate in your assessment of methodological limitations that these issues have been included, and consider this when assessing the other components. Alternatively, you may choose to leave out questions in a critical appraisal tool that are covered by other CERQual components.

Step 3: make a judgement about the seriousness of your concerns and justify this judgement

Once you have assessed methodological limitations for each review finding, you should categorise any concerns that you have identified as either of the following:

  • No or very minor concerns

  • Minor concerns

  • Moderate concerns

  • Serious concerns

You should begin with the assumption that there are no concerns regarding methodological limitations for the body of data contributing to each review finding. In practice, minor concerns will not lower our confidence in the review finding, while serious concerns will lower our confidence. Moderate concerns may lead us to consider lowering our confidence in our final assessment of all four CERQual components.

Where you have concerns about methodological limitations, describe these concerns in the CERQual Evidence Profile in sufficient detail to allow users of the review findings to understand the reasons for the assessments made. The Evidence Profile presents each review finding along with the assessments for each CERQual component, the overall CERQual assessment for that finding and an explanation of this overall assessment. For more information, see the second paper in this series [7].

Examples of assessing methodological limitations

In Table 1, we give examples of how methodological limitations can be assessed for a selection of review findings. These examples illustrate how different, and differing degrees of, methodological weaknesses can affect the overall assessment of methodological limitations for a review finding.

Implications when methodological limitations are identified

Concerns about methodological limitations may not only have implications for our confidence in a review finding but can also point to ways of improving future research. Firstly, where serious methodological limitations have been identified, this may indicate the need for future primary researchers to use more appropriate methods or to report their methods more clearly. You should also consider updating the review once this research is available.

Conclusions

Concerns regarding methodological limitations may lower our confidence in review findings and are therefore part of the CERQual assessment. However, it is also important to remember that this is just one component of the CERQual approach. Having concerns about methodological limitations may not necessarily lead to a downgrading of overall confidence in a review finding, as this will be assessed alongside the other three CERQual components.

In this paper, we have described how the methodological limitations component have been used so far and have provided guidance to review authors and others on how to assess this component. However, more work is needed to determine which criteria critical appraisal tools should include and to explore how different methodological weaknesses might impact upon an overall assessment of confidence. We currently recommend that whichever tool is used, you provide a transparent description of their assessments of methodological limitations and how this impacts our confidence in a review finding. We expect the methodological limitations component, as well as the CERQual approach more generally, to develop as we gain experience and feedback from increased practical application of the approach.

Open peer review

Peer review reports for this article are available in Additional file 2.

References

  1. Lewin S, Glenton C, Munthe-Kaas H, Carlsen B, Colvin CJ, Gulmezoglu M, Noyes J, Booth A, Garside R, Rashidian A. Using qualitative evidence in decision making for health and social interventions: an approach to assess confidence in findings from qualitative evidence syntheses (GRADE-CERQual). PLoS Med. 2015;12(10):e1001895.

    Article  PubMed  PubMed Central  Google Scholar 

  2. Guyatt GH, Oxman AD, Vist G, Kunz R, Brozek J, Alonso-Coello P, Montori V, Akl EA, Djulbegovic B, Falck-Ytter Y, et al. GRADE guidelines: 4. Rating the quality of evidence—study limitations (risk of bias). J Clin Epidemiol. 2011;64(4):407–15.

    Article  PubMed  Google Scholar 

  3. Higgins JP, Altman DG, Gotzsche PC, Juni P, Moher D, Oxman AD, Savovic J, Schulz KF, Weeks L, Sterne JA, et al. The Cochrane Collaboration’s tool for assessing risk of bias in randomised trials. BMJ. 2011;343:d5928.

    Article  PubMed  PubMed Central  Google Scholar 

  4. Colvin CJ, Garside R, Wainwright M, Lewin S, Bohren M, Glenton C, Munthe-Kaas HM, Carlsen B, Tuncalp Ö, Noyes J, et al. Applying GRADE-CERQual to qualitative evidence synthesis findings—paper 4: how to assess coherence. Implement Sci. 2018;13(Suppl 1): https://doi.org/10.1186/s13012-017-0691-8.

  5. Glenton C, Carlsen B, Lewin S, Munthe-Kaas HM, Colvin CJ, Tuncalp Ö, Bohren M, Noyes J, Booth A, Garside R, et al. Applying GRADE-CERQual to qualitative evidence synthesis findings—paper 5: how to assess adequacy of data. Implement Sci. 2018;13(Suppl 1): https://doi.org/10.1186/s13012-017-0692-7.

  6. Noyes J, Booth A, Lewin S, Carlsen B, Glenton C, Munthe-Kaas HM, Colvin CJ, Garside R, Bohren M, Rashidian A, et al. Applying GRADE-CERQual to qualitative evidence synthesis findings—paper 6: how to assess relevance of the data. Implement Sci. 2018;13(Suppl 1): https://doi.org/10.1186/s13012-017-0693-6.

  7. Lewin S, Bohren M, Rashidian A, Glenton C, Munthe-Kaas HM, Carlsen B, Colvin CJ, Tuncalp Ö, Noyes J, Booth A, et al. Applying GRADE-CERQual to qualitative evidence synthesis findings—paper 2: how to make an overall CERQual assessment of confidence and create a Summary of Qualitative Findings Table. Implement Sci. 2018;13(Suppl 1): https://doi.org/10.1186/s13012-017-0689-2.

  8. Ames HMR, Glenton C, Lewin S. Parents’ and informal caregivers’ views and experiences of communication about routine childhood vaccination: a synthesis of qualitative evidence. Cochrane Database Syst Rev. 2017;(2) Art. No.: CD011787.

  9. Aslam RW, Hendry M, Carter B, Noyes J, Rycroft Malone J, Booth A, Pasterfield D, Charles JM, Craine N, Tudor Edwards R, et al. Interventions for preventing unintended repeat pregnancies among adolescents (protocol). Cochrane Database Syst Rev. 2015;(1) Art. No.: CD011477.

  10. Bohren MA, Hunter EC, Munthe-Kaas HM, Souza JP, Vogel JP, Gulmezoglu AM. Facilitators and barriers to facility-based delivery in low- and middle-income countries: a qualitative evidence synthesis. Reprod Health. 2014;11(1):71.

    Article  PubMed  PubMed Central  Google Scholar 

  11. Bohren MA, Vogel JP, Hunter EC, Lutsiv O, Makh SK, Souza JP, Aguiar C, Saraiva Coneglian F, Diniz AL, Tuncalp O, et al. The mistreatment of women during childbirth in health facilities globally: a mixed-methods systematic review. PLoS Med. 2015;12(6):e1001847. discussion e1001847

    Article  PubMed  PubMed Central  Google Scholar 

  12. Colvin CJ, de Heer J, Winterton L, Mellenkamp M, Glenton C, Noyes J, Lewin S, Rashidian A. A systematic review of qualitative evidence on barriers and facilitators to the implementation of task-shifting in midwifery services. Midwifery. 2013;29(10):1211–21.

    Article  PubMed  Google Scholar 

  13. Glenton C, Colvin CJ, Carlsen B, Swartz A, Lewin S, Noyes J, Rashidian A. Barriers and facilitators to the implementation of lay health worker programmes to improve access to maternal and child health: qualitative evidence synthesis. Cochrane Database Syst Rev. 2013;10:CD010414.

    Google Scholar 

  14. Munabi-Babigumira S, Glenton C, Lewin S, Fretheim A, Nabudere H. Factors that influence the provision of intrapartum and postnatal care by skilled birth attendants in low- and middle-income countries: a qualitative evidence synthesis. Cochrane Database Syst Rev. 2017;(11) Art. No.: CD011558.

  15. Munthe-Kaas HM, Hammerstrøm KT, et al. Effekt av og erfaringer med kontinuitetsfremmende tiltak i barnevernsinstitusjoner. Oslo: Norwegian Knowledge Centre for the Health Services; 2013.

    Google Scholar 

  16. O’Brien TD, Noyes J, Spencer LH, Kubis HP, Hastings RP, Edwards RT, Bray N, Whitaker R. ‘Keep fit’ exercise interventions to improve health, fitness and well-being of children and young people who use wheelchairs: mixed-method systematic review protocol. J Adv Nurs. 2014;70(12):2942–51.

    Article  PubMed  Google Scholar 

  17. Rashidian A, Shakibazadeh E, Karimi-Shahanjarini A, Glenton C, Noyes J, Lewin S, Colvin C, Laurant M. Barriers and facilitators to the implementation of doctor-nurse substitution strategies in primary care: qualitative evidence synthesis (protocol). Cochrane Database Syst Rev. 2013;2:CD010412.

    Google Scholar 

  18. Whitaker R, Hendry M, Booth A, Carter B, Charles J, Craine N, Edwards RT, Lyons M, Noyes J, Pasterfield D, et al. Intervention now to eliminate repeat unintended pregnancy in teenagers (INTERUPT): a systematic review of intervention effectiveness and cost-effectiveness, qualitative and realist synthesis of implementation factors and user engagement. BMJ Open. 2014;4(4):e004733.

    Article  PubMed  PubMed Central  Google Scholar 

  19. Downe S, Finlayson K, Tuncalp, Metin Gulmezoglu A. What matters to women: a systematic scoping review to identify the processes and outcomes of antenatal care provision that are important to healthy pregnant women. BJOG. 2016;123(4):529–39.

    Article  CAS  PubMed  Google Scholar 

  20. Odendaal WA, Goudge J, Griffiths F, Tomlinson M, Leon N, Daniels K. Healthcare workers’ perceptions and experience on using mHealth technologies to deliver primary healthcare services: qualitative evidence synthesis (protocol). Cochrane Database Syst Rev. 2015;(11) Art. No.: CD011942.

  21. Lewin S, Booth A, Glenton C, Munthe-Kaas HM, Rashidian A, Wainwright M, Bohren MA, Tuncalp Ö, Colvin CJ, Garside R, et al. Applying GRADE-CERQual to qualitative evidence synthesis findings: introduction to the series. Implement Sci. 2018;13(Suppl 1): https://doi.org/10.1186/s13012-017-0688-3.

  22. Denzin N. Qualitative inquiry under fire: toward a new paradigm dialogue. USA: Left Coast Press; 2009.

    Google Scholar 

  23. Hammersley M. The issue of quality in qualitative research. Int J Res Method Educ. 2007;30(3):287–305.

    Article  Google Scholar 

  24. Smith J. The problem of criteria for judging interpretive inquiry. Educ Eval Policy Anal. 1984;6(4):379–91.

    Article  Google Scholar 

  25. Smith J, Deemer D. The problem of criteria in the age of relativism. In: Denzin N, Lincoln Y, editors. Handbook of Qualitative Research. London: Sage; 2000.

    Google Scholar 

  26. Saini M, Shlonsky A. Systematic synthesis of qualitative research. Oxford: Oxford University Press; 2012.

    Book  Google Scholar 

  27. Santiago-Delefosse M, Gavin A, Bruchez C, Roux P, Stephen SL. Quality of qualitative research in the health sciences: analysis of the common criteria present in 58 assessment guidelines by expert users. Soc Sci Med. 2016;148:142–51.

    Article  CAS  PubMed  Google Scholar 

  28. Hannes K. Chapter 4: critical appraisal of qualitative research. In: Noyes J, Booth A, Hannes K, Harden A, Lewin S, Lockwood C, editors. Supplementary Guidance for Inclusion of Qualitative Research in Cochrane Systematic Reviews of Interventions Version 1 (Updated August 2011). London: Cochrane Qualitative and Implementation Methods Group. http://cqrmg.cochrane.org/supplemental-handbook-guidance; 2011.

    Google Scholar 

  29. Noyes J, Booth A, Flemming K, Garside R, Harden A, Lewin S, Pantoja T, Hannes K, Harris J, Cargo M, et al. Cochrane Qualitative and Implementation Methods Group Guidance paper 2: Methods for assessing methodological limitations, data extraction and synthesis, and confidence in synthesized qualitative findings. J Clin Epidemiol. 2017. In press. https://doi.org/10.1016/j.jclinepi.2017.06.020.

  30. CASP. Qualitative appraisal checklist for qualitative research. Available at: http://www.casp-uk.net/#!casp-tools-checklists/c18f8: Critical Appraisal Skills Programme; 2011.

  31. Walsh D, Downe S. Appraising the quality of qualitative research. Midwifery. 2006;22(2):108–19.

    Article  PubMed  Google Scholar 

Download references

Acknowledgements

Our thanks for their feedback to those who participated in the GRADE-CERQual Project Group meetings in January 2014 or June 2015 or gave comments to the paper: Elie Akl, Heather Ames, Zhenggang Bai, Rigmor Berg, Jackie Chandler, Karen Daniels, Hans de Beer, Kenny Finlayson, Bela Ganatra, Stephen Giles, Chan Kam-wa, Manasee Mishra, Susan Munabi-Babigumira, Andy Oxman, Tomas Pantoja, Vicky Pileggi, Kent Ranson, Rebecca Rees, Holger Schünemann, Elham Shakibazadeh, Anna Selva, Birte Snilstveit, James Thomas, Hilary Thomson, Judith Thornton, Joseph D. Tucker and Joshua Vogel. Thanks also to Sarah Rosenbaum for developing the figures used in this series of papers and to members of the GRADE Working Group for their input. The guidance in this paper has been developed in collaboration and agreement with the GRADE Working Group (http://www.gradeworkinggroup.org).

Funding

This work, including the publication charge for this article, was supported by funding from the Alliance for Health Policy and Systems Research, WHO (http://www.who.int/alliance-hpsr/en/). Additional funding was provided by the Department of Reproductive Health and Research, WHO (www.who.int/reproductivehealth); Norad (Norwegian Agency for Development Cooperation: www.norad.no), the Research Council of Norway (www.forskningsradet.no) and the Cochrane Methods Innovation Fund. SL is supported by funding from the South African Medical Research Council (www.mrc.ac.za). The funders had no role in the study design, data collection and analysis, preparation of the manuscript or the decision to publish.

Availability of data and materials

Additional materials are available on the GRADE-CERQual website (www.cerqual.org)

To join the CERQual Project Group and our mailing list, please visit our website: http://www.cerqual.org/contact/. Developments in CERQual are also made available via our Twitter feed: @CERQualNet.

About this supplement

This article has been published as part of Implementation Science Volume 13 Supplement 1, 2018: Applying GRADE-CERQual to Qualitative Evidence Synthesis Findings. The full contents of the supplement are available online at https://implementationscience.biomedcentral.com/articles/supplements/volume-13-supplement-1.

Author information

Authors and Affiliations

  1. Norwegian Institute of Public Health, Oslo, Norway

    Heather Munthe-Kaas, Claire Glenton, Simon Lewin & Signe Flottorp

  2. UNDP/UNFPA/ UNICEF/WHO/World Bank Special Programme of Research, Development and Research Training in Human Reproduction, Department of Reproductive Health and Research, World Health Organization, Geneva, Switzerland

    Meghan A. Bohren & Özge Tunçalp

  3. Health Systems Research Unit, South African Medical Research Council, Cape Town, South Africa

    Simon Lewin

  4. School of Social Sciences, Bangor University, Bangor, UK

    Jane Noyes

  5. School of Health and Related Research (ScHARR), University of Sheffield, Sheffield, UK

    Andrew Booth

  6. European Centre for Environment and Human Health, University of Exeter Medical School, Exeter, UK

    Ruth Garside

  7. Division of Social and Behavioural Sciences, School of Public Health and Family Medicine, University of Cape Town, Cape Town, South Africa

    Christopher J. Colvin & Megan Wainwright

  8. Department of Health Management and Economics, School of Public Health, Tehran University of Medical Sciences, Tehran, Iran

    Arash Rashidian

  9. Information, Evidence and Research Department, Eastern Mediterranean Regional Office, World Health Organization, Cairo, Egypt

    Arash Rashidian

  10. Uni Research Rokkan Centre, Bergen, Norway

    Benedicte Carlsen

Authors
  1. Heather Munthe-Kaas
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Meghan A. Bohren
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Claire Glenton
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Simon Lewin
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Jane Noyes
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Özge Tunçalp
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Andrew Booth
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Ruth Garside
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Christopher J. Colvin
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Megan Wainwright
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Arash Rashidian
    View author publications

    You can also search for this author in PubMed Google Scholar

  12. Signe Flottorp
    View author publications

    You can also search for this author in PubMed Google Scholar

  13. Benedicte Carlsen
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

All authors participated in the conceptual design of the CERQual approach. HMK and MB wrote the first draft of the manuscript. All authors contributed to the writing of the manuscript. All authors have read and approved the manuscript.

Corresponding author

Correspondence to Heather Munthe-Kaas.

Ethics declarations

Ethics approval and consent to participate

Not applicable. This study did not undertake any formal data collection involving humans or animals.

Consent for publication

Not applicable

Competing interests

The authors declare that they have no competing interests.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Additional files

Additional file 1:

Key definitions relevant to CERQual. (PDF 619 kb)

Additional file 2:

Open peer review reports. (PDF 99 kb)

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Munthe-Kaas, H., Bohren, M.A., Glenton, C. et al. Applying GRADE-CERQual to qualitative evidence synthesis findings—paper 3: how to assess methodological limitations. Implementation Sci 13 (Suppl 1), 9 (2018). https://doi.org/10.1186/s13012-017-0690-9

Download citation

  • Published:

  • DOI: https://doi.org/10.1186/s13012-017-0690-9

Keywords

Implementation Science

ISSN: 1748-5908