Skip to main content

Applying GRADE-CERQual to qualitative evidence synthesis findings–paper 6: how to assess relevance of the data

Abstract

Background

The GRADE-CERQual (Confidence in Evidence from Reviews of Qualitative research) approach has been developed by the GRADE (Grading of Recommendations Assessment, Development and Evaluation) Working Group. The approach has been developed to support the use of findings from qualitative evidence syntheses in decision-making, including guideline development and policy formulation.

CERQual includes four components for assessing how much confidence to place in findings from reviews of qualitative research (also referred to as qualitative evidence syntheses): (1) methodological limitations, (2) coherence, (3) adequacy of data and (4) relevance. This paper is part of a series providing guidance on how to apply CERQual and focuses on CERQual’s relevance component.

Methods

We developed the relevance component by searching the literature for definitions, gathering feedback from relevant research communities and developing consensus through project group meetings. We tested the CERQual relevance component within several qualitative evidence syntheses before agreeing on the current definition and principles for application.

Results

When applying CERQual, we define relevance as the extent to which the body of data from the primary studies supporting a review finding is applicable to the context (perspective or population, phenomenon of interest, setting) specified in the review question. In this paper, we describe the relevance component and its rationale and offer guidance on how to assess relevance in the context of a review finding. This guidance outlines the information required to assess relevance, the steps that need to be taken to assess relevance and examples of relevance assessments.

Conclusions

This paper provides guidance for review authors and others on undertaking an assessment of relevance in the context of the CERQual approach. Assessing the relevance component requires consideration of potentially important contextual factors at an early stage in the review process. We expect the CERQual approach, and its individual components, to develop further as our experiences with the practical implementation of the approach increase.

Background

The GRADE-CERQual (Confidence in Evidence from Reviews of Qualitative research) approach has been developed by the GRADE (Grading of Recommendations Assessment, Development and Evaluation) Working Group. The approach has been developed to support the use of findings from qualitative evidence syntheses in decision-making, including guideline development and policy formulation. GRADE-CERQual (hereafter referred to as CERQual) includes four components for assessing how much confidence to place in findings from reviews of qualitative research (also referred to as qualitative evidence syntheses): (1) methodological limitations, (2) coherence, (3) adequacy of data and (4) relevance. This paper focuses on one of these four components, relevance.

When carrying out a CERQual assessment, we define relevance as the extent to which the body of data from the primary studies supporting a review finding is applicable to the context (including for example the perspective or population, phenomenon of interest, setting) specified in the review question [1]. The relevance component in CERQual is analogous to the indirectness domain used in the GRADE approach for assessing confidence in findings from systematic reviews of effectiveness [2].

Aim

The aim of this paper, part of a series (Fig. 1), is to describe what we mean by relevance of data in the context of a qualitative evidence synthesis and to give guidance on how to operationalise this component in the context of a review finding as part of the CERQual approach. This paper should be read in conjunction with the papers describing the other three CERQual components [3,4,5] and the paper describing how to make an overall CERQual assessment of confidence and create a Summary of Qualitative Findings table [6]. Key definitions for the series are provided in Additional file 1.

Fig. 1
figure 1

Overview of the GRADE-CERQual series of papers

How CERQual was developed

The initial stages of the process for developing CERQual, which started in 2010, are outlined elsewhere [1]. Since then, we have further refined the current definitions of each component and the principles for application of the overall approach using a number of methods. When developing CERQual’s relevance component, we undertook informal searches of the literature, including Google and Google Scholar, for definitions and discussion papers related to the concept of relevance and to related concepts such as internal and external validity. We carried out similar searches for the other three components. We presented an early version of the CERQual approach in 2015 to a group of methodologists, researchers and end users with experience in qualitative research, GRADE or guideline development. We further refined the approach through training workshops, seminars and presentations during which we actively sought, collated and shared feedback; by facilitating discussions of individual CERQual components within relevant organisations; through applying the approach within diverse qualitative evidence syntheses [7,8,9,10,11,12,13,14,15,16,17]; and through supporting other teams in using CERQual [18, 19]. As far as possible, we used a consensus approach in these processes. We also gathered feedback from CERQual users through an online feedback form and through short individual discussions. The methods used to develop CERQual are described in more detail in the first paper in this series [20].

Assessing relevance

As described above, relevance in the context of CERQual is defined as the extent to which the body of data from the primary studies supporting a review finding is applicable to the context specified in the review question [1]. Relevance is the CERQual component that is anchored to the context specified in the review question. How the review question and objectives are expressed, how a priori subgroup analyses are specified and how theoretical considerations inform the review design are therefore critical to making an assessment of relevance when applying CERQual.

This paper primarily targets review authors who apply CERQual whilst undertaking their own review. We recognise additional circumstances when it is desirable to apply CERQual retrospectively to a review conducted by others. We envisage two scenarios when applying CERQual retrospectively (i) using the original review question or (ii) adapting the context specified in the original review question to reflect a particular context of interest (for example, an a posteriori identified ‘subgroup’ population or intervention type). These circumstances will feature in subsequent guidance, following additional methodological development and testing. For the remainder of this paper, we focus primarily on review authors who apply CERQual when undertaking their own review. (See also Additional file 2 in paper 2 in this series [6] for more information about applying CERQual to a synthesis conducted by others.)

Description of and rationale for the relevance component

Evidence-informed health policymaking requires “relevant research” [21] that relates to the review question, which may be defined from initiation of the review or may emerge iteratively during the review process, and to the type of question that is being asked. The methodological literature similarly refers to the degree of fit of included studies to the structured review question as one type of internal validity [22]. Internal validity, a term commonly used by systematic review authors, therefore maps onto the ‘internal relevance’ of included studies to the review question in a qualitative evidence synthesis.

In most qualitative evidence syntheses, the inclusion criteria for studies closely mirror the review question. Included studies must, by definition, be directly relevant to the review question. In other cases, the body of data from the primary studies may not be completely relevant because of differences between these data and the context of the review question, for instance differences in the focus of the study question, perspective or population, the phenomenon of interest or intervention, the setting or the timeframe. In some cases, the body of data from the included primary studies only partially address the context specified by the review question. This occurs if the body of data from the included studies covers only a subgroup of the population or a subtype of the intervention. In other cases, relevant studies are missing entirely. Under these circumstances, review authors sometimes extend the scope of eligible studies to include studies linked to the review question conceptually or by analogy.

CERQual focuses on the assessment of the internal relevance of the body of evidence from included studies contributing to a review finding, as mapped against the context of the review question. This assessment of relevance is not intended to make externally referent claims regarding the transferability, generalisability or applicability (terms that we take to mean the same) of a review finding. Wider external relevance (a concept that maps onto external validity––a term also commonly used by review authors conducting quantitative systematic reviews of the effectiveness of interventions) of a review finding is addressed in part by the overall CERQual assessment. This overall assessment, based on judgements for all four CERQual components, seeks to establish the extent to which a synthesis finding is a reasonable representation of the phenomenon of interest [1, 6]. An overall CERQual assessment communicates the extent to which the synthesis finding is likely to be substantially different from the phenomenon of interest, as defined in the review question. For completeness, an explanation of external relevance can be found in Additional file 2.

When assessing relevance in the context of CERQual, our aim is not to judge whether some absolute standard of relevance has been achieved, but to judge whether there are grounds for concern regarding relevance. Having identified any such concerns, we must consider whether these are sufficiently serious to lower our confidence in the review finding. We are likely to have concerns about relevance when the context of the included studies does not adequately reflect the context determined by the review question.

Guidance on how to assess relevance of data in the context of a review finding (Fig. 2)

Step 1: clarify the review question and context

When designing any review and developing the protocol, you should consider and decide which contextual factors are important for the review question [23]. By ‘context’, we refer to a complex and interacting composite that includes, but is not restricted to, the perspective, the population, the phenomenon of interest and the setting (Table 1). Review questions can be articulated using frameworks such as SPICE (Setting Perspective, Phenomenon of Interest, Evaluation) or adapted versions of PICO (Population, Phenomenon of Interest, Comparison, Outcomes) [24].

Fig. 2
figure 2

Steps when assessing methodological limitations

Table 1 Contextual factors to consider when refining the review scope and specifying the question

A priori identification of factors considered to be relevant to an individual review may also be facilitated, before commencement of a review, by reference to an available context framework or theory. Alternatively, a decision on likely factors may be informed by an existing review in a similar topic area. Incorporating insights from external frameworks, theories or comparable reviews at the design stage may help with a later assessment of relevance. For example, a specific theoretical perspective (such as a behavioural theory [25]) or lens (such as an equity lens [26]) may be adopted to help define and refine the review question and important contextual factors. Identification, selection and application of an appropriate theory are critical to this process, and Cochrane Guidance is available for review authors to aid in selecting social theories in reviews [27].

Depending on the methodology selected, you may specify the review question(s) a priori in the protocol or may develop question(s) iteratively as part of the review process, with the final question(s) being documented later. Information about the context of the review question may appear in several sections of the review protocol, including the structured review question, inclusion/exclusion criteria and any parameters set for the search and retrieval of studies [28]. The review’s inclusion and exclusion criteria are thus an important additional source of information when assessing relevance [28]. Typically, these criteria include the geographical and temporal setting and the characteristics of the population, such as gender, ethnicity, religion and other demographic and cultural factors––the micro-context for the review (Table 1).

If the review explores how an intervention is implemented, then the review’s inclusion and exclusion criteria are also likely to include the characteristics of the intervention and of those administering the intervention, and possibly also the organisational characteristics within which the intervention is delivered (the meso-context). For other review questions, particularly those related to policy, the political and legislative background context (the macro-context) are equally, if not more, important.

You may also augment the core review question, for instance by pre-specifying specific subgroups of the population or setting (for instance, young men or rural health facilities) that you will consider separately in the analysis. Pre-specification of subgroup analyses offers review authors the opportunity to apply CERQual to subgroup level findings. However, we recognise that review authors may only uncover meaningful groupings as they conduct the review itself.

If you are applying CERQual to someone else’s review and the original question and context remain unchanged, then step 1 would apply in the same way. At present, we have no empirical base for guidance on applying CERQual to someone else’s review when aspects of the original context of the review question have been changed to reflect a new context. We have yet to undertake the methodological development and testing for this specific application of CERQual.

Step 2: decide on the appropriateness and implications of the study inclusion strategy

As a review author, you need to make an informed judgement on how and why you selected primary studies in relation to your review question. In a qualitative evidence synthesis, you can choose different strategies when identifying and selecting studies to synthesise [24]. For some syntheses, you may include all eligible primary studies, for example, when the qualitative synthesis is coterminous with an effectiveness review. For exploratory or interpretive syntheses, only a sample of the eligible studies may be included. When subsequently assessing relevance, you should reflect on how the sample was located and on the underpinning principles that determined its selection. For example, was the sample selected to explain the attitudes or behaviours of a particular group or to demonstrate the applicability of a theory across groups? In making decisions on how to select the sample of included studies, you may trade-off such factors as the methodological strengths and limitations of available studies (see paper 3 in this series [5]), the relevance of the evidence to the review question and, for instance, the geographical coverage of studies in relation to the review question. For example, if the question is global, review authors may purposively select studies from diverse settings. Conducting an initial knowledge map of potential studies to identify important contextual factors [29], together with other important elements, may help to design a sampling frame to inform decisions about studies for potential inclusion [8].

Pragmatically, the sampling strategy seeks to optimise the trade-off between the number of available studies that meets the inclusion criteria versus the time available to synthesise studies. A key consideration relates to whether or not a review is required to inform an imminent decision. Where this is the case, the decision maker needs to draw upon the most relevant available evidence and studies may be sampled on that basis. A further consideration is where dissemination bias might limit the identification of relevant evidence to address the question for a specific context. Recent work outlines the conceptual basis for the effects of dissemination bias in qualitative research and its likely extent [30, 31], while the final paper in this series [32] addresses how dissemination bias in qualitative research might impact upon a CERQual assessment.

Step 3: gather information about relevance in the included studies

Next, you need to gather information from included primary studies to help you to identify similarities between the context of the studies supporting each review finding and the context specified in the review question. Information about context in the primary studies that maps against the context of interest as specified in the review protocol may be reported throughout the primary study. This step is therefore a ‘search, find and subsequently extract’ exercise. As far as possible, you should ensure that this information is gathered as part of the screening or subsequent data extraction process.

Step 4: assess the body of data that contributes to each review finding and decide whether you have concerns about relevance

An assessment of relevance is best facilitated by access to subject-specific knowledge either amongst the review authors or from an expert advisory group. To assess the relevance of the data, you need to identify similarities and differences between the context of the studies supporting each review finding and the context specified in the review question. You should assess relevance for each review finding individually and not for the review as a whole. Table 2 provides a non-exhaustive list of potentially important contextual factors when assessing relevance.

Table 2 Identifying similarities and differences between the context specified in the review question and the context specified in the primary studies contributing to a review finding

We have identified three types of threat to relevance—data that are indirectly relevant to the review question, data that are only partially relevant and data that are of unclear relevance:

  1. 1.

    (Some of) the underlying data are of indirect relevance: this assessment is made where a review team is unable to identify studies that fully represent the context of the review, but is able to identify studies that correspond with some factors from the context of the review question but not with others. In other words, one or more aspects of context are substituted with another in these studies. For example, if a synthesis is looking at factors affecting the implementation of public health campaigns for bird flu but finds no, or very little, evidence of direct relevance to bird flu, they may choose to include indirect evidence from studies of swine flu campaigns to address the original synthesis question. This use of indirect evidence is based on the assumption that there are likely to be sufficient common factors between the implementation of the respective public health campaigns. In contrast, other contextual factors such as animal versus bird welfare and containment in a flu epidemic are likely to be very different. Review authors therefore need to be cautious when interpreting indirect evidence where some contextual factors are similar and others are not in order to ensure that the use of indirect evidence is not misleading.

  2. 2.

    (Some of) the underlying data are of partial relevance: this is assigned where a part of the context of the larger review question (e.g. a population subgroup) is addressed directly by a review finding but where evidence is lacking for the complete context specified in the review question. For example, a review question seeks to determine Muslim women’s experiences of divorce and child custody. A finding that only includes evidence from Jordanian Muslem women may be assessed as “partially relevant” for Muslim women in general. In some circumstances, it may be more appropriate and meaningful to rephrase a finding to incorporate the aspects of context that indicate that the finding is for a particular ‘subgroup’ in relation to context. Using the same example, the evidence would be reassessed as “directly relevant” for a finding relating to Jordanian Muslim women specifically when this group was pre-specified as a subgroup of interest. Decisions on the level of granularity used when framing the findings depends upon the purpose and intended audience for your specific review.

  3. 3.

    It is unclear whether the underlying data is relevant (unclear relevance): this assessment should be reserved for situations where you have identified, typically a priori, important factors that influence the interpretation of review findings but where you are unable to identify those factors from included studies. For example, you have reason to believe that different age groups respond differently to health appointment reminders sent via mobile phone. However, the contributing qualitative studies do not describe the ages of the informants who contributed to specific findings, only supplying a general description of the demographics for all study participants.

Review authors should consider whether any characteristics related to each contextual factor previously identified as important, and reported in primary studies contributing to a finding, are directly, partially or indirectly relevant or of unknown relevance to the context specified in the review question. You may find it helpful to assign each study contributing to a review finding a relevance rating (directly relevant; indirectly relevant; partially relevant; of unclear relevance, with a short explanation). Tables 3, 4, 5 and 6 provide visual examples.

Table 3 CERQual assessments of relevance in the context of a review finding––examples of no or very minor concerns
Table 4 CERQual assessments of relevance in the context of a review finding––examples of minor concerns
Table 5 CERQual assessments of relevance in the context of a review finding––examples of moderate concerns
Table 6 CERQual assessments of relevance in the context of a review finding––examples of serious concerns

You might have fewer concerns about relevance where the contexts of the studies contributing data to a finding match the context of the review question or a pre-identified subgroup. You are not, however, seeking a perfect fit between the included studies and the context of the review question. As with the other CERQual components, the emphasis is on characteristics that trigger significant concerns (see paper 2 in this series on making an overall CERQual assessment [6]). In many cases, you will assess the relevance of review findings that are supported by studies that are not derived from the exact setting for which the decision is intended. For example, even if one or more individual studies derive from the same country as the decision-making context, you cannot consider these studies to be fully representative of that country. Where multiple studies exist, derived directly from the decision-making context, these studies may represent different time periods within different political and economic settings. In general, we would have no concerns regarding relevance for a review finding where we judge there to be no important differences between the context of the data contributing to the review finding and the context specified in the review question.

Our confidence in a review finding may, however, be reduced where the relation between the contexts of the primary studies and that specified in the review question is not apparent. For example, there may be differences in the perspective or population, the phenomenon of interest or intervention, the context or the timeframe. In addition, in a review where studies with a direct relation are not found, you may include studies that imperfectly represent aspects of the context. For example, you may use studies conducted with immediately pre- or post-adolescents to inform a review question relating to adolescents if you have been unable to find any studies of the adolescents themselves.

The relevance of a review finding is not related to the number of primary studies contributing data to that finding. For instance, where a finding is based on data from a single study that matches the contextual factors identified in the review question, you would not have concerns about relevance. In contrast, you may have concerns about relevance for a review finding derived from multiple studies if the contexts of those studies do not match the context of the review question. How to assess adequacy of data as part of the CERQual approach is addressed elsewhere in this series [4].

Step 5: make a judgement about the seriousness of your concerns and justify this judgement

Having completed the process described above, you should decide whether any concerns that you have identified should be categorised as:

  • No or very minor concerns regarding relevance

  • Minor concerns regarding relevance

  • Moderate concerns regarding relevance

  • Serious concerns regarding relevance

You should begin with the assumption that there are no concerns regarding relevance for the body of data contributing to each review finding. In practice, very minor or minor concerns will usually not lower our confidence in the review finding, while serious concerns will lower our confidence. Moderate concerns may lead us to consider lowering our confidence, as part of our final assessment of all four CERQual components. Where you have concerns about relevance, you should describe these concerns in the CERQual Evidence Profile in sufficient detail to allow users of the review findings to understand the reasons for the assessments made (for more information, see [6]).

Implications when concerns about relevance are identified

Concerns about relevance may not only impact on our confidence in a review finding but also may point to ways of improving future research. First of all, these concerns may indicate the need for more primary research that specifically addresses the evidence gap. The review team should also consider whether the review should be updated when this new research becomes available.

Secondly, where the evidence does not match the specificity of the review question, you should also consider contextual or conceptual justifications for including wider evidence to address the review question. For example, in the absence of relevant evidence on what works concerning public health risk communication in swine flu, you could consider using indirect evidence from risk communication in bird flu. You need to also consider whether the use of indirect evidence will impact on your assessment of the other CERQual components.

Conclusions

Assessing the relevance component requires that you consider potentially important contextual factors at an early stage in the review process. Concerns about relevance impact on assessments of confidence in review findings and are therefore integral to the CERQual approach. However, it is also important to remember that relevance is just one component of the CERQual approach. Having concerns about relevance may not necessarily lead to a downgrading of overall confidence in a review finding as it will be assessed alongside the other three CERQual components.

This paper describes the current thinking of CERQual developers in order to prompt review authors and others to consider issues relating to relevance of findings from qualitative evidence syntheses. However, the CERQual approach in general, and the relevance component in particular, continues to evolve and, in turn, to be informed by ongoing work on the application of research findings more generally.

Open peer review

Peer review reports for this article are available in Additional file 3.

References

  1. Lewin S, Glenton C, Munthe-Kaas H, Carlsen B, Colvin CJ, Gulmezoglu M, Noyes J, Booth A, Garside R, Rashidian A. Using qualitative evidence in decision making for health and social interventions: an approach to assess confidence in findings from qualitative evidence syntheses (GRADE-CERQual). PLoS Med. 2015;12(10):e1001895.

    Article  PubMed  PubMed Central  Google Scholar 

  2. Guyatt GH, Oxman AD, Kunz R, Woodcock J, Brozek J, Helfand M, Alonso-Coello P, Falck-Ytter Y, Jaeschke R, Vist G, et al. GRADE guidelines: 8. Rating the quality of evidence––indirectness. J Clin Epidemiol. 2011;64(12):1303–10.

    Article  PubMed  Google Scholar 

  3. Colvin CJ, Garside R, Wainwright M, Lewin S, Bohren M, Glenton C, Munthe-Kaas HM, Carlsen B, Tuncalp Ö, Noyes J, et al. Applying GRADE-CERQual to qualitative evidence synthesis findings––paper 4: how to assess coherence. Implement Sci. 2018;13(Suppl 1) https://doi.org/10.1186/s13012-017-0691-8.

  4. Glenton C, Carlsen B, Lewin S, Munthe-Kaas HM, Colvin CJ, Tuncalp Ö, Bohren M, Noyes J, Booth A, Garside R, et al. Applying GRADE-CERQual to qualitative evidence synthesis findings––paper 5: how to assess adequacy of data. Implement Sci. 2018;13(Suppl 1) https://doi.org/10.1186/s13012-017-0692-7.

  5. Munthe-Kaas HM, Bohren M, Carlsen B, Glenton C, Lewin S, Colvin CJ, Tuncalp Ö, Noyes J, Booth A, Garside R, et al. Applying GRADE-CERQual to qualitative evidence synthesis findings––paper 3: how to assess methodological limitations. Implement Sci. 2018;13(Suppl 1) https://doi.org/10.1186/s13012-017-0690-9.

  6. Lewin S, Bohren M, Rashidian A, Glenton C, Munthe-Kaas HM, Carlsen B, Colvin CJ, Tuncalp Ö, Noyes J, Booth A, et al. Applying GRADE-CERQual to qualitative evidence synthesis findings––paper 2: how to make an overall CERQual assessment of confidence and create a Summary of Qualitative Findings table. Implement Sci. 2018;13(Suppl 1) https://doi.org/10.1186/s13012-017-0689-2.

  7. Ames HMR, Glenton C, Lewin S. Parents’ and informal caregivers’ views and experiences of communication about routine childhood vaccination: a synthesis of qualitative evidence. Cochrane Database Syst Rev. 2017;2:CD011787.

    PubMed  Google Scholar 

  8. Aslam RW, Hendry M, Carter B, Noyes J, Rycroft Malone J, Booth A, Pasterfield D, Charles JM, Craine N, Tudor Edwards R, et al. Interventions for preventing unintended repeat pregnancies among adolescents (protocol). Cochrane Database Syst Rev. 2015;(1):Art. No.: CD011477.

  9. Bohren MA, Hunter EC, Munthe-Kaas HM, Souza JP, Vogel JP, Gulmezoglu AM. Facilitators and barriers to facility-based delivery in low- and middle-income countries: a qualitative evidence synthesis. Reprod Health. 2014;11(1):71.

    Article  PubMed  PubMed Central  Google Scholar 

  10. Bohren MA, Vogel JP, Hunter EC, Lutsiv O, Makh SK, Souza JP, Aguiar C, Saraiva Coneglian F, Diniz AL, Tuncalp O, et al. The mistreatment of women during childbirth in health facilities globally: a mixed-methods systematic review. PLoS Med. 2015;12(6):e1001847. discussion e1001847

    Article  PubMed  PubMed Central  Google Scholar 

  11. Colvin CJ, de Heer J, Winterton L, Mellenkamp M, Glenton C, Noyes J, Lewin S, Rashidian A. A systematic review of qualitative evidence on barriers and facilitators to the implementation of task-shifting in midwifery services. Midwifery. 2013;29(10):1211–21.

    Article  PubMed  Google Scholar 

  12. Glenton C, Colvin CJ, Carlsen B, Swartz A, Lewin S, Noyes J, Rashidian A. Barriers and facilitators to the implementation of lay health worker programmes to improve access to maternal and child health: qualitative evidence synthesis. Cochrane Database Syst Rev. 2013;10:CD010414.

    Google Scholar 

  13. Munabi-Babigumira S, Glenton C, Lewin S, Fretheim A, Nabudere H. Factors that influence the provision of intrapartum and postnatal care by skilled birth attendants in low- and middle-income countries: a qualitative evidence synthesis. Cochrane Database Syst Rev. 2017;(11):Art. No.: CD011558.

  14. Munthe-Kaas HM, Hammerstrøm KT, et al. Effekt av og erfaringer med kontinuitetsfremmende tiltak i barnevernsinstitusjoner. Norwegian Knowledge Centre for the Health Services: Oslo; 2013.

    Google Scholar 

  15. O'Brien TD, Noyes J, Spencer LH, Kubis HP, Hastings RP, Edwards RT, Bray N, Whitaker R. ‘Keep fit’ exercise interventions to improve health, fitness and well-being of children and young people who use wheelchairs: mixed-method systematic review protocol. J Adv Nurs. 2014;70(12):2942–51.

    Article  PubMed  Google Scholar 

  16. Rashidian A, Shakibazadeh E, Karimi- Shahanjarini A, Glenton C, Noyes J, Lewin S, Colvin C, Laurant M. Barriers and facilitators to the implementation of doctor-nurse substitution strategies in primary care: qualitative evidence synthesis (protocol). Cochrane Database Syst Rev. 2013;2:CD010412.

    Google Scholar 

  17. Whitaker R, Hendry M, Booth A, Carter B, Charles J, Craine N, Edwards RT, Lyons M, Noyes J, Pasterfield D, et al. Intervention Now To Eliminate Repeat Unintended Pregnancy in Teenagers (INTERUPT): a systematic review of intervention effectiveness and cost-effectiveness, qualitative and realist synthesis of implementation factors and user engagement. BMJ Open. 2014;4(4):e004733.

    Article  PubMed  PubMed Central  Google Scholar 

  18. Downe S, Finlayson K, Tuncalp O, Metin Gulmezoglu A. What matters to women: a systematic scoping review to identify the processes and outcomes of antenatal care provision that are important to healthy pregnant women. BJOG. 2016;123(4):529–39.

    Article  CAS  PubMed  Google Scholar 

  19. Odendaal WA, Goudge J, Griffiths F, Tomlinson M, Leon N, Daniels K. Healthcare workers’ perceptions and experience on using mHealth technologies to deliver primary healthcare services: qualitative evidence synthesis (protocol). Cochrane Database Syst Rev. 2015;(11):Art. No.: CD011942.

  20. Lewin S, Booth A, Glenton C, Munthe-Kaas HM, Rashidian A, Wainwright M, Bohren MA, Tuncalp Ö, Colvin CJ, Garside R, et al. Applying GRADE-CERQual to qualitative evidence synthesis findings: introduction to the series. Implement Sci. 2018;13(Suppl 1) https://doi.org/10.1186/s13012-017-0688-3.

  21. Lavis JN, Oxman AD, Lewin S, Fretheim A. SUPPORT Tools for evidence-informed health Policymaking (STP). Health Res Policy Syst/BioMed Central. 2009;7(Suppl 1):I1.

    Article  Google Scholar 

  22. Bambra C. Real world reviews: a beginner’s guide to undertaking systematic reviews of public health policy interventions. J Epidemiol Community Health. 2011;65(1):14–9.

    Article  CAS  PubMed  Google Scholar 

  23. Hannes K, Harden A. Multi-context versus context-specific qualitative evidence syntheses: combining the best of both. Res Synth Methods. 2011;2(4):271–8.

    Article  PubMed  Google Scholar 

  24. Booth A. Searching for qualitative research for inclusion in systematic reviews: a structured methodological review. Syst Rev. 2016;5:74.

    Article  PubMed  PubMed Central  Google Scholar 

  25. Gardner B, Whittington C, McAteer J, Eccles MP, Michie S. Using theory to synthesise evidence from behaviour change interventions: the example of audit and feedback. Soc Sci Med. 2010;70(10):1618–25.

    Article  PubMed  Google Scholar 

  26. Main C, Thomas S, Ogilvie D, Stirk L, Petticrew M, Whitehead M, Sowden A. Population tobacco control interventions and their effects on social inequalities in smoking: placing an equity lens on existing systematic reviews. BMC Public Health. 2008;8:178.

    Article  PubMed  PubMed Central  Google Scholar 

  27. Noyes J, Hendry M, Booth A, Chandler J, Lewin S, Glenton C, Garside R. Current use was established and Cochrane guidance on selection of social theories for systematic reviews of complex interventions was developed. J Clin Epidemiol. 2016;75:78–92.

    Article  PubMed  Google Scholar 

  28. Barroso J, Sandelowski M, Voils CI. Research results have expiration dates: ensuring timely systematic reviews. J Eval Clin Pract. 2006;12(4):454–62.

    Article  PubMed  PubMed Central  Google Scholar 

  29. Evidence for Policy and Practice Information and Co-ordinating Centre. EPPI-Centre Methods for Conducting Systematic Reviews. London: University of London; 2007.

    Google Scholar 

  30. Toews I, Booth A, Berg RC, Lewin S, Glenton C, Munthe-Kaas HM, Noyes J, Schroter S, Meerpohl JJ. Further exploration of dissemination bias in qualitative research required to facilitate assessment within qualitative evidence syntheses. J Clin Epidemiol. 2017;88:133–9.

  31. Toews I, Glenton C, Lewin S, Berg RC, Noyes J, Booth A, Marusic A, Malicki M, Munte-Kaas HM, Meerpohl JJ. Extent, awareness and perception of dissemination bias in qualitative research: an explorative survey. PLoS One. 2016;11(8):e0159290.

    Article  PubMed  PubMed Central  Google Scholar 

  32. Booth A, Lewin S, Glenton C, Munthe-Kaas HM, Meerpohl J, Rees R, Noyes J, Rashidian A, Berg R, Nyakango B, et al. Applying GRADE-CERQual to qualitative evidence synthesis findings––paper 7: understanding the potential impacts of dissemination bias. Implement Sci. 2018;13(Suppl 1) https://doi.org/10.1186/s13012-017-0694-5.

  33. Kavanagh J, Oliver S, Lorenc T. Reflections on developing and using PROGRESS-Plus. Equity Update. 2008;2(1):1–3.

    Google Scholar 

  34. Burford BJ, Welch V, Waters E, Tugwell P, Moher D, O'Neill J, Koehlmoos T, Petticrew M. Testing the PRISMA-Equity 2012 reporting guideline: the perspectives of systematic review authors. PLoS One. 2013;8(10):e75122.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  35. Pfadenhauer LM, Mozygemba K, Gerhardus A, Hofmann B, Booth A, Lysdahl KB, Tummers M, Burns J, Rehfuess EA. Context and implementation: a concept analysis towards conceptual maturity. Z Evid Fortbild Qual Gesundhwes. 2015;109(2):103–14.

    Article  PubMed  Google Scholar 

  36. Lewin S, Hendry M, Chandler J, Oxman AD, Michie S, Shepperd S, Reeves BC, Tugwell P, Hannes K, Rehfuess EA, et al. Assessing the complexity of interventions within systematic reviews: development, content and use of a new tool (iCAT_SR). BMC Med Res Methodol. 2017;17(1):76.

    Article  PubMed  PubMed Central  Google Scholar 

  37. Hoffmann TC, Glasziou PP, Boutron I, Milne R, Perera R, Moher D, Altman DG, Barbour V, Macdonald H, Johnston M, et al. Better reporting of interventions: template for intervention description and replication (TIDieR) checklist and guide. BMJ. 2014;348:g1687.

    Article  PubMed  Google Scholar 

Download references

Acknowledgements

Our thanks for their feedback to those who participated in the GRADE-CERQual Project Group meetings in January 2014 or June 2015 or gave comments to the paper: Elie Akl, Heather Ames, Zhenggang Bai, Rigmor Berg, Karen Daniels, Hans de Beer, Kenny Finlayson, Bela Ganatra, Susan Munabi-Babigumira, Andy Oxman, Vicky Pileggi, Kent Ranson, Rebecca Rees, Holger Schünemann, Elham Shakibazadeh, Birte Snilstveit, James Thomas, Hilary Thompson, Judith Thornton and Josh Vogel. Thanks also to Sarah Rosenbaum for developing the figures used in this series of papers and to the members of the GRADE Working Group for their input. The guidance in this paper has been developed in collaboration and agreement with the GRADE Working Group (http://www.gradeworkinggroup.org).

Funding

This work, including the publication charge for this article, was supported by funding from the Alliance for Health Policy and Systems Research, WHO (www.who.int/alliance-hpsr/en/). Additional funding was provided by the Department of Reproductive Health and Research, WHO (http://www.who.int/reproductivehealth/en/); Norad (Norwegian Agency for Development Cooperation: www.norad.no), the Research Council of Norway (www.forskningsradet.no); and the Cochrane Methods Innovation Fund. SL is supported by funding from the South African Medical Research Council (www.mrc.ac.za). The funders had no role in study design, data collection and analysis, preparation of the manuscript or the decision to publish.

Availability of data and materials

Additional materials are available on the GRADE-CERQual website (www.cerqual.org)

To join the CERQual Project Group and our mailing list, please visit our website: http://www.cerqual.org/contact/. Developments in CERQual are also made available via our Twitter feed: @CERQualNet.

About this supplement

This article has been published as part of Implementation Science Volume 13 Supplement 1, 2018: Applying GRADE-CERQual to Qualitative Evidence Synthesis Findings. The full contents of the supplement are available online at https://implementationscience.biomedcentral.com/articles/supplements/volume-13-supplement-1.

Author information

Authors and Affiliations

Authors

Contributions

All authors participated in the conceptual design of the CERQual approach. JN and AB wrote the first draft of the manuscript. All authors contributed to the writing of the manuscript. All authors have read and approved the manuscript.

Corresponding author

Correspondence to Andrew Booth.

Ethics declarations

Ethics approval and consent to participate

Not applicable. This study did not undertake any formal data collection involving humans or animals.

Consent for publication

Not applicable

Competing interests

The authors declare that they have no competing interests.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Additional files

Additional file 1:

Key definitions relevant to CERQual. (PDF 619 kb)

Additional file 2:

External relevance of data. (PDF 643 kb)

Additional file 3:

Open peer review reports. (PDF 133 kb)

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Noyes, J., Booth, A., Lewin, S. et al. Applying GRADE-CERQual to qualitative evidence synthesis findings–paper 6: how to assess relevance of the data. Implementation Sci 13 (Suppl 1), 4 (2018). https://doi.org/10.1186/s13012-017-0693-6

Download citation

  • Published:

  • DOI: https://doi.org/10.1186/s13012-017-0693-6

Keywords