Use of the GRADE approach in health policymaking and evaluation: a scoping review of nutrition and physical activity policies
Implementation Science volume 15, Article number: 37 (2020)
Nutrition and physical activity policies have the potential to influence lifestyle patterns and reduce the burden of non-communicable diseases. In the world of health-related guidelines, GRADE (Grading of Recommendations, Assessment, Development and Evaluation) is the most widely used approach for assessing the certainty of evidence and determining the strength of recommendations. Thus, it is relevant to explore its usefulness also in the process of nutrition and physical activity policymaking and evaluation.
The purpose of this scoping review was (i) to generate an exemplary overview of documents using the GRADE approach in the process of nutrition and physical activity policymaking and evaluation, (ii) to find out how the GRADE approach has been applied, and (iii) to explore which facilitators of and barriers to the use of GRADE have been described on the basis of the identified documents. The overarching aim of this work is to work towards improving the process of evidence-informed policymaking in the areas of dietary behavior, physical activity, and sedentary behavior.
A scoping review was conducted according to current reporting standards. MEDLINE via Ovid, the Cochrane Library, and Web of Science were systematically searched up until 4 July 2019. Documents describing a body of evidence which was assessed for the development or evaluation of a policy, including documents labeled as “guidelines,” or systematic reviews used to inform policymaking were included.
Thirty-six documents were included. Overall, 313 GRADE certainty of evidence ratings were identified in systematic reviews and guidelines; the strength of recommendations/policies was assessed in four documents, and six documents mentioned facilitators or barriers for the use of GRADE. The major reported barrier was the initial low starting level of a body of evidence from non-randomized studies when assessing the certainty of evidence.
This scoping review found that the GRADE approach has been used for policy evaluations, in the evaluation of the effectiveness of policy-relevant interventions (policymaking), as well as in the development of guidelines intended to guide policymaking. Several areas for future research were identified to explore the use of GRADE in health policymaking and evaluation.
According to the most recent report by the Global Burden of Disease study group, non-communicable diseases (NCDs), such as cardiovascular disease, cancer, and type 2 diabetes, accounted for 64.8% of total deaths worldwide in 2017 . Major risk factors for the development of NCDs are obesity, dyslipidemia, insulin resistance, and hypertension [1,2,3]. Suboptimal dietary behavior and physical inactivity, including increased sedentary behavior, are related to these risk factors. Therefore, adherence to an optimal diet and physical activity recommendations can contribute to the prevention of obesity and several NCDs .
On a public health level, policies are important instruments that can be used to change people’s behavior . While the definition of what constitutes a policy remains a topic of debate, for the purpose of this review, we use the term “policy” both for (a) interventions implemented at a population level, as well as for (b) recommendations on a specific course of action to be implemented by a government or other public body (see Table 1).
Policies have the potential to influence lifestyle patterns within a population . Certain good-practice characteristics of successful policies on optimal diets and physical activity have been identified in recent research, such as the use of the theory of change, analyses of target populations and target behavior, content development and management, multidimensionality, and appropriate consideration of practitioners and settings . For example, interventions such as mass media campaigns, school programs, or changes of the environment through communities showed to be effective in increasing physical activity levels among a population, especially when they operate on multiple levels and contain multiple components . However, to be maximally effective, they require regular evaluation and improvement. The Policy Evaluation Network (PEN), a consortium funded by the European Union, was founded to help improve policymaking in the areas of dietary behavior, physical activity, and sedentary behavior by evaluating existing policies [6, 11]. This scoping review is part of the PEN-project, with a special focus on policies that explicitly target, or are likely to impact upon, population health, in particular through NCDs.
To increase the acceptance and impact of polices, it might be helpful if they adhered to principles widely accepted, for instance, similar to health guidelines, i.e., they should be transparent, structured, and evidence-informed [12, 13]. Within the PEN-project, it is explored if these guideline standards, which also include the assessment of the certainty of evidence, should be considered when developing and evaluating a policy .
The Grading of Recommendations, Assessment, Development and Evaluation (GRADE) approach and the GRADE Evidence to Decision (EtD) frameworks are important methods for the development and evaluation of recommendations for practice and can therefore also help to make evidence-informed decisions on a population level [15,16,17]. GRADE is a well-recognized tool to facilitate structured decision-making and is potentially the most widely used approach for assessing the certainty of evidence and determining the strength of a recommendation . It is standard practice to use the GRADE approach to form trustworthy evidence-informed recommendations, especially in the development of guidelines in the broader field of medicine. For this reason, it seems sensible to explore its potential in the process of developing and evaluating policies concerned with nutrition and physical activity .
The primary aim of this scoping review is to investigate how and to what degree the GRADE approach has contributed to policy development in the context of nutrition and physical activity. To this end, the following steps were taken: (i) an overview of documents using the GRADE approach in the context of nutrition and physical activity policymaking and evaluation was compiled, (ii) methods of applying the GRADE approach (e.g., by rating the certainty of evidence or the strength of a recommendation) were recorded, and (iii) facilitators of and barriers to the use of GRADE described in the identified documents were identified and explored.
This scoping review was conducted according to the methodology of the Joanna Briggs Institute’s (JBI) Reviewers’ Manual 2015 and is reported following the PRISMA (Preferred Reporting Items for Systematic reviews and Meta-Analyses) extension for scoping reviews, the PRISMA-ScR Statement (Additional file 1) [19, 20]. To complement our scoping review, a survey was conducted by contacting European health policymaking institutions to further investigate the use of GRADE in nutrition and physical activity policymaking and evaluation and to potentially identify additional policy documents.
First, policy evaluations in the form of systematic reviews and other relevant documents published in peer-reviewed journals were identified through searches in the databases MEDLINE, Web of Science, and the Cochrane Library. The search strategy was developed with the support of an information specialist and adapted for each database accordingly. Databases were searched up until 4 of July 2019, and the search was not restricted to any language. The MEDLINE (Ovid) search strategy is presented in Additional file 2. Moreover, forward citation tracking and reference checking from included documents were performed.
Secondly, an internet-based search through the Google™ search engine and through Google Scholar™ was performed to identify gray literature and different relevant institutions and organizations, such as the Food and Agriculture Organization for the United Nations (FAO), World Health Organization (WHO), World Cancer Research Fund International (WCRF), the National Institute for Health and Care Excellence (NICE), and the Campbell Collaboration. Following this, the types of documents relevant for policymaking and which address dietary behavior, physical activity, and/or sedentary behavior, such as guidelines, reports, or systematic reviews, were searched on the websites of relevant organizations. Documents were found both through the Google search directly and also retrieved from the websites of those organizations and institutions identified in the search.
Thirdly, an extensive overview of existing policies including policy evaluations from the NOURISHING-database by the WCRF was considered . Lastly, science platforms and databases such as ProQuest, Dimensions, BASE, Metager, Oaister, and the Political Science Database (Berkeley) were searched.
Because of the lack of a standard definition of the term “policy” across various areas of health policy and policy decision-making, the current search also aimed to identify documents that might not have used the term “policies” or “policy evaluations.” We focused on documents that described a body of evidence which was assessed for the development or evaluation of a policy, including documents labeled as “guidelines,” or systematic reviews used to inform policymaking . Furthermore, we established criteria for interventions and/or recommendations (i.e., policies) to be eligible. Documents had to meet the following criteria to be included in this scoping review:
At least one intervention or recommendation of a policy document targeted the PEN-related outcomes: dietary behavior, physical activity, or sedentary behavior.
Recommendations targeted a population rather than individuals. This means that at least one intervention or recommendation had to fit in one of the following categories: school program (only if mandatory to school children, e.g., more physical exercise lessons or change in school meal offerings), mass media (e.g., commercials, poster campaigns), infrastructure (e.g., standing desks), financial incentive (e.g., taxation on sugar or fat), change of offerings (e.g., portion/package sizes), nutrition labeling, and population education (in the sense of recommendations, e.g., guidelines). These categories were established by reviewing a set of pre-identified documents.
Specific measures or action plans (potential policies) that were or could be implemented by a government or an inter-governmental organization such as the WHO (e.g., labeling and school programs).
In addition to meeting criteria i–iii, GRADE was used to assess the certainty of evidence or the strength of recommendations or reasons for not using GRADE were explicitly noted.
There were three exclusion criteria:
Since the GRADE approach is primarily applicable in the context of evidence syntheses within systematic reviews, health technology assessments (with or without meta-analyses) or guidelines, surveys, and primary studies (e.g., randomized controlled trials (RCTs) or non-randomized studies (NRSs)) were excluded.
Documents concerning the methodological aspects of policy implementation were excluded.
Documents about interventions that specifically target individuals instead of a population were also excluded.
Selection process of sources of evidence
First, title and abstract screening was performed by one reviewer (JZ). Only clearly irrelevant references were excluded at this stage. Second, for all potentially relevant references, full-text publications were obtained and checked for final inclusion by two reviewers (JZ, LS) independently. Uncertainties were resolved through discussion with a third author (JJM).
For included documents, one reviewer (JZ) extracted the following characteristics: name of first author, year of publication, document type, study aim, outcome of interventions (physical activity, dietary behavior, sedentary behavior, multi-component), design of primary studies included in the document, policy/intervention(s), target group/study setting, use of GRADE, GRADE ratings for relevant outcomes (including down- and upgrading factors), facilitators, and barriers to apply GRADE. The data extraction form was piloted and proved to be applicable to all types of retrieved documents. Data extraction was verified by a second reviewer (LS).
Data categorization and presentation
The documents were categorized according to policy evaluation, evidence of effectiveness for policymaking, or a recommendation to guide policymaking (see Table 1). Furthermore, we categorized the documents according to the applied interventions (type of policy) in the studies (e.g., school program and mass media), according to the setting of the studies (e.g., workplace and community) and according to PEN-relevant outcomes (dietary behavior, physical activity, and sedentary behavior). All these characteristics were collected for each document and were displayed in Table 2. In further tables, we displayed the GRADE ratings, facilitators, and barriers for the use of GRADE (Table 3), as well as the strengths of recommendations (Additional file 4).
To complement the scoping review, a survey was conducted by contacting Ministries of Health or other responsible institutions that play a role in health policymaking in all 53 European countries, as listed by the WHO (Additional file 3) . Contact information was searched through the FAO websites, the WHO websites, and respective websites of national governments. An e-mail was sent in English with a brief description of the project and the question whether the GRADE approach had been used for nutrition and physical activity policymaking in these countries.
The database search retrieved 6164 documents (MEDLINE via Ovid, 4782; Cochrane Library, 309; Web of Science, 1073). Another 19 documents were retrieved through further reference checking and from the gray literature. After removal of duplicates (n = 511), title and abstract screening of 5672 documents was performed, from which 5443 documents were excluded. The remaining 229 documents were reviewed in full-text screening. The eligibility criteria were met by 36 documents which were included in this scoping review (Fig. 1) [23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58]. The science databases and platforms did not yield any additional documents relevant for this scoping review.
An overview of the use of GRADE in policymaking and policy evaluation
The characteristics of the included documents are displayed in Table 2. All documents were published in English, and they were either systematic reviews with meta-analyses (n = 10; 28%), systematic review protocols (n = 8; 22%), systematic reviews without meta-analyses (n = 9; 25 %), guidelines (n = 6; 17%), an umbrella review (n = 1; 3%), or papers on guideline making (n = 2; 5%). From these findings, 17 documents (47%) were Cochrane reviews (including five Cochrane review protocols). Four documents were international guidelines by the WHO and one guideline by NICE (physical activity and the environment ). Furthermore, two sets of national-level guideline recommendations to guide policymaking were found: the Canadian 24-Hour Movement Guidelines for Children and Youth [52, 53]. Documents were published between 2011 and 2019, and half of those (n = 18) were published within the last 3 years. Settings of the studies included communities, schools, or workplaces. The majority of recommendations/interventions addressed the general population through various forms of mass media. Furthermore, the included documents examined various combinations of interventions addressing dietary behavior, physical activity, and sedentary behavior, and also combinations of these with other health-related topics. Most of the studies investigated interventions related to dietary behavior alone (39%), followed by physical activity (17%). There was only one study that assessed interventions aiming to diminish sedentary behavior alone (3%), and all other documents (41%) were based on interventions that targeted various outcomes, focusing on multicomponent interventions that addressed general health and aspects such as alcohol or tobacco consumption.
Many of the documents included did not review implemented policies and focused rather on interventions that have the potential to drive policy in the future. Seven of the 36 documents include studies in which the effects of implemented nutrition and physical activity policies were examined, including nutrition labeling [27, 49], SSB-taxes , the WHO Health Promoting Schools framework , taxation of fat content in food , sodium restriction , and taxation of unprocessed sugar or sugar-added foods . Five of these are based on policies issued by governments (see Table 2). Twenty-two of the included documents applied the GRADE approach for grading the certainty of evidence for PEN-related outcomes (for instance, increase of physical activity, change of body-mass index (BMI), and sugar or salt intake) (see Table 3). The remaining 14 documents were either systematic review protocols (n = 8, [26, 33, 35, 36, 40, 44, 48, 50]) where GRADE will be used for grading the certainty of evidence, one recommendation to guide policymaking which used GRADE-ADOLOPMENT (GRADE Evidence to Decision frameworks for adoption, adaptation, and de novo development of trustworthy recommendations) (n = 1, ), documents in which GRADE was used on another development stage of the document (i.e., recommendations where GRADE ratings of included systematic reviews were evaluated, n = 2, [52, 53]), or documents in which the barriers against the use of GRADE were stated (n = 3, [30, 32, 42]) (see Table 4).
Table 3 shows the ratings of the certainty of evidence for all PEN-related outcomes for each study (max. seven ratings per identified document are presented in the table), and the reasons for down- and upgrading. Overall, there were 313 individual outcome ratings (107 GRADE ratings were done for the NICE guideline alone ): 34% of very low, 38% of low, 21% of moderate, and 7% of high certainty of evidence. The main reasons for downgrading were high risk of bias, imprecision (e.g., for wide confidence interval, small sample size, and incomplete outcome data), or there was an initial low starting level of a body of evidence from non-randomized study design type (Table 3).
The WHO guidelines were the only documents which, in addition to rating the certainty of evidence for each outcome, also provided the “strength of recommendations” for each recommendation included in the guideline (Additional file 4).
Facilitators and barriers for the use of GRADE from identified documents
In two documents, an adaption of GRADE was used for primary outcomes [24, 30]. In three documents, the authors mentioned the GRADE approach or presented specific barriers that prevented them from applying GRADE [30, 32, 42], and in one document, the GRADE-ADOLOPMENT approach was used  (see Table 4).
Barriers for the use of GRADE for the assessment of the certainty of evidence were mentioned only in six documents. Reasons cited by authors for their decision not to use GRADE included, among others, incomplete, insufficient, or inaccessible data [24, 32, 42]. In another document, it was mentioned that difficulties in the application of the GRADE approach were seen due to the research topic, where RCTs are difficult to conduct, or because of subjective outcome measures such as quality of life . Dyson et al.  mentioned some general challenges with the GRADE approach, such as the initial low starting level of a body of evidence from cohort studies and the difficulties of applying GRADE for individualized interventions. Instead, Dyson et al. adopted a grading system very similar to the GRADE system, in which recommendations on the certainty of evidence are provided on a scale from 1 to 4 (strong to very low strength) . The motivation for using this adapted approach is not discussed. In the NICE guideline, a modified version of GRADE was used where NRSs started as “high quality evidence” when the study design was the most feasible one for the examined intervention . Only one document mentioned facilitators for the use of GRADE; however, this was for another GRADE methodology, namely the GRADE-ADOLOPMENT, which was used by Okely et al. for the development of new guidelines . There was no mention of facilitators or barriers for assessing the strengths of recommendations in any of the included documents. Facilitators and barriers are presented in Table 4.
Complementary survey among European health policymakers
No documents were retrieved through the survey sent to ministries and authorities in 53 European countries. Staff members from eleven countries replied to the request, stating that their national policies were not developed according to the GRADE approach (Additional file 3). Representatives from those countries, namely Austria, Lithuania, Luxemburg, Monaco, the Netherlands, Northern Ireland, and Poland, replied that they do not use GRADE in any way for policymaking. The German Nutrition Society used a newly developed grading system called NutriGrade for their protein guideline . As a response to the survey, Bosnia and Herzegovina showed explicit interest in using GRADE for future policymaking. Iceland and Norway stated that they do not use GRADE for policymaking but in some cases for national clinical guidelines. Monaco replied that they refer to the WHO or French national recommendations. Furthermore, the Nordic Nutrition Recommendations use WHO guidelines, and Norway and Iceland reported that their policies are either based on the Nordic Nutrition Recommendations or they make policies which are based on WHO guidelines directly. No reasons were given whatsoever why GRADE has not been used directly.
Summary of findings
To the best of our knowledge, this scoping review is the first to examine if and to what extent the GRADE approach has been used in nutrition and physical activity policymaking and policy evaluation in the areas of dietary behavior, physical activity, and sedentary behavior. There were no findings of actual policies or formulation processes from governmental bodies; however, five systematic reviews included evidence from interventions that were based on policies issued by a government or governmental organization (see Table 2). This scoping review found that the GRADE approach has been used for policy evaluations, in the evaluation of the effectiveness of policy-relevant interventions, as well as for guidelines and recommendations intended to guide policymaking, where the evidence for a recommendation or an intervention has to be assessed. These results may contribute to improving the process of evidence-informed policymaking in the areas of dietary behavior, physical activity, and sedentary behavior.
Almost half of the included documents that used the GRADE approach to assess the certainty of evidence are Cochrane Reviews (n = 17). The reason for this is that Cochrane adopted the GRADE approach for the evaluation of the certainty of evidence in systematic reviews . The strength of recommendations has only been determined in WHO guidelines, and no barriers to its use were mentioned in these cases. Already in 2003, the WHO selected the GRADE approach as the method of choice to be used in their guideline development . This explains why the guidelines identified in the context of this scoping review are mostly WHO guidelines.
Although guideline and recommendation development are a major part of policymaking, a number of perceived concerns may prevent institutions from adopting available methods. With respect to applying GRADE to policies affecting dietary behavior, physical activity, and sedentary behavior, concerns that are frequently raised include the assessment of risk of bias and the initial low starting level of a body of evidence obtained from NRSs. According to the GRADE approach, the certainty of evidence is initially determined by study design: a body of evidence from RCTs starts with a high certainty, whereas a body of evidence from NRSs, such as prospective observational studies, starts with low certainty due to confounding and selection bias issues associated with NRSs [63, 64]. However, with the advent of tools that use the concept of a target trial as reference point, such as the ROBINS-I (risk of bias in non-randomized studies of interventions), initial certainty of the evidence can also be high for bodies of evidence from NRSs [63, 64]. Nevertheless, as stated above, it is likely that NRSs would be downrated to low certainty because of confounding and selection bias .
There has been a long debate regarding what constitutes best evidence in nutrition research and whether it emerges from RCTs in which the effects of a dietary change on disease, intermediate or surrogate markers, or recognized risk markers are evaluated . However, most dietary intervention RCTs are of short duration and do not target patient-relevant outcomes such as morbidity or mortality. RCTs, if well-designed and conducted, provide robust answers to the research questions they address and are considered the ideal methodology for causal inference. NRSs, on the other hand, provide less-robust information regarding causality but are usually considered more applicable for nutrition research. Additionally, all common dietary assessment methods used in nutrition research, such as food consumption records, 24 h recalls, dietary records, dietary history, and food frequency questionnaires suffer from inherent limitations, for example, because the measures are subjective or the results are difficult to reproduce .
Apart from the WHO guidelines and the NICE guideline, there were no publications closer to the legislative policymaking process (e.g., policy briefs, policy documents, proposals to inform laws or regulations) that used GRADE. The legislative process underlying policymaking is complex and time-consuming. In addition, there are specific challenges associated with translating empirical findings from the lab into actionable policies in government [66,67,68,69]. The absence of documents could be due to various factors: for example, key documents not being publicly available (e.g., internal position papers by political parties), documents being informed by GRADE not explicitly using the term, or the search strategy being specifically tailored to identify those types of policy documents. It could also be due to GRADE not being employed to inform policymakers’ decisions, for example, due to issues in uptake of evidence in the policymaking process [67,68,69], inadequate knowledge-translation strategies [70,71,72], or an unawareness about GRADE. Future research needs to address this issue.
Strengths and limitations
This scoping review has several strengths. Firstly, the inclusion criteria and the search strategy planned for this scoping review were broad, including an extensive search of gray literature such as databases of the WHO or the WCRF in addition to more traditional sources. This access to a wide variety of documents in the research field yields a better overview than previously available. Secondly, we performed a rigorous approach to screening and data extraction, which was performed by two reviewers independently. Thirdly, the survey among national policymakers of the European region brought relevant input and insights into this study by giving an idea about the current use of GRADE within policymaking institutions.
Limitations of this scoping review in our view are as follows. Firstly, developing a suitable search strategy was challenging. Though the search was systematic and as broad as possible, it proved to be difficult to grasp all the facets of the topic. Secondly, only documents were included in which GRADE was applied or reasons were provided for not using GRADE. A different approach with inclusion of documents that could have applied GRADE but did not do so and a research design enabling an in-depth investigation of policymakers’ views (e.g., a mixed methods approach) might have provided further insights into this topic. These approaches would have allowed for a thorough investigation of the extent of evidence use without structured consideration of certainty of evidence and strength of recommendations in policymaking and evaluation. However, this was beyond the scope of this project. Thirdly, a standard definition of the terms “policy” and “health policy” was difficult to determine since various definitions can be found in the literature. Only a few of the included documents (n = 7) actually examined real policies. In these cases, the authors did not distinguish between the types of included studies, that is, if an intervention was carried out in an experimental/artificial setting or based on a law/regulation. Lastly, a document was included even when only one of the interventions in the review was considered a policy intervention although the majority of the remaining interventions were not. In this case, ratings for the certainty of evidence with GRADE are likely to be higher than for NRSs in real-life settings as it is much easier to conduct RCTs in experimental settings.
Findings from other studies
This is the first scoping review on the use of GRADE in nutrition and physical activity policy evaluation, but other publications have also addressed the complexity of interventions in the public health field, which might prevent researchers and policymakers from applying this approach. Public health and health policy interventions are often thought of as events in systems , whereby complex intervention (e.g., characterized by several interacting active components intended to influence multiple behaviors ) are implemented in an interactive system (e.g., characterized by changes of the system due to emergent properties, adaptivity, or feedback mechanisms [73, 75]). However, there is specific guidance on how the GRADE approach can be applied in these cases [76, 77].
A study conducted by Rehfuess and Akl in 2013 examined the experience with applying the GRADE approach to public health interventions . In their survey performed in 2013, NICE, which is the national institute that provides guidance and advice to improve health and social care in the UK, stated that the most important reason for not using GRADE when investigating (complex) public health interventions was the fact that most of the evidence originates from NRSs . However, since 2014, NICE included the GRADE approach in their manual for guideline development, also for public health guidelines . Since bodies of NRSs start with low certainty of evidence, it seemed that this would hinder the application of GRADE in such cases . This issue was also brought up in the findings of this scoping review as mentioned above.
Recently, Blake and colleagues showed that from 32 food-based dietary guidelines published since 2010, none of the guidelines used the GRADE approach to evaluate the certainty of evidence or graded the strength of recommendations . However, the authors also acknowledged that methods may not be directly applicable to the types of studies included in food-based dietary guidelines. Of note, GRADE has established working groups focusing on an approach for public health, modeling studies, or environmental exposure studies.
Implications that follow for the broader research field
It has been pointed out that the use of the GRADE approach in particular in relation to bias assessment is challenging (e.g., users of GRADE may inappropriately double count the risk of confounding and selection bias so the certainty of evidence is downgraded too much) [60, 80]. Further, the usefulness of GRADE to assess the evidence of effectiveness in domains of health policy other than dietary behavior, physical activity, and sedentary behavior needs to be explored. In general, knowledge translation and exchange strategies (i.e., methods of bringing scientific findings to policymakers) have provided mixed results regarding the use or uptake of scientific evidence in the policymaking process [70,71,72]. One key issue has been the perceived limited relevance, generalizability, and applicability of research findings to a specific setting . Future research should focus on assessing the views of policymakers on the usefulness of GRADE to inform their decisions.
Moreover, beyond the evidence of effectiveness in a given context, public health and health policy decision-makers have to consider numerous additional aspects [82,83,84,85], for example, stakeholder interests, ethical concerns, equity/equality considerations, financial implications, or societal and cultural values and norms. These have to be taken into account along with scientific evidence of effectiveness which is often not the most important consideration . In addition, there are individual, group, or systemic factors such as own experiences of policymakers, the political system, institutional mechanisms (structures, processes, or regulations), or contextual, societal factors [69, 86]. The GRADE EtD frameworks might be a suitable approach to address these issues. They might help policymakers to identify, prioritize, and address issues beyond benefit-harm considerations in the healthcare decision-making process in a structured and transparent way such as resource use and cost-effectiveness, equity, acceptability, and feasibility [15, 17]. However, we did not identify any documents which used the EtD frameworks.
This scoping review was the first attempt to shed some light on how the GRADE approach has been used in nutrition and physical activity policymaking and evaluation. In conclusion, we show that the GRADE approach is able to support the processes of health policymaking (e.g., guidelines) and policy evaluation (e.g., systematic reviews evaluating policy interventions) by facilitating a structured and transparent use of evidence, and that it has already been used in the areas of dietary behavior, physical activity, and sedentary behavior. The aforementioned barriers regarding the use of the GRADE approach have been recognized by the GRADE working group as well, since new tools such as ROBINS-I have been endorsed for the assessment of the certainty of evidence, especially to help decision-makers and policymakers in the area of public health. GRADE is being used as a standard method during guideline development in various institutions, which is an important part in the process of informing policymaking. Nevertheless, our survey shows that GRADE is not being used regularly for actual policymaking. We believe the evaluation of policies should be consistent with standardized methods in order to have the best possible impact on improving health and quality of life, and the GRADE approach could play an important role in this.
Availability of data and materials
All data generated or analyzed during this study are included in this published article [and its supplementary information files].
Body mass index
Evidence to Decision
Food and Agriculture Organization for the United Nations
Grading of Recommendations Assessment, Development and Evaluation
GRADE Evidence to Decision frameworks for adoption, adaptation, and de novo development of trustworthy recommendations
Joanna Briggs Institute
National Institute for Health and Care Excellence
Policy Evaluation Network
Preferred Reporting Items for Systematic reviews and Meta-Analyses
Randomized controlled trials
Risk of bias in non-randomized studies of interventions
World Cancer Research Fund International
World Health Organization
Stanaway JD, Afshin A, Gakidou E, Lim SS, Abate D, Abate KH, et al. Global, regional, and national comparative risk assessment of 84 behavioural, environmental and occupational, and metabolic risks or clusters of risks for 195 countries and territories, 1990–2017: a systematic analysis for the Global Burden of Disease Study 2017. Lancet. 2018;392(10159):1923–94.
Global, regional, and national comparative risk assessment of 79 behavioural, environmental and occupational, and metabolic risks or clusters of risks, 1990-2015: a systematic analysis for the Global Burden of Disease Study 2015. Lancet (London, England). 2016;388(10053):1659–724.
Ezzati M, Riboli E. Behavioral and dietary risk factors for noncommunicable diseases. N Engl J Med. 2013;369(10):954–64.
Afshin A, Sur PJ, Fay KA, Cornaby L, Ferrara G, Salama JS, et al. Health effects of dietary risks in 195 countries, 1990–2017: a systematic analysis for the Global Burden of Disease Study 2017. Lancet. 2019;393(10184):1958–72.
Woods C, Mutrie N. Putting Physical Activity on the Policy Agenda; 2012. p. 92–104.
Lakerveld J, Woods C, Hebestreit A, Brenner H, Flechtner-Mors M, Harrington JM, et al. Advancing the evidence base for public policies impacting on dietary behaviour, physical activity and sedentary behaviour in Europe: The Policy Evaluation Network promoting a multidisciplinary approach. Food Policy. 2020;101873.
World Health Organization; Health Policy [Available from: http://www.who.int/topics/health_policy/en/]. Accessed 9 Dec 2019.
de Leeuw E, Clavier C, Breton E. Health policy--why research it and how: health political science. Health Res Policy Syst. 2014;12:55.
Brug J, van der Ploeg HP, Loyen A, Ahrens W, Allais O, Andersen LF, et al. Determinants of diet and physical activity (DEDIPAC): a summary of findings. Int J Behav Nutr Phys Act. 2017;14(1):150.
Puggina A, Aleksovska K, Buck C, Burns C, Cardon G, Carlin A, et al. Policy determinants of physical activity across the life course: a ‘DEDIPAC’ umbrella systematic literature review. Eur J Pub Health. 2018;28(1):105–18.
PEN. Effectiveness of existing policies for lifestyle interventions - Policy Evaluation Network (PEN) 2017 [Available from: https://www.jpi-pen.eu/]. Accessed 9 Dec 2019.
WHO Handbook of guideline development. 2014. [Available from: https://apps.who.int/medicinedocs/en/m/abstract/Js22083en/]. Accessed 9 Dec 2019.
Qaseem A, Forland F, Macbeth F, Ollenschläger G, Phillips S, van der Wees P, et al. Guidelines International Network: Toward International Standards for Clinical Practice Guidelines. Ann Intern Med. 2012;156(7):525–31.
Guyatt G, Oxman AD, Akl EA, Kunz R, Vist G, Brozek J, et al. GRADE guidelines: 1. Introduction-GRADE evidence profiles and summary of findings tables. J Clin Epidemiol. 2011;64(4):383–94.
Alonso-Coello P, Schünemann HJ, Moberg J, Brignardello-Petersen R, Akl EA, Davoli M, et al. GRADE Evidence to Decision (EtD) frameworks: a systematic and transparent approach to making well informed healthcare choices. 1: Introduction. BMJ (Clinical research ed). 2016;353:i2016–i.
Guyatt GH, Oxman AD, Vist GE, Kunz R, Falck-Ytter Y, Alonso-Coello P, et al. GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ (Clinical research ed). 2008;336(7650):924–6.
Moberg J, Oxman AD, Rosenbaum S, Schunemann HJ, Guyatt G, Flottorp S, et al. The GRADE Evidence to Decision (EtD) framework for health system and public health decisions. Health Res Policy Syst. 2018;16(1):45.
Goldet G, Howick J. Understanding GRADE: an introduction. J Evid Based Med. 2013;6(1):50–4.
The Joanna Briggs Institute. Joanna Briggs Institute Reviewers’ Manual: 2015 edition / Supplement, 2015.
Tricco AC, Lillie E, Zarin W, O’Brien KK, Colquhoun H, Levac D, et al. PRISMA Extension for Scoping Reviews (PRISMA-ScR): checklist and explanation. Ann Intern Med. 2018;169(7):467–73.
The NOURISHING-database; World Cancer Research Fund International (WCRF) [Available from: https://www.wcrf.org/int/policy/nourishing-database.
Balshem H, Helfand M, Schunemann HJ, Oxman AD, Kunz R, Brozek J, et al. GRADE guidelines: 3. Rating the quality of evidence. J Clin Epidemiol. 2011;64(4):401–6.
Al-Khudairy L, Loveman E, Colquitt JL, Mead E, Johnson RE, Fraser H, et al. Diet, physical activity and behavioural interventions for the treatment of overweight or obese adolescents aged 12 to 17 years. Cochrane Database Syst Rev. 2017;6:Cd012691.
Baker PRA, Francis DP, Soares J, Weightman AL, Foster C. Community wide interventions for increasing physical activity. Cochrane Database Syst Rev. 2015;1.
Balogun OO, O’Sullivan EJ, McFadden A, Ota E, Gavine A, Garner CD, et al. Interventions for promoting the initiation of breastfeeding. Cochrane Database Syst Rev. 2016;11.
Carducci B, Oh C, Keats EC, Gaffey MF, Roth DE, Zulfiqar AB. Protocol: Impact of the food environment on diet-related health outcomes in school-age children and adolescents in low- and middle-income countries: a systematic review. Campbell Collaboration. 2018.
Crockett RA, King SE, Marteau TM, Prevost AT, Bignardi G, Roberts NW, et al. Nutritional labelling for healthier food or non-alcoholic drink purchasing and consumption. Cochrane Database Syst Rev. 2018;2.
Cushing CC, Brannon EE, Suorsa KI, Wilson DK. Systematic review and meta-analysis of health promotion interventions for children and adolescents using an ecological framework. J Pediatr Psychol. 2014;39(8):949–62.
Dobbins M, Husson H, DeCorby K, LaRocca RL. School-based physical activity programs for promoting physical activity and fitness in children and adolescents aged 6 to 18. Cochrane Database Syst Rev. 2013;2.
Dyson PA, Twenefour D, Breen C, Duncan A, Elvin E, Goff L, et al. Diabetes UK evidence-based nutrition guidelines for the prevention and management of diabetes. Diabet Med. 2018;35(5):541–7.
Elvsaas IKO, Giske L, Fure B, Juvet LK. Multicomponent lifestyle interventions for treating overweight and obesity in children and adolescents: a systematic review and meta-analyses. J Obes. 2017;2017:5021902.
Erickson J, Sadeghirad B, Lytvyn L, Slavin J, Johnston BC. The scientific basis of guideline recommendations on sugar intake: a systematic review. Ann Intern Med. 2017;166(4):257–67.
Flatz A, Pfeifer N, Radtke T, Kriemler S, Klerings I, Wolfenden L, et al. Interventions implemented through sporting organisations for promoting healthy behaviour or improving health outcomes. Cochrane Database Syst Rev. 2016;5.
Freak-Poli RL, Cumpston M, Peeters A, Clemes SA. Workplace pedometer interventions for increasing physical activity. Cochrane Database Syst Rev. 2013;4:CD009209.
Heise TL, Katikireddi SV, Pega F, Gartlehner G, Fenton C, Griebler U, et al. Taxation of sugar-sweetened beverages for reducing their consumption and preventing obesity or other adverse health outcomes. Cochrane Database Syst Rev. 2016;8.
Heise TL, Romppel M, Molnar S, Buchberger B, Berg A, Gartlehner G, et al. Community gardening, community farming and other local community-based gardening interventions to prevent overweight and obesity in high-income and middle-income countries: protocol for a systematic review. BMJ Open. 2017;7(6):e016237.
Hodder RK, Stacey FG, Wyse RJ, O’Brien KM, Clinton-McHarg T, Tzelepis F, et al. Interventions for increasing fruit and vegetable consumption in children aged five years and under. Cochrane Database Syst Rev. 2017;9.
Hollands GJ, Shemilt I, Marteau TM, Jebb SA, Lewis HB, Wei Y, et al. Portion, package or tableware size for changing selection and consumption of food, alcohol and tobacco. Cochrane Database Syst Rev. 2015;2017(3) (no pagination).
Langford R, Bonell CP, Jones HE, Pouliou T, Murphy SM, Waters E, et al. The WHO Health Promoting School framework for improving the health and well-being of students and their academic achievement. Cochrane Database Syst Rev. 2014;4.
Lhachimi SK, Pega F, Heise TL, Fenton C, Gartlehner G, Griebler U, et al. Taxation of the fat content of foods for reducing their consumption and preventing obesity or other adverse health outcomes. Cochrane Database Syst Rev. 2016;10.
Martin A, Fitzsimons C, Jepson R, Saunders DH, van der Ploeg HP, Teixeira PJ, et al. Interventions with potential to reduce sedentary time in adults: systematic review and meta-analysis. Br J Sports Med. 2015;49(16):1056–63.
Matwiejczyk L, Mehta K, Scott J, Tonkin E, Coveney J. Characteristics of effective interventions promoting healthy eating for pre-schoolers in childcare settings: an umbrella review. Nutrients. 2018;10(3):01.
McLaren L, Sumar N, Barberio AM, Trieu K, Lorenzetti DL, Tarasuk V, et al. Population-level interventions in government jurisdictions for dietary sodium reduction. Cochrane Database Syst Rev. 2016;9:CD010166.
Mosdol A, Lidal IB, Straumann GH, Vist GE. Targeted mass media interventions promoting healthy behaviours to reduce risk of non-communicable diseases in adult, ethnic minorities. Cochrane Database Syst Rev. 2017;2:CD011683.
NICE guideline. Physical activity and the environment March 2018 [Available from: https://www.nice.org.uk/guidance/ng90]. Accessed 25 Nov 2019.
Oakman J, Neupane S, Proper KI, Kinsman N, Nygard CH. Workplace interventions to improve work ability: a systematic review and meta-analysis of their effectiveness. Scand J Work Environ Health. 2018;44(2):134–46.
Okely AD, Ghersi D, Hesketh KD, Santos R, Loughran SP, Cliff DP, et al. A collaborative approach to adopting/adapting guidelines - The Australian 24-Hour Movement Guidelines for the early years (Birth to 5 years): an integration of physical activity, sedentary behavior, and sleep. BMC Public Health. 2017;17(Suppl 5):869.
Pfinder M, Katikireddi SV, Pega F, Gartlehner G, Fenton C, Griebler U, et al. Taxation of unprocessed sugar or sugar-added foods for reducing their consumption and preventing obesity or other adverse health outcomes. Cochrane Database Syst Rev. 2016;8.
von Philipsborn P, Stratil JM, Burns J, Busert LK, Pfadenhauer LM, Polus S, et al. Environmental interventions to reduce the consumption of sugar-sweetened beverages and their effects on health. Cochrane Database Syst Rev. 2019;6(6):Cd012292.
Salam RA, Das JK, Hoodbhoy Z, Nathani KR, Z.A. B. Protocol: Effects of lifestyle modification interventions to prevent and manage child and adolescent obesity: a systematic review. Campbell Collaboration. 2018.
Shrestha N, Kukkonen-Harjula KT, Verbeek JH, Ijaz S, Hermans V, Pedisic Z. Workplace interventions for reducing sitting at work. Cochrane Database Syst Rev. 2018;6:Cd010912.
Tremblay MS, Leblanc AG, Carson V, Choquette L, Connor Gorber S, Dillman C, et al. Canadian Physical Activity Guidelines for the Early Years (aged 0-4 years), Applied physiology, nutrition, and metabolism = Physiologie appliquee, nutrition et metabolisme. 2012;37(2):345–69.
Tremblay MS, Carson V, Chaput JP, Connor Gorber S, Dinh T, Duggan M, et al. Canadian 24-Hour Movement Guidelines for Children and Youth: An Integration of Physical Activity, Sedentary Behaviour, and Sleep. Appl Physiol Nutr Metab. 2016;41(6 Suppl 3):S311–27.
Verweij LM, Coffeng J, van Mechelen W, Proper KI. Meta-analyses of workplace physical activity and dietary behaviour interventions on weight outcomes. Obes Rev. 2011;12(6):406–29.
WHO Guidelines Approved by the Guidelines Review Committee. Guideline: assessing and managing children at primary health-care facilities to prevent overweight and obesity in the context of the double burden of malnutrition: updates for the Integrated Management of Childhood Illness (IMCI). Geneva: World Health Organization Copyright (c) World Health Organization; 2017.
Guideline. Potassium Intake for Adults and Children. Geneva: World Health Organization; 2012. Recommendations and remarks. [Available from: https://www.ncbi.nlm.nih.gov/books/NBK132453/?report=classic]. Accessed 25 Nov 2019.
Guideline. Sodium Intake for Adults and Children. Geneva: World Health Organization; 2012. [Available from: https://www.ncbi.nlm.nih.gov/books/NBK133309/]. Accessed 25 Nov 2019.
Guideline. Sugars Intake for Adults and Children. Geneva: World Health Organization; 2015. [Available from: https://www.ncbi.nlm.nih.gov/books/NBK285537/]. Accessed 25 Nov 2019.
World Health Organization. Countries of Europe [Available from: http://www.euro.who.int/en/countries].
Schwingshackl L, Knuppel S, Schwedhelm C, Hoffmann G, Missbach B, Stelmach-Mardas M, et al. Perspective: NutriGrade: a scoring system to assess and judge the meta-evidence of randomized controlled trials and cohort studies in nutrition research. Adv Nutr (Bethesda, Md). 2016;7(6):994–1004.
JPT H, S G, editors. Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011]. The Cochrane Collaboration; 2011.
Schünemann HJ, Fretheim A, Oxman AD. Improving the use of research evidence in guideline development: 9. Grading evidence and recommendations. Health Res Policy Syst. 2006;4:21.
Schunemann HJ, Cuello C, Akl EA, Mustafa RA, Meerpohl JJ, Thayer K, et al. GRADE guidelines: 18. How ROBINS-I and other tools to assess risk of bias in nonrandomized studies should be used to rate the certainty of a body of evidence. J Clin Epidemiol. 2018.
Sterne JA, Hernan MA, Reeves BC, Savovic J, Berkman ND, Viswanathan M, et al. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ (Clinical research ed). 2016;355:i4919.
Maki KC, Slavin JL, Rains TM, Kris-Etherton PM. Limitations of observational evidence: implications for evidence-based dietary recommendations. Adv Nutr (Bethesda, Md). 2014;5(1):7–15.
Shim J-S, Oh K, Kim HC. Dietary assessment methods in epidemiologic studies. Epidemiol Health. 2014;36:e2014009–e.
Saltelli A, Giampietro M. What is wrong with evidence based policy, and how can it be improved? Futures. 2017;91:62–71.
Moat KA, Lavis JN, Abelson J. How contexts and issues influence the use of policy-relevant research syntheses: a critical interpretive synthesis. Milbank Q. 2013;91(3):604–48.
Liverani M, Hawkins B, Parkhurst JO, et al. PLoS One. 2013;8(10):e77404–e.
LaRocca R, Yost J, Dobbins M, Ciliska D, Butt M. The effectiveness of knowledge translation strategies used in public health: a systematic review. BMC Public Health. 2012;12:751.
Busert LK, Mutsch M, Kien C, Flatz A, Griebler U, Wildner M, et al. Facilitating evidence uptake: development and user testing of a systematic review summary format to inform public health decision-making in German-speaking countries. Health Res Policy Syst. 2018;16(1):59.
Petkovic J, Welch V, Jacob MH, Yoganathan M, Ayala AP, Cunningham H, et al. The effectiveness of evidence summaries on health policymakers and health system managers use of evidence from systematic reviews: a systematic review. Implement Sci. 2016;11(1):162.
Hawe P, Shiell A, Riley T. Theorising interventions as events in systems. Am J Community Psychol. 2009;43(3-4):267–76.
Lewin S, Hendry M, Chandler J, Oxman AD, Michie S, Shepperd S, et al. Assessing the complexity of interventions within systematic reviews: development, content and use of a new tool (iCAT_SR). BMC Med Res Methodol. 2017;17(1):76.
Petticrew M, Knai C, Thomas J, Rehfuess EA, Noyes J, Gerhardus A, et al. Implications of a complexity perspective for systematic reviews and guideline development in health decision making. BMJ Glob Health. 2019;4(Suppl 1):e000899.
Montgomery P, Movsisyan A, Grant SP, Macdonald G, Rehfuess EA. Considerations of complexity in rating certainty of evidence in systematic reviews: a primer on using the GRADE approach in global health. BMJ Glob Health. 2019;4(Suppl 1):e000848.
Rehfuess EA, Akl EA. Current experience with applying the GRADE approach to public health interventions: an empirical study. BMC Public Health. 2013;13:9.
National Institute for Health and Care Excellence (NICE). Developing NICE guidelines: the manual [Available from: https://www.nice.org.uk/process/pmg20/chapter/introduction-and-overview]. Accessed 9 Dec 2019.
Blake P, Durão S, Naude CE, Bero L. An analysis of methods used to synthesize evidence and grade recommendations in food-based dietary guidelines. Nutr Rev. 2018;76(4):290–300.
Schwingshackl L, Knuppel S, Schwedhelm C, Hoffmann G, Missbach B, Stelmach-Mardas M, et al. Reply to JJ Meerpohl et al. Adv Nutr (Bethesda, Md). 2017;8(5):790–1.
Green LW, Glasgow RE. Evaluating the relevance, generalization, and applicability of research: issues in external validation and translation methodology. Eval Health Prof. 2006;29(1):126–53.
Guindo LA, Wagner M, Baltussen R, Rindress D, van Til J, Kind P, et al. From efficacy to equity: literature review of decision criteria for resource allocation and healthcare decisionmaking. Cost Effect Resource Alloc. 2012;10(1):9.
Gu Y, Lancsar E, Ghijben P, Butler JR, Donaldson C. Attributes and weights in health care priority setting: a systematic review of what counts and to what extent. Soc Sci Med. 2015;146:41–52.
Stratil J, Baltussen R, Scheel I, Nacken A, Rehfuess E. Development of the WHO-INTEGRATE evidence-to-decision framework: an overview of systematic reviews of decision criteria for health decision-making. Cost Effectiveness and Resource Allocation. forthcoming.
Youngkong S, Kapiriri L, Baltussen R. Setting priorities for health interventions in developing countries: a review of empirical studies. Tropical Med Int Health. 2009;14(8):930–9.
Elliott H, Popay J. How are policy makers using evidence? Models of research utilisation and local NHS policy making. J Epidemiol Community Health. 2000;54(6):461–8.
We thank Eva Rehfuess and Peter von Philipsborn from the LMU Munich for their valuable contributions to the manuscript. Furthermore, we thank the information specialist Edith Motschall from the University of Freiburg for her help and expertise in the development of the search strategy.
The PEN project is funded by the Joint Programming Initiative “A Healthy Diet for a Healthy Life” (JPI HDHL), a research and innovation initiative of EU member states and associated countries. The funding agencies supporting this work are (in alphabetical order of participating countries) France: Institut National de la Recherche Agronomique (INRA); Germany: Federal Ministry of Education and Research (BMBF); Ireland: Health Research Board (HRB); Italy: Ministry of Education, University and Research (MIUR); The Netherlands: The Netherlands Organisation for Health Research and Development (ZonMw); New Zealand: The University of Auckland, School of Population Health; Norway: The Research Council of Norway (RCN); and Poland: The National Centre for Research and Development (NCBR).
Ethics approval and consent to participate
Consent for publication
JZ: No conflict of interest
LS: Member of GRADE Working Group
AM: Member of GRADE Working Group
JMSt: Developer of the WHO-INTEGRATE EtD framework
CS: No conflict of interest
JMS: No conflict of interest
SF: No conflict of interest
WA: No conflict of interest
DKG: Member of GRADE Working Group
HJS: Co-chair of the GRADE working group; Director, McMaster GRADE Centre
JJM: Member of GRADE Working Group; Director, Freiburg GRADE Center
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Preferred Reporting Items for Systematic reviews and Meta-Analyses extension for Scoping Reviews (PRISMA-ScR).
Search in MEDLINE via Ovid, 4 July 2019.
Survey among European health policymakers.
WHO – Strengths and recommendations according to GRADE.
About this article
Cite this article
Zähringer, J., Schwingshackl, L., Movsisyan, A. et al. Use of the GRADE approach in health policymaking and evaluation: a scoping review of nutrition and physical activity policies. Implementation Sci 15, 37 (2020). https://doi.org/10.1186/s13012-020-00984-2