- Open Access
- Open Peer Review
The GRADE evidence-to-decision framework: a report of its testing and application in 15 international guideline panels
- Ignacio Neumann1, 2,
- Romina Brignardello-Petersen1, 3,
- Wojtek Wiercioch1,
- Alonso Carrasco-Labra1, 3,
- Carlos Cuello1,
- Elie Akl4,
- Reem A. Mustafa1, 5,
- Waleed Al-Hazzani1,
- Itziar Etxeandia-Ikobaltzeta1, 7,
- Maria Ximena Rojas8,
- Maicon Falavigna9,
- Nancy Santesso1,
- Jan Brozek1, 6,
- Alfonso Iorio1,
- Pablo Alonso-Coello1, 10 and
- Holger J. Schünemann1, 6Email author
© Neumann et al. 2016
- Received: 21 December 2015
- Accepted: 20 June 2016
- Published: 15 July 2016
Judgments underlying guideline recommendations are seldom recorded and presented in a systematic fashion. The GRADE Evidence-to-Decision Framework (EtD) offers a transparent way to record and report guideline developers’ judgments. In this paper, we report the experiences with the EtD frameworks in 15 real guideline panels.
Following the guideline panel meetings, we asked methodologists participating in the panel to provide feedback regarding the EtD framework. They were instructed to consider their own experience and the feedback collected from the rest of the panel. Two investigators independently summarized the responses and jointly interpreted the data using pre-specified domains as coding system. We asked methodologists to review the results and provide further input to improve the structure of the EtDs iteratively.
The EtD framework was well received, and the comments were generally positive. Methodologists felt that in a real guideline panel, the EtD framework helps structuring a complex process through relatively simple steps in an explicit and transparent way. However, some sections (e.g., “values and preferences” and “balance between benefits and harms”) required further development and clarification that were considered in the current version of the EtD framework.
The use of an EtD framework in guideline development offers a structured and explicit way to record and report the judgments and discussion of guideline panels during the formulation of recommendations. In addition, it facilitates the formulation of recommendations, assessment of their strength, and identifying gaps in research.
- Clinical practice guidelines
- Evidence to decisions framework
Clinical practice guidelines are an efficient way to bridge the gap between research evidence, expert experience, and decision-making [1–4]. In order to be trustworthy, guidelines need to be explicit regarding the methods used to summarize the evidence and rate its certainty (also known as quality of the evidence or confidence in the effect estimates) and about the judgments involved in moving from evidence to recommendations [5–7]. Unfortunately, such judgments are rarely recorded and presented in a systematic way.
The Grading of Recommendations Assessment, Development and Evaluation (GRADE) working group (www.gradeworkinggroup.org), a collaborative of over 500 scientists, clinicians, and people with other backgrounds, has developed an approach to assessing the certainty in the body of evidence summarized in systematic reviews that support a decision or guideline recommendation, called the GRADE approach [8, 9]. This approach is used by over 90 organizations, including the World Health Organization, the National Institutes of Health and Care Excellence, the Canadian Task Force for the Preventive Services, numerous professional organizations, and the Cochrane Collaboration. There are over 20,000 citations to GRADE’s methodological work, and thousands of recommendations have been developed following the GRADE approach. GRADE can be used to evaluate different types of evidence, including evidence for intervention effects (including multiple treatment comparisons), test accuracy, prognosis, and resources.
In the context of its “Developing and Evaluating Communication Strategies to Support Informed Decisions and Practice Based on Evidence (DECIDE)” project, GRADE has developed strategies for the targeted communication of evidence-based recommendations to different stakeholders . One of such strategies is the GRADE Evidence to Decision (EtD) Framework, meant to structure development of both recommendations (e.g., clinical recommendations) and decisions (e.g., coverage or public health decisions) using the GRADE approach. Each EtD framework includes detailed sections describing (a) the question and background, (b) an assessment of the evidence, and (c) the conclusions. A summary of the effects of alternative management strategies on patient-important outcomes, data about patients’ values and preferences, and information about resource utilization are part of the assessment section [11–13]. The EtD framework also features information regarding acceptability and feasibility of the alternative management strategies, as well their impact on health equity. This data are obtained from current systematic reviews when available or, more often, updating of previous reviews or the de novo systematic reviews. Finally, The EtD framework provides an explicit record of the judgments made by guideline panelists that ultimately determine the direction and strength of recommendations or decisions. In this paper, we report on the first experience with the EtD framework for clinical recommendations in real guideline panels.
The EtD frameworks
Evolution of the EtD framework
Used on the World Health Organization and Spanish Guidelines program
Used in Kingdom of Saudi Arabia, World Allergy Organization, Colombian guideline program and rare diseases guidelines
Did not include “prioritization of the problem”
Included a question evaluating the priority of the problem
Included a single question evaluating the balance between health benefits and harms of the intervention of interest
The original question was disaggregated into 3 questions: (1) the magnitude of the health benefits, (2) the magnitude of the health harms, and (3) the balance between health benefits and harms.
Included a single question evaluating the variability of patients’ values and preferences
A judgment regarding the uncertainty about patients’ values and preferences was added.
Included a single question evaluating the cost of the intervention in relation to the benefits
The original question was disaggregated into 2 questions: (1) what is the cost of the intervention and (2) how large is this cost in relation to the benefits.
Included equity as an additional criterion
Acceptability and feasibility were added to equity
Guideline panels on which the GRADE Evidence to Decision (EtD) Framework was tested
Kingdom of Saudi Arabia Guideline Programa
1. Management of allergic rhinitis in adults and children
2. Antithrombotic Treatment of Patients with Non-valvular Atrial Fibrillation
3. Use of Screening Strategies for Detection of Breast Cancer (used interactive EtD in GRADEpro during panel meeting)
4. Screening and Treatment of Precancerous Lesions for Cervical Cancer Prevention
5. Diagnosis of Suspected First Deep Vein Thrombosis of Lower Extremity
6. Timing of Initiation of Dialysis
7. Role of Vitamin D, Calcium and Exercise in Fracture Prevention in Elderly
8. Use of Thrombolytic Therapy in Acute Stroke
9. Prevention of Venous Thromboembolism in Stroke
10. Treatment of Venous Thromboembolism
World Health Organization Guideline Programb
11. The use of bedaquiline in the treatment of multidrug resistant tuberculosis (used interactive EtD in GRADEpro during panel meeting)
WAO Guideline Programc
12. Guidelines for allergic diseases prevention (GLAD-P) (used interactive EtD in GRADEpro during panel meeting)
Colombian Guidelines programd
13. Clinical practice guidelines for evidence-based care of HIV infection in children
Spanish Guidelines programe
14. Spanish Guidelines program: Childhood asthma
15. Hemophilia (used interactive EtD in GRADEpro during panel meeting)
We will describe its main features of the EtD here (Additional file 1: Table S1). The header of the table provides details about the clinical question, clearly outlining its components following the PICO framework and any relevant background information. The first column offers the factors or criteria being evaluated framed as questions (e.g., is the problem a priority?). The second column identifies the required judgments by panels and allows recording of these judgments for each criterion. The relevant research evidence and additional considerations upon which the judgments are based redocumented in third column and fourth column, respectively. While the research evidence should be based on systematic reviews of the literature, additional considerations allow capturing specific considerations and information about the local context in which the recommendation will be applied. Finally, the framework captures the judgments leading to the direction and strength of the recommendation or decision. In addition, there is free space to include subgroup and implementation considerations, as well as information regarding futures updates, monitoring, and research priorities. In the four guidelines that used GRADEpro, GRADE’s online tool to assess evidence and develop recommendations, the information was shown on a large screen during the panel meeting and judgments and additional considerations were added online to the EtD during the panel meetings . The version of the EtD in GRADEpro corresponded to the stage of the EtD development (shown in Additional file 1: Table S1) while the current version of GRADEpro that is available on the web (www.gradepro.org) includes the fully developed static and interactive EtD.
We used the EtD frameworks in 15 international guideline panels (Table 2). Two guideline panels were organized by international agencies (World Health Organization and World Allergy Organization); 12 panels were in the context of national guideline programs (10 in the Kingdom of Saudi Arabia, 1 in Colombia, and 1 in Spain); and 1 panel was part of an effort to produce establish methods for developing guidance on rare diseases (Italy and Germany).
Summary of the results
Section in the EtD table
Order of columns and rows
The order was not a relevant issue for most panels
Criteria in EtD
One methodologist expressed concerns regarding the overlap between the sections “acceptability” and “feasibility” with “values and preferences” and “resource considerations”.
Another methodologist considered the section “priority” redundant.
According to another methodologist, the criteria of the table were not easy to apply to recommendations addressing multiple comparisons.
Two methodologists stated that allowing modifications to the answer options might be needed to accommodate different contexts and scenarios.
Although for the majority of panels there were no issues with this column, one methodologist stated that the difference between the purposes of the columns “available evidence” and “additional consideration” was not clear.
One methodologist suggested using this column to summarize the available evidence as a general narrative statement.
One methodologist considered this section redundant (with respect to information that is present in the main text of the guideline).
Three methodologists expressed concerns regarding how the question is presented, specifically, they suggested making the PICO structure more explicit (and using the exact terms that P-I-C-O stand for).
One methodologist considered this section particularly relevant, as being explicit about the perspective may help to make transparent the decisions made to formulate recommendations.
Overall certainty of the Evidence
In general, this section was well evaluated by methodologists, with no major suggestion for improvement. Two methodologists made minor wording suggestions.
Another methodologist suggested expanding the content to include more information about each particular outcome.
Values and preferences (“Uncertainty about how much people value the main outcomes”)
This section posed significant difficulties in several panels. Four methodologists suggested differentiating between “variability” and “uncertainty” of patients’ values and preferences.
Another methodologist suggested making explicit the source of patients’ values and preferences. The order of presenting and discussing values and preferences was also questioned and subsequently changed in the EtD framework.
Balance of benefits and harms
There were major difficulties in 6 guideline panels. According to methodologists, panelists had problems answering consistently the questions about the size of the effect.
Additionally, two methodologists considered the questions of this section redundant.
Three guideline panels (all without health economists) struggled answering the question about the relationship between incremental cost and benefits. These panels proposed “is the treatment cost-effective?” as a better alternative. The only guideline panel with health economists considered the questions of this section too superficial.
Three guideline panels struggled with this question. A more clear definition of health equity and more guidance on how to answer the question were considered necessary.
Two methodologists suggested adding the option “no effect on health equity” to the answers options.
Two methodologists expressed problems when trying to identify the relevant stakeholders. More guidance was considered necessary.
One methodologist considered the wording of this section confusing when the recommendation under discussion involved two active treatments.
Another methodologist suggested de-emphasizing the option “no recommendation”
Two panels struggled to decide what to include in the remarks. More guidance was considered necessary.
One methodologist considered that subgroups should be more explicit in the table.
According to methodologists, this section was used in different ways across guideline panels. More guidance in what to include was considered necessary.
Monitoring and evaluation
In general, there was agreement regarding the importance of this section.
Two investigators (IN and RBP) independently identified themes from the feedback obtained from the methodologists. They used the pre-specified domains as coding system. Then, the two investigators jointly interpreted the data and organized the themes into the following categories: general comments about the framework and comments about specific sections within the framework. All participants were invited to review and comment on a draft report of the results.
Context and methodologists
The topics covered by guideline panels were diverse, including cardiovascular diseases (5 guidelines), asthma and allergy (3 guidelines), infectious diseases (2 guidelines), cancer screening and diagnosis (2 guidelines), and others (3 guidelines). Most of the guidelines were focused on adults (11 guidelines) (Table 2).
Ten methodologists led the guideline development on the 15 panels, and for all guidelines, a second methodologist provided backup support. Seven of the panel leads were physicians, two dentists, and one registered nurse, all of them with postgraduate training in health research methods or a related discipline. All guideline methodologists had formal training and experience in conducting systematic reviews and in using the GRADE approach to formulate recommendations. They were all members of the GRADE working group. For 13 out of the 15 guidelines, one highly experienced guideline methodologist led the panel discussions supported by another guideline methodologist. Most of the highly experienced methodologists had led multiple systematic reviews and high level guideline panels at major international organizations (e.g., WHO) or professional societies with large guideline programs (e.g., American Thoracic Society or American College of Chest Physicians). The gender distribution was six males and four females, and the mean age was 37.8 years (standard deviation 6.3). All of them provided feedback about the EtD framework.
General comments about the EtD framework
Table 3 summarizes the results by themes. In general, the EtD framework was well received and the comments were mostly positive. The methodologists felt that in a real guideline panel, the EtD framework helps structuring a complex group process through relatively simple steps in an explicit and transparent way. Further, two methodologists commented that the explicitness of the framework could help having an accurate idea of the underlying evidence and protecting against inappropriate recommendations.
Two methodologists stated that the wording of the questions and options to answer were suboptimal for recommendations comparing two active interventions (instead of one active intervention against placebo/no treatment). Also, three methodologists raised concerns regarding the length of the framework and the relevance of some sections in specific circumstances. In particular, considerations regarding feasibility, accessibility, and equity were considered less relevant in the context of clinical recommendations. Finally, two methodologists stated that in order to take full advantage of the ETD framework, it is necessary to be familiar with the GRADE approach and with the framework itself. Therefore, for new users, previous training may be helpful for successful implementation and use. For example, one methodologist stated, “Until you see the process you don’t appreciate all the real work and value of the forms and filling out information before the panel meeting, etc. You must go through the process and use the [EtD] and see the outcome.”
Comments about specific sections
The sections “values and preferences” and “balance of benefits and harms” posed difficulties in several panels. Regarding the “values and preferences” section, the GRADE approach requires guideline panelists to judge whether there is important variability in how patients’ value the main outcomes and to what extent there is important uncertainty about this. In case of any of the two, panels should consider issuing a weak recommendation. The main concern with this question was that the answer options did not differentiate between the variability and uncertainty in patients’ values and preferences, creating some confusion among panelists. Four methodologists suggested dividing this question in two.
Regarding the balance between benefits and harms, this section features three questions requiring panelists to judge the magnitude of desirable and undesirable effects, as well the relation between the two. Panelists struggled judging the size of the benefits and harms consistently. Two methodologists considered the three questions redundant and suggested to drop the questions about the size of the benefit and harms and maintain only the one addressing their balance.
This is the first study evaluating the use of the GRADE EtD framework in the context of real guideline development. Given the widespread application of GRADE, this study provides important information and rationale for utilizing EtDs in guidelines. In general, the response to the use of the EtD framework was positive, although some sections (i.e., “values and preferences” and “balance between benefits and harms”) required improvements, in particular the choice of answer options. These improvements have been implemented in the current final option of GRADE EtD frameworks (www.gradepro.org) [11–13].
There are some limitations to this study. The use of iteratively developed formats of the EtD framework in different guideline panels could have introduced some variability in the provided answers. However, it was necessary as part of the development and improvement of the EtDs. For example, early testing with the World Health Organization and World Allergy Organization panels revealed difficulties with providing responses to the available answer options. In order to make best use of the EtDs, they were modified for use in the work of later guideline panels. While methodologists were prospectively instructed before the panel meetings to record challenges with the EtDs during the panel meetings, there was no independent evaluation of challenges with the EtD frameworks. However, the data were recorded without delay and we believe that recall of the encountered issues does not lead to bias as all methodologists had an interest in improving the frameworks.
The strengths of this study include the strategies taken to ensure the trustworthiness of our findings: prospective data collection, independent and duplicate analysis, and member checking of the conclusions. The most important strength of this work is the application of the EtD frameworks in the context of real guideline development by international panels in a broad range of topics, including diagnostic and therapeutic questions.
Our findings strongly suggest that the EtD framework can enhance the transparency of guideline developers’ judgments, allowing decision makers to truly assess the recommendation. This is not only corroborated by this work but by the adoption of the EtD approach by major organizations such as WHO . What happens within guideline panels is seldom reported in current guidelines [22–26]. Hence, when recommendations are analyzed at a deeper level, some decisions of the panel might seem arbitrary and unjustified. Providing details about the reasons and judgments behind such decisions enrich the recommendations and also may facilitate the adaptation of international guidelines to local contexts. In the EtDs, each criterion is considered by working trough the EtD sequentially and then in a summary before formulating a recommendation. This avoids unnecessarily moving back and forth between criteria and wasting time. In addition, each criterion requires a judgment that determines the direction and strength of a recommendation, which is often not found in guidelines. This information will facilitate updating (by seeing judgments and evaluating if the should change) and adaptation of guidelines (by making others aware of the judgments).
The EtD framework also offers the opportunity to identify and potentially avoid inappropriate recommendations. One potential problem of recommendations is the mismatch between the strength of the recommendation and the underlying information about the certainty in effect estimates (also known as confidence in the evidence or quality of the evidence), the balance between the benefits and harms, values and preferences, and resource considerations . The EtD framework, on the one hand, allows users to judge to what extent the recommendation corresponds to the underlying information and, on the other hand, may help preventing guideline developers from inappropriately grading the strength of the recommendation . Finally, the EtD framework may also be a valuable research tool, since it would allow a detailed comparison of conflicting recommendations from different organizations.
In conclusion, the use of an EtD framework in guideline development offers an explicit and transparent way to record and report the judgments and discussion of guideline panelists. The EtD framework may facilitate the assessment of recommendations and also their potential adaptation and implementation. As result of this work, a new section was added to the EtD framework. This section allows summarizing the judgments of guideline panelists just prior to the deliberations regarding the direction and strength of recommendations. The currently latest version of the EtD resulting from this and other work including an interactive version of the EtD based on this and other work is available in GRADE’s online GRADEpro Guideline Development Tool (www.gradepro.org) and described elsewhere [11–13]. While the current formats of EtD frameworks are already used widely, future use and research will inform changes that will have to be made to the EtDs. To name some areas of relevance, additional research could identify how to best integrate issues arising from potential conflicts of interest, how to structure research recommendations, how to address agreement of panel members with information provided by systematic reviews, how to document alterations in judgments about the certainty in the evidence by panel members as a result of indirectness of the evidence , and how to structure implementation considerations. The GRADE working group will further evaluate and develop the EtD frameworks. Doing this, the evaluation of the EtD frameworks in real guideline development processes should continue to provide information for improvement.
We thank all panelists and methodologists of the guidelines in which this EtD framework was tested.
This project has been funded by the European Union’s Seventh Framework Programme for research, technological development and dissemination under grant agreement no. 258583 (www.decide-collaboration.eu), the Ministry of Health of the Kingdom of Saudi Arabia, and the McMaster GRADE center.
HJS, PAC and WW conceived of the study and participated in its design and coordination. IN and RBP participated in its design and conducted the analysis. ACL, CC, EA, RAM, WA, IEI, MXR, MF, NS, JB and AI provided substantial contributions to the interpretation of the findings. IN and HJS drafted the manuscript. PAC, WW, RBP, ACL, CC, EA, RAM, WA, IEI, MXR, MF, NS, JB and AI revised the manuscript critically. All authors read and approved the final manuscript.
The authors declare that they have no competing interests.
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.
- Woolf SH, Grol R, Hutchinson A, Eccles M, Grimshaw J. Clinical guidelines: potential benefits, limitations, and harms of clinical guidelines. BMJ. 1999;318(7182):527–30.View ArticlePubMedPubMed CentralGoogle Scholar
- Eccles MP, Grimshaw JM, Shekelle P, Schunemann HJ, Woolf S. Developing clinical practice guidelines: target audiences, identifying topics for guidelines, guideline group composition and functioning and conflicts of interest. Implemen Sci. 2012;7:60.View ArticleGoogle Scholar
- Shekelle P, Woolf S, Grimshaw JM, Schunemann HJ, Eccles MP. Developing clinical practice guidelines: reviewing, reporting, and publishing guidelines; updating guidelines; and the emerging issues of enhancing guideline implementability and accounting for comorbid conditions in guideline development. Implemen Sci. 2012;7:62.View ArticleGoogle Scholar
- Woolf S, Schunemann HJ, Eccles MP, Grimshaw JM, Shekelle P. Developing clinical practice guidelines: types of evidence and outcomes; values and economics, synthesis, grading, and presentation and deriving recommendations. Implemen Sci. 2012;7:61.View ArticleGoogle Scholar
- Brouwers MC, Kho ME, Browman GP, Burgers JS, Cluzeau F, Feder G, Fervers B, Graham ID, Grimshaw J, Hanna SE, et al. AGREE II: advancing guideline development, reporting and evaluation in health care. J Clin Epidemiol. 2010;63(12):1308–11.View ArticlePubMedGoogle Scholar
- Institute of Medicine (US) Committee on Standards for Developing Trustworthy Clinical Practice Guidelines; Editors: Robin Graham, Michelle Mancher, Dianne Miller Wolman, Sheldon Greenfield, and Earl Steinberg. Washington (DC): National Academies Press (US); 2011 (available at: http://www.nap.edu/openbook.php?record_id=13058)
- Schunemann HJ, Wiercioch W, Etxeandia I, Falavigna M, Santesso N, Mustafa R, Ventresca M, Brignardello-Petersen R, Laisaar KT, Kowalski S, et al. Guidelines 2.0: systematic development of a comprehensive checklist for a successful guideline enterprise. CMAJ. 2014;186(3):E123–142.View ArticlePubMedPubMed CentralGoogle Scholar
- Atkins D, Best D, Briss PA, Eccles M, Falck-Ytter Y, Flottorp S, Guyatt GH, Harbour RT, Haugh MC, Henry D, et al. Grading quality of evidence and strength of recommendations. BMJ. 2004;328(7454):1490.View ArticlePubMedGoogle Scholar
- Schünemann HJ, Best D, Vist G, Oxman AD. Letters, numbers, symbols and words: how to communicate grades of evidence and recommendations. CMAJ. 2003;169(7):677–80.PubMedPubMed CentralGoogle Scholar
- Treweek S, Oxman AD, Alderson P, Bossuyt PM, Brandt L, Brozek J, Davoli M, Flottorp S, Harbour R, Hill S, et al. Developing and Evaluating Communication Strategies to Support Informed Decisions and Practice Based on Evidence (DECIDE): protocol and preliminary results. Implement Sci. 2013;8:6.View ArticlePubMedPubMed CentralGoogle Scholar
- Schunemann HJ, Mustafa R, Brozek J, Santesso N, Alonso-Coello P, Guyatt G, Scholten R, Langendam M, Leeflang MM, Akl EA, et al. GRADE Guidelines: 16. Development of the GRADE Evidence to Decision (EtD) frameworks for tests in clinical practice and public health. J Clin Epidemiol 2016. doi: 10.1016/j.jclinepi.2016.01.032.
- Alonso-Coello P, Schunemann H, Moberg J, Brignardello-Petersen R, Akl E, Davoli M, Treweek S, Mustafa R, Rada G, Rosenbaum S, Morelli A, Guyatt GH, Oxman AD. GRADE Evidence to Decision frameworks: 1. Introduction. BMJ in press.Google Scholar
- Alonso-Coello P, Oxman AD, Moberg J, Brignardello-Petersen R, Akl E, Davoli M, Treweek S, Mustafa R, Vandvik, P, Meerpohl, J, Guyatt, GH, Schunemann H. GRADE Evidence to Decision frameworks: 2. Clinical practice guidelines. BMJ in press.Google Scholar
- World Health Organization. The use of bedaquiline in the treatment of multidrug-resistant tuberculosis. Interim policy guidance http://apps.who.int/iris/bitstream/10665/75146/1/9789241548441_eng.pdf. Accessed 8 Dec 2014. In.; 2013.
- Schunemann HJ, Hill SR, Kakad M, Bellamy R, Uyeki TM, Hayden FG, Yazdanpanah Y, Beigel J, Chotpitayasunondh T, Del Mar C, et al. WHO Rapid Advice Guidelines for pharmacological management of sporadic human infection with avian influenza A (H5N1) virus. Lancet Infect Dis. 2007;7(1):21–31.View ArticlePubMedGoogle Scholar
- Schunemann HJ, Hill SR, Kakad M, Vist GE, Bellamy R, Stockman L, Wisloff TF, Del Mar C, Hayden F, Uyeki TM, et al. Transparent development of the WHO rapid advice guidelines. PLoS Med. 2007;4(5):e119.View ArticlePubMedPubMed CentralGoogle Scholar
- Santesso N, Schunemann H, Blumenthal P, De Vuyst H, Gage J, Garcia F, Jeronimo J, Lu R, Luciani S, Quek SC, et al. World Health Organization Guidelines: use of cryotherapy for cervical intraepithelial neoplasia. Int J Gynaecol Obstet. 2012;118(2):97–102.View ArticlePubMedGoogle Scholar
- Saudi Arabian Clinical Practice Guideline on Allergic Rhinits in Asthma (available at: http://www.moh.gov.sa/endepts/Proofs/Pages/Guidelines.aspx)
- Schünemann HJ, Brozek J. GRADEpro Guideline Development Tool. In. Hamilton, Canada: McMaster University; 2015.Google Scholar
- Schünemann H, Wiecioch W, Brozek J, Etxeandia-Ikobaltzetaen I, Mustafa R, Manja V, Brignadello-Peterson R, Neumann I, Falavigna M, Al-Hazzani W, et al. GRADE Evidence to Decision Frameworks for adoption, adaptation and de novo development of trustworthy recommendations: GRADE-ADOLOPMENT J Clin Epidemiol 2016, accepted for publication.Google Scholar
- WHO handbook for guideline development. Geneva, Switzerland: World Health Organization; 2014.Google Scholar
- Alonso-Coello P, Irfan A, Sola I, Gich I, Delgado-Noguera M, Rigau D, Tort S, Bonfill X, Burgers J, Schunemann H. The quality of clinical practice guidelines over the last two decades: a systematic review of guideline appraisal studies. Qual Saf Health Care. 2010;19(6):e58.PubMedGoogle Scholar
- Raine R, Sanderson C, Black N. Developing clinical guidelines: a challenge to current methods. BMJ. 2005;331(7517):631–3.View ArticlePubMedPubMed CentralGoogle Scholar
- Oxman AD, Fretheim A, Schunemann HJ. Improving the use of research evidence in guideline development: introduction. Health Res Policy Syst. 2006;4:12.View ArticlePubMedPubMed CentralGoogle Scholar
- Oxman AD, Schunemann HJ, Fretheim A. Improving the use of research evidence in guideline development: 14. Reporting guidelines. Health Res Policy Syst. 2006;4:26.View ArticlePubMedPubMed CentralGoogle Scholar
- Wilson KC, Irwin RS, File Jr TM, Schunemann HJ, Guyatt GH, Rabe KF. Reporting and publishing guidelines: article 12 in Integrating and coordinating efforts in COPD guideline development. An official ATS/ERS workshop report. Proc Am Thorac Soc. 2012;9(5):293–7.View ArticlePubMedGoogle Scholar
- Neumann I, Santesso N, Akl EA, Rind DM, Vandvik PO, Alonso-Coello P, Agoritsas T, Mustafa RA, Alexander PE, Schunemann H, et al. A guide for health professionals to interpret and use recommendations in guidelines developed with the GRADE approach. J Clin Epidemiol. 2016;72:45–55.View ArticlePubMedGoogle Scholar
- Alexander PE, Brito JP, Neumann I, Gionfriddo MR, Bero L, Djulbegovic B, Stoltzfus R, Montori VM, Norris SL, Schunemann HJ, et al. World Health Organization strong recommendations based on low-quality evidence (study quality) are frequent and often inconsistent with GRADE guidance. J Clin Epidemiol. 2016;72:98–106.View ArticlePubMedGoogle Scholar
- Schunemann HJ. Interpreting GRADE's levels of certainty or quality of the evidence: GRADE for statisticians, considering review information size or less emphasis on imprecision? J Clin Epidemiol. 2016;75:6–15.View ArticlePubMedGoogle Scholar