Table 4 Summary of selected papers
From: Health systems guidance appraisal—a critical interpretive synthesis
Author(s) | Title | Organization/locationa | Purpose of the study and relevance of its contribution | Concepts extracted |
|---|---|---|---|---|
Murray and Frenk [72] | A WHO framework for health system performance assessment | WHO/Switzerland | This paper discusses how variations in health outcomes across different countries are related to differences in health system performance (design, content, and management of health systems) and proposes a framework to assess and advance the understanding of health system performance. | Timeliness; scope; evidence-based; stakeholder involvement; ethical; outcomes; operationalization; costs; feasibility; socio-culturally acceptable; politically sound; external factors |
German et al. [71] | Updated guidelines for evaluating public health surveillance systems | CDC/USA | The paper aims to provide an operational framework and guidelines for evaluating the quality, efficiency, and usefulness of a public health surveillance systems. | Timeliness; scope; evidence-based; stakeholder involvement; problem definition; resources; outcomes/impact evaluation; feasibility; socio-culturally acceptable; generalizability; sustainability |
Davies and Littlejohns [53] | Views of directors of public health about NICE appraisal guidance: framework and guidelines for evaluating results of a postal survey | NICE/UK | The aim is to explore the view of Directors of Public Health with regards to the development, implementation, and dissemination of appraisal guidance for health technologies within the UK health system. | Prioritization; timeliness; scope; evidence-based; stakeholder involvement; presentation; feasibility; affordability; socio-culturally acceptable; politically sound |
APA [80] | Criteria for practice guideline development and evaluation | American Psychological Association / USA | The paper is designed to promote quality and consistency in practice guideline development and to describe the criteria by which practice guidelines are developed, evaluated and reviewed. | Prioritization; relevance; transparency; evidence-based; stakeholder involvement; ethical; outcomes/impact evaluation; presentation; feasibility; flexibility; external factors |
Shaw and Kalo [54] | A background for national quality policies in health systems | WHO / Switzerland | The paper aims to outline some of the values, forms, and concepts which affect national approaches for the improvement of quality as a central element for reform of health systems and health service delivery. | Prioritization; transparency; evidence-based; stakeholder involvement; ethical; operationalization; resources; effectiveness; dissemination plan; affordability; socio-culturally acceptable; politically sound; external factors |
Wilson [81] | How to find the good and avoid the bad or ugly: a short guide to tools for rating quality of health information on the internet | European Commission / Belgium | The aim is to report methods, a set of criteria for good practice and tools for evaluating and rating the quality of health information. | Ethical; feasibility |
Arah et al. [55] | Conceptual frameworks for health systems performance: a quest for effectiveness, quality, and improvement | University of Amsterdam / Nederlands | The aims are to understand the underlying concepts of national and international performance frameworks for health systems (case studies: UK, Canada, Australia, the USA, the WHO, and the OECD); to explore health system efficiency and performance indicators; and examine how and in what context the resultant performance data can be used to drive improvement. | Prioritization; relevance; timeliness; scope; evidence-based; stakeholder involvement; ethical; outcomes; resources; effectiveness; outcomes/impact evaluation; feasibility; affordability; socio-culturally acceptable; politically sound; generalizability |
Murray and Evans [86] | Health systems performance assessment: debates, methods and empiricism | WHO / Switzerland | The aim is to explore the role that the WHO plays in providing advice to member states on how best to organize, manage, and strengthen their health systems. It discusses how recommendations for clinical practice decisions differ from health system recommendations. | Outcomes; evidence-based |
Travis et al. [87] | Overcoming health-systems constraints to achieve the millennium development goals | WHO / Switzerland | The paper uses the Millennium Development Goals as a reference point to explore the advantages and disadvantages of approaches to health system strengthening through the lens of individual service or disease specific initiatives. | Prioritization; evidence-based; stakeholder involvement; outcomes; operationalization; resources; cost-effectiveness; feasibility; affordability; external factors; generalizability; sustainability |
Lomas et al. [78] | Conceptualizing and combining evidence for health system guidance | Canadian Health Services Research Foundation / Canada | This review examines how guidance developers and policy-makers view evidence as well as how different forms of evidence can be combined to produce health system guidance. | Relevance; transparency; evidence-based; stakeholder involvement; outcomes; ethical; problem definition; effectiveness; feasibility; flexibility |
Arah et al. [64] | A conceptual framework for the OECD health care quality indicators project | University of Amsterdam / Nederlands | The aim is to provide a sound conceptual framework that defines what is meant by quality of health care and to place it within a wider performance framework which acknowledges the key health policy goals adopted by the OECD and its member countries as they formally assess and “incentivize” the performance of their health care systems. | Relevance; timeliness; stakeholder involvement; ethical; problem definition; cost; effectiveness; feasibility; affordability; socio-culturally acceptable; politically sound; external factors; sustainability |
Oxman et al. [65] | Improving the use of research evidence in guideline development: reporting guidelines | Norwegian Knowledge Centre for the Health Services / Norway | The aim is explore the standard formats for wide variety of WHO policies, recommendations or guidelines and how these recommendations should be formulated and reported. It emphasizes that the information needed to judge the quality of guidance and determine its applicability and adaptability should be reported. | Relevance; timeliness; transparency; evidence-based; ethical; presentation; outcomes; problem definition; operationalization; cost-effectiveness; outcomes/impact evaluation; dissemination plan; affordability; generalizability |
Schünemann et al. [88] | Improving the use of research evidence in guideline development: 1. guidelines for guidelines | McMaster University / Canada | This report from the WHO Advisory Committee on Health Research is aimed at providing advice to the WHO on the use of more rigorous processes to ensure that the best available research evidence informs health care recommendations. | Prioritization; evidence-based; stakeholder involvement; ethical; outcomes; competing interests; presentation; problem definition; operationalization; costs; benefits/harm; process evaluation; outcomes/impact evaluation; updating; flexibility; generalizability |
Islam [56] | Health systems assessment approach: a how-to manual | USAID / USA | The aim of this report is to enable USAID Missions to assess a country’s health system during early phases of program development or sector planning. Using a performance indicator-based and health indices approach, the report is designed to provide a rapid and comprehensive assessment of key health systems functions (Governance, Health financing, Health service delivery, Human resources, Pharmaceutical management, Health information systems) and propose recommendations for improvement. | Prioritization; relevance; timeliness; scope; evidence-based; stakeholder involvement; ethical; resources; effectiveness; affordability; sustainability |
National Health and Medical Research Council (NHMRC) [57] | Standards and procedures for externally developed guidelines | Australian Government / Australia | The aims is to inform external persons and organizations of the procedures and key steps to be followed in developing, implementing, and evaluating guidelines that are intended for submission to the NHMRC for approval. | Prioritization; relevance; timeliness; evidence-based; stakeholder involvement; outcomes; presentation; problem definition; resources; effectiveness; cost-effectiveness; outcomes/impact evaluation; dissemination plan; feasibility; flexibility; socio-culturally acceptable; politically sound |
Øvretveit and Klazinga [61] | Guidance on developing quality and safety strategies with a health system approach | WHO / Denmark | The aim is to provide tools and approaches to help national policy advisers and policy-makers to create and implement a national quality strategy, drawing attention to the need for sustainable longer term public health measures in order to improve their health systems and engage member states in a constructive dialogue. | Prioritization; relevance; timeliness; scope; transparency; evidence-based; stakeholder involvement; ethical; presentation; operationalization; resources; process evaluation; outcomes/impact evaluation; feasibility; affordability; socio-culturally acceptable; politically sound; sustainability |
Van der Sluijs et al. [62] | Exploring the quality of evidence for complex and contested policy decisions | Utrecht University / The Nederlands | The aim is to provide a deeper understanding and increased awareness of the phenomenon of uncertainty and its policy implications, by discussing some key quality aspects of knowledge production and use especially in complex policy issues. | Relevance; timeliness; evidence-based; stakeholder involvement; outcomes; problem definition; process evaluation; outcomes/impact evaluation; socio-culturally acceptable |
Hoffman et al. [83] | The use of research evidence in two international organizations’ recommendations about health systems | McMaster University / Canada | This study systematically compares health systems recommendations by international organizations (WHO and the World Bank) to the research evidence that was available at the time of their formulation. These recommendations about health systems (on technical guidance for example) have the potential to link research to action by acting as mediators between the best available research evidence and policy options. | Prioritization; relevance; transparency; evidence-based; stakeholder involvement; ethical; outcomes/impact evaluation; feasibility; affordability; politically sound |
NHS [60] | How to use NICE guidance to commission high-quality services | NHS / UK | The guide is aimed for people involved in commissioning health and social care services and public health programs in the UK and provides guidance that can help to support the commissioning of new productive, efficient, and high-quality services, provides an implementation tool for planning and prioritizing services, and a framework for the evaluation or redesign of existing services and the decommissioning of ineffective interventions. | Prioritization; relevance; scope; evidence-based; stakeholder involvement; ethical; presentation; problem definition; operationalization; cost-effectiveness; outcomes/impact evaluation; generalizability; sustainability |
NHS [66] | Methods for the development of NICE public health guidance | NHS / UK | The paper describes the philosophical and methodological principles which govern the production of guidance for public health practice by NICE and the key components involved. | Relevance; stakeholder involvement; ethical; outcomes; presentation; problem definition; generalizability |
Oxman et al. [75] | SUPPORT tools for evidence-informed health policymaking: what is evidence-informed policymaking? | Norwegian Knowledge Centre for the Health Services / Norway | This article focuses on how to better use research evidence (what constitutes evidence? What is its role in health policy?) to inform decisions about how best to organize health systems, including arrangements for delivering, financing and governing health services, and strategies for bringing about change. | Prioritization; transparency; evidence-based; stakeholder involvement; ethical; outcomes; benefits/harms; politically sound |
Perpiñán et al. [84] | Quality assessment of economic evaluations in health care: a checklist and user guide | Murcia University / Spain | The aim is to promote the efficiency in the process of incorporating new health technologies, as well as to guide their implementation in health systems by reporting an instrument composed of a user guide and a 12-criteria checklist in which a score is assigned to each items. | Problem definition; cost; feasibility; socio-culturally acceptable; politically sound |
Savigny and Adam [89] | Systems thinking for health systems strengthening | Alliance for Health Policy and Systems Research, WHO | This report offers a practical systems thinking approach to decipher the complexity of health systems, identify health systems challenges, and then applies that understanding to design better interventions to strengthen health systems and improve health. | Scope; relevance; transparency; evidence-based; stakeholder involvement; ethical; outcomes; problem definition; cost; resources; effectiveness; cost-effectiveness; benefits/harms; process evaluation; outcomes/impact evaluation; feasibility; affordability; politically sound |
WHO [69] | Practical guidance for scaling up health service innovations | WHO / Switzerland | The aim is to identify general principles and make specific suggestions on the process of scaling up successfully tested health services innovations and discusses the strategic choices that facilitate and hinder the process. | Timeliness; evidence-based; stakeholder involvement; ethical; operationalization; resources; effectiveness; outcomes/impact evaluation; feasibility; affordability; socio-culturally acceptable; politically sound; sustainability |
Moher et al. [76] | Guidance for developers of health research reporting guidelines | Faculty of Medicine / University of Ottawa | The aim is to update and expand upon efforts to outline a strategy for developing reporting guidelines and shows that reporting guidelines is associated with improvements in the quality of reporting health research. An 18-step checklist on how to develop a reporting guideline is provided. | Transparency; evidence-based; stakeholder involvement; presentation; problem definition; resources; dissemination plan |
Swanson et al. [90] | Toward a consensus on guiding principles for health systems strengthening | Brigham Young University / USA | The paper proposes a list of ten guiding principles necessary for the development and communication of clear and consistent frameworks for policy, practice, and evaluation with the overall goal of strengthening health system. | Scope; transparency; evidence-based; stakeholder involvement; ethical; problem definition; operationalization; effectiveness; cost-effectiveness; outcomes/impact evaluation; affordability; socio-culturally acceptable; politically sound |
Etienne et al. [91] | Health systems financing: the path to universal coverage | WHO / Switzerland | This report provides practical guidance on ways to finance health care by transforming available evidence-based practices into a menu of options for raising sufficient resources and removing financial barriers to access. Emphasis is placed on moving toward universal coverage to optimize health service provision. | Relevance; scope; evidence-based; ethical; outcomes; feasibility; operationalization; flexibility; timeliness; cost; resources; effectiveness; cost-effectiveness; outcomes/impact evaluation; affordability; politically sound; external factors |
German BACKUP Initiative (GIZ) [82] | Guidance on integrating gender-specific issues into health systems strengthening activities | Federal Ministry of Economic Cooperation and Development / Germany | The aim is to advise and assist organizations that are planning to apply to the German BACKUP Initiative for technical support on how to analyze and integrate gender-related issues into Health Systems Strengthening (HSS) activities. A checklist used to plan for technical support and develop applications that take into consideration specific gender dimensions in the different components of a health system is provided. | Ethical; stakeholder involvement; socio-culturally acceptable; politically sound |
Sheikh et al. [92] | Building the field of health policy and systems research: framing the questions | Public Health Foundation of India / India | This paper discusses the state-of-the-art in Health Policy and Systems Research (HPSR), addresses the current challenges and opportunities for the field, and lays out what is needed to build capacity in HPSR and support local policy development and health systems strengthening. | Prioritization; relevance; scope; stakeholder involvement; outcomes; problem definition; operationalization; feasibility; flexibility; socio-culturally acceptable; politically sound; generalizability |
WHO [11] | Health system strengthening: improving support to policy dialogue around national health policies, strategies and plans | WHO / Switzerland | This report reviews experiences with conducting and supporting policy dialogue for the development or renewal of comprehensive policies, strategies, and plans to improve health service delivery, health outcomes and strengthen health systems. | Prioritization; relevance; scope; evidence-based; stakeholder involvement; outcomes; operationalization; costs; resources; effectiveness; feasibility; affordability; politically sound; external factors; generalizability; sustainability |
Atun [73] | Health systems, systems thinking and innovation | Faculty of Medicine, London Imperial College / UK | The aim is to discuss factors that influence the achievement of health system performance and efficiency and provide an understanding of why many well-intentioned policies and managerial decisions aimed at improving health systems do not achieve desired outcomes but lead to unexpected or unintended consequence. | Scope; evidence-based; ethical; problem definition; operationalization; costs; resources; effectiveness; feasibility; affordability; socio-culturally acceptable; politically sound; external factors |
Bosch-Capblanch and Allen [70] | Health systems strengthening and conflict: transformation in fragile states | Tropical and Public Health Institute / Switzerland | The aim is to inform programming, policy, advocacy, research, and the civil society on the best ways to strengthen health systems. Main challenges encountered in producing methods to develop health systems guidance are discussed. | Timeliness; evidence-based; costs; effectiveness; feasibility; politically sound |
Bosch-Capblanch et al. [4] | Guidance for evidence-informed policies about health systems: rationale for and challenges of guidance development | Tropical and Public Health Institute / Switzerland | The aim is to assess extent to which the need for health systems guidance is part of national policies and plans and assess how guidance is currently formulated. Conceptual and the methodological challenges in the development and use of health systems guidance and ways to address them are discussed. | Prioritization; relevance; timeliness; evidence-based; stakeholder involvement; ethical; outcomes; presentation; problem definition; cost-effectiveness; outcomes/impact evaluation; dissemination plan; feasibility; external factors |
Lavis et al. [5] | Guidance for evidence-informed policies about health systems: linking guidance development to policy development | McMaster University / Canada | The aim is to discuss the importance of contextual factors in shaping decisions about health systems and discusses the need to work through all the pros and cons of different options before adopting specific health systems guidance. It also shows the need for division of labor between national/global guidance and policy developers to support evidence-informed policymaking about health systems. | Prioritization; relevance; scope; transparency; evidence-based; stakeholder involvement; problem definition; costs; effectiveness; outcomes/impact evaluation; feasibility; politically sound |
Lewin et al. [63] | Guidance for evidence-informed policies about health systems: assessing how much confidence to place in the research evidence | Norwegian Knowledge Centre for the Health Services / Norway | The aim is to assess how much confidence to place in the types of evidence available on health systems interventions that inform judgments for health systems strengthening. The factors that are important when developing recommendations on policy options regarding health systems interventions are discussed. | Relevance; scope; transparency; evidence-based; stakeholder involvement; ethical; operationalization; cost-effective; feasibility; affordability; socio-culturally acceptable; politically sound |
Moga et al. [67] | Development of a quality appraisal tool for case series studies using a modified Delphi technique | Institute of Health Economics / Canada | The aim is to outline the process of development of a checklist for quality appraisal of case series studies using a modified Delphi technique. Criteria and items that are useful for appraising the quality of case series are reported. | Relevance; transparency; stakeholder involvement; outcomes; problem definition |
Orem et al. [74] | Do guidelines influence the implementation of health programs? - Uganda’s experience | WHO / Uganda | The aim is to describe the processes of development, implementation, dissemination, and evaluation of health planning, services management, and clinical guidelines within the health sector in Uganda, with the goal of understanding how these processes facilitate or abate the utility of guidelines. | Prioritization; relevance; scope; evidence-based; stakeholder involvement; ethical; outcomes; problem definition; operationalization; costs; effectiveness; outcomes/impact evaluation; dissemination plan; generalizability |
Peters and Bennett [93] | Better guidance is welcome, but without blinders | John Hopkins School of Public Health / USA | This paper discusses the challenges related to engendering greater structure and systematization in the development of health system guidance and to the application of evidence to policy. | Evidence-based; stakeholder involvement; benefits/harms; feasibility; flexibility; politically sound; generalizability |
WHO [58] | Guidance on assessing health system building blocks | WHO / Switzerland | The aim is to provide an overview of the key opportunities and challenges facing a health system (health issues, systemic issues, political/policy issues), and how to address and assess them. It discusses indicators that can be used to assess each of the health systems building blocks and provides a how-to-manual. | Prioritization; scope; evidence-based; stakeholder involvement; outcomes; presentation; operationalization; resources; outcomes/impact evaluation; feasibility; politically sound |
Bryce et al. [79] | A common evaluation framework for the African health initiative | The Johns Hopkins Bloomberg School of Public Health / USA | The aim is to describe a common evaluation framework for the cross-site initiative to improve population health by strengthening health systems and evaluating the results. Some core elements, inputs, and processes required for strengthening health systems in Africa are discussed. | Stakeholder involvement; ethical; resources; cost-effectiveness; process evaluation; outcomes/impact evaluation; feasibility; politically sound; external factors; generalizability |
Philips et al. [77] | Protocol for development of the guideline for reporting evidence based practice educational interventions and teaching (GREET) statement | University of South Australia / Australia | The aim is to develop reporting guidelines for evidence-based practice educational interventions and teachings to enable their consistent and transparent reporting in health care. Criteria for appraising practice education for health professional disciplines are provided. | Evidence-based; stakeholder involvement; costs; effectiveness; outcomes/impact evaluation |
International Centre for Applied Health Evidence (ICAHE) [68] | The iCAHE guideline checklist | University of South Australia / Australia | The aim is to provide a checklist (items, criteria and domains) to assist in the process of development and appraisal of guidelines. | Relevance; timeliness; evidence-based; stakeholder involvement; presentation; problem definition; feasibility |
Funk et al. [59] | Checklist for evaluating a mental health policy | WHO / Switzerland | The aim is to explore the processes that are likely to lead to the success of a mental health policy. A framework for the assessment of the quality of the processes and content of mental health policy recommendations is proposed. | Prioritization; scope; transparency; evidence-based; stakeholder involvement; ethical; problem definition; resources; feasibility; affordability; external factors; transferability |