Skip to main content


  • Systematic review
  • Open Access
  • Open Peer Review

Health systems guidance appraisal—a critical interpretive synthesis

  • 1,
  • 1Email author,
  • 2,
  • 3 and
Implementation Science201611:9

  • Received: 25 September 2015
  • Accepted: 17 January 2016
  • Published:
Open Peer Review reports



Health systems guidance (HSG) are systematically developed statements that assist with decisions about options for addressing health systems challenges, including related changes in health systems arrangements. However, the development, appraisal, and reporting of HSG poses unique conceptual and methodological challenges related to the varied types of evidence that are relevant, the complexity of health systems, and the pre-eminence of contextual factors. To address this gap, we are conducting a program of research that aims to create a tool to support the appraisal of HSG and further enhance HSG development and reporting. The focus of this paper was to conduct a knowledge synthesis of the published and grey literatures to determine quality criteria (concepts) relevant for this process.


We applied a critical interpretive synthesis (CIS) approach to knowledge synthesis that enabled an iterative, flexible, and dynamic analysis of diverse bodies of literature in order to generate a candidate list of concepts that will constitute the foundational components of the HSG tool. Using our review questions as compasses, we were able to guide the search strategy to look for papers based on their potential relevance to HSG appraisal, development, and reporting. The search strategy included various electronic databases and sources, subject-specific journals, conference abstracts, research reports, book chapters, unpublished data, dissertations, and policy documents. Screening the papers and data extraction was completed independently and in duplicate, and a narrative approach to data synthesis was executed.


We identified 43 papers that met eligibility criteria. No existing review was found on this topic, and no HSG appraisal tool was identified. Over one third of the authors implicitly or explicitly identified the need for a high-quality tool aimed to systematically evaluate HSG and contribute to its development/reporting. We identified 30 concepts that may be relevant to the appraisal of HSG and were able to cluster them into three meaningful domains: process principles, content, and context principles.


Our study showed the role that the quality criteria play in the development, appraisal, and reporting of HSG and demonstrated the link and resonance within and between the various concepts in the three domains.


  • Health systems guidance
  • Guidance development
  • Guidance appraisal
  • Guidance reporting
  • Health systems challenges
  • Health systems arrangements


Roemer [1] defined a health system as the combination of resources, organizations, and financing and management bodies that culminate in the delivery of health services to a population. A redefinition of the term health system was proposed by the World Health Organization [2] to consist of all organizations, people, actions, and activities whose primary purpose is to promote, restore, and maintain health. This was further modified—to be more comprehensive and explicit—by a group convened by the World Health Organization’s (WHO) Alliance for Health Policy and Systems Research, to refer to delivery and financial and governance arrangements for health care and population health services and the broader context in which they are negotiated, implemented, and reformed [3].

Health systems guidance (HSG) are systematically developed statements produced at global, national, and regional levels to assist with decisions about appropriate options for addressing health systems challenges (including related changes in health systems arrangements), the implementation of these options, and the monitoring and evaluation of the implementation efforts [4]. The need to develop and use evidence-informed approaches to address national and global health system issues through the use of HSG has been articulated by many [5]. For example, the Millennium Development Goals (MDGs) reflect eight international goals (e.g., reduce child mortality, improve maternal health, and combat HIV/AIDS, malaria, and other diseases) and were developed by the United Nations in 2000. However, their attainment has been hindered by weak health systems and lack of system-specific recommendations [6, 7]. Formulating recommendations to address root causes has the potential to clear the path toward better health outcomes in line with health goals [8].

The terms guidelines and guidance both refer to any document containing a recommendation on a course of action. However, in contrast to the word “guidelines,” the term “guidance” is used in the health system context in order to make explicit the difference between the process of supporting evidence-informed judgments for health system issues from that of clinical judgments as would be expected with clinical practice guidelines (CPGs). As a policy-oriented product, HSG represents the whole body of knowledge that informs policy decisions on how health system issues should be classified or prioritized, appropriate health system governance, optimal financial arrangements, system organization, and the design and delivery of effective health programs and services [912].

The main bodies tasked with developing HSG are international and intergovernmental organizations (e.g., WHO, Pan American Health Organization), local ministries of health, and special national committees or agencies providing support to ministries of health. Their production is usually linked with these high-level health sector entities and less frequently in decentralized structures at the sub-national level. HSG may be aligned with specific goal-oriented taxonomies (e.g., WHO building blocks), unique national/regional governance, or tailored to financial/delivery arrangements with the common goal to optimize health care systems. However, while these organizations have demonstrated an interest in the development, appraisal, and reporting of this type of guidance as described in the WHO’s Handbook for Supporting the Development of Health Systems Guidance [13], for example, many of their members and collaborators (who are also members of the Appraisal of Guidelines for Research and Evaluation for Health Systems (AGREE-HS) team) contend that these intensions have not been matched by adequate action or results, in part, due to the lack of experience in developing HSG.

Indeed, in comparison to CPGs, the development of HSG pose different conceptual and methodological challenges related to the varied types of evidence that are relevant, valued, and considered; the complexity of health systems’ internal and external relations that must be addressed; and the pre-eminence of contextual factors that directly influence the design and adoptability of recommendations [4]. As with CPGs, HSG statements or recommendations should be justified by assessments of the quality of evidence supporting them, the potential for unintended consequences, and by discussions of implementation and contextual issues. For example, the full implementation of HSG recommendations can be further hampered by “bottlenecks” like health system fragmentation and capacity limitations; these limitations should also be addressed in the guidance documents [14, 15].

The potential for positive impacts by HSG as a decision tool to improve health systems is great. However, as with CPGs [16], this potential is only as good as the quality of the HSG. Indeed, the ability to impact and optimize health system performance and efficiency through the development and adoption of HSG is hampered by the dearth of tools to guide their development, appraisal, and reporting. As a consequence, this leaves knowledge users at a loss when choosing the highest quality and/or most appropriate guidance or in creating new guidance in circumstances where there is none or navigating circumstances where the existing guidance is not credible or of poor quality. In contrast to the development, appraisal, and reporting methods for CPGs, the development of HSG is still at a rudimentary stage [4]. There has been some work related to the use of “evidence briefs” to assist policymakers and stakeholders with working through a health system problem, options for addressing it, and key implementation considerations, informed by the best available data and research evidence [17]. However, there is a need for systematically and transparently developed guidance that can feed into such context-specific documents. With the rising trends that encourage bridging the gap between research and policy and practice, this is a significant research gap in both the science and practice of knowledge translation. HSG can provide this bridge between research synthesis and policy needs for evidence.

The creation of high-quality HSG requires tools to support the development and reporting of high-quality guidance and tools able to differentiate between high- and low-quality reports. There is also value in establishing acceptable quality thresholds and creating common methodologies and nomenclature among the HSG community (developers, users, and researchers). At present, however, there is no universally acceptable gold standard approach for appraising HSG, although there are some tools (for example the Handbook for Supporting the Development of Health System Guidance) to support their development [13] and reporting. To address these gaps, we are conducting a program of research with the international health systems guidance community that aims to create a tool to support the appraisal of HSG and further enhance HSG development and reporting.

The first step to this program of research, and the focus of this paper, was to conduct a knowledge synthesis of the published and grey literatures to determine concepts (items, criteria, or domains) related to HSG development, reporting, and quality. The results of the synthesis are to provide the foundational components of our HSG tool and to serve as a conceptual status report for the research community.


With the goal of generating a list of concepts that can be used to develop, appraise, and report HSG, we conducted an initial search to identify existing concepts, tools, templates, or checklists that have been used or could be used to describe, differentiate, or appraise the quality of HSG. The aim of this initial search was to determine if reviews on this topic exist, estimate an initial number of relevant papers available, highlight additional and useful search terms, and clarify inclusion and exclusion criteria and the most appropriate knowledge synthesis approach [18, 19].

After contemplating the variety of knowledge synthesis approaches available, a critical interpretive synthesis (CIS) approach was considered the most appropriate for three reasons. First, CIS is a systematic approach that facilitates the analysis of complex and diverse bodies of literature including qualitative, quantitative, and theoretical papers [2026]. The available literature on HSG is highly heterogeneous and methodologically diverse, comprising of a mix of empirical qualitative and quantitative papers and non-empirical papers. A wider range of evidence and study designs are typical occurrences in health systems/health policy research and HSG is a nascent domain where good thinking is likely to be captured in expert opinion/views, editorial comment, policy documents, political statements, experiences of stakeholders, theoretical/discussion papers, and other colloquial forms of evidence. In contrast to CIS, conventional systematic reviews, for example, have been criticized for excluding forms of evidence traditionally considered as non-experimental [20, 2730].

Second, the objective of a CIS is to develop new concepts and theories through a typically interpretive mode of inquiry. This is in contrast to more conventional systematic review approaches where the mode of inquiry is more aggregative and aimed at testing theories by collating, compiling, pooling, and summarizing common outcomes across a range of studies [1921, 2527, 31, 32]. Since our review aims at generating a candidate list of items, criteria, or domains for HSG development, appraisal, and reporting, this theory-generating approach that CIS promotes was deemed a good fit.

Third, CIS offers a more flexible, iterative, dynamic, and reflective approach requiring investigators to assess the extent to which new information or data are provided with each additional paper considered. It applies a relatively loosely defined set of processes for critically analyzing and synthesizing literature [19, 20, 23, 27, 28, 30]. This differs from the more conventional systematic review methods which have clearly stated study protocols, an exhaustive search of all available literature, standardized data extraction templates, and explicit quality appraisal checklists, as well as pre-determined focused questions, strict inclusion and exclusion criteria, and specified data boundaries [20, 23, 2730].

With the unique conceptual and methodological challenges related to the development of HSG [4], it was essential to have a review method that was iterative, flexible, and dynamic. Our review was not simply aimed at summarizing this complex literature on HSG, so we sought to produce a logical and insightful interpretation of a purposefully sampled body of evidence. As such, the aim was to yield a comprehensive interpretation without requiring exhaustive identification of relevant items of literature as in a conventional systematic review [20, 29, 30].

Review question(s)

As per CIS methodological standards, our review questions served as compasses rather than anchors [20, 29, 33] allowing for the concepts of HSG to be derived from synthesis of the literature and constantly modifying them in an iterative manner throughout the review.

Our guiding review questions were as follows:
  1. 1.

    How have authors variously defined HSG quality in the literature?

  2. 2.

    How have authors interpreted and used criteria (or tools or instruments, checklists, systems, etc.): to describe or define HSG quality or reporting requirements, to appraise HSG quality, or to differentiate between HSG on the basis of quality?

  3. 3.

    What methods have authors and stakeholders used to develop these criteria?

  4. 4.

    What methods have authors and stakeholders used to address health system issues/challenges?


Literature search

In searching the literature, our goal was to select papers based on their potential relevance to HSG appraisal and quality, while including other papers that, though not directly relevant to health systems, were deemed important for the purpose of our review. Instead of including an exhaustive number of papers, our plan was to provide a comprehensive sampling frame of potentially relevant papers using emergent eligibility criteria [25, 32]. As per the standards of a CIS, the boundaries of our inclusion and exclusion criteria were modifiable, dynamic, and continuously shifting [20, 32, 34]. The goal was to populate the concept of HSG with new concepts rather than finding papers that reiterated ideas already captured in previously reviewed papers.

Eligibility criteria

  1. 1.

    Study content: (a) Papers that evaluated HSG or papers that report on criteria/tools that have been documented as important indicators of HSG quality and (b) papers that reported on methods for addressing health system issues/challenges.

  2. 2.

    Time frame: Papers are eligible if published in or after the year 2000 (when the first World Health Report on health systems [2] was published).

  3. 3.

    Context: Unrestricted. We sought papers that considered HSG evaluation and health system issues/challenges in various contexts (low-, middle-, and high-income countries).

  4. 4.

    Study design: Any study design.

  5. 5.

    Language: English, French, and Spanish


A combination of key and free text terms were used to search through various databases. Terms used were health systems, health policy, guidelines, guidance, health services arrangement, health services organization, health system issues, health system challenges, tools, instruments, criteria, items, domains, evaluation, appraisal, quality, and standards.

The search strategy included various electronic databases and sources: Cumulative Index to Nursing and Allied Health Literature (CINAHL), Cochrane Library, Excerpta Medica dataBASE (EMBASE), Google Scholar, Health Systems Evidence, Latin American and Caribbean Health Sciences Literature (LILACS), PubMed, Virtual Health Library, and Web of Science. We also searched subject-specific as well as regional electronic sources: Australia’s National Health and Medical Research Council, Evidence Best Practices for Public Health, University of Massachusetts National Guidelines Clearinghouses, National Institute of Health and Care Excellence (NICE), WHO’s Evidence Informed Policy Network (EVIPNet), WHO EURO’s Health Evidence Network, and Guidelines International Network (GIN) directories.

We also searched for other grey literature, including conference abstracts (Global Symposium on Health Systems Research, Canadian Association for Health Services and Policy Research, Health Systems and Process Improvement Conference, Canada’s Health Leadership Conference, Health Systems Reform in Asia Conference, International Society on Priorities in Health Care). Further, research reports, book chapters, unpublished data, dissertations, and policy documents that were nominated by members of the team or found in unique holdings of health sciences libraries in Canada were also included. Additional papers were identified by manually searching bibliographies, while more were obtained by hand searching some key journals (i.e., Health Policy and Planning, Health Services Research and Policy, Health Research Policy and Systems, Global Healthcare Systems, Health Systems and Reform, and Health Policy). This was a complementary search strategy to account for papers not included in electronic databases or with search terms that do not allow them to be easily identified [34, 35].

Finally, we also contacted (through emails, phone, Skype or in person) experts, colleagues, and members of our research team with a known interest on this topic to identify additional papers (published, unpublished, or ongoing). To increase the scope of these key informants, we also asked initial contacts to refer us to others who can provide more information. Relevant papers were then imported to Endnote bibliographic software.

A two-step sampling process was used. First, a purposeful sampling approach [36, 37] was used to investigate the literature to determine the range of unique candidate concepts—or theoretical domains—associated with HSG. We stopped sampling at the saturation point where looking at new literature no longer contributed additional concepts [25, 37]. Second, a theoretical sampling strategy was used to interrogate the literature relevant to each of the identified concepts. Theoretical sampling does not occur at a single point in the research process but is a recurrent feature, which aligns it well with the dynamic CIS methodology. Similar to the first approach, we stopped sampling at the saturation point where looking at new literature no longer contributed additional descriptions of the identified concepts. The intent here was to thoroughly capture the depth of the concept across the literature in order to generate and develop the theoretical underpinnings of the ideas rather than collecting numerous citations of identical concepts and/or descriptions. Screening titles, abstracts, and full text was completed independently and in duplicates (DAA and SA). Disagreements were resolved by consensus.

Quality appraisal

We did not consider it appropriate to undertake quality appraisal for three principal reasons: firstly, the literature on HSG is diverse and complex; secondly, the available literature could not be hierarchically ordered in terms of study design, importance, or relevance; and thirdly, very little consensus exists on how or whether to perform quality appraisal in a CIS methodology [27, 32, 38, 39]. Instead, we chose to evaluate papers for inclusion based on a judgment of their relevance and likely contribution to concept development and theory [23, 26, 32, 40]. Our assumption was that some methodologically weak papers are theoretically and conceptually pertinent [25, 29, 34]. The goal of our review was to generate key concepts relevant to the appraisal of HSG, so we applied a flexible relevance boundary in order to be as inclusive and comprehensive as possible and include papers that could contribute to this theory generation.

Data extraction and analysis

As a result of the nature of the data, we did not use a standardized data extraction template to retrieve the relevant elements of this review from all the retained papers. Instead, and as per CIS methodology, we gave latitude for a more narrative data retrieval approach. It was, however, possible to extract some key elements from all the papers including author(s), the year of publication, geographical location of the study or affiliation of the author, purpose of the study/paper, its relevance and a summary of its main conceptual contributions, and types of items, criteria, or domains considered. Data were extracted independently and in duplicate (DAA and SA) with disagreements resolved via consensus. The extracted data were then compiled in summary form.

Again due to the anticipated diversity of methodological designs and implementation of the different items, criteria, and domains that were extracted from the papers, a narrative approach to data synthesis was executed. The data extracted were sorted and categorized into groups with themes. We sought to collect similar concepts within the same group. This systematic method of recording themes, and making connections between themes and the data collected within a comprehensive category system [41, 42], is an advocated concept-development coding approach [43]. The information was imported into the analysis software NVivo version 9 in order to analyze groups of themes that depicted similar patterns in information. We did not organize the concepts in any hierarchical order but reported the concepts that were identified in each of the papers in this review.

Modeling from the health policy analysis triangle framework that was developed by Walt and Gilson [44], we were able to connect the concepts (constructs) together into meaningful domains. The framework considers all the essential elements that interact to shape policymaking by demonstrating that health policy should focus on the processes contingent on developing and implementing change, on the content of health policy reform, and on the context within which the policy is promulgated, as well as on the actors involved in the policy reform [44, 45]. The health policy analysis triangle framework is a highly simplified model of an extremely complex set of interrelationships between the different elements of the model (process, content, context, and actors) with each element influencing or being influenced by the other [45, 46]. Development, appraisal, and reporting of HSG play an important role in health policy making by providing options and recommendations to address a health systems issue. Therefore, similar to health policymaking, HSG development, appraisal, and reporting can be seen to occur in a series of discrete yet interconnected components of process, content, and context.

Results and discussion

No existing knowledge synthesis was found on the topic, and no existing HSG appraisal tool (draft or final version) was identified. We identified a total of 43 papers that met our eligibility criteria and reported on concepts (items, criteria, domains) considered directly or conceptually relevant to HSG and/or their quality (see Fig. 1 for a flowchart of the study selection).
Fig. 1
Fig. 1

Flowchart of study selection

Forty-seven percent of the retained studies were technical reports, 32 % were concept papers, 13 % were quantitative studies, and 8 % used a mixed methods approach. The authors held affiliations at the following organizations: 33 % at universities, 25 % at the WHO, 23 % at research institutes, 12 % at government agencies (United Kingdom’s National Health Services [NHS], United States’ Centre for Disease Control [CDC], United States Agency for International Development [USAID], United Kingdom’s National Institute for Health and Care Excellence [NICE]), and 7 % at national ministries of health. Fifty-eight percent (58 %) of the papers were lead authored by an individual based in Europe, 30 % in North America, 8 % in Australia, 2 % in Asia, and 2 % in Africa. Over one third of the authors implicitly or explicitly identified the need for a high-quality tool aimed to systematically evaluate HSG and contribute to development and reporting of HSG. Thirty concepts were identified that are considered to be a good fit for, and may be relevant to, the development, appraisal, and reporting of HSG. The Appendix provides a description of the papers, their objectives, relevance to the HSG process, and the concepts extracted.

Process principles, content, and context principles

Ostrom [47] stated that identifying key concepts and relationships among them culminates in the generation of a theory, which connects these key concepts. We were able to organize the concepts and identify relationships within and across them. Through an iterative process, we clustered the concepts together into three meaningful domains: process principles, content, and context principles.

Process principles

Process principles represent the methodological elements and the defining principles that demonstrate the development integrity of the HSG. They refer to the ways in which the HSG is initiated, developed, and formulated and can be looked upon as the “who” and “how” of the guidance [44, 46]. Here, we find the procedures and principles that were employed by the HSG developers in coming up with the guidance recommendations. Process principles are critical in the development of HSG because the methodological strategy is crucial for guidance appraisal and for differentiating across HSG of varying quality. They are also important because they can articulate the tactics that could be utilized in the development and reporting of HSG to optimize quality. Process principles also depict the subjective and/or objective belief systems in place that represent the preferences (what ought to be) of individuals, groups, or populations on the course(s) of action for addressing a health systems challenge. Process principles also refer to how the guidance is made public in a way that is consistent and comparative as this can facilitate their comprehension, which will further enhance their uptake and aid in easing their application. Therefore, when developing, appraising, or reporting HSG, a conceptual understanding of the process is fundamental. Table 1 shows concepts for HSG process principles.
Table 1

Process principles

1. Prioritization

The guidance fits in properly and is consistent with current health system priority areas within all applicable system levels and sectors by targeting a priority topic/jurisdiction/population [4, 5, 5359]. The guidance addresses these specific local priority areas with a clearly documented/demonstrated need and also informs policy decisions on how to further prioritize across competing areas [4, 53, 54, 59, 60]. The origin of the mandate to develop the guidance [59] is also reported (for example, guidance that is mandated by a top official like the Minister of Health is considered to be of high priority).

2. Relevance

The guidance recommendations should be relevant to, appropriate to, and valid for the health system issue being addressed and relevant to the target population [5, 5557, 6063]. The recommendations are relevant to the setting within which the guidance will operate, the institutional needs of that system/sub-system, as well as local, national, and potentially global needs [54, 6468].

3. Timeliness

The recommendations are available in a timely manner in relation to when the policy decisions are made or timely in relation to the health system issue being addressed [55, 56, 61, 69, 70]. The guidance is timely and usable by the broad range of health systems stakeholders [53, 68, 71] since some policy decisions are sometimes made within crucial corresponding time frames or as windows of opportunity open and close [4, 62].

4. Scope

The guidance is comprehensive and covers all relevant/appropriate (direct and indirect) health system levels, sub-systems, and sectors [5, 8, 56, 61, 63, 71, 72]. This also includes the various relevant sub-systems/components (hospitals, regional health authorities, and public health units etc.) within the health system. Identifying the scope is important because these various components are interlinked, interdependent, and interact at various interfaces for overall health system performance [53, 55, 5860, 7174].

5. Transparency

Systematic, replicable, and transparent processes are applied in developing and reporting the guidance [52, 61, 75, 76]. These processes are systematic and transparent enough for the methods of development/reporting of the guidance to be reproducible [59, 74]. In order to paint a clear picture to knowledge users and target populations, sufficient details on these processes are provided [5, 54, 63, 65, 67].

6. Evidence- based

The best available research evidence informs the recommendations [4, 5, 54, 55, 57, 58, 60, 61, 63, 6971, 7377]. The type(s) of evidence that was used to generate the guidance is/are stated, and this can range from well-established scientific methodologies or it can also be non-experimental (for example, colloquial evidence, anecdotal evidence, or preliminary models) [8, 62, 63, 70, 75, 78]. The evidence is context sensitive enough to resonate with local realities [4, 5, 54, 56, 57, 60, 61, 63, 6971, 74, 75].

7. Stakeholder involvement

Alternative views on the policy issue and the complementary expertise of a multidisciplinary group of relevant stakeholders are considered in the development of the guidance [4, 5, 8, 5355, 5864, 6769, 71, 7577, 79]. Guidance developers, those involved in the implementation and evaluation of the guidance, and those who will be affected by the guidance recommendations are involved in the development process [54, 56, 57, 66, 71, 74, 80].

8. Ethical

The recommendations reflect considerations of an ethical lens [4, 8, 54, 56, 76] and align with applicable ethical principles and values (for example, equity, equality, human rights, liberty, efficiency, autonomy, dignity, beneficence) [8, 55, 63, 65, 74, 80, 81]. The guidance adequately promotes fairness and equality in terms of age, ability, culture, gender, socioeconomic status, religion, occupation, language, ethnicity, race, or sexual orientation among the target population [60, 61, 66, 69, 79, 82].

9. Outcomes

The guidance describes all the anticipated effects/outcomes as well as the appropriate indicators that can be used to measure the effects/outcomes [8, 54, 57, 58, 66, 67, 74, 75]. Adequate rationale regarding the choice of the outcomes and the indicators selected is provided [8, 58]. Considering potential uncertainties that may result, alternative outcomes and outcome indicators are also identified. Performance thresholds, targets, and standards that are considered acceptable are also identified [8, 58, 6567, 76].

10. Competing interests

A declaration of competing interests (for example, financial, academic, professional) by the guidance developers, whether direct or indirect, is/are made in advance [57, 74, 80]. The author’s positions, roles, and affiliations are clearly stated [65]. Any reported or identified conflicts of interest are managed, with a description of the approaches used to curb any influence clearly documented [57, 80]. It is also clear that the views of any funding body involved have not influenced the development process of the guidance [65].

11. Presentation

The recommendations are clear, succinct, unambiguous, and presented in a readable and consistent format [57, 60, 76, 81], with key recommendations easily identifiable [53, 57, 58, 66]. The guidance is presented in a manner that is uniform, user-friendly, and easy to navigate [4, 53, 61, 64, 68, 80]. It contains an executive summary, full text, a complete list of relevant references, a glossary of terms, and full meaning of abbreviations and contact information of authors. Words or phrases denote an aspirational rather than a mandatory intent [80].

We were able to point out patterns and draw linkages between the process principles concepts. We offer five examples here. First, guidance should be relevant to and developed for health system areas with a clearly demonstrated and documented need, and the feedback of appropriate stakeholders will further highlight priority areas, which may potentially lead to timely interventions. Second, systematic and transparent approaches have to be applied to search for and identify relevant evidence, which should also be available in a timely manner. Third, ensuring that stakeholders from all applicable health system and sub-system levels are involved in the HSG process, that relevant evidence is sought and that appropriate outcomes are chosen can enhance the comprehensiveness of the guidance recommendations. Fourth, having clearly defined and consistent outcomes and declaring and managing interests as well as engaging in a participatory approach that incorporates the various perspectives of multiple stakeholders is also important for transparency of HSG. Finally, the ethical lens applied will be impacted by the quality of the evidence available, the composition of the involved stakeholders, the outcomes/indicators selected, the health system levels/sector involved, and whether it is a priority area that is relevant to the setting.


Content (the “what” of the HSG) represents the topics, subjects, and substance that demonstrate the content integrity of the guidance recommendations [44, 46]. Here, we find statements about the impetus of the endeavor that provides direction on the HSG objectives and goals. Keeping in mind that in the absence of an appropriate economic lens, even the best-designed HSG may not achieve adequate success; we also find here the myriad of economic factors that come with the HSG process and need to be taken into account. In addition, within the content of the HSG, it is also important to consider and reflect on the fact that not all guidance recommendations go into practice as planned and events may occur that may derail intended actions. We also find here assessment considerations, usually ongoing throughout the HSG process, and which refer to those elements that assist in determining whether the guidance process was properly followed and/or records the impact/outcomes of the HSG. The content of the HSG also articulates the operationalization of the proposed recommendations and carries information on how to best reach the target users of the guidance. Table 2 shows concepts for HSG content.
Table 2


12. Problem definition

The health systems challenge (for example, financial, governance, or delivery arrangements) and its causes are clearly articulated (including any links/integration with other policy problems on the government’s agenda) [5, 75, 83]. The nature, causes, magnitude, frequency, and intensity of the problem, the populations, and jurisdictions that are affected are clearly described [57, 60, 62, 64, 66, 71]. Appropriate rationale exists to justify that either new guidance is needed or existing guidance of acceptable quality can be adapted and used to address the problem [4, 56].

13. Operationalization

The recommended “solutions” are operationalized sufficiently with the conceptualization, operational guidance, and the mode of delivery of the options clearly stated [8, 54, 60]. For example, the guidance provides instructional support for their successful operation and staff training that corresponds with the guidance expectations. Training recommendations could be in the form of a course, a workshop, accompanying manuals, or consultancy services that staff can refer to during the implementation phase in order to standardize practice [8, 54, 58, 60, 61, 63, 73, 74]. If technical assistance (research institutes, consulting firms, NGOs) is required, this is identified and documented [8].

14. Costs

The guidance clearly documents a tentative budget required to implement the guidance recommendations [8, 64, 65, 74, 77, 84]. The potential financial costs (including downstream costs) of the operation are stated so that decision makers can assess the feasibility of the guidance implementation and evaluate whether the cost of implementing the guidance will be worth its potential impacts [5, 64, 73, 74].

15. Resources

The inputs and resources required to implement the recommendations are clearly defined and they have to be proportionate to the health system problem that is being addressed [55, 57, 61, 71, 73, 79]. Some of these resources could be time, infrastructure, administrative capacity, information, equipment, supplies, healthcare professionals, training etc. [55, 56, 58, 59, 61, 69, 76]. The guidance provides a description of the amount, frequency, and duration of the inputs and resources required [54, 57, 71].

16. Effectiveness

The guidance reports whether the anticipated goals and objectives have been achieved elsewhere or in a similar setting/condition, either through evidence from evaluation studies done at other sites (if available) or from expert opinion [5456, 69]. In describing this effectiveness, the guidance makes projections on how and why the objectives and goals will be achieved in the current setting [64, 69, 75].

17. Cost-effectiveness

The recommendations are attentive to value for money considerations [57, 63, 84]. Sound local or applicable evidence (wherever available) on the cost-effectiveness of the guidance recommendations are provided [54, 57, 60, 63, 79]. These traditionally report costs, direct and indirect program inputs/resources, and outcomes to guide health policy decisions and provide benchmark(s) or threshold(s) that the health system is willing to accept or support in relation to other competing health system priorities [54, 60, 73].

18. Benefits/harms

Description of the potential unintended consequences (positive and negative) of the guidance is provided or an assessment/judgment of the potential benefits/harms are made [60, 67, 75]. Descriptions of the populations or institutions that may experience significant impacts are identified [57, 60, 65].

19. Dissemination plan

Strategies for communicating the guidance are included with a clear dissemination framework, the mode of delivery, and the integrity of the avenue used for dissemination been properly reported [54, 65, 81]. The proposed strategies for disseminating the guidance are tailored to the relevant audiences (for example, a formal written report, user-friendly summary, oral presentation, poster, press release, booklet, workbook, films, pocket card) [57, 74].

20. Process evaluation

This involves recommendations for evaluating the structure and process of implementation as well as corresponding challenges [61, 68, 80]. This evaluation examines the extent to which the guidance recommendations were implemented as planned, and also provides a way to monitor the process and make adjustments and improvements to implementation strategies [74, 79]. It documents the inputs, services, and activities that were implemented, and can identify potential strengths, weaknesses, opportunities, and threats to the implementation process [62, 79].

21. Outcomes/impact evaluation

An assessment of the outcome/impact of the guidance is recommended to determine whether the course of action was a success or failure. There are recommendations on measuring the results, or outcomes of the guidance in a way that determines whether the changes observed in relation to the health system challenge being addressed can be attributed to the guidance [58, 61, 80, 82]. There are also recommendations for an impact evaluation to look at the short- and long-term deeper primary and secondary changes that resulted from the guidance [57, 69, 74, 77].

22. Updating

Recommendations for periodic updates are made and the procedure to update the guidance is provided with explicit timelines on anticipated review [57, 68], appropriate expiration date of the guidance, and an explanation of the rational for the proposed time frames [80]. Setting time frames for periodic updates ensures that guidance producers revisit the recommendations and respond accordingly to potential health system changes and emerging challenges. Also, the recommendations should be current, and the evidence (for example, systematic reviews) on which they are based is considered recent and up-to-date [76, 80].

As with the process principles, we were also able to point out patterns and draw linkages between the content concepts. For example, the way the problem is defined can provide direction on the costs and resources required for implementation of the solution, reveal some potential unintended consequences, and guide the operational plan. The operational plan can also provide hints on some potential unintended consequences. A clear problem definition will also inform the process and outcomes/impact evaluation of the HSG. Designing the operational options can inform the costs and resources required. The costs, resources, and operational plan will in turn influence whether the HSG will be effective and/or cost-effective. The updating plan is also contingent on the health system issue being addressed, the effectiveness of the guidance recommendation, and the resources required.

Context principles

Context principles represent those systemic factors like the local technical, situational, structural, institutional political, and socio-cultural components of the health policy environment that can impact HSG recommendations [44, 46]. Like health policymaking, HSG does not occur in a vacuum so it is important to pay attention to the variety of contextual factors that may have significant impact on how the guidance is developed, adapted, and implemented by the end users [48, 49]. The process principles and the content of the HSG all have to be contextualized; therefore, context principles impact these other clusters of the concepts. Context principles refer to the usability in context, and here, we find concepts related to those system components that make up the setting within which the HSG is to be used. A clear overview of the context or setting that will be impacted by the guidance recommendations is essential. This will provide an understanding of why some guidance recommendations may work in some settings and not others. We find here factors that will enhance and facilitate the adherence to HSG recommendations as per protocol. We also find here factors that represent the values and moral fabric of the society and how this can either facilitate or impede the HSG recommendations. Table 3 shows concepts for HSG context principles.
Table 3

Context principles

23. Feasibility

The guidance recommendations are realistic and the actions are pragmatic [4, 5, 53, 55, 57, 59, 61, 63, 68, 70, 81, 84]. The guidance describes facilitators and barriers for implementation [58, 69]. It is clearly demonstrated that the implementation of the guidance is feasible within the proposed practice environment, and the recommendations match local capacities and expectations [4, 8, 53, 61, 63, 69, 71, 80].

24. Affordability

The guidance recommendations are affordable within the financial structure and budgetary allocations of the health system [53, 54, 56, 69]. Potential sources of local government funding and donor organizations are identified. For policy issues in which there may be several sources of funding, the guidance also considers the level of coordination among the donors and between the donors and the local government [56, 58, 59, 61, 69].

25. Flexibility

The guidance is flexible and adaptable to the expertise of the user and the varying local conditions. It acknowledges the importance of professional judgment and discretion and provides recommendations that users can adapt in accordance with their own individual circumstances and needs [57, 71, 80]. The recommendations steer away from the adoption of rigid approaches so as not to inappropriately or unnecessarily limit those in charge of applying them [57, 71].

26. Socio- culturally acceptable

Considering the diversity of values in many regions, the recommendations are robust under societal and cultural scrutiny by adopting a socio-cultural perspective [54, 57, 6163, 69, 73]. It recognizes socio-cultural expectations and provides an understanding of the role that socio-cultural factors will play in the success of the guidance recommendations [54, 55, 69, 71, 73].

27. Politically sound

The political acceptability of the recommendations is considered in order to assess if they align with political interests/commitments [54, 57, 63, 64, 70, 75, 79, 81, 82]. Implementation of guidance can stir swings in the national mood, lead to changes in the balance of organized forces, such as interest groups, or influence outcome of events within the government, for instance an election [85]. Therefore, options proposed that are in sync with the political climate may garner adequate support from top policy/government officials [58, 73].

28. External factors

Determinants of health system performance that lie outside the formal architecture of the health system but will influence the performance of its functions are considered; for example, judicial system, social system, recession, corruption, state of the economy [48, 54, 73, 79]. These are non-health system factors originating from other local institutional organizations that impact on the usual operations of the health system [54, 59, 64, 80].

29. Generalizability

The recommendations are transferable to other settings with similar health system features; for example other countries or regions [59, 65, 66, 79]. Judgments are made about the applicability of the recommendations beyond its original context (setting or population) to ensure that contexts with similar institutional, socioeconomic, and political demographics facing an identical health system challenges can adapt and use the guidance [55, 60, 65, 74].

30. Sustainability

The guidance provides an indication of the sustainability of the effects of the recommendations to show that long-term outcomes can be continuously achieved and maintained at an acceptable level [56, 61, 64, 69]. Due to constantly evolving health system issues, looming budget cuts, fluctuating resources, rising costs of new technologies, an ageing population, shifting burdens of diseases etc., it is crucial to develop recommendations that will stand the test of time [56, 60, 69].

We were able to point out patterns and draw linkages between the context principle concepts as well. We offer some examples here. For HSG to be feasible and sustainable, it should be affordable and resonate with local values (political and socio-cultural). Information on affordability, feasibility, and sustainability may also determine whether the guidance recommendations can be transferable to other settings. Providing socio-culturally appropriate recommendations may also indicate which other health systems can adapt and use the HSG. Also, the HSG should be flexible enough to accommodate shifts in values (e.g., political, socio-cultural). Additionally, HSG that reflects the socio-cultural preferences of the target population, that is affordable, feasible, sustainable, and transferable to comparable settings, can amass support from politicians. The external factors that originate from other institutional systems may impact the feasibility and sustainability of the guidance and may be influenced by the political and socio-cultural climate.

Synthesis across domains

As is expected of a Critical Interpretive Synthesis (CIS), we were able to point out relationships between the concepts across the three clusters as shown in Fig. 2. We highlight some relationships:
Fig. 2
Fig. 2

Framework of health systems guidance concepts

  • ➢ Uptake of guidance can be enhanced if it is addressing a priority area for which evidence-based reports of effectiveness and cost-effectiveness exist.

  • ➢ Consultations with appropriate stakeholders are also crucial for feasibility of implementation and sustainability of the HSG because sometimes, the individuals tasked with implementing guidance recommendations may not be committed to them and this may influence adherence. Their input into the HSG process and support for the recommendations may alleviate this concern.

  • ➢ HSG outcome(s) chosen will influence the way the problem is defined and provides information that will be useful for evaluating the HSG.

  • ➢ Including an ethical viewpoint will also impact how the problem is defined and will influence cost-effectiveness thresholds.

  • ➢ Systematic and transparent processes may provide an impetus for donor involvement (affordability).

  • ➢ Information on affordability can determine whether the health system issues can be addressed in a timely manner and inform judgments on how to evaluate the process.

  • ➢ The HSG should be flexible enough to accommodate constantly evolving evidence and changing health system priorities.

  • ➢ The guidance should provide socio-culturally appropriate solutions that are relevant to the applicable levels/sectors of the health system.

  • ➢ HSG that is in line with ethical principles, addresses priority issues, and is timely will be appealing to politicians.

  • ➢ The external alignment of guidance may affect the effectiveness and impact the operational considerations.

  • ➢ The outcomes selected, the system level/sectors involved, or the ethical values in place may expose the external factors that may be pertinent.

  • ➢ Some determinants of generalizability of the guidance are the stakeholders involved, the evidence used, transparency of the process, ethical lens considered, and outcome/indicators selected


This project is the first phase of a multistage approach to create an internationally useful HSG tool, AGREE for Health Systems (AGREE-HS) that will inform the development, reporting, and appraisal of HSG. Modeling after the paradigm to create a tool for clinical practice guidelines [50], our first step was to conduct a review of the published and grey literatures to identify concepts related to HSG quality. In this vein, it was our expectation that the receptiveness, adoption, and diffusion of HSG recommendations depend on the perception of their quality, and with this study, we aimed to identify those core components of good quality HSG.

We found a total of 30 potential HSG appraisal concepts that have the capacity to discriminate between high- and low-quality guidance and direct their development and reporting. We found no existing tools to support HSG appraisal and found few studies describing concepts that were directly tested to appraise HSG. Indeed, the papers we examined reflected a variety of study designs and goals; none reflected methods used to develop appraisal methods. However, the data from our studies show a convergence of ideas in the HSG research community about what constitutes good and useable HSG. Together, these data can provide the foundation of a tool that guides HSG developers in the types of information important to report in a HSG document and tactics for optimal execution. These concepts also address issues of appropriateness and completeness as well as information, which are important components in the uptake of guidance recommendations.

One strength of this study is that we used a sound knowledge synthesis strategy, the critical interpretive synthesis (CIS) approach. We acknowledged the paucity of data related to HSG appraisal, as well as the diversity in the literature sources and types of available data. Also, given that HSG is not a highly bounded topic, we did not rely on a narrow initial research question; instead, we were refining it as the review progressed. The various methodological stages did not proceed as discrete entities as there was a constant to-ing and fro-ing throughout the review and synthesis process; at times, we were concurrently searching, sampling, critiquing, and analyzing. The CIS approach is both systematic and iterative with an interpretive approach to analysis and synthesis of data that allowed us to capture and critically analyze an in-depth depiction of how to differentiate between HSG on the basis of quality, among other considerations. Through CIS methodology, we were able to further identify and include additional papers that were not directly related to HSG but made valuable theoretical contributions to the process of appraising guidance [20, 27].

A weakness of our study is in the nature of the CIS strategy, which exposes us to the risk that certain concepts may have been missed (calling saturation too soon, for example). If we have missed important elements, we anticipate they will be identified in the second stage of our research. Also, the health policy analysis triangle framework is made up of four elements: process, content, context, and actors. But for our analysis, the original framework was modified as we included “actors” in the process principles. Actors (the who of policy) refer to local, national, or international individuals or groups (governmental and non-governmental) involved in the policy process [44, 45]. We found that actors could be incorporated under the process principles concept “participatory.” Additionally and at first glance, it appears that there is the risk that some concepts may lead to potential contradictions. For example, in some jurisdictions, achieving an ethical HSG may not be aligned with achieving a feasible HSG (e.g., in contexts where there is a bias toward citizens because of race or sexual orientation). Currently, the data provides no guidance on how to reconcile a situation like this.

In our analysis, we showed the role that the process principles, content, and context principles plays in the development, appraisal, and reporting of HSG and demonstrated the link and resonance within and between their various concepts. We pointed out patterns and drew linkages between the concepts in order to show that the concepts and domains do not typically occur in a linear, independent, or discrete manner. The interaction between these three clusters is an imperative consideration because they all influence the guidance process and can facilitate or impede the success of HSG recommendations.

Some of the concepts (criteria or items) and domains identified in this review may not be applicable to every jurisdiction or country so they are not intended to serve as a blueprint for all health systems to strictly apply. Guidance documents are administrative instruments that provide recommendations and implementation options typically in a step-by-step format but do not have force of law and, as such, allow for accommodations in approach [51, 52]. They are intended to provide recommendations on how to comply with governing statutes and regulations and assist staff and managers on how institutional mandates and objectives should be implemented in a manner that is fair, consistent, and effective [52]. Therefore, development of high-quality HSG will impact the type of recommendations being formulated, the degree to which they get implemented, the methods of dissemination, and the extent to which they impact on the usual operations of the health system [51].

Different players in the HSG process play different roles at different times and under different circumstances. Developers of guidance may only be looking at process principles and content, while end users in the field may focus more on usability of the guidance in their individual contexts. Thus, for example, to optimize this dual perspective and facilitate the division of labor, there is interest at the WHO to produce workbooks where the global guidance they develop (process principles and content) will be complemented by having a companion workbook for those at the receiving end (context principles). In this case, the role of the WHO will be the development of global guidance, while the country’s role (or health system’s role) will be to take this global guidance and then apply it to their local context. This makes our framework of health systems guidance concepts quite practical.

Our study has elicited 30 unique concepts. It is unclear the extent to which HSG stakeholders will view all concepts as equally important to our proposed tool, or, moreover, given the potential roles of the tool (development, appraisal, reporting), if there are varying levels of priority as a function of purpose. The next phase (stage 2) of our research program is aimed to create the beta version of the AGREE-HS by having international stakeholders prioritize these concepts and using those results to populate the tool. Our final phase (stage 3) will involve the usability testing of the beta version of the tool.



Appraisal of Guidelines for Research and Evaluation for Health Systems


Centre for Disease Control


Cumulative Index to Nursing and Allied Health Literature


Critical Interpretive Synthesis


Clinical Practice Guidelines


Denis Ako-Arrey


Excerpta Medica dataBASE


Evidence Informed Policy Network


Guidelines International Network


health systems guidance


John Lavis


Latin American and Caribbean Health Sciences Literature


Melissa Brouwers


Millennium Development Goals


Mita Giacomini


National Health Services


National Institute of Health and Care Excellence


Saira Akram


United States Agency for International Development


World Health Organization



The authors would like to acknowledge the contributions of the members of the AGREE-HS team: Andy Haines—London School of Hygiene & Tropical Medicine, UK; Carmen Mihaela Dolea—World Health Organization, Switzerland; Fadi El-Jardali—American University of Beirut, Lebanon; Francoise Cluzeau—National Institute for Health and Care Excellence, UK; Govin Permanand Govin—World Health Organization, Denmark; Iván Darío Flórez Gómez—Universidad de Antioquia, Colombia; Jillian Ross—Cancer Care Ontario, Canada; Luis Gabriel Cuervo—Pan American Health Organization, USA; Mike Wilson—McMaster University, Canada; Pablo Perel—London School of Hygiene and Tropical Medicine, UK; Paidgraig Warde—Cancer Care Ontario, Canada; Pierre Ongolo—Centre for Development of Best Practices in Health, Cameroon; Sheila McNair—Program for Evidence Based Care, Canada; Ulysses Panisset—World Health Organization, Switzerland; Xavier Bosch-Capblanch—Swiss Tropical and Public Health Institute, Switzerland; and Yaolong Chen—Lanzhou University, China. This project was supported by the Canadian Institutes of Health Research (Canada). DAA received a student award from Knowledge Translation (KT) Canada. We would also like to acknowledge SA for her role as a second reviewer during the screening and data extraction.

Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (, which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver ( applies to the data made available in this article, unless otherwise stated.

Authors’ Affiliations

McMaster University, Juravinski Hospital Site, G Wing, 2nd Floor, Room 207, 711 Concession Street, Hamilton, ON, L8V 1C3, Canada
McMaster University, MML-417, 1280 Main St. West, Hamilton, ON, L8S 4L6, Canada
McMaster University, CRL-218, 1280 Main Street West, Hamilton, ON, L8S 4K1, Canada


  1. Roemer M. I. National health systems of the world, vol. 1: the countries. New York: Oxford University Press; 1991. p. 97.Google Scholar
  2. World Health Organization. The World Health Report 2000. Health systems: improving performance. Geneva, 2000. [accessed 9 July 2013] Available from: Scholar
  3. Hoffman S, Rottingen J-A, Bennett S, Lavis J, Edge J, Frenk J. Background paper on conceptual issues related to health systems research to inform a WHO global strategy on health systems research—a working paper in progress. Geneva: World Health Organisation; 2012. p. 2013.Google Scholar
  4. Bosch-Capblanch X, Lavis JN, Lewin S, Atun R, Røttingen JA, Dröschel D, et al. Guidance for evidence-informed decisions about health systems: rationale for and challenges of guidance development. PLoS Med. 2012;9:e1001185.PubMed CentralView ArticlePubMedGoogle Scholar
  5. Lavis JN, Røttingen JA, Bosch-Capblanch X, Atun R, El-Jardali F, Gilson L, et al. Guidance for evidence-informed policies about health systems: linking guidance development to policy development. PLoS Med. 2012;9:e1001186.PubMed CentralView ArticleGoogle Scholar
  6. Remme JHF, Adam T, Becerra-Posada F, D’Arcangues C, Devlin M, Gardner C, et al. Defining research to improve health systems. PLoS Med. 2010;7(11):e1001000.PubMed CentralView ArticlePubMedGoogle Scholar
  7. Fryatt R, Mills A, Nordstrom A. Financing of health systems to achieve the health Millennium Development Goals in low-income countries. Lancet. 2010;375:419–26.View ArticlePubMedGoogle Scholar
  8. Murray C, Frenk J. A WHO framework for health system performance assessment. Bull World Health Organ. 2000;78:717–31.PubMed CentralPubMedGoogle Scholar
  9. World Health Organization. Fifty-eighth World Health Assembly, A58/22, 1–7. Geneva: Author; 2005. WHA58.34: Ministerial Summit on Health Research.Google Scholar
  10. Bosch-Capblanch X. Handbook for developing health systems guidance. Swiss: TPH; 2010. Analytical framework for health systems guidance.Google Scholar
  11. World Health Organization. Health system strengthening: improving support to policy dialogue around national health policies, strategies and plans. Report by the Secretariat to the 64th World Health Assembly. A64/12. Geneva: World Health Organization; 2011. Retrieved on October 4th 2013 from Scholar
  12. Wilson MG, Moat KA, Lavis JN. The global stock of research evidence relevant to health systems policymaking. Health Res Policy Syst. 2013;11(32):10–1186.Google Scholar
  13. Bosch-Capblanch X. Handbook for supporting the development of health system guidance. Basel: Swiss Centre for International Health; 2011.Google Scholar
  14. Travis P, Bennett S, Haines A, Pang T, Bhutta Z, Hyder A, et al. Overcoming health-systems constraints to achieve the Millennium Development Goals. Lancet. 2004;364:900–6.View ArticlePubMedGoogle Scholar
  15. Chopra M, Lawn J, Sanders D, Barron P. Achieving the health millennium development goals for South Africa: challenges and priorities. Lancet. 2009;374:1023–31.View ArticlePubMedGoogle Scholar
  16. Brouwers M, Kho ME, Browman GP, Burgers JS, Cluzeau F, Feder G, et al .Appraisal of Guidelines for Research and Evaluation. AGREE II Instrument. The AGREE Next Steps Consortium. 2009. Available from: [Accessed: 2013 Aug 23].
  17. SUPPORT. Supporting policy relevant reviews and trials. 2012. Available: Accessed 30 January 2013.Google Scholar
  18. Brettle A. Information skills training: a systematic review of the literature. Health Info Libr J. 2003;20 Suppl 1:3–9.View ArticlePubMedGoogle Scholar
  19. Heaton J, Corden A, Parker G. ‘Continuity of care’: a critical interpretive synthesis of how the concept was elaborated by a national research programme (2012). Int J Integrated Care. 2012;12.Google Scholar
  20. Dixon-Woods M, Agarwal S, Jones D, Young B, Sutton A. Synthesising qualitative and quantitative evidence: a review of possible methods. J Health Serv Res Policy. 2005;10(1):45–53.View ArticlePubMedGoogle Scholar
  21. Dixon-Woods M, Kirk MD, Agarwal MS, Annandale E, Arthur T, Harvey J, et al. Vulnerable groups and access to health care: a critical interpretive review. National Coordinating Centre NHS Service Delivery Organ RD (NCCSDO). 2005.Google Scholar
  22. Taylor TL, Killaspy H, Wright C, Turton P, White S, Kallert TW, et al. A systematic review of the international published literature relating to quality of institutional care for people with longer term mental health problems. BMC Psychiatry. 2009;9(1):55.PubMed CentralView ArticlePubMedGoogle Scholar
  23. Barnett-Page E, Thomas J. Methods for the synthesis of qualitative research: a critical review. BMC Med Res Methodol. 2009;9(1):59.PubMed CentralView ArticlePubMedGoogle Scholar
  24. Gysels MH, Evans C, Higginson IJ. Patient, caregiver, health professional and researcher views and experiences of participating in research at the end of life: a critical interpretive synthesis of the literature. BMC Med Res Methodol. 2012;12(1):123.PubMed CentralView ArticlePubMedGoogle Scholar
  25. Entwistle V, Firnigl D, Ryan M, Francis J, Kinghorn P. Which experiences of health care delivery matter to service users and why? A critical interpretive synthesis and conceptual map. J Health Serv Res Policy. 2012;17(2):70–8.PubMed CentralView ArticlePubMedGoogle Scholar
  26. Kazimierczak KA, Skea ZC, Dixon-Woods M, Entwistle VA, Feldman-Stewart D, N’dow JM, et al. Provision of cancer information as a “support for navigating the knowledge landscape”: findings from a critical interpretive literature synthesis. Eur J Oncol Nurs. 2013;17(3):360–9.View ArticlePubMedGoogle Scholar
  27. Dixon-Woods M, Cavers D, Agarwal S, Annandale E, Arthur A, Harvey J, et al. Conducting a critical interpretive synthesis of the literature on access to healthcare by vulnerable groups. BMC Med Res Methodol. 2006;6:35. doi:10.1186/1471-2288-6-35.PubMed CentralView ArticlePubMedGoogle Scholar
  28. Dixon-Woods M, Bonas S, Booth A, Jones DR, Miller T, Sutton AJ, et al. How can systematic reviews incorporate qualitative research? A critical perspective. Qual Res. 2006;6(1):27–44.View ArticleGoogle Scholar
  29. Talseth AG, Gilje FL. Nurses’ responses to suicide and suicidal patients: a critical interpretive synthesis. J Clin Nurs. 2011;20(11–12):1651–67.View ArticlePubMedGoogle Scholar
  30. Morrison LG, Yardley L, Powell J, Michie S. What design features are used in effective e-health interventions? A review using techniques from critical interpretive synthesis. Telemedicine and e-Health. 2012;18(2):137–44.View ArticlePubMedGoogle Scholar
  31. Noblit GW, Hare RD. Meta-ethnography: synthesising qualitative studies. London: Sage; 1988.Google Scholar
  32. Markoulakis R, Kirsh B. Difficulties for university students with mental health problems: a critical interpretive synthesis. Rev High Educ. 2013;37(1):77–100.View ArticleGoogle Scholar
  33. Eakin JM, Mykhalovskiy E. Reframing the evaluation of qualitative health research: reflections on a review of appraisal guidelines in the health sciences. J Eval Clin Pract. 2003;9:187–94.View ArticlePubMedGoogle Scholar
  34. Flemming K. Synthesis of quantitative and qualitative research: an example using critical interpretive synthesis. J Adv Nurs. 2010;66(1):201–17.View ArticlePubMedGoogle Scholar
  35. Dickersin K, Scherer R, Lefebvre C. Identifying relevant studies for systematic reviews. BMJ. 1994;309(6964):1286–91.PubMed CentralView ArticlePubMedGoogle Scholar
  36. Seale C. The quality of qualitative research. London: Sage Publications; 1999.Google Scholar
  37. Annandale E, Harvey J, Cavers D, Dixon-Woods M. Gender and access to healthcare in the UK: a critical interpretive synthesis of the literature. Evid Policy. 2007;3(4):463–86.View ArticleGoogle Scholar
  38. Sandelowski M, Docherty S, Emden C. Focus on qualitative methods. Qualitative meta-synthesis: issues and techniques. Res Nurs Health. 1997;20:365–71.View ArticlePubMedGoogle Scholar
  39. Dixon-Woods M, Shaw R, Agarwal S, Smith J. The problem of appraising qualitative research. Qual Saf Health Care. 2004;13:223–5.PubMed CentralView ArticlePubMedGoogle Scholar
  40. Gough D. Weight of evidence: a framework for the appraisal of the quality and relevance of evidence. In: Furlong J, Oancea A, editors. Special Edition of Research Papers in Education, 22 (2): 213–228. 2007.Google Scholar
  41. Burnard P. A method of analysing transcripts in qualitative research. Nurse Educ Today. 1991;11:461–6.View ArticlePubMedGoogle Scholar
  42. Graneheim UH, Lundman B. Qualitative content analysis in nursing research: concepts, procedures and measures to achieve trustworthiness. Nurse Educ Today. 2004;24:105–12.View ArticlePubMedGoogle Scholar
  43. Strauss A, Corbin J. Basics of qualitative research: grounded theory procedures and techniques. 2nd ed. Newbury Park: Sage; 1998.Google Scholar
  44. Walt G, Gibson L. Reforming the health sector in developing countries: the central role of policy analysis. Health Policy Plan. 1994;9:353–70.View ArticlePubMedGoogle Scholar
  45. Walt G, Shiffman J, Schneider H, Murray SF, Brugha R, Gilson L. ‘Doing’ health policy analysis: methodological and conceptual reflections and challenges. Health Policy Plan. 2008;23(5):308–17.PubMed CentralView ArticlePubMedGoogle Scholar
  46. Buse K, Mays N, Walt G. Making health policy. Berkshire: Open University Press; 2005.Google Scholar
  47. Ostrom E. Institutional rational choice: an assessment of the institutional analysis and development framework. In: Sabatier PA, editor. Theories of the policy process. 2nd ed. Boulder: Westview Press; 2007. p. 21–64.Google Scholar
  48. Collins C, Green A, Hunter D. Health sector reform and the interpretation of policy context. Health Policy Plan. 1999;47:69–83.View ArticleGoogle Scholar
  49. WHO, World Health Organization. Sound choices: enhancing capacity for evidence-informed health policy. Geneva. 2007Google Scholar
  50. Brouwers MC, Kho ME, Browman GP, Burgers JS, Cluzeau F, Feder Get al. AGREE II: advancing guideline development, reporting and evaluation in health care. CMAJ 2010. DOI:10.1503/cmaj.090449Google Scholar
  51. World Health Organization. “Guidelines for WHO guidelines.” EIP/GPE/EqC/2003.1. Geneva: Author; 2003. Retrieved Jan 28, 2013. < EqC_2003_1.pdf.Google Scholar
  52. Health Canada. Recognition and use of standards under the medical devices regulations. 2009. Retrieved on October 1st, 2013 from Scholar
  53. Davies E, Littlejohns P. Views of directors of public health about NICE appraisal guidance: results of a postal survey. J Public Health. 2002;24(4):319–25.View ArticleGoogle Scholar
  54. Shaw CD, Kalo I. A background for national quality policies in health systems. 2002. Retrieved on August 19th 2013 from Scholar
  55. Arah OA, Klazinga NS, Delnoij DMJ, Ten Asbroek AHA, Custers T. Conceptual frameworks for health systems performance: a quest for effectiveness, quality, and improvement. Int J Qual Health Care. 2003;15(5):377–98.View ArticlePubMedGoogle Scholar
  56. Islam M. Health systems assessment approach: a how-to manual. 2007. Retrieved on August 17th 2013 from Scholar
  57. National Health and Medical Research Council - NHMRC. Standards and procedures for externally developed guidelines. 2007. Retrieved on August 16th 2013 from Scholar
  58. World Health Organization - WHO. The health systems assessment approach: a how to manual: guidance on assessing health system building blocks. 2012. Retrieved on October 4th 2013 from Scholar
  59. Funk M, Drew N, Faydi E. Checklist for evaluating a mental health policy. Retrieved August 19th 2013 from
  60. National Health Services - NHS. How to use NICE guidance to commission high quality services. 2009. Retrieved October 4th 2013 from Scholar
  61. Øvretveit J, Klazinga N. Guidance on developing quality and safety strategies with a health system approach. 2008. Retrieved on August 17th 2013 from Scholar
  62. Van der Sluijs JP, Petersen AC, Janssen PH, Risbey JS, Ravetz JR. Exploring the quality of evidence for complex and contested policy decisions. Environ Res Lett. 2008;3(2):024008. anual.View ArticleGoogle Scholar
  63. Lewin S, Bosch-Capblanch X, Oliver S, kl EA, Vist GE, Lavis JN, et al. Guidance for evidence-informed policies about health systems: assessing how much confidence to place in the research evidence. PLoS Med. 2012;9(3):e1001187.PubMed CentralView ArticleGoogle Scholar
  64. Arah OA, Westert GP, Hurst J, Klazinga NS. A conceptual framework for the OECD health care quality indicators project. Int J Qual Health Care. 2006;18 suppl 1:5–13.View ArticlePubMedGoogle Scholar
  65. Oxman AD, Fretheim A, Schünemann HJ. Improving the use of research evidence in guideline development: 14. Reporting guidelines. Health Res Policy Syst. 2006;4:26.PubMed CentralView ArticlePubMedGoogle Scholar
  66. National Health Services - NHS. Methods for the development of NICE public health guidance (second edition). 2009. Retrieved October 4th 2013 from Scholar
  67. Moga C, Guo B, Schopflocher D, Harstall C. Development of a quality appraisal tool for case series studies using a modified delphi technique. Edmonton: Institute of Health Economics (IHE); 2012.Google Scholar
  68. International Centre for Allied Health Evidence - iCAHE. The iCAHE guideline quality checklist. Retrieved on August 19th 2013 from
  69. World Health Organization - WHO. Practical guidance for scaling up health service innovations. 2009. Retrieved on October 5th 2013 from Scholar
  70. Bosch-Capblanch X, Allen C. Health systems strengthening and conflict: transformation in fragile states. 2012. Retrieved on August 17th 2013 from Scholar
  71. German RR, Lee LM, Horan JM, Milstein RL, Pertowski CA, Waller MN. Updated guidelines for evaluating public health surveillance systems: recommendations from the Guidelines Working Group. MMWR Morb Mortal Wkly Rep Recomm Rep. 2001;50:1–35.Google Scholar
  72. Murray CJ, Frenk J. A WHO framework for health system performance assessment. Evidence and information for policy, World Health Organization. 1999.Google Scholar
  73. Atun R. Health systems, systems thinking and innovation. Health Policy Plan. 2012;27 suppl 4:iv4–8.View ArticlePubMedGoogle Scholar
  74. Orem JN, Wavamunno JB, Bakeera SK, Criel B. Do guidelines influence the implementation of health programs?—Uganda’s experience. Implement Sci. 2012;7(1):1–16.View ArticleGoogle Scholar
  75. Oxman AD, Lavis JN, Lewin S, Fretheim A. SUPPORT Tools for evidence-informed health Policymaking (STP) 1: What is evidence-informed policymaking? Health Res Policy Syst. 2009;7 Suppl 1:S1.PubMed CentralView ArticlePubMedGoogle Scholar
  76. Moher D, Schulz KF, Simera I, Altman DG. Guidance for developers of health research reporting guidelines. PLoS Med. 2010;7(2):e1000217.PubMed CentralView ArticleGoogle Scholar
  77. Phillips AC, Lewis LK, McEvoy MP, Galipeau J, Glasziou P, Hammick M, et al. Protocol for development of the guideline for reporting evidence based practice educational interventions and teaching (GREET) statement. BMC Med Educ. 2013;13(1):9.PubMed CentralView ArticlePubMedGoogle Scholar
  78. Lomas J, Culyer T, Mccutcheon C, Law S, Tetroe J. Conceptualizing and combining evidence for health system guidance. 2005. Retrieved on October 4th 2013 from Scholar
  79. Bryce J, Requejo JH, Moulton LH, Ram M, Black RE. A common evaluation framework for the African Health Initiative. BMC Health Serv Res. 2013;13 Suppl 2:S10.PubMed CentralView ArticlePubMedGoogle Scholar
  80. American Psychological Association -APA-. Criteria for practice guideline development and evaluation. Am Psychol. 2002;57:1048–51.View ArticleGoogle Scholar
  81. Wilson P. How to find the good and avoid the bad or ugly: a short guide to tools for rating quality of health information on the internet. BMJ. 2002;324(7337):598.PubMed CentralView ArticlePubMedGoogle Scholar
  82. German BACKUP Initiative - GIZ. Guidance on integrating gender-specific issues into health systems strengthening activities. 2011. Retrieved on October 4th 2013 from Scholar
  83. Hoffman S, Lavis J, Bennett S. The use of research evidence in two international organizations’ recommendations about health systems. Healthcare Policy. 2009;5(1):66–86.PubMed CentralPubMedGoogle Scholar
  84. Perpiñán Abellán JM, Sánchez Martínez FI, Martínez Pérez JE. Quality assessment of economic evaluations in health care: a checklist and user guide. Rev Esp Salud Publica. 2009;83(1):71–84.View ArticleGoogle Scholar
  85. Kingdon JW, Thurber JA. Agendas, alternatives, and public policies (vol. 45). Boston: Little, Brown. Chicago; 1984.Google Scholar
  86. Murray CJ, Evans DB. Health system performance assessment: debates, methods, and empiricism. Geneva: World Health Organization; 2003. Retrieved on August 17th 2013 from Scholar
  87. Travis P, Bennett S, Haines A, Pang T, Bhutta Z, Hyder AA, et al. Overcoming health-systems constraints to achieve the Millennium Development Goals. Lancet. 2004;364(9437):900–6.View ArticlePubMedGoogle Scholar
  88. Schünemann HJ, Fretheim A, Oxman AD, WHO Advisory Committee on Health Research. Improving the use of research evidence in guideline development: 1. Guidelines for guidelines. Health Res Policy Syst. 2006;4(13):1–6.Google Scholar
  89. Savigny D, Adam T (Eds.). Systems thinking for health systems strengthening. World Health Organization. 2009. Retrieved on August 17th 2013 from
  90. Swanson RC, Bongiovanni A, Bradley E, Murugan V, Sundewall J, Betigeri A, et al. Toward a consensus on guiding principles for health systems strengthening. PLoS Med. 2010;7(12):e1000385.PubMed CentralView ArticlePubMedGoogle Scholar
  91. Etienne C, Asamoa-Baah A, Evans DB. Health systems financing: the path to universal coverage. World Health Organization. (2010): Retrieved on August 17th 2013 from
  92. Sheikh K, Gilson L, Agyepong IA, Hanson K, Ssengooba F, Bennett S. Building the field of health policy and systems research: framing the questions. PLoS Med. 2011;8(8):e1001073.PubMed CentralView ArticlePubMedGoogle Scholar
  93. Peters DH, Bennett S. Better guidance is welcome, but without blinders. PLoS Med. 2012;9(3):e1001188.PubMed CentralView ArticlePubMedGoogle Scholar


© Ako-Arrey et al. 2016