We’re sorry, something doesn't seem to be working properly.
Please try refreshing the page. If that doesn't work, please contact support so we can address the problem.
Making sense of complexity in context and implementation: the Context and Implementation of Complex Interventions (CICI) framework
Implementation Sciencevolume 12, Article number: 21 (2017)
The effectiveness of complex interventions, as well as their success in reaching relevant populations, is critically influenced by their implementation in a given context. Current conceptual frameworks often fail to address context and implementation in an integrated way and, where addressed, they tend to focus on organisational context and are mostly concerned with specific health fields. Our objective was to develop a framework to facilitate the structured and comprehensive conceptualisation and assessment of context and implementation of complex interventions.
The Context and Implementation of Complex Interventions (CICI) framework was developed in an iterative manner and underwent extensive application. An initial framework based on a scoping review was tested in rapid assessments, revealing inconsistencies with respect to the underlying concepts. Thus, pragmatic utility concept analysis was undertaken to advance the concepts of context and implementation. Based on these findings, the framework was revised and applied in several systematic reviews, one health technology assessment (HTA) and one applicability assessment of very different complex interventions. Lessons learnt from these applications and from peer review were incorporated, resulting in the CICI framework.
The CICI framework comprises three dimensions—context, implementation and setting—which interact with one another and with the intervention dimension. Context comprises seven domains (i.e., geographical, epidemiological, socio-cultural, socio-economic, ethical, legal, political); implementation consists of five domains (i.e., implementation theory, process, strategies, agents and outcomes); setting refers to the specific physical location, in which the intervention is put into practise. The intervention and the way it is implemented in a given setting and context can occur on a micro, meso and macro level. Tools to operationalise the framework comprise a checklist, data extraction tools for qualitative and quantitative reviews and a consultation guide for applicability assessments.
The CICI framework addresses and graphically presents context, implementation and setting in an integrated way. It aims at simplifying and structuring complexity in order to advance our understanding of whether and how interventions work. The framework can be applied in systematic reviews and HTA as well as primary research and facilitate communication among teams of researchers and with various stakeholders.
The effectiveness of complex interventions, as well as their success in reaching all relevant target populations, is critically influenced by their implementation in a given context; indeed, effectiveness, implementation and context are inextricably linked [1, 2]. To date, however, limited information on implementation and contextual factors is reported in primary studies. Likewise, systematic reviews and health technology assessments (HTA) fail to capture context and implementation in appropriate ways, which constitutes a major barrier to appraising transferability and applicability of findings .
Insufficient understanding of context and implementation thus contributes to the critical gap between research and practice. Policy makers and practitioners are often challenged with making decisions in relation to the evaluation and implementation of complex interventions. Complex interventions usually comprise multiple components, which may act independently or interdependently, with the ‘active ingredient(s)’ being difficult to specify . Moreover, the boundaries between what constitutes the intervention, its implementation and context are often blurred [2, 4], with interactions taking place between all three . These complex interventions challenge current approaches to the conceptualisation and assessment of the intervention and the way it is implemented in context. For example, a low emission zone to reduce particulate matter air pollution and related health impacts comprises several necessary and interacting components, i.e., regulatory components (e.g., the passing of a law and its enforcement), infrastructure components (e.g., signposting) and educational components (e.g., information through mass media and other channels). A low emission zone is not implemented within one particular organisation or sector but in entire cities, such as Munich [6, 7] or London [8, 9], often requiring cooperation across multiple sectors (e.g., transport, urban development, environment, health). Relevant contextual factors include geographical aspects (e.g., existing public transport infrastructure) and socio-economic aspects (e.g., disposable income for purchasing a newer car). Importantly, this intervention is not taking place in isolation, but its impacts on particulate matter air pollution and health may be compromised or enhanced by other ongoing interventions implemented nationally (e.g., a national guideline regulating industry emissions) or locally (e.g., incentives to promote walking and cycling). Understanding whether such an intervention really makes a difference requires a conceptualisation of the intervention within the system and an appropriate assessment of context and implementation.
Numerous frameworks, models and theories for assessing implementation have been developed, together with a smaller number to assess context. In 2015, Nilsen provided a taxonomy to differentiate between categories of theories, models and frameworks in order to facilitate the appropriate selection and application of these theoretical approaches . This taxonomy distinguishes between three overarching aims: describing and/or guiding the process of translating research into practise (process models); understanding and/or explaining what influences implementation outcomes (determinant frameworks, classic theories, implementation theories) and approaches facilitating the evaluation of an implementation effort (evaluation frameworks) . Widely cited approaches, such as the Reach Effectiveness Adoption Implementation Maintenance (RE-AIM) framework or the PRECEDE-PROCEED framework  can thus be considered evaluation frameworks, whereas the Stetler Model or the Quality Implementation Framework  can be classified as process models. In contrast, the Promoting Action on Research Implementation in Health Services (PARiHS) framework , the Consolidated Framework for Implementation Research (CFIR)  or the Conceptual Model by Greenhalgh  represent determinant frameworks. According to Nilsen, classic theories, such as the Theory of Diffusion of Innovations , various social cognitive and social network or organisational theories, have been developed in psychology, sociology or organisational sciences with a realm that goes beyond implementation but can usefully explain selected aspects of implementation . On the other hand, various implementation theories, such as the Implementation Climate Theory  or the Absorptive Capacity Theory , are directly concerned with implementation and regard behaviours that accompany or facilitate the implementation of evidence on an individual or community level [19–26]. These theories have been developed within the comparatively new field of implementation sciences, either de novo or by modifying existing theories .
Most existing frameworks, models and theories are primarily concerned with implementation, while context plays a minor role. The two aspects are rarely assessed in an integrated way, although selected frameworks have attempted to do so (e.g., [14, 27]). Importantly, uses of the terms ‘context’ and ‘implementation’ and their meanings vary widely in the health literature, lacking consensual definitions and descriptions as well as clearly delineated boundaries. Context is often used synonymously with setting and environment [13, 28], embracing static (e.g., physical environment) and dynamic aspects (e.g., relationships, networks) as well as the theory underpinning the intervention and its implementation . Implementation is considered to be a process, a constellation of processes, efforts or the means or methods of fitting, assimilating or putting into use an intervention—either evidence-based or theory-based—in an organisation or a setting [14, 30, 31]. Moreover, many of these frameworks, theories and models are specific rather than generic, e.g., focussing only on organisational context, and therefore do not lend themselves to application across diverse health interventions. Furthermore, few of these frameworks appear suitable for use within systematic reviews and HTAs. Indeed, many of these frameworks, models and theories are relatively abstract in nature and lack guidance on how to operationalise them in practise.
Our objective was, therefore, to develop a framework to facilitate structured and comprehensive conceptualisation and assessment of context and implementation of complex health interventions. We constructed the Context and Implementation of Complex Interventions (CICI) framework for use in systematic reviews and HTAs. However, given that primary research, evidence synthesis and evidence-based policy and practise are inextricably linked , the framework is likely to be equally relevant for primary research.
The CICI framework was developed as part of the EU-funded project Integrated health technology assessment for the evaluation of complex technologies (INTEGRATE-HTA) (www.integrate-hta.eu). The overall aim of the project was to develop concepts and methods for HTA to enable a patient-centred, integrated assessment of the effectiveness, the economic, social, cultural, legal, and ethical aspects of complex health technologies, which take context and implementation into account. The CICI framework was developed in a three-step process, each involving a degree of overlap and iteration (Fig. 1).
In the first step, we undertook purposive literature searches for models, theories and frameworks concerned with context and/or implementation. We critically examined the publications identified in this way and, using our understanding of complex interventions within complex systems, we built an initial framework. In parallel, methods for logic models  as well as methods to undertake an assessment of effectiveness, economic, socio-cultural, ethical and legal aspects were developed within INTEGRATE-HTA. Starting in step 1 and continuing, we aimed to come to agreement across the project regarding the domains of the CICI framework as well as the aspects covered by each domain, so that all methods could be used in a coherent and complementary fashion (Fig. 1). As a research team, we applied the initial framework within three rapid assessments of complex interventions (i.e., improved household stoves and fuels for developing countries, specialist palliative care and e-learning interventions to increase evidence-based healthcare competencies in healthcare professionals). These rapid assessments explored whether the broad principles of the framework might apply across very different types of health interventions. The rapid assessments unveiled inconsistencies in the interpretation of the terms context and implementation and the characteristics assigned to each of these among members of the research team.
Consequently, we examined the conceptual maturity of both concepts (i.e., context and implementation). A mature concept is well defined and clearly described and has delineated boundaries as well as documented pre-conditions and outcomes . To explore and advance conceptual maturity, we chose pragmatic utility (PU) concept analysis [34–36]. PU concept analysis comprises three major steps, i.e., selection of literature, organisation and structuring of the selected literature and asking analytical questions of the literature. In step 1, we conducted systematic searches to identify a comprehensive set of publications that describe relevant models, theories and frameworks of context and implementation. For this purpose, we conducted two separate searches, a standard systematic search for context in EMBASE and MEDLINE and an innovative forward tracking search for implementation in Google Scholar. The latter was based on the landmark review by Damschroder et al. (2009) and identified any publication citing either Damschroder et al. (2009) or the individual models, theories or frameworks included therein. This resulted in considerably improved efficiency of our searches without a significant loss of information . In step 2, we structured the literature according to the field of application, the field in which the concepts were originally developed, definitions, characteristics as well as contained model(s) (e.g., Consolidated Framework for Advancing Implementation Research (CFIR)) whenever an extension or refinement of an existing framework was undertaken. Finally, in step 3, we inductively and deductively developed questions that were asked of the literature. These questions included, for example, the interactions between the concepts of context, implementation and the implementation agents, the influence of time and the conceptualisation of an implementation success. These questions were answered in order to reveal potential inconsistencies in the conceptualisation of terms. Both concepts (context and implementation) were also analysed in terms of their conceptual maturity, comprising definitions, characteristics, described pre-conditions and outcomes, and boundaries. More details on the concept analysis are published elsewhere . Based on the findings of the concept analysis, we revised the initial framework.
The revised framework underwent extensive application. As part of the INTEGRATE-HTA project, a demonstration HTA of reinforced home-based palliative care (rHBPC) (i.e., home care with an additional element of carer support) was undertaken. The same revised framework was applied by different researchers, both within and outside of INTEGRATE-HTA. It was used (i) as part of the system-based logic model used in scoping the demonstration HTA (JB, AR, LB, LP, PW, AG, ER, KBL, KM and others) ; (ii) as the so called best-fit framework  in a qualitative systematic review (LP, AB, ER, LB and others) ; (iii) as the basis for a data extraction tool in a systematic review of effectiveness (JB, SP, LP, ER and others) [40–42]; and (iv) as the structure for an applicability assessment (SP, LP, AG, ER and others) (Polus et al., submitted manuscript), with most of these described in more detail below.
For the application of the revised framework in quantitative systematic reviews, we developed a tool for inclusion within the respective data extraction forms, with questions corresponding to the domains of the framework. This data extraction tool was used within a systematic review assessing the effectiveness of home-based palliative care undertaken as part of INTEGRATE-HTA (Burns et al., submitted manuscript), as well as a review on the effectiveness of interventions to reduce lead in consumer products and drinking water  and a Cochrane review of interventions to reduce ambient air pollution .
In a qualitative systematic review of contextual enablers and barriers to the implementation of home-based palliative care , we used the revised framework as a best-fit framework, meaning that findings emerging deductively from the data were structured according to the framework; where findings did not fit, the framework was modified inductively. To facilitate this process, we developed a further data extraction tool based on domains of the framework, with three questions to represent each of the domains.
We also applied the revised framework within an applicability assessment, which served to examine whether the findings of the demonstration HTA would be applicable in distinct decision-making settings. To identify contextual and implementation factors affecting applicability, we conducted semi-structured consultations with palliative care experts from three European countries. The domains of the revised framework provided the structure for the consultation guide, which included an open question on each domain (Polus et al., submitted manuscript).
Drawing on recognised approaches for examining applicability, coherence, completeness, usefulness and ease of application [43, 44], we compiled seven criteria (Table 1) to guide our appraisal of whether the framework could actually deliver on what we had set out to achieve, i.e., to facilitate the structured and comprehensive conceptualisation and assessment of context and implementation of complex health interventions. We collected feedback on the above from all of the researchers who had participated in the various applications, either verbally or in writing. At the same time, the framework was externally peer-reviewed by three experts with intentions to apply the framework. These experts, characterised by different methodological backgrounds and activities across different fields (medical and social policy), provided detailed written feedback; in one case, this was pursued further through an in-depth discussion.
Based on lessons learnt from the multiple applications as well as the detailed feedback from the external peer reviewers, we modified the framework, resulting in the CICI framework presented here (Fig. 2).
We then developed a worked example by applying the framework to an exemplary complex intervention included in the abovementioned Cochrane review of interventions to reduce ambient air pollution: the Air Pollution Act on the Marketing, Sale and Distribution of Coal, was initially introduced in Ireland in 1987 and enacted in the city of Dublin from 1990. Since we were also aiming to demonstrate the added value of the CICI framework as compared to other established and widely used frameworks, we also applied the CFIR  and the PARiHS framework  to the same intervention. In populating the three frameworks, we started with the primary study included with our Cochrane review  and then conducted specific searches of the grey literature to identify relevant information in newspaper articles, government documents, city council reports as well as lobbyism reports.
Initial framework (step 1)
The initial framework comprised two dimensions—context and implementation—with eight domains (i.e., locational, geographical, epidemiological, socio-cultural, socio-economic, ethical, legal and political) and four domains (i.e., provider, organisation and structure, funding and policy), respectively. The intervention, informed by theory, lies at the heart of the CICI framework with its reach and effectiveness being affected by context and implementation.
Towards the revised framework (step 2)
Based on the systematic searches, we included 35 publications developing, proposing or describing theories, models or conceptual frameworks for implementation (see Additional file 1) and 17 publications doing the same for context (see Additional file 2) . The concept of implementation is situated within an advancing and well-connected field of implementation science. The concept of context, on the other hand, is less studied and applied in diverse ways across different fields, among them implementation science. Setting—a term that is sometimes used synonymously with context—emerged as an additional and distinct concept of interest. Our searches had not been designed to capture all relevant aspects, which precluded a formal concept analysis. The definitions of context, setting and implementation as advanced by concept analysis are presented in Table 2. The concept analysis led to a clear conceptual distinction between context and setting as well as a more comprehensive formulation of characteristics for each of the domains of context and implementation. These were incorporated in the revised framework.
Towards the CICI framework (step 3)
The revised framework was applied to different complex interventions using various methodological approaches (i.e., HTA, quantitative systematic reviews, qualitative systematic review, applicability assessment). We mapped feedback received from the researchers applying the revised framework in HTA, quantitative and qualitative reviews as well as an applicability assessment together with our own experiences against the criteria in Table 1. Overall, as summarised in Table 3, the revised framework showed good internal coherence and completeness, applicability and user-friendliness and was found to contribute to theory advancement and development. The test cases did, however, also reveal that selected aspects under specific domains of context and implementation had been missing.
In general, the peer reviewers felt that the guidance was useful and comprehensive but questioned whether the framework would be taken up in practise, given the additional workload required. They also expressed some concern with respect to the definition of the implementation domains. Related to this, they issued a collective call for a clearer distinction between the context and implementation dimensions and greater elaboration of definitions of individual domains. As a consequence, the implementation dimension was significantly revised. In contrast, the context dimension remained largely unchanged apart from the locational domain being subsumed within a new “setting” dimension.
The CICI framework comprises three dimensions—context, implementation and setting—which are described in detail in the following section. The context dimension comprises seven domains, i.e., geographical, epidemiological, socio-cultural, socio-economic, ethical, legal and political. The implementation dimension comprises five domains, i.e., implementation theory, implementation process, implementation strategies as well as implementation agents and outcomes. Setting refers to the specific physical location, in which the intervention is put into practise. In the setting dimension, context, implementation and the intervention, which are described according to intervention theory, design and delivery characteristics , interact with one another and often co-evolve over time. Clearly, the intervention itself is critical. This dimension and its three domains intervention theory, intervention design (including components and execution) as well as intervention delivery are described in Table 4; they have been defined and described in detail elsewhere . The CICI framework fundamentally builds on and is applied alongside this conceptualisation of the intervention but focuses on an operationalisation of context, setting and implementation.
Macro, meso or micro levels
Depending on the nature of the intervention, the interactions between intervention, implementation and context in a given setting can occur at a macro level (e.g., policies or regulations across a whole health system or country), meso level (e.g., introduction of new treatment guidelines in a specific hospital or of sanitation systems in a specific village) or micro level (e.g., promotion of health-protective behaviours among individuals or families). Interventions are typically implemented simultaneously within different settings and at multiple levels. It is, however, not necessarily useful or meaningful to conduct an analysis at all levels, and the relevant level will depend on the assessed intervention and the system, in which it exists. The macro level refers to everything surrounding a community or organisation ; this can include the regional, national or international environment. The meso level refers to a community or organisation . A community is defined by its function (e.g., employer, religious entity), geography (e.g., village, neighbourhood), shared interests or characteristics (e.g., ethnicity, culture) or a combination of these [48, 49], with members sharing a sense of identity or connection [50, 51]. An organisation is defined by its structure and size [52–55], organisational culture [14, 52, 56–62] and climate [14, 63], networks and relationships [64–66]. All of these community or organisational characteristics jointly influence implementation climate [14, 15], system readiness for change [14, 17, 67] and capacity for change [15, 62, 68, 69] and thus the chances of an intervention being implemented successfully. The micro level refers to the level of direct action, i.e., where the intervention is delivered by a specialised palliative care team at the patient’s home, in which a room has to be remodelled in order to deliver physiotherapy to a patient.
Apart from the intervention of direct interest, the context and the way the intervention of interest is implemented may be advanced or compromised by other interventions occurring independently. For example, an organisation promoting water flushing before tap water consumption to avoid exposure to lead in drinking water in a specific community (meso level) can be facilitated by an ongoing national campaign on the risks of lead contamination of drinking water (macro level). These other interventions are likely to influence the reach and effectiveness of the intervention under investigation in a specific setting.
The geographical context refers to the broader physical environment, landscapes and resources, both natural and transformed by humans (e.g., infrastructure), available in a given setting. The supply of rHBPC, for example, might be hindered by the availability of services for geographically isolated potential recipients of palliative care.
The epidemiological context refers to the distribution of diseases or conditions, the attributable burden of disease, as well as determinants of needs in populations , including demographics [70, 71]. Psychosocial and physical needs of lay caregivers of palliative care patients as well as the needs of palliative care patients themselves would fall into this category.
The socio-cultural context comprises explicit and implicit behaviour patterns, including their embodiment in symbols and artefacts; the essential core of culture consists of historically derived and selected ideas and values that are shared among members of a group . It not only refers to the conditions in which people are born, grow, live, work and age but also embraces the social roles a human being takes on as a family member, community member or citizen and the relationships inherent to these roles. Constructs such as knowledge, beliefs, conceptions, customs, institutions and any other capabilities and habits acquired by a group are included in this domain . An example could be families and communities that fulfil specific roles in the provision of home-based palliative care (HBPC). The wife of a palliative care patient could feel obliged to care for her husband due to social expectations; being an informal carer in HBPC may lead to loss of employment or income.
The socio-economic context comprises the social and economic resources of a community and the access of a population to these resources [14, 74]. This could, for example, comprise the potential loss of income among lay caregivers of palliative care patients as they enter the caregiver role.
The ethical context comprises reflections of morality, which encompasses norms, rules, standards of conduct and principles that guide the decisions and behaviour of individuals and institutions . Ethical, socio-cultural and legal aspects are strongly interrelated [76–78]. For example, ensuring autonomy and valid consent for patients receiving palliative care at home with their informal carers (both are care receivers) can be challenging given their different needs, preferences, legal position, and socio-cultural perspectives.
The political context focuses on the distribution of power, assets and interests within a population, as well as the range of organisations involved, their interests and the formal and informal rules that govern interactions between them . The domain also comprises the health care system and its accessibility (e.g., delivery of services, leadership and governance, health information, human resources and financing). For example, the increasing political pressure to ensure equitable access to palliative care for those in rural areas has led to greater interest in telemedicine . In this case, interactions with the geographical and the socio-cultural domains were observed.
The legal context is concerned with the rules and regulations that have been established to protect a population’s rights and societal interests . Formally, these have to be passed by a competent legislative body like a parliament. Legal norms can mostly be enforced with order and compulsion, which distinguishes them from ethical and social norms . A legal issue arising in HBPC on the micro level is, for example, the sharing of information with relatives who wish to be informed about the medical condition of the palliative care patient. This might contradict the legal framework, in which care is delivered as well as socio-cultural and ethical norms of information sharing.
An implementation theory attempts to explain the causal mechanisms of implementation ; it is therefore analogous to a programme theory, which attempts to explain the causal mechanisms linking an intervention and its outcomes. An implementation theory formalises how change needs to be executed in order for the implementation effort to be successful [70, 82, 83] and underpins both the implementation process and implementation strategies .
The implementation process refers to the social processes, through which interventions are operationalised in an organisation or community . It contains the tactics and methods used by change leaders [62, 82]. The implementation process is an active, multistage, iterative and dynamic process that does not usually occur in a linear fashion . At specific points, corrections, refinements or expansions [14, 82] are undertaken by implementation agents  in order to successfully implement an intervention.
The first stage is characterised by the exploration of organisational needs, intervention-organisational fit as well as capacity and readiness assessment in a given setting [12, 86]. Once the decision to adopt an intervention has been made [31, 53], structural changes can be undertaken in the setting in order to facilitate the implementation effort . This requires thorough planning and preparation, a stage during which specific implementation strategies, the engagement of implementation agents as well as the implementation process itself are planned [14, 53, 82]. This is followed by initial implementation [54, 62, 82, 87]. At this stage, staff should be educated and information disseminated; interventions can be pilot-tested and consequently adapted . Full implementation begins, when the intervention becomes integrated into practise at all levels and for all implementation agents and intervention recipients [54, 82]. At this stage, processes and procedures supporting the intervention execution are in place, and the system, although never completely stable, has largely been recalibrated to accommodate and support the intervention . The stage of evaluation and reflection aims to assess the process as well as the outcomes with reference to the intended goals, and to learn from the implementation for the setting in question, as well as for future implementation efforts in other settings [12, 14, 31, 62]. Evaluation and reflection often occur relatively late in the implementation process; however, ideally, this starts from the very beginning. The final stage relates to sustainment of a successfully implemented intervention and refers to the continued use of the intervention in the organisation or community [53, 54, 86].
The implementation process consists of specific implementation strategies, which encompass all methods and means to ensure the adoption and sustainment of interventions . Drawing on the implementation theory and epistemological considerations, implementation strategies comprise a set of activities that are chosen and tailored to fit a specific context [52, 88] or to create such a context . These may change over time. Implementation strategies can contain multiple components and as such may be considered an intervention in its own right. Implementation strategies should be named, preferably according to the literature in order to identify similar efforts , and should be described in terms of their components . Furthermore, implementation strategies can be described in terms of their theory, the implementation agents, the concrete action, the action target, their temporality, their dose as well as the implementation outcome(s) affected .
Implementation agents comprise all individuals and organisations engaged with (i) deciding to implement a given intervention (e.g., funders, administrators), (ii) implementing this intervention (e.g., providers, advocates, physicians, nurses) or (iii) being the target or otherwise affected by an intervention (e.g., patients and their families, consumers) . These implementation agents can be located inside or outside (e.g., external change agents) of the organisation, in or through which an intervention is implemented. These individuals have particular personality attributes, skills, knowledge, beliefs as well as attitudes that exert their influence on the implementation of an intervention. The success of implementation is highly dependent on the buy-in of individuals who become key stakeholders in both the intervention and the implementation effort . Thus, stakeholders should be carefully involved with the planning, execution and evaluation of the implementation effort. An implementation effort can be promoted and facilitated by single implementation leaders [14, 15] and/or implementation teams .
Individual implementation agents, or teams that are actively involved in funding, administering or implementing an intervention, are usually embedded in an organisation or community that critically influences their attitudes and behaviours (see description of meso level above). This organisational or community level determines the overall implementation climate and capacity for change and thus the ability of individual implementation agents to contribute to an implementation effort. As such, it plays a significant role in implementing an intervention successfully. This is especially true, where one organisation or group within an organisation is selected among several alternatives, or where a new organisation is created to facilitate implementation.
An implementation outcome is the result or implication of the implementation effort and forms part of good monitoring and evaluation practises. Important implementation outcomes are adoption, uptake, acceptability, implementation cost, penetration, sustainability [14, 89] and dissemination to other contexts .
Operationalisation of the CICI framework
CICI framework tools
To facilitate the pragmatic application of the CICI framework, we developed a generic checklist (Table 5). This checklist can be modified with respect to a specific intervention and the level(s), at which its implementation is to be assessed (i.e., macro, meso, micro); it can and should also be adapted and refined towards specific uses in primary research, evidence synthesis and evidence-based policy and practise. The checklist comprises questions regarding which factors of a respective dimension (i.e., context, implementation, setting) exert their influence, and how this influence affects implementation success and, ultimately, intervention effectiveness. A table describing the aspects within the domains can be found in Additional file 3.
When used in a systematic review or HTA, the CICI framework can be operationalised through the use of a data extraction tool. Generic data extraction tools for qualitative and quantitative systematic reviews, as applied in several systematic reviews, are available (see Additional files 4 and 5). The expert consultation guide is also accessible for use (see Additional file 6).
Lessons learnt from a worked example
The worked example of the CICI framework, available in Additional file 7 and also including a simple logic model of the intervention, demonstrates that the framework can be readily applied to a complex public health intervention that operates across multiple settings and engages multiple implementation agents across different organisations and sectors. Issues identified in scientific articles and grey literature documents could easily be categorised under relevant domains, and the CICI framework also facilitated explicit consideration of potential interactions occurring between domains and dimensions and across various levels. These interactions were listed in our tabular presentation of findings and can thus be made transparent. It should be noted, however, that a more in-depth analysis of interactions would necessitate a thorough textual presentation.
The assessment of context shed light on factors that influence implementation success and, ultimately, the effectiveness of the intervention in reducing ambient particulate matter air pollution and associated mortality; for example, the lack of a strong coal lobby and collaboration across different ministries positively influenced implementation, whereas historically grown preferences for the use of solid fuels for heating and concerns about the loss of jobs in the coal industry initially presented challenges.
The CICI framework accommodated with ease the consideration that the intervention was implemented in more than one setting: the Irish government itself, the local authority that is obliged to adopt and enforce the regulation, the coal vendor that needs to register with the Environmental Protection Agency as well as the single household within the enforcement zone that reacts by changing to a different fuel or heating system. All of these settings involve different implementation agents, require different implementation strategies and allow the assessment of different implementation outcomes.
In contrast, we had difficulties with applying the CFIR and PARiHS to our intervention of choice, as both assume that those delivering and those receiving the intervention (e.g., doctors, nurses) are nested within the organisation implementing the change (e.g., hospitals, health centres); consequently, the detailed assessment of context exclusively relates to this organisational context. Instead, for public health/complex interventions, there is often an organisational or even sectoral disconnection between the implementing organisation(s) and those receiving the intervention, and context is concerned with the society at large. Consequently, as we applied the CFIR and PARiHS to the Air Pollution Act, many relevant aspects of context and implementation were not captured at all, while a substantial proportion of the CFIR and PARiHS domains remained unpopulated. Importantly, from our application of the CICI framework, we learnt that context as a versatile, dynamic construct evolves over time, as the intervention unfolds and as implementation strategies change, e.g., the issuing of additional taxes or grants. A further difficulty related to the terminology used, where terms used by the CFIR and PARIHS frameworks (e.g., patients) are grounded in a clinical and/or health services perspective and only partially applicable to public health interventions.
Numerous frameworks, theories and models assessing implementation and, to a lesser extent, context have been published over the past decades. Few of these frameworks address implementation and context in an integrated fashion, and very few offer pragmatic guidance and worked examples of how to apply the framework, theory or model in practise. To our knowledge, the CICI framework is the first attempt to provide guidance for assessing setting, context and implementation of complex interventions in a comprehensive manner, for integrating the views of different disciplines (economics, ethics, sociology, law) and for adapting a concept of context that is embedded in a broad public health perspective rather than a narrower organisational perspective. Through systematic literature searches and concept analysis, the CICI framework builds on previous knowledge and presents a strong theoretical basis; through empirical applications across several distinct complex health interventions, it has proven its value across a range of different methodological approaches (e.g., qualitative and quantitative systematic reviews, primary qualitative research). It can serve both as a determinant framework that seeks to conceptualise, describe and understand the multiple influences on implementation outcomes, and as an evaluation framework that clarifies the context, setting and implementation aspects to be assessed when examining implementation success or lack thereof. The CICI framework includes step-by-step pragmatic guidance, a generic checklist to be adapted for different purposes and data extraction tools as well as a graphical representation to facilitate its application in practise.
Theoretical underpinnings and scope
Previously published theories, frameworks or models, as identified from our systematic searches are based on different theoretical underpinnings, although these are often not made explicit. We are aware of at least two other concept analyses of context . McCormack and colleagues looked at context as part of the Promoting Action on Research Implementation in Health Services (PARiHS) framework ; Squires and colleagues have not yet published their concept analysis . To our knowledge, no concept analysis of implementation has been undertaken to date.
Our concept analysis, which was a critical building block for the development of the CICI framework, attempted to capture all relevant aspects of context, implementation and setting, keeping a complex interventions perspective in mind. In terms of previous frameworks, models or theories, Damschroder’s Consolidated Framework for advancing Implementation Research, Greenhalgh’s landmark review, Roger’s Diffusion of Innovation Theory, and Klein and Sorra’s Theory of Innovation  and Ajzen’s Theory of Planned Behaviour were particularly influential in the development of the CICI framework. Integrating insights gained in previously published frameworks, theories or models, it adds a macro perspective that seeks to capture all aspects of complex interventions. In moving beyond the organisational context, the framework is particularly suitable for public health and other interventions that are assessed or analysed on a macro level or on several levels concurrently. Previously published approaches primarily focus on implementation, although the details of implementation theory, implementation strategy or a comprehensive implementation process are often neglected (e.g., [14, 53, 84, 85]). Context often plays a subordinate role; where it is considered explicitly, it is usually restricted to an organisational context [27, 44, 91]. As demonstrated in the worked example, for complex interventions context extends much beyond the organisational context due to the number of settings, in which the intervention is delivered, as well as the number of implementation agents and strategies active across different sectors. Moreover, the ethical, socio-economic or epidemiological context, concepts that are regularly used in HTA and are known to have a considerable impact on the uptake, reach and effectiveness of an intervention, are rarely considered. The CICI framework highlights this breadth and depth of the influence of context from a societal perspective and shows that context can act at one or several different levels (micro, meso and macro), thereby adding granularity and flexibility to the assessment of a complex intervention. Lastly, the CICI framework pays tribute to competing interventions that might have a considerable impact on the uptake, reach and effectiveness of the intervention being investigated.
Over the last few decades, considerable work has been undertaken to elaborate and define elements of organisational context as well as behaviours that influence an implementation effort (REFS). This work is referenced throughout the relevant domains, and, where applicable to the intervention in question, we would advise users of the CICI framework to thoroughly examine the constructs and sub-constructs elaborated in these other frameworks. For example, respective subdomains within the implementation strategy domain of the CICI framework align with the work of Proctor and colleagues . In this way, the CICI framework can serve as a comprehensive starting point for examining context and implementation; where useful, granularity within specific domains can be added by bringing in insights from previously published frameworks.
As stated repeatedly throughout this paper, an intervention and its implementation interact in multiple ways within any given setting and context, with these interactions taking place at multiple levels. Going further, it may be argued that implementation and context are constitutive parts of the intervention itself. Therefore, any structural separation of these dimensions may be considered somewhat artificial but, at the same time, is necessary in order to facilitate a structured description and a feasible assessment. As noted before, it is hard to fully account for the complex and interdependent relationships between the various domains of context and implementation, although we found that these can be tracked and made transparent using the tools presented in this paper. It is important to note that these interact in a complex adaptive system, where significant changes at a micro, meso or macro level can become manifest over time . We encourage researchers conducting systematic reviews or HTA to use the CICI framework in conjunction with a system-based logic model , since both were developed in a coherent manner as two complementary tools. The logic model is designed to facilitate a detailed description of the theory, design and delivery of an intervention and to take into account the interactions between the intervention, its implementation and the surrounding system; the latter is the focus of the CICI framework. Worked examples of combined applications of both tools are the systematic reviews on HBPC, interventions to reduce lead in consumer products and drinking water as well as interventions to reduce ambient air pollution [40–42].
Need for detailed assessment and reporting
The success in populating the dimensions and domains of the CICI framework with evidence critically depends on detailed assessment and reporting of information in primary studies. In our various applications of the framework as well as in the worked example, a significant problem was that primary studies reported little detail on context and implementation; this was less problematic when using primary qualitative or mixed-method studies. Unfortunately, our experience confirms what is known from the literature: reporting of information on context and implementation of health interventions is consistently poor . While many of the widely used reporting guidelines, such as STROBE, SQUIRE, TREND and the extension of CONSORT for Pragmatic Trials, acknowledge context, most only require researchers to report on the setting. More specific reporting guidelines, such as the Reporting guidelines for implementation and operational research, the revised Criteria for Reporting the Development and Evaluation of Complex Interventions in healthcare (CReDECI 2) , the REporting of studies Conducted using Observational Routinely collected Data statement (RECORD)  and Developing Standards for Reporting Phase IV Implementation studies (StaRI)  provide helpful guidance on how to report implementation. Additional information regarding context can moreover be found in process evaluations that attest to the evolving, adaptive dynamics of context .
Methodological strengths and limitations
The combination of systematic searches to identify existing frameworks, theories and models of context and implementation and formal methods to derive definitions and detailed characteristics of the relevant concepts provides a strong theoretical basis for the CICI framework and constitutes one of its major strengths. Some limitations pertaining to the searches are described elsewhere, for example the lack of explicit searches in databases of management and organisational studies . It must also be noted that the setting dimension is less well conceptualised, mostly because our search strategy was designed to yield insights into the concepts of context and implementation. The realisation that setting, which plays a critical role in many fields, in particular in health promotion, should be treated as a separate dimension only became apparent in step 3. While the literature tends to conceptualise setting as having physical boundaries , this may fall short of acknowledging developments during the last decade (e.g., telemedicine, Internet- or smart phone-based interventions). Importantly, we did not employ any formal method of consensus building, such as a Delphi approach. The iterative development of the framework through regular interactions with other methodological work packages across the INTEGRATE-HTA project and external peer review of the framework cannot compensate entirely for this but demonstrates good general agreement with the principles of the framework from a broad group of experts across many different disciplines.
Another considerable strength of the CICI framework is that it underwent extensive testing across different types of interventions and using a range of methodological approaches (i.e., HTA, effectiveness reviews, qualitative review and applicability assessment). While the CICI framework was developed as a generic framework with broad applicability and flexibility, it is not intended to be a straitjacket: depending on the intervention being examined and the scope of the assessment, the CICI framework can focus on selected aspects or domains of the context, implementation and setting dimensions and thus can be tailored to the needs imposed by the intervention or research question. In summary, according to the feedback received, the framework performed well against seven pre-specified criteria. A limitation is, however, that the latest version of the framework has not yet been fully tested. We are in the process of applying the CICI framework in several of our own primary research and evidence synthesis projects and will critically examine its performance. In addition, we would greatly value feedback from others applying the framework and suggestions on how to make it more useful.
As the complexities that emerge in relation to implementing effective health interventions become ever more apparent, it becomes increasingly important to undertake the systematic conceptualisation and assessment of context and implementation. Complexity is not only inherent in the intervention but also a consequence of interactions between the intervention and its implementation in context. The CICI framework constitutes one way of structuring this complexity in order to advance our understanding of whether and how interventions work, while keeping in mind that it is impossible to look for and discover everything . When used in conjunction with a logic model, the CICI framework can help researchers and research organisations that undertake systematic reviews and HTAs, such as Cochrane and the European network for HTA (EUnetHTA), make sense of complexity and develop, prioritise and hopefully answer some of the many questions that arise when implementing complex interventions. Operationalised as a checklist and/or graphical representation, the framework can help researchers and research organisations to communicate with the policy and practise audiences they intend to reach. Finally, while developed primarily for evidence synthesis, the usefulness of the framework depends on the evidence available. Therefore, we hope that the scientific community will support adoption of the CICI framework for primary research, where it may be used for conceptualising interventions in complex systems taking the influence of setting, context and implementation on intervention reach and effectiveness into account.
Consolidated framework for implementation research
Context and Implementation of Complex Interventions
Consolidated Standard of Reporting Trials
- CReDECI 2:
Criteria for Reporting the Development and Evaluation of Complex Interventions in healthcare
European network for health technology assessment
Home-based palliative care
Health technology assessment
Promoting Action on Research Implementation in Health Services
Practical, Robust Implementation and Sustainability Model
Reach Effectiveness Adoption Implementation Maintenance framework
REporting of studies Conducted using Observational Routinely collected health Data
Reinforced home-based palliative care
Standards for Reporting Phase IV Implementation studies
STrengthening the Reporting of OBservational studies in Epidemiology
Transparent Reporting of Evaluations with Nonrandomized Design
World Health Organisation
Waters E, Hall BJ, Armstrong R, Doyle J, Pettman TL, de Silva-Sanigorski A. Essential components of public health evidence reviews: capturing intervention complexity, implementation, economics and equity. J Public Health (Oxf). 2011;33:462–5.
Wells M, Williams B, Treweek S, Coyle J, Taylor J. Intervention description is not enough: evidence from an in-depth multiple case study on the untold role and impact of context in randomised controlled trials of seven complex interventions. Trials. 2012;13:95.
Medical Research Council. A framework for development and evaluation of RCTs for complex interventions to improve health. London: Medical Research Council; 2000.
Van Herck P, Vanhaecht K, Deneckere S, Bellemans J, Panella M, Barbieri A, Sermeus W. Key interventions and outcomes in joint arthroplasty clinical pathways: a systematic review. J Eval Clin Pract. 2010;16:39–49.
Rychetnik L, Frommer M, Hawe P, Shiell A. Criteria for evaluating evidence on public health interventions. J Epidemiol Community Health. 2002;56:119–27.
Morfeld PSR, Builtjes PJH, Groneberg DA, Spallek M. Introduction of a low-emission zone and the effect on air pollutant concentration of particulate matter (PM10)—a pilot study in Munich. Zbl Arbeitsmed. 2013;63:104–15.
Fensterer V, Küchenhoff H, Maier V, Wichmann H-E, Breitner S, Peters A, Gu J, Cyrys J. Evaluation of the impact of low emission zone and heavy traffic ban in Munich (Germany) on the reduction of PM10 in ambient air. Int J Environ Res Public Health. 2014;11:5094.
Atkinson RW, Barratt B, Armstrong B, Anderson HR, Beevers SD, Mudway IS, Green D, Derwent RG, Wilkinson P, Tonne C, Kelly FJ. The impact of the congestion charging scheme on ambient air pollution concentrations in London. Atmos Environ. 2009;43:5493–500.
Tonne C, Beevers S, Armstrong B, Kelly F, Wilkinson P. Air pollution and mortality benefits of the London Congestion Charge: spatial and socioeconomic inequalities. Occup Environ Med. 2008;65:620–7.
Nilsen P. Making sense of implementation theories, models and frameworks. Implement Sci. 2015;10:53.
Green L, Kreuter M. Health program planning: an educational and ecological approach. New York: McGrawhill; 2005.
Meyers DC, Durlak JA, Wandersman A. The Quality Implementation Framework: a synthesis of critical steps in the implementation process. Am J Community Psychol. 2012;50:462–80.
Kitson A, Harvey G, McCormack B. Enabling the implementation of evidence based practice: a conceptual framework. Quality in Health Care. 1998;7:149–58.
Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci. 2009;4.
Greenhalgh T, Robert G, Macfarlane F, Bate P, Kyriakidou O. Diffusion of Innovations in Service Organizations: systematic review and recommendations. Milbank Q. 2004;82:581–629.
Rogers EM. Diffusion of innovations. 5th ed. Free Press; 2003.
Klein KJ, Sorra JS. The challenge of innovation implementation. Acad Manag Rev. 1996;21:1055–80.
Zahra SA, George G. Absorptive capacity: a review, reconceptualization, and extension. Acad Manag Rev. 2002;27:185–203.
Mathisen GE, Einarsen S, Jorstad K, Bronnick KS. Climate for work group creativity and innovation: Norwegian validation of the team climate inventory (TCI). Scand J Psychol. 2004;45:383–92.
Goh SC, Quon TK, Cousins JB. The organizational learning survey: a re-evaluation of unidimensionality. Psychol Rep. 2007;101:707–21.
Lubomski LH, Marsteller JA, Hsu YJ, Goeschel CA, Holzmueller CG, Pronovost PJ. The team checkup tool: evaluating QI team activities and giving feedback to senior leaders. Jt Comm J Qual Patient Saf. 2008;34:619–23. 561.
Duckers ML, Wagner C, Groenewegen PP. Developing and testing an instrument to measure the presence of conditions for successful implementation of quality improvement collaboratives. BMC Health Serv Res. 2008;8:172.
Good L, Nelson D. Effects of person-group and intragroup attitude similarity on perceived group attractiveness and cohesiveness. Psychon Sci. 1971;25:215–7.
Marsick VJ, Watkins KE. Demonstrating the value of an organization’s learning culture: the dimensions of the learning organization questionnaire. Adv Dev Hum Resour. 2003;5:132–51.
Helfrich CD, Li YF, Mohr DC, Meterko M, Sales AE. Assessing an organizational culture instrument based on the Competing Values Framework: exploratory and confirmatory factor analyses. Implement Sci. 2007;2:13.
Fineout-Overholt E, Melnyk B. Organizational Culture and Readiness for System-Wide Implementation of EBP (OCR-SIEP) scale. Gilbert: ARCC llc Publishing; 2006.
Stetler CB, Damschroder LJ, Helfrich CD, Hagedorn HJ. A guide for applying a revised version of the PARIHS framework for implementation. Implement Sci. 2011;6.
McCormack B, Kitson A, Harvey G, Rycroft-Malone J, Titchen A, Seers K. Getting evidence into practice: the meaning of ‘context’. J Adv Nurs. 2002;38:94–104.
Pawson R, Greenhalgh T, Harvey G, Walshe K. Realist review—a new method of systematic review designed for complex policy interventions. J Health Serv Res Policy. 2005;10 Suppl 1:21–34.
Rabin BA, Brownson RC, Haire-Joshu D, Kreuter MW, Weaver NL. A glossary for dissemination and implementation research in health. J Public Health Manag Pract. 2008;14:117–23.
Simpson DD. A framework for implementing sustainable oral health promotion interventions. J Public Health Dent. 2011;71:S84–94.
Rehfuess EA, Durao S, Kyamanywa P, Meerpohl JJ, Young T, Rohwer A, Cebha, consortium. An approach for setting evidence-based and stakeholder-informed research priorities in low- and middle-income countries. Bull World Health Organ. 2016;94:297–305.
Rohwer A, Pfadenhauer LM, Burns J, Brereton L, Gerhardus A, Booth A, Oortwijn W, Rehfuess EA. Logic models help make sense of complexity in systematic reviews and health technology assessments. J Clin Epidemiol. 2016.
Morse JM, Mitcham C, Hupcey JE, Tason MC. Criteria for concept evaluation. J Adv Nurs. 1996;24:385–90.
Morse J, Hupcey J, Mitcham C, Lenz E. Choosing a strategy for concept analysis in nursing research: moving beyond Wilson. In: Gift AG, editor. Clarifying concepts in nursing research. New York: Springer; 1997. p. 73–96.
Morse JM. Exploring pragmatic utility: concept analysis by critically appraising the literature. In: Rodgers BL, Knafl K, editors. Concept development in nursing: foundations, techniques, and applications. 2nd ed. Philadelphia: W.B. Saunders; 2000. p. 333–52.
Pfadenhauer LM, Mozygemba K, Gerhardus A, Hofmann B, Booth A, Lysdahl KB, Tummers M, Burns J, Rehfuess EA. Context and implementation: a concept analysis towards conceptual maturity. Z Evid Fortbild Qual Gesundhwes. 2015;109:103–14.
Brereton L, Wahlster P, Lysdahl KB, Mozygemba K, Burns J, Chilcott JB, Ward S, Brönneke JB, Tummers M, Van Hoorn R, Van Der Wilt GJ, Pfadenhauer L, Gerhardus A, Rowher A, Rehfuess E, Oortwijn W, Refolo P, Sacchini D, Leppert W, Blazeviciene A, Preston L, Clark J, Goyder E, on behalf of the INTEGRATE-HTA team. Integrated assessment of home based palliative care with and without reinforced caregiver support: ‘a demonstration of INTEGRATE-HTA methodological guidances’. 2016.
Carroll C, Booth A, Cooper K. A worked example of “best fit” framework synthesis: a systematic review of views concerning the taking of some potential chemopreventive agents. BMC Med Res Methodol. 2011;11:29.
Burns J, Boogaard H, Turley R, Pfadenhauer LM, van Erp AM, Rohwer AC, Rehfuess E. Interventions to reduce ambient particulate matter air pollution and their effect on health (Protocol). Cochrane Database Syst Rev. 2014; Issue 1. Art. No.: CD010919. doi:10.1002/14651858.CD010919.
Burns J, Polus S, Brereton L, Ward S, Chilcott J, Pfadenhauer L, Rehfuess E. Effectiveness of home-based and reinforced home-based palliative care: a systematic review with semi-structured expert consultations. in submission process 2016.
Pfadenhauer LM, Burns J, Rohwer A, Rehfuess EA. Effectiveness of interventions to reduce exposure to lead through consumer products and drinking water: a systematic review. Environ Res. 2016;147:525–36.
Ostrom E. An assessment of the institutional analysis and development framework. In: Sabatier PA, editor. Theories of the policy process. 1st ed. Colorado: Westview Press; 1999.
Kitson AL, Rycroft-Malone J, Harvey G, McCormack B, Seers K, Titchen A. Evaluating the successful implementation of evidence into practice using the PARiHS framework: theoretical and practical challenges. Implementation science : IS. 2008;3:1–1.
Dockery DW, Rich DQ, Goodman PG, Clancy L, Ohman-Strickland P, George P, Kotlov T, Committee HEIHR. Effect of air pollution control on mortality and hospital admissions in Ireland. Res Rep Health Eff Inst. 2013;3–109.
Rohwer A, Pfadenhauer LM, Burns J, Brereton L, Gerhardus A, Booth A, Oortwijn W, Rehfuess EA: Logic models help make sense of complexity in systematic reviews and health technology assessments. J Clin Epidemiol. 2016, in press.
Mendel P, Meredith LS, Schoenbaum M, Sherbourne CD, Wells KB. Interventions in organizational and community context: a framework for building evidence on dissemination and implementation in health services research. Adm Policy Ment Health. 2008;35:21–37.
Fellin P. Understanding American communities. In: Rothamn J, Erlich J, Tropman J, editors. Strategies of community organizations. Itasca: Peacock; 1995.
Scott W. Conceptualizing organizational fields: Linking organizations and societal systems. In: Derlien H, Gerhardt U, Scharpf F, editors. Systemrationalitat und Partial Interesse [System Rationality and Partial Interests]. Baden Baden: Nomos Verlagsgesellschaft; 1994. p. 203–21.
Schulz AJ, Israel BA, Lantz P. Instrument for evaluating dimensions of group dynamics within community-based participatory research partnerships. Evaluation and Program Planning. 2003;26:249–62.
DiMaggio PJ, Powell WW. The iron cage revisited: institutional isomorphism and collective rationality in organizational fields. Am Sociol Rev. 1983;48:147–60.
Aarons GA, Fettes DL, Hurlburt MS, Palinkas LA, Gunderson L, Willging CE, Chaffin MJ. Collaboration, negotiation, and coalescence for interagency-collaborative teams to scale-up evidence-based practice. J Clin Child Adolesc Psychol. 2014;43:915–28.
Aarons GA, Hurlburt M, Horwitz SM. Advancing a conceptual model of evidence-based practice implementation in public service sectors. Adm Policy Ment Health. 2011;38:4–23.
Fixsen DL, Blase KA, Naoom SF, Wallace F. Core implementation components. Res Soc Work Pract. 2009;19:531–40.
Tomoaia-Cotisel A, Scammon DL, Waitzman NJ, Cronholm PF, Halladay JR, Driscoll DL, Solberg LI, Hsu C, Tai-Seale M, Hiratsuka V, et al. Context matters: the experience of 14 research teams in systematically reporting contextual factors important for practice change. Ann Fam Med. 2013;11 Suppl 1:S115–23.
Bergstrom A, Peterson S, Namusoko S, Waiswa P, Wallin L. Knowledge translation in Uganda: a qualitative study of Ugandan midwives’ and managers’ perceived relevance of the sub-elements of the context cornerstone in the PARIHS framework. Implement Sci. 2012;7:117.
Helfrich CD, Li Y-F, Sharp ND, Sales AE. Organizational readiness to change assessment (ORCA): Development of an instrument based on the Promoting Action on Research in Health Services (PARIHS) framework. Implement Sci. 2009;4.
Bandura A. Self-efficacy: toward a unifying theory of behavioral change. Psychol Rev. 1977;84:191–215.
Glisson C, Landsverk J, Schoenwald S, Kelleher K, Hoagwood KE, Mayberg S, Green P, Research Network on Youth Mental H. Assessing the organizational social context (OSC) of mental health services: implications for research and practice. Adm Policy Ment Health. 2008;35:98–113.
Bandura A. Social foundations of thought and action: a social cognitive theory. Prentice Hill: Englewood Cliffs, New Jersey; 1986.
Lehman WE, Greener JM, Simpson DD. Assessing organizational readiness for change. J Subst Abuse Treat. 2002;22:197–209.
Packard T. Organizational change: a conceptual framework to advance the evidence base. J Hum Behav Soc Environ. 2013;23.
Cane J, O’Connor D, Michie S. Validation of the theoretical domains framework for use in behaviour change and implementation research. Implement Sci. 2012;7:37.
Aunger R, Curtis V. Consolidating behaviour change theory. London: LSHTM/Hygiene Centre for Unilever PLC; 2007.
Rogers EM. Diffusion of innovations. 4th ed. New York: Free Press; 1995.
Fishbein M, Ajzen I. Belief, attitude, intention, and behavior: an introduction to theory and research. Reading, MA: Addison-Wesley; 1975.
Weiner BJ, Lewis MA, Linnan LA. Using organization theory to understand the determinants of effective implementation of worksite health promotion programs. Health Educ Res. 2009;24:292–305.
Emmons KM, Weiner B, Fernandez ME, Tu S-P. Systems antecedents for dissemination and implementation: a review and analysis of measures. Health Educ Behav. 2012;39:87–105.
Flottorp SA, Oxman AD, Krause J, Musila NR, Wensing M, Godycki-Cwirko M, Baker R, Eccles MP. A checklist for identifying determinants of practice: a systematic review and synthesis of frameworks and taxonomies of factors that prevent or enable improvements in healthcare professional practice. Implement Sci. 2013;8.
Hage E, Roo JP, van Offenbeek MAG, Boonstra A. Implementation factors and their effect on e-health service adoption in rural communities: a systematic literature review. BMC Health Serv Res. 2013;13.
Castro F, Barrera M, Martinez C. The cultural adaptation of prevention interventions: resolving tensions between fidelity and fit. Prev Sci. 2004;5:41–5.
Sabatier PA. Theories of the policy process. Westview Press; 2007.
Mozygemba K, Hofmann B, Lysdahl KB, Pfadenhauer L, Van Der Wilt GJ, Gerhardus A. Guidance to assess socio-cultural aspects. In: Lysdahl KB, Mozygemba K, Burns J, Chilcott JB, Brönneke JB, Hofmann B, editors. Guidance for assessing effectiveness, economic aspects, ethical aspects, socio-cultural aspects and legal aspects in complex technologies. 2015.
Victora C, Armstrong Schellenberg J, Huicho L, Amaral J, El Arifeen S, Pariyo G, Manzi F, Scherpbier R, Bryce J, Habicht J. Context matters: interpreting impact findings in child survival evaluations. Health Policy Plan. 2005;20:i18–1.
European Network for Health Technology Assessment (EUnetHTA): HTA Core Model for screening technologies. Work Package 4 Core HTA EUnetHTA Joint Action 2010 - 2012. 2011.
Lehoux P, Williams-Jones B. Mapping the integration of social and ethical issues in health technology assessment. Int J Technol Assess Health Care. 2007;23:9–16.
Potter BK, Avard D, Graham ID, Entwistle VA, Caulfield TA, Chakraborty P, Kennedy C, McGuire M, Griener GG, Montgomery M, et al. Guidance for considering ethical, legal, and social issues in health technology assessment: application to genetic screening. Int J Technol Assess Health Care. 2008;24:412–22.
Statens Beredning för Medicinsk Utvärdering (SBU). Utvärdering av metoder i hälso- og sjukvården, en handbok. Etiska och sociala aspekter. Stockholm: SBU (Statens beredning för medicinsk utvärdering); 2014.
Nash R, Hudson A, Luttrell C. Mapping political context: A toolkit for Civil Society Organisations Overseas Development Institute—Research and Policy in Development Programme; 2006.
Bradford NK, Young J, Armfield NR, Herbert A, Smith AC. Home telehealth and paediatric palliative care: clinician perceptions of what is stopping us? BMC Palliat Care. 2014;13:29.
Brönneke JB, Hofmann B, Lysdahl KB, Van Der Wilt GJ, Buchner B. Guidance to assess legal aspects. In: Lysdahl KB, Mozygemba K, Burns J, Chilcott JB, Brönneke JB, Hofmann B, editors. Guidance for assessing effectiveness, economic aspects, ethical aspects, socio-cultural aspects and legal aspects in complex technologies. 2015.
Metz A, Bartley L. Active implementation frameworks for program success. Zero to One. 2012;32:11–8.
Proctor EK, Powell BJ, McMillen JC. Implementation strategies: recommendations for specifying and reporting. Implement Sci. 2013;8.
May CR, Mair F, Finch T, MacFarlane A, Dowrick C, Treweek S, Rapley T, Ballini L, Ong BN, Rogers A, et al. Development of a theory of implementation and integration: normalization process theory. Implement Sci. 2009;4.
May C. Towards a general theory of implementation. Implement Sci. 2013;8.
Avgar AC, Litwin AS, Pronovost PJ. Drivers and barriers in health IT adoption: a proposed framework. Appl Clin Inform. 2012;3:488–500.
Palmer D, Kramlich D. An introduction to the multisystem model of knowledge integration and translation. ANS Adv Nurs Sci. 2011;34:29–38.
Damschroder LJ, Hagedorn HJ. A guiding framework and approach for implementation research in substance use disorders treatment. Psychol Addict Behav. 2011;25:194–205.
Chaudoir SR, Dugan AG, Barr CHI. Measuring factors affecting implementation of health innovations: a systematic review of structural, organizational, provider, patient, and innovation level measures. Implement Sci. 2013;8.
Squires JE, Graham ID, Hutchinson AM, Linklater S, Brehaut JC, Curran J, Ivers N, Lavis JN, Michie S, Sales AE, et al. Understanding context in knowledge translation: a concept analysis study protocol. J Adv Nurs. 2015;71:1146–55.
Rycroft-Malone J, Seers K, Chandler J, Hawkes CA, Crichton N, Allen C, Bullock I, Strunin L. The role of evidence, context, and facilitation in an implementation trial: implications for the development of the PARIHS framework. Implement Sci. 2013;8.
May CR, Johnson M, Finch T. Implementation, context and complexity. Implement Sci. 2016;11:141.
Luoto J, Shekelle PG, Maglione MA, Johnsen B, Perry T. Reporting of context and implementation in studies of global health interventions: a pilot study. Implement Sci. 2014;9:1–9.
Mohler R, Kopke S, Meyer G. Criteria for reporting the development and evaluation of complex interventions in healthcare: revised guideline (CReDECI 2). Trials. 2015;16:204.
Benchimol EI, Smeeth L, Guttmann A, Harron K, Moher D, Petersen I, Sorensen HT, von Elm E, Langan SM, Committee RW. The REporting of studies Conducted using Observational Routinely-collected health Data (RECORD) statement. PLoS Med. 2015;12, e1001885.
Pinnock H, Epiphaniou E, Sheikh A, Griffiths C, Eldridge S, Craig P, Taylor SJ. Developing standards for reporting implementation studies of complex interventions (StaRI): a systematic review and e-Delphi. Implement Sci. 2015;10:1–10.
Moore GF, Audrey S, Barker M, Bond L, Bonell C, Hardeman W, Moore L, O’Cathain A, Tinati T, Wight D, Baird J. Process evaluation of complex interventions: Medical Research Council guidance. BMJ. 2015;350.
World Health Organization, Division of Health Promotion Education and Communication. Health promotion glossary. Geneva: World Health Organization; 1998.
Peters M, Harmsen M, Laurant M, Wensing M. Ruimte voor verandering? Knelpunten en mogelijkheden voor verbeteringen in de patiëntenzorg [Room for improvement? Barriers to and facilitators for improvement of patient care]. Nijmegen: Centre for Quality of Care Research (WOK), Radboud University Nijmegen Medical Centre; 2002.
May C, Finch T, Mair F, Ballini L, Dowrick C, Eccles M, Gask L, MacFarlane A, Murray E, Rapley T, et al. Understanding the implementation of complex interventions in health care: the normalization process model. BMC Health Serv Res. 2007;7:148.
We would like to thank all those who contributed to the development of the CICI framework, in particular, the stakeholders who participated in the demonstration HTA and all the members of the INTEGRATE-HTA project. We would moreover like to thank Maggie Hendry, Claudia Wild and Ingrid Wildbacher for their valuable feedback on the previous versions of the CICI framework.
The research leading to these results has received (partial) funding from the European Union Seventh Framework Programme under grant agreement no. 306141.
Availability of data and materials
LMP, AG, KM, KBL, AB, BH, PW, SP, JB, LB and ER all made substantial contributions to the conception and design of the research. LMP, AG, KM, KBL, BH and JB collected the data. LMP, AG, KM, KBL, AB, BH, PW, SP, JB, LB and ER were involved in the analysis and interpretation of the data. PW, SP, JB and LB applied the framework and provided valuable feedback. ER, AB and BH provided valuable conceptual input throughout the development and the application of the framework. LMP and ER led the drafting of the manuscript, and all authors have reviewed the draft critically and suggested revisions, given final approval of the version to be published and agreed to be accountable for all aspects of the work.
The authors declare that they have no competing interests.
Consent for publication
Ethics approval and consent to participate
Publications included into concept analysis of context. (DOC 79 kb)
Publications included into concept analysis of implementation. (DOC 93 kb)
Dimensions, Domains and Constructs of Context, Implementation and Setting. (DOC 484 kb)
Data Extraction Tool for Quantitative Systematic Reviews. (DOC 54 kb)
Data Extraction Tool for Qualitative Systematic Reviews. (DOC 55 kb)
Expert Consultation Guide. (DOC 57 kb)
Worked example of three frameworks (CICI, CFIR, PARIHS). (DOCX 143 kb)
About this article
- Complex intervention
- Concept analysis
- Systematic review
- Health technology assessment
- Public health