Open Access

Evaluating the implementation of a national clinical programme for diabetes to standardise and improve services: a realist evaluation protocol

  • S. McHugh1Email author,
  • M. L. Tracey1,
  • F. Riordan1,
  • K O’Neill1,
  • N. Mays2 and
  • P. M. Kearney1
Implementation Science201611:107

https://doi.org/10.1186/s13012-016-0464-9

Received: 16 June 2016

Accepted: 30 June 2016

Published: 28 July 2016

Abstract

Background

Over the last three decades in response to the growing burden of diabetes, countries worldwide have developed national and regional multifaceted programmes to improve the monitoring and management of diabetes and to enhance the coordination of care within and across settings. In Ireland in 2010, against a backdrop of limited dedicated strategic planning and engrained variation in the type and level of diabetes care, a national programme was established to standardise and improve care for people with diabetes in Ireland, known as the National Diabetes Programme (NDP). The NDP comprises a range of organisational and service delivery changes to support evidence-based practices and policies. This realist evaluation protocol sets out the approach that will be used to identify and explain which aspects of the programme are working, for whom and in what circumstances to produce the outcomes intended.

Methods/design

This mixed method realist evaluation will develop theories about the relationship between the context, mechanisms and outcomes of the diabetes programme. In stage 1, to identify the official programme theories, documentary analysis and qualitative interviews were conducted with national stakeholders involved in the design, development and management of the programme. In stage 2, as part of a multiple case study design with one case per administrative region in the health system, qualitative interviews are being conducted with frontline staff and service users to explore their responses to, and reasoning about, the programme’s resources (mechanisms). Finally, administrative data will be used to examine intermediate implementation outcomes such as service uptake, acceptability, and fidelity to models of care.

Discussion

This evaluation is using the principles of realist evaluation to examine the implementation of a national programme to standardise and improve services for people with diabetes in Ireland. The concurrence of implementation and evaluation has enabled us to produce formative feedback for the NDP while also supporting the refinement and revision of initial theories about how the programme is being implemented in the dynamic and unstable context of the Irish healthcare system.

Keywords

Realist evaluation Protocol Diabetes Implementation

Background

Diabetes is a major public health and health service challenge worldwide with global prevalence estimated to increase from 2.8 % in 2000 to 4.4 % in 2030, an increase from 171 million people to 366 million people in 30 years [1]. The most recent Global Burden of Disease study estimates that diabetes is the seventh leading cause of years lived with disability worldwide [2]. Diabetes is associated with reduced quality of life and life expectancy [3, 4]. There are also significant societal and health service costs associated with the condition; global health expenditure on diabetes was estimates to be at least US$376 billion in 2010, rising to US$490 billion by 2030 [5].

Optimal diabetes care

The need for organised coordinated implementation of strategies to improve diabetes care and reduce disease burden has long been recognised. In 1989, health departments from across Europe, including Ireland, signed the St. Vincent Declaration, a set of standards and goals to improve diabetes care [6]. The onus was placed on individual governments to implement strategies to meet the agreed targets. Over the next three decades, a number of countries developed national and regional multidimensional programmes to improve the monitoring and management of diabetes and to enhance the coordination of care within and across settings [710].

Consensus exists on what constitutes good quality diabetes care. Substantial evidence supports treatments to manage diabetes and slow the progression of complications [1115]. National and international guidelines recommend the regular monitoring and management of blood glucose levels, blood pressure, kidney function, body mass index and smoking status, as well as routine foot surveillance, retinopathy screening and patient self-management education [1618]. At a system level, the organisational features of high-quality diabetes care include regular review, patient registration and recall [1921]. There has been a shift towards multidisciplinary shared management of complex patients across primary and secondary care settings, and structured management of stable diabetes in primary care with suitable organisational support [21, 22]. While quality improvement strategies targeting professionals and patients improve diabetes care and patient outcomes, strategies which target the entire system of chronic disease management, such as case management, team changes and patient registry, are associated with the largest benefits [23].

Diabetes services in Ireland

Over the last two decades in Ireland, a plethora of policies and reports have repeatedly called for evidence-based service developments seen in other countries [24]. A number of diabetes initiatives have emerged, led by healthcare professionals, to improve diabetes care at a local level but with inconsistent implementation of a comprehensive diabetes service nationally. The balance of care between primary and secondary care settings varies and includes traditional hospital-based management, shared care between GPs and hospitals, and structured primary care-led management. Care in general practice ranges from ad hoc opportunistic management to structured care characterised by patient registration, regular recall and review coordinated by practice nurses [25]. A national survey of GPs reported that less than half used a patient register and diabetes guidelines or engaged in routine recall of patients with diabetes. Less than 10 % had a formal shared protocol or ever had a joint meeting with the hospital diabetes team. There was deficient access to allied health services such as podiatry, dietetics and eye screening [26]. Within the hospitals, not all diabetes services are led by an endocrinologist. Endocrinology-led services in Ireland had more developed subspecialty clinics and greater access to specialist allied health professionals. However, waiting times for these services were longer and discharge rates to primary care were lower than for non-specialty led services [27]. The provision of structured diabetes care in general practice and shared care between settings has produced favourable results in Ireland in terms of processes and outcomes of care [2831]. However, these models of care are not common-place and there is a dearth of information on the quality of routine diabetes management at a national level.

The National Diabetes Programme: a complex intervention to standardise care

Against this backdrop of variation in the type and quality of diabetes care, and a lack of dedicated strategic national planning and programme implementation, in 2010, a clinical programme for diabetes was established to standardise and improve care for people with diabetes in Ireland, known as the National Diabetes Programme (NDP) [32]. The NDP was one of a number of clinical care programmes set up under the auspices of the Directorate of Clinical Strategy and Programme in the Health Service Executive (HSE), the national health service in Ireland. The overarching goals of these programmes are to improve access to services, quality and safety of care, and cost effectiveness. These goals are achieved by bringing together representatives from various clinical disciplines to develop standardised patient pathways and evidence-based models of care [32]. Similar to the other programmes, the diabetes programme has a defined governance structure with a national clinical lead and programme manager, a clinical advisory group, and a national working group with the joint involvement of healthcare providers in primary, secondary and tertiary care [33]. There are also four regional Diabetes Services Implementation Groups (DSIGs), which are multidisciplinary regional networks established to inform the development and implementation of the National Diabetes Programme.

The specific aim of the NDP is to ‘save the eyes, limbs and lives of people with diabetes’ [33]. Like other large-scale service delivery innovations [34], a change in patient outcomes was to be achieved through the coordinated reorganisation of existing services, and the introduction of new services and supports for people with diabetes. Dedicated work streams were established for the implementation of a national retinopathy screening programme, a national model of care for the screening and treatment of diabetic foot disease, and a national model of integrated care for the management of diabetes across primary, secondary and tertiary care settings (Fig. 1).
Fig. 1

Intervention components of the National Programme for Diabetes

Service innovations such as the NDP lead to both intended and unintended consequences. Evaluation provides an opportunity not only to identify and track these but also to identify multilevel barriers to implementation and conditions that make success and sustainability more likely. However, process evaluation is difficult to apply to complex services spanning organisational boundaries [35]. The NDP represents a number of complex interventions introduced into a complex social system, making it an appropriate subject for a realist evaluation. The realist evaluation approach, developed by Pawson and Tilley, goes beyond looking at whether a programme works or not, to try and understand which aspects of the programme work, for whom, in what circumstances [36, 37]. There is an inherent acknowledgement that a programme will work differently in different contexts; the aim is to understand what it is about a programme that leads to different outcomes [38].

According to Pawson and colleagues, complex service interventions are based on an underlying hypothesis of how the intervention will bring about an outcome [37, 39]. The first step of a realist evaluation is to identify and articulate these theories, known as programme theories. A programme provides a resource, an opportunity or a constraint, that influences the decision-making process of its intended target group. It is this decision-making process that determines whether an outcome is achieved or not; complex interventions are active, that is they only work through stakeholders’ reasoning and responses. This underlying interaction between what a programme provides and the reasoning of its intended targets is known as a mechanism. Understanding and explaining the often invisible implicit mechanisms are core functions of a realist evaluation [37, 40]. Mechanisms are argued to be triggered, to a greater or lesser extent in certain favourable and unfavourable contexts, leading to intended and unintended outcomes. The programme theory articulates a theoretical relationship between a context, mechanism and outcome, known as a ‘C-M-O’ configuration [37].

In this paper, we present the protocol for an evaluation of the NDP that adopts a realist approach. The aim of the evaluation is to identify and explain which aspects of the programme are working (or not working), for whom and in what circumstances to produce outcomes. The evaluation will examine three ongoing work streams of the NDP which have been prioritised since its inception in 2010: the introduction of a national diabetic retinopathy screening programme (initiated in 2013); the establishment of a national model of foot care for people with diabetes (staff recruited in 2013); and the development of a national model of integrated care for diabetes (staff recruited in 2013). The aim of this paper is to outline in detail the stages, methods and data being used in the evaluation, as well as some of the challenges to, and opportunities for, providing formative feedback to the NDP.

Methods/design

This prospective evaluation follows the research stages outlined by Pawson and Tilley: (1) elicit and formulate the programme theory underlying the NDP and its work streams (national retinopathy screening programme, national foot care model and national model of integrated care), (2) collect data to test these initial theories, (3) analyse data to interrogate the theories and (4) interpret analysis to refine or revise the initial programme theories [36] (Fig. 2).
Fig. 2

Realist research cycle (adapted with permission from [63])

Realist evaluation is method neutral and most studies employ both quantitative and qualitative research methods [4144]. We are using mixed methods depending on the stage of the study and the theory component (context, mechanism or outcome) under scrutiny; for example documentary analysis and qualitative interviews are considered useful for identifying the official programme theory and context, qualitative interviews are also appropriate for exploring stakeholders’ responses to and reasoning about the programme (mechanisms), and quantitative administrative data allow examination of outcomes such as service uptake [37]. Table 1 summarises the methods being used during data collection at stage 1 and stage 2.
Table 1

Data collection during stages 1 and 2 to formulate and refine programme theories

Stage

Methods

Stage 1: Elicit and formulate the programme theory

• Documentary analysis of published and unpublished documents pertaining to the establishment, development and implementation of the National Programme for Diabetes and the three programme interventions.

• Semi-structured interviews with national level programme developers (n = 19).

Stage 2: Data Collection

Implementation of the National Programme for Diabetes

• Multiple case study design (n = 4 cases)

• Semi-structured interviews with theoretically sampled stakeholders in each area (n = approx. 15 per case).

Further data for each case will be gathered through a number of sub-studies conducted at a local and national level to build a rich case description and allow for embedded analysis of each programme component.

Retinopathy screening programme

• Audit of registration, consent and uptake among a convenience sample of two large primary care centres (n = 22 GPs, approx. 600 people with diabetes) and a smaller rural practice in one region (n = 2 GPs, approx. 100 people with diabetes)

• Semi-structured interviews with people with diabetes from those practices

National foot care model

• Semi-structured interviews with podiatrists including those recruited as part of the programme.

• Cross-sectional study of administrative activity data submitted by podiatrists as part of the National Programme for Diabetes.

National model of integrated care

• National survey of Diabetes Nurse Specialists (DNSs) including those recruited as part of the programme.

• Follow-up interviews and focus groups

• Cross-sectional analysis of administrative activity data submitted by DNSs recruited as part of the National Programme for Diabetes

Stage 1: elicit and formulate the programme theory

Design

As a theory-based evaluation approach, the first step of the realist evaluation is to clarify the ‘programme theory’; that is how the NDP and its three work streams are expected to cause or contribute to outcomes. The programme theory articulates the relationship between a context, mechanism and outcomes of the national programme components, known as ‘C-M-O’ configurations.

Data collection

Three data sources were used to develop the initial programme theory. Firstly, a documentary analysis was carried out to establish the official programme theory, expectations and rationale for establishing the Programme. Documents included published and unpublished material such as strategy documents from interest groups, media coverage, press releases, national service plans, NDP website, and official documentation on the role and function of the programmes. An additional file outlines the type and source of documents (see Additional file 1). These data were also useful for mapping the context in which the programme was being designed and implemented.

Secondly and concurrently, we conducted qualitative interviews with a purposive sample of stakeholders involved at a national level in the design, development and management of the Programme. All members of the national diabetes working group were invited to take part (membership between July 2014 and January 2015) as well as former clinical leads and programme managers. The national diabetes working group comprises representatives from endocrinology, general practice, diabetes nurse specialists and practice nurses, dietetics, podiatry, community pharmacy, public health, patient advocacy and health service management. Members also represent different parts of the country. Following an initial invitation via email, all participants were contacted individually by a member of the research team (MT) to outline the study and arrange a convenient time and place for interview.

A semi-structured topic guide was developed informed by initial findings from the documentary analysis and previously published realist evaluations [38, 42]. The topic guide was piloted with a convenience sample of two participants involved in diabetes care, who were not members of the national working group but were aware of the work of the Programme. Minor amendments were made to the prompts and probes used within the topic guide. The topic guide addressed participants’ role in the Programme, why the programme was established, the planned changes and how they were being implemented, progress to date, anticipated barriers and facilitators, and expected outcomes (see Additional file 2).

Face-to-face interviews were conducted (by MT) with 19 participants between July 2014 and January, 2015 (average duration 1 h). Participants received an information sheet and signed a consent form prior to the interview. These participants are implementers within their own local diabetes service as well as being involved in the design of programme at a national level. Thus, in addition to discussing planned implementation and expected outcomes (official programme theory), participants discussed their own experience of implementation, perceived outcomes in their area, and barriers encountered. Data collection and analysis were iterative to allow the gathering of further data on emergent themes and the topic guide was modified to accommodate emergent lines of inquiry.

Thirdly, following a presentation of preliminary findings, a short survey was conducted among attendees at the annual conference held by the NDP (November, 2015). Attendees, including healthcare professionals, patient representatives, health service managers and policy makers involved in or affected by the national programme, were invited to complete open-ended questions about which aspects of the national programme were working well, which aspects were not working as well, and why. Respondents were asked to indicate their professional role and the area of the country in which they worked. Thirty attendees completed the questionnaire (approximately 25 % response rate). Gathering the opinions of those involved in implementation from around the country allowed for further refinement and corroboration of the initial programme theories based on national stakeholders’ accounts.

Data analysis

Interviews were audio-recorded, transcribed verbatim and imported into NVivo 10 software [45]. The framework approach [46] was used to systematically identify contexts (C), mechanisms (M), and outcomes (O) in the interview transcripts and documents, and chart hypothetical relationships between them (C-M-O configurations) to formulate programme theories for each programme component.

The Framework approach is sufficiently open to allow for novel themes to emerge inductively during analysis [47]. First, transcripts were read and re-read (familiarisation), followed by open coding to identify contexts, mechanisms and outcomes. Emergent concepts which did not fit explicitly with the C-M-O framework were also coded. Two researchers (MT and SMH) open-coded three interviews of staff from different parts of the country. The research team then met to compare and contrast codes, clarify understanding of contexts, mechanisms and outcomes, and agree on an initial coding framework. Two independent coders (FR, KON, researchers who had recently joined the research team) were invited to analyse four interviews (from different professional backgrounds and locations) to further refine the coding framework. This coding framework was applied to subsequent transcripts by the research team. Framework development was a dynamic process with regular meetings to discuss new codes or merging existing codes, assumptions, and ideas about C-M-O configurations.

Having openly coded all of the transcripts, data were sorted and synthesised by theme bringing similar concepts together (thematic charting). At this stage, themes were sorted under the individual programme components: national working group (SMH), retinopathy screening (MT), national foot care model (KON) and national model of integrated care (FR). Each researcher led on the synthesis of codes and development of themes for a different programme intervention. This facilitated data management but also enabled data immersion necessary to develop a programme theory about the relationship between contexts, mechanisms and outcomes for that intervention. In some instances, participants themselves outlined partial C-M-O configurations during interviews (e.g. between contexts and outcomes, mechanisms and outcomes); these relationships were refined or revised by examining other participant interviews. However, in most cases, the research team developed C-M-O configurations based on the analysis of all interviews, starting with a synthesis of the proposed outcomes and working backwards to build a theory about the mechanism that led to that outcome and the context that triggered the mechanism.

Open-ended responses to the conference survey were coded using the same approach although emergent themes tended to reiterate, and overlap with enabling and disabling contexts identified during the interviews. The themes were used to reinforce or refine the initial C-M-O configurations.

Memos were used and shared throughout the analysis to note assumptions, events and changes in the NDP, coding definitions, hunches and early impressions [48]. The language and expressions of the participants were maintained as far as possible, using in vivo codes, to avoid losing the meaning and context. The results were presented to the wider research team for discussion.

Stage 2: data collection to test programme theories

The aim of stage 2, which is currently underway, is to collect data to test the C-M-O configurations developed in stage 1. A multiple case study design is being used. Case studies are often used in realist evaluation [34, 4951]; this approach emphasises the in-depth study of phenomena in their real-life context, and the importance of theory to inform the design, selection and interpretation of case studies [52].

Case selection

A case was defined as a geographical area within one of the four HSE administrative regions (Fig. 3).
Fig. 3

Map of administrative regions in the Health Service Executive (HSE)

A number of criteria were used to select a case area. Firstly, cases had to have received an intervention from the NDP (retinopathy screening programme, integrated care nurse and/or podiatrist). Second, it emerged during stage 1 analysis that the presence or absence of a diabetes initiative (either a primary care-led diabetes initiative, an existing community diabetes nurse specialist (DNS) service, or an established diabetic retinopathy (DR) screening initiative) was an important contextual factor; therefore, we theoretically selected cases on this basis. More detail on the underlying rationale for case selection is available (see Additional file 3).

We mapped these criteria for all areas, starting with the allocation of interventions by the NDP, and discussed the selection of cases within the research team. Table 2 outlines the final selection of four cases and their characteristics.
Table 2

Case study selection & sampling criteria

Case 1

Case 2

Case 3

Case 4

NDP interventions

• Received 2 components

Existing infrastructure and engagement

• Prior DR screening initiative delivered in the community, open to patients in a select number of general practices (no universal access)

• Existing primary care diabetes initiative with voluntary enrolment by some general practices

NDP interventions

• Received 3 components

Existing infrastructure and engagement

• Previous population-based retinopathy screening initiative offered to all general practices in the area

• Existing primary care diabetes initiative with voluntary participation from some practices

• Community DNS

NDP interventions

• Received 3 components

Existing infrastructure and engagement

• Previous hospital service which provided eye screening for those referred to service, no population-based screening programme in the community

• Existing primary care diabetes initiative with voluntary participation from general practices

• Community DNS

NDP interventions

• Received 3 components

Existing infrastructure and engagement

• Previous hospital service which provided eye screening for those referred to service, no population-based screening programme in the community

• No existing primary care-led diabetes initiative

• Community DNS

Data collection

Within each case, multiple sources of data will be used to test the C-M-O configurations developed in stage 1.

Qualitative

Qualitative interviews will be undertaken with key stakeholder groups purposively sampled in each case and this work is ongoing. Table 3 outlines the expected number of participants per case. In each area, local members of the regional DSIG will be invited to participate. Participants will be invited to suggest other stakeholders such as health service managers that they engage with. These managers may or may not be formally involved with the DSIG but have a role in deploying and coordinating resources. Integrated care nurses and podiatrists appointed as part of the implementation of the NDP will be invited to participate in an interview or focus group, as well as diabetes nurse specialists (hospital and community-based) and podiatrists previously in post. A purposive sample of GPs and practice nurses will be recruited from practices enrolled/not enrolled in primary care-led diabetes initiatives. Participants will be recruited through educational meetings, continuing professional development groups and local DSIGs. People with diabetes will be recruited using a web-based entry form and dedicated telephone line (participant portal). To ensure we have patient representation specific to our cases, we will ask participating healthcare professionals to publicise our study and the participant portal to patients, and display sign-up posters in local clinics. People with diabetes will also be recruited through local education and awareness events run by a national patient advocacy group (Diabetes Ireland).
Table 3

Stage 2 sample per case

 

Number per case

Total

DSIG member

2

8

Endocrinologist

1

4

General practitioner (GP)

2

8

Practice nurse

2

8

Diabetes nurse specialist/integrated care nurse

2

8

Podiatrists

2

8

Ophthalmologist

1

4

Patient representative

2

8

Health service manager

1

4

Total

15

60

A theory-driven topic guide has been devised for interviews in stage 2, based on the programme theories developed during stage one. The topic guide has been tailored to the stakeholder group being interviewed (hospital specialist, GP, practice nurse, specialist nurse, podiatrist, person with diabetes). During the interview, we will have an active and explicit role in explaining the contexts and outcomes of interest, to ensure a shared understanding of the terminology and purpose of the questions. In the context of our developed theories, participants will be invited to explain how their experience fits with that theory and reflect on what may explain the outcomes in their area [53]. The topic guide has been piloted with a convenience sample of one GP and two practice nurses, staff who would be most familiar with or in receipt of most programme components. Written consent will be obtained prior to each interview, and all interviews will be audio-recorded and transcribed verbatim. Thematic analysis of the interviews will be guided by the initial programme theories identified in stage 1. However, analysis will be open to emergent themes to facilitate further theory refinement.

Quantitative

To assess programme outcomes, administrative data and healthcare professional surveys will be analysed (see Table 1). For the national retinopathy screening programme, the outcomes being examined are registration, consent and uptake. These will be examined using local audits of clinical records in general practice. For the national foot care model, case-specific activity data including the number and risk profile of patients will be analysed. For the national model of integrated care, a national survey of diabetes nurse specialists (including integrated care nurses) is being conducted. This will be supplemented with case-specific analysis of activity data, including the number of GPs engaging with the integrated care nurse service and the number of patient consultations.

Stage 3: analyse data to interrogate theories

In realist evaluation, the unit of analysis is the theories hypothesising the mechanisms by which an intervention produces certain outcomes in a particular context [54]. A matrix will be used to analyse and synthesise both the qualitative and quantitative data available for each case (administrative data, survey data, transcripts) [55, 56]. A matrix will be constructed for each programme theory relating to various programme components (retinopathy screening, national foot care model, and national model of integrated care). Following the example of O’Cathain and colleagues [57], the columns of the matrix will contain the contexts, mechanisms and outcomes for a given theory. Each row in the matrix will represent a different case (see Additional file 4 for an example). This approach will facilitate within-case analysis, highlighting similarities or discrepancies between data sources which may lead to further data collection or analysis [56]. It will also facilitate cross-case analysis to identify patterns across cases. NVivo 10 software will be used to store and manage data [45].

Stage 4: interpret analysis to refine or revise the initial programme theories

In light of the analysis in stage 3, the programme theories underpinning the NDP will be refined. The original programme theories will be assessed against the evidence emerging within cases and then between cases to more accurately reflect the experiences of those delivering and receiving the changes introduced by the NDP, and the outcomes realised in different contexts.

Ethical considerations

This study has been approved by the Clinical Research Ethics Committee of the Cork Teaching Hospitals. Each participant in the study is asked for written informed consent prior to conducting the interviews and focus groups. Informed consent has also been sought from survey participants, and data have been anonymised for analysis and reporting. Permission has been granted by the NDP to analyse activity data submitted to the programme. Anonymity will be assured at each case study site and all participants will be given a unique ID number. Initial programme theories were presented to members of the national working group for comment. Results from later stages will be fed back to case study participants in the form of a case report. Any potentially identifiable information will be removed prior to reporting and publishing the findings.

Discussion

Realist evaluation, which allows for the study of context and its influence on outcomes, is appropriate for examining the implementation of the NDP, given the history of regional variation in diabetes services in Ireland. This paper outlines the protocol for a mixed methods evaluation to explore which aspects of the programme are working, for whom and in which circumstances.

Geographic case studies are often difficult to define [52], and this has been a particular challenge in this study given the ill-defined boundaries of health services in Ireland. Catchment areas for health services and hospitals are often fluid, and the organisational structures within the health service have gone through several recent reconfigurations. Furthermore, the results of stage 1 of this study suggest variation in diabetes services within regions and counties depending on the local resources, infrastructure and engagement from stakeholders such as GPs and local management. We have selected cases for stage 2 on the basis of these preliminary findings. The aim of this study is to understand how the NDP is working, for whom and in what circumstances. Therefore, the cases are considered instrumental as opposed to intrinsic [58], that is they are being used to gain a deeper understanding of programme implementation as a whole, as opposed to focusing on the uniqueness of the individual case itself.

We have used a number of strategies to enhance the rigour of this study. Data collection tools including topic guides and surveys have been extensively piloted. Different triangulation techniques will be used to strengthen the validity of findings, including the use of mixed methods, multiple data sources (interviews and documents) and researchers from different disciplinary perspectives (health services research, epidemiology, public health, clinical medicine) [59]. While the limitations of member checking as a strategy to verify overall results have been highlighted, it is considered appropriate to enhance validity in case study research, as case reports maintain the contextual information that allows participants to relate their experiences to synthesised results [52, 60].

Throughout this study, data will be collected and analysed concurrently within each stage to allow emergent lines of enquiry to be explored [60]. The research team has endeavoured to be responsive to the implementation of the programme and changeable context in which it is being rolled out. For example, in October 2015, the Department of Health in Ireland agreed a new contract with GPs which provided financial reimbursement for two structured diabetes review visits in general practice per year. This scheme is known as the Diabetes Cycle of Care. Patients with type 2 diabetes who have a medical card or a GP visit card, which entitles them to free GP care in Ireland, are eligible to be registered by their GP for the scheme. This is a significant influential factor in the context of the NDP. Although not part of the initial programme theories, given its recent introduction, we have adapted our topic guide to explore how the introduction of this financial incentive may influence implementation.

Complex social interventions such as the NDP achieve their outcomes by active input from various stakeholders. Qualitative research is an important part of exploring the reasoning and responses of stakeholders to a programme [37]. Similar to other realist evaluations [61], the results of our interviews with national programme stakeholders, who were also local implementers with context-specific experience, further refined ‘official’ programme theories about which aspects are working, in which circumstances and why. This evaluation builds on previous work by the research team which analysed the many diabetes care policies in Ireland, thereby providing information on some of the contextual factors that preceded the national programme [24]. Pawson suggests that by defining clearly the boundaries of case studies, evaluators are then able to harness the potential of administrative data, for example, relevant to quantifying the outcomes of programmes in realist evaluation [37]. Collaboration with the NDP has enabled us to analyse such administrative information where available. However, we are limited by the lack of a diabetes register in Ireland or national databases on the quality of diabetes care, and patients’ health service interactions and outcomes.

There is increasing interest in the evaluation of health policy and health service implementation. In particular, there is increasing emphasis on theory-based evaluations which aim to establish the context and mechanisms that facilitate successful implementation rather than simply focusing on the achievement of specific endpoints [35]. Realist evaluation has been used for this purpose to study a diverse range of service changes including the introduction of an integrated care pathway for palliative care [41], a multifaceted maternity care programme [49], ‘communities of practice’ [38], oncology teams [54] and quality improvement in primary care [51]. There are very few evaluations of the implementation of programmes or service interventions in Ireland, and to our knowledge, this is the first realist evaluation of a programme in Ireland.

The NDP is constantly moving between planning for future work streams and ongoing implementation of the current work streams. Therefore, the programme offers a potentially unique opportunity to evaluate and inform the implementation of changes in the Irish health system as they emerge and evolve. For example, there has been phased recruitment of integrated care diabetes nurse specialists (known as integrated care nurses (ICNs)) to support the implementation of the national model of integrated care as resources have been secured at national level. A protocol has been developed to clarify the role of the ICN, partly in response to barriers to implementation highlighted in the evaluation. There is close collaboration between the national working group and research team; the principal investigator (PK) is a member of the working group which provides an opportunity to provide formative feedback on implementation to those responsible. Our results should also provide insights relevant to the implementation of other clinical care programmes in Ireland operating in similar contexts. Furthermore, we hope that the findings will be relevant to programmes in other countries, some of which are also evaluating implementation of new care programmes [62].

Declarations

Acknowledgements

The authors are grateful to the past and present members of the ESPIRIT research team who contributed to data management and discussions regarding the analysis and interpretation of the data: Caragh Flannery, Julie Barrett and Cora Grant. The authors are also grateful to all of the participants who have participated in the evaluation and those who facilitated access to data.

Funding

This work was funded by the Health Research Board Research Leader Award (RL/2013/7) awarded to PMK. PMK, MT, KON and FR are funded through this award. SMH is funded by the Centre for Ageing Research and Development in Ireland (CARDI). The funding body had no role in the study design, collection, analysis or interpretation of data.

Authors’ contributions

SMH designed the study, conducted data analysis and wrote the initial draft of the paper. MT contributed to the study design, carried out data collection, conducted data analysis and contributed to revising the manuscript. FR contributed to the study design, carried out data collection, conducted data analysis and contributed to revising the manuscript. KON contributed to the study design, carried out data collection, conducted data analysis and contributed to revising the manuscript. NM contributed to the study design and revised the manuscript. PMK contributed to the study design and contributed to revising the manuscript. All authors have given final approval of the version to be published and agree to be accountable for all aspects of the work.

Competing interests

The authors declare that they have no competing interests.

Consent for publication

Not applicable.

Ethics approval and consent to participate

This study has been approved by the Clinical Research Ethics Committee of the Cork Teaching Hospitals. Each participant in the study is asked for written informed consent prior to conducting the interviews and focus groups. Informed consent has also been sought from survey participants, and data have been anonymised for analysis and reporting. Permission has been granted by the NDP to analyse activity data submitted to the programme.

Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Authors’ Affiliations

(1)
Department of Epidemiology & Public Health, Western Gateway Complex, University College Cork
(2)
Department of Health Services Research & Policy, London School of Hygiene and Tropical Medicine

References

  1. Wild S, Roglic G, Green A, Sicree R, King H. Global prevalence of diabetes: estimates for the year 2000 and projections for 2030. Diabetes Care. 2004;27(5):1047–53.View ArticlePubMedGoogle Scholar
  2. Vos T, Barber RM, Bell B, Bertozzi-Villa A, Biryukov S, Bolliger I, Charlson F, Davis A, Degenhardt L, Dicker D, et al. Global, regional, and national incidence, prevalence, and years lived with disability for 301 acute and chronic diseases and injuries in 188 countries, 1990–2013: a systematic analysis for the Global Burden of Disease Study. Lancet. 2013;386(9995):743–800.View ArticleGoogle Scholar
  3. Franco OH, Steyerberg EW, Hu FB, Mackenbach J, Nusselder W. Associations of diabetes mellitus with total life expectancy and life expectancy with and without cardiovascular disease. Arch Intern Med. 2007;167(11):1145–51.View ArticlePubMedGoogle Scholar
  4. Wexler D, Grant R, Wittenberg E, Bosch J, Cagliero E, Delahanty L, Blais M, Meigs J. Correlates of health-related quality of life in type 2 diabetes. Diabetologia. 2006;49(7):1489–97.View ArticlePubMedGoogle Scholar
  5. Zhang P, Zhang X, Brown J, Vistisen D, Sicree R, Shaw J, Nichols G. Global healthcare expenditure on diabetes for 2010 and 2030. Diabetes Res Clin Pract. 2010;87(3):293–301.View ArticlePubMedGoogle Scholar
  6. Diabetes Care and Research in Europe. The Saint Vincent Declaration. Diabet Med. 1990;7(4):360–360.Google Scholar
  7. Rothe U, Müller G, Schwarz PEH, Seifert M, Kunath H, Koch R, Bergmann S, Julius U, Bornstein SR, Hanefeld M, et al. Evaluation of a diabetes management system based on practice guidelines, integrated care, and continuous quality management in a federal state of Germany: a population-based approach to health care research. Diabetes Care. 2008;31(5):863–8.View ArticlePubMedGoogle Scholar
  8. Adolfsson ET, Rosenblad A, Wikblad K. The Swedish National Survey of the Quality and Organization of Diabetes Care in Primary Healthcare—Swed-QOP. Primary Care Diabetes. 2010;4(2):91–7.View ArticlePubMedGoogle Scholar
  9. Looker HC, Nyangoma SO, Cromie DT, Olson JA, Leese GP, Black MW, Doig J, Lee N, Lindsay RS, McKnight JA, et al. Rates of referable eye disease in the Scottish National Diabetic Retinopathy Screening Programme. Br J Ophthalmol. 2014;98(6):790–5.View ArticlePubMedPubMed CentralGoogle Scholar
  10. Gudbjörnsdottir S, Cederholm J, Nilsson PM, Eliasson B. The National Diabetes Register in Sweden: an implementation of the St. Vincent Declaration for Quality Improvement in Diabetes Care. Diabetes Care. 2003;26(4):1270–6.View ArticlePubMedGoogle Scholar
  11. DCCT: The Diabetes Control and Complications Trial Research Group. The effect of intensive treatment of diabetes on the development and progression of long-term complications in insulin-dependent diabetes mellitus. N Engl J Med. 1993;329(14):977–86.View ArticleGoogle Scholar
  12. UKPDS: UK Prospective Diabetes Study Group. Intensive blood-glucose control with sulphonyureas or insulin compared with conventional treatment and risk of complications in patients with type 2 diabetes. Lancet. 1998;352(9131):837–53.View ArticleGoogle Scholar
  13. Holman R, Paul S, Bethel M, Matthews D, Neil H. 10-year follow-up of intensive glucose control in type 2 diabetes. N Engl J Med. 2008;359(15):1577–89.View ArticlePubMedGoogle Scholar
  14. Holman RR, Paul SK, Bethel MA, Neil HAW, Matthews DR. Long-term follow-up after tight control of blood pressure in type 2 diabetes. N Engl J Med. 2008;359(15):1565–76.View ArticlePubMedGoogle Scholar
  15. Cholesterol Treatment Trialists Collaborators. Efficacy and safety of cholesterol-lowering treatment: prospective meta-analysis of data from 90056 participants in 14 randomised trials of statins. Lancet. 2008;366(9493):1267–78.Google Scholar
  16. NICE. Type 2 diabetes in adults: management. London: National Institute for Health and Care Excellence; 2015.Google Scholar
  17. Harkins V. A practical guide to integrated type 2 diabetes care. Dublin: Irish College of General Practitioners; 2016.Google Scholar
  18. American Diabetes Association. 4. Foundations of care: education, nutrition, physical activity, smoking cessation, psychosocial care, and immunization. Diabetes Care. 2015;38(Supplement 1):S20–30.View ArticleGoogle Scholar
  19. Renders CM, Valk GD, Griffin S, Wagner E, Eijk J, Assendelft W: Interventions to improve the management of diabetes mellitus in primary care, outpatient and community settings.Cochrane Database Syst Rev. 2000;Issue 4:Art. No.: CD001481Google Scholar
  20. Wagner EH, Austin BT, Von Michael K. Organizing care for patients with chronic illness. Milbank Q. 1996;74(4):511–44.View ArticlePubMedGoogle Scholar
  21. Griffin S, Kinmonth AL. Diabetes care: the effectiveness of systems for routine surveillance for people with diabetes. Cochrane Database Syst Rev. 2000;2:CD000541.Google Scholar
  22. Kahn R, Anderson JE. Improving diabetes care: the model for health care reform. Diabetes Care. 2009;32(6):1115–8.View ArticlePubMedPubMed CentralGoogle Scholar
  23. Tricco AC, Ivers NM, Grimshaw JM, Moher D, Turner L, Galipeau J, Halperin I, Vachon B, Ramsay T, Manns B, et al. Effectiveness of quality improvement strategies on the management of diabetes: a systematic review and meta-analysis. Lancet. 2012;379(9833):2252–61.View ArticlePubMedGoogle Scholar
  24. Mc Hugh SM, Perry IJ, Bradley C, Brugha R. Developing recommendations to improve the quality of diabetes care in Ireland: a policy analysis. Health Res Policy Syst. 2014;12(1):53.View ArticlePubMedPubMed CentralGoogle Scholar
  25. Mc Hugh S, O’Mullane M, Perry IJ, Bradley C. Barriers to, and facilitators in, introducing integrated diabetes care in Ireland: a qualitative study of views in general practice. BMJ Open. 2013;3(8):e003217.View ArticlePubMedPubMed CentralGoogle Scholar
  26. Mc Hugh S, O’Keeffe J, Fitzpatrick A, de Siún A, O’Mullane M, Perry I, Bradley C. Diabetes care in Ireland: a survey of general practitioners. Prim Care Diabetes. 2009;3(4):225–31.View ArticlePubMedGoogle Scholar
  27. O’Donnell M, de Siún A, O’Mullane M, Smith D, Bradley C, Finucane F, Dinneen S. Differences in the structure of outpatient diabetes care between endocrinologist-led and general physician-led services. BMC Health Serv Res. 2013;13(1):493–8.View ArticlePubMedPubMed CentralGoogle Scholar
  28. Mc Hugh SM, Marsden P, Brennan C, Murphy K, Croarkin C, Moran J, Harkins V, Perry I. Counting on commitment; the quality of primary care-led diabetes management in a system with minimal incentives. BMC Health Serv Res. 2011;11(1):384–93.View ArticleGoogle Scholar
  29. Brennan C, Harkins V, Perry I. Management of diabetes in primary care: a structured-care approach. Eur J Gen Pract. 2008;14(3):117–22.View ArticlePubMedGoogle Scholar
  30. Smith S, Bury G, O’Leary M, Shannon W, Tynan A, Staines A, Thompson C. The North Dublin randomized controlled trial of structured diabetes shared care. Fam Pract. 2004;21(1):39–45.View ArticlePubMedGoogle Scholar
  31. Collins MM, O’Sullivan T, Harkins V, Perry IJ. Quality of life and quality of care in patients with diabetes experiencing different models of care. Diabetes Care. 2009;32(4):603–5.View ArticlePubMedPubMed CentralGoogle Scholar
  32. Royal College of Physicians Ireland. About the National Clinical Programmes [https://www.rcpi.ie/national-clinical-programmes/]. 01.03.16
  33. Health Service Executive. National Clinical Programme for Diabetes [http://www.hse.ie/eng/about/Who/clinical/natclinprog/diabetesprogramme/]. 01.03.16
  34. Greenhalgh T, Humphrey C, Hughes J, Macfarlane F, Butler C, Pawson R. How do you modernize a health service? A realist evaluation of whole-scale transformation in London. Milbank Q. 2009;87.Google Scholar
  35. Lamont T, Barber N, Pury JD, Fulop N, Garfield-Birkbeck S, Lilford R, Mear L, Raine R, Fitzpatrick R. New approaches to evaluating complex health and care systems. BMJ. 2016;352. i154.Google Scholar
  36. Pawson R, Tilley N. Realistic evaluation. London: Sage; 1997.Google Scholar
  37. Pawson R. The science of evaluation: a realist manifesto. London: Sage; 2013.View ArticleGoogle Scholar
  38. Ranmuthugala G, Cunningham FC, Plumb JJ, Long J, Georgiou A, Westbrook JI, Braithwaite J. A realist evaluation of the role of communities of practice in changing healthcare practice. Implement Sci. 2011;6:49.View ArticlePubMedPubMed CentralGoogle Scholar
  39. Pawson R, Greenhalgh T, Harvey G, Walshe K. Realist review—a new method of systematic review designed for complex policy interventions. J Health Serv Res Policy. 2005;10 suppl 1:21–34.View ArticlePubMedGoogle Scholar
  40. Dalkin SM, Greenhalgh J, Jones D, Cunningham B, Lhussier M. What’s in a mechanism? Development of a key concept in realist evaluation. Implement Sci. 2015;10(1):1–7.View ArticleGoogle Scholar
  41. Dalkin SM, Jones D, Lhussier M, Cunningham B. Understanding integrated care pathways in palliative care using realist evaluation: a mixed methods study protocol. BMJ Open. 2012;2:e001533. doi:https://doi.org/10.1136/bmjopen-2012-001533.
  42. Abhyankar P, Cheyne H, Maxwell M, Harris FM, McCourt C. A realist evaluation of a normal birth programme. Evid Based Midwifery. 2013;11(4):112–9.Google Scholar
  43. Rycroft-Malone J, Fontenla M, Bick D, Seers K. A realistic evaluation: the case of protocol-based care. Implement Sci. 2010;5(1):38–52.Google Scholar
  44. Goicolea I, Hurtig AK, San Sebastian M, Vives-Cases C, Marchal B. Developing a programme theory to explain how primary health care teams learn to respond to intimate partner violence: a realist case-study. BMC Health Serv Res. 2015;15:228.View ArticlePubMedPubMed CentralGoogle Scholar
  45. QSR International Pty Ltd. NVivo qualitative data analysis software, Version 10. 2012.Google Scholar
  46. Ritchie J, Lewis J. Qualitative research practice: a guide for social science students & researchers. Thousand Oaks: Sage Publications; 2003.Google Scholar
  47. Mays N, Pope C. Qualitative research in health care, 3rd edn. Malden, MA: Blackwell Publishing; 2006Google Scholar
  48. Charmaz K. Constructing Grounded Theory: a practical guide through qualitative analysis. London: Sage Publications Ltd; 2006.Google Scholar
  49. Cheyne H, Abhyankar P, McCourt C. Empowering change: realist evaluation of a Scottish Government programme to support normal birth. Midwifery. 2013;29(10):1110–21.View ArticlePubMedGoogle Scholar
  50. Tremblay D, Touati N, Roberge D, Denis J-L, Turcotte A, Samson B. Conditions for production of interdisciplinary teamwork outcomes in oncology teams: protocol for a realist evaluation. Implement Sci. 2014;9(1):1–11.View ArticleGoogle Scholar
  51. Schierhout G, Hains J, Si D, Kennedy C, Cox R, Kwedza R, O’Donoghue L, Fittock M, Brands J, Lonergan K, et al. Evaluating the effectiveness of a multifaceted, multilevel continuous quality improvement program in primary health care: developing a realist theory of change. Implement Sci. 2013;8(1):1–15.View ArticleGoogle Scholar
  52. Yin RK. Case study research: design and methods. 4th ed. Thousand Oaks, CA: Sage; 2009.Google Scholar
  53. Pawson R. Theorizing the interview. Br J Sociol. 1996;47(2):295–314.View ArticleGoogle Scholar
  54. Randell R, Greenhalgh J, Hindmarsh J, Dowding D, Jayne D, Pearman A, Gardner P, Croft J, Kotze A. Integration of robotic surgery into routine practice and impacts on communication, collaboration, and decision making: a realist process evaluation protocol. Implement Sci. 2014;9(1):52.View ArticlePubMedPubMed CentralGoogle Scholar
  55. Miles MB, Huberman AM. Qualitative data analysis: an expanded sourcebook: Sage; 1994Google Scholar
  56. O’Cathain A, Murphy E, Nicholl J. Three techniques for integrating data in mixed methods studies. BMJ. 2010;341:c4587.View ArticlePubMedGoogle Scholar
  57. O’Cathain A, Murphy E, Nicholl J. Multidisciplinary, interdisciplinary, or dysfunctional? Team working in mixed-methods research. Qual Health Res. 2008;18(11):1574–85.View ArticlePubMedGoogle Scholar
  58. Stake R. Case Studies. In: Denzin N, Lincoln Y, editors. Strategies of qualitative inquiry. Thousand Oaks: Sage Publications; 1998.Google Scholar
  59. Farmer T, Robinson K, Elliott SJ, Eyles J. Developing and implementing a triangulation protocol for qualitative health research. Qual Health Res. 2006;16(3):377–94.View ArticlePubMedGoogle Scholar
  60. Morse JM, Barrett M, Mayan M, Olson K, Spiers J. Verification strategies for establishing reliability and validity in qualitative research. Int J Qual Methods. 2008;1(2):13–22.View ArticleGoogle Scholar
  61. Bartlett YK, Haywood A, Bentley CL, Parker J, Hawley MS, Mountain GA, Mawson S. The SMART personalised self-management system for congestive heart failure: results of a realist evaluation. BMC Medical Informatics and Decision Making. 2014;14(1):1–13.View ArticleGoogle Scholar
  62. Raaijmakers LGM, Kremers SPJ, Schaper NC, de Weerdt I, Martens MK, Hesselink AE, de Vries NK. The implementation of national action program diabetes in the Netherlands: lessons learned. BMC Health Serv Res. 2015;15(1):1–8.View ArticleGoogle Scholar
  63. Marchal B, Belle S, Olmen J, Hoerée T, Kegels G. Is realist evaluation keeping its promise? A literature review of methodological practice in health systems research. Evaluation 2012;18.Google Scholar

Copyright

© McHugh et al. 2016

Advertisement