Delivering stepped care: an analysis of implementation in routine practice
© Richards et al; licensee BioMed Central Ltd. 2012
Received: 26 April 2011
Accepted: 16 January 2012
Published: 16 January 2012
In the United Kingdom, clinical guidelines recommend that services for depression and anxiety should be structured around a stepped care model, where patients receive treatment at different 'steps,' with the intensity of treatment (i.e., the amount and type) increasing at each step if they fail to benefit at previous steps. There are very limited data available on the implementation of this model, particularly on the intensity of psychological treatment at each step. Our objective was to describe patient pathways through stepped care services and the impact of this on patient flow and management.
We recorded service design features of four National Health Service sites implementing stepped care (e.g., the types of treatments available and their links with other treatments), together with the actual treatments received by individual patients and their transitions between different treatment steps. We computed the proportions of patients accessing, receiving, and transiting between the various steps and mapped these proportions visually to illustrate patient movement.
We collected throughput data on 7,698 patients referred. Patient pathways were highly complex and very variable within and between sites. The ratio of low (e.g., self-help) to high-intensity (e.g., cognitive behaviour therapy) treatments delivered varied between sites from 22:1, through 2.1:1, 1.4:1 to 0.5:1. The numbers of patients allocated directly to high-intensity treatment varied from 3% to 45%. Rates of stepping up from low-intensity treatment to high-intensity treatment were less than 10%.
When services attempt to implement the recommendation for stepped care in the National Institute for Health and Clinical Excellence guidelines, there were significant differences in implementation and consequent high levels of variation in patient pathways. Evaluations driven by the principles of implementation science (such as targeted planning, defined implementation strategies, and clear activity specification around service organisation) are required to improve evidence on the most effective, efficient, and acceptable stepped care systems.
Evidence-based medicine (EBM) is the 'conscientious, explicit, and judicious use of current best evidence in making decisions about the care of individual patients' . Production of clinical guidelines is a conventional method of operationalising EBM and ensuring that clinical and cost-effective 'health technologies' are used in routine service settings. However determining the clinical and cost-effectiveness of health technologies does not provide a blueprint for their delivery in practice . Service delivery and organisation is the focus of health services research (HSR), which aims to "identify the most effective ways to organize, manage, finance, and deliver high-quality care; reduce medical errors; and improve patient safety" http://archive.ahrq.gov/about/whatis.htm.
In the United Kingdom (UK), the National Institute for Health and Clinical Excellence (NICE) clinical guidelines for depression [3, 4] used evidence synthesis to summarise available data on the effectiveness of individual health technologies and outline 'what works for whom.' However, despite their basis in health technology assessment, the guidelines also made strong recommendations about service delivery and organisation, suggesting that services for depression should be structured around a stepped care model [5, 6].
The adoption of the stepped care model within all NICE guidelines for common mental health problems, including now anxiety , is a reasoned response to the challenges of effective depression care, but was not based on the same, detailed evidence synthesis as recommendations about individual treatments. There are very limited data available on the operation of the stepped care model and significant questions remain about implementation . These include the optimal number of steps and the range of treatments within steps; the proportion of patients who might bypass low-intensity treatments and be referred directly to higher intensity treatments; the process of decision-making about 'stepping up' to higher intensity of care; and the degree to which stepped care systems should be responsive to local context.
One of the core tensions for those designing services for depression is the balance between 'stepped' and 'stratified' models of care. Each model is based on the idea that depression services should deliver both low-intensity and high-intensity treatments. However, the way in which patients access those different interventions is more contested. In a stepped model, the system is self-correcting, [5, 6] in that although most patients are assigned to low-intensity interventions initially, those failing to benefit are 'stepped up' to higher intensity treatment. This has the advantage of maximising the proportion of patients who might benefit from low-intensity interventions, but may potentially delay access to appropriate care for some patients. In contrast, a stratified model seeks to assign patients to particular steps on the basis of their presenting characteristics (such as initial depression severity) to better target interventions at patients likely to benefit . This is potentially a better method to ensure timely delivery of the appropriate intensity of intervention, but is dependent on accurate knowledge of the types of patients who are most likely to benefit from a particular intensity of intervention (so-called 'aptitude treatment interactions') . Although a combination of 'stepping' and 'stratification' is likely to be required, the relative importance of the two different mechanisms was not explicit in the NICE guidelines.
The NICE guidelines have underpinned a six-year £700 m UK investment in psychological therapies, the Improving Access to Psychological Therapies (IAPT) programme http://www.iapt.nhs.uk. Understanding implementation is thus a priority to ensure effective use of these resources. We wanted to investigate how mental health services in the UK implemented the service delivery and organisation recommendations of NICE so that we could provide advice to policy makers on the likely shape of stepped care service delivery systems in routine practice.
What different models of stepped care are implemented in practice?
How do patients access and flow through the different models?
What proportion of patients are managed at each step, and what proportion 'stepped up'?
The study was part of a larger operational research project to develop a decision and modelling aid for services designing stepped care organisational systems http://www.sdo.nihr.ac.uk/files/project/109-final-report.pdf.
The study was observational in design, using routine data in four NHS service sites consisting of between 22 and 46 mental health workers. Each site was recruited to this study on the basis that they were aiming to set in place services to provide patients with a low- and high-intensity treatment options as described in the NICE guidelines, organised in a stepped care manner within the context of their existing resources and personnel. In the UK, during the data collection period, mental health services for people with depression and anxiety were not delivered to a standard blueprint but could be provided by a range of National Health Service (NHS) organizations including primary care or specialist care NHS Trusts. In some cases all treatment services could be provided by one organization that may be a primary or specialist care organization. In other cases, several organizations could provide different elements of a psychological therapies service.
In a previous stage of the project (see http://www.sdo.nihr.ac.uk/files/project/109-final-report.pdf), we facilitated a consensus workshop at each site to help teams design their stepped care models. We used the 'constituency approach'  to enable sites to build consensus gradually, providing participants with skilled facilitation around clear inputs and defined outputs in a carefully structured process . Our team provided the structured environment for sites to reach their own design consensus, at no point influencing the sites' specific choices of new service configuration. Following this exercise, sites set up implementation teams locally to restructure their services.
Our conceptual framework was the 'patient pathway' , i.e., the route that a patient will take from their first contact with a clinical service, including referral, to the completion of their treatment. Given the potential diversity of organizations able to deliver these services in the NHS, we recorded service level descriptors for each site, including which professionals referred into the stepped care system; who made the initial assessment of the patient; how patients were allocated to steps; whether self referral was permitted; and whether there was a single point of entry to the system for all patients, or whether patients could 'bypass' aspects of the service. Where possible we extracted individual level patient data from information systems used by sites to manage their activity, although clinicians were asked to collect additional information on treatments received by patients where existing data systems did not collect this. All sites used different information systems and had varying levels of information collected by clinicians. We collected data on how and when patients in each site accessed, received and transited between the various treatment options, including the type of treatments, the step/intensity of the treatments, and the patient's final status at the end of the project time (i.e., dropped out of treatment, completed treatment or still remained in treatment). These data were extracted from information systems and supplementary record sheets onto a standard proforma by clerks in each site, trained and supervised by project research workers who also validated their data extractions against source data. We also recorded demographic data on patient gender, age, ethnicity, employment status, main mental health problem identified by clinicians, and medication use.
Data were anonymised by local NHS service data clerks and submitted to the research team as a flat database file and cleaned. We analysed categorical data, including demographics as frequency counts and percentages (with ranges where applicable), continuous data as means, and standard deviations. For those variables where there were missing data, we excluded these cases from the analysis of that variable, and the total number of cases used for each variable is presented in the results with percentages of data missing. We analysed information about patient pathways using start and end points and whether they were stepped up or down. We then mapped patient flow visually to illustrate patient movement through these exemplar stepped care systems.
The limited time span of the project did not allow us to collect final endpoint status data about the proportion of patients who remained in treatment at the end of the project, particularly in high-intensity treatment which by its nature is of longer duration than low-intensity.
Ethical approval for the study was given by the South West Multisite Ethics Committee in the UK (05/MRE06/71).
Population characteristics of the sites
Type of organisation
Average IMD Ranka
Primary care Trust A
Primary care Trust B
Primary care Trust C
Primary care Trust
Primary care Trust D
Primary care Trust E
Primary care Trust F
Primary care Trust G
Characteristics of stepped care services by site
Site A Northern Urban/Rural
Site B Northern Urban
Site C Southern Urban/Rural
Site D Southern Urban
Primary Care led with Specialist Trust partner
Qualified mental health practitioner face to face
Para-professional or qualified mental health worker face to face
Senior mental health worker in a GP based clinic triaging direct to steps from written information or following telephone or face to face appointment
Para-professional or psychologist face to face
Step 2 low-intensity
Guided self-help delivered by para-professionals; group classes by para-professionals and qualified mental health workers
Guided self-help delivered by para-professionals; group classes by para-professionals and qualified mental health workers
Self-directed cCBT,1 guided self-help and group classes by para-professionals and qualified mental health workers
Guided self-help delivered by para-professionals, cCBT, group classes by para-professionals
Step 3 high-intensity
Short-term evidence-based psychological interventions delivered by a mental health practitioner or practice-based counsellor
Short-term psychological interventions including brief CBT delivered by a mental health practitioner
Short-term psychological interventions including brief CBT and group work delivered by trained para-professionals and mental health practitioners
Psychology, and counselling
Step 4 high-intensity
Complex evidence-based psychological interventions delivered by psychological services, CMHT, 2 or the psychiatric service
Specialised psychological treatment delivered by community mental health teams, psychology and psychotherapy services working within the specialist mental health trust
Psychology, psychotherapy and community mental health teams
Step 5 high-intensity
Crisis teams, self-harm liaison and in-patient admission by specialist clinical teams
Crisis teams, self-harm liaison and in-patient admission by specialist clinical teams
Models of stepped care
Each of the sites had developed a model of stepped care within their available resources that they regarded as reflecting their local contexts and best serving their local population needs (Table 2). All sites allowed patient self-referral, but most restricted this to access to low-intensity treatment steps (i.e., patients could self refer for access to written self help materials and computer based treatments at step two, but not treatments involving access to contact with mental health workers at steps three, four, and five). All sites allowed a range of professionals--general practitioners (GPs), other primary care staff, and secondary mental healthcare staff--to refer, although site A also accepted referrals from the UK 'third sector' (i.e., non-governmental organisations who deliver mental health services). The majority of patients were assessed by para-professional workers in the sites, although site C offered a triage service run by professionally qualified mental health workers (e.g., mental health nurses) who made decisions about allocation to all steps including a computerised treatment programme that was designed to be used by the patient independent of any contact with the service--'step one.' Sites A, C, and D allowed professionals to refer to different steps based on their clinical assessment; site B had a single point of entry (with a few exceptions) with the intention that all patients would be initially allocated to low-intensity treatments.
Demographic characteristics of the patients
We collected data on almost 7,698 patients referred to four NHS sites operating stepped care services. Large amounts of demographic data were missing from service information systems at all sites, including age (75% missing), gender (60% missing), ethnicity (70% missing), employment status (42% missing), previous history (55% missing), identified problem (42% missing), medication status (54% missing), and sickness status (47% missing). On the basis of the available data, the patient populations accessing the stepped care services were largely female (63% to 67% across sites) and aged 20 to 39 (56% to 62%) with most employed (50% to 62%). Ethnic minority rates varied more widely (81% to 96% caucasian British). Absence from work through illness also varied widely (11% to 27%), with between 41% and 66% reporting a previous treatment history for anxiety or depression and between 48% and 65% taking psychotropic medication. All patients with available data were identified as experiencing anxiety, depression, or a mixed combination.
Service delivery and patient pathways
Summary of number of patients referred and accessing each step by site
Step 1 b
Step 2 b
Step 3 b
Step 4 b
In marked contrast to site A, site B developed a more stepped model. Few patients were allocated immediately to high-intensity treatment and the overall ratio of low- to high-intensity treatments was 2.6:1, the opposite pattern to site A. Of those patients who were allocated to and subsequently received a treatment, one-third received high-intensity treatment compared to two-thirds in site A. In site B, where entry was not controlled by experienced professionals, these experienced workers were more able to utilise their time in the delivery of high-intensity treatment.
Site D was a complex mixture of 'stepped' and 'stratified' delivery. Although the ratio of low- to high-intensity treatments actually received by patients favoured low-intensity interventions (1.4:1), many more patients were assessed at a high-intensity level. The main differentiating factor in site D was the ability of GPs to make a direct referral to high-intensity primary care psychology. As a consequence, there were two very clear entry points for patients. Although more patients were referred directly to low-intensity compared to high-intensity workers, a considerable number of patients could omit this step.
In summary, the ratio of low- and high-intensity treatments received by patients varied across the sites and ranged from 22:1 (site C including unsupported step one activity), through 2.1:1 (site B), 1.4:1 (site D) to 0.5:1 (mainly high-intensity treatments in site A). In essence, this means that of people treated by site C, 20 times more patients received a low- rather than a high-intensity treatment, whereas in site A twice as many patients received high- as opposed to a low-intensity treatment, a forty-fold allocation difference between the two sites. Furthermore, between sites, the numbers of patients allocated directly to high-intensity treatment varied from 3% to 45%. Rates of stepping up to eventual high-intensity treatment were less than 10% in all sites, although a small number of patients moved from step three to step four within the high-intensity phase of their treatment.
For the three sites where we have referral data, all 'lost' considerable numbers of patients between referral and assessment, ranging from 21% to 34% of referrals, and a similar number were not treated after assessment (Figure 4). Referrals that were not assessed were never seen by the services at all, and we have no data on what happened to these patients. An unscheduled discontinuation rate (i.e., where patients drop out of treatment without agreement with a therapist) of 30% was common across treatment steps and sites, although there were variations between sites and steps. As noted in the methods section, the low rates of both scheduled (i.e., agreed) and unscheduled discontinuation in high-intensity treatment in some sites are most likely a product of the longer treatment times for high-intensity resulting in a proportion of patients remaining in treatment at the project end date and whose final pathway status was not recorded.
We have reported throughput data on almost 8,000 patients referred to four NHS sites operating stepped care services for common mental health problems, mapping patients' entry and exit from the systems as well as recording the types of therapeutic inputs received. Our data illustrate the considerable variation in the design and implementation of care systems in response to the recommendation for stepped care in the NICE guidelines , which lacked explicit detail concerning the optimal model for delivery.
The variation in models was significant. Although it is helpful to place them on an operational continuum from stepped to stratified, this categorisation does not do sufficient justice to their complexity and diversity. While the design of the systems as 'stepped' or 'stratified' is a key dimension that dramatically influences the performance of stepped care systems, staff availability and professional referral behaviour can subvert initial plans in important ways. For example, lack of experienced workers meant that site C became essentially a low-intensity-only service, whereas site D saw a 'stepped' model bypassed by a significant number of patients, although this might also be influenced by patient population characteristics, such as the proportion of patients diagnosed with post traumatic stress disorder where low-intensity treatments are not known to be effective.
In contrast, however, there were some interesting similarities in patient flow between sites. The levels of attrition between referral and assessment have been observed previously in stepped care services [14–16] and are a well-known phenomena in psychological therapies services [17–19]. In the two sites where there was closer balance between low- and high-intensity provision (sites B and D), we observed a consistent 'stepping up' rate of less than 10%, similar to that reported in one of the first Improving Access to Psychological Therapies (IAPT) demonstration sites over three years .
Scheduled completion rates for all treatments were variable between sites and steps (Figures 2 and 3). Site D achieved less treatment completion at high-intensity, where only one-third of cases were recorded as completing their high-intensity treatment, although our data are limited by project end dates as cited earlier. Nonetheless, low-intensity treatment is intrinsically shorter than high-intensity treatment, and it is conceivable that patients will be more likely complete a shorter course of low-intensity treatment than the longer high-intensity option. This observation requires further investigation since the concept that treatments should be 'least restrictive' for the patient [5, 9, 20] is at the heart of the original concept of stepped care.
Our study is of four 'early implementer' sites attempting to reconfigure their services to more closely reflect NICE guidelines . As a consequence, the data may have limited generalisability. Missing demographic data (a product of poor local information systems and haphazard clinical data entry highly prevalent in UK mental health services) limits knowledge of the characteristics of the populations served by these sites.
A more significant limitation is our inability to utilise clinical outcome data to ascertain the outcomes for patients, rather than their flow through the systems. Clinicians and information systems were unable to provide us with this data. Recent evaluations of Australian 'Better Access' mental health systems have similarly suffered from a lack of outcome data, basing published evaluations on a mere 15% of all patients treated . However, it should be noted that the bulk of treatments delivered were evidence-based, as summarised in the NICE guidelines , and our analysis assumes that outcomes would be broadly in line with those reported in the guidelines. However, this remains to be confirmed, especially in the context of a stepped care system where patients may receive a number of treatments in a serial fashion.
That clinical services do not use routine data is of greater concern, given the supposed centrality of outcome monitoring in stepped care to support clinical decision-making and ensure that services are responsive when patients do not benefit from initial 'steps.' The assessment of treatment effect to aid clinical decision-making appears deficient in these services. However, the use of formal psychometric measures is not the only way to assess treatment progress. Clinical judgement may be applied by mental health workers using clinical interview assessments. Clearly, for research purposes this is less easy to quantify without routine outcome measures, but their absence does not necessarily invalidate the stepped care process given our patient pathway data. Whether the use of such measures within a formal clinical decision-making algorithm would have reduced the variation in patient pathways could be a potential subject for further research.
We were also ignorant as to any additional service options available to patients and referrers outside the services we studied. Other resources available may alter who is referred to these services and we cannot assume that the proportions of patients 'needing' step two or step three interventions would be the same in all sites.
The study highlights variability in the implementation of stepped care for depression. This is to be expected, to the degree that the NICE guidelines were not explicit about a number of issues, and provided no formal 'blueprint' for the organisation and delivery of services. Our observations of variability are confirmed by the report on the first-year IAPT service . Of course, some local variation is desirable, but the very different service delivery models may not be desirable in the long term. It is important that evidence from implementation studies such as that reported here impact on later iterations of the depression guidelines, to maximise standardisation where appropriate and ensure that models of service delivery have a solid basis in evidence, complementing the evidence base relating to the health technologies which are delivered within these services.
Our data suggest that the principal driver of patient flow through stepped care systems is the allocation to initial treatments. The rate of stepping up was low, no matter how the patients were assessed or how many were allocated directly to high-intensity treatment. Not dissimilar proportions of patients were stepped up in systems which allocated large numbers directly to high-intensity treatments, allowed referrers to make direct referral to high-intensity treatment or direct most patients to low-intensity treatment. The two services which stepped fewer patients from low- to high-intensity treatment included one where lack of resources led to very little high-intensity treatment provision, and another where initial allocations to high-intensity treatment was almost 50% of referrals assessed.
Although service planners may seek to design services that reflect their desired balance between stepping and stratification, they must be aware that patient flow is highly sensitive to other factors, including the background of the workers at each step of the service. Triage or assessment by a professionally qualified workforce may lead to more people receiving high-intensity treatment, providing that option is available. Service managers may need to plan on the basis that whatever the initial allocation rates of patients to low- or high-intensity treatments, providing sufficient high-intensity resource is available, less than 10% of patients may be stepped up from low- to high-intensity treatment. It is, therefore, important to resource all available steps sufficiently to allow patients to be stepped up from low- to high-intensity treatment and to prevent situations arising where patients might be inappropriately 'held' at a low-intensity step.
Finally, stepped care systems do not seem to differ from the often observed attrition rates to psychological therapies at all stages in the patient pathway. Access to care has not traditionally received the same research focus as issues of treatment effectiveness. That is now beginning to change [7, 23, 24] but there is an urgent need to understand the reasons for these levels of attrition and ensure the findings are used to inform the design of stepped care systems in the future.
Although implementation science is relatively new to the evidence-based movement, its use in ensuring that effective treatments and organisational models are put in place consistently is now recommended by research funders . Stepped care would seem to be a prime example of a recommended organisational system idea being interpreted and applied in very different ways. The application of core principles of implementation science (such as targeted planning, implementation strategies and clear activity specification) around service organisation is urgently required.
We would like to thank the site-based NHS audit clerks who collected and collated the data on patient throughputs and to Kerry Cipriano and Joanne English who provided administrative support. We would like to acknowledge the contribution of the NHS clinical staff, patients, managers, and commissioners who were involved in our four NHS clinical service sites. The study was funded by the National Institute for Health Research (NIHR). The views and opinions expressed herein are those of the authors and do not necessarily reflect those of the NHS, the NIHR or the Department of Health.
- Sackett D, Rosenberg W, Gray J, Haynes B, Richardson W: Evidence-based medicine: what it is and what it is not. British Medical Journal. 1996, 312: 71-2. 10.1136/bmj.312.7023.71.PubMed CentralView ArticlePubMed
- Grol R, Wensing M, Eccles M: Improving Patient Care. The Implementation of Change in Clinical Practice. 2005, London: Elsevier
- National Institute for Health and Clinical Excellence: Depression: the treatment and management of depression in adults (update) CG90. London. 2009
- National Institute for Health and Clinical Excellence: Depression in Adults with a Chronic Physical Health Problem: treatment and management CG91. London. 2009
- Haaga D: Introduction to the special section on stepped care models in psychology. Journal of Consulting and Clinical Psychology. 2000, 68: 547-8.View ArticlePubMed
- Sobell M, Sobell L: Stepped care as a heuristic approach to the treatment of alcohol problems. Journal of Consulting and Clinical Psychology. 2000, 68: 573-9.View ArticlePubMed
- Gulliford M, Figueroa-Munoz J, Morgan M, Huges D, Gibson B, Beech R: What does 'access to health care' mean?. Journal of Health Services Research and Policy. 2002, 7: 186-8. 10.1258/135581902760082517.View ArticlePubMed
- Pilling S, Whittington C, Taylor C, Kendrick T: Identification and care pathways for common mental health disorders: summary of NICE guidance. British Medical Journal. 2011, 342:
- Bower P, Gilbody S: Stepped care in psychological therapies: access, effectiveness and efficiency: narrative literature review. British Journal of Psychiatry. 2005, 186: 11-7. 10.1192/bjp.186.1.11.View ArticlePubMed
- Richards D: United Kingdom perspective. Depression in primary care: evidence and practice. Edited by: Gilbody S, Bower P. 2010, Cambridge: Cambridge University Press, 127-36.View Article
- Connally T, Conlan E, Deutsch S: Organizational effectiveness: a multiple constituency approach. Academy of Management Review. 1980, 5: 211-17.
- Murphy M, Black N, Lamping D, McKee C, Sanderson C, Askham J: Consensus development methods, and their use in clinical guideline development. Health Technology Assessment. 1998, 2 (3): 1-88.
- Department of Health: Patient Pathways London. 2007, Available from: http://webarchive.nationalarchives.gov.uk/+/www.dh.gov.uk/en/Healthcare/Primarycare/Treatmentcentres/DH_4097263
- Richards D, Suckling R: Improving access to psychological therapies: Phase IV prospective cohort study. British Journal of Clinical Psychology. 2009, 48: 377-96. 10.1348/014466509X405178.View ArticlePubMed
- Clark D, Layard R, Smithies R, Richards D, Suckling R, Wright B: Improving access to psychological therapy: Initial evaluation of two UK demonstration sites. Behaviour Research and Therapy. 2009, 47 (11): 910-20. 10.1016/j.brat.2009.07.010.PubMed CentralView ArticlePubMed
- Richards D, Borglin G: Implementation of Psychological Therapies for Anxiety and Depression in Routine Practice: Two Year Prospective Cohort Study. Journal of Affective Disorders. 2011, 133: 51-60. 10.1016/j.jad.2011.03.024.View ArticlePubMed
- Barrett M, Chua W, Crits-Christoph P, Gibbons M, Casiano D, Thompson D: Early withdrawal from mental health treatment: implications for psychotherapy practice. Psychotherapy. 2008, 45: 247-67.PubMed CentralView ArticlePubMed
- Gilbert N, Barkham M, Richards A, Cameron I: The effectiveness of a primary care mental health service delivering brief psychological interventions: a benchmarking study using the CORE system. Primary Care Mental Health. 2005, 3: 241-51.
- Wierzbicki M, Pekarik G: A meta-analysis of psychotherapy dropout. Professional Psychology: Research and Practice. 1993, 24: 190-5.View Article
- Davison G: Stepped care: doing more with less?. Journal of Consulting and Clinical Psychology. 2000, 68: 580-5.View ArticlePubMed
- Pirkis J, Bassilios B, Fletcher J, Sanderson K, Spittal M, King K: Clinical improvement after treatment provided through the Better Outcomes in Mental Health Care (BOiMHC) programme: Do some patients show greater improvement than others?. Aust N Z JPsychiatry. 2011, 45: 289-98. 10.3109/00048674.2010.539195.View Article
- Glover G, Webb M, Evison F: Improving Access to Psychological Therapies. A review of the progress made by sites in the first rollout year. 2010, Durham: North East Public Health Observatory
- Dixon-Woods M, Kirk D, Agarwal S, Annandale E, Arthur T, Harvey J: Vulnerable groups and access to health care: A critical interpretive review. 2005, London: NIHR-SDO
- Dowrick C, Gask L, Edwards S, Aseem S, Bower P, Burroughs H: Researching the mental health needs of hard-to-reach groups: managing multiple sources of evidence. BMC Health Services Research. 2009, 9:
- Medical Research Council: Developing and evaluating complex interventions: new guidance London. 2008
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.