- Systematic review
- Open Access
- Open Peer Review
Learning from the emergence of NIHR Collaborations for Leadership in Applied Health Research and Care (CLAHRCs): a systematic review of evaluations
Implementation Science volume 13, Article number: 111 (2018)
Collaborations for Leadership in Applied Health Research and Care (CLAHRCs) were funded by NIHR in England in 2008 and 2014 as partnerships between universities and surrounding health service organisations, focused on improving the quality of healthcare through the conduct and application of applied health research. The aim of this review is to synthesise learning from evaluations of the CLAHRCs.
Fifteen databases including CINAHL, MEDLINE, EMBASE and PsycINFO were searched to identify any evaluations of CLAHRCs. Current and archived CLAHRC websites and the reference lists of retrieved articles were scanned to identify any additional evaluations. Searches were restricted to English language only. Any publications from evaluations of the CLAHRCs were eligible for inclusion if they fulfilled at least one of three pre-specified inclusion criteria. A narrative synthesis was undertaken.
Twenty-six evaluations (reported in 37 papers) were deemed eligible for inclusion. Evaluations focused on describing and exploring the formative partnerships, vision, values, structures and processes of CLAHRCs; the nature and role of boundaries; the deployment of knowledge brokers and hybrid roles to support knowledge mobilisation; patient and public involvement; and capacity building. The relative lack of data about the early impact of CLAHRCs on health care provision or outcomes is notable.
Much of the evaluative focus on CLAHRCs has been on how they have been organised and on the development of theory around their emergent properties. Evidence is lacking on the impact of CLAHRCs particularly in relation to the knowledge mobilisation processes and practices adopted. Further evaluation of CLAHRCs and other similar research and practice partnerships is warranted and should focus on which knowledge mobilisation approaches work where, how and why.
PROSPERO (Registration number: CRD42016042945).
Healthcare has long seen significant investment in the production of research evidence to inform decisions and choices around the delivery and organisation of services. However, making use of research-based knowledge routinely has been a challenge and one that has been described as the ‘second translation gap’ . Growing recognition of the need to accelerate the generation and uptake of knowledge in health systems has led to a focus on the development of new models of research and practice partnership [2, 3]. Such collective knowledge mobilisation processes are increasingly viewed as integral to the development of learning health systems which seek to improve care through a continuous cycle of knowledge production and implementation .
In the USA, the Veterans Health Administration through its Health Services Research and Development Service and the Quality Enhancement Research Initiative has been at the forefront of efforts to enhance partnered research [5, 6]. This has been mirrored in other geographical settings, such as the establishment of Advanced Health Research and Translation Centres by the National Health and Medical Research Council in Australia . In the UK, a report by the Chief Medical Officer’s Clinical Effectiveness Group in 2007 recommended that the National Health Service (NHS) should better utilise higher education to support initiatives to enhance the uptake of applied health research into routine practice . This recommendation prompted the development of new models of research and practice partnership. In 2008, the National Institute for Health Research (NIHR) funded nine ‘pilot’ Collaborations for Leadership in Applied Health Research and Care (CLAHRCs): collaborative partnerships between universities and surrounding NHS organisations, focused on improving patient outcomes through the conduct and application of applied health research. Each CLAHRC was required to obtain ‘matched funding’ from partners to the value of the NIHR investment. The aim was to create and embed approaches to research and its application that are specifically designed to take account of the way that health care is delivered across sectors and a clearly defined geographical area.
In 2014, a further round of funding was awarded to 13 CLAHRCs across England with the same matched funding requirements . Each CLAHRC has developed independently within a local context with key service stakeholders and researchers playing an important role in shaping the focus for research and improvement. The CLAHRCs therefore represent an ongoing nationwide experiment to improve collaboration between academic and health partners, and consequently to increase research impact for the benefit of patients.
In 2010, the NIHR Service Delivery and Organisation Programme (now known as the Health Services and Delivery Research (HS&DR) Programme) commissioned independent longitudinal research evaluations of the pilot CLAHRCs through an open call . The call asked for evaluations that reflected ‘the dynamics, processes, emergent properties and diverse impacts of the CLAHRCs’ as they developed . Applications that drew on the ‘broad diversity of evaluation approaches including exploratory, descriptive, experimental, programme and economic evaluation approaches’ were to be encouraged. The call also indicated that funded evaluations were expected to contribute to the growing international knowledge base on research use and impact and to generate evidence with broader applicability for the development of other research and practice partnerships beyond the CLAHRCs.
There have been no such evaluations commissioned by NIHR since these in 2010, and given that the second round of CLAHRC funding was not referred as ‘pilot’ funding, it might be assumed that NIHR have been convinced of the ‘value’ of CLAHRCs through the pilot funding round. None of the commissioned evaluations published their final reports before the second round of funding although it is possible that unpublished early findings were fed in informally to NIHR as part of the commissioning process for the 2014 funding round.
NIHR also required routine performance information from CLAHRCs, which was focused on research metrics used for other types of NIHR funding (e.g. biomedical research). These metrics included numbers of publications, numbers of funded students awarded higher degrees, additional research funding leveraged, impact on health care and patients through ‘case studies’ .
Our aim with this review is to synthesise what has been learnt through evaluation (and published) about the process and impact of the CLAHRCs. We have focused on published papers because of the requirement from the funded evaluations, and of CLAHRCs generally, to contribute to knowledge. Specifically, we are interested in what evaluations tell us about how CLAHRCs work and are organised; how they have assessed any emergent impacts of CLAHRCs; and what strengths and limitations are apparent in the ways by which CLAHRCs have been evaluated to date.
The protocol was registered in PROSPERO (Registration number CRD42016042945).
Data sources and searches
We searched the following databases: CINAHL, MEDLINE, EMBASE, PsycINFO, Cochrane Methodology Register, Cochrane Central Register of Controlled Trials, Cochrane Database of Systematic Reviews, Database of Abstracts of Reviews of Effects, Health Technology Assessment, NHS Economic Evaluation Database, HMIC Health Management Information Consortium, SPORTDiscus, Scopus, TRiP database and PROSPERO. All the searches were restricted to English language only and were conducted in June 2016. Update searches were conducted up to June 2018 using the same search terms and databases. Details of the search strategies are available in Additional file 1.
As our focus was on identifying evaluations of CLAHRCs, we also searched for eligible studies in current and archived CLAHRC websites since we were aware that some CLAHRCs had carried out internal evaluations. Reference lists of retrieved articles were scanned to identify any additional studies.
Any published empirical papers drawing on data from an evaluation of CLAHRCs or some aspect of them were eligible for inclusion if they fulfilled at least one of the following criteria:
an external or internal evaluation of the CLAHRC(s) or CLAHRC process,
an exploration of the CLAHRC(s) as a novel organisational form and
development of theory using the CLAHRC(s) as a research setting, i.e. including empirical data.
As our focus was on identifying evaluations of CLAHRCs as an entity, any evaluations that were based around a single project conducted within a CLAHRC were excluded from the review. This included descriptive accounts aiming to showcase the achievements of an individual project without providing rigorous evidence and/or critical analysis of these achievements; and/or (2) theory-building accounts that use a single project as an empirical illustration of a broader theoretical issue.
References were loaded onto the systematic review web app Rayyan QCRI  for title and abstract screening. Study selection was performed independently by one researcher and checked by a second. All full text studies that were provisionally excluded were discussed collectively by the research team.
Data extraction and quality assessment
From the primary output paper for each identified evaluation, details of the type and main findings were extracted and assessed by one researcher and checked by a second. As NIHR funded studies are extensively peer reviewed and quality assured prior to publication, we did not undertake separate quality assessments for all four NIHR funded evaluations. The other included CLAHRC evaluations are presented descriptively with any major limitations in reporting highlighted.
Method of synthesis
As the NIHR funded evaluations were mixed methods, and the other included evaluations were largely qualitative, we performed a narrative synthesis of the evidence. Consistent with an integrative approach to synthesising evidence, the narrative synthesis aimed to present a descriptive summary of findings across studies and then to generate, across reported findings, a number of themes relevant to the aims of this review. The original commissioning brief anticipated that evaluations may address organisational form, structure and processes, funding arrangements, nature of formative partnerships, engagement of health care users and the general public, emerging impacts and potential for sustainability of change . We used these themes as a guiding framework to help answer our research questions on organisation, impact and evaluation. An iterative process of adaptation and refinement was undertaken by two researchers to generate initial themes, and these were further refined via consensus discussions with the full research team.
Given the interdisciplinary nature of CLAHRC work and the resulting diversity of papers being reviewed, it was particularly important to minimise individual disciplinary biases when synthesising the literature. This was accomplished through regular reflective discussions within the research team (which included two organisation and management scholars and two health services researchers) as well as through internal review from academic colleagues and CLAHRC managers in the role of ‘critical friends’.
After de-duplication, we identified a total of 2045 records through database searching and a further 10 records through other web based sources. Titles and abstracts were screened, and 61 full text papers were assessed for inclusion (see Fig. 1: PRISMA flow diagram).
We excluded 24 papers on eligibility grounds. Given our stated focus on emergent impacts, we did not include papers that presented descriptive accounts of CLAHRC(s) processes (n = 7) or those that were conceptual and not based on empirical data (n = 6). We also excluded studies that were based around a single project conducted within a CLAHRC (n = 4) rather than addressing a CLAHRC as a whole. Five protocols and two papers unrelated to the evaluation of CLAHRCs were also excluded. We checked the reference lists of excluded papers to ensure we had identified and included all relevant evaluations.
In total, 26 studies (reported in 37 papers) were deemed eligible for inclusion. We included all four NIHR funded independent evaluations of CLAHRCs [12,13,14,15] (also reported in a further 11 papers [16,17,18,19,20,21,22,23,24,25,26]). Details of the included evaluations and their associated outputs are presented in Table 1. The four NIHR funded evaluations were all longitudinal and mixed-methods by design and drew their conclusions from the analysis of more than one CLAHRC. Table 2 summarises the main findings from the each of the four NIHR funded evaluations. A further 22 studies [27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48] of aspects and processes of individual CLAHRCs were also identified, and these are presented in Table 3.
Synthesis of findings
Five prominent themes were identified from the literature: organisational form and emergent properties, the nature and role of boundaries, the deployment of knowledge brokers and other hybrid roles to support knowledge mobilisation, engagement of health care users and the general public in the form of patient and public involvement (PPI), and capacity building. We describe each of these themes in turn.
Organisational form and emergent properties
All the NIHR funded evaluations highlight the influence of local context and the interplay between local research producers and the key health service actors in shaping the initial design and organisational form of each CLAHRC. Drawing on a comparison of all nine CLAHRCs, five different knowledge translation ‘archetypes’ have been proposed to represent the different ways of achieving the balance between research production and research implementation (see Lockett et al., Table 2) . However, Fitzgerald and Harvey caution that the rigid structural design of a CLAHRC may adversely impact its performance, particularly if the adopted form does not readily facilitate the intended function of knowledge mobilisation .
According to Soper et al.,  key features of the CLAHRCs include a range of knowledge mobilisation approaches, efforts to promote cultural change and freedom to experiment, learn and adapt while Rycroft-Malone et al.  identified collaborative action, relationship building, engagement, motivation, knowledge exchange and learning as key mechanisms important to the processes and outcomes of CLAHRCs.
The way each CLAHRC developed was highly influenced by the vision and beliefs of their leaders; they shaped the type of resulting social networks, and the way different groups worked together . Senior leaders and managers played an important formative role in selecting, enacting and interpreting different knowledge mobilisation practices . Many were well-known clinical academics and relied on existing relationships to support early mobilisation activity. But in doing so, they may also have restricted the development of novel, integrated approaches to the production and implementation of applied health research . Despite this, it was shown that the CLAHRC initiative led to the development of relationships that span the ‘research to practice’ divide and have been able to work across professional and organisational boundaries .
Nature and role of boundaries
Multiple types of boundaries were highlighted across evaluations. Rycroft-Malone et al.  suggest that the way in which CLAHRCs had developed their organisational form resulted in the reinforcement, rather than resolution, of boundaries between research and practice, between higher education and health services and between communities. They argue that the different perspectives which individuals and groups brought to the issue were a function of, and perpetuated, professional and epistemic boundaries . The geographic delineation of the CLAHRCs resulted, in turn, in physical and spatial boundaries. Similarly, Kislov describes the boundary between the research and implementation activities that gives rise to discontinuities in knowledge sharing within one CLAHRC,  whereas Currie et al.  describe epistemic differences and power struggles unfolding between health services researchers and organisation scientists in relation to the CLAHRC activities.
Analysis by Scarbrough et al.  focused on the differences between ‘bridging’ and ‘blurring’ approaches to boundary spanning. Where a CLAHRC framed knowledge mobilisation as the dissemination of high-quality evidence into practice, ‘bridging mechanisms’ were utilised to overcome the boundaries between research and practice. In contrast, where greater emphasis was placed on the integration of research practices with practical concerns, ‘blurring’ of boundaries occurred to a much greater extent. Scarbrough et al.  argue that reliance on these different mechanisms seems to reflect the relative extent of ‘epistemic’ differences between the communities involved as well as the specific local configurations of contextual factors. Furthermore, they suggest both approaches could be used simultaneously as what determines their appropriateness is ‘not the model per se, but rather the interplay between an initiative’s specific context and unfolding role-enactment and work-practices’ .
Whilst the evaluative literature focused mainly on developing the theory around the concepts of boundaries and boundary spanning, some useful practical implications were also drawn. CLAHRCs should ‘diagnose’ the existing professional and organisational context when implementing knowledge mobilisation projects,  actively facilitate the negotiation of concepts, approaches, and objectives that are interpreted in conflicting ways by different groups, create incentives to support productive joint working, and articulate the overarching goals and philosophy of a collaborative enterprise at early stages . Drawing on the internal evaluation of one CLAHRC, Martin and colleagues demonstrate that deep-seated institutional divisions between CLAHRC members were ‘overcome’ by concerted action resulting from the External Advisory Review .
Deployment of knowledge brokers and other hybrid roles
A number of evaluations explored the use of knowledge brokering and ‘hybrid’ roles to support knowledge mobilisation within the CLAHRCs. These types of roles are often proposed as a means to overcome ‘boundaries’. Although often seen as a promising solution to the problem of bridging the second translational gap, evaluations highlight that there is often lack of support and recognition for these roles at an organisational level, and that formidable professional boundaries, existing organisational norms and lack of institutionalised career pathways for knowledge brokers may make such roles difficult to sustain in the longer term [29, 48]. The potential of formalised knowledge brokering roles can also be decreased by over-formalisation, infrequency of interaction, competition for recognition and resources, low trust and lack of rewards . Scarbrough et al.  also show that in more decentralised structures, lack of clarity of the nature of the role specifications may limit the effectiveness of knowledge brokering.
In their study of clinicians seconded to roles as formalised knowledge brokers, Kislov et al.  describe the strategies such clinicians deploy to surmount challenges associated with bridging multiple boundaries: (1) relying on additional boundary ‘bridges’, (2) conforming to existing ways of doing things and (3) shifting from ‘facilitating’ to ‘doing’. Their analysis sheds new light on the limitations of clinicians as designated knowledge brokers, demonstrating that, paradoxically, professional authority can sometimes become an impediment to the successful realisation of all dimensions of knowledge brokering.
In a broader study into the evolution of formalised knowledge brokering roles over time, Kislov et al.  discuss how knowledge brokers accumulate, convert and mobilise different forms of ‘capital’ to achieve legitimacy with multiple stakeholder groups. Unintended (and largely unexpected) consequences of legitimation include exclusion of some stakeholder groups (for example, academic researchers) from bridging the gap between research and practice as well as the gradual transformation of ‘knowledge brokers’ into ‘managers’, with a corresponding decrease in their brokering activities on the ground.
Finally, at an individual level of analysis, Spyridonidis et al.  describe that by creating hybrid physician-manager roles that make sense to professionals, so as to enable knowledge mobilisation, some (the ‘innovators’) easily nested this role within their existing professional identity. Others (the ‘sceptics’) found it much harder and believed that it might erode their professional autonomy. Many who initially resisted (the ‘late majority’) eventually came around, once they could redefine the role as one more around clinical leadership.
Engagement of health care users and the general public
None of the NIHR funded evaluations had a particular focus on PPI; other evaluations of relevant structures and processes were also relatively scarce. Soper et al.  did interview PPI representatives from two CLAHRCs as part of their case studies. They suggest that where the CLAHRC had existing expertise and relations they could, and did, build strong relations with such stakeholders.
Three studies investigated how PPI was enacted and how patient and professional roles developed over time in CLAHRCs [27, 41, 43]. One of these describe how patients were able to draw on elements of organisational culture (such as an emphasis on non-hierarchical, multidisciplinary collaboration) to help them collaborate with healthcare professionals,  whilst another explores how patients’ views on PPI differ from those of healthcare professionals . This latter study highlights the need to not only take patient voices into account but also to track the dynamic social processes and networks through which PPI can make a contribution to health-care improvement efforts. Given the ostensible requirement for collaborative partnership with patients, it is likely that authentic efforts to achieve this in practice will result in the same complexities being encountered as covered in themes above, requiring the same attention and consideration to navigate.
Increasing the capacity to undertake and use applied health research in the NHS and to foster a culture of collaboration between the academic and service delivery sectors was one of the key objectives that CLAHRCs were required by NIHR to address. Soper et al. surveyed NHS and academic staff across six CLAHRCs and found that both NHS and academic respondents strongly supported both of these aims. Although these aims were well understood, there was considerable uncertainty about how best to achieve them in practice, and CLAHRCs themselves felt that 5 years was too short a time in which to embed their approach and change the ‘norms’ of the service .
A small individual evaluation exploring capacity building in one CLAHRC suggests criteria for judging the success of capacity building secondment arrangements . The study describes an experiential model of capacity development and reports different experiences of academic and clinical secondees. The academic secondees reported considerable personal development, but there was no evidence that secondments led to further involvement in research. Clinical secondees benefited from ongoing clinical engagement helping to maintain their credibility with staff whose practice they sought to influence. Findings also suggest that secondees required mentorship from host teams and support from managers in seconding organisations to maximise the benefits to individual secondees and to the organisations involved.
A significant investment was made in independent external evaluations of the ‘pilot’ CLAHRC initiative by NIHR. In addition, others (mainly funded through individual CLAHRCs) also carried out and published evaluations. To our knowledge, this review represents the first attempt to systematically capture learning from these sources. Evaluations have largely focused on describing and exploring the leadership, vision, values, structures and processes of CLAHRCs, the nature and role of boundary spanning and hybrid roles, the deployment of knowledge brokers and other hybrid roles to support knowledge mobilisation.
The relative lack of data about the early impact of CLAHRCs on health care provision or outcomes, whilst understandable due to the inevitable time lag between an intervention and its impact, is notable. To date, no systematic assessment of impact appears to have been made nor do there appear to be any plans in place to assess this. Assessing outcomes and sustainability requires a sufficient timeframe, and it would be difficult to expect that the NIHR funded evaluations could fully address these issues so early in the development of CLAHRCs. However, reflecting on the impact of the CLAHRCs was an original commissioning aim, and the opportunity to at least develop and share formative learning on the nature and type of impacts appears to have been missed. As no further funding for independent evaluations was made available by NIHR beyond that for the initial ‘pilot’ phase of CLAHRCs, longitudinal insights are also lacking.
This opportunity foregone may be a feature of the funded evaluations themselves and may reflect a preoccupation with the need on the part of the evaluators to generate high quality academic outputs over providing more pragmatic insights into what works, how and why. Indeed, we have found that much of the evaluative focus has led to the development of theory around emergent properties and processes. Whilst theory provides a foundation for further scientific insight, evidence on the impact of many of the emergent properties of CLAHRCs, particularly in relation to the knowledge mobilisation processes and roles that were adopted remains sparse. There is a large body of practical experience and learning that CLAHRCs will have gained from their work. However, much of this learning is currently ‘locked up’  with the CLAHRCs themselves, undermining the further development of international knowledge base on research use and impact. Indeed, Davies et al.  highlight that these new models of partnership, which have aimed to improve the research to practice gap, have instead perpetuated a gap in our understanding of the effects of knowledge mobilisation in practice.
The role of capacity building as both a contributing process and an intended outcome in itself needs to be further examined. Given that capacity building was one of the three main objectives of the CLAHRCs [50, 51], relative scarcity of empirical data on how it was (or was not) enacted in practice, highlights a significant area for future research. In light of Gerrish et al.  reporting different experiences of academic and clinical secondees, it is necessary to better differentiate pathways to capacity building depending on the target group(s) involved (e.g. academics, clinicians, managers or hybrid roles) as well as recognise that enhanced capacity should be considered not only at individual level, but also at the level of teams and organisations. In particular, it is important to understand not only the impacts on capacity of the partner organisations, (the focus of [Soper’s evaluation ), but also develop and test ways of developing the capacity of academics themselves to deliver co-production projects. The latter should take into account both capacity to produce impact through knowledge mobilisation and capacity to produce high-quality research despite conflicting priorities and workload pressures.
We recognise that the range of knowledge mobilisation approaches adopted by CLAHRCs reflects the different personal, professional and organisational contexts in which they have evolved. As such, knowledge mobilisation is inherently complex, and the mechanisms through which activities produce intended (or unintended) outcomes can be highly context-dependent, making any evaluation challenging. Given the problem of attribution and the time lag between the end of an intervention and its medium- and long-term outcomes, the preference for formative, as opposed to summative, evaluations in the extant literature is hardly surprising. In addition, the intermediary position of knowledge mobilisation at the conflict-laden interface of policymaking, management, science and professional practice is likely to further politicise any evaluation attempts and affect the utilisation of their outputs.
Multiple questions remain about the ways in which evaluations could inform the actual practices of knowledge mobilisation despite the political tensions described above. If further evaluation is to be helpful to those involved in current and future collaborative partnerships such as CLAHRCs, as well as those developing methods of collaboration and co-production between research users and producers more generally, there remains a need to move beyond ‘cataloguing’  to testing and linking these adopted and adapted strategies to impacts and outcomes. This should include novel methodological work developing or critically analysing the use of quantitative, qualitative and mixed methods to deliver timely, relevant and rigorous summative evaluations of deliberate knowledge mobilisation strategies in a range of settings and contexts. However, who should do this remains unclear. Knowledge production and mobilisation are a key focus in many of the current CLAHRCs and, understandably, any further reflection, self-evaluation and/or critical examination of the process of research itself may not be seen as a major priority. Any locally funded evaluation is also likely to be under-resourced as a result.
The relative lack of data on the practical implications and evidence-based ‘lessons learnt’ (beyond those developed within one CLAHRC ) for those who are actually ‘doing’ CLAHRC business is notable, despite the particular emphasis on sharing formative learning with the CLAHRCs within the original commissioning brief . The developing academic literature (where this might not be expected to constitute a key element) does not appear to be complemented by publicly accessible literature with a more pragmatic ‘how to do’ focus. The benefit to practice of the large funding invested in evaluations of the pilot CLAHRCs by NIHR is not evident from this analysis in terms of outputs or timing, especially given that the second round of CLAHRCs started in 2014, before any of the findings from the NIHR funded evaluations were published.
Some other topics have received relatively little evaluative attention: role CLAHRCs can or should play in supporting sustainability and scale-up, the nature and extent of collaboration between and across CLAHRCs, the effects of co-production on the nature, scope and quality of research conducted by CLAHRCs. The Directors of the early CLAHRCs also identified challenges from their perspective; these included maintaining and sustaining resources dependent on matched funding arrangements, ensuring that a full range of NHS professional groups are engaged, the need to demonstrate both academic outputs and improvements in care . But these too appear not to have been given much attention in funded evaluations.
It is widely recognised that there is a need for greater evaluation of the outcomes of patient involvement, [55, 56] but this synthesis demonstrates that identifying the dynamic processes and networks through which PPI can make a contribution to health-care improvement efforts within organisations like CLAHRCs is also crucial. Given that health research in the UK operates with a more explicit distinction of the roles of ‘patients’ and ‘professionals’ (in contrast to, for example, integrated knowledge translation or community-based participatory research efforts in North America ), it is vital to understand how collaborative organisations such as CLAHRCs can effectively extend this collaboration to encompass service users as well as service providers.
This review is not without limitations. First, we have deliberately focused on the emergence of one specific type of large-scale knowledge mobilisation initiative, and we are conscious that the findings of this review are to a certain degree shaped by the UK context. At the same time, our findings are likely to be applicable to a range of knowledge mobilisation partnerships (and their evaluations) internationally as the institutional pressures are similar across high-income countries [51, 58]. Second, due to our focus on empirical papers that report evaluation findings, we have excluded a number of conceptual papers that have been directly informed by their authors’ experience of designing and or working within CLAHRCs. Finally, we have decided against making formal judgements about the methodological rigour of individual evaluations as criteria for assessing research quality vary broadly depending on the epistemological position of the assessor;  instead, this review has adopted a pragmatic, pluralistic and epistemologically tolerant approach.
There is still much to learn about how the processes adopted and adapted by each CLAHRC actually deliver impact. CLAHRCs (and indeed other similar research and practice partnerships internationally) remain a rich and fertile research setting for those interested in the mechanisms, practices and consequences of knowledge mobilisation approaches and in the effects of models of research and practice partnership more generally. However, if future evaluations are to be more useful, then they need to heed the lessons of the past and deliver learning on mobilisation processes and impacts in a timely manner that can inform and influence the on-going development of such partnerships, thus bridging the gap between implementation science and the practice of implementation. We summarise our recommendations for further evaluation of research and practice partnerships as follows:
Emphasis should be placed on comparative evaluations that are embedded across research and practice partnerships (nationally or internationally), facilitating greater contextual understanding of what works, where, how and why.
Evaluations should explicitly capture, analyse and report knowledge mobilisation strategies employed and their impacts.
Given the complex multi-stakeholder context of research and practice partnerships, evaluations should aim to report perspectives on impact from different partners.
Reporting of unintended outcomes as well as contextual and/or political factors affecting mechanisms of impact should be encouraged.
Capacity building and PPI should be evaluated and reported taking into account the diversity of audiences and patient populations involved.
Evaluation outputs should themselves be accessible to non-academic audiences and generate actionable insights to surface pragmatic and experiential knowledge.
Much of the evaluative focus on CLAHRCs has been on how they have been organised and on the development of theory around emergent properties. Evidence is lacking, however, on the impact of CLAHRCs, particularly in relation to the knowledge mobilisation processes and practices adopted. Further evaluation focused on which knowledge mobilisation approaches work, where, how and why in research and practice partnerships is warranted.
Cooksey D. A review of UK health research funding. London: The Stationery Office; 2006.
Mitton C, Adair CE, McKenzie E, Patten SB, Waye Perry B. Knowledge transfer and exchange: review and synthesis of the literature. Milbank Q. 2007;85(4):729–68.
Gagliardi AR, Berta W, Kothari A, Boyko J, Urquhart R. Integrated knowledge translation (IKT) in health care: a scoping review. Implement Sci. 2016;11:38.
Lessard L, Michalowski W, Fung-Kee-Fung M, Jones L, Grudniewicz A. Architectural frameworks: defining the structures for implementing learning health systems. Implement Sci. 2017;12(1):78.
Stetler CB, Mittman BS, Francis J. Overview of the VA Quality Enhancement Research Initiative (QUERI) and QUERI theme articles: QUERI series. Implement Sci. 2008;3:8.
Kilbourne AM, Atkins D. Partner or perish: VA health services and the emerging bi-directional paradigm. J Gen Intern Med. 2014;29(Suppl 4):817–9.
Advanced Health Research and Translation Centres [https://www.nhmrc.gov.au/research/advanced-health-research-and-translation-centres]. Accessed 10 Aug 2018.
Tooke JC. Report of the high level group on clinical effectiveness. A report to sir Liam Donaldson chief medical officer. London: Department of Health; 2007.
National Institute for Health Research. Collaborations for Leadership in Applied Health Research and Care (CLAHRCs). London: National Institute for Health Research; 2016.
NIHR Service delivery and organisation Programme. Evaluating partnerships between universities and NHS organisations: learning from the collaborations for leadership in applied Health Research and care (CLAHRCs). Call for proposals. In: Research Brief (CLA258). London: NIHR Service Delivery and Organisation Programme; 2010.
Ouzzani M, Hammady H, Fedorowicz Z, Elmagarmid A. Rayyan-a web and mobile app for systematic reviews. Syst Rev. 2016;5(1):210.
Lockett A, El Enany N, Currie G, Oborn E, Barrett M, Racko G, Bishop S, Waring J. A formative evaluation of Collaboration for Leadership in Applied Health Research and Care (CLAHRC): institutional entrepreneurship for service innovation. Health Serv Deliv Res. 2014;2(31)
Rycroft-Malone J, Burton C, Wilkinson J, Harvey G, McCormack B, Baker R, Dopson S, Graham I, Staniszewska S, Thompson C, et al. Collective action for knowledge mobilisation: a realist evaluation of the Collaborations for Leadership in Applied Health Research and Care. Health Serv Deliv Res. 2015;3(44)
Scarbrough H, D'Andreta D, Evans S, Marabelli M, Newell S, Powell J, Swan J. Networked innovation in the health sector: comparative qualitative study of the role of Collaborations for Leadership in Applied Health Research and Care in translating research into practice. Health Serv Deliv Res. 2014;2(13)
Soper B, Hinrichs S, Drabble S, Yaqub O, Marjanovic S, Hanney S, Nolte E. Delivering the aims of the Collaborations for Leadership in Applied Health Research and Care: understanding their strategies and contributions. Health Serv Deliv Res. 2015;3(25)
Currie G, El Enany N, Lockett A. Intra-professional dynamics in translational health research: the perspective of social scientists. Soc Sci Med. 2014;114:81–8.
Currie G, Lockett A, El Enany N. From what we know to what we do: lessons learned from the translational CLAHRC initiative in England. J Health Serv Res Policy. 2013;18(3):27–39.
Oborn E, Barrett M, Prince K, Racko G. Balancing exploration and exploitation in transferring research into practice: a comparison of five knowledge translation entity archetypes. Implement Sci. 2013;8:104.
Racko G. Knowledge exchange in the UK CLAHRCs: the enabling role of academics and clinicians’ social position. J Health Organ Manage. 2018;32(2):246–62.
D’Andreta D, Marabelli M, Newell S, Scarbrough H, Swan J. Dominant cognitive frames and the innovative power of social networks. Organ Stud. 2016;37(3):293–321.
D'Andreta D, Scarbrough H, Evans S. The enactment of knowledge translation: a study of the Collaborations for Leadership in Applied Health Research and Care initiative within the English National Health Service. J Health Serv Res Policy. 2013;18(3):40–52.
Evans S, Scarbrough H. Supporting knowledge translation through collaborative translational research initiatives: 'bridging' versus 'blurring' boundary-spanning approaches in the UK CLAHRC initiative. Soc Sci Med. 2014;106:119–27.
Rycroft-Malone J, Burton CR, Wilkinson J, Harvey G, McCormack B, Baker R, Dopson S, Graham ID, Staniszewska S, Thompson C, et al. Collective action for implementation: a realist evaluation of organisational collaboration in healthcare. Implement Sci. 2016;11:17.
Rycroft-Malone J, Wilkinson J, Burton CR, Harvey G, McCormack B, Graham I, Staniszewska S. Collaborative action around implementation in Collaborations for Leadership in Applied Health Research and Care: towards a programme theory. J Health Serv Res Policy. 2013;18(3):13–26.
Ling T, Soper B, Marjanovic S, Celia C, Yaqub O, Reding A, Milewa T, Hanney S: Delivering the aims of the CLAHRCs: evaluating CLAHRCs' strategies and contributions. Interim report: Phase I. Cambridge: RAND Europe; 2011.
Soper B, Yaqub O, Hinrichs S, Marjanovich S, Drabble S, Hanney S, Nolte E. CLAHRCs in practice: combined knowledge transfer and exchange strategies, cultural change, and experimentation. J Health Serv Res Policy. 2013;18(3):53–64.
Ariss S, Cooke J, Smith C, Reed J, Nancarrow S. NIHR CLAHRC for South Yorkshire internal evaluation report November 2011: executive summary. Sheffield: NIHR CLAHRC for South Yorkshire; 2012.
Caldwell SE, Mays N. Studying policy implementation using a macro, meso and micro frame analysis: the case of the Collaboration for Leadership in Applied Health Research & Care (CLAHRC) programme nationally and in north West London. Health Res Policy Syst. 2012;10:32.
Chew S, Armstrong N, Martin G. Institutionalising knowledge brokering as a sustainable knowledge translation solution in healthcare: how can it work in practice? Evid Pol. 2013;9:335–51.
Cooke J, Ariss S, Smith C, Read J. On-going collaborative priority-setting for research activity: a method of capacity building to reduce the research-practice translational gap. Health Res Policy Syst. 2015;13:25.
Fitzgerald L, Harvey G. Translational networks in healthcare? Evidence on the design and initiation of organizational networks for knowledge mobilization. Soc Sci Med. 2015;138:192–200.
Gerrish K, Piercy H. Capacity development for knowledge translation: evaluation of an experiential approach through Secondment opportunities. Worldviews Evid-Based Nurs. 2014;11(3):209–16.
Heaton J, Day J, Britten N. Inside the "black box" of a knowledge translation program in applied Health Research. Qual Health Res. 2015;25(11):1477–91.
Heaton J, Day J, Britten N. Collaborative research and the co-production of knowledge for practice: an illustrative case study. Implement Sci. 2016;11:20.
Howe C, Randall K, Chalkley S, Bell D. Supporting improvement in a quality collaborative. Br J Healthc Manag. 2013;19:434–42.
Jordan M, Rowley E, Morriss R, Manning N. An analysis of the research team-service user relationship from the service user perspective: a consideration of 'The three Rs' (roles, relations, and responsibilities) for healthcare research organisations. Health Expect. 2015;18(6):2693–703.
Kislov R, Walshe K, Harvey G. Managing boundaries in primary care service improvement: a developmental approach to communities of practice. Implement Sci. 2012;7:97.
Kislov R. Boundary discontinuity in a constellation of interconnected practices. Public Adm. 2014;92(2):307–23.
Kislov R, Hodgson D, Boaden R. Professionals as knowledge brokers: the limits of authority in healthcare collaboration. Public Adm. 2016;94(2):472–89.
Kislov R, Hyde P, McDonald R. New game, old rules? Mechanisms and consequences of legitimation in boundary spanning activities. Organ Stud. 2017;38(10):1421–44.
Marston C, Renedo A. Understanding and measuring the effects of patient and public involvement: an ethnographic study. Lancet. 2013;382:S69.
Martin GP, McNicol S, Chew S. Towards a new paradigm in health research and practice? Collaborations for Leadership in Applied Health Research and Care. J Health Organ Manage. 2013;27(2):193–208.
Renedo A, Marston C, Spyridonidis D. Patient and public involvement in healthcare quality improvement: how organisations can help patients and professionals to collaborate. Public Manage Rev. 2015;17(1):17-34.
Reed JE, Howe C, Doyle C, Bell D. Simple rules for evidence translation in complex systems: a qualitative study. BMC Med. 2018;16(1):92.
Spyridonidis D, Hendy J, Barlow J. Understanding hybrid roles: the role of identity processes amongst physicians. Public Adm. 2015;93(2):395–411.
Spyridonidis D, Hendy J, Barlow J. Leadership for knowledge translation: the case of CLAHRCs. Qual Health Res. 2015;25(11):1492–505.
Waterman H, Boaden R, Burey L, Howells B, Harvey G, Humphreys J, Rothwell K, Spence M. Facilitating large-scale implementation of evidence based health care: insider accounts from a co-operative inquiry. BMC Health Serv Res. 2015;15:60.
Wright N. First-time knowledge brokers in health care: the experiences of nurses and allied health professionals of bridging the research-practice gap. Evid Policy. 2013;9(4):557–70.
Carlile PR. Transferring, translating, and transforming: an integrative framework for managing knowledge across boundaries. Organ Sci. 2004;15(5):555–68.
Kislov R, Harvey G, Walshe K. Collaborations for Leadership in Applied Health Research and Care: lessons from the theory of communities of practice. Implement Sci. 2011;6:64.
Kislov R, Waterman H, Harvey G, Boaden R. Rethinking capacity building for knowledge mobilisation: developing multilevel capabilities in healthcare organisations. Implement Sci. 2014;9:166.
Davies HTO, Powell AE, Nutley SM. Mobilising knowledge to improve UK health care: learning from other countries and other sectors - a multimethod mapping study. Health Serv Deliv Res. 2015;3(27)
Kislov R, Wilson P, Boaden R. The 'dark side' of knowledge brokering. J Health Serv Res Policy. 2017;22(2):107–12.
NHS Confederation. Integrating research into practice : the Collaborations for Leadership in Applied Health Research and Care (CLAHRC) experience. In: Health service research network briefing. vol. 245. London: NHS Confederation; 2012.
Mockford C, Staniszewska S, Griffiths F, Herron-Marx S. The impact of patient and public involvement on UK NHS health care: a systematic review. Int J Qual Health Care. 2012;24(1):28–38.
Brett J, Staniszewska S, Mockford C, Herron-Marx S, Hughes J, Tysall C, Suleman R. Mapping the impact of patient and public involvement on health and social care research: a systematic review. Health Expect. 2014;17(5):637–50.
Jull J, Giles A, Graham ID. Community-based participatory research and integrated knowledge translation: advancing the co-creation of knowledge. Implement Sci. 2017;12(1):150.
French CE, Ferlie E, Fulop NJ. The international spread of Academic Health Science Centres: a scoping review and the case of policy transfer to England. Health Policy. 2014;117(3):382–91.
Johnson P, Buehring A, Cassell C, Symon G. Evaluating qualitative management research: towards a contingent criteriology. Int J Manag Rev. 2006;8(3):131–56.
Rycroft-Malone J, Wilkinson JE, Burton CR, Andrews G, Ariss S, Baker R, Dopson S, Graham I, Harvey G, Martin G, et al. Implementing health research through academic and clinical partnerships: a realistic evaluation of the Collaborations for Leadership in Applied Health Research and Care (CLAHRC). Implement Sci. 2011;6:74.
Rowley E, Morriss R, Currie G, Schneider J. Research into practice: Collaboration for Leadership in Applied Health Research and Care (CLAHRC) for Nottinghamshire, Derbyshire, Lincolnshire (NDL). Implement Sci. 2012;7:40.
The authors thank information specialist Shannon Robalino for assistance with the development and administration of the database searches for this review.
All authors are in receipt of funding from the NIHR CLAHRC Greater Manchester. The views expressed in this article are those of the authors and not necessarily those of the NHS, NIHR or the Department of Health.
Availability of data and materials
All available data can be obtained from the corresponding author.
Ethics approval and consent to participate
Consent for publication
PMW is Deputy Editor in Chief of Implementation Science. All decisions relating to this manuscript were made by another senior editor. All authors are in receipt of funding from the NIHR CLAHRC Greater Manchester, one of 13 CLAHRCs funded in England.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Kislov, R., Wilson, P.M., Knowles, S. et al. Learning from the emergence of NIHR Collaborations for Leadership in Applied Health Research and Care (CLAHRCs): a systematic review of evaluations. Implementation Sci 13, 111 (2018) doi:10.1186/s13012-018-0805-y
- Learning health systems
- Knowledge mobilisation