- Study protocol
- Open Access
- Open Peer Review
A realist evaluation of the role of communities of practice in changing healthcare practice
Implementation Science volume 6, Article number: 49 (2011)
Healthcare organisations seeking to manage knowledge and improve organisational performance are increasingly investing in communities of practice (CoPs). Such investments are being made in the absence of empirical evidence demonstrating the impact of CoPs in improving the delivery of healthcare. A realist evaluation is proposed to address this knowledge gap. Underpinned by the principle that outcomes are determined by the context in which an intervention is implemented, a realist evaluation is well suited to understand the role of CoPs in improving healthcare practice. By applying a realist approach, this study will explore the following questions: What outcomes do CoPs achieve in healthcare? Do these outcomes translate into improved practice in healthcare? What are the contexts and mechanisms by which CoPs improve healthcare?
The realist evaluation will be conducted by developing, testing, and refining theories on how, why, and when CoPs improve healthcare practice. When collecting data, context will be defined as the setting in which the CoP operates; mechanisms will be the factors and resources that the community offers to influence a change in behaviour or action; and outcomes will be defined as a change in behaviour or work practice that occurs as a result of accessing resources provided by the CoP.
Realist evaluation is being used increasingly to study social interventions where context plays an important role in determining outcomes. This study further enhances the value of realist evaluations by incorporating a social network analysis component to quantify the structural context associated with CoPs. By identifying key mechanisms and contexts that optimise the effectiveness of CoPs, this study will contribute to creating a framework that will guide future establishment and evaluation of CoPs in healthcare.
With a focus on knowledge sharing and learning, communities of practice (CoPs) are being promoted in the healthcare sector as a means of improving practice and patient care . By definition, a CoP is a group of people 'who share a concern, a set of problems, or a passion about a topic, and who deepen their knowledge and expertise on this area by interacting on an ongoing basis' . It is argued that CoPs nurture and harness knowledge, particularly in terms of promoting the exchange of tacit knowledge, and drive innovation to help individuals and organisations improve practice and performance [3–5]. Such claims have contributed to the widespread adoption of CoPs in healthcare and other sectors seeking to effectively manage knowledge in order to improve organisational performance.
Reflecting the increased uptake of CoPs in the healthcare sector, the number of peer-reviewed papers reporting on CoPs is steadily increasing , and includes the publication of two systematic reviews in 2009 [7, 8]. Despite this increase, there is a lack of empirical evidence demonstrating the impact of CoPs in improving healthcare practice. Much of the published literature is limited to describing the establishment or activities of CoPs [6, 9]. If organisations and sponsors are to foster CoPs for their value in knowledge management and for improving organisational performance, there is a need to understand better the role of CoPs in improving healthcare practice. To this end, a realist evaluation of CoPs is proposed.
In contrast to traditional evaluation methods that examine the success of an intervention based on whether or not a predefined outcome has been achieved, the realist approach seeks to answer the questions -- how, why, and when does the intervention work ? A realist evaluation is a theory-driven approach to understanding what it is about a program that achieves a particular outcome in one setting and a different outcome in another. It is well suited for social interventions where outcomes are determined through stakeholder action and interaction, which in turn is likely to be influenced by social and cultural norms [10, 11]. Underpinned by the principle that context (C) will trigger mechanisms (M) to yield outcomes (O), a realist evaluation goes beyond focussing purely on inputs and outputs; it involves exploring and identifying the mechanisms by which the inputs are converted inside a 'black box' into outputs, and recognises the need for particular conditions (or contexts) to be present inside the black box for the causal mechanisms to be triggered and yield a particular outcome. The relationship between context, mechanism, and outcome is presented as a 'CMO configuration' . Based on these principles, the objective of this study is to identify CMO configurations that will explain the role of CoPs in improving healthcare practice. This objective will be achieved by seeking to answer the following questions: What outcomes do CoPs achieve in the healthcare sector? Do these outcomes translate into improved practice in healthcare? What are the contexts and mechanisms by which CoPs impact on improved practice in healthcare?
This realist evaluation will be conducted in four stages as shown in Table 1 corresponding to the four components of the realist evaluation cycle (theory, hypotheses generation, observations, and program specifications) as described by Pawson and Tilley . The stages will be undertaken sequentially such that the findings from each stage will inform the next stage, and the final stage will involve reviewing the findings from stage three to confirm, modify, or reject the theory-based hypotheses generated in stage two .
An opportunistic sample of four CoPs will be identified, and sponsors, facilitators, and members of each CoP will be interviewed and surveyed using semi-structured interviews and online surveys. These four CoPs will act as case studies to enable the in-depth exploration required to understand the happenings within the 'black box' linking CoPs to improved practice.
Stage one: theory
The first stage of the realist evaluation involves developing candidate theories on the role of CoPs in improving healthcare practice and potential CMO configurations. This begins with a systematic search and review of the healthcare literature identifying characteristics of CoPs and outcomes achieved. By focusing the review specifically on the healthcare sector, the information collected will be context specific. Our systematic review  identified particular features by which CoPs in the healthcare sector differ (Table 2), providing us with a starting point for formulating contexts and potential mechanisms by which CoPs in the healthcare sector influence change in practice.
The next activity in stage one is to interview sponsors and facilitators with the objective of identifying CMOs that will, in turn, be used to generate candidate theories on CMO configurations explaining the role of CoPs in improving healthcare practice. Additional data will be collected on resources offered to members and the means by which impact is assessed. The questions used to guide these interviews are in Additional File 1.
When collecting data from published literature and from interviews, contexts will be defined as the settings in which the CoP operates. To a large extent, these will be the characteristics of the CoP in terms of constitution of membership, level of maturity of the CoP, and activities organised by the CoP. Contexts will be also be determined by examining the connections, interactions, and knowledge flow that occur within each CoP. Mechanisms will be defined as the factors and resources that the CoP offers its members to influence a change in behaviour or action [12, 13]. A mechanism may be an enabler or a disabler depending on the context.
To date, there is no consistency in the way in which outcomes of CoPs are defined or measured . For the purpose of this study, an outcome will be defined as a change in behaviour or work practice that occurred, influenced by participating in a CoP activity or through accessing resources provided by the CoP. The change may be to a process (such as adoption of a new system or process, or reduced time to achieve a goal that is related to improved care); an innovation (such as development of a new product or technology that will improve the delivery of healthcare); or change in level of customer (patient) satisfaction . Financial outcomes will not be considered due to the focus of this study on clinical practice. Individual as well as organisational level outcomes will be sought, recognising that improved organisational performance is achieved through changing the work practice of individuals who contribute to the organisation.
The end product of stage one will be a list of CMOs and possible CMO configurations that explain the role of CoPs in improving healthcare. Figure 1 presents a preliminary list based on background research.
Stage two: Hypotheses generation
The second (hypotheses generating) stage involves rephrasing the CMO configurations theorised in stage one as hypotheses for testing in stage three. These hypotheses will be framed around the theoretical relationships between specific CMOs that could explain the different outcomes of CoPs, depending on the context in which the CoP operates.
Stage three: Observation
The hypotheses developed in stage two will be tested during stage three. All members of the participating CoPs will be asked to respond to a survey that will seek their level of agreement (using a five-point Likert scale) with hypotheses (see Additional File 2). Testing these hypotheses will help identify CMO configurations that occur with regularity, and provide possible explanations for the role of CoPs in improving healthcare practice.
The second part of the third stage will involve obtaining contextual information on the connections, relationships, and knowledge exchange that occur within a CoP. These are, by definition, essential elements of a CoP and form the context in which the hypotheses are tested. As identified in the literature and presented in Table 2 regular communication, interaction, and knowledge exchange between members are characteristics associated with CoPs. These elements have been linked to improved organisational performance through the concept of social capital as follows: Social capital is created by developing connections among practitioners that foster 'relationships that build a sense of trust and mutual obligation, and (by) creating a common language and contexts that can be shared by community members' . The connections, relationships, and common contexts that generate social capital, in turn, positively impact on organisational performance . The study will utilise social network analysis (SNA) methods to examine the professional connections and relationships within the CoP, represented by the strength of ties, so as to understand how the CoP network features may relate to improved healthcare practice. The social interactions that occur within the CoP will also be examined, recognising their role in the exchange of tacit knowledge .
Questions used to collect data on professional and social connections and knowledge exchange will be based on social network questions used by other researchers [16–18], modified where necessary to achieve the objectives of this study (See Additional File 3). The questionnaire will be validated by testing on a convenience sample of ten people with clinical practice and health management roles similar to those of members of the CoPs. Reliability will be tested using Kappa coefficient methods on test and retest of the instrument five days apart.
The network data collected in the study will be analysed using UCInet . The NetDraw feature of this software allows visual examination of each of the relationships (i.e., professional connections, social interactions, and information and knowledge flow) for strength of connectedness based on the frequency of contact. It also aids the identification of cliques (or subgroups), cut-points (referring to a person whose departure will result in a break in flow of information/knowledge), and isolated individuals .
When surveying CoP members, knowledge will be defined as 'internali(s)ed or understood information that can be used to make decisions' . Knowledge will be differentiated from information by the fact that '(k)nowledge is information possessed in the mind of individuals: it is personali(s)ed information (which may or may not be new, unique, useful, or accurate) related to facts, procedures, concepts, interpretations, ideas, observations, and judgements' . The reason for differentiating knowledge from information is to examine how much of the information that CoPs provide their members is translated into knowledge that influences change in their own work and practice.
The analysis functions in UCInet will be used to quantify the connectivity and stability of the community by measuring degree, closeness and betweenness centrality, reciprocity of relationships, and multiplexity . Degree centrality is the number of persons (or nodes) to which a particular person is directly linked; a higher score indicates a well connected person . This measure will help identify key persons in the community, with the facilitator likely to score highly. A high average density score at the CoP level indicates a high level of direct links or interactions between members of the CoP. Closeness centrality recognises the importance of indirect connections for exchange of resources (such as knowledge) and measures the shortest path connecting a key node (CoP member, in this case) to any other node . Betweenness centrality also takes into account the importance of indirect links in maintaining links between nodes not otherwise connected . This, too, is relevant in terms of examining the flow of resources (such as information or knowledge) . A CoP scoring highly in the knowledge exchange relationship would indicate high connectivity with little threat to knowledge exchange due to lost links. Reciprocity of each relationship will also be examined to identify bidirectional links, with suggestions that high level of reciprocity is characteristic of a more stable network . The knowledge relationship will be examined further for path length, to assess the efficiency of information and knowledge flow and exchange within the CoP and will help identify how best to optimise this process .
As this study examines multiple relationships (that is, professional connections, social interactions, and information and knowledge flow), multiplexity will be examined as an indication of the strength of the link between members; with members linked by more than one relationship said to have stronger ties than those linked by one relationship [24, 28, 29].
Stage four: Program specification
The fourth and final stage is program specification, during which the theorised role of CoPs in improving healthcare and potential CMO configurations from stage one will be reviewed in light of the findings in stage three. The CMO configurations that were supported with regularity will form the basis for specifying possible explanations for the role of CoPs in improving healthcare practice.
This paper describes a protocol that uses mixed methods to examine systematically and understand how, why, and when CoPs improve healthcare practice. Realist evaluation is being used increasingly in the healthcare sector, recognising the fact that programs and interventions requiring behavioural change operate within a complex social and cultural context, and that the operating context plays an important role in determining impact. In such circumstances, the traditional approach of evaluating success based on whether or not a pre-defined outcome is achieved does not provide decision makers with sufficient information to assess the value of the program outside the context in which it was tested. There is a need for methods that are able to tease out the mechanisms by which a program results in change, and study the interactions between these causal mechanisms and context [10, 30].
Following the application of a realist approach to evaluate a modernisation initiative in the UK, Greenhalgh et al. discussed the difficulties in identifying the mechanisms of change and drawing realist conclusions around CMO configurations. They refer to this process as typically requiring 'a three-hour face-to-face meeting as well as lengthy email exchanges and numerous iterations and counteriterations' . Work undertaken to date on this project affirms the difficulty of identifying mechanisms and outcomes, and generating the list presented in this paper has required lengthy discussions and iterations. Since our proposal also includes using SNA methods to examine the connections and knowledge exchanges within the CoP as a means of providing contextual information, we need to strike a fine balance between making significant demands on participants' time and securing the high response rates required for SNA. As a means of achieving this balance, we have chosen to limit the in-depth interviews and discussions to facilitators and sponsors of CoPs. Members will participate in testing the hypotheses generated by the discussions and will respond to the SNA questions. To help this process further and taking into consideration the length of the hypotheses testing survey, the SNA survey will be administered at a later date.
A challenging aspect of developing this study protocol was identifying and defining an outcome that would demonstrate the impact of CoPs in improving work practice. A finding from our systematic review  was that the vast majority of existing research had assessed impact though self-reported perceived benefits, with very limited effort to substantiate these claims through triangulation. This study will attempt to overcome this limitation by defining an outcome as a demonstrated change in work practice at the individual member level as well as the organisation level. The difficulty in drawing conclusions around CMO configurations will be addressed to some extent by looking for patterns that occur with regularity supporting the occurrence of such causal interactions.
The realist evaluation method, by seeking to understand how, why, and when a program works, is well suited for separating out and examining the multiple components in a program individually and in the context of the program. This feature is particularly useful given the difficulty experienced in directly attributing outcome to a CoP in studies that have measured and reported outcomes from multi-faceted interventions . CoPs offer more than one resource to their members with the intention of facilitating knowledge creation and sharing. Knowing the role that each of these components play in influencing change in healthcare practice will help maximise value and return on investment in CoPs.
This study further enhances the value of realist evaluations by incorporating a SNA component to quantify the structural context associated with CoPs. To our knowledge, these two methods have not been previously combined. By examining the connections and relationships that occur within the community or network, SNA methods quantify the structural component of the context within which CoPs operate.
Overall, this paper proposes a research study to understand the complexity of CoPs, taking into consideration its multi-component nature and the influence of context in determining impact. The systematic approach proposed will help identify key mechanisms that operate within particular contexts, which in turn will help optimise the establishment and effectiveness of CoPs. The study will contribute to creating a framework that will guide the future development and evaluation of CoPs in the healthcare sector .
le May A: Introducing communities of practice. Communities of practice in health and social care. Edited by: le May A. 2009, Wiley-Blackwell, 3-16.
Wenger E, McDermott R, Snyder WM: Cultivating Communities of Practice. 2002, Boston, Massachusetts: Harvard Business School Press
American Productivity & Quality Centre: Building and sustaining communities of practice: Continuing success in knowledge management. 2001, Texas: American Productivity and Quality Centre (APQC)
Lesser EL, Storck J: Communities of practice and organizational performance. IBM Syst J. 2001, 40: 831-841.
Schenkel A, Teigland R: Improved organizational performance through communities of practice. J knowledge management. 2008, 12: 106-118. 10.1108/13673270810852421.
Ranmuthugala G, Plumb J, Cunningham F, Georgiou A, Westbrook J, Braithwaite J: Communities of practice in the health sector: A systematic review of the peer-reviewed literature. 2010, Sydney: University of New South Wales, Australian Institute of Health Innovation
Li LC, Grimshaw JM, Nielsen C, Judd M, Coyte PC, Graham ID: Use of communities of practice in business and health care sectors: A systematic review. Implement Sci. 2009, 4:
Fung-Kee-Fung M, Watters J, Crossley C, Goubanova E, Abdulla A, Stern H, Oliver TK: Regional collaborations as a tool for quality improvements in surgery: a systematic review of the literature. Ann Surg. 2009, 249: 565-572. 10.1097/SLA.0b013e31819ec608.
Braithwaite J, Westbrook JI, Ranmuthugala G, Cunningham F, Plumb J, Wiley J, Ball D, Huckson S, Hughes C, Johnston B: The development, design, testing, refinement, simulation and application of an evaluation framework for communities of practice and social-professional networks. BMC Health Serv Res. 2009, 9: 162-10.1186/1472-6963-9-162.
Pawson R, Tilley N: Realistic evaluation. 1997, London: Sage
Hunt KS, Sridharan S: A Realist evaluation approach to unpacking the impacts of the sentencing guidelines. Am J Eval. 2010, 31: 463-485. 10.1177/1098214010371817.
Thistleton L: A realistic evaluation of the work of a speech and language therapy service in primary schools (the first schools project) using the perceptions of some of the important stakeholders (teachers, SLTs, and parents). 2008, PhD thesis. The University of Birmingham, School of Education
Kazi M: A realist evaluation of practice: the NSPCC's Shield project and intensive analysis. Realist evaluation in practice: Health and social work. Edited by: Kazi M. 2003, London, California, New Delhi: Sage Publications Ltd, 95-109.
Rhodes J, Lok P: An integrative model of organizational learning and social capital on effective knowledge transfer and perceived organizational performance. J Workplace Learning. 2008, 20: 245-258. 10.1108/13665620810871105.
Helms R, Bosua R, Ignatio R: Impact assessment of knowledge sharing bottlenecks:The Knowledge Sharing Environment Model (KSEM). 2009, Presented at the 20th Australasian Conference on Information Systems, Melbourne, Australia
Cross R, Parker A: The hidden power of social networks. 2004, Boston, Massachusetts, USA: Harvard Business School Publishing Corporation
Sales AE, Estabrooks CA, Valente TW: The impact of social networks on knowledge transfer in long-term care facilities: Protocol for a study. Implement Sci. 2010, 5: 49-10.1186/1748-5908-5-49.
Ferlie E, Fitzgerald L, McGivern G, Dopson S, Exworthy M: Networks in health care: A comparative study of their management, impact and performance. 2010, London: King's College London, Accessed 13 March 2011, [http://www.sdo.nihr.ac.uk/files/project/102-final-report.pdf]
Borgatti S, Everett M, Freeman L: UCInet 6.3 For Windows: Software for social network analysis. 2010, Lexington, KY
Helms R: Redesigning communities of practice using knowledge network analysis. Hands-On Knowledge Co-Creation and Sharing: Practical Methods and Techniques. Edited by: Kazi A, Wohlfart L, Wolf P. Stuttgart. 2007, Germany: KnowledgeBoard, 251-274.
Zins C: Conceptual approaches for defining data, information, and knowledge. J Am Soc Inf Sci Technol. 2007, 58: 479-493. 10.1002/asi.20508.
Alavi M, Leidner DE: Review: Knowledge management and knowledge management systems: Conceptual foundations and research issues. MIS Quarterly. 2001, 25: 107-136. 10.2307/3250961.
Creswick N, Westbrook JI: Social network analysis of medication advice-seeking interactions among staff in an Australian hospital. Int J Med Inform. 2010, 79: e116-125. 10.1016/j.ijmedinf.2008.08.005. Epub 2008 Nov 2012
Provan KG, Nakama L, Veazie MA, Teufel-Shone NI, Huddleston C: Building community capacity around chronic disease services through a collaborative interorganizational network. Health Educ Behav. 2003, 30: 646-662. 10.1177/1090198103255366.
Provan KG, Fish A, Sydow J: Interorganizational networks at the network level: A review of the empirical literature on whole networks. J Management. 2007, 33: 479-516.
Hanneman R, Riddle M: Chapter 8: Embedding. Introduction to social network methods. 2005, Riverside, CA: University of California
Grippa A: A social network scorecard to monitor knowledge flow across communication media. Knowledge management and practice. 2009, 7: 317-328. 10.1057/kmrp.2009.24.
Holschuh J, Segal S: Factors related to multiplexity in support networks of persons with severe mental illness. Social Networks and Health. 2002, 8: 293-321.
Provan KG, Milward HB: Do networks really work? A framework for evaluating public-sector organizational networks. Public Admin Review. 2001, 61: 414-423. 10.1111/0033-3352.00045.
Berwick DM: The science of improvement. J Amer Med Assoc. 2008, 299: 1182-1184. 10.1001/jama.299.10.1182.
Greenhalgh T, Humphrey C, Hughes J, Macfarlane F, Butler C, Pawson R: How do you modernize a health service? A realist evaluation of whole-scale transformation in London. Milbank Q. 2009, 87: 391-416. 10.1111/j.1468-0009.2009.00562.x.
Hildreth PM, Kimble C: The duality of knowledge. Information Research. 2002, 8:
Lathlean J, le May A: Communities of practice: an opportunity for interagency working. J Clin Nurs. 2002, 11: 394-398. 10.1046/j.1365-2702.2002.00630.x.
Kimball L, Ladd A: Facilitator toolkit for building and sustaining virtual communities of practice. Idea Group Publishing. 2004
Render ML, Brungs S, Kotagal U, Nicholson M, Burns P, Ellis D, Clifton M, Fardo R, Scott M, Hirschhorn L: Evidence-based practice to reduce central line infections. Joint Comm J Qual Patient Saf. 2006, 32: 253-260.
Chandler L, Fry A: Can communities of practice make a meaningful contribution to sustainable service improvement in health and social care?. J Integr Care. 2009, 17: 41-48.
Jiwa M, Deas K, Ross J, Shaw T, Wilcox H, Spilsbury K: An inclusive approach to raising standards in general practice: working with a 'community of practice' in Western Australia. BMC Med Res Methodol. 2009, 9: 13-10.1186/1471-2288-9-13.
Acknowledgements and funding
This research is supported by the Australian Research Council's Discovery Project funding scheme (DP 0986493), and has been approved by the Human Research Ethics Committee, University of New South Wales (HREC 09085). The authors would like to acknowledge and thank Sue Huckson and Scott Bennett for their input into identifying CMOs as they relate to CoPs; and David Greenfield and Nerida Creswick for their valuable input at various stages of this study.
The authors declare that they have no competing interests.
JB and JIW conceptualised the overarching research project and are the chief investigators of the research grant funding this research activity. GR developed the study protocol presented in this paper in consultation will all other co-authors and wrote the first draft. All authors provided input into various aspects of the study, provided ongoing critique and approved the final version of the manuscript.