A realist evaluation of the role of communities of practice in changing healthcare practice
© Ranmuthugala et al; licensee BioMed Central Ltd. 2011
Received: 22 March 2011
Accepted: 23 May 2011
Published: 23 May 2011
Healthcare organisations seeking to manage knowledge and improve organisational performance are increasingly investing in communities of practice (CoPs). Such investments are being made in the absence of empirical evidence demonstrating the impact of CoPs in improving the delivery of healthcare. A realist evaluation is proposed to address this knowledge gap. Underpinned by the principle that outcomes are determined by the context in which an intervention is implemented, a realist evaluation is well suited to understand the role of CoPs in improving healthcare practice. By applying a realist approach, this study will explore the following questions: What outcomes do CoPs achieve in healthcare? Do these outcomes translate into improved practice in healthcare? What are the contexts and mechanisms by which CoPs improve healthcare?
The realist evaluation will be conducted by developing, testing, and refining theories on how, why, and when CoPs improve healthcare practice. When collecting data, context will be defined as the setting in which the CoP operates; mechanisms will be the factors and resources that the community offers to influence a change in behaviour or action; and outcomes will be defined as a change in behaviour or work practice that occurs as a result of accessing resources provided by the CoP.
Realist evaluation is being used increasingly to study social interventions where context plays an important role in determining outcomes. This study further enhances the value of realist evaluations by incorporating a social network analysis component to quantify the structural context associated with CoPs. By identifying key mechanisms and contexts that optimise the effectiveness of CoPs, this study will contribute to creating a framework that will guide future establishment and evaluation of CoPs in healthcare.
With a focus on knowledge sharing and learning, communities of practice (CoPs) are being promoted in the healthcare sector as a means of improving practice and patient care . By definition, a CoP is a group of people 'who share a concern, a set of problems, or a passion about a topic, and who deepen their knowledge and expertise on this area by interacting on an ongoing basis' . It is argued that CoPs nurture and harness knowledge, particularly in terms of promoting the exchange of tacit knowledge, and drive innovation to help individuals and organisations improve practice and performance [3–5]. Such claims have contributed to the widespread adoption of CoPs in healthcare and other sectors seeking to effectively manage knowledge in order to improve organisational performance.
Reflecting the increased uptake of CoPs in the healthcare sector, the number of peer-reviewed papers reporting on CoPs is steadily increasing , and includes the publication of two systematic reviews in 2009 [7, 8]. Despite this increase, there is a lack of empirical evidence demonstrating the impact of CoPs in improving healthcare practice. Much of the published literature is limited to describing the establishment or activities of CoPs [6, 9]. If organisations and sponsors are to foster CoPs for their value in knowledge management and for improving organisational performance, there is a need to understand better the role of CoPs in improving healthcare practice. To this end, a realist evaluation of CoPs is proposed.
In contrast to traditional evaluation methods that examine the success of an intervention based on whether or not a predefined outcome has been achieved, the realist approach seeks to answer the questions -- how, why, and when does the intervention work ? A realist evaluation is a theory-driven approach to understanding what it is about a program that achieves a particular outcome in one setting and a different outcome in another. It is well suited for social interventions where outcomes are determined through stakeholder action and interaction, which in turn is likely to be influenced by social and cultural norms [10, 11]. Underpinned by the principle that context (C) will trigger mechanisms (M) to yield outcomes (O), a realist evaluation goes beyond focussing purely on inputs and outputs; it involves exploring and identifying the mechanisms by which the inputs are converted inside a 'black box' into outputs, and recognises the need for particular conditions (or contexts) to be present inside the black box for the causal mechanisms to be triggered and yield a particular outcome. The relationship between context, mechanism, and outcome is presented as a 'CMO configuration' . Based on these principles, the objective of this study is to identify CMO configurations that will explain the role of CoPs in improving healthcare practice. This objective will be achieved by seeking to answer the following questions: What outcomes do CoPs achieve in the healthcare sector? Do these outcomes translate into improved practice in healthcare? What are the contexts and mechanisms by which CoPs impact on improved practice in healthcare?
Four-stage approach to the realist evaluation of communities of practice
• Systematic review of the literature
• Semi-structured interviews with sponsors and facilitators of CoPs
• Formulate CMO configurations
• Qualitative -- Identify themes and categorise as outcomes, mechanisms, and contextual factors
• Formulate potential CMO configurations
Provide the theoretical basis for the realist evaluation
2. Hypotheses generation
• Generate hypotheses based on CMO configurations
Rephrase CMO configurations into hypotheses
Formulate hypotheses to be tested during stage three
• Online survey of CoP members
• Identify CMO configurations that occur with regularity
• Social Network Analysis
Test and accept, reject, or modify hypotheses
Examine structure of professional and social relationships and flow of information and knowledge within the CoP
4. Program specification
• Review analysis from stage three
Refine theorised CMO configurations based on testing of hypotheses
Specify CMO configurations that explain how, when and why CoPs improve healthcare practice
An opportunistic sample of four CoPs will be identified, and sponsors, facilitators, and members of each CoP will be interviewed and surveyed using semi-structured interviews and online surveys. These four CoPs will act as case studies to enable the in-depth exploration required to understand the happenings within the 'black box' linking CoPs to improved practice.
Stage one: theory
Characteristics of communities of practice identified from the literature 
Findings from the literature review
Membership and practice
• One becomes a member through shared practice 
• CoPs help establish professional identity 
• Members have a common goal or purpose 
• Membership often crossed geographical, professional, and/or organisational boundaries 
• The focus of the group may vary over time 
Activities and communication methods
• Members exchange knowledge through formal and informal processes. Formal methods of interaction include face-to-face meetings within or external to usual workplace and/or virtual methods that include communication via email and/or blogs 
• Social interaction, in person or through the use of communication technology, is an important feature of a CoP identity 
• Spontaneous origin or established as a management initiative 
• CoPs have five stages of development: potential, coalescing, maturing, stewardship, and transformation 
Determinants of success
• A committed facilitator 
• Shared purpose 
• Commitment and enthusiasm from the members 
• A CoP with self-selected membership may be more successful than a CoP with externally appointed members 
• Regular communication with, and interaction between members 
• Developing relationships through face-to-face interactions, even to start with, is important 
• Infrastructure to support the work of the CoP in terms of ease of access to knowledge or evidence 
The next activity in stage one is to interview sponsors and facilitators with the objective of identifying CMOs that will, in turn, be used to generate candidate theories on CMO configurations explaining the role of CoPs in improving healthcare practice. Additional data will be collected on resources offered to members and the means by which impact is assessed. The questions used to guide these interviews are in Additional File 1.
When collecting data from published literature and from interviews, contexts will be defined as the settings in which the CoP operates. To a large extent, these will be the characteristics of the CoP in terms of constitution of membership, level of maturity of the CoP, and activities organised by the CoP. Contexts will be also be determined by examining the connections, interactions, and knowledge flow that occur within each CoP. Mechanisms will be defined as the factors and resources that the CoP offers its members to influence a change in behaviour or action [12, 13]. A mechanism may be an enabler or a disabler depending on the context.
To date, there is no consistency in the way in which outcomes of CoPs are defined or measured . For the purpose of this study, an outcome will be defined as a change in behaviour or work practice that occurred, influenced by participating in a CoP activity or through accessing resources provided by the CoP. The change may be to a process (such as adoption of a new system or process, or reduced time to achieve a goal that is related to improved care); an innovation (such as development of a new product or technology that will improve the delivery of healthcare); or change in level of customer (patient) satisfaction . Financial outcomes will not be considered due to the focus of this study on clinical practice. Individual as well as organisational level outcomes will be sought, recognising that improved organisational performance is achieved through changing the work practice of individuals who contribute to the organisation.
Stage two: Hypotheses generation
The second (hypotheses generating) stage involves rephrasing the CMO configurations theorised in stage one as hypotheses for testing in stage three. These hypotheses will be framed around the theoretical relationships between specific CMOs that could explain the different outcomes of CoPs, depending on the context in which the CoP operates.
Stage three: Observation
The hypotheses developed in stage two will be tested during stage three. All members of the participating CoPs will be asked to respond to a survey that will seek their level of agreement (using a five-point Likert scale) with hypotheses (see Additional File 2). Testing these hypotheses will help identify CMO configurations that occur with regularity, and provide possible explanations for the role of CoPs in improving healthcare practice.
The second part of the third stage will involve obtaining contextual information on the connections, relationships, and knowledge exchange that occur within a CoP. These are, by definition, essential elements of a CoP and form the context in which the hypotheses are tested. As identified in the literature and presented in Table 2 regular communication, interaction, and knowledge exchange between members are characteristics associated with CoPs. These elements have been linked to improved organisational performance through the concept of social capital as follows: Social capital is created by developing connections among practitioners that foster 'relationships that build a sense of trust and mutual obligation, and (by) creating a common language and contexts that can be shared by community members' . The connections, relationships, and common contexts that generate social capital, in turn, positively impact on organisational performance . The study will utilise social network analysis (SNA) methods to examine the professional connections and relationships within the CoP, represented by the strength of ties, so as to understand how the CoP network features may relate to improved healthcare practice. The social interactions that occur within the CoP will also be examined, recognising their role in the exchange of tacit knowledge .
Questions used to collect data on professional and social connections and knowledge exchange will be based on social network questions used by other researchers [16–18], modified where necessary to achieve the objectives of this study (See Additional File 3). The questionnaire will be validated by testing on a convenience sample of ten people with clinical practice and health management roles similar to those of members of the CoPs. Reliability will be tested using Kappa coefficient methods on test and retest of the instrument five days apart.
The network data collected in the study will be analysed using UCInet . The NetDraw feature of this software allows visual examination of each of the relationships (i.e., professional connections, social interactions, and information and knowledge flow) for strength of connectedness based on the frequency of contact. It also aids the identification of cliques (or subgroups), cut-points (referring to a person whose departure will result in a break in flow of information/knowledge), and isolated individuals .
When surveying CoP members, knowledge will be defined as 'internali(s)ed or understood information that can be used to make decisions' . Knowledge will be differentiated from information by the fact that '(k)nowledge is information possessed in the mind of individuals: it is personali(s)ed information (which may or may not be new, unique, useful, or accurate) related to facts, procedures, concepts, interpretations, ideas, observations, and judgements' . The reason for differentiating knowledge from information is to examine how much of the information that CoPs provide their members is translated into knowledge that influences change in their own work and practice.
The analysis functions in UCInet will be used to quantify the connectivity and stability of the community by measuring degree, closeness and betweenness centrality, reciprocity of relationships, and multiplexity . Degree centrality is the number of persons (or nodes) to which a particular person is directly linked; a higher score indicates a well connected person . This measure will help identify key persons in the community, with the facilitator likely to score highly. A high average density score at the CoP level indicates a high level of direct links or interactions between members of the CoP. Closeness centrality recognises the importance of indirect connections for exchange of resources (such as knowledge) and measures the shortest path connecting a key node (CoP member, in this case) to any other node . Betweenness centrality also takes into account the importance of indirect links in maintaining links between nodes not otherwise connected . This, too, is relevant in terms of examining the flow of resources (such as information or knowledge) . A CoP scoring highly in the knowledge exchange relationship would indicate high connectivity with little threat to knowledge exchange due to lost links. Reciprocity of each relationship will also be examined to identify bidirectional links, with suggestions that high level of reciprocity is characteristic of a more stable network . The knowledge relationship will be examined further for path length, to assess the efficiency of information and knowledge flow and exchange within the CoP and will help identify how best to optimise this process .
As this study examines multiple relationships (that is, professional connections, social interactions, and information and knowledge flow), multiplexity will be examined as an indication of the strength of the link between members; with members linked by more than one relationship said to have stronger ties than those linked by one relationship [24, 28, 29].
Stage four: Program specification
The fourth and final stage is program specification, during which the theorised role of CoPs in improving healthcare and potential CMO configurations from stage one will be reviewed in light of the findings in stage three. The CMO configurations that were supported with regularity will form the basis for specifying possible explanations for the role of CoPs in improving healthcare practice.
This paper describes a protocol that uses mixed methods to examine systematically and understand how, why, and when CoPs improve healthcare practice. Realist evaluation is being used increasingly in the healthcare sector, recognising the fact that programs and interventions requiring behavioural change operate within a complex social and cultural context, and that the operating context plays an important role in determining impact. In such circumstances, the traditional approach of evaluating success based on whether or not a pre-defined outcome is achieved does not provide decision makers with sufficient information to assess the value of the program outside the context in which it was tested. There is a need for methods that are able to tease out the mechanisms by which a program results in change, and study the interactions between these causal mechanisms and context [10, 30].
Following the application of a realist approach to evaluate a modernisation initiative in the UK, Greenhalgh et al. discussed the difficulties in identifying the mechanisms of change and drawing realist conclusions around CMO configurations. They refer to this process as typically requiring 'a three-hour face-to-face meeting as well as lengthy email exchanges and numerous iterations and counteriterations' . Work undertaken to date on this project affirms the difficulty of identifying mechanisms and outcomes, and generating the list presented in this paper has required lengthy discussions and iterations. Since our proposal also includes using SNA methods to examine the connections and knowledge exchanges within the CoP as a means of providing contextual information, we need to strike a fine balance between making significant demands on participants' time and securing the high response rates required for SNA. As a means of achieving this balance, we have chosen to limit the in-depth interviews and discussions to facilitators and sponsors of CoPs. Members will participate in testing the hypotheses generated by the discussions and will respond to the SNA questions. To help this process further and taking into consideration the length of the hypotheses testing survey, the SNA survey will be administered at a later date.
A challenging aspect of developing this study protocol was identifying and defining an outcome that would demonstrate the impact of CoPs in improving work practice. A finding from our systematic review  was that the vast majority of existing research had assessed impact though self-reported perceived benefits, with very limited effort to substantiate these claims through triangulation. This study will attempt to overcome this limitation by defining an outcome as a demonstrated change in work practice at the individual member level as well as the organisation level. The difficulty in drawing conclusions around CMO configurations will be addressed to some extent by looking for patterns that occur with regularity supporting the occurrence of such causal interactions.
The realist evaluation method, by seeking to understand how, why, and when a program works, is well suited for separating out and examining the multiple components in a program individually and in the context of the program. This feature is particularly useful given the difficulty experienced in directly attributing outcome to a CoP in studies that have measured and reported outcomes from multi-faceted interventions . CoPs offer more than one resource to their members with the intention of facilitating knowledge creation and sharing. Knowing the role that each of these components play in influencing change in healthcare practice will help maximise value and return on investment in CoPs.
This study further enhances the value of realist evaluations by incorporating a SNA component to quantify the structural context associated with CoPs. To our knowledge, these two methods have not been previously combined. By examining the connections and relationships that occur within the community or network, SNA methods quantify the structural component of the context within which CoPs operate.
Overall, this paper proposes a research study to understand the complexity of CoPs, taking into consideration its multi-component nature and the influence of context in determining impact. The systematic approach proposed will help identify key mechanisms that operate within particular contexts, which in turn will help optimise the establishment and effectiveness of CoPs. The study will contribute to creating a framework that will guide the future development and evaluation of CoPs in the healthcare sector .
Acknowledgements and funding
This research is supported by the Australian Research Council's Discovery Project funding scheme (DP 0986493), and has been approved by the Human Research Ethics Committee, University of New South Wales (HREC 09085). The authors would like to acknowledge and thank Sue Huckson and Scott Bennett for their input into identifying CMOs as they relate to CoPs; and David Greenfield and Nerida Creswick for their valuable input at various stages of this study.
- le May A: Introducing communities of practice. Communities of practice in health and social care. Edited by: le May A. 2009, Wiley-Blackwell, 3-16.
- Wenger E, McDermott R, Snyder WM: Cultivating Communities of Practice. 2002, Boston, Massachusetts: Harvard Business School Press
- American Productivity & Quality Centre: Building and sustaining communities of practice: Continuing success in knowledge management. 2001, Texas: American Productivity and Quality Centre (APQC)
- Lesser EL, Storck J: Communities of practice and organizational performance. IBM Syst J. 2001, 40: 831-841.View Article
- Schenkel A, Teigland R: Improved organizational performance through communities of practice. J knowledge management. 2008, 12: 106-118. 10.1108/13673270810852421.View Article
- Ranmuthugala G, Plumb J, Cunningham F, Georgiou A, Westbrook J, Braithwaite J: Communities of practice in the health sector: A systematic review of the peer-reviewed literature. 2010, Sydney: University of New South Wales, Australian Institute of Health Innovation
- Li LC, Grimshaw JM, Nielsen C, Judd M, Coyte PC, Graham ID: Use of communities of practice in business and health care sectors: A systematic review. Implement Sci. 2009, 4:
- Fung-Kee-Fung M, Watters J, Crossley C, Goubanova E, Abdulla A, Stern H, Oliver TK: Regional collaborations as a tool for quality improvements in surgery: a systematic review of the literature. Ann Surg. 2009, 249: 565-572. 10.1097/SLA.0b013e31819ec608.View ArticlePubMed
- Braithwaite J, Westbrook JI, Ranmuthugala G, Cunningham F, Plumb J, Wiley J, Ball D, Huckson S, Hughes C, Johnston B: The development, design, testing, refinement, simulation and application of an evaluation framework for communities of practice and social-professional networks. BMC Health Serv Res. 2009, 9: 162-10.1186/1472-6963-9-162.PubMed CentralView ArticlePubMed
- Pawson R, Tilley N: Realistic evaluation. 1997, London: Sage
- Hunt KS, Sridharan S: A Realist evaluation approach to unpacking the impacts of the sentencing guidelines. Am J Eval. 2010, 31: 463-485. 10.1177/1098214010371817.View Article
- Thistleton L: A realistic evaluation of the work of a speech and language therapy service in primary schools (the first schools project) using the perceptions of some of the important stakeholders (teachers, SLTs, and parents). 2008, PhD thesis. The University of Birmingham, School of Education
- Kazi M: A realist evaluation of practice: the NSPCC's Shield project and intensive analysis. Realist evaluation in practice: Health and social work. Edited by: Kazi M. 2003, London, California, New Delhi: Sage Publications Ltd, 95-109.
- Rhodes J, Lok P: An integrative model of organizational learning and social capital on effective knowledge transfer and perceived organizational performance. J Workplace Learning. 2008, 20: 245-258. 10.1108/13665620810871105.View Article
- Helms R, Bosua R, Ignatio R: Impact assessment of knowledge sharing bottlenecks:The Knowledge Sharing Environment Model (KSEM). 2009, Presented at the 20th Australasian Conference on Information Systems, Melbourne, Australia
- Cross R, Parker A: The hidden power of social networks. 2004, Boston, Massachusetts, USA: Harvard Business School Publishing Corporation
- Sales AE, Estabrooks CA, Valente TW: The impact of social networks on knowledge transfer in long-term care facilities: Protocol for a study. Implement Sci. 2010, 5: 49-10.1186/1748-5908-5-49.PubMed CentralView ArticlePubMed
- Ferlie E, Fitzgerald L, McGivern G, Dopson S, Exworthy M: Networks in health care: A comparative study of their management, impact and performance. 2010, London: King's College London, Accessed 13 March 2011, [http://www.sdo.nihr.ac.uk/files/project/102-final-report.pdf]
- Borgatti S, Everett M, Freeman L: UCInet 6.3 For Windows: Software for social network analysis. 2010, Lexington, KY
- Helms R: Redesigning communities of practice using knowledge network analysis. Hands-On Knowledge Co-Creation and Sharing: Practical Methods and Techniques. Edited by: Kazi A, Wohlfart L, Wolf P. Stuttgart. 2007, Germany: KnowledgeBoard, 251-274.
- Zins C: Conceptual approaches for defining data, information, and knowledge. J Am Soc Inf Sci Technol. 2007, 58: 479-493. 10.1002/asi.20508.View Article
- Alavi M, Leidner DE: Review: Knowledge management and knowledge management systems: Conceptual foundations and research issues. MIS Quarterly. 2001, 25: 107-136. 10.2307/3250961.View Article
- Creswick N, Westbrook JI: Social network analysis of medication advice-seeking interactions among staff in an Australian hospital. Int J Med Inform. 2010, 79: e116-125. 10.1016/j.ijmedinf.2008.08.005. Epub 2008 Nov 2012View ArticlePubMed
- Provan KG, Nakama L, Veazie MA, Teufel-Shone NI, Huddleston C: Building community capacity around chronic disease services through a collaborative interorganizational network. Health Educ Behav. 2003, 30: 646-662. 10.1177/1090198103255366.View ArticlePubMed
- Provan KG, Fish A, Sydow J: Interorganizational networks at the network level: A review of the empirical literature on whole networks. J Management. 2007, 33: 479-516.
- Hanneman R, Riddle M: Chapter 8: Embedding. Introduction to social network methods. 2005, Riverside, CA: University of California
- Grippa A: A social network scorecard to monitor knowledge flow across communication media. Knowledge management and practice. 2009, 7: 317-328. 10.1057/kmrp.2009.24.View Article
- Holschuh J, Segal S: Factors related to multiplexity in support networks of persons with severe mental illness. Social Networks and Health. 2002, 8: 293-321.View Article
- Provan KG, Milward HB: Do networks really work? A framework for evaluating public-sector organizational networks. Public Admin Review. 2001, 61: 414-423. 10.1111/0033-3352.00045.View Article
- Berwick DM: The science of improvement. J Amer Med Assoc. 2008, 299: 1182-1184. 10.1001/jama.299.10.1182.View Article
- Greenhalgh T, Humphrey C, Hughes J, Macfarlane F, Butler C, Pawson R: How do you modernize a health service? A realist evaluation of whole-scale transformation in London. Milbank Q. 2009, 87: 391-416. 10.1111/j.1468-0009.2009.00562.x.PubMed CentralView ArticlePubMed
- Hildreth PM, Kimble C: The duality of knowledge. Information Research. 2002, 8:
- Lathlean J, le May A: Communities of practice: an opportunity for interagency working. J Clin Nurs. 2002, 11: 394-398. 10.1046/j.1365-2702.2002.00630.x.View ArticlePubMed
- Kimball L, Ladd A: Facilitator toolkit for building and sustaining virtual communities of practice. Idea Group Publishing. 2004
- Render ML, Brungs S, Kotagal U, Nicholson M, Burns P, Ellis D, Clifton M, Fardo R, Scott M, Hirschhorn L: Evidence-based practice to reduce central line infections. Joint Comm J Qual Patient Saf. 2006, 32: 253-260.
- Chandler L, Fry A: Can communities of practice make a meaningful contribution to sustainable service improvement in health and social care?. J Integr Care. 2009, 17: 41-48.View Article
- Jiwa M, Deas K, Ross J, Shaw T, Wilcox H, Spilsbury K: An inclusive approach to raising standards in general practice: working with a 'community of practice' in Western Australia. BMC Med Res Methodol. 2009, 9: 13-10.1186/1471-2288-9-13.PubMed CentralView ArticlePubMed
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.