- Short report
- Open Access
- Open Peer Review
Community capacity to acquire, assess, adapt, and apply research evidence: a survey of Ontario's HIV/AIDS sector
Implementation Sciencevolume 6, Article number: 54 (2011)
Community-based organizations (CBOs) are important stakeholders in health systems and are increasingly called upon to use research evidence to inform their advocacy, program planning, and service delivery. To better support CBOs to find and use research evidence, we sought to assess the capacity of CBOs in the HIV/AIDS sector to acquire, assess, adapt, and apply research evidence in their work.
We invited executive directors of HIV/AIDS CBOs in Ontario, Canada (n = 51) to complete the Canadian Health Services Research Foundation's "Is Research Working for You?" survey.
Based on responses from 25 organizations that collectively provide services to approximately 32,000 clients per year with 290 full-time equivalent staff, we found organizational capacity to acquire, assess, adapt, and apply research evidence to be low. CBO strengths include supporting a culture that rewards flexibility and quality improvement, exchanging information within their organization, and ensuring that their decision-making processes have a place for research. However, CBO Executive Directors indicated that they lacked the skills, time, resources, incentives, and links with experts to acquire research, assess its quality and reliability, and summarize it in a user-friendly way.
Given the limited capacity to find and use research evidence, we recommend a capacity-building strategy for HIV/AIDS CBOs that focuses on providing the tools, resources, and skills needed to more consistently acquire, assess, adapt, and apply research evidence. Such a strategy may be appropriate in other sectors and jurisdictions as well given that CBO Executive Directors in the HIV/AIDS sector in Ontario report low capacity despite being in the enviable position of having stable government infrastructure in place to support them, benefiting from long-standing investment in capacity building, and being part of an active provincial network. CBOs in other sectors and jurisdictions that have fewer supports may have comparable or lower capacity. Future research should examine a larger sample of CBO Executive Directors from a range of sectors and jurisdictions.
Community-based organizations (CBOs) are important stakeholders in the health sector [1, 2] as they not only provide a wide spectrum of programs and services to the members of their community but also play an advocacy role for broader system-level supports. As we have outlined in more detail in a previous paper , CBOs are typically not-for-profit organizations that: are guided by a specific mission (i.e., an overall goal) shaped by commonly held values within the community they serve; have a governance structure consisting of board members elected from the members in the community; and deliver a specific set of programs or services that are shaped by the mission and values of the organization. Because they are key health system stakeholders, it is important to support their capacity to find and use relevant and high-quality research evidence. Doing so will help ensure that programs, services, and advocacy are informed by the best available evidence.
However, there are many potential challenges related to research use. Barriers that have been consistently identified across sectors include: the complexity of research evidence, organizational barriers, lack of available time, poor access to current literature, lack of timely research, lack of experience and skills for critical appraisal, unsupportive culture for research, lack of actionable messages in research reports, and limited resources for implementation [4–8]. Given these barriers, it is not surprising that a lack of uptake of research evidence has been noted in many different sectors [9–13].
While there are strategies for supporting the use of research by managers of healthcare organizations and by policy makers in government [14, 15], there is still an important gap in the availability of a specific strategy for CBOs . Many existing strategies for supporting the use of research evidence are based on experience and anecdotal evidence rather than on rigorous evidence of effects [8, 14, 16]. Moreover, strategies designed for supporting the use of research evidence by healthcare organizations and governments may not be relevant to the specific contexts and capacity of CBOs. The transferability of these strategies to CBOs is difficult to determine without first collecting evidence about their current capacity to find and use research evidence and then grappling with how to align capacity-building strategies with local realities [17–21].
In order to begin to fill this gap, we conducted an assessment of the capacity of HIV/AIDS CBOs in one Canadian jurisdiction (Ontario). The role of CBOs is particularly important for addressing the HIV/AIDS epidemic in Ontario as approximately half of all people living with HIV/AIDS in Canada reside in that province [22, 23]. The province has a strong network of HIV/AIDS CBOs to respond to the local HIV epidemic, with 51 organizations serving 19 different regions at the time of this study (there are now 46 organizations) . As such, this network provides essential on-the-ground support for people living with HIV/AIDS, especially for the most vulnerable .
We invited Executive Directors of HIV/AIDS CBOs that are funded by the AIDS Bureau of the Ontario Ministry of Health and Long-Term Care and members of the Ontario AIDS Network (n = 51) to complete an online survey. We asked each Executive Director to work with one or more managers and front-line staff to complete the survey, allowing them to respond to areas that they are most knowledgeable about but also be informed by the specific knowledge of other managers and staff.
The CBO Executive Directors in Ontario were invited to complete an adapted version of the Canadian Health Services Research Foundation's "Is Research Working for You?" survey, which has been previously validated [26, 27]. The survey is organized into four areas of assessment: (1) Acquire: Can your organization find and obtain the research findings it needs?; (2) Assess: Can your organization assess research findings to ensure they are reliable, relevant, and applicable to you?; (3) Adapt: Can your organization present the research to decision makers in a useful way?; and (4) Apply: Are there skills, structures, processes, and a culture in your organization to promote and use research findings in decision making ? Each domain is then broken down into subsections that ask how well their organization performs specific tasks, and each question uses a five-point Likert scale (question anchors are 1 = Don't do, 2 = Do poorly, 3 = Do inconsistently, 4 = Do with some consistency, and 5 = Do well).
We analyzed the results of the survey by calculating response frequencies for each question. We did not conduct statistical comparisons between organizations due to the small sample available for the survey. Ethics approval was obtained from the University of Toronto to complete this research.
Executive Directors from 25 organizations completed the survey (response rate = 49%). These organizations collectively provide services to approximately 32,000 clients per year with 290 full-time equivalent staff. Participant/organizational characteristics are summarized in Table 1, and the results for each question are presented in Tables 2, 3, 4 to 5. The survey respondents were predominantly from organizations with an exclusive focus on HIV/AIDS. In addition, the sample appears balanced based on region served (approximately half were based in the greater Toronto area) and includes organizations with varying scale (as measured by the number of full-time equivalent staff) and service volume (as measured by the number of clients served each year). Lastly, the organizations included in our sample provide HIV/AIDS services to a broad array of populations, including women, youth, men who have sex with men, injection drug users, specific ethno-racial communities, as well as people living with HIV/AIDS.
Overall, we found organizational capacity to acquire, assess, adapt, and apply research evidence to be low, with a couple of notable exceptions where the majority of organizations provided ratings of "Do with some consistency" or "Do well." Capacity for applying research evidence appears to be relatively strong, with approximately half of organizations indicating they "Do with some consistency" or "Do well" on all but two questions (related to time and engaging on-staff researchers) for the subdomain related to whether their decision-making processes have a place for research (Table 5, subsection VII). In addition, most CBOs indicated that they have a corporate culture that is supportive of research use (64% selected "Do with some consistency" or "Do well").
Few organizations rated any of the questions related to acquiring, assessing, or adapting research evidence higher than "Inconsistent." Areas of relative strength include identifying research evidence through peer networks, "grey" literature reports, and through websites. Capacity was lowest for the domains related to: acquiring research (subsection I); assessing the reliability, quality, relevance, and applicability of research evidence (subsections III and IV); and summarizing results in a user-friendly way (subsection V). A higher proportion of organizations selected the "Don't do" or "Do poorly" response options in these domains. Overall, the results indicate that CBOs are not only limited in skilled staff with time, incentive, and resources for summarizing results in a user-friendly way and assessing the quality and reliability of research, but they also have limited arrangements with external experts to help them.
This study gives initial insight into the capacity of CBOs to acquire, assess, adapt, and apply research in their organizations, which can provide a starting point for developing their capacity to find and use research evidence. Efforts to build capacity among CBOs should also draw on key facilitators for supporting the use of research evidence that have been cited for other groups of stakeholders, such as ongoing interactions between researchers, managers, and policy makers and ensuring research is available in a timely manner (e.g., through searchable databases of research syntheses) . In addition, many strategies to support the use of research evidence highlight the importance of clear summaries that are tailored to the specific audience and highlight the take-home messages from research [7, 8, 21, 28, 29].
The primary strength of this study is that it provides a baseline assessment of CBO capacity to acquire, assess, adapt, and apply research, which can be used for identifying areas for future capacity-building strategies. In addition, the data from this study can be used in future evaluations of organizational capacity after capacity-building initiatives have been implemented. However, the results should be interpreted with caution given the low response rate and the small sample size. In addition, the generalizability of our results to other Canadian and international jurisdictions may be limited as we only surveyed organizations in one Canadian jurisdiction.
Given the limited capacity to find and use research evidence, we recommend a capacity-building strategy for HIV/AIDS CBOs that focuses on providing the tools, resources, and skills needed to help CBOs more consistently acquire, assess, adapt, and apply research evidence. Such a strategy may also be appropriate in other sectors and jurisdictions given that CBO Executive Directors in the HIV/AIDS sector in Ontario report low capacity despite being in the enviable position of having stable government infrastructure in place to support them, benefiting from long-standing investment in capacity building, and being part of an active provincial network. CBOs in other sectors and jurisdictions that have fewer supports may have comparable or lower capacity.
Capacity building does not mean that CBOs need to do everything themselves. For instance, capacity building can focus on efforts to facilitate "pull" , such as developing and maintaining "one-stop shopping" websites that provide access to high-quality and relevant systematic reviews of the literature with user-friendly summaries (e.g., Health Systems Evidence [http://www.healthsystemsevidence.org] for health system managers and policy makers and http://www.health-evidence.ca for public health) [30, 31], and/or "rapid response" services, such as those developed through the Ontario HIV Treatment Network for HIV/AIDS CBOs  and the Health Technology Inquiry Service at the Canadian Agency for Drugs and Technologies in Health . Additional efforts could also include facilitating access to research evidence by providing more links to external experts who can assist CBOs with identifying relevant research evidence and assessing its quality, reliability, and local applicability. Such efforts to support the use of research evidence will require in-depth consultation with CBOs to develop approaches that reflect their specific needs.
World Health Organization: Ottawa Charter for Health Promotion. 1986, Copenhagen: World Health Organization, Division of Health Promotion, Education & Communication
World Health Organization Regional Office for Europe: Health 21 Health for All in the 21st Century. 1999, Copenhagen: World Health Organization Regional Office for Europe
Wilson MG, Lavis JN, Travers R, Rourke SB: Community-based knowledge transfer and exchange: Helping community-based organizations link research to action. Implementation Science. 2010, 5: 33-10.1186/1748-5908-5-33.
Dobbins M, Rosenbaum P, Plews N, Law M, Fysh A: Information transfer: What do decision-makers want and need from researchers. Implementation Science. 2007, 2: 20-10.1186/1748-5908-2-20.
Haynes RB, Haines A: Getting research findings into practice: Barriers and bridges to evidence based clinical practice. BMJ. 1998, 317: 273-276.
Innvaer S, Vist GE, Trommald M, Oxman AD: Health policy-makers' perceptions of their use of evidence: A systematic review. Journal of Health Services Research and Policy. 2002, 7: 239-244. 10.1258/135581902320432778.
Lavis JN, Davies HTO, Oxman AD, Denis JL, Golden-Biddle K, Ferlie E: Towards systematic reviews that inform health care management and policy-making. Journal of Health Services Research and Policy. 2005, 10: S1:35-S1:48.
Mitton C, Adair CE, McKenzie E, Patten SB, Wayne Perry B: Knowledge Transfer and Exchange: Review and Synthesis of the Literature. Milbank Quarterly. 2007, 85: 729-768. 10.1111/j.1468-0009.2007.00506.x.
Graham ID, Logan J, Harrison MB, Straus SE, Tetroe J, Caswell W: Lost in translation: Time for a map?. Journal of Continuing Education in the Health Professions. 2007, 26: 13-24.
Grol RP, Dalhuijsen J, Thomas S, Veld C, Rutten G, Mokking H: Attributes of clinical guidelines that influence use of guidelines in general practice: Observational study. BMJ. 1998, 317: 858-861.
Lavis JN: Ideas at the margin or marginalized ideas? Nonmedical determinants of health in Canada. Health Affairs. 2002, 21: 107-112. 10.1377/hlthaff.21.2.107.
McGlynn E, Asch SM, Adams J, Keesey J, Hicks J, DeCristofaro A: The quality of health care delivered to adults in the United States. New England Journal of Medicine. 2003, 348: 2635-2645. 10.1056/NEJMsa022615.
Schuster M, McGlynn E, Brook RH: How good is the quality of health care in the United States?. Milbank Quarterly. 1998, 76: 517-563. 10.1111/1468-0009.00105.
Walshe K, Rundall TG: Evidence-based management: From theory to practice in health care. Milbank Quarterly. 2001, 79: 429-457. 10.1111/1468-0009.00214.
Rundall TG, Martelli PF, Arroyo L, McCurdy R, Graetz I, Neuwirth EB: The informed decisions toolbox: tools for knowledge transfer and performance improvement. Journal of Healthcare Management. 2007, 52: 325-341.
Young SK: Evidence-based management: A literature review. Journal of Nursing Management. 2002, 10: 145-151. 10.1046/j.1365-2834.2002.00309.x.
Miller RL, Shinn M: Learning from communities: overcoming difficulties in dissemination of prevention and promotion efforts. Am J Community Psychol. 2005, 35: 169-183. 10.1007/s10464-005-3395-1.
McDonald PW, Viehbeck S: From evidence-based practice making to practice-based evidence making: creating communities of (research) and practice. Health Promotion Practice. 2007, 8: 140-144. 10.1177/1524839906298494.
Lavis JN, Robertson D, Woodside JM, McLeod CB, Abelson J: How can research organizations more effectively transfer research knowledge to decision makers?. Milbank Quarterly. 2003, 81: 221-248. 10.1111/1468-0009.t01-1-00052.
Lavis JN, Posada FB, Haines A, Osei E: Use of research to inform public policymaking. The Lancet. 2004, 364: 1615-1621. 10.1016/S0140-6736(04)17317-0.
Jacobson N, Butterill D, Goering P: Development of a framework for knowledge translation: Understanding user context. Journal of Health Services & Research Policy. 2003, 8: 94-99. 10.1258/135581903321466067.
Public Health Agency of Canada: HIV/AIDS Epi Updates. 2006, Ottawa: Centre for Infectious Disease Prevention and Control, Public Health Agency of Canada
Public Health Agency of Canada: HIV/AIDS Epi Updates. 2007, Ottawa, ON: Centre for Infectious Disease Prevention and Control, Public Health Agency of Canada
Ontario AIDS Network. Ontario AIDS Network: Members List: 2011, [http://ontarioaidsnetwork.on.ca/site/oan-community/members/]
Crook J, Browne G, Roberts J, Gafni A: Impact of Support Services Provided by a Community-Based AIDS Service Organization on Persons Living With HIV/AIDS. Journal of the Association of Nurses in AIDS Care. 2005, 16: 39-49. 10.1016/j.jana.2005.05.003.
Canadian Health Services Research Foundation: Is Research Working for you? A Self-Assessment Tool and Discussion Guide for Health Services and Policy Organizations. 2005, Ottawa: Canadian Health Services Research Foundation
Kothari A, Edwards N, Hamel N, Judd M: Is research working for you? Validating a tool to examine the capacity of health organizations to use research. Implementation Science. 2009, 4: 46-10.1186/1748-5908-4-46.
Lavis JN, Lomas J, Hamid M, Sewankambo NK: Assessing country-level efforts to link research to action. Bulletin of the World Health Organization. 2006, 84: 620-628. 10.2471/BLT.06.030312.
Lavis JN, Wilson MG, Grimshaw JM, Haynes RB, Ouimet M, Raina P: Supporting the use of health technology assessments in policy making about health systems. International Journal of Technology Assessment in Health Care. 2010, 26: 405-414. 10.1017/S026646231000108X.
Health-Evidence: 2009, [http://www.health-evidence.ca/]
McMaster Health Forum. Health Systems Evidence. McMaster Health Forum: Hamilton, Canada, 2010 [http://www.healthsystemsevidence.org/]
Ontario HIV Treatment Network. Rapid Response Service: 2011, [http://www.ohtn.on.ca/Pages/Knowledge-Exchange/Rapid-Response-Service.aspx]
CADTH CADTH: Health Technology Inquiry Service: 2011, [http://cadth.ca/en/products/rapid-response/rapid-response-service]
Acknowledgements and funding
The study and manuscript preparation was funded by The Ontario HIV Treatment Network. SBR, MGW, and JB each receive a salary from the Ontario HIV Treatment Network. JNL receives salary support as Canadian Research Chair of Knowledge Transfer and Exchange. We would also like to thank Tarik Bereket for coordinating the survey administration.
Some of the organizations included in the study sample may receive funding from the Ontario HIV Treatment Network. JNL was Chair and then a member of the Board of Directors of an ASO in Ontario but he played no role in administering or completing the survey.
MGW contributed to the conception and design of the study, the data analysis and interpretation, drafted the original manuscript, and incorporated revisions from the study team. SBR contributed to the conception and design of the study, the data analysis and interpretation, and helped draft and revise the manuscript. JNL contributed to the conception and design of the study, the data interpretation, and provided revisions to the manuscript. JB contributed to the conception and design of the study and provided revisions to the manuscript. RT contributed to the conception and design of the study and provided revisions to the manuscript. All authors read and approved the final manuscript.