- Study protocol
- Open Access
Organizational supports used by private child and family serving agencies to facilitate evidence use: a mixed methods study protocol
Implementation Science volume 12, Article number: 49 (2017)
Challenges to evidence use are well documented. Less well understood are the formal supports—e.g., technical infrastructure, inter-organizational relationships—organizations may put in place to help overcome these challenges. This study will identify supports for evidence use currently used by private child and family serving agencies delivering publicly funded behavioral health and/or human services; examine contextual, organizational, and managerial factors associated with use of such supports; and determine how identified supports affect evidence use by staff at multiple levels of the organization.
We will use a sequential explanatory mixed methods design, with study activities occurring in two sequential phases: In phase 1, quantitative survey data collected from managers of private child and family serving agencies in six states (CA, IN, KY, MO, PA, and WI) and analyzed using both regression and qualitative comparative analysis (QCA) will identify organizational supports currently being used to facilitate evidence use and examine the contextual, organizational, and managerial factors associated with the use of such supports. In phase 2, data from phase 1 will be used to select a purposive sample of 12 agencies for in-depth case studies. In those 12 agencies, semi-structured interviews with key informants and managers, focus groups with frontline staff, and document analysis will provide further insight into agencies’ motivation for investing in organizational supports for evidence use and the facilitators and barriers encountered in doing so. Semi-structured interviews with managers and focus groups with frontline staff will also assess whether and how identified supports affect evidence use at different levels of the organization (senior executives, middle managers, frontline supervisors, and frontline staff). Within- and between-case analyses supplemented by QCA will identify combinations of factors associated with the highest and lowest levels of staff evidence use.
This study will inform efforts to improve sustainment, scale-up, and spread of evidence by providing insight into organizational and managerial strategies that facilitate evidence use, the contexts in which these strategies are most effective, and their effect on evidence use by staff at different levels of the organization.
Effective use of research evidence, or knowledge that has been subjected to testing and found to be credible [1, 2], by health and human service practitioners can reduce disparities in the costs and quality of care, enhance service effectiveness and worker satisfaction, and improve outcomes experienced by vulnerable children and families [3–6]. Recognizing the importance of research evidence for improving organizational performance and service outcomes, policymakers in many countries have made evidence-informed health and human services a priority [7, 8]. In the USA, an increasing number of state and local governments now link the use of research evidence regarding “effective” programs and practices, including evidence-based treatments (EBTs), to funding decisions or service reimbursement .
Despite the importance of research evidence use for achieving positive outcomes for vulnerable children and families, research-to-practice gaps persist across settings, conditions, and population groups . In health care, for example, only about half of recommended care practices are implemented [11, 12]. Uptake of research recommendations is even lower for prevention and behavior change programs and for managerial practices such as goal setting and performance feedback [13, 14]. In the human services sector, national survey evidence suggests that only 25% of child welfare agency programs and practices are evidence based . In all states, an increasingly large proportion of publicly funded services for child welfare-involved children and families are delivered via purchase of service contracts with private agencies [16, 17]. Although these private agencies are heterogeneous as a population , the majority provide both behavioral health and human services to vulnerable children and families . However, only a small proportion of agency administrators and practitioners report regularly using research to inform their daily work [20, 21].
Challenges to the use of research evidence, and particularly to the adoption, implementation, and sustainment of EBTs, are well documented [8, 22, 23]. Less studied are the formal supports—i.e., staff positions, infrastructural supports, and special initiatives—organizations may put in place to help overcome these challenges. For example, in some clinical settings, knowledge broker positions (i.e., intermediaries accountable for encouraging knowledge use) have successfully been used to support evidence-informed decision-making . Ties to opinion leaders and researchers outside of the organization [25, 26], a supportive technical infrastructure [27, 28] and an organizational climate that rewards evidence use [24, 29] have also been identified as affecting evidence use.
When present, these supports may signify an organizational commitment to improving service quality and program outcomes. However, these supports typically require significant managerial and workforce investment and can be costly to develop, implement, and sustain. Agencies may vary in their capacity and/or willingness to utilize such supports and in their approach to systematizing the use of research evidence by managers and other staff. Currently, we know little about the extent to which health and human service agencies use such supports to facilitate evidence use by staff. We also know little about the contextual and organizational factors associated with agency use of such supports or the extent to which these supports affect evidence use by staff at different levels of the organization.
The current study will address this gap by collecting qualitative and quantitative data from private child and family serving agency managers and staff to address the following aims:
Aim 1: Identify formal organizational supports used by private child and family serving agencies to facilitate evidence use
Aim 2: Examine contextual, organizational, and managerial factors affecting agency use of formal supports
Aim 3: Determine how formal organizational supports affect use of research evidence at multiple levels of the organization
Study activities will be guided by the conceptual framework in Fig. 1. This model was developed following a review of the literature on evidence use in the human services, public health, and associated scholarly domains and is informed by concepts from resource dependence theory , institutional theory [31, 32], theories of leadership [33, 34], and the strategic management literature [35, 36].
Organizational supports that facilitate evidence use
We propose to examine four types of formal supports that agencies may use to facilitate evidence use. These supports include linkage and exchange efforts, technical infrastructure, other knowledge management infrastructure, and strategic alignment.
Linkage and exchange efforts refer to formal ties to knowledge brokers  outside of the agency who can assist in acquiring, assessing, adapting, or applying evidence in decision-making or practice [38–41]. These knowledge brokers can include researchers, professional associations, consultants, or research use networks. The use of knowledge brokers to assist with distilling and disseminating research to practitioners is increasingly common in Canada and the UK [42–44] and has also been seen increased uptake in the US health care sector  but has not been systematically examined in the US behavioral health or human service sectors.
Technical infrastructure includes internal data systems and/or other tools designed to facilitate access to and use of evidence by agency staff [46–48]. Research suggests that investment in technical infrastructure may be necessary to facilitate evidence use [49, 50]. At a basic level, agencies may provide staff with computers that permit use of free online resources or develop virtual libraries to promote staff access to research evidence. Agencies may also invest in performance measurement systems designed to collect and report data on program- or agency-level performance indicators or in client management information systems that provide real-time data on client service utilization and outcomes.
Other knowledge management infrastructure includes other agency resources allocated for the purpose of building capacity to use or promote evidence use. These supports include having formal positions (full or part-time) responsible for supporting evidence use, e.g., internal knowledge brokers or other staff formally assigned to retrieve, translate, and disseminate research knowledge within the agency or research agency programs and initiatives. Agencies may also promote staff training and continuing education on specific research topics.
Finally, strategic alignment  refers to other formal efforts intended to establish an organizational culture and climate, i.e., workers’ perceptions of norms and expectations in their work environment [52, 53], that prioritizes evidence use. Examples include emphasis on the importance of evidence use in the agency’s mission or strategic plan or establishing policies and practices that promote accountability for evidence use (e.g., incorporating a requirement for research evidence use into staff performance reviews or compensation plans).
Factors hypothesized to affect agency use of formal organizational supports
As shown in Fig. 1, we conceptualize agency use of formal organizational supports to facilitate evidence use as affected by factors at the environmental, agency, and managerial levels.
Contextual and organizational factors hypothesized to affect agency use of such supports are informed by two complementary macro-theoretical perspectives on organizational behavior, resource dependence theory and institutional theory [30, 31, 54]. Consistent with resource dependence theory and institutional theory, we posit that agencies will invest in formal organizational supports for evidence use if leaders perceive this investment as enhancing agencies’ ability to secure resources vital to organizational maintenance and survival (e.g., providing a competitive edge with clients or funders) [55, 56] or if they face institutional pressure to do so from funders, policy-makers, competitors, and accrediting and educational bodies whose support agencies need in order to be perceived as “legitimate” [57–59]. Specific factors hypothesized to affect agency investment in organizational supports for evidence use include inter-organizational competition , accreditation requirements promoting evidence-informed practice, state or local government policies requiring contractor use of evidence-based practice, and agency size [61, 62].
Managerial characteristics hypothesized to affect agency use of such supports are identified from the strategic management literature [35, 36], theories of leadership [33, 34], and prior research on evidence use in health care [38, 63, 64]. Specifically, we propose to examine three types of managerial characteristics hypothesized to affect agency use of formal organizational supports to facilitate evidence use: boundary spanning activities [65, 66] (i.e., management of relations with external entities), leadership behaviors  (e.g., the actions leaders take to motivate staff and implement plans), and managers’ education, training, and attitudes towards evidence use [38, 63, 64]. These characteristics, particularly leadership behaviors and attitudes towards evidence use, may also directly affect evidence use by agency staff.
We will use a sequential explanatory mixed methods design , with study activities occurring in two sequential phases: In phase 1, quantitative survey data collected from managers of private child and family serving agencies in six states will examine organizational supports currently being used by agencies to facilitate evidence use (aim 1) and identify contextual, organizational, and managerial factors associated with agency use of such supports (aim 2). In phase 2, data from phase 1 will be used to select a purposive sample of 12 agencies for in-depth case studies. In those 12 agencies, key informant interviews, focus groups, and document analysis will provide further insight into agencies’ motivation for investing in organizational supports for evidence use and the facilitators and barriers encountered in doing so (aim 2). Semi-structured interviews with managers (executives, middle managers, and frontline supervisors) and focus groups with frontline staff will examine whether and how identified supports affect evidence use at different levels of the organization (aim 3). An overview of our proposed study design is provided in Fig. 2.
Phase 1: quantitative survey of private agency executives
In phase 1, we will administer a web-based survey to private child and family serving agency executives in six states that vary in size and sociopolitical context (CA, IN, KY, MO, PA, and WI). Included agencies in six states will enhance generalizability of study findings and allow us to better control for differences in local institutional context. It will also ensure that we have a sufficient sample size to conduct proposed analyses.
Sampling strategy and recruitment procedures
Eligible agencies in each state include all private agencies serving the child welfare population that are members of a state or national association of private providers. Prior evidence suggests that public agencies contract with such agencies for a broad spectrum of services, ranging from core child welfare services such as foster care and case management to behavioral health and health care services [18, 69]. In the absence of a comprehensive database identifying our study population, we will partner with the major membership associations representing private child and family serving agencies in state and federal policymaking to distribute survey invitations to their members, as well as private non-member agencies on their association listservs. Our partners include six state membership associations of private child and family serving agencies as well as the Alliance for Strong Families and Communities (the Alliance). Based on 2015 membership, we anticipate a sample size of approximately 448 agencies.
A single executive in each agency will receive an electronic letter of invitation that provides additional information on the study’s purpose and a hyperlink to the survey. Typically, this executive will be the individual identified by our study partners as their primary contact for that agency. Respondents will be asked to complete the survey only once per agency. To facilitate tracking, each agency will receive a unique survey link. The survey portal will remain open for 12 weeks. Upon completion of the survey, responding agencies will be entered in a raffle to win one of 45 $100 electronic gift cards (or equivalent donation to their agencies). Respondents will also be asked if their agency is interested in participating in the phase 2 of the study.
Survey instrument development
An overview of survey domains and content is provided in Table 1 reflecting measures previously validated either in child welfare or other health and human service contexts as well as prior research conducted by the study investigators [18, 70–75]. To ensure questions are of high salience to agency executives, the survey instrument will be piloted with an expert panel comprised of 5–6 private agency administrators and other key stakeholders from states not targeted for survey participation, and refined to ensure it takes no more than 20 min to complete.
The phase 1 unit of analysis is the private child and family serving agency. Factor analysis will examine the underlying factor structure of any quantitative measures not previously used in this context . Univariate and bivariate analyses will descriptively examine the prevalence of different formal organizational supports to facilitate evidence use across the sample of private child welfare agencies (aim 1). Subsequent analyses will explore the extent to which contextual, agency, and managerial factors identified in Fig. 1 are associated with the presence and use of different organizational supports for evidence use (aim 2). These analyses will occur in three stages: First, multiple regression will identify contextual, organizational, and managerial factors significantly associated with agency use of organizational supports to facilitate evidence use.
Next, qualitative comparative analysis (QCA) will be used to identify specific combinations of contextual, agency, and managerial factors associated with high vs. low levels of organizational support for evidence use [77–79]. QCA is a set-theoretic method based on Boolean algebra that is increasingly used in sociological, management, and health services research to explore complex social phenomena [80, 81]. The primary benefit of QCA for the proposed research is the technique’s ability to identify conditions or combinations of conditions that are necessary vs. sufficient for agency investment in organizational supports for evidence use. A condition is considered necessary if it must be present in order for an outcome to occur; however, the presence of a necessary condition does not ensure the outcome will occur. For example, agencies may need to be a certain size in order to invest in management information systems or other technical infrastructure. However, even if agency size is necessary, it may not be sufficient for agency investment in technical infrastructure in the absence of other conditions such as institutional pressures or managerial support for evidence use. Conditions or combinations of conditions are considered sufficient if they consistently produce an outcome of interest when present. In our study, there may be multiple combinations of contextual, organizational, and managerial factors that are sufficient but not necessary for agency investment in organizational supports for evidence use. For example, having either an executive with a positive attitude towards research or funders that link evidence use to payments could each be sufficient to ensure agency investment in supports for evidence use. This assumption of causal heterogeneity, i.e., different combinations of conditions can lead to the same outcome , is a strength of QCA and will allow for development of models that better reflect the reality of complex organizational phenomena than if we relied solely on regression-based techniques.
Configurational solutions identified through QCA will be inputted into regression models to identify configurations of factors that are significantly associated with agency use of formal organizational supports even after controlling for other agency characteristics. Regression analyses will be conducted using Stata 13.0 . QCA will be conducted using either the fsQCA software  or the FUZZY module in Stata , depending on the specific constructs being tested.
Phase 2: mixed methods case studies of 12 agencies
In phase 2, our primary aim is to examine the ways in which identified organizational supports affect the use of evidence at multiple levels of the agency (aim 3). We will also explore in more detail the contextual, organizational, and managerial factors that may affect agency use of such supports and use of research at different levels of the agency (aim 2). To achieve this objective, we will employ a multiple case study design, with the agency as the unit of analysis.
In phase 2, eligible agencies include all private child and family serving agencies from phase 1 that expressed an interest in participating in phase 2. Eligible agencies will be stratified by prevalence of formal organizational supports for evidence use, and a maximum variation sampling procedure  will be used to identify a diverse sample of 12 private agencies for in-depth case study analysis. Within each of these 12 agencies, qualitative data will be collected from respondents at multiple levels of the organization (e.g., agency executives, middle managers, frontline supervisors, and frontline staff).
Semi-structured interviews with key informants and managers
We propose to conduct semi-structured qualitative interviews with key informants and managers at multiple levels (e.g., executive team, middle management, frontline supervisors) within each agency (~8–10 interviews per agency). Qualitative interviews with 2–3 key informants in each agency will provide insight into agency strategic priorities, motivation for investing in identified organizational supports, and facilitators and barriers to putting these supports in place. Qualitative interviews with managers (additional ~6–8 total) will provide insight into how respondents at different levels of the agency use research evidence and the ways in which available organizational supports affect evidence use. Interviews will also further explore contextual, organizational, and managerial factors—particularly leadership behaviors and staff training and resources—that may affect perceived utility of these organizational supports and overall evidence use (see Table 2).
All interviews will last no longer than 45 min and will be conducted using a semi-structured interview guide tailored to the respondent’s role within the agency. With respondents’ permission, all interviews will be recorded and transcribed verbatim.
Focus groups with frontline staff
Focus groups rely on group interaction to generate insights and can be effective at encouraging participation from individuals reluctant to be interviewed on their own . In phase 2, focus groups with frontline staff will examine the extent to which staff currently use research evidence and the ways in which identified organizational supports and leader behaviors do or do not affect evidence use (see Table 2). Depending on agency size, we anticipate conducting 2–3 focus groups per agency, with 5–8 participants per focus group. An experienced moderator will introduce the topics to be addressed, monitor dynamics of group discussion to ensure all views are represented, and ensure the discussion stays on track. Focus groups will be conducted in absence of other agency personnel, and the script will be tailored to create an atmosphere in which staff feel safe and comfortable sharing their opinions (see Table 2). A second researcher will be present to take detailed notes on the discussion. Focus groups will last approximately 90–100 min and, with respondents’ permission, will be recorded and transcribed verbatim.
As appropriate during the semi-structured interview process, key informants will be asked to provide access to documents that clarify organizational policies, priorities, or supports for evidence use. These documents will provide a low-cost method of augmenting respondents’ descriptions of their organizational contexts and available supports for evidence use [84, 86]. These documents may include but are not limited to program manuals, quality improvement plans and practice protocols, the agency’s response to a request for proposals that seeks funding for training or other infrastructure development activities, or publicly available information on the agency’s structure and services. With respondents’ and agencies’ permission, these documents will be uploaded into NVivo for analysis.
Phase 2 analyses will occur in multiple stages. First, all qualitative data will be uploaded into the qualitative software QSR NVivo 10.0 for analysis . Case study analyses of these qualitative data will be guided by pattern-matching logic . Procedurally, this analysis will involve three steps: coding, within-case analysis, and cross-case analysis. In the first step, all qualitative data will be coded in NVivo. Our initial codebook will be informed by constructs identified in Fig. 1 and the results of phase 1 data collection but may subsequently be refined to include emergent constructs identified from the data.
In the next step, we will conduct within- and cross-case analyses. Specifically, coded data will be analyzed to identify themes in the coded data for each construct, the degree to which these themes emerge in the data (“strength”) and the degree to which each construct positively or negatively affects evidence use by staff (“valence”). We will also assess the degree to which observed relationships within and across cases are consistent with the conceptual model outlined in Fig. 1 .
Finally, phase 2 qualitative data will be calibrated and analyzed using QCA. Our objective in applying QCA will be to identify combinations of factors associated with high levels of staff research use and low levels of staff research use. Given the limited number of cases in our sample, limited diversity may prevent us from identifying meaningful combinations of factors associated with the desired outcomes. If that is the case, we will forego QCA and rely purely on the within- and cross-case analysis results. Results will be shared with participating agencies and used to refine the conceptual model in Fig. 1.
Prior research has identified numerous barriers to evidence use by practitioners. Research evidence can be costly for practitioners to access. The quality of available evidence and its perceived relevance to local organizational and practitioner needs can also vary considerably [40, 43, 90]. In particular, many EBTs are developed and tested with specific client populations in relatively resource rich settings; however, the contexts in which practitioners are expected to translate these EBTs are often significantly more heterogeneous in terms of available resources, client characteristics, and supportive infrastructure [91–93]. Strategies for beginning to address such concerns include the use of hybrid research designs that simultaneously assess program effects and implementation and the development of processes that allow for structured (rather than ad hoc) adaptation of EBTs to local contexts [94, 95]. However, equally critical for sustainment, scale-up, and spread of evidence to lower resource settings are a better understanding of the contexts in which evidence is being used and the extent to which effective strategies for facilitating evidence use may vary across these contexts.
This study identifies organizational supports (e.g., technical infrastructure, knowledge management infrastructure, linkage and exchange efforts) being used by private child and family serving agencies to promote evidence use. The study also examines the contexts in which these supports are more prevalent, identifies multilevel factors driving agency investment in these supports, and explores the conditions under which these supports may affect evidence use at different levels of the organization. Key strengths include the study’s focus on private child and family serving agencies, who play a critical role in delivery of publicly funded behavioral health and/or human services but are under-examined in the literature; the use of a multi-state sample that allows for systematic examination of how different institutional and market factors influence agencies’ support for evidence use; and the use of a configurational comparative approach to inform analyses, which will allow for identification of solutions that better reflect the reality of complex organizational phenomena.
In all states, public and private agencies are challenged to use evidence to improve organizational performance, including child and family outcomes [96–99]. Results will provide insight into strategies that may be effective for scaling evidence use at multiple levels of the organization and across different institutional environments and, ultimately, improving outcomes for vulnerable children and families.
Alliance for Strong Families and Communities
Qualitative comparative analysis
Higgs J, Jones M. Will evidence-based practice take the reasoning out of practice? In: Higgs J, Jones M, editors. Clinical reasoning in the health professionals. 2nd ed. Oxford: Butterworth Heineman; 2000. p. 307–15.
Rycroft-Malone J, Seers K, Titchen A, Harvey G, Kitson A, McCormack B. What counts as evidence in evidence-based practice? J Adv Nurs. 2004;47:81–90.
Cabassa LJ, Baumann A. A two-way street: bridging implementation science and cultural adaptations of mental health treatments. Implement Sci. 2013;8:90.
Napoles AM, Santoyo-Olsson J, Stewart AL. Methods for translating evidence-based behavioral interventions for health-disparity communities. Prev Chronic Dis. 2013;10:E193. doi:10.5888/pcd10.130133.
Grol R. Successes and failures in the implementation of evidence-based guidelines for clinical practice. Med Care. 2001;39:1146–54.
Aarons GA, Sommerfeld D, Hecht D, Silvosky J, Chaffin M. The impact of evidence-based practice implementation and fidelity monitoring on staff turnover: evidence for a protective effect. J Consult Clin Psychol. 2009;77:270–80.
Dill K, Shera W. Implementing evidence-informed practice: international perspectives. Toronto: Canadian Scholars’ Press Inc; 2012.
Gray M, Joy E, Plath D, Webb S. Implementing evidence-based practice: a review of the empirical research literature. Res Soc Work Pract. 2013;23:157–66.
Littell J, Shlonsky A. Toward evidence-informed policy and practice in child welfare. Res Soc Work Pract. 2010;20:723–5.
Carrilio T, Packard T, Clapp J. Nothing in—nothing out: barriers to the use of performance data in social service programs. Adm Soc Work. 2003;27:61–76.
McGlynn E, Asch S, Adams J, Keesey J, Hicks J, DeCristofaro A, Kerr E. The quality of health care delivered to adults in the United States. N Engl J Med. 2003;348:2635–45.
Mangione-Smith R, DeCristofaro A, Setodji C, Keesey J, Klein D, Adams J, Schuster MA, McGlynn E. The quality of ambulatory care delivered to children in the United States. N Engl J Med. 2007;357:1515–23.
Locke E, Latham G. Goal setting: a motivational technique that works. Englewood Cliffs: Prentice-Hall; 1984.
Rousseau D. Is there such a thing as evidence-based management? Acad Manage Rev. 2006;31:256–68.
Horwitz SM, Hurlburt MS, Goldhaber-Fiebert J, Palinkas L, Rolls-Reutz J, Zhang J, Fisher E, Landsverk J. Exploration and adoption of evidence-based practice by U.S. child welfare agencies. Child Youth Serv Rev. 2014;39:147–52.
Gronbjerg KA, Salamon LM. Devolution, marketization, and the changing shape of government-nonprofit relations. In: Salamon LM, editor. The state of nonprofit America. 2nd ed. Washington: Brookings Institution; 2012.
Smith S. The political economy of contracting and competition. In: Hasenfeld Y, editor. Human services as complex organizations. Los Angeles: Sage Publications; 2009. p. 139–60.
Chuang E, Collins-Camargo C, McBeath B, Wells R, Bunger A. An empirical typology of private child and family serving agencies. Child Youth Serv Rev. 2014;38:101–12.
Smith S. Cross-sector nonprofit-government financing. In: Steuerle EBCE, editor. Nonprofits and government: collaboration and conflict. 3rd ed. Washington: Urban Institute Press; 2016.
Collins-Camargo C, Sullivan D, Murphy A. Use of data to assess performance and promote outcome achievement by public and private child welfare agency staff. Child Youth Serv Rev. 2011;33:330–9.
Chagnon F, Pouliot L, Malo C, Gervais M, Pigeon M. Comparison of determinants of research knowledge utilization by practitioners and administrators in the field of child and family social services. Implement Sci. 2010;5:41.
Barwick M, Boydell K, Stasiulis E, Ferguson H, Blase K, Fixsen D. Research utilization among children’s mental health providers. Implement Sci. 2008;3:19.
Aarons GA, Hurlburt MS, Horwitz SM. Advancing a conceptual model of evidence-based practice implementation in public service sectors. Adm Policy Ment Health. 2011;38:4–23.
Ellen M, Leon G, Bouchard G, Lavis J, Ouimet M, Grimshaw J. What supports do health systems organizations have in place to facilitate evidence-informed decision-making? A qualitative study. Implement Sci. 2013;8:84.
Flodgren G, Parmelli E, Doumit G, Gattellari M, O’Brien M, Grimshaw J, Eccles M. Local opinion leaders: Effects on professional practice and healthcare outcomes. Cochrane Database Syst Rev. 2011;(8):CD000125. doi:10.1002/14651858.CD000125.pub4.
Oliver K, Innvar S, Lorenc T, Woodman J, Thomas J. A systematic review of barriers to and facilitators of the use of evidence by policymakers. BMC Health Serv Res. 2014;14:2.
Ivers N, Jamtvedt G, Flottorp S, Young J, Odgaard-Jensen J, French S, O’Brien M, Johansen M, Grimshaw J, Oxman A. Audit and feedback: effects on professional practice and healthcare outcomes. Cochrane Database Syst Rev. 2012;6:CD000259. doi:10.1002/14651858.CD000259.pub3.
Quinn E, Huckel-Schneider C, Campbell D, Seale H, Milat A. How can knowledge exchange portals assist in knowledge management for evidence-informed decision making in public health? BMC Public Health. 2014;14:443.
Weiner BJ, Belden C, Bergmire D, Johnston M. The meaning and measurement of implementation climate. Implement Sci. 2011;6:78–89.
Pfeffer J, Salancik G. The external control of organizations: a resource dependence perspective. Stanford: Stanford University Press; 2003.
DiMaggio P, Powell W. The iron cage revisited: institutional isomorphism and collective rationality in organizational fields. Am Sociol Rev. 1983;48:147–60.
Meyer J, Rowan B. Institutionalized organizations: formal structure as myth and ceremony. Am J Sociol. 1977;83:340–63.
Bass B, Avolio B. Full range leadership development: manual for the multifactor leadership questionnaire. Redwood City: Mind Garden Inc; 1997.
Fiedler F. A contingency model of leadership effectiveness. Adv Exp Soc Psychol. 1964;1:149–90.
Bourgeois L. Strategy and environment: a conceptual integration. Acad Manage Rev. 1980;5:25–39.
Jemison D. The importance of boundary spanning roles in strategic decision making. J Manag Stud. 1984;21:131–52.
Hargadon A. Brokering knowledge: linking learning and innovation. Res Organ Behav. 2002;24:41–85.
Innvaer S, Vist G, Trommald M, Oxman A. Health policy-makers’ perceptions of their use of evidence: a systematic review. J Health Serv Res Policy. 2002;7:239–44.
Ward V, House A, Hamer S. Knowledge brokering: the missing link in the evidence to action chain. Evid Policy. 2011;5:267–79.
Buckley H, Tonmyr L, Lewig K, Jack S. Factors influencing the uptake of research evidence in child welfare: a synthesis of findings from Australia, Canada, and Ireland. Child Abuse Rev. 2014;23:5–16.
Provan KG, Beagles J, Mercken L, Leischow S. Awareness of evidence-based practices by organizations in a publicly funded smoking cessation network. J Public Adm Res Theory. 2012;23:133–53.
Shera W, Dill K. Promoting evidence-informed practice in child welfare in Ontario: progress, challenges, and future directions. Res Soc Work Pract. 2012;22:204–13.
Stevens M, Liabo K, Frost S, Roberts H. Using research in practice: a research information service for social care practitioners. Child Fam Soc Work. 2005;10:67–75.
Lavis J, Robertson D, Woodside J, McLeod C, Abelson J. How can research organizations more effectively transfer research knowledge to decision makers? Milbank Q. 2003;81:221–48.
Mays GP, Hogg R, Castellanos-Cruz D, Hoover A, Fowler L. Public health research implementation and translation: evidence from practice-based research networks. Am J Prev Med. 2013;45:752–62.
LeRoux K, Wright N. Does performance measurement improve strategic decision making? Findings from a national survey of nonprofit social service agencies. Nonprofit Volunt Sect Q. 2010;39:571–87.
Bakken S. An informatics infrastructure is essential for evidence-based practice. J Am Med Inform Assoc. 2001;8:199–201.
Bakken S, Currie L, John R. Information and communication technology infrastructure for evidence-based practice. In: Holzemer W, editor. Improving health through nursing research. West Sussex: Wiley-Blackwell; 2010.
Syed-Ikhsan S, Rowland F. Knowledge management in a public organization: a study on the relationship between organizational elements and the performance of knowledge transfer. J Knowl Manag. 2004;8:95–111.
Mithas S, Ramasubbu N, Sambamurthy V. How information management capability influences firm performance. MIS Q. 2011;35:237–56.
White K, Griffith JH. The well-managed health care organization. 7th ed. Chicago: Health Administration Press; 2010.
Glisson C, James LR. The cross-level effects of culture and climate in human service teams. J Organ Behav. 2002;23:767-94.
Schneider B, Ehrhart M, Macey W. Organizational culture and climate. Annu Rev Psychol. 2013;64:361–88.
Tolbert P. Institutional environments and resource dependence: sources of administrative structure in institutions of higher education. Adm Sci Q. 1985;30:1–13.
Hillman A, Withers M, Collins B. Resource dependence theory: a review. J Manag. 2009;35:1404–27.
Oliver C. Determinants of interorganizational relationships: integration and future directions. Acad Manage Rev. 1990;15:241–65.
Lee M. Children and family service agencies’ experience with the initial Council on Accreditation process: an exploratory study. Families in Society: The Journal of Contemporary Social Services. 2013;94:23–30.
Thyer B. What is evidence-based practice? In: Roberts A, Yeager K, editors. Foundations of evidence-based social work practice. New York: Oxford University Press; 2006.
Saufl N, Fieldus M. Accreditation: a ‘voluntary’ regulatory requirement. J Perianesth Nurs. 2003;18:152–9.
Ulrich D, Barney J. Perspectives in organizations: resource dependence, efficiency, and population. Acad Manage Rev. 1984;9:471–81.
Sharfman M, Wolf G, Chase R, Tansik D. Antecedents of organizational slack. Acad Manage Rev. 1988;13:601–14.
Kimberly JR. Organizational size and the structuralist perspective: a review, critique, and proposal. Adm Sci Q. 1976;21:571–97.
Nutley S, Walter I, Davies H. Using evidence: how research can inform public services. Bristol: Policy Press; 2007.
Squires J, Estabrooks C, Gustavsson P, Wallin L. Individual determinants of research utilization by nurses: a systematic review update. Implement Sci. 2011;6:1.
Thompson J. Organizations in action. New York: McGraw-Hill; 1967.
Aldrich H, Herker D. Boundary spanning roles and organization structure. Acad Manage Rev. 1977;2:217–30.
Banaszak-Holl J, Nembhard I, Taylor L, Bradley EH. Leadership and management: a framework for action. In: Burns L, Bradley E, Weiner B, editors. Shortell & Kaluzny’s health care management: organization design and behavior. 6th ed. Clifton Park: Delmar; 2012.
Creswell J, Plano Clark V. Designing and conducting mixed methods research. 2nd ed. Thousand Oaks: Sage; 2011.
McBeath B, Collins-Camargo C, Chuang E. Portrait of private agencies in the child welfare system: principal results from the National Survey of Private Child and Family Serving Agencies, Report by the National Quality Improvement Center on the Privatization of Child Welfare Services. Funded by the U.S. Health and Human Services Administration for Children and Families, Children’s Bureau. 2011.
Collins-Camargo C, Armstrong M, McBeath B, Chuang E. Promoting cross-sector partnerships in child welfare: qualitative results from a five-state strategic planning process. Child Welfare. 2013;92:33–63.
Collins-Camargo C, Chuang E, McBeath B, Bunger A. Private child welfare agency managers’ perceptions of the effectiveness of different performance management strategies. Child Youth Serv Rev. 2014;38:133–41.
Chuang E, Dill J, Morgan J, Konrad TR. A configurational approach to the relationship between high performance work practices and frontline health care worker outcomes. Health Serv Res. 2012;47:1460–81.
Chuang E, Wells R. The role of inter-agency collaboration in facilitating receipt of behavioral health services for youth involved with child welfare and juvenile justice. Child Youth Serv Rev. 2010;32:1814–22.
McBeath B, Briggs HE, Aisenberg E. The role of child welfare managers in promoting agency performance through experimentation. Child Youth Serv Rev. 2009;31:112–8.
Chuang E, Wells R, Alexander J, Green S. How outpatient substance abuse treatment unit director activities may affect provision of community outreach services. Drugs: Education, Prevention, and Policy. 2013;20:149–59.
Litwin M. How to assess and interpret survey psychometrics. 2nd ed. Thousand Oaks: Sage Publications; 2002.
Rihoux B, Ragin C, editors. Configurational comparative methods: qualitative comparative analysis (QCA) and related techniques. Thousand Oaks: Sage Publications; 2009.
Ragin C. Redesigning social inquiry: fuzzy sets and beyond. Chicago: University of Chicago Press; 2008.
Schneider C, Wagemann C. Set-theoretic methods for the social sciences: a guide to qualitative comparative analysis. Cambridge: Cambridge University Press; 2012.
Thygeson N, Solberg L, Asche S, Fontaine P, Pawlson L, Scholle S. Using fuzzy set qualitative comparative analysis (fs/QCA) to explore the relationship between medical ‘homeness’ and quality. Health Serv Res. 2012;47:22–45.
Kane H, Lewis MA, Williams P, Kahwati L. Using qualitative comparative analysis to understand and quantify translation and implementation. Transl Behav Med. 2014;4:201–8.
StataCorp. Stata statistical software: release 13. College Station: StataCorp LP; 2013.
Longest KC, Vaisey S. Fuzzy: a program for performing qualitative comparative analysis (QCA) in Stata. Stata J. 2008;8:79–104.
Patton M. Qualitative research and evaluation methods. Thousand Oaks: Sage Publications; 2002.
Krueger R, Casey M. Focus groups: a practical guide for applied research. 4th ed. Thousand Oaks: Sage Publications; 2009.
Bowen G. Document analysis as a qualitative research method. Qual Res J. 2009;9:27–40.
Bazeley P, Jackson K. Qualitative data analysis with NVivo. 2nd ed. London: Sage; 2013.
Yin R. Case study research. 3rd ed. Thousand Oaks: Sage Publications; 2003.
Miles MB, Huberman AM. Qualitative data analysis: an expanded sourcebook. Thousand Oaks: Sage Pubns; 1994.
Gough D. Research for practice in child neglect. In: Taylor J, Daniel B, editors. Neglect: practice issues for health and social care. London: Jessica Kingsley; 2004. p. 43–56.
Kerner J. Knowledge translation versus knowledge integration: a funder’s perspective. J Contin Educ Heal Prof. 2006;26:72–80.
Kitson A, Rycroft-Malone J, Harvey G, mcCormack B, Seers K, Titchen A. Evaluating the successful implementation of evidence into practice using the PARIHS framework: theoretical and practical challenges. Implement Sci. 2008;3:1.
Glasgow R, Emmons K. How can we increase translation of research into practice? Types of evidence needed. Annu Rev Public Health. 2007;28:413–33.
Curran G, Bauer M, Mittman B, Pyne J, Stetler C. Effectiveness-implementation hybrid designs: combining elements of clinical effectiveness and implementation research to enhance public health impact. Med Care. 2012;50:217–26.
Aarons GA, Green A, Palinkas L, Self-Brown S, Whitaker D, Lutzker J, Silovsky J, Hecht D, Chaffin M. Dynamic adaptation process to implement an evidence-based child maltreatment intervention. Implement Sci. 2012;7:32.
Collins-Camargo C, McBeath B, Ensign K. Privatization and performance-based contracting in child welfare: recent trends and implications for social service administrators. Adm Soc Work. 2011;35:494–516.
McBeath B, Collins-Camargo C, Chuang E. The role of the private sector in child welfare: historical reflections and a contemporary snapshot based on the National Survey of Private Child and Family Serving Agencies. J Publ Child Welfare. 2012;6:459–81.
Milward HB, Provan KG. Managing the hollow state: collaboration and contracting. Public Manag Rev. 2003;5:1–18.
Smith S, Lipsky M. Nonprofits for hire: the welfare state in the age of contracting. Cambridge: Harvard University Press; 1993.
Estabrooks C, Squires J, Strandberg E, Nilsson-Kajermo K, Scott S, Profetto-McGrath J, Harley D, Wallin L. Towards better measures of research utilization: a collaborative study in Canada and Sweden. J Adv Nurs. 2011;67:1705–18.
Weiss C, Murphy-Graham E, Birkeland S. An alternate route to policy influence: how evaluations affect D.A.R.E. Am J Eval. 2005;26:12–30.
Palinkas L, Garcia A, Aarons GA, Finno-Velasquez M, Holloway I, Mackie T, Leslie LK, Chamberlain P. Measuring use of research evidence: the Structured Interview for Evidence Use (SIEU). Res Soc Work Pract. 2014;26:550-64.
McBeath B, Austin M. The organizational context of research-minded practitioners: challenges and opportunities. Res Soc Work Pract. 2015;25:446–59.
Squires J, Estabrooks C, Newburn-Cook C, Gierl M. Validation of the conceptual research utilization scale: an application of the standards for educational and psychological testing in healthcare. BMC Health Serv Res. 2011;11:107.
Adams T, Heeringa SG. Outpatient substance abuse treatment survey: technical documentation for OSATSS-5, 1999–2000. Ann Arbor: Institute for Social Research; 2001.
Heeringa SG. Outpatient drug abuse treatment studies: technical documentation. Ann Arbor: Institute for Social Research, University of Michigan; 1996.
Lizarondo L, Grimmer-Somers K, Kumar S. A systematic review of the individual determinants of research evidence use in allied health. J Multidiscip Healthc. 2011;4:261–72.
We would like to thank the directors and staff of the associations for private child and family serving agencies for their partnership and support on this project: Bernadette Bianchi, Mary Chant, Cathleen Graham, Linda Hall, Polina Makievsky, Phyllis Richards, Michelle Sanborn, Carroll Schroeder, and Lynn Thull. We would also like to thank our expert panel members for input provided on survey instruments and interview protocols: Valerie Hicks, Mary Hollie, Gregory Kurth, Mike Shaver, and Kara Teeple. Finally, we appreciate our partnership with Kristen Lwin and her team at Practice and Research Together related to dissemination efforts.
This work is supported by the William T Grant Foundation (ID #186134). The views presented here are solely the responsibility of the authors and do not necessarily represent the official views of the William T Grant Foundation.
Availability of data and materials
EC, CC, and BM are responsible for study conception and design and for drafting and critical review of the manuscript. All authors read and approved the final manuscript.
The authors declare that they have no competing interests.
Consent for publication
Ethics approval and consent to participate
This study, including procedures for obtaining informed consent, was approved by the Institutional Review Board at the University of California Los Angeles (IRB #16-000579). Informed consent will be obtained from study participants prior to data collection.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Chuang, E., Collins-Camargo, C. & McBeath, B. Organizational supports used by private child and family serving agencies to facilitate evidence use: a mixed methods study protocol. Implementation Sci 12, 49 (2017). https://doi.org/10.1186/s13012-017-0580-1
- Children and adolescents
- Mixed methods
- Evidence use
- Organizational supports