Skip to main content

Table 1 Steps for the decision sampling framework and an illustrative case study

From: The decision sampling framework: a methodological approach to investigate evidence use in policy and programmatic innovation

Methodological approach

Decision sampling framework

Illustrative case study

1. Identify the policy domain and scope of inquiry

a. Articulate the system-wide innovation or policy of explicit interest

Decisions sampled included related to the development, implementation, or sustainment of universal screening programs that sought to identify and treat the trauma of children entering foster care

b. Identify the scope of the decisions relevant to the policy or programmatic domain of interest

The scope included decisions about system-wide interventions that sought to identify and treat trauma of children and adolescents entering foster care.

2. Select appropriate qualitative method and develop data collection instrument

a. Identify research paradigm and qualitative method appropriate for the research question and decision-making process

The illustrative case study engaged a post-positivist orientation to the research question posed. Due to the exploratory nature of this project, 60-min semi-structured interviews were conducted

b. Develop interview guide that explicitly crosswalks with core and relevant constructs from the decision sciences, anchoring the interview on the discrete decision point

Cross-walking with tenants of decision analysis, developed a 26 question interview guide that included questions in following domains:

• Decision points

• Choices considered

• Evidence and expertise regarding chances and outcomes

• Outcomes prioritized (which implicitly reflect values)

• Group process (including focus, formality, frequency, and function associated with the decision-making process)

• Explicit values as described by the respondent

• Trade-offs considered in the final decision

3. Identify sampling framework

a. Identify key informants who participate in decision-making processes for selected system-wide innovation or policy

Decisions were sampled from narratives provided by mid-level administrators from Medicaid, child welfare, and mental health agencies with roles developing policy for the provision of trauma-informed services for children in foster care. Our team then employed a key informant framework for sampling, asking to recruit individuals into the study who participated in recent decisions regarding efforts to build a trauma-informed system of care for children in foster care

b. Sample one or more decisions from each informant

Informants were asked to describe a recent and important decision relevant to implementation. Our approach sampled a single decision per participant

4. Conduct semi-structured interview

a. Develop and execute study-specific recruitment

Recruited study participants: The team initially contacted a public sector mid-level manager in the child welfare agency who was an expert in developing protocols for provision and oversight of mental health services for children in foster care [44]. This initial contact assisted the research team in acquiring permission for participation through the respective child welfare agency

b. Execute study-specific data collection

Semi-structured interviews were conducted over the phone by at least two trained members of the trained research team trained, including (1) a sociologist and health services researcher (TM) and (2) public health researchers (BF, AS, ER); interviews were approximately 60 min in length and were recorded with notes taken by one researcher and memos written by both researchers immediately following the interview. Of those recruited into this study, one individual declined participation. Primary data collection for the semi-structured interviews and member-checking group interviews occurred between January 2017 and August 2019

5. Conduct data analysis

a. Engage a modified framework analysis (see the “Methods” section, analytic approach)

Employed modified framework analysis, engaging the seven steps (as articulated in the Methods, analytic approach) by trained investigators (TM, AS, ER, BF), including:

• Trained analyst checked transcripts for accuracy and de-identified

• Trained analysts familiarized themselves with data, listening to recordings and reading transcripts

• Reviewed interview transcripts generating codebook and conducted interdisciplinary team reviews employing emergent and a priori codes, with coding consensus. Entered codes into a mixed methods software program, DeDoose.TM

• Developed matrix to index codes thematically

• Summarize excerpted codes for each thematic area

• Provide excerpted data to support thematic summary in the decision-specific matrices

• Systematically analyze across matrices by decision point

6. Conduct data validation

a. Engage strategies to validate analyses and reduce investigator-introduced biases (e.g., member-checking focus groups)

Conducted member-checking group interviews with a subset of decision-makers who participated in phase 1 semi-structured interviews

• Identified and recruited sample: participants were selected among the 32 decision-makers who spoke to screening and assessment in their phase 1 interviews. Respondents were selected because they brought lived experience that would facilitate assessing the utility of the model given prior experiences in relevant decision-making. A total of 8 decision-makers participated in 4 group interviews

• Developed and conducted group interview guides.

• Group interview participants were provided a standardized slide set that:

 • Introduced the purpose of the study and summarized the qualitative findings from the decision sampling framework

 • Presented a Monte Carlo simulation model and results that were developed to synthesize available information analytically and facilitate conversation

 • After presentation of the decision sampling findings, respondents were asked “Does this match your experience?”, “Do you want to change anything?”, and “Do you want to add anything?”

• Conducted analyses: each member-checking group interview transcript was analyzed following completion [45, 46]. We used an immersion-crystallization approach in which two study team members (TM, AS) listened to and read each group interview to identify important concepts and engaged open coding and memos to identify themes and disconfirming evidence [45, 46]. The study team members used open-codes to gain new insights about the synthesized findings from the group interviews. After the initial analysis and coding was complete, the researchers re-engaged the data to investigate for disconfirming evidence. Throughout this process, the researchers sought connections to identify themes through persistent engagement with the group interview text and in regular discussion with the interdisciplinary team