From: Eight characteristics of rigorous multilevel implementation research: a step-by-step guide
Characteristic | Recommendations | In-paper resources |
---|---|---|
To conduct rigorous, high-quality multilevel implementation research... | ||
(1) Map and operationalize the specific multilevel context for defined populations and settings | 1a. Create and include a list or map of contextual levels most salient to the research question(s) and population(s) under study | ▪ Table 2 ▪ Additional File 1 ▪ Glossary |
(2) Define and state the level of each construct under study | 2a. For each construct, define its substantive meaning and the level at which it resides/population unit with which it is associated 2b. For each construct, provide an explanation or “mini theory” that explains why the construct is assigned to its specific level/population unit | ▪ Additional File 2 ▪ Glossary |
(3) Describe how constructs relate to each other within and across levels | 3a. Include a figure or narrative that describes the study’s theoretical model, including the level of each construct and the hypothesized relationships between constructs 3b. When hypothesized relationships cross levels, researchers should describe the processes through which higher-level antecedents influence lower-level consequents (i.e., top-down processes) or how lower-level antecedents shape higher-level consequents (i.e., bottom-up processes) 3c. Clarify each construct’s location in the study theoretical model relative to other constructs (e.g., is it an antecedent, mediator, consequent, primary or secondary endpoint, etc.) | ▪ Fig. 1 ▪ Additional File 3 ▪ Glossary |
(4) Specify the temporal scope of each phenomenon at each relevant level | 4a. Provide a detailed explanation of the expected temporal dynamics within the study at each level, using visual aids as needed, to include the following: i. When investigators expect to observe change in each relevant outcome at each relevant level (e.g., of system- or organization-level implementation strategies) ii. How frequently and when constructs will be measured to capture these changes iii. How changes in outcomes at different levels align with each other in the research design iv. The theoretical rationale for these choices | ▪ Additional File 4 |
(5) Align measurement choices and construction of analytic variables with the levels of theories selected (and hypotheses generated, if applicable) | 5a. Align the levels of theory and measurement; for unit-level constructs, determine whether the construct is a global, shared, or configural property of the unit and use this to align measures with theory 5b. For shared constructs, address the following: i. Include a specific referent that indicates who and/or what is being rated ii. Effectively communicate these referents to participants in measurement instruments iii. Ensure respondents who are asked to report on shared constructs can report on them, and that they are the appropriate persons to ask iv. Provide evidence that individuals within a unit reflect (and can report on) a shared phenomenon or experience v. When shared constructs are measured quantitatively using individual responses, aggregate the individual responses into unit-level scores of shared constructs | ▪ Additional File 5 ▪ Glossary |
(6) Use a sampling strategy consistent with the selected theories or research objectives and sufficiently large and variable to examine relationships at requisite levels | 6a. Design and justify a multilevel sampling plan, ensuring there is the following: i. A large enough sample at each level to rigorously test hypotheses or make theory-based inferences ii. Adequate variability within the sample at each level to rigorously test hypotheses or make theory-based inferences iii. Adequate representativeness of the achieved sample at each level (for quantitative) 6b. When reporting study findings for quantitative studies, include the following: i. The distribution and range of within-unit sample sizes ii. The distribution and range of within-unit response rates iii. A comparison of the characteristics of unit members who responded versus those who did not respond iv. The theoretical or empirical rationale for exclusion of units (as applicable) | ▪ Additional File 6 ▪ Glossary |
(7) Align analytic approaches with the chosen theories (and hypotheses, if applicable), ensuring that they account for measurement dependencies and nested data structures | 7a. Directly acknowledge dependencies (i.e., correlated observations/nesting) within the proposed study design, articulate what analytic method has been selected to account for those dependencies, and provide a rationale for the choice of analytic method with reference to specific characteristics of the data and strengths of the selected method/model 7b. For quantitative, ensure that variables enter statistical models at the level warranted and scrutinize choices related to centering, standardization, and calculation of effect sizes to confirm they reflect the study’s multilevel design; for randomized studies, the variable representing randomization to condition (i.e., exposure) should enter the statistical model at the level at which randomization occurs 7c. Be transparent and thorough in reporting details of the analytic approach 7d. Consider developing and sharing crosswalks that specify research questions and justify the use of data collection tools and their accompanying analytic techniques, defining their multilevel purpose and (anticipated) contributions, including “explicit connections” or “intentional redundancies” among quantitative and qualitative approaches 7e. Consider making final analytic tools accessible to end users of multilevel research reports (e.g., qualitative interview guides, statistical code) | ▪ Additional File 2 ▪ Additional File 7 ▪ Glossary |
(8) Ensure inferences are made at the appropriate level | 8a. Carefully craft and check language within research reports and presentations to ensure atomistic and ecological fallacies are not present | ▪ Additional File 8 ▪ Glossary |