Table 1 Summary of multilevel study characteristics and recommendations for implementation researchers

To conduct rigorous, high-quality multilevel implementation research...

(1) Map and operationalize the specific multilevel context for defined populations and settings

1a. Create and include a list or map of contextual levels most salient to the research question(s) and population(s) under study

▪ Table 2

▪ Additional File 1

▪ Glossary

(2) Define and state the level of each construct under study

2a. For each construct, define its substantive meaning and the level at which it resides/population unit with which it is associated

2b. For each construct, provide an explanation or “mini theory” that explains why the construct is assigned to its specific level/population unit

▪ Additional File 2

▪ Glossary

(3) Describe how constructs relate to each other within and across levels

3a. Include a figure or narrative that describes the study’s theoretical model, including the level of each construct and the hypothesized relationships between constructs

3b. When hypothesized relationships cross levels, researchers should describe the processes through which higher-level antecedents influence lower-level consequents (i.e., top-down processes) or how lower-level antecedents shape higher-level consequents (i.e., bottom-up processes)

3c. Clarify each construct’s location in the study theoretical model relative to other constructs (e.g., is it an antecedent, mediator, consequent, primary or secondary endpoint, etc.)

▪ Fig. 1

▪ Additional File 3

▪ Glossary

(4) Specify the temporal scope of each phenomenon at each relevant level

4a. Provide a detailed explanation of the expected temporal dynamics within the study at each level, using visual aids as needed, to include the following:

i. When investigators expect to observe change in each relevant outcome at each relevant level (e.g., of system- or organization-level implementation strategies)

ii. How frequently and when constructs will be measured to capture these changes

iii. How changes in outcomes at different levels align with each other in the research design

iv. The theoretical rationale for these choices

▪ Additional File 4

(5) Align measurement choices and construction of analytic variables with the levels of theories selected (and hypotheses generated, if applicable)

5a. Align the levels of theory and measurement; for unit-level constructs, determine whether the construct is a global, shared, or configural property of the unit and use this to align measures with theory

5b. For shared constructs, address the following:

i. Include a specific referent that indicates who and/or what is being rated

ii. Effectively communicate these referents to participants in measurement instruments

iii. Ensure respondents who are asked to report on shared constructs can report on them, and that they are the appropriate persons to ask

iv. Provide evidence that individuals within a unit reflect (and can report on) a shared phenomenon or experience

v. When shared constructs are measured quantitatively using individual responses, aggregate the individual responses into unit-level scores of shared constructs

▪ Additional File 5

▪ Glossary

(6) Use a sampling strategy consistent with the selected theories or research objectives and sufficiently large and variable to examine relationships at requisite levels

6a. Design and justify a multilevel sampling plan, ensuring there is the following:

i. A large enough sample at each level to rigorously test hypotheses or make theory-based inferences

ii. Adequate variability within the sample at each level to rigorously test hypotheses or make theory-based inferences

iii. Adequate representativeness of the achieved sample at each level (for quantitative)

6b. When reporting study findings for quantitative studies, include the following:

i. The distribution and range of within-unit sample sizes

ii. The distribution and range of within-unit response rates

iii. A comparison of the characteristics of unit members who responded versus those who did not respond

iv. The theoretical or empirical rationale for exclusion of units (as applicable)

▪ Additional File 6

▪ Glossary

(7) Align analytic approaches with the chosen theories (and hypotheses, if applicable), ensuring that they account for measurement dependencies and nested data structures

7a. Directly acknowledge dependencies (i.e., correlated observations/nesting) within the proposed study design, articulate what analytic method has been selected to account for those dependencies, and provide a rationale for the choice of analytic method with reference to specific characteristics of the data and strengths of the selected method/model

7b. For quantitative, ensure that variables enter statistical models at the level warranted and scrutinize choices related to centering, standardization, and calculation of effect sizes to confirm they reflect the study’s multilevel design; for randomized studies, the variable representing randomization to condition (i.e., exposure) should enter the statistical model at the level at which randomization occurs

7c. Be transparent and thorough in reporting details of the analytic approach

7d. Consider developing and sharing crosswalks that specify research questions and justify the use of data collection tools and their accompanying analytic techniques, defining their multilevel purpose and (anticipated) contributions, including “explicit connections” or “intentional redundancies” among quantitative and qualitative approaches

7e. Consider making final analytic tools accessible to end users of multilevel research reports (e.g., qualitative interview guides, statistical code)

▪ Additional File 2

▪ Additional File 7

▪ Glossary

(8) Ensure inferences are made at the appropriate level

8a. Carefully craft and check language within research reports and presentations to ensure atomistic and ecological fallacies are not present

▪ Additional File 8

▪ Glossary