Table 2 Methodological status of controlled studies

Horbar et al. [84] (2004)

Randomized controlled

Randomization (computer generated), allocation concealment (investigators, prior to intervention), baseline (13 of 14 measures similar, no statistical testing), blinding (statistician), ITT (done), follow-up (100%)

Voluntary participation in collaborative: 114/178 hospitals eligible participated.

Curley et al. [102]

(1998)

Randomized controlled

Randomization (blocked), allocation concealment (NS), baseline (18 of 19 similar), blinding (NS), ITT (NS), follow-up (NS)

Used a convenience sample for one measure; controlled for potential covariates in analyses; questionable construct validity for provider satisfaction.

Carlhed et al. [26] (2006)

Controlled before

Allocation (matched then randomized), allocation concealment (controls), baseline (7 of 7 similar), blinding (controls), ITT (NS), follow-up (NS)

Intervention group hospitals self-selected, whereas control hospitals were hospitals that did not self-select; no group differences at baseline; registry had continuous monitoring; no reason to believe proposition of patients with contraindications systematically differed.

Doran et al. [56] (2002)

Controlled before

Allocation (participant preference, attempts to randomize), allocation concealment (NS), baseline (NS), blinding (external reviewers), ITT (NS), follow-up (time 1: 85%, time 2: 74%; higher control group attrition)

Selection: sample may be biased towards those who responded most quickly; measurement: unlikely, external reviewers blinded to group allocation and not part of study, reported methods to avoid bias; attrition/exclusion: differences between intervention group and those who withdrew, greater drop-out in the control group; gave description of sample, but did not compare group characteristics; performance: unlikely, analyses at team level.

Hermida and Robalino [19] (2002)

Controlled before

Allocation (matched then randomized), allocation concealment (NS), baseline (higher outcomes in intervention group), blinding (NS), ITT (NS), follow-up (NS)

Howard et al. [21] (2007)

Controlled before

Allocation (matched, wait-list control), allocation concealment (NS), baseline (2 of 6 similar - controls, 5 of 6 similar - delayed comparison), blinding (NS), ITT (NS), follow-up (NS)

Provided information on non-responders; selection: self-selection, 43/58 participated, group differences at baseline; provide evidence against regression to the mean and selection bias in the wait-list controls; no information on quality of the data source.

Bookbinder et al. [64] (2005)

Controlled before

Allocation (location - unit type), allocation concealment (NS), baseline (3 of 21 similar), blinding (NS), ITT (NS), follow-up (NS)

Measurement: no baseline data; developed tools with interrater reliability; attrition bias: short survival of patients on the oncology unit; one tool could not completed: use was limited to 50 patients on intervention unit; selection: loss to follow up on comparison unit; performance: not possible to control for extraneous variables; referral to consultation team, exposure of staff to other educational offerings, cultural and leadership styles.

Brickman et al. [27] (1998)

Controlled before

Allocation (location - hospital, unclear if 'randomization' occurred), allocation concealment (NS), baseline (NS), blinding (NS), ITT (NS), follow-up (NS)

Performance: changing processes.

Horbar et al. [23] (2001)

Controlled before

Allocation (project participation), allocation concealment (NS), baseline (9 of 9 similar), blinding (NS), ITT (NS), follow-up (attrition in control)

Selection: self-selection of institutions.

Wang et al. [113] (2003)

Controlled before

Allocation (location - unit type), allocation concealment (NS), baseline (10 of 12 similar), blinding (NS), analyses (covariates), ITT (NS), follow-up (NS)

Selection: allocated by unit type, differences between groups on baseline characteristics and outcome measures, controlled for characteristics in analyses; clinical significance of differences in question; no attrition bias; performance: likely with different unit types being compared; source of inventory data quality is not known.

Isouard [112] (1999)

Controlled before

Allocation (location - hospital), allocation concealment (NS), baseline (3 of 3 similar), blinding (NS), analyses (no covariates), ITT (NS), follow-up (NS)

Selection: well defined criteria for selection for AMI.

Cable [37] (2001)

Interrupted time series

Data points (pre - 42-47 months/data points, post 22 to 27 months/data points), blinding (NS), analyses (ARIMA, switching replication), ITT (NS), follow-up (100%)

Measurement: change in catheterization tray, which affected catheterization events.

Berriel-Cass et al. [20] (2006)

Interrupted time series

Baseline (retrospective, NS case mix; pre - 7/8 months/data points, post - 23/24 months/data points), blinding (NS), analyses (pre-post comparisons), ITT (NS), follow-up (NS)

Carey and Teeters [75] (1995)

Interrupted time series

Baseline (pre - 6 months/data points, post - 15 months/data points), blinding (NS), analyses (np charts, no inferential statistics), ITT (NS), follow-up (NS)

Selection/attrition: NA; performance/measurement: nurses may have increased reporting after training program, rather than the intervention being efficacious; unclear as to whether there was a change in intervention midway or after training program.

Harris et al. [38] (2000)

Interrupted time series

Baseline (pre - 3 years/6 data points, post - 3 years/6 data points), blinding (NS), analyses (no inferential statistics), ITT (NS), follow-up (NS)

Performance: physicians were already beginning to establish criteria before implementation; selection: no information about the sample.

Bartlett et al. [85] (2002)

Interrupted time series

Baseline (1. pre - 20 weeks/data points, post - 20 weeks/data points; 2. pre - 10 weeks/6 data points, post - 25 weeks/14 data points), blinding (NS), analyses (no inferential statistics), ITT (NS), follow-up (100%)

Selection/attrition: unlikely; measurement/performance: team-self and director-reported 'significant improvements', attempts to blind director to team identity.

Fox et al. [24] (2006)

Interrupted time series

Baseline (pre - 15 months/5 data points, post - 27 months/9 data points), blinding (NS), analyses (no inferential statistics), ITT (NS), follow-up (100%)

Time series controls for selection, but does not for history, instrumentation, and testing; no testing and instruments using review of charts; difficult to determine if there were any historical events that may have influenced results.

Allison and Toy [82] (1996)

Interrupted time series

Baseline (pre - 6 years/data points, post - 5 years/data points), blinding (NS), analyses (no inferential statistics), ITT (NS), follow-up (NS)

Measurement/instrumentation: unclear as to how some of the data was collected.

Halm et al. [40] (2004)

Cohort

Cohort (matched, separate pre- post cohorts, 30 of 37 similar), blinding (NS), ITT (NS), follow-up (NS)

Selection: acknowledges pre-post comparison of separate groupings of patients who met criteria of CAP; samples matched for age, race, sex, severity of diseases, co-morbidities, etc.

Berenholtz et al. [39] (2004)

Cohort

Cohort (different ICU types, baseline NS), blinding (NS), ITT (NS), follow-up (NS)

Selection: no description of population; may not have accounted for other confounding factors such as antibiotic use and location of catheter insertion.

Brown et al. [42] (2006)

Case-control

Cohort (prospective, case mix 3 of 4 similar, before-after comparisons), blinding (NS), analyses (regression)

Participants matched on post-data; performance: defined eras and care; selection bias: no loss to follow up, matched on most confounding variables; no masking regarding exposure and outcome.

Houston et al. [43] (2003)

Case-control

Cohort (matched - chart review, NS case mix), blinding (NS), analyses (no inferential statistics)

Bromenshenkel et al. [41] (2000)

Case-control

Cohort (chart review, NS case mix; pre-post comparisons), blinding (NS), analyses (no inferential statistics)

No information on comparability of cases and controls for confounding variables, or if data collection was masked with regard to disease status of participant.