Objective: Because the results of a meta-analysis are used to formulate the highest level recommendation in clinical practice guidelines, clinicians should be mindful of problems inherent in this technique. Rather than reviewing meta-analysis in abstract, general terms, we believe readers can gain a more concrete understanding of the problems through a detailed examination of one meta-analysis. The meta-analysis on which we focus is that conducted by an American College of Chest Physicians/American Association for Respiratory Care/American College of Critical Care Medicine Task Force on ventilator weaning.
Data source: Two authors extracted data from all studies included in the Task Force's meta-analysis. DATA SYNTHESIS AND OVERVIEW: The major obstacle to reliable internal validity and, thus, reliable external validity (generalizability) in biological research is systematic error, not random error. If systematic errors are present, averaging (as with a meta-analysis) does not decrease them--instead, it reinforces them, producing artifact. The Task Force's meta-analysis commits several examples of the three main types of systematic error: selection bias (test-referral bias, spectrum bias), misclassification bias (categorizing reintubation as weaning failure, etc.), and confounding (pressure support treated as unassisted breathing). Several additional interpretative errors are present.
Conclusions: An increase in study size, as achieved through the pooling of data in a meta-analysis, is mistakenly thought to increase external validity. On the contrary, combining heterogeneous studies poses considerable risk of systematic error, which impairs internal validity and, thus, external validity. The strength of recommendations in clinical practice guidelines is based on a misperception of the relative importance of systematic vs. random error in science.