Background The validity and applicability of before-after studies compared to randomised controlled trials (RCTs) of fluconazole prophylaxis for very low birthweight (VLBW) infants is uncertain.
Objectives The aim was to examine whether the study design (before-after studies compared to RCTs) affected the estimate of effect size yielded in meta-analyses and to explore possible causes for any differences detected.
Methods A systematic review and meta-analysis of before-after studies, which assessed the effect of fluconazole prophylaxis on the incidence of invasive fungal infection in VLBW infants, was undertaken. Data were compared with estimates generated from meta-analyses of RCTs. Funnel plots were examined for evidence of publication bias.
Results Meta-analysis of 11 before-after studies found a reduced risk of invasive fungal infection following introduction of fluconazole prophylaxis: RR 0.19 (95% CI 0.13 to 0.27). This estimate is significantly lower than the estimate generated from meta-analysis of RCTs: RR 0.48 (95% CI 0.31 to 0.73). Inspection of a funnel plot of before-after studies revealed that smaller studies with large effects sizes contributed an excess of data points.
Conclusions Publication bias may be an important cause of effect size estimate inflation of before-after studies. Data from before-after studies of antifungal prophylaxis for VLBW infants should be interpreted and applied cautiously. Evidence to guide policy and practice for should instead be derived from well-designed RCTs.
Statistics from Altmetric.com
Invasive fungal infection is an important cause of mortality and morbidity in very low birthweight (VLBW) infants. Because diagnosis and treatment are often delayed, recent research attention has focused on prevention.1 Systemic antifungal prophylaxis using fluconazole has been widely studied but further evaluation of its effect is needed.2 3
The randomised controlled trial (RCT) is the ‘gold standard’ method for assessing the effect of healthcare interventions but developing and performing RCTs is a lengthy, difficult and expensive process.4 Researchers may instead opt to undertake observational or quasi-experimental studies in an attempt to address important clinical questions more cheaply and quickly. However, observational studies, especially before-after (epoch-comparison) studies using historical controls, are subject to various forms of bias that can distort the estimates of effect sizes compared with RCTs of the same intervention.5 Concern exists that over-reliance on before-after studies may inappropriately misguide practice and skew the research agenda if RCTs are considered unnecessary because of ‘overwhelming’ observational data.6 This issue is particularly relevant to the evaluation of neonatal care interventions where RCTs usually need to be multicentre and often multinational, to recruit enough participants to assess clinically meaningful outcomes.
What is already known on this topic
Before-after studies are quicker and cheaper methods of assessing interventions than randomised controlled trials (RCTs) but are subject to various biases that may result in overestimation of effect sizes.
What this study adds
Before-after studies of fluconazole prophylaxis for very low birthweight infants have significantly and substantially overestimated the effect size compared with RCTs.
Publication bias may be an important contributor to effect size inflation in meta-analysis of observational data.
With regard to whether prophylactic fluconazole prevents invasive fungal infection in VLBW infants, several RCTs and before-after studies have been undertaken but substantial uncertainty and variation in practice remains.2 3 7 8 The key question for clinicians and consumers who wish to resolve this uncertainty is—should further RCTs be undertaken or are data from before-after studies sufficiently reliable and valid to inform policy and practice? To address this question, a systematic review of before-after studies was conducted and the findings were compared with those from meta-analyses of RCTs.7 The aim was to examine whether the study design affected the estimates of effect sizes and to explore possible causes of any differences detected.
Inclusion criteria for the systematic review and meta-analyses are described in table 1.
The electronic search or Medline and EMBASE (1980–2009) used the following text words and MeSH terms: (infant, newborn OR infant, premature OR infant, very low birth weight OR infant, extremely low birth weight OR infan* OR neonat* OR VLBW OR ELBW) and (antifungal agents OR candida OR candidiasis OR fungemia OR fluconazole). No language restriction was applied. Reference lists of all potentially eligible studies and the reference lists of all RCTs identified in the Cochrane review of antifungal prophylaxis for VLBW infants were screened.7 The abstracts presented at the annual meetings of the Society for Pediatric Research, European Society for Pediatric Research and the Federation of Infection Societies between 2000 and 2009 were looked at.
For each potentially eligible study, both authors independently abstracted information on setting, design and primary outcomes (invasive fungal infection and mortality). RevMan 4.2 was used to calculate RRs with 95% CI for the individual studies and a pooled RR with 95% CI using the Mantel-Haenszel method. The statistical validity of combining the results of the studies by inspection of the forest plot was assessed for evidence of heterogeneity of the outcomes and by calculating the χ2 of the RR estimates. Publication bias by generating and inspecting a funnel plot was assessed (box 1).9 To assess statistical differences between RR estimates, RR ratios and 95% CI were calculated using the method described by Altman and Bland.10
Box 1 Funnel plot
Scatter plot of effect size versus study size and precision.
Data points for individual studies should scatter symmetrically like an inverted funnel around the pooled estimate of effect.
Asymmetry due to excess of data points from small studies with large effect sizes (independently of event rates) is suggestive of publication bias.
Eleven studies fulfilled the prespecified eligibility criteria (excluding reports that included overlapping participants).11,–,21 Four of the studies have so far been reported as conference abstracts.16,–,18 20 The study characteristics are summarised in table 2.
Most of the studies were undertaken within the past decade in neonatal intensive care centres in North America and Europe. In total, 3360 VLBW infants participated—1528 in the control phases and 1832 in the intervention phases. Most infants had additional specific risk factors for invasive fungal infection (central venous line access, broad spectrum antibiotic exposure or fungal colonisation). A total of 2201 participants were ELBW infants. In four studies, only ELBW infants were eligible for participation and in one study participation was restricted to infants of BW less than 750 g. Subgroup data for ELBW infants were provided in three other studies.
No information on baseline characteristics of the comparison groups was available for two studies. Most of the remaining studies reported that baseline demographic characteristics (at least baseline BW and gestational age (GA)) were similar in infants in both phases. Significant differences in BW and/or GA between the comparison groups were found in two studies.13 18 Only limited data on other demographic of clinical care characteristics, including exposure to putative risk factors for invasive fungal infection, were available. In one study, infants in the control era had higher ‘clinical risk index for babies’ scores and more exposure to antibiotics.14 Multivariate analysis adjusting for these factors did not alter the study findings. In two other studies, fewer infants in the control era had received antenatal steroids and surfactant.12 20 In one study, control infants had more frequent exposure to cephalosporins.21
The control and intervention phases generally lasted from one to three years. During the control era, none of the infants received systemic antifungal prophylaxis. It is unclear whether some infants in some centres received oral/topical non-absorbed prophylaxis during control eras. During the intervention eras, fluconazole dose regimens followed published schedules and varied from 3 mg/kg/dose to 6 mg/kg/dose, usually for 4–6 weeks after birth.
Invasive fungal infection was generally defined as mycological culture from a normally sterile site—usually blood, urine and cerebrospinal fluid. Only three reports specified that the samples were collected using methods to minimise contamination from surface colonising organisms (blood sampled from a peripheral site and urine sampled by suprapubic aspiration or in-out aseptic urethral catheterisation). The incidence of invasive fungal infection during the control eras of the studies ranged from 6.3% to 20.4% (mean 10.3%) compared with a range of 0–5.4% (mean 1.9%) during the intervention eras. All studies, except one,15 found a significant reduction in the incidence of invasive fungal infection during the intervention era.
Data on all-cause mortality were available from eight studies. Only one study reported a significant reduction in mortality.12 Overall, 16% of participants died during the control era compared with 13% following introduction of fluconazole prophylaxis.
Meta-analysis of the 11 before-after studies that reported data on the incidence of invasive fungal infection found a significant effect of fluconazole prophylaxis in VLBW infants (RR 0.19, 95% CI 0.13 to 0.27) and in subgroup analysis of ELBW infants (RR 0.13, 95% CI 0.08 to 0.22) (figure 1). There was no evidence of statistical heterogeneity in these meta-analyses.
Inspection of a funnel plot revealed substantial asymmetry and an excess of data points for small studies with large effect size reductions (figure 2).
Meta-analysis of eight studies which provided data found a statistically significant reduction in all-cause mortality (RR 0.80, 95% CI 0.66 to 0.97) (figure 3).
Cochrane review of RCTs
The Cochrane review of systemic antifungal prophylaxis for VLBW infants identified five RCTs in which 656 infants in total participated.7 Meta-analysis found a significant effect on the incidence of invasive fungal infection (RR 0.48, 95% CI 0.31 to 0.73; five trials) but not on all-cause mortality (RR 0.74 95% CI 0.51 to 1.09; four trials).
Comparison of effect estimates
Invasive fungal infection
The estimated RR was significantly lower in meta-analysis of before-after studies than RCTs: RRs ratio 0.40 (95% CI 0.32 to 0.49).
The estimates of effect size were not significantly different: RRs ratio 1.08 (95% CI 0.89 to 1.29).
Before-after studies have significantly overestimated the effect of fluconazole prophylaxis on the incidence of invasive fungal infection in VLBW infants compared with RCTs of the same intervention. Ten of 11 before-after studies compared with two of five RCTs found a significant beneficial effect. Meta-analysis of before-after studies suggests that VLBW infants who receive fluconazole prophylaxis have less than one-fifth the risk of acquiring invasive fungal infection. This compares with the summary RR from RCTs of about one-half.
With regard to mortality, the meta-analyses have similar point estimates. Meta-analysis of before-after studies suggested that the adoption of fluconazole prophylaxis reduced mortality by 20%. This estimate is not significantly different from the estimate generated by meta-analysis of RCTs (26% reduction). More than five times as many VLBW infants have participated in before-after studies than in RCTs. This sample size discrepancy may have contributed to the finding of a significant effect in meta-analysis of before-after studies but not of RCTs despite similar point estimates. Funnel plot analysis did not suggest publication bias, consistent with mortality being regarded as a secondary outcome measure for these studies.
Several factors are likely to have contributed to the differences between the findings of before-after studies and RCTs:
Funnel plot inspection suggests that the over-estimation of effect size may be partly the result of publication bias. Studies that show a significant effect are more likely to be submitted and accepted for publication in journals and conferences than studies that do not show any effect.22 Evidence exists that observational studies and especially before-after studies, are more likely to be affected by publication bias than RCTs.23 The requirement to undergo a priori methodological review and ethical evaluation and the introduction of initiatives such as prospective trial registries may have helped to reduce publication bias of RCTs. However, these regulatory safeguards are not generally applied to before-after studies, which are usually retrospective reviews of outcomes before and after adoption of a new intervention.
Regression towards the mean
The mean incidence of invasive fungal infection in VLBW infants during the control eras of the included studies was 10%, comparable with the incidence detected in controls who participated in RCTs,7 but much higher than the 1–4% incidence reported in population-based multicentre studies.4 5 Clinicians in those neonatal units where invasive fungal infection was more common than average may have been more inclined to be ‘early adopters’ of targeted fluconazole prophylaxis. However, because of temporal and geographical clustering of invasive fungal infection in neonatal intensive care units, it is likely that the incidence of invasive fungal infection in those units in the years following the control phase would have tended towards the mean population incidence even if no intervention was applied. This statistical phenomenon is known as regression towards the mean and contributes to the apparent effects of many interventions assessed in studies without contemporaneous controls.24
Confounding variables in historical controls
In the absence of random allocation, the apparent effect of an intervention may be related to known or unknown differences between the comparison groups. In the before-after studies the comparison groups were generally of similar BW and GA. However, other factors that may have independently affected the risk of developing invasive fungal infection differed between the groups in some studies.25 26 For example, exposure to antenatal corticosteroids and surfactant and receipt of broad-spectrum antibiotics was less common among control groups in some studies.12 14 20 21 Furthermore, for most studies it is unclear whether other possible risk factors such as exposure to H2 receptor blockers or duration of parenteral nutrition also differed. Two studies attempted to adjust for the potential confounding effects of prognostic variables and risk factors using a multivariate analysis.12 14 However, even after adjustment interpreting the findings is problematic as only known and recorded factors can be included and incomplete or inconsistent categorisation may cause residual confounding.
Other changes in infection control practices
Temporal changes in other infection control practices such as handwashing policy, staff-infant ratios and antibiotic-prescribing policies, as well as other management practices like enteral feeding strategies may contribute to varying degrees to changes in the risk of invasive fungal infection. In addition, changes in the policy and practice for investigation and diagnosis of invasive fungal infection may alter the detected incidence even of the true incidence in unchanged.
Ascertainment bias in diagnosing infection
A further specific concern with antimicrobial prophylaxis is ascertainment bias. Fluconazole prophylaxis reduces the sensitivity of microbiological culture. This issue affects both RCTs and before-after studies. For this reason, evaluation of other clinical outcomes, especially more patient-important outcomes such as mortality and adverse neurodevelopmental outcomes, is needed to reliably assess the effect of fluconazole prophylaxis. Of the eight before-after studies that reported all-cause mortality, only one study detected a significant reduction in mortality.12 However, this may be related to the fact that infants in this study were also significantly less likely to have received antenatal steroids and surfactant in the control era than during the intervention era.
Lack of intention-to-treat analysis
To increase the generalisability of RCT findings, outcomes for all participants are assessed irrespective of whether they actually received the intended intervention or not (‘intention-to-treat analysis’). For retrospective observational studies, this methodological requirement is less likely to be adhered to. If infants in the intervention era who were eligible for prophylaxis but did not receive it are omitted from outcome assessment then this may cause inflation of the effect size estimate.
This analysis suggests that the findings of before-after studies of fluconazole prophylaxis should not be regarded as sufficiently valid to inform practice as they substantially over-estimate effect sizes in part because of publication bias. Clinicians and consumers who need more reliable and robust evidence to inform their practice should instead focus efforts on developing large pragmatic RCTs.
Thanks Dr Bob Phillips and Dr Peter Fowlie for their constructive criticism.
Competing interests In 2002, the authors received a grant from Pfizer Ltd, a manufacturer of fluconazole, to support a UK national surveillance study of invasive fungal infection in VLBW infants.
Provenance and peer review Not commissioned; not externally peer reviewed.
Detail has been removed from this case description or these case descriptions to ensure anonymity. The editors and reviewers have seen the detailed information available and are satisfied that the information backs up the case the authors are making.
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.