Survival of very low birthweight infants (VLBWI) depends on birth weight (BW) and gestational age (GA), but also on other perinatal factors and physiological conditions of the individual infants, in particular disease severity in the first hours of life.¹ Illness severity scores were thus developed with the aim of quantifying the clinically obvious fact that infants of the same GA and BW differ in their risk of dying.^2,3 Although the initial goal of computing the risk of death for individual subjects has not been realised,⁴ illness severity scores are increasingly used to allow fair comparisons of outcome across different hospitals by “adjusting outcome rates for initial severity of the illness”⁴—that is, for risk adjustment. The availability of reliable and valid instruments to measure the severity of illness may allow the conduction of unbiased comparisons in benchmarking⁵ and quality of care studies. Moreover, they can better define populations of neonates within clinical trials, outcome evaluations, or resource utilisation studies.⁶

CRIB (clinical risk index for babies)⁷ and SNAPPE (score for neonatal acute physiology—perinatal extension)⁸ are the most commonly used scores, and their performance has been extensively validated.^2,3,6,9–12 However, both scores have some limitations and were developed almost a decade ago, before widespread use of surfactant and antenatal steroids, when mortality was higher.

SNAPPE⁸ (developed and mainly used in the United States and Canada) can be applied to neonates of all BW and all GA, whereas CRIB⁷ (developed in the United Kingdom and mainly used in Europe) can only be applied to VLBWI. CRIB, on the other hand, has the disadvantage of using some data that can be determined by the clinician. Apart from this, published data on which score is best are conflicting. Only three studies have compared CRIB and SNAPPE by applying them to the same set of neonates. The first was on 222 VLBWI admitted to a single neonatal intensive care unit (NICU) in Finland,¹³ and concluded that CRIB is superior to SNAPPE in its ability to predict in-hospital death. The second was on 476 VLBWI in eight units in the United States,⁶ and found non-significant differences between the two scores, with SNAPPE being only slightly better. The third was a retrospective study on 280 VLBWI in two NICUs in Sweden, and found similar results for the two scores.¹⁴

Both CRIB and SNAPPE have recently been updated. In 2001 a revised and shortened version of SNAPPE, called SNAPPE-II, was published,¹⁵ which uses only nine items collected over 12 hours from admission instead of the original 34 collected over 24 hours. In 2003, CRIB was also updated, using only data (five items) available up to one hour from admission.¹⁶

The aim of this study was to compare the ability of CRIB, CRIB-II, and SNAPPE-II to predict in-hospital mortality in a cohort of VLBWI admitted to 12 NICUs participating in a regional network in Italy in 1999–2001.

PATIENTS AND METHODS

The study was a prospective study planned to compare CRIB with SNAPPE-II; after publication of the CRIB-II score, relevant data were analysed and an assessment of CRIB-II was included. A cohort of infants (BW < 1500 g, GA 23–32 weeks), admitted to the 12 level III NICUs in Lombardy (Northern Italy) that participated in the Network Neonatale Lombardo in 1999–2001, were included in the analysis. Neonates of GA < 23 weeks or BW < 400 g, those with lethal congenital anomalies, those who died in the delivery room or were moribund on admission to the NICU (arbitrarily defined as dying within 10 hours of life and receiving only comfort care), and late admissions (> 12 hours from birth) were excluded.

The outcome measure was in-hospital death. Neonates transferred from participating NICUs were tracked to ascertain their outcome; those transferred after 20 days of life or transported back to level I or level II units were considered to be alive. The outcome of two transferred infants remained unknown, and they were excluded from the analysis. The final sample was 720 VLBWI.

Calculation of scores

Data used to calculate the scores were prospectively collected, abstracted from charts by one or two trained observers at each centre, and recorded on a customised web database (www.neonatalnet.org) as part of a regional neonatal network. CRIB⁷ was calculated from six items: BW; GA; highest and lowest fractional inspired oxygen (Fio₂) needed to keep a normal arterial oxygen saturation (88–95%) excluding the delivery room; worst base excess; congenital anomalies. The data collection window was the first 12 hours of life. SNAPPE-II¹⁵ was calculated from nine items: BW; being small for GA (SGA); Apgar at five minutes; urine output; lowest mean blood pressure; worst Pao₂/Fio₂ ratio; lowest pH; occurrence of seizures; lowest temperature. The data collection window was the first 12 hours after admission to the NICU. SNAPPE-II data were abstracted following the original description of SNAPPE⁸ (data collection was started in 1999, when only abstracts on SNAPPE-II had been published); weights were given to the values after SNAPPE-II publication. To minimise errors in data collection, original values were recorded, and coding and scores were calculated by computer.¹⁷

Following the original SNAPPE recommendations,⁸ items not recorded in charts were treated in the score calculation as a normal value. For CRIB, the only item that could be missing was worst base excess: in this case, following the same line of reasoning and CRIB description,⁷ a normal value was assumed.

CRIB-II¹⁶ is calculated from five items: sex; BW; GA; worst base excess; temperature at admission. Given that CRIB-II was published after collection of the data, missing data were not assumed to be normal and the case was deleted. Thus 720 neonates were available for the three way comparison.

Statistical methods

Discrimination—that is, the ability of the scores to correctly predict life or death—was assessed by calculating receiver operating characteristic curves and their associated area under the curve (AUC).¹⁸ An AUC value of 0.5 indicates no ability to discriminate, and larger values indicate increasing ability. A value of 0.8 is considered good.

As a model could have a high AUC, yet systematically overestimate or underestimate risk for some groups of infants, the Hosmer-Lemeshow (HL) test¹⁹ was used to measure the goodness of fit of the models. This test compares observed and expected mortality across several strata (usually 10) of the score. A non-significant p value of the HL statistic indicates a model with a constant discriminatory ability across strata.

The effect of adding other variables to the scores was assessed with multiple logistic regression models, using in-hospital death as the dependent variable, and calculating as above the AUC and the HL test. Because of the sampling method used (cluster sampling), methods that allow robust estimation of variance were used.

All calculations were carried out with the statistical package Stata 7 (College Station, Texas, USA).

RESULTS

Table 1 gives the basic characteristics of the neonates.

The median CRIB value was 1 (interquartile range (IQR) 1–4) for infants who survived and 10 (7–14) for those who did not. The corresponding values were 7 (5–9) and 14 (11–16) for CRIB-II and 20 (9–33) and 56.5 (39–71) for SNAPPE-II.

Figure 1 shows the receiver operating characteristic curves comparing the discriminating ability of CRIB, CRIB-II, and SNAPPE-II. CRIB and CRIB-II showed significantly greater discrimination than SNAPPE-II (AUC 0.90 (0.015) for CRIB, 0.91 (0.016) for CRIB-II, and 0.84 (0.020) for SNAPPE-II; p < 0.0004), but had a worse goodness of fit (HL p value  =  0.045 for CRIB, 0.04 for CRIB-II, and 0.52 for SNAPPE-II).

Exclusion of babies weighing 400–499 g (n  =  15) yielded an AUC of 0.898 for CRIB, 0.905 for CRIB-II, and 0.835 for SNAPPE-II (p  =  0.0015).

All the scores did not fully estimate the risk of death, because in addition to the scores, several other variables were significantly associated with survival in multiple logistic regression models. Antenatal steroid prophylaxis, caesarean section, singleton birth, an Apgar score ⩾ 7 at five minutes, not being SGA, and not having any congenital anomaly, were significantly associated (or tended to significance) with a better survival in VLBWI (tables 2–4). The influence of these factors was greatest with SNAPPE-II, and least with CRIB-II.

A higher GA was associated with a better survival in CRIB and SNAPPE-II models, and with a worse survival in the CRIB-II model. For the other factors, the direction and magnitude of the odds ratios were similar. Only congenital anomalies, which are included in CRIB but not in SNAPPE-II and CRIB-II, had a clearly smaller odds ratio in CRIB. For CRIB and SNAPPE-II, it must be noted that some factors are highly significant even though they are already included in the score (GA and congenital anomalies for CRIB, SGA, and Apgar score at five minutes for SNAPPE-II), probably indicating poor weighting for these items. For both scores, the addition of these simple factors substantially increased both discrimination (CRIB-based model: AUC  =  0.931; SNAPPE-II-based model: AUC  =  0.913) and goodness of fit (CRIB: HL p value  =  0.20; SNAPPE-II: p  =  0.53). For comparison, a logistic model including only these variables and BW, but without CRIB, CRIB-II, or SNAPPE-II, had an AUC of 0.907 with a good fit (HL test p value  =  0.49).

DISCUSSION

This study addressed two important questions: what are the results of one widely used (CRIB) and two new (CRIB-II and SNAPPE-II) scores in an unselected sample of VLBWI ⩽ 32 weeks in terms of discrimination, goodness of fit, and comparative performance; is risk adjustment using these scores complete, so that their use effectively removes the confounding due to severity of illness and allows unbiased comparisons between different groups of neonates.

This is the first study to compare CRIB, CRIB-II, and SNAPPE-II for their ability to correctly predict in-hospital death in VLBWI. We show that CRIB and CRIB-II performed better than SNAPPE-II; however, the addition of other perinatal factors considerably improved discrimination of all scores.

The worse SNAPPE-II performance in VLBWI cannot be ascribed to its unexpectedly poor performance, because the discriminatory ability obtained in this study for all scores was in line with published data.^6,7,13–16 In fact, in our study the scores in VLBWI were as in the original papers.^7,15,16 The cohort in which our study was carried out (infants of BW 400–1499 g and GA ⩽ 32 weeks) was slightly different from the original CRIB cohort⁷ (infants of GA < 31 weeks or BW < 1500 g) and CRIB-II cohort¹⁶ (infants of GA ⩽ 32 weeks). On the other hand, SNAPPE⁸ and SNAPPE-II¹⁵ were developed and validated on all acute neonatal admissions, irrespective of GA or BW. Although these differences should not influence the risk adjusted mortality prediction (our inclusion criteria defined a subset of those with the three scores), some SNAPPE-II items may have a lower sensitivity in VLBWI. Richardson et al¹⁵ have acknowledged that urine output is less useful in VLBWI than in bigger neonates, probably because of difficulties in obtaining complete and precise spontaneous urine collection in a relatively short timespan (12 hours from admission) in tiny babies. A recent report confirms that urine output is an unreliable item.²⁰ A similar problem is found with seizures: in our sample, only five babies had convulsions in the first 12 hours. Also this item is probably more useful in larger babies, where admission to NICU reflects a larger variety of causes.

For CRIB and SNAPPE-II, a possible cause of reduced discrimination is the weight given to levels of physiological derangement for some items: the fact that low Apgar score and being SGA (for SNAPPE-II) and congenital anomalies and GA (for CRIB) are highly significant and increase discrimination when added to the models implies that these factors, already present, are inadequately weighted by the scores.

For SNAPPE-II, the problem could again be because the weights were derived from samples in which VLBWI represented a minority, and for CRIB because the weights were derived a decade ago, when a different mix of risk factors (with different relative importance) probably influenced mortality.

Moreover, in CRIB some items are determined by the care team. For instance, CRIB takes into account both highest and lowest appropriate Fio₂ used during the first 12 hours, and it is possible that prophylactic surfactant administration could obliterate this difference, and thus reduce CRIB performance. In this sample, 48.4% of the newborns received surfactant at any time, and only 12.5% of these in the delivery room. We cannot therefore comment on the performance of CRIB when a prophylactic surfactant strategy is used. However, the problem of using variables that can be arbitrarily determined by doctors, such as Fio₂,²¹ was one of the main reasons for developing CRIB-II, which only uses information available within one hour of admission and independent of the care provided. Our data show that the new score has a discrimination that is comparable to that of the “old” CRIB, but potentially less confounded by treatment given by the team.

A surprising finding with CRIB-II is that GA is almost significant (table 4), with death risk increasing with increasing GA. This suggests that the BW–GA–sex matrix used in CRIB-II is not well calibrated for our sample. This interpretation is strengthened by the observation that risk adjustment using all scores is imperfect, and even after accounting for “severity of illness”, other important prenatal and perinatal factors influence VLBWI survival, including antenatal steroid prophylaxis, Apgar score at five minutes, type of delivery, multiple birth, and congenital anomalies. These factors are known to influence survival in VLBWI, and a logistic model including only these variables and BW, but without CRIB, CRIB-II, or SNAPPE-II, showed excellent discrimination. CRIB-II and SNAPPE-II do not take into account congenital anomalies, but the large estimated effect on mortality (an odds ratio of 5.01 for CRIB-II and 4.92 for SNAPPE-II) when added to the model suggests that this information should be included in a severity score.²²

The imperfect adjustment has been reported before.^6,23,24 For antenatal steroid prophylaxis, Richardson et al¹⁵ argue against including it in the score, because this would excuse poor obstetrical management, and “it may obscure the ill effects of improper treatment”. This line of reasoning cannot, however, be applied to other factors which do not depend on medical decisions, such as multiple birth, congenital anomalies, or being SGA. The important contribution to risk of these factors means that the use of CRIB, SNAPPE-II, or, to a lesser degree, CRIB-II alone cannot guarantee an unbiased risk adjustment, thus partly losing the reason for using these scores.

In conclusion, this study shows that CRIB and CRIB-II were superior to SNAPPE-II in VLBWI, but their goodness of fit was not good. CRIB-II appeared to be as accurate as CRIB in this sample, and was less confounded by other factors. All scores do not fully estimate the risk of death—that is, in addition to CRIB, CRIB-II, or SNAPPE-II, other perinatal factors influence survival and should be included in any analysis to improve risk adjustment.