METHODS

We based our methodology on that of the national Confidential Enquiry into Stillbirth and Death in Infancy (CESDI),¹ which has been operational since 1992. In CESDI, a panel of representative experts reviews a defined set of cases and identifies notable aspects of suboptimal care. However, we wished to evaluate only the postnatal (paediatric) aspects of care, to include other cases than deaths, and to identify factors notable for excellence, as well as those suggesting suboptimal care. We therefore developed a novel methodology, based on confidential enquiry, which allowed us to capture the full spectrum of quality.

We chose the following index (“sentinel”) cases:

• Neonatal deaths (0–27 postnatal days)

• Neonatal seizures

• Babies requiring transfer for specialist care at a referral hospital

After approval by the multicentre research ethics committee, we evaluated 87 anonymised cases of babies born between 1 February 1999 and 31 March 2000. From the three smaller units, we took all available cases, whereas from the largest unit we selected every fifth case chronologically that matched the criteria, to give comparable numbers. Eight dimensions of care, listed in table 2, were each rated on an ordinal scale of 0–4 from seriously deficient to exemplary. The dimension of “transfer” relates to the preparation of the baby for transfer, not the transport itself. A copy of the proforma, including definitions for each point on the scale, is available on the website (www.archdischild.com/supplemental). Case notes were assessed by four raters, supplied by four of the participating units. The notes were fully anonymised in relation to the names of the personnel making the clinical notes, the hospitals from which the notes came, and the identities of the babies. To maximise uniformity of rating, there was a “calibration” meeting using dummy cases drawn from a hospital that did not participate in the comparison. The factors, and their grading, were recorded on the proforma and all documents returned centrally for analysis.

If predefined standards of concordance were not met, the notes were rated by a panel independent of any of the participating hospitals, and this rating was taken as definitive. The criteria were either three or more points discordance on any one dimension, or four or more dimensions discordant by 2 points. The panel consisted of one of each of the following disciplines from the former Northern Region and one from elsewhere: paediatrician, neonatologist, neonatal nurse/midwife, an ANNP; and one independent chairperson. There were at least eight people present for all panel meetings. The panel met six times and deliberated on 26 cases (28% of those assessed).

The five comparator hospitals each had different models of neonatal care (table 1), but all were based on hierarchies of medical staff. They included a regional centre, two district general hospitals within the former Northern Region, one district general hospital from the south of England, and one from Scotland.

We could not perform a prior power calculation because we would have needed knowledge of the likely spread of scores and a view as to what might constitute a relevant difference. As neither of these could be satisfied, the basis of the investigation was that, if no difference between different structures of neonatal care could be shown—on the proposed dimensions—when they were subjected to the intense scrutiny of confidential enquiry, then there would be no reason on quality grounds to prefer one model of healthcare delivery over another.

Table 1 shows the characteristics of the participating hospitals; it can be seen that the comparator hospitals encompassed a broad spectrum of models of neonatal care. A paediatric service on site means the 24 hour availability of junior paediatricians but does not necessarily imply that maternity services are on the same campus as the main general hospital facilities. “Full 2nd on call cover” means the 24 hour availability, seven days a week, of a specialist registrar, “senior” senior house officer, or staff grade doctor. It was “partial” for hospital B in that, although there were middle grade staff, they were not available 24 hours, seven days a week.

RESULTS

A total of 87 cases were examined. The arithmetic means of the scores were taken for each relevant dimension of care for each case, and the mean for each case was averaged for each hospital. Table 2 shows the means for each dimension and for each hospital. There were too few cases involving terminal care to allow meaningful comparison, so the data are shown for completeness, but have not been included in the means at the foot of the table, which are therefore based on the other seven dimensions. Babies given terminal care do not equate to the total hospital figures for neonatal death, for either Ashington or any other unit. Transfers varied between hospitals, with those performing intensive care transferring the least, so the overall means are shown both with and without this dimension included. It can be seen that the rankings of the hospitals, including Ashington, are unaffected by the inclusion or exclusion of this dimension. The lowest score for any single case was 0.49 and the highest 3.07; neither case was from Ashington. The mean overall score for the five comparator hospitals was 2.22, while that of Ashington was 2.37, and overall Ashington ranked second out of the six hospitals.

Although not a formal outcome of this study, perinatal mortality and neonatal mortality data for Ashington and the former Northern Region are shown in table 3. For Northumberland as a whole, the population based perinatal mortality rate fell from a three year average of 10.7 (1995–1997) to 8.1 (1998–2000) against a regional background of 9.3 to 8.6 respectively per 1000 births.²

DISCUSSION

We have shown that, over a range of dimensions, the quality of care delivered by ANNPs at Ashington compared favourably with the full spectrum of models of medical care in a variety of locations and settings. We are confident that the service is safe in terms of mortality, as rates of both perinatal and neonatal death, for Ashington and for Northumberland as a whole, compare favourably with the regional averages, and have decreased over the time scale in question.

We wished to assess the quality of the process of care because this is what all babies experience if they pass through the neonatal unit. However, we did not include “routine” cases, where arguably it is easiest to deliver good quality care, but preferred to find those that may be thought most likely to test the system, hence our choice of “sentinel” cases. We recognise that differential patterns of in utero referral will have influenced the case mix, skewing the proportion of sick babies toward the two hospitals (D and E) undertaking neonatal intensive care. However, Ashington and the smaller comparator hospitals (A, B, and C) had similar thresholds for in utero transfer, relating more to the level of their own facilities rather than the presence of medical staff. Even with the most stringent referral policies, very preterm babies are born unexpectedly in smaller units, and both Ashington and the smaller comparator hospitals had to cope with these, as is evident from the gestation and birthweight data in table 2. More mature babies can become unexpectedly unwell in any setting.

We chose to use confidential enquiry as the tool for assessing quality of care because it has become well established through the Confidential Enquiry into Stillbirth and Death in Infancy (CEMACH: www.cemach.org.uk, and it is therefore familiar to those working in a maternity service; it has also been used to assess quality of care in other settings.³ However, the traditional CESDI methodology¹ used in maternity services focuses on adverse events, and grades care on a scale from 0 (no suboptimal care) to 3 (suboptimal care where “different management would reasonably have been expected to alter the outcome”). This perspective has the obvious limitation that it misses the opportunity of capturing a full spectrum of quality, from excellent to very poor, which we wished to ascertain; hence our modification of the CESDI methodology. We also departed from CESDI methodology in using individual raters, to make more manageable the logistics of undertaking scrutiny of a large number of cases. This allowed us to confine the independent panel meetings to those cases on which the individual raters were discordant, and it will be of interest to others who may wish to use this methodology to see that the panel needed to review over a quarter of the cases scrutinised.

The main drawbacks of using confidential enquiry to study quality are the possibility of inconsistency between the raters in scoring each dimension, the lack of explicit criteria or norms for what constitutes “quality”, and the reliance on the written record for assessing what actually happened. There is also the difficulty of maintaining complete blindness to the hospital where the notes originated. We tried to address consistency and criteria in two ways. We gave the raters practice using other cases to calibrate their rating against each other, and we developed a structured approach to the assessment of each dimension, modelled on that originally used in the CESDI 27/28 week case-control study.⁴ However, the judgments about quality on each dimension remained, in the end, those of the rater’s own expertise. For this reason, we also engaged the reference panel of independent experts, with the same spread of disciplines as the raters themselves, who could arbitrate when there was discordance between the raters. The limitations of the necessary reliance on written records are well recognised, but poor record keeping is in itself an important quality indicator, although we chose not to try to assess it as a separate dimension in this study. Blinding is subject to the same limitations as in other confidential enquiry studies, and we acknowledge this as a potential weakness. Rater bias can never completely be excluded, but we examined the raters’ data and the reference panel results closely, and we are confident that we have no evidence of such bias.

One of the strengths of this study was the spectrum of comparator hospitals, although our inability to identify them in this paper (because of confidentiality) may also be seen as a limitation. At the time the study was conceived, there was no consensus about the “best” way of delivering neonatal care to a population. The criteria recently published by the British Association of Perinatal Medicine (BAPM)⁵ represent a consensus view on standards rather than a blueprint for service configuration, and a variety of models of provision may allow the standards to be met. In planning this study, we took the view that it was essential to set the performance of the ANNPs at Ashington in the context of a wide range of types of provision, and to ensure that more than one of the comparator units was from outside the Northern & Yorkshire Region. We believe that this adds a measure of generalisability to our findings, which offsets the fact that the former Northern Region has a long standing managed clinical network for neonatal intensive care that is different from the rest of England.

Recent UK studies have compared neonatal nurse practitioners with junior paediatric staff in the routine examination of the newborn in a general hospital⁶ and for neonatal resuscitation in a tertiary neonatal intensive case setting.⁷ Both these “head to head” studies found that ANNPs performed at least as well as junior doctors, but neither were able, with their more limited remits, to address wider questions of quality in relation to the more complex areas of practice that we examined. In contrast, we set out to evaluate the whole service package, and, although our method was very labour intensive, we hope it will encourage others to use confidential enquiry to measure the quality of overall care.

What are the factors that enable the ANNPs at Ashington to deliver good quality care? The practitioners do not change every six months as junior doctors do, and consequently have developed long term team relationships with their midwifery and senior obstetric colleagues. They work within well developed guidelines. They are supported by the consultants from the regional centre in Newcastle, one of whom has specific responsibility for Ashington in any given month, yet they have a high degree of professional autonomy. They engage in continuing professional development, including neonatal life support courses, and can therefore demonstrate proficiency and competency independently of evaluations such as this. Finally, the delivery rate provides sufficient workload to allow both skill retention and the steady acquisition of further experience, which we believe would not necessarily be the case for a unit delivering less than a thousand babies a year.

In conclusion, we have shown that quality of care can be assessed in a robust manner by means of confidential enquiry. When applied to the model of care provided at Ashington, we have shown that the quality falls in the upper half of the spectrum for models of neonatal provision that are typical in the United Kingdom at present. This implies that it is not always necessary to colocate maternity services with medical paediatric services, and that good quality maternity services do not have to rely on resident junior paediatric doctors.

CONTRIBUTORSHIP

MPWP, together with the Evaluation Group, conceived the study, supervised its execution, and wrote the paper. KB undertook the data collection and analysis, and wrote the paper.