Objective: To evaluate the National Congenital Anomaly System (NCAS).
Methods: The NCAS in England and Wales based at the Office for National Statistics and the various regional registers that exchange data with it were examined, based on guidelines for evaluating public health surveillance systems, published by the Centres for Disease Control (CDC). Data relating to congenital anomaly notifications received from 1991 to 2002 were analysed.
Main outcome measures: The main outcome measures were based on CDC standards and included the level of usefulness of the system, simplicity, flexibility, data quality, acceptability, sensitivity, representativeness, timeliness, and stability of the system.
Results: The NCAS has two main tiers: the “passive” system of voluntary notifications and the anomaly registers, but many reporting sources within these. It receives about 7000 notifications a year. It is inflexible and has variable data quality. The voluntary nature of reporting affects the system’s acceptability. The sensitivity as compared with two regional registers (Trent and Wales) is about 33%. The congenital anomaly registers reporting to the NCAS achieve high levels of coverage and completeness. From 2003, they cover 42% of all births and account for the major proportion of the notifications.
Conclusions: The NCAS serves the important function of monitoring birth defects in England and Wales, but is not currently operating in a timely or effective way. It should be adapted to meet its main objectives more effectively. More regional anomaly registers should be instituted and existing registers supported through central funds.
- CDC, Centres for Disease Control
- CUSUM, cumulative sum technique
- NCAS, National Congenital Anomaly System
- ONS, Office for National Statistics
- PCT, Primary Care Trust
- TOPs, terminations of pregnancy
- birth defects
- congenital anomalies
- National Congenital Anomaly System
Statistics from Altmetric.com
- CDC, Centres for Disease Control
- CUSUM, cumulative sum technique
- NCAS, National Congenital Anomaly System
- ONS, Office for National Statistics
- PCT, Primary Care Trust
- TOPs, terminations of pregnancy
The National Congenital Anomaly System (NCAS) records congenital anomalies for all births (live and stillbirths) in England and Wales.1 Congenital anomalies cause nearly a third of all deaths in infants, and one quarter of deaths within the first week of life.2 The NCAS is one of 10 national congenital anomaly surveillance systems world wide and contributes data to the International Clearinghouse for Birth Defects Monitoring Systems.3 None of these systems has been evaluated systematically, although the Australian system is currently under review.
In a recent paper, Boyd et al4 examined under-ascertainment in the NCAS by birth defect, geographical area, and deprivation. They also discussed the non-uniformity of NCAS ascertainment levels and their impact, and the implications of the lack of data on terminations resulting from fetal anomalies. The aim of our study was to evaluate whether the NCAS fulfilled its stated objectives and to what extent it conformed to the guidelines for the evaluation of surveillance systems developed by the Centres for Disease Control (CDC), Atlanta, USA. We present the results of the evaluation and a discussion on difficulties encountered in applying the guidelines to a national congenital anomaly surveillance system.
The NCAS had previously been evaluated in 1993 by a working group of the Registrar General’s Medical Advisory Committee. A report based on this evaluation was published in 1995.5 This report had made 14 recommendations to improve the working of the system. Some of these—for example, extending notification time scales, exchanging information with anomaly registers, renaming the system NCAS, and reviewing the system’s costs—were implemented.
The report also suggested identifying potential customers for the outputs of the NCAS, and to garner additional resources for its operation through them. However, this could not be implemented fully. Another recommendation was to obtain regular expert opinion on the coding of anomalies, but without training the coders concurrently; this recommendation did not have the desired impact. The key recommendations—on including data on legal abortions in surveillance, developing safer systems for data transfer, and capturing data from birth notifications electronically—were not implemented. They remained unresolved, and are included in the recommendations that emerge from this evaluation.
Criteria for evaluation
All attributes under the CDC guidelines for the evaluation of surveillance systems were studied. Results are presented according to the CDC framework (Box 1).6
Box 1: Evidence on the performance of the NCAS based on CDC guidelines
Level of usefulness of the system
Evaluation of the system for the attributes of:
– Data quality
– Positive predictive value
Data were collected through interviews with data providers and users of the system (NHS staff in midwifery, public health and child health departments, academic researchers, and information managers in NHS trusts) using a semistructured questionnaire. Seven of the eight congenital anomaly registers were visited. Within the Office for National Statistics (ONS), data entry operators, data managers, and other stakeholders were also interviewed.
All publications and reports produced by the NCAS since 1991 were reviewed, as were annual reports of the congenital anomaly registers.
Usefulness of the NCAS
The usefulness of the NCAS was studied by determining the extent to which it meets its stated objectives and the criteria for usefulness as outlined by the CDC (table 1).
The simplicity of the NCAS was measured on the basis of the criteria described below.6 The NCAS is not a simple system as assessed by these.
The number of organisations receiving case reports
There are two main tiers of information: the regional and local registers and the “passive” system of voluntary notification through SD56 forms.1 The registers collect their information from multiple sources. Within the passive system, the reporting agencies can be acute trusts, child health systems, public health departments within primary care trusts, or midwifery units. They are required to complete and send in SD56 forms.
Amount and type of data necessary to establish the case definition
Several thousand birth defects have been identified, which occur with varying frequency and often in combination. The correct identification and coding of an anomaly depends on the quality of information received through notifications. The demographic and behavioural information required on the SD56 forms is not extensive, yet large proportions of key items on the forms are incomplete (table 2).
Level of integration with other systems
The system stands alone and is not integrated with the national births and deaths databases. This integration was also recommended in the earlier report.5 If successful, the integration would enable ONS to collect less demographic information at congenital anomaly notification as this could be obtained from the birth record, and therefore reduce the burden on data suppliers.7 It would also improve the data quality and allow analyses of survival in babies with congenital anomalies.
Method of data analysis and dissemination
The NCAS produces surveillance statistics monthly, quarterly, and annually. The cumulative sum technique (CUSUM) is used for analysis, based on a comparison of numbers (rather than rates) of anomalies reported from an area compared with those reported for the same area during the previous year or quarter, and raises alarms if the number of cases reported is more than expected. Alarm letters are sent to the relevant Primary Care Trust (PCT) (previously health authority) for investigation.
A flexible surveillance system can adapt to changing information needs or operating conditions with little time, personnel, or allocated funds.6 The NCAS is relatively inflexible. Any changes require considerable additional time and resources. This is primarily because it is based on a 16 bit Fox-pro database.
Data quality reflects the completeness, accuracy, and validity of the data recorded in the surveillance system.6 Like most surveillance systems, the NCAS relies on more than simple case counts. An analysis to see the percentage of unknown and blank responses over the last five years in selected key fields in the NCAS database (table 2) revealed that a large proportion of records had information missing.
Acceptability reflects the willingness of persons and organisations to participate in the surveillance system.6 Quantitative measures of acceptability included in the study were: agency participation rate, reporting rates from trusts, and completeness of reporting forms.
Agency participation rates
At present, the NCAS receives SD56 based notifications from a 100 contacts in various NHS organisations in England. The eight registers provide notifications electronically. The list of contacts in the areas not covered by registers is still based on the old health authorities which ceased to exist from September 2001.8 The list is now being updated to cover all PCTs and Strategic Health Authorities.
Reporting rates from trusts
These vary across the country. Figure 1 depicts the considerable under-reporting of cases to ONS that took place before the CARIS register (Wales) and the East Midlands and South Yorkshire (or Trent) register began sharing information in 1998 and 1999 respectively.9,10 The Mersey register started reporting to ONS from 2000. It is clear from fig 1 that the inception of registers in an area increases the notification rates two to threefold. The Eastern region, where there is no register, has very low notification rates, comparable to the national rates for the passive system.
Completeness of report forms
The report forms vary in their quality and degree of completeness, which affects the quality of data in the NCAS (table 2).
Qualitative assessment of acceptability
The various data providers in NHS organisations were interviewed through a semistructured questionnaire to determine their views on the NCAS, and their role in the information provision. The issues identified were their lack of awareness about the exact nature of the NCAS, a need for greater public health input and support, and the often inadequate information on congenital anomalies provided on the birth notification forms.
The coordinators of regional registers felt that the NCAS is omitting a major proportion of anomalies by not including terminations of pregnancy (TOPs) data in its database. Congenital anomalies are increasingly detected in the antenatal period. By not having TOPs data, the NCAS is not detecting important trends in occurrence of anomalies and potential clusters, thus limiting its usefulness and acceptability.
The sensitivity of a surveillance system is its ability to detect “cases”. At the level of case reporting, sensitivity refers to the proportion of cases of a disease (or other health related event) detected by the surveillance system.6
Based on a prevalence rate of 2–3%, we would expect between 12 000 and 18 000 congenital anomalies in 2001.11,12 Of this expected number, 6989 (39–58%) were notified to the NCAS through the passive system (via SD56 forms) and the regional registers.
In table 3, the sensitivity was calculated for four years (1998–2001), comparing the passive system with two of the regional registers. The sensitivity is about 33%, except in Trent in 1998, which was the year it started.
The sensitivity of the system was also measured for a specific disorder, by comparing the number of notifications of Down syndrome on the NCAS with those on the National Down Syndrome Cytogenetic Register, based on an analysis by Botting (in table 4).13 There was under-ascertainment of 50–60% for Down syndrome in the NCAS.13
Positive predictive value
Positive predictive value is the proportion of reported cases that actually have the health related event (congenital anomaly) under surveillance.6 In assessing the positive predictive value, the primary emphasis is placed on confirmation of cases reported through the surveillance system.6 This requires accessing anomaly information from hospital records, which could not be done because of ethical, logistic, and time constraints.
A representative public health surveillance system accurately describes the occurrence of a health related event over time and its distribution in the population by place and person.6
In 1998, the registers that fed into the NCAS covered 33% of all births in England and Wales. From 2003, the addition of two new registers has increased the coverage to 42% (fig 2). The registers provided 53% of all congenital anomaly notifications received by the NCAS in 2002.
Timeliness reflects the speed between steps in a public health surveillance system.6 There is disparity between the desired (4–6 weeks) and actual (8–12 weeks) amount of time required for the system to receive data, and to release the data, which is currently running at about six to seven months, but should be three months.5
Factors affecting the NCAS time scales include delayed recognition of symptoms and time spent on submission and confirmation of laboratory tests. Delays are also incurred because of movement of records—for example, records being sent to child health—before they are notified to ONS.
Stability refers to the reliability (the ability to collect, manage, and provide data properly without failure) and availability (the ability to be operational when it is needed) of the public health surveillance system. The NCAS has undergone several changes in the past few years, but it is stable in information technology terms.
The NCAS is a complex system that meets some of the CDC guidelines, but there are areas where its operation could be improved. There are methodological problems to applying the CDC criteria to this type of surveillance system, which have also been discussed by Gazarian et al,14 and there are practical problems for surveillance that arise from the recent changes to NHS structures.8
The NCAS is inflexible. The completeness of data is particularly poor for certain key fields—for example, gestational age and occupation of parents. The acceptability of the system to the data providers depends on the extent of local support received by them for supplying the congenital anomaly information, which is generally poor. The sensitivity of the passive system, as compared with the local registers is about 33%, and when compared with expected figures, the national system detects 39–59% of estimated anomalies.
The NCAS covers all of England and Wales, but does not obtain adequate notification rates in areas not covered by the registers, possibly because of its voluntary nature. The registers cover 42% of all births, and account for most of the notifications to the NCAS. The establishment of a register leads to a two to threefold increase in notifications from that area, and areas with no registers have low notification rates. The stability of the NCAS is its key strength. The time lags in the NCAS are largely due to delayed reporting.
Of the nine performance attributes, acceptability, positive predictive value, and representativeness were the hardest to evaluate in this study. It was difficult to analyse reporting rates from the various reporting units as a measure of acceptability, because of the recent reorganisation of the NHS, which has led to the creation of new PCTs, and a redefinition of boundaries and roles of the former health authorities (now Strategic Health Authorities).8 Area based measures of reporting rates were used, as the database of NHS contacts was no longer relevant.
Concurrent with the recommendation made in the MAG report5 and by Boyd et al,4 the coordinators of regional registers felt that the NCAS is omitting a major proportion of anomalies by not including TOPs data in its database. It is thus likely to miss important trends in occurrence of congenital anomalies and potential clusters.
It was not possible to verify “true values” for fields such as gestational age or occupation of parents, or verify if a reported case actually had the stated anomaly, by going back to the hospital records, because of the complicated information trail from reporting trusts to the ONS. Hence, the positive predictive value for the system could not be estimated.
What is already known on this topic
Surveillance for congenital anomalies in England and Wales has been ongoing since 1964, when it was established following the thalidomide epidemic of limb defects
It is a voluntary system that relies on the efforts of staff within NHS trusts whose primary responsibility is patient care
What this study adds
The NCAS has evolved considerably since the time it was first developed to detect visible congenital anomalies at birth, and is now a two tier system comprising anomaly registers and the passive system of information sent from NHS trusts directly
The data quality is variable, largely because of its voluntary nature. Areas where there are anomaly registers achieve high coverage of anomalies
Although fig 2 shows the geographic coverage of the registers and the areas where the passive system operates, the representativeness of the NCAS could not be estimated accurately. Instead, the coverage of the system was measured on the basis of the number of births that occurred in that area. For the registers, this was a relatively easy process, but the inability to calculate birth rates for each reporting unit in the passive system did not allow an accurate estimation of the coverage of the passive system.
The NCAS has evolved considerably since it was first developed to detect visible congenital anomalies at birth.1 Its primary purpose was to detect changes in the frequency of reporting any particular anomaly or group of anomalies rather than trying to estimate the prevalence at birth. The absence of TOPs data is a major omission, which should be amended by ONS. However, the NCAS was not designed to monitor abortions. Its original role also did not envisage recording abnormalities detected later in infancy and beyond, or providing high quality epidemiological data on congenital anomalies. Yet, it is struggling to adapt itself to perform these functions, and is an important part of the public health response to the occurrence of congenital anomalies. The steps that the ONS can take to improve the NCAS database and its operations are shown in box 2.
Box 2: Recommendations based on the NCAS evaluation
Information received by the NCAS
The NCAS should invest time and resources to update and maintain the database of its reporting sources
The registers and NHS trusts should be encouraged to send notifications electronically
All attempts should be made to secure the data on termination of pregnancies and this should be included in the anomalies database, and analysed routinely
Information processed by the NCAS
Coders working on the NCAS should receive formal training and upgrades
Information received from registers should not be recoded by the ONS coders
The NCAS should reconfigure its monitoring groups in line with EUROCAT
The CUSUM technique and baseline figures in the NCAS need to be reviewed
The monthly surveillance should be replaced by quarterly and annual analysis
The ONS should consider a two tier system, where registers’ data are analysed differently from data received from the passive system, in addition to the combined analysis, and fed back to the registers quarterly
The two systems should be compared annually, as part of the system’s data quality checks
Some of the missing data, such as information on the last menstrual period, can be calculated from the date of birth and the gestation age, and this should be performed regularly
Linkage of the NCAS with the births and deaths databases should be carried out annually until the system is upgraded
When the current 16 bit system is upgraded, it should be made compatible with the national births and deaths databases
Information sent out by the NCAS
Statistical analysis for surveillance should be performed on the basis of Strategic Health Authorities, with information on number of cases of anomalies by PCT. The NCAS should make a provision for providing details of cases to directors of public health in PCTs and Strategic Health Authorities, if there is a need for further exploration and analysis
The current format of Alarm Letters should be discontinued, and replaced by a system where surveillance information is only sent out to directors of public health for further investigation if a true increase in number of cases is detected
Demographic data obtained by linkage with the births and deaths databases at ONS should be reported back to local registers
The format of the annual MB3 series needs to be reviewed and made more user friendly and meaningful to the data providers
Regional registers have more complete data and function efficiently because of the small area of operation, local contacts, and effective practices that ensure high participation rates. As suggested by Boyd et al,4 the existing registers should be allocated central funding, and new registers should be established in areas where there are none at present. These could then feed into a more streamlined national system.
We thank Nigel Physick, Pam McMinn, and Elaine Tower at the Office for National Statistics for help with extracting the data required for this paper, and the coordinators of the various registers for their views and insight on the system. We gratefully acknowledge the use of congenital anomaly data notified by the Trent and Wales registers.
Competing interests: ND works at the Office for National Statistics and is Director of the NCAS.
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.