Receiver operating characteristic (ROC) analysis: Basic principles and applications in radiology
Introduction
To address the clinical problems in everyday radiology practice, a large and ever expanding array of imaging modalities is available. This raises the question of what particular test to use for what purpose. There is thus, a need for a method to compare the diagnostic accuracy of the various tests in an objective manner. Over the last two decades, receiver operating characteristic (ROC) analysis has increasingly been used for this purpose, notably in radiology and clinical chemistry 1, 2. Originally developed in the early 1950s for the analysis of RADAR signal detection, ROC analysis was first applied in psychophysical research 1, 3, 4. In the 1960s, Dr Lee Lusted was the first to recognize a possible role for ROC analysis in medical decision making 5, 6.
In this article we will describe the principles underlying ROC analysis and explain the advantages of this method over conventional analysis, which uses comparison of sensitivity and specificity values. We will provide practical information on how to use and interpret ROC curves. The major applications of ROC analysis will be discussed and their limitations will be addressed.
Section snippets
Sensitivity and specificity: Need for ROC analysis
The traditional measures to quantify the diagnostic accuracy of a test are sensitivity and specificity. These parameters describe the fractions of patients (diseased and non-diseased) that are classified correctly. The sensitivity or true positive fraction (TPF) describes the fraction of diseased patients that actually has a positive test result. The specificity or true negative fraction (TNF) describes the probability of a negative test result in non-diseased individuals. Sensitivity and
The ROC curve: Basic principles
The choice of the threshold value influences both sensitivity and specificity. For the ideal diagnostic test, the probability distributions of test results indicating presence or absence of disease do not overlap and the chosen threshold value is in between these distributions (Fig. 1). The resulting sensitivity and specificity are both 100%. For most diagnostic tests, however, the probability distributions of diseased and normal overlap. Any threshold value will lead to the misclassification
Practical aspects of ROC analysis
ROC analysis can be performed for tests that provide either continuous data or rating-scale data. A rating scale for confidence judgements will generally produce a meaningful curve if five rating categories are used 8, 9. Several computer programs are available to estimate a smooth ROC curve through the observed operating points. The most widely used computer software package is the one developed by Metz et al. [10]. These computer programs estimate a binormal ROC curve. This binormal model is
Applications of ROC analysis: Comparing tests or observers
Which test or observer discriminates best between presence and absence of disease? The discriminative ability of a test is determined by the amount of overlap between the probability distributions of the test results of diseased and non-diseased patients. This overlap determines the shape and position of the ROC curve. If the probability distributions of diseased and non-diseased are identical, i.e. they overlap completely, the TPF and FPF are equal at any threshold value. The test has no
Applications of ROC analysis: Optimizing the threshold value
Another potential use of the ROC curve is in optimizing the threshold value of a test. The ROC curve comprises all possible combinations of sensitivity and specificity at all possible threshold values. This offers the opportunity to assess the optimal threshold value to be used in clinical practice.
In practice, choosing an optimal threshold value based on ROC analysis is practicable only for continuous data, e.g. Doppler velocity parameters for carotid artery stenosis or CT-density for
References (25)
A computer program for non-parametric receiver operating characteristic analysis
Comput Methods Programs Biomed
(1993)- et al.
Comparison of receiver operating characteristic curves on the basis of optimal operating points
Acad Radiol
(1996) ROC methodology in radiologic imaging
Invest Radiol
(1986)In pursuit of a piece of the ROC
Radiology
(1997)Receiver operating characteristic analysis (ROC) methodology: The state of the art
Crit Rev Diagn Imaging
(1989)ROC analysis applied to the evaluation of medical imaging techniques
Invest Radiol
(1979)Logical analysis in roentgen diagnosis
Radiology
(1960)Signal detectability and medical decision making
Science
(1971)- et al.
CT and MR distinction of adenomas and nonadenomas of the adrenal gland
J Comput Assist Tomogr
(1994) Some practical issues of experimental design and data analysis in radiological ROC studies
Invest Radiol
(1989)
The use of continuous and discrete judgements in receiver operating characteristic studies of diagnostic imaging techniques
Invest Radiol
Cited by (269)
Prediction of bitterant and sweetener using structure-taste relationship models based on an artificial neural network
2022, Food Research InternationalCitation Excerpt :For the CNN model, the 2D image of the molecule was converted from the SMILES structure. The 2D image size of the optimized model evaluated by six indicators (Section 2.6), such as accuracy, precision, sensitivity, specificity, Matthew’s correlation coefficient (MCC) and area under curve (AUC) by receiver operating characteristics (ROC) (Van Erkel & Peter, 1998), was 32 × 32. These images were then converted into RGB color images in PNG format with 32 × 32 pixels (Fig. 1(B)).
Semi-supervised automatic seizure detection using personalized anomaly detecting variational autoencoder with behind-the-ear EEG
2022, Computer Methods and Programs in BiomedicineApplication of quantitative structure-activity relationship to food-derived peptides: Methods, situations, challenges and prospects
2021, Trends in Food Science and Technology