Receiver operating characteristic (ROC) analysis: Basic principles and applications in radiology

https://doi.org/10.1016/S0720-048X(97)00157-5Get rights and content

Abstract

Receiver operating characteristic (ROC) analysis is a widely accepted method for analyzing and comparing the diagnostic accuracy of radiological tests. In this paper we will explain the basic principles underlying ROC analysis and provide practical information on the use and interpretation of ROC curves. The major applications of ROC analysis will be discussed and their limitations will be addressed.

Introduction

To address the clinical problems in everyday radiology practice, a large and ever expanding array of imaging modalities is available. This raises the question of what particular test to use for what purpose. There is thus, a need for a method to compare the diagnostic accuracy of the various tests in an objective manner. Over the last two decades, receiver operating characteristic (ROC) analysis has increasingly been used for this purpose, notably in radiology and clinical chemistry 1, 2. Originally developed in the early 1950s for the analysis of RADAR signal detection, ROC analysis was first applied in psychophysical research 1, 3, 4. In the 1960s, Dr Lee Lusted was the first to recognize a possible role for ROC analysis in medical decision making 5, 6.

In this article we will describe the principles underlying ROC analysis and explain the advantages of this method over conventional analysis, which uses comparison of sensitivity and specificity values. We will provide practical information on how to use and interpret ROC curves. The major applications of ROC analysis will be discussed and their limitations will be addressed.

Section snippets

Sensitivity and specificity: Need for ROC analysis

The traditional measures to quantify the diagnostic accuracy of a test are sensitivity and specificity. These parameters describe the fractions of patients (diseased and non-diseased) that are classified correctly. The sensitivity or true positive fraction (TPF) describes the fraction of diseased patients that actually has a positive test result. The specificity or true negative fraction (TNF) describes the probability of a negative test result in non-diseased individuals. Sensitivity and

The ROC curve: Basic principles

The choice of the threshold value influences both sensitivity and specificity. For the ideal diagnostic test, the probability distributions of test results indicating presence or absence of disease do not overlap and the chosen threshold value is in between these distributions (Fig. 1). The resulting sensitivity and specificity are both 100%. For most diagnostic tests, however, the probability distributions of diseased and normal overlap. Any threshold value will lead to the misclassification

Practical aspects of ROC analysis

ROC analysis can be performed for tests that provide either continuous data or rating-scale data. A rating scale for confidence judgements will generally produce a meaningful curve if five rating categories are used 8, 9. Several computer programs are available to estimate a smooth ROC curve through the observed operating points. The most widely used computer software package is the one developed by Metz et al. [10]. These computer programs estimate a binormal ROC curve. This binormal model is

Applications of ROC analysis: Comparing tests or observers

Which test or observer discriminates best between presence and absence of disease? The discriminative ability of a test is determined by the amount of overlap between the probability distributions of the test results of diseased and non-diseased patients. This overlap determines the shape and position of the ROC curve. If the probability distributions of diseased and non-diseased are identical, i.e. they overlap completely, the TPF and FPF are equal at any threshold value. The test has no

Applications of ROC analysis: Optimizing the threshold value

Another potential use of the ROC curve is in optimizing the threshold value of a test. The ROC curve comprises all possible combinations of sensitivity and specificity at all possible threshold values. This offers the opportunity to assess the optimal threshold value to be used in clinical practice.

In practice, choosing an optimal threshold value based on ROC analysis is practicable only for continuous data, e.g. Doppler velocity parameters for carotid artery stenosis or CT-density for

References (25)

  • S Vida

    A computer program for non-parametric receiver operating characteristic analysis

    Comput Methods Programs Biomed

    (1993)
  • EJ Halpern et al.

    Comparison of receiver operating characteristic curves on the basis of optimal operating points

    Acad Radiol

    (1996)
  • CE Metz

    ROC methodology in radiologic imaging

    Invest Radiol

    (1986)
  • AJ Dwyer

    In pursuit of a piece of the ROC

    Radiology

    (1997)
  • JA Hanley

    Receiver operating characteristic analysis (ROC) methodology: The state of the art

    Crit Rev Diagn Imaging

    (1989)
  • JA Swets

    ROC analysis applied to the evaluation of medical imaging techniques

    Invest Radiol

    (1979)
  • LB Lusted

    Logical analysis in roentgen diagnosis

    Radiology

    (1960)
  • LB Lusted

    Signal detectability and medical decision making

    Science

    (1971)
  • AR Van Erkel et al.

    CT and MR distinction of adenomas and nonadenomas of the adrenal gland

    J Comput Assist Tomogr

    (1994)
  • CE Metz

    Some practical issues of experimental design and data analysis in radiological ROC studies

    Invest Radiol

    (1989)
  • HE Rockette et al.

    The use of continuous and discrete judgements in receiver operating characteristic studies of diagnostic imaging techniques

    Invest Radiol

    (1992)
  • Metz CE, Kronman HB. Computer programs ROCFIT, LABROC1, CORROC2, CLABROC, INDROC, ROCPWRPC. Available from CE Metz,...
  • Cited by (269)

    • Prediction of bitterant and sweetener using structure-taste relationship models based on an artificial neural network

      2022, Food Research International
      Citation Excerpt :

      For the CNN model, the 2D image of the molecule was converted from the SMILES structure. The 2D image size of the optimized model evaluated by six indicators (Section 2.6), such as accuracy, precision, sensitivity, specificity, Matthew’s correlation coefficient (MCC) and area under curve (AUC) by receiver operating characteristics (ROC) (Van Erkel & Peter, 1998), was 32 × 32. These images were then converted into RGB color images in PNG format with 32 × 32 pixels (Fig. 1(B)).

    View all citing articles on Scopus
    View full text