Probabilities of hospital mortality provide meaningful information in many contexts, such as in discussions of patient prognosis by intensive care physicians, in patient stratification for analysis of clinical trial data by researchers, and in hospital reimbursement analysis by insurers. Use of probabilities as binary predictors based on a cut point can be misleading for making treatment decisions for individual patients, however, even when model performance is good overall. Alternative models for estimating severity of illness in intensive care unit (ICU) patients, while demonstrating good agreement for describing patients in the aggregate, are shown to differ considerably for individual patients. This suggests that identifying patients unlikely to benefit from ICU care by using models must be approached with considerable caution.