A brief account (via simulation) of the ROC (and its AUC)

True and false predictions

While we might not know group membership for an individual, we assume that they do indeed belong to one of the two groups. When we base a prediction of group membership based on a test, we may or may not be right. There are four scenarios. It is possible that our prediction is (1) a true positive (the patient has the condition and that is what we predict), (2) a false positive (the patient does not have the condition, but we predict they do have it), (3) a false negative (the patient has the condition but we believe they are healthy), or (4) a true negative (the patient is healthy and that is our prediction.) A “good” test is one that maximizes true positive predictions while minimizing false positive predictions.

We can actually only assess the quality of the test if we know the true group membership of the individuals. So, our plan is to take measurements on this known sample, make predictions based on the test, and see how our predictions match up to reality. The ROC is one way to characterize how well our test matches up to reality.

Binary decision informed by a continuous measure

While we make a binary decision about group membership - either we think they have the condition or they do not - the underlying measure that is used to make that determination may be continuous, like a score. For example, a hypothetical test might return a score between -10 and 10. We can pick a threshold anywhere along the continuum that will form the basis of our prediction. For example, we might say that any score > 0 indicates the condition is present, otherwise it is not. This simple test will be useful as a tool to discriminate between the disease and non-disease groups if that threshold indeed distinguishes the groups.

This is probably best demonstrated with a simple simulation. The sample we will generate has 100 individuals, around 40% who have the condition in question. The average score for the non-disease group is set at -5, and the average score for the disease group is 5. Both have variance 3.5:

library(simstudy)

# define data

defx <- defData(varname = "condition", formula = .4, dist = "binary")
defx <- defData(defx, "x", formula = "-5 + 10*condition", 
                variance = 3.5, dist = "normal")

# generate data

set.seed(1873)
dx <- genData(100, defx)
head(dx)

##    id condition     x
## 1:  1         0 -5.83
## 2:  2         1  4.66
## 3:  3         1  4.23
## 4:  4         0 -3.87
## 5:  5         1  1.78
## 6:  6         0 -4.87

Looking at the plot below, a threshold of zero appears to do an excellent job of distinguishing the groups. All of those with the condition (depicted in red) are above the threshold, whereas all of those without the condition (depicted in green) fall below the threshold:

The world is not always so neat and clean

Of course, we don’t usually have a measure or test that separates the groups so cleanly. Let’s say the average of the disease group is 2.5 and the non-disease group is -3. The threshold of zero still works pretty well, but it is not perfect. Some with the disease fall below the threshold (false negatives), and some without the disease lie above the threshold (false positives). In fact, only 87% of those with the disease are correctly identified (true positives), while 13% of those without the condition are incorrectly identified has having the disease (false positives).

defx <- updateDef(defx, changevar = "x", newformula="-3 + 5.5*condition",
                  newvariance = 6)
dx <- genData(100, defx)

Area under the ROC

The AUC is, well, the area under the ROC. The maximum AUC will be 1 when there is complete separation (there is an example of this below), and the minimum is 0.5 (depicted by the diagonal line) when there is no separation by the test measure (again, an example will follow). We can estimate this area by integrating an approximate function defined by the data between 0 and 1.

f <- approxfun(x = roc$false.pos, y=roc$true.pos)
integrate(f, lower = 0, upper = 1)

## 0.957 with absolute error < 0.00011

There is actually a meaningful interpretation of the AUC, that is described in a classic 1982 paper by Hanley & McNeil (if you want a deeper understanding of the issues, this paper is not a bad place to start - there is, of course, a huge literature on the topic of ROCs). The AUC is actually equivalent to the probability that the test measure of a random draw from the diseased group will be greater than the test measure of a random draw from the healthy group. So, an AUC = 0.90 indicates that 90% of the time we draw a test measure from the disease group and non-disease group, the measure from the disease group will be greater.

Here is a simple function that returns a value of TRUE if the random draw from the disease group is greater:

randcomp <- function(ds) {
  ds[condition == 1, sample(x, 1)] > ds[condition == 0, sample(x, 1)]
}

And here is the proportion of 1000 draws where the measure from the disease group draws is greater (this is expected to be close to the AUC, which was estimated above to be 0.957):

mean(sapply(1:1000, function(x) randcomp(dx)))

## [1] 0.958

Of course, R has several packages that provide ROCs and calculate AUCs. I’m using package pROC here just to show you that my AUC estimate is not totally crazy:

library(pROC)
roc_obj <- roc(response = dx$condition, predictor = dx$x)
auc(roc_obj)

## Area under the curve: 0.958

Minimal separation

When the test measure for each group is equally distributed, there is unlikely to be any threshold for which the proportion of true positives exceeds the proportion of false positives. If this is the case, we should probably look for another test measure - or be prepared to make a lot of mistakes in the non-disease group.

defx <- updateDef(defx, changevar = "x", newformula="0+0*condition",
                  newvariance = 8)
dx <- genData(100, defx)

As we move the threshold lower, both the proportion of true positives and false positives steadily increase:

As a result, the ROC hangs fairly close to the diagonal lower bound.

We would expect the AUC to be fairly close to 0.5, which it is:

f <- approxfun(x = roc$false.pos, y=roc$true.pos)
integrate(f, lower = 0, upper = 1)

## 0.623 with absolute error < 4.5e-05

mean(sapply(1:1000, function(x) randcomp(dx)))

## [1] 0.613

Complete separation

At the other extreme, the mean of the disease group is high enough so that there is no overlap between the two groups. In this case, the curve follows along Y-axis before going across the X-axis. We can achieve 100% true positives and no false positives if threshold is set at some point that is below the minimum of the disease group, and above the maximum of the non-disease group. Zero will be the ideal cut-off point for this example.

defx <- updateDef(defx, changevar = "x", newformula="-4+8*condition",
                  newvariance = 3.5)
dx <- genData(100, defx)

As expected the AUC is equal to 1:

f <- approxfun(x = roc$false.pos, y=roc$true.pos)
integrate(f, lower = 0, upper = 1)

## 0.996 with absolute error < 9.2e-05

mean(sapply(1:1000, function(x) randcomp(dx)))

## [1] 1