Receiver operating characteristics curve

From NorthShore Analytics
Jump to: navigation, search

ROC curves are an essential tool for assessing the quality of a classification model. Table 3.24 illustrates the relationship between the actual and predicted outcomes as reflected by such curves.

Table 3.24. Tabularized relations between truth/falseness of the null hypothesis and outcomes of the test.

ROC curve graphs are used for internal research purposes and for presentation to technical audiences familiar with this concept. The ROC curve for a survival model describing the mortality risk in heart failure patients is presented as an example in Fig. 3.16.

Figure 3.16. Sample receiver operating characteristic (ROC) curve graph.

An example of a plot combining ROC, \(F_1\) score and Matthews' correlation coefficient was presented earlier in Fig. 3.6

For historical reasons, the Clinical Analytics team has found it more instructive and easily digestible for executive and practitioner audiences to employ combined lift curve / positive predictive value graphs as a tool for visualizing the quality of a predictive model. lift curve / positive predictive value graphs are presented double-scaled with true positive rate plotted on the left in red, and positive predictive value on the right in blue. The abscissa (x-axis) represents the percentage of the (test) population that was classified by the model as having an outcome of interest. The combined graph for a logistic heart failure admission prediction model is presented as an example in Fig. 3.17.

Figure 3.17. Sample lift curve graph.

As follows from Fig. 3.17, should the riskiest 5% of the patients selected by the model be chosen for intervention, true positive rate in that population will be approximately 50%. This rate reaches approximately 10% in the general population, hence the lift achieved by applying the model for the top 5% is close to 5.