Prediction Impact Curve: A New Graphical Approach Integrating Intervention Effects in the Evaluation of Prediction Model Utility Open Access

Campbell, Will (2014)

Permanent URL: https://etd.library.emory.edu/concern/etds/fx719m62f?locale=en
Published

Abstract

Traditional measures of model performance generally address discrimination and calibration, while novel measures focus on the potential for risk models to change medical decisions. This document first provides a review of current traditional and novel model performance measures. Then, we propose a graphical approach, the prediction impact curve, which evaluates the performance of risk models in terms of their expected preventive effect in the population. Using simulated data and estimates from the literature, we illustrate how the prediction impact curve is used to estimate the expected reduction in events when using a risk model to assign individuals to a preventive intervention and how to compare nested risk models. We apply the prediction impact curve to the Atherosclerosis Risk in Communities (ARIC) Study to illustrate its application toward primary prevention of coronary heart disease. We estimated that if the ARIC cohort received statin intervention at baseline, 5% of events were expected to be prevented when evaluated at a cut-off threshold of 20% predicted risk. Additionally, we estimated that an average of 15% of events were expected to be prevented when considering performance across all possible thresholds. We conclude that the prediction impact curve is a useful and intuitive graphical approach for assessing the expected performance of risk models and is most beneficial when considered alongside existing measures of model performance.

Table of Contents

Chapter I: Literature Review ...............................................1

Traditional Measures of Model Performance ...........................1

Overall performance measures and goodness-of-fit .................1

Calibration .......................................................................2

Discrimination...................................................................3

Limitations of AUC .............................................................3

Novel Measures of Model Performance ..................................4

Reclassification measures ...................................................4

Net reclassification improvement .........................................5

Integrated discrimination improvement..................................6

Limitations of NRI and IDI ...................................................7

Traditional decision analysis ................................................8

Decision curve analysis and net benefit .................................9

Limitations of decision curve analysis ..................................10

Chapter II: Manuscript ......................................................11

Abstract .........................................................................12

Introduction ....................................................................13

Materials and Methods ......................................................14

Results ..........................................................................18

Discussion ......................................................................21

Funding and Acknowledgements .........................................23

Chapter III: Discussion and Future Direction .........................24

Appendix .......................................................................26

Figure Legend .................................................................26

References .....................................................................27

Figures and Tables ...........................................................29

IRB Exemption Letter ........................................................34

About this Master's Thesis

Rights statement
  • Permission granted by the author to include this thesis or dissertation in this repository. All rights reserved by the author. Please contact the author for information regarding the reproduction and use of this thesis or dissertation.
School
Department
Degree
Submission
Language
  • English
Research Field
Keyword
Committee Chair / Thesis Advisor
Partnering Agencies
Last modified

Primary PDF

Supplemental Files