Prediction Impact Curve: A New Graphical Approach Integrating Intervention Effects in the Evaluation of Prediction Model Utility Público

Abstract

Traditional measures of model performance generally address discrimination and calibration, while novel measures focus on the potential for risk models to change medical decisions. This document first provides a review of current traditional and novel model performance measures. Then, we propose a graphical approach, the prediction impact curve, which evaluates the performance of risk models in terms of their expected preventive effect in the population. Using simulated data and estimates from the literature, we illustrate how the prediction impact curve is used to estimate the expected reduction in events when using a risk model to assign individuals to a preventive intervention and how to compare nested risk models. We apply the prediction impact curve to the Atherosclerosis Risk in Communities (ARIC) Study to illustrate its application toward primary prevention of coronary heart disease. We estimated that if the ARIC cohort received statin intervention at baseline, 5% of events were expected to be prevented when evaluated at a cut-off threshold of 20% predicted risk. Additionally, we estimated that an average of 15% of events were expected to be prevented when considering performance across all possible thresholds. We conclude that the prediction impact curve is a useful and intuitive graphical approach for assessing the expected performance of risk models and is most beneficial when considered alongside existing measures of model performance.

Table of Contents

Chapter I: Literature Review ...............................................1

Traditional Measures of Model Performance ...........................1

Overall performance measures and goodness-of-fit .................1

Calibration .......................................................................2

Discrimination...................................................................3

Limitations of AUC .............................................................3

Novel Measures of Model Performance ..................................4

Reclassification measures ...................................................4

Net reclassification improvement .........................................5

Integrated discrimination improvement..................................6

Limitations of NRI and IDI ...................................................7

Traditional decision analysis ................................................8

Decision curve analysis and net benefit .................................9

Limitations of decision curve analysis ..................................10

Chapter II: Manuscript ......................................................11

Abstract .........................................................................12

Introduction ....................................................................13

Materials and Methods ......................................................14

Results ..........................................................................18

Discussion ......................................................................21

Funding and Acknowledgements .........................................23

Chapter III: Discussion and Future Direction .........................24

Appendix .......................................................................26

Figure Legend .................................................................26

References .....................................................................27

Figures and Tables ...........................................................29

IRB Exemption Letter ........................................................34

About this Master's Thesis

Rights statement

• Permission granted by the author to include this thesis or dissertation in this repository. All rights reserved by the author. Please contact the author for information regarding the reproduction and use of this thesis or dissertation.

School	Rollins School of Public Health
Department	Epidemiology
Degree	MPH
Submission	Master's Thesis
Language	English
Research Field	Health Sciences, Epidemiology
Palabra Clave	Prediction research Epidemiology Model performance
Committee Chair / Thesis Advisor	Janssens, A Cecile, Emory University
Partnering Agencies	Does not apply (no collaborating organization)

Última modificación

Primary PDF

Thumbnail	Title	Date Uploaded	Actions
	Prediction Impact Curve: A New Graphical Approach Integrating Intervention Effects in the Evaluation of Prediction Model Utility ()	2018-08-28 11:17:23 -0400	Press to Select an action Download

Supplemental Files

Thumbnail	Title	Date Uploaded	Actions