AN EVALUATION PLAN FOR A PILOT OF CDC EMR ALERTING SERVICE PROTOTYPE Open Access
Ye, Ye (2011)
Abstract
Background: As a critical component of public health surveillance, public health authorities disseminate alerts to healthcare providers to increase their awareness of potential public health threats and to enable timely and effective responses. The variety of communication channels and the diversity of message formats, however, lead the receptors to a paradoxical situation: too much useless information, or too little relevant information. To address these challenges, the CDC EMR alerting team developed a Public Health Alert Repository System, and planned to conduct a real pilot to evaluate its performance. A comprehensive evaluation plan was needed to guarantee the completeness and validity of the test results.
Method: This evaluation plan was designed by taking account of the objectives of major stakeholders, selecting an evaluation framework, identifying evaluation elements, and clarifying measurement methods.
Results: Based on the logic model framework, stakeholders' objectives (actionable alerts; consumption of alert; integration with the decision support system; clinician action performed; sensitivity, specificity, and PPV of matching algorithm; and local customization) were mapped with five evaluation elements, including system quality; quality of the alerts (sensitivity, PPV, and specificity); clinician use of the alerts ("% of matched alerts that are clicked"); user perception (a user perception questionnaire); and impact (public health impact: local customization; health care impact: "% of positive specimen stool results" and "number of specimen stools per patient"). To test the validity of these indicators, the plan also suggested calculating their correlations, including the correlation between an objective indicator (% of matched alerts that are clicked) and a subjective indicator (the score of the "read alerts" question), and the correlations between many input, process, and outcome indicators.
Conclusion: In this study, stakeholders' objectives were successfully translated into a measurable evaluation matrix, where feasible measurement methods, study design, and data resources were identified. The causal relationships between input, process, and outcome, and the correlations between objective indicators and subjective indicators, were also recommended to be used for checking the validity of many indicators. The development process of this evaluation plan and many of its results may be possibly adapted for other system evaluations.
Table of Contents
Table of Contents
Chapter I Introduction...1
Problem statement...2
Purpose statement...6
Significance statement...6
Chapter II Review of literature...8
Evaluation frameworks...9
Study designs...10
Simple before-after evaluation...10
Controlled before-after evaluation...11
Randomized controlled trial...12
Considerations for study design selection...13
Considerations for sample size...14
Evaluation methods...15
Subjective methods...16
Objective methods...18
Aligning study methods with research questions...19
Studies...20
Chapter III Methodology...23
Involving stakeholders...24
Methods to define the scope of the evaluation...24
Choosing an evaluation framework...25
Selecting a study design...25
Using both subjective and objective methods...27
Chapter IV Results...28
Objectives for EMR alerting service...29
Evaluation elements...31
System quality...34
Quality of alerts...35
Clinician use of the alerts...38
User perception...40
Impact...45
Correlations between indicators...51
Chapter V Conclusions, Implications, and Recommendations...56
Summary of study...57
Conclusion and Implication...61
Recommendations...62
Recommendations for EMR alerting
service...62
Discussion on PPV...63
Recommendations for evaluation design...64
Recommendations for user perception questionnaire design...64
References...66
About this Master's Thesis
| School | |
|---|---|
| Department | |
| Subfield / Discipline | |
| Degree | |
| Submission | |
| Language |
|
| Research Field | |
| Keyword | |
| Committee Chair / Thesis Advisor | |
| Committee Members | |
| Partnering Agencies |
Primary PDF
| Thumbnail | Title | Date Uploaded | Actions |
|---|---|---|---|
|
|
AN EVALUATION PLAN FOR A PILOT OF CDC EMR ALERTING SERVICE PROTOTYPE () | 2018-08-28 11:45:38 -0400 |
|
Supplemental Files
| Thumbnail | Title | Date Uploaded | Actions |
|---|