Evaluation of agreement measures among groups of raters with an application to the interpretation of kidney obstruction. Open Access
Zinsli, Alison (Spring 2018)
Published
Abstract
Kidney obstruction prevents the kidneys from properly draining which can lead to loss of function if left untreated. The Department of Nuclear Medicine at Emory University is developing a decision supporting software called RENEX to assist radiologists in limiting their errors and arriving at the correct diagnosis when interpreting the renal scans. In the absence of a gold standard, experts assessment of kidney obstruction is considered to be the best available standard. The objective of this study aims to quantify the agreement among experts and residents with and without the RENEX intervention and to address the question of whether the RENEX intervention helps the residents perform similar to the experts by quantifying the agreement between groups.
Three experts and three residents with and without the RENEX educational intervention interpreted data from 50 patients for both their left and right on degree of obstruction. They classified obstruction on a continuous scale from -1 to 1 and could be categorized into three groups: unobstructed [-1, -0.2), undetermined [-0.2, 0.2] and obstructed (0.2, 1]. Agreement was evaluated within groups using the concordance correlation coefficient (CCC) and weighted kappa. Further analysis was done to determine whether a resident can replace an expert in the interpretation of kidney obstruction. Since the same patient is evaluated multiple times the observations are correlated and a bootstrap methodology was used to calculate accurate standard error and confidence intervals.
The agreement index of CCC for experts for the left and right kidney are 0.819 (0.619, 0.937) and 0.866 (0.706, 0.935), respectively. Whereas, residents with and without the use of RENEX CCC agreement for the left kidney is 0.314 (0.132, 0.492) and 0.726 (0.504, 0.874), respectively; and 0.347 (0.107, 0.558) and 0.680 (0.435, 0.835) for the right kidney. There is a significant difference between the agreement of experts and residents (p<0.001, left and right kidney) but when residents use RENEX the difference in agreement is no longer significantly different (p=0.505, p=0.050; left and right kidney, respectively). A similar pattern can be seen when an expert is replaced by a resident; the agreement is better when the resident uses RENEX.
In conclusion, not only did residents agreement improve, but RENEX also helped improve the accuracy of the resident’s classification compared to the expert’s. The methodology that was used in determining whether a resident can replace an expert is generally applicable to other similar studies.
Table of Contents
Introduction ................................................................................ 1
Methods ...................................................................................... 5
The Data ............................................................................. 5
Concordance Correlation Coefficient ................................. 5
Weighted Kappa .................................................................. 7
Bootstrap Confidence Intervals ........................................... 8
Exploratory Analysis .......................................................... 10
Agreement for obstruction classification within groups ..... 11
Replacement analysis to determine resident accuracy ........ 12
Comparing agreement between groups ............................... 12
Results ........................................................................................ 13
Exploratory ......................................................................... 13
Agreement for obstruction classification within groups ..... 15
Replacement analysis to determine resident accuracy ........ 18
Discussion ................................................................................... 21
References ................................................................................... 24
Appendix ..................................................................................... 26
About this Master's Thesis
| School | |
|---|---|
| Department | |
| Degree | |
| Submission | |
| Language |
|
| Research Field | |
| Keyword | |
| Committee Chair / Thesis Advisor | |
| Committee Members | |
| Partnering Agencies |
Primary PDF
| Thumbnail | Title | Date Uploaded | Actions |
|---|---|---|---|
|
|
Evaluation of agreement measures among groups of raters with an application to the interpretation of kidney obstruction. () | 2018-04-09 11:39:53 -0400 |
|
Supplemental Files
| Thumbnail | Title | Date Uploaded | Actions |
|---|