Assessing Observer Agreement for Categorical Observations Público
Gao, Jingjing (2010)
Published
Abstract
Assessment of observer agreement is based on the similarity between readings made on the same subject by different observers. The assessment of agreement on categorical observations is traditionally based on kappa or weighted kappa coefficients (Cohen, 1960, 1968). However, kappa statistics have been criticized because they attain implausible values when the marginal distributions are skewed and/or unbalanced (Feinstein and Cicchetti, 1990), and also because they depend on the prevalence of the underlying condition (Kraemer, 1979; Thompson and Walter, 1988).
We propose new scaled indices called the coefficients of individual agreement (CIAs) for the assessment of individual observer agreement by comparing the observed disagreement between two observers to the disagreement between replicated observations made by the same observer on the same subject. This is based on the notion that the agreement between the two observers is usually not expected to be better than the agreement between replicated observations of the same observer, and hence, a satisfactory agreement is established if these quantities are similar. We investigate two scenarios: (1) when the data can be considered as unmatched replicated measurements; (2) when the data consist of matched repeated observations measured by the same observer under different conditions.
In addition, we consider two situations: (1) a symmetric assessment of agreement between two observers, and (2) an assessment of the agreement of a new observer with a reference. We propose simple methods for the estimation of the new agreement coefficients as well as their standard errors. The reliability of the estimation methods are examined via simulation studies. Two biomedical examples are used to illustrate the new concepts and methods.Table of Contents
Contents
1 Introduction.....................................................................................1
1.1 Background...................................................................................2
1.2 Existing Methods for Quantitative Data...............................................7
1.2.1 Aggregated Approaches with Unscaled Indices...................................7
1.2.1.1 Mean Squared Deviation..............................................................8
1.2.1.2 Coverage Probability and Total Deviation Index...............................10
1.2.2 Aggregated Approaches with Scaled Indices.....................................11
1.2.2.1 Intraclass Correlation Coefficient (ICC)..........................................12
1.2.2.2 Concordance Correlation Coefficient (CCC).....................................15
1.2.3 Discussion...................................................................................17
1.3 Existing Methods for Qualitative Data..................................................20
1.3.1 Kappa Statistics...........................................................................20
1.3.1.1 Two Observers and Two Categories...............................................21
1.3.1.2 Two Observers and Multiple Nominal Categories................................22
1.3.1.3 Two Observers and Multiple Ordinal Categories.................................23
1.3.1.4 Multiple Observers and Two Categories...........................................25
1.3.1.5 Multiple Observers and Multiple Categories.......................................26
1.3.2 ICC for Binary Observations.............................................................27
1.3.3 CCC for Categorical Observations.....................................................30
1.3.4 Limitations of Kappa Statistics.........................................................31
2 Coefficient of Individual Agreement........................................................37
2.1 Motivation......................................................................................38
2.2 Definition of Coefficients...................................................................40
2.2.1 CIA for Continuous Observations......................................................40
2.2.2 A General Approach for Two Observers..............................................42
2.2.3 Estimation....................................................................................44
2.2.4 Extension to More Than Two Observers..............................................44
2.3 Comparison of CIA and CCC for Replicated Quantitative Data.....................46
3 Assessing Observer Agreement for Studies Involving Binary Observations........48
3.1 Introduction.....................................................................................49
3.2 Definition of Coefficients.....................................................................51
3.2.1 Definition.......................................................................................51
3.2.2 Interpretation and Properties of the CIAs.............................................52
3.2.3 Estimation......................................................................................53
3.2.3.1 Parametric Approach......................................................................53
3.2.3.2 Nonparametric Approach.................................................................55
3.2.4 Standard Error.................................................................................60
3.3 A Latent Class Model for Diagnostic Agreement.........................................63
3.4 An Example........................................................................................69
3.4.1 Mammography Data..........................................................................69
3.4.1.1 Description...................................................................................69
3.4.1.2 Data Summary..............................................................................70
3.4.1.3 Results........................................................................................71
3.4.2 A Content Analysis...........................................................................72
3.4.2.1 Description...................................................................................72
3.4.2.2 Data Summary..............................................................................72
3.4.2.3 Results........................................................................................73
3.5 Simulations........................................................................................75
3.5.1 Simulation Process............................................................................75
3.5.2 Simulation Set-up.............................................................................76
3.5.3 Simulation Results.............................................................................76
3.6 Sample Size Calculation........................................................................81
3.6.1 Introduction.....................................................................................81
3.6.2 Individual Level.................................................................................82
3.6.2.1 Variance.......................................................................................82
3.6.2.2 Covariance....................................................................................85
3.6.3 Mean Level .....................................................................................88
3.6.4 Variance for CIAs..............................................................................89
3.6.5 Sample Size Calculation......................................................................90
3.6.6 Sample Size Calculation Simulation........................................................92
3.6.7 Sample Size Calculation Example...........................................................92
4 Assessing Observer Agreement for Studies Involving Nominal Categorical
Observations............................................................................................94
4.1 Definition of Coefficients........................................................................95
4.1.1 Definition..........................................................................................95
4.1.2 Estimation.........................................................................................97
4.1.2.1 Parametric Method...........................................................................97
4.1.2.2 Non-parametric Method.....................................................................98
4.1.3 Standard Error....................................................................................99
4.2 An Example..........................................................................................102
4.3 Simulations..........................................................................................104
4.3.1 Simulation Process..............................................................................104
4.3.1.1 Step 1: Generate Population..............................................................104
4.3.1.2 Step 2: Calculate True Values............................................................107
4.3.1.3 Step 3: Select Sample......................................................................110
4.3.1.4 Step 4: Estimate N and R..................................................................110
4.3.2 Simulation Results..............................................................................110
5 Assessing Observer Agreement for Studies Involving Ordinal Categorical
Observations............................................................................................114
5.1 Definition of Coefficients........................................................................115
5.1.1 Definition..........................................................................................115
5.1.2 Estimation.........................................................................................119
5.1.2.1 Parametric Method...........................................................................119
5.1.2.2 Non-parametric Method.....................................................................120
5.2 An Example..........................................................................................123
5.3 Simulations..........................................................................................126
5.3.1 Simulation Process..............................................................................126
5.3.2 Simulation Results...............................................................................127
6 Assessing Observer Agreement for Data with Matched Repeated
Measurements............................................................................................129
6.1 Introduction..........................................................................................130
6.2 Notations..............................................................................................132
6.3 Extended CIAs for Assessing Observer Agreement for Matched Repeated
Continuous Measurements.............................................................................133
6.4 Extended CIAs for Assessing Observer Agreement for Matched Repeated
Binary Measurements....................................................................................135
6.4.1 Definition of Coefficients........................................................................135
6.4.2 Estimation...........................................................................................137
6.4.3 Examples.............................................................................................139
6.4.3.1 Carotid Stenosis Screening Study..........................................................140
6.4.3.2 Mammography Study...........................................................................142
6.4.4 Simulations...........................................................................................143
6.4.4.1 Simulation Process...............................................................................143
6.4.4.2 Simulation Results................................................................................146
7 Summary and Future Research.......................................................................148
7.1 Summary and Discussion.............................................................................149
7.2 Future Work.............................................................................................153
Appendix.......................................................................................................157
A.1 Figures....................................................................................................158
A.2 Tables.....................................................................................................164
A.3 The moment-generating function for the Binomial distribution.............................209
Bibliography...................................................................................................211
About this Dissertation
| School | |
|---|---|
| Department | |
| Degree | |
| Submission | |
| Language |
|
| Research Field | |
| Palabra Clave | |
| Committee Chair / Thesis Advisor | |
| Committee Members |
Primary PDF
| Thumbnail | Title | Date Uploaded | Actions |
|---|---|---|---|
|
|
Assessing Observer Agreement for Categorical Observations () | 2018-08-28 10:13:27 -0400 |
|
Supplemental Files
| Thumbnail | Title | Date Uploaded | Actions |
|---|