A New Permutation-Based Method for Assessing Observer Agreement with Replicated and Repeated Observations Open Access
Pan, Yi (2011)
Abstract
Abstract
The area of observer agreement has rapidly developed over the last
half-century. A substantial number of coefficients and approaches
have been developed and used to assess the agreement between
different observers or methods of measurement. In this
dissertation, a new permutation-based coefficient for the
evaluation of agreement between two observers making replicated and
repeated binary or quantitative measurements is introduced. The new
coefficient of individual equivalence (CIE) compares the observed
disagreement between the observers to its expected value under the
hypothesis of individual equivalence. This hypothesis states that
for each subject, the conditional distributions of the readings of
the two observers are identical. Therefore, from a statistical
viewpoint, it does not matter which observer makes the reading on
this subject. In other words, under individual equivalence the
observers can be used interchangeably.
Let K and L denote the number of replicated observations that
are available from observers X and Y, respectively, on a given
subject. Then the expected disagreement under individual
equivalence for this subject is based on the K +L choose K possible
assignments of X 's and Y 's to the K +L observations made on this
subject. Under individual equivalence, all these assignments have
the same probability. Simple methods for nonparametric estimation
of the new coefficient and its standard error are derived for both
binary and continuous outcomes. Furthermore, model-based approaches
are developed for estimation of the CIE for binary and continuous
assessments. Model-based methods for estimation of the CIE from
repeated binary outcomes are also
discussed. Simulation studies confirm the validity of the estimated
coefficient and its standard error. Finally, the new coefficient is
compared with the of individual agreement (CIA), Kappa statistic
and the concordance correlation coefficient(CCC). Examples with
binary and continuous outcomes are used to illustrate the new
coefficient. One example involves the evaluation of mammograms by
ten radiologists and another one compares magnetic resonance
angiography (MRA) techniques for noninvasive screening of carotid
stenosis to an invasive intra-arterial angiogram (IA)
method.
Table of Contents
1 Background and Motivation 1
1.1 What is Agreement Study? . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Motivating Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.2.1 Mammogram Example . . . . . . . . . . . . . . . . . . . . . . 4
1.2.2 Carotid Stenosis Example . . . . . . . . . . . . . . . . . . . . 5
1.3 Existing Methods for Discrete Outcomes . . . . . . . . . . . . . . . . 6
1.3.1 The Cohen's Kappa Coefficient . . . . . . . . . . . . . . . . . 6
1.3.2 The Weighted Kappa Coefficient . . . . . . . . . . . . . . . . 10
1.3.3 Critiques on Kappa Coefficients . . . . . . . . . . . . . . . . . 12
1.4 Existing Methods for Continuous Outcomes . . . . . . . . . . . . . . 16
1.4.1 Limits of Agreement . . . . . . . . . . . . . . . . . . . . . . . 17
1.4.2 Intraclass Correlation Coefficient . . . . . . . . . . . . . . . . 18
1.4.3 Concordance Correlation Coefficient . . . . . . . . . . . . . . . 21
1.4.4 Coefficient of Individual Agreement . . . . . . . . . . . . . . . 23
1.5 Existing Methods for Repeated Binary Outcomes . . . . . . . . . . . 26
1.5.1 Logistic Regression Modeling Agreement Proportion . . . . . . 26
1.5.2 Extended Kappa Coefficient . . . . . . . . . . . . . . . . . . . 27
1.5.3 Other Agreement Measurement for Repeated Binary Measurements
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
1.5.4 Agreement Measurement for Repeated Continuous Measurements 29
1.6 Bayesian Approaches for Evaluating Agreement . . . . . . . . . . . . 30
1.7 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
2 Introduction of the Coefficient of Individual Equivalence 36
2.1 Motivation for introducing the Coefficient of Individual Equivalence
(CIE) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
2.2 Overview and Definition of the Coefficient of Individual Equivalence . 38
2.3 An Alternative Expression for the CIE . . . . . . . . . . . . . . . . . 41
2.4 Nonparametric Estimation of the CIE . . . . . . . . . . . . . . . . . . 41
2.5 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
3 Coefficient of Individual Equivalence for Replicated Binary Measurements
46
3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
3.2 The Coefficient of Individual Equivalence for Binary Observation . . . 47
3.2.1 Non-Parametric Estimation of CIE for G(X, Y ) = E(X−Y )2 =
P(X 6= Y ) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
3.2.2 Minimum and Maximum Values of CIE . . . . . . . . . . . . . 49
3.2.2.1 Minimum Value of CIE . . . . . . . . . . . . . . . . 49
3.2.2.2 Maximum Value of CIE . . . . . . . . . . . . . . . . 49
3.3 The Adjusted CIE (Definition and Estimation) . . . . . . . . . . . . . 51
3.3.1 General Definition . . . . . . . . . . . . . . . . . . . . . . . . 51
3.3.2 The Standard Error of the Adjusted CIE . . . . . . . . . . . . 51
3.3.3 Asymptotic Property of CIE . . . . . . . . . . . . . . . . . . . 52
3.3.4 Interpretation of CIEA . . . . . . . . . . . . . . . . . . . . . . 54
3.4 Simulation Studies . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
3.4.1 Data Generation . . . . . . . . . . . . . . . . . . . . . . . . . 54
3.4.2 Simulation Process . . . . . . . . . . . . . . . . . . . . . . . . 55
3.4.3 Simulation Results . . . . . . . . . . . . . . . . . . . . . . . . 56
3.5 An Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
3.6 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
4 Coefficient of Individual Equivalence for Quantitative Replicated
Measurements 68
4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
4.2 Definition and Estimation of the Coefficient of Individual Equivalence
(CIE) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
4.2.1 Estimation of CIE with G(X, Y ) = E(X − Y )2 using Linear
Mixed Effects Models for Normally Distributed Observations . 69
4.2.2 Minimum and Maximum Values of CIE . . . . . . . . . . . . . 71
4.2.2.1 Minimum Value of CIE . . . . . . . . . . . . . . . . 71
4.2.2.2 Maximum Value of CIE . . . . . . . . . . . . . . . . 71
4.3 Adjusted CIE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
4.3.1 Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
4.3.2 Large Sample Distribution and Standard Error of d CIEA . . . 72
4.3.3 Interpretation of CIEA . . . . . . . . . . . . . . . . . . . . . . 73
4.4 Simulation Studies . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
4.5 Carotid Stenosis Example . . . . . . . . . . . . . . . . . . . . . . . . 79
4.6 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
5 Comparison of CIE/CIEA to CIA, Kappa and CCC 87
5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
5.2 CIEA vs. CIA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88
5.2.1 Equality of d CIEA and dCIA when K=L . . . . . . . . . . . . 88
5.2.1.1 Binary Measurements . . . . . . . . . . . . . . . . . 88
5.2.1.2 Continuous Measurements: Nonparametric Estimation 89
5.2.1.3 Continuous Measurements: Parametric Estimation using
Full Model Only . . . . . . . . . . . . . . . . . . 90
5.2.2 Comparison between d CIEA and dCIA when K 6= L . . . . . . 91
5.3 CIEA vs. Kappa for Binary Observations . . . . . . . . . . . . . . . . 93
5.4 CIEA, CIA vs. CCC for Quantitative Data . . . . . . . . . . . . . . . 96
6 Model-based Estimation of the Coefficient of Individual Equivalence
for Replicated and Repeated Binary Measurements 99
6.1 Model-Based Estimation of CIEA for Replicated Binary Outcomes . . 100
6.1.1 Identity Link . . . . . . . . . . . . . . . . . . . . . . . . . . . 101
6.1.1.1 Estimation of CIEA using identity link . . . . . . . . 101
6.1.1.2 Variance Components Approach with Identity Link . 102
6.1.2 Logit Link . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104
6.1.2.1 Cumulative Gaussian Approximation . . . . . . . . . 105
6.1.2.2 Adaptive Gauss-Hermite Quadrature . . . . . . . . . 108
6.1.3 Bayesian Method . . . . . . . . . . . . . . . . . . . . . . . . . 111
6.1.4 Simulations . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112
6.1.5 Mammogram Study . . . . . . . . . . . . . . . . . . . . . . . . 118
6.2 CIEA for Repeated Binary Outcomes . . . . . . . . . . . . . . . . . . 119
6.2.1 Repeated Outcomes with the Identity Link . . . . . . . . . . . 120
6.2.2 Repeated Outcomes with the Logit Link . . . . . . . . . . . . 122
6.2.3 Simulation Studies . . . . . . . . . . . . . . . . . . . . . . . . 126
6.2.3.1 Data Generation . . . . . . . . . . . . . . . . . . . . 126
6.2.3.2 True Values . . . . . . . . . . . . . . . . . . . . . . . 127
6.2.4 Mammogram Study . . . . . . . . . . . . . . . . . . . . . . . . 132
6.3 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135
7 Summary and Future Research 138
Bibliography 141
About this Dissertation
| School | |
|---|---|
| Department | |
| Degree | |
| Submission | |
| Language |
|
| Research Field | |
| Keyword | |
| Committee Chair / Thesis Advisor | |
| Committee Members |
Primary PDF
| Thumbnail | Title | Date Uploaded | Actions |
|---|---|---|---|
|
|
A New Permutation-Based Method for Assessing Observer Agreement with Replicated and Repeated Observations () | 2018-08-28 11:50:22 -0400 |
|
Supplemental Files
| Thumbnail | Title | Date Uploaded | Actions |
|---|