Evaluating Agreement Among Observers or Methods of Measurement forQuantitative Data Open Access

Wiener, Jeffrey Benjamin (2009)

Permanent URL: https://etd.library.emory.edu/concern/etds/1544bp48c?locale=en
Published

Abstract

Evaluating Agreement Among Observers or Methods of Measurement for Quantitative Data By Jeffrey Wiener

Agreement measures are used to compare measurements of a specific variable made by different observers or methods, and to evaluate whether a substantial difference exists between these sources of measurement. Assessing agreement is applicable to method comparison or observer reliability studies in both the biomedical and psychosocial sciences. Frequently, a reference method or gold standard exists which is considered to be the most accurate of those available.

First, we explore and evaluate multiple unscaled measures of agreement between quantitative measurements by two observers with and without replications. Two scaled coefficients of agreement based on a general disagreement function which makes no distributional assumptions are described, one for the case of no applicable reference method, and the second for the case where one observer is considered a reference. We develop methods of inference for these coefficients, evaluating them against previously developed methods, and also define the asymptotic distribution of the coefficients and assess the robustness of the estimation methods. Next, we extend the described coefficients of agreement to the case where a set of two or more observers are selected at random from a pool of potential observers.

Finally, we model agreement using a disagreement function as our outcome variable. The effects of subject-specific covariates are examined. We apply these methods to a behavioral intervention study on medication adherence in HIV-positive children and to a carotid stenosis screening method comparison study.

Table of Contents

Table of Contents

1. Introduction...1

2. Approaches for Evaluating Agreement Between Two Observers...5

2.1 Introduction and notation...5 2.2 Existing methods...6

2.2.1 Mean Squared Deviation (MSD)...6 2.2.2 Intraclass Correlation Coefficient (ICC)...7 2.2.3 Concordance Correlation Coefficient (CCC)...10 2.2.4 Total Deviation Index (TDI)...11 2.2.5 Coverage Probability (CP)...12

2.3 Evaluation of existing methods...13 2.4 Discussion...15

3. A General Approach for Evaluating Agreement Between Two Observers with Replicated Measurements...22

3.1 Introduction and notation...22 3.2 Coefficients of Individual Agreement...23 3.3 Methods of inference for the Coefficients of Individual Agreement using the MSD...28

3.3.1 Method A - assuming independence between estimated mean square errors...28 3.3.2 Method B - general method using subject-specific estimates...30 3.3.3 Method C - estimation and inference using variance components...31

3.4 Simulation study - performance and comparison of estimates...34 3.5 Sample size estimation...36 3.6 Examples...37

3.6.1 Bland and Altman Systolic Blood Pressure (SBP) Data...37 3.6.2 Carotid Stenosis Data...38

3.7 Robustness of estimates and standard errors...38 3.8 Extension to J ≥ 2 observers...40 3.9 Discussion...42

4. A General Approach for Evaulating Agreement Between Observers Selected at Random...50

4.1 Introduction...50 4.2 Coefficients of Individual Agreement for random observers...51 4.3 Estimation and inference...54

4.3.1 Estimation and inference for ψN...54 4.3.2 Estimation and inference for ψR...55 4.3.3 Estimation and inference using variance components...57

4.4 Simulation study - performance and comparison of estimates...58 4.5 Examples...60

4.5.1 Bland and Altman Blood Pressure (SBP) Data...60 4.5.2 Carotid Stenosis Data...60

4.6 Discussion...61

5. Modeling Measures of Agreement 65 5.1 Introduction and notation...65

5.2 Pediatric Impact adherence data...66 5.3 Models...67

5.3.1 Two observers with covariates - least squares method...67 5.3.2 More than two observers, no reference - mixed model...68 5.3.3 More than two observers with a reference method - mixed model...69 5.3.4 Penalized spline regression models...70

5.4 Carotid stenosis data...72 5.5 Discussion...73

6. Summary...81 Bibliography...82

About this Dissertation

Rights statement
  • Permission granted by the author to include this thesis or dissertation in this repository. All rights reserved by the author. Please contact the author for information regarding the reproduction and use of this thesis or dissertation.
School
Department
Degree
Submission
Language
  • English
Research Field
Keyword
Committee Chair / Thesis Advisor
Committee Members
Last modified

Primary PDF

Supplemental Files