Validity, Model-Data Fit, and Person Response Functions in Educational Assessment Open Access

Walker, Angela Adrienne (2016)

Permanent URL:


Background: A test score alone is not sufficient to represent a person's level of knowledge and skills on a measured construct. Validity of the interpretation and use of test scores is based on an underlying theory (APA/AERA/NCME, 2014), and measurement models based on item response theory provide one way to evaluate validity. Good model-data fit at the group and individual levels is critical for establishing the validity of test score interpretation and use. Procedures for examining model-data fit have been developed and are used in educational testing. However in practice, these procedures are limited to ensuring adequate item-level fit and global person-level fit (i.e., person fit over all test takers). Procedures ensuring adequate individual person-level fit are not conducted for most educational tests. Furthermore, communicating person fit information to educational stakeholders who use test scores to make important educational decisions is also absent.

Purpose: This study explores an approach for examining and communicating individual person-level fit. The research questions addressed by the study are

1. How do person response functions and person-level model-data fit contribute to the validation of inferences regarding person scores?

2. What existing methods of creating person response functions can be utilized in practice for validating the inferences of scores on educational tests?

Methods: A review and critique of the literature provided the conceptual foundation for the study. I built upon this foundation by conducting three empirical applications that used real and simulated test data. A two-step, statistical and graphical, procedure was used to detect and illustrate individual person misfit. Specifically, person fit was examined statistically with person fit indices and visually with person response functions.

Findings: Individual person fit analyses and person response functions together have promise for inclusion in quality checking because they can illustrate test score trustworthiness in a clear way. Person response functions can be used as a tool to help researchers and practitioners understand individual person misfit in educational tests.

Significance: Individual person fit analyses provide information that validates the claims of test score meaning. This is a necessary, and currently missing, piece of validity information in large-scale educational testing practice.

Table of Contents

Chapter One: Introduction. 1

Statement of the Problem. 6

Purpose of the Study. 7

General Research Questions. 8

Theoretical Framework. 9

Summary. 16

Chapter Two: Review of the Literature. 17

Validity: Meaning and Interpretation of Test Scores. 17

Model-data Fit and Person Fit in the Rasch Model 28

Person Response Functions and Model-Data Fit 32

Uses of Person Response Functions. 40

Summary. 53

Chapter Three: Exploring Person Fit with an Approach Based on Multilevel Logistic Regression 57

Method. 65

Results. 71

Discussion. 78

Chapter Four: Exploring Aberrant Responses Using Person Fit and Person Response Functions 83

Purpose. 84

Method. 93

Results. 96

Discussion. 101

Chapter Five: Using Person Fit Statistics and Person Response Functions to Validate Theta Estimates from Computer Adaptive Tests. 105

Purpose. 107

Method. 114

Results. 125

Discussion. 131

Chapter Six: Discussion. 137

References. 148

Appendix A. 185

Appendix B. 186

Appendix C. 187

About this Dissertation

Rights statement
  • Permission granted by the author to include this thesis or dissertation in this repository. All rights reserved by the author. Please contact the author for information regarding the reproduction and use of this thesis or dissertation.
  • English
Research Field
Committee Chair / Thesis Advisor
Committee Members
Last modified

Primary PDF

Supplemental Files