Validity, Model-Data Fit, and Person Response Functions in Educational Assessment Open Access
Walker, Angela Adrienne (2016)
Published
Abstract
Background: A test score alone is not sufficient to represent a person's level of knowledge and skills on a measured construct. Validity of the interpretation and use of test scores is based on an underlying theory (APA/AERA/NCME, 2014), and measurement models based on item response theory provide one way to evaluate validity. Good model-data fit at the group and individual levels is critical for establishing the validity of test score interpretation and use. Procedures for examining model-data fit have been developed and are used in educational testing. However in practice, these procedures are limited to ensuring adequate item-level fit and global person-level fit (i.e., person fit over all test takers). Procedures ensuring adequate individual person-level fit are not conducted for most educational tests. Furthermore, communicating person fit information to educational stakeholders who use test scores to make important educational decisions is also absent.
Purpose: This study explores an approach for examining and communicating individual person-level fit. The research questions addressed by the study are
1. How do person response functions and person-level model-data fit contribute to the validation of inferences regarding person scores?
2. What existing methods of creating person response functions can be utilized in practice for validating the inferences of scores on educational tests?
Methods: A review and critique of the literature provided the conceptual foundation for the study. I built upon this foundation by conducting three empirical applications that used real and simulated test data. A two-step, statistical and graphical, procedure was used to detect and illustrate individual person misfit. Specifically, person fit was examined statistically with person fit indices and visually with person response functions.
Findings: Individual person fit analyses and person response functions together have promise for inclusion in quality checking because they can illustrate test score trustworthiness in a clear way. Person response functions can be used as a tool to help researchers and practitioners understand individual person misfit in educational tests.
Significance: Individual person fit analyses provide information that validates the claims of test score meaning. This is a necessary, and currently missing, piece of validity information in large-scale educational testing practice.
Table of Contents
Chapter One: Introduction. 1
Statement of the Problem. 6
Purpose of the Study. 7
General Research Questions. 8
Theoretical Framework. 9
Summary. 16
Chapter Two: Review of the Literature. 17
Validity: Meaning and Interpretation of Test Scores. 17
Model-data Fit and Person Fit in the Rasch Model 28
Person Response Functions and Model-Data Fit 32
Uses of Person Response Functions. 40
Summary. 53
Chapter Three: Exploring Person Fit with an Approach Based on Multilevel Logistic Regression 57
Method. 65
Results. 71
Discussion. 78
Chapter Four: Exploring Aberrant Responses Using Person Fit and Person Response Functions 83
Purpose. 84
Method. 93
Results. 96
Discussion. 101
Chapter Five: Using Person Fit Statistics and Person Response Functions to Validate Theta Estimates from Computer Adaptive Tests. 105
Purpose. 107
Method. 114
Results. 125
Discussion. 131
Chapter Six: Discussion. 137
References. 148
Appendix A. 185
Appendix B. 186
Appendix C. 187
About this Dissertation
| School | |
|---|---|
| Department | |
| Degree | |
| Submission | |
| Language |
|
| Research Field | |
| Keyword | |
| Committee Chair / Thesis Advisor | |
| Committee Members |
Primary PDF
| Thumbnail | Title | Date Uploaded | Actions |
|---|---|---|---|
|
|
Validity, Model-Data Fit, and Person Response Functions in Educational Assessment () | 2018-08-28 11:49:29 -0400 |
|
Supplemental Files
| Thumbnail | Title | Date Uploaded | Actions |
|---|