Characterizing and Mitigating Human Bias in Holistic Reviews with Interactive Visualization Open Access

Abstract

People are susceptible to various forms of bias, which can have a significant impact in high-stakes decision-making contexts where subjective judgment is involved. In domains like university admissions and job hiring, biases such as implicit racial or gender bias can contribute to undesirable outcomes such as unfair evaluations of candidates with comparable qualifications, leading to unequal access to opportunities and systematic underrepresentation of certain groups. Although education-based methods, such as training courses, have been employed to mitigate these biases, their effectiveness often remains limited due to the subconscious nature of such biases, i.e., individuals are often unaware that they are behaving differently because these biases are unintentional. As a result, people often lack the real-time awareness needed to recognize and correct biased behavior during actual decision-making processes. In this dissertation, I aim to fill this gap by developing real-time mitigation strategies that leverage interactive visualizations to support reviewer awareness during the decision-making process. My research focuses on bias in holistic review processes, including bias embedded in application materials like letters of recommendation and bias reflected in reviewer behavior during applicant evaluations, across both university admissions and hiring settings. By surfacing patterns in language and interaction through visualization, this work supports timely reflection and promotes fairer evaluation practices. Collectively, the work in this dissertation contributes both practical tools and generalizable design principles to foster bias-aware decision-making in high-stakes evaluative contexts.

Table of Contents

1 Introduction 1

1.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

1.2 Thesis Statement and Research Questions . . . . . . . . . . . . . . . 3

1.3 Summary of Dissertation Research . . . . . . . . . . . . . . . . . . . 4

2 Related Work 7

2.1 Implicit Association . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

2.2 Bias in Admissions and Hiring . . . . . . . . . . . . . . . . . . . . . . 8

2.2.1 Bias in Letters of Recommendation . . . . . . . . . . . . . . . 9

2.3 Bias Mitigation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

2.3.1 Bias Mitigation in Visual Analytics . . . . . . . . . . . . . . . 11

3 Characterizing and Mitigating Bias in Letters of Recommendation 13

3.1 Experimental Design: Overview . . . . . . . . . . . . . . . . . . . . . 15

3.1.1 Task and Procedure . . . . . . . . . . . . . . . . . . . . . . . 15

3.1.2 Materials . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

3.2 Preliminary Studies . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

3.2.1 Refining the Dictionary . . . . . . . . . . . . . . . . . . . . . . 20

3.2.2 Testing Interventions . . . . . . . . . . . . . . . . . . . . . . . 21

3.2.3 Determining Sample Size . . . . . . . . . . . . . . . . . . . . . 23

3.3 Main Study Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

3.3.1 Participants . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

3.3.2 Stimuli and Conditions . . . . . . . . . . . . . . . . . . . . . . 25

3.3.3 Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

3.4 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

3.4.1 Effects of Letter Language and Interventions . . . . . . . . . . 28

3.4.2 Association between IAT Score and Evaluation Behavior . . . 30

3.4.3 Feedback on Interventions. . . . . . . . . . . . . . . . . . . . . 31

3.4.4 Exploratory Analyses . . . . . . . . . . . . . . . . . . . . . . . 34

3.5 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

3.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39

4 Characterizing and Mitigating Bias in Application Review Behavior

in University Admissions 40

4.1 Formative Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

4.1.1 Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

4.1.2 Prior Review Process . . . . . . . . . . . . . . . . . . . . . . . 43

4.1.3 Process Awareness Needs . . . . . . . . . . . . . . . . . . . . . 44

4.1.4 Design Goals . . . . . . . . . . . . . . . . . . . . . . . . . . . 44

4.2 System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45

4.2.1 Rating Page . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45

4.2.2 Summary Page . . . . . . . . . . . . . . . . . . . . . . . . . . 47

4.3 Controlled Study Methodology . . . . . . . . . . . . . . . . . . . . . . 49

4.3.1 Participants . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49

4.3.2 Dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49

4.3.3 Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50

4.3.4 Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51

4.3.5 Hypotheses . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51

4.4 Controlled Study Results . . . . . . . . . . . . . . . . . . . . . . . . . 52

4.4.1 H1: Effects of Applicant Gender and Race . . . . . . . . . . . 53

4.4.2 H2: Implicit Biases . . . . . . . . . . . . . . . . . . . . . . . . 54

4.4.3 H3: Effect of the Intervention . . . . . . . . . . . . . . . . . . 55

4.5 Case Study Methodology . . . . . . . . . . . . . . . . . . . . . . . . . 59

4.5.1 Dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60

4.5.2 Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61

4.5.3 Analysis and Coding . . . . . . . . . . . . . . . . . . . . . . . 61

4.6 Case Study Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62

4.6.1 Awareness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62

4.6.2 Interaction Analysis . . . . . . . . . . . . . . . . . . . . . . . 64

4.6.3 System Usability . . . . . . . . . . . . . . . . . . . . . . . . . 67

4.7 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68

4.8 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74

5 Characterizing and Mitigating Bias in Application Review Behavior

in Job Hiring 75

5.1 Experimental Design . . . . . . . . . . . . . . . . . . . . . . . . . . . 76

5.1.1 Participants . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77

5.1.2 Dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77

5.1.3 Applicant Demographic Characteristics Assignment . . . . . . 79

5.1.4 Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80

5.1.5 Hypotheses . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81

5.2 Study Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82

5.2.1 H1: Effects of Applicant Gender and Race . . . . . . . . . . . 82

5.2.2 H2: Implicit Biases . . . . . . . . . . . . . . . . . . . . . . . . 84

5.2.3 H3: Effect of the Intervention . . . . . . . . . . . . . . . . . . 85

5.3 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90

5.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92

6 Discussion and Future Work 94

6.1 The Role of Visualization Beyond Self-Awareness . . . . . . . . . . . 94

6.2 Interpretation of Reviewer Behavior . . . . . . . . . . . . . . . . . . . 95

6.3 Positional Reflections . . . . . . . . . . . . . . . . . . . . . . . . . . . 96

6.4 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97

7 Conclusion 100

Bibliography 102

About this Dissertation

Rights statement

• Permission granted by the author to include this thesis or dissertation in this repository. All rights reserved by the author. Please contact the author for information regarding the reproduction and use of this thesis or dissertation.

School	Laney Graduate School
Department	Computer Science and Informatics
Degree	Ph.D.
Submission	Dissertation
Language	English
Research Field	Computer Science
Keyword	Bias Mitigation Interactive Visualization
Committee Chair / Thesis Advisor	Emily Wall, Emory University
Committee Members	Ho, Joyce , Emory University Klein, Lauren , Emory University Kulkarni, Chinmay , Emory University

Last modified

Primary PDF

Thumbnail	Title	Date Uploaded	Actions
	Characterizing and Mitigating Human Bias in Holistic Reviews with Interactive Visualization ()	2025-11-02 00:08:40 -0400	Press to Select an action Download

Supplemental Files

Thumbnail	Title	Date Uploaded	Actions