Can NLP Models Aid in Behavioral Economics Decision-Making? Open Access

Ahmed, Junaid (Spring 2021)

Permanent URL:


Natural Language Processing (NLP) models have seen rapid improvements in the last two years. Literature has indicated that these models are capable of reasoning, and in certain cases, reason better than humans. While behavioral economics tends to focus exclusively on human subjects, this study seeks to evaluate how NLP models fare in comparison to human subjects during cognitive bias tasks. More specifically, we evaluate how RoBERTa responds to fill-in-the-blank questions based on the conjunction fallacy. We use the conjunction fallacy due to its mathematical falsifiability and ease-of-testing. The hypothesis guiding this study is that RoBERTa outperforms human subjects. From this study, we conclude that RoBERTA does not outperform human subjects in aggregate, but shows promise for individuals prone to the conjunction fallacy, suggesting that there is value in future research. Moving forward, we recommend that other NLP models undergo similar tests across a greater range of cognitive biases to more accurately assess whether there is potential for using NLP models as external aids to decision making.

Table of Contents

1 Introduction 1

2 Background 2

2.1 The Model 2

2.1.1 Machine Learning 2

2.1.2 Natural Language Processing (NLP) 3

2.1.3 RoBERTa 3

2.1.4 Evidence for Logical and Inferential Reasoning Capabilities 5

2.1.5 Testing for Logical and Inferential Reasoning Capabilities 7

2.1.6 Analysis of Tests 14

2.2 Conjunction Fallacy 17

2.2.1 Justification 17

2.2.2 Application 18

3 Methodology 19

3.1 Hypothesis 19

3.2 Experimental Design 20

3.2.1 Questions for RoBERTa 20

3.2.2 Questions for Survey Participants 21

4 Data 22

5 Discussion 30

5.1 Results 30

5.1.1 Hypothesis Testing 30

5.2 Sources of Error 32

6 Conclusion 33

6.1 Ethical Concerns 34

7 Bibliography 35

8 Appendix 40

About this Honors Thesis

Rights statement
  • Permission granted by the author to include this thesis or dissertation in this repository. All rights reserved by the author. Please contact the author for information regarding the reproduction and use of this thesis or dissertation.
  • English
Research Field
Committee Chair / Thesis Advisor
Committee Members
Last modified

Primary PDF

Supplemental Files