The Generative Nature of Commonsense Knowledge: Insights from Machine Learning Open Access

Abstract

The study of commonsense has received little attention for lack of accounts of how it might be

represented. Recent advances in machine learning are resulting in rich knowledge bases that

(unwittingly) might offer some insight into this elusive phenomenon. This paper assesses one precomputed

model, RoBERTa, for suitability as a working model of human commonsense knowledge by

testing it against variation in human agreement. We examine the contribution of statistical and

structural properties of language to this performance, including frequency and cooccurrence-based

representations, as well as part of speech and syntactic structure. We conclude that RoBERTa is a

suitable model for language prediction: the model’s predictions closely reflected human agreement and

cannot be explained by simple linguistic features. In investigating the range of possible responses to a

particular context, we find that these responses illustrate the impact of categorical organization on

precise context sensitivity and conclude that this demonstrates the hallmarks of commonsense

knowledge. After exploring the contribution of the static component of RoBERTa’s knowledge, the main

finding of this paper is that the knowledge base that directly facilitates both human agreement and the

model’s measure of fit is by its very nature generative, and only truly exists in representation as it is

applied. This paper discusses the role of implicit learning and predictive processing as potential

frameworks within which to substantiate this meta-theoretic observation.

Table of Contents

TABLE OF CONTENTS

ABSTRACT 2

INTRODUCTION 4

THE CONTRIBUTIONS OF LINGUISTIC STRUCTURE AND DISTRIBUTION 9

METHODS 10

   PARTICIPANTS 10

   MATERIALS 10

   PROCEDURE 12

RESULTS 13

DISCUSSION 15

CONTEXT-SENSITIVITY AND THE STRUCTURE OF THE KNOWLEDGE BASE 15

METHODS 17

   PARTICIPANTS 17

   MATERIALS 17

   PROCEDURE 17

RESULTS 17

DISCUSSION 19

UNDERSTANDING ROBERTA’S KNOWLEDGE BASE 19

METHODS 23

   PARTICIPANTS 23

   MATERIALS 23

   PROCEDURE 23

RESULTS 24

   CLOZE TASK 24

   SIMILARITY ANALYSES 26

DISCUSSION 27

GENERAL DISCUSSION 27

CONCLUSION 29

REFERENCES 30

About this Master's Thesis

Rights statement

• Permission granted by the author to include this thesis or dissertation in this repository. All rights reserved by the author. Please contact the author for information regarding the reproduction and use of this thesis or dissertation.

School	Laney Graduate School
Department	Psychology
Subfield / Discipline	Cognition and Development
Degree	M.S.
Submission	Master's Thesis
Language	English
Research Field	Psychology, Cognitive Language, Linguistics
Keyword	Semantic Memory Artificial Intelligence Natural Language Processing Commonsense Knowledge
Committee Chair / Thesis Advisor	Phillip Wolff, Emory University Robyn Fivush, Emory University Harold Gouzoules, Emory University

Last modified

Primary PDF

Thumbnail	Title	Date Uploaded	Actions
	The Generative Nature of Commonsense Knowledge: Insights from Machine Learning ()	2020-10-27 09:42:18 -0400	Press to Select an action Download

Supplemental Files

Thumbnail	Title	Date Uploaded	Actions