What Transformers Might Know About the Physical World: T5 and the Origins of Knowledge Public

Haohan, Shi (Spring 2021)

Permanent URL: https://etd.library.emory.edu/concern/etds/c534fp98b?locale=fr
Published

Abstract

We find that knowledge of animals and objects' physical properties does not depend on direct or indirect perceptual experience. Rather, such knowledge can emerge from inferential processes driven by the statistical properties of language. Here we investigate the latent knowledge of the T5 encoder-decoder model with respect to various physical properties of animals and objects. Such networks represent model organisms for the origins of knowledge in a learning system without innate knowledge or access to perceptual information. We proposed and evaluated three hypotheses about what T5 might know about the physical world: 1) that T5 might understand physical dimensions much like humans understand these dimensions, 2) that it has no understanding of the perceptual dimensions of experience, and 3) that it understands some dimensions of experience better than others and potentially uses the better-understood dimensions to understand the less well-understood dimensions. The results from Study 1- 4 show that knowledge of the size, weight, and shape—but not color— agrees closely with that of humans. Moreover, agreement with human judges increased with the network's size, suggesting that disagreement with humans might ultimately disappear as the size of the networks is increased. However, Study 5 shows that T5's understanding of size might rely on its understanding of weight, which supports our third hypothesis regarding T5's understanding of dimensions in physical world. 

Table of Contents

1 Introduction 1

1.1 Related Works . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

1.2 Transformers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

2 Experiment 10

2.1 Study 1: T5’s Knowledge in Animal Size . . . . . . . . . . . . . . .10

2.1.1 Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .11

2.1.2 Results and Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .13

2.2 Study 2: T5’s Knowledge in Animal Weight . . . . . . . . . . . .14

2.2.1 Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .14

2.2.2 Results and Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .15

2.3 Study 3: T5’s Knowledge in Animal Shape . . . . . . . . . . . . .15

2.3.1 Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .16

2.3.2 Results and Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .18

2.4 Study 4: T5’s Knowledge in Animal Color . . . . . . . . . . . . . .19

2.4.1 Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

2.4.2 Results and Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .20

2.5 Study 5: T5’s Knowledge in Object Weight and Size . . . .20

2.5.1 Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

2.5.2 Results and Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .22

3 General Discussion 24

References 27

About this Honors Thesis

Rights statement
  • Permission granted by the author to include this thesis or dissertation in this repository. All rights reserved by the author. Please contact the author for information regarding the reproduction and use of this thesis or dissertation.
School
Department
Subfield / Discipline
Degree
Submission
Language
  • English
Research Field
Mot-clé
Committee Chair / Thesis Advisor
Committee Members
Dernière modification

Primary PDF

Supplemental Files