Prosody in Speech as a Source of Referential Information: The Case of Pitch Conveying Color Brightness Open Access

Abstract

Prosody, or the timing, rhythm, and intonation of a spoken message, conveys a wealth of information crucial for effective vocal communication. In addition to informing linguistic structure and speakers' emotional state, prosody also conveys referential information that listeners integrate into their mental representations of objects and events described using spoken language. However, the communicative conditions under which speakers recruit prosody to convey referential detail and the cognitive mechanisms underlying such use of prosody have not been systematically investigated. Given evidence for non-arbitrary auditory-visual correspondences across multiple dimensions, this dissertation uses the mapping between pitch and color brightness to assess the possibility that the use of prosody to convey referential information is an instantiation of a more general level of cross-modal association that influences perceptual processing. In a series of four experiments, this dissertation examines (1) the extent to which systematic auditory-visual correspondences manifest in prosody to convey visual details of linguistic referents (Experiments 1 and 2), (2) the extent to which listeners infer referential information from these prosodic cues (Experiment 3), and (3) the extent to which communicative context modulates speakers' and listeners' use of prosody to resolve referential ambiguity (Experiment 4). I conclude that prosody can be conceptualized as a type of vocal gesture, as it provides referential detail about objects and events in the world and can be recruited to resolve ambiguity in the accompanying propositional content. That prosody persists as a source of referential information in spoken language suggests that it serves a non-redundant role alongside linguistic content to maintain a maximally efficient and expressive communicative system.

Table of Contents

I.Introduction. . . . . 1

a. Functional significance of prosody: Traditional characterizations . . . . . 2

i. Prosody and speaker emotion . . . . . 2

ii. Prosody as a cue to linguistic structure . . . . . 4

b. Encoding prosody in lexical representations . . . . .6

c. Prosodic cues to meaning . . . . . 7

i. Prosody and semantic disambiguation . . . . . 8

ii. Beyond emotional meaning . . . . . 9

iii. Prosody and word learning . . . . . 10

iv. Prosody as a source of referential information . . . . . 11

d. Neural mechanisms underlying non-arbitrary sound-meaning mappings . . . 12

e. Effects of communicative demand on prosody use . . . . . 14

i. Sources of variation in referential prosody perception and production . . . . . 15

f. Overview of the dissertation . . . . . 15

II. Experiment 1 . . . . . 16

a. Method . . . . . 17

i. Participants . . . . . 17

ii. Stimuli . . . . . 17

iii. Procedure . . . . . 17

iv. Acoustic analyses . . . . . 19

b. Results and discussion . . . . . 20

III. Experiment 2 . . . . . 22

a. Method . . . . . 22

i. Participants . . . . . 22

ii. Stimuli . . . . . 22

iii. Procedure . . . . . 22

b. Results and discussion . . . . . 23

i. F0 . . . . . 25

ii. Amplitude . . . . . 25

iii. Duration . . . . . 25

IV. Experiment 3 . . . . . 27

a. Method . . . . . 28

i. Participants . . . . . 28

ii. Stimuli . . . . . 28

iii. Procedure . . . . . 29

b. Results and discussion . . . . . 29

i. F0 . . . . . 30

ii. Amplitude . . . . . 30

iii. Duration . . . . . 31

V. Experiment 4 . . . . . 33

a. Method . . . . . 35

i. Participants . . . . . 35

ii. Stimuli . . . . . 35

iii. Procedure . . . . . 36

b. Results and discussion . . . . . 40

i. Listener performance . . . . . 40

ii. Speaker performance . . . . . 42

1. F0 . . . . . 43

2. Amplitude . . . . . 43

3. Duration . . . . . 43

iii. Relation between listener and speaker performance . . . . . 46

1. F0 . . . . . 46

2. Amplitude and duration . . . . . 47

iv. Individual differences in prosody perception and production . . . . . 49

VI. Summary of Results across Experiments . . . . . 51

VII. General Discussion . . . . . 52

a. Systematic cross-modal mappings in spoken language . . . . . 53

b. Neural substrates of cross-modal mappings in spoken language . . . . . 57

c. Referential prosody and grounded views of spoken language processing . . . 59

d. Automaticity and context-dependence of referential prosody use . . . . . 60

e. Parallels between referential prosody and co-speech gesture . . . . . 63

f. Prosody as a source of referential information: Implications for language evolution . . . . . 66

VIII.Conclusion. . . . . 69

IX.References. . . . . 70

X.Appendices. . . . . 88

About this Dissertation

Rights statement

• Permission granted by the author to include this thesis or dissertation in this repository. All rights reserved by the author. Please contact the author for information regarding the reproduction and use of this thesis or dissertation.

School	Laney Graduate School
Department	Psychology
Degree	PhD
Submission	Dissertation
Language	English
Research Field	Psychology, General Psychology, Experimental Psychology, Cognitive
Keyword	spoken language vocal gesture referential information cross-modal correspondences prosody
Committee Chair / Thesis Advisor	Nygaard, Lynne, Emory University Namy, Laura, Emory University
Committee Members	Barsalou, Lawrence, Emory University Rodman, Hillary R, Emory University Fivush, Robyn, Emory University Stout, Dietrich, Emory University

Last modified

Primary PDF

Thumbnail	Title	Date Uploaded	Actions
	Prosody in Speech as a Source of Referential Information: The Case of Pitch Conveying Color Brightness ()	2018-08-28 13:12:52 -0400	Press to Select an action Download

Supplemental Files

Thumbnail	Title	Date Uploaded	Actions