The sound of meaning: Physical, perceptual, and neural correlates of sound to shape mapping in natural language Open Access
List, Sara (Spring 2019)
Published
Abstract
One of the fundamental properties of language is the assumed arbitrariness of the relationship between sound and meaning. Recent behavioral, linguistic, and neuroscience studies have challenged this assumption through the demonstration of sound-symbolic crossmodal correspondences, or non-arbitrary associations between sound and meaning. Research, including psychophysical and preliminary fMRI studies, indicates that sound symbolism may result from multisensory interactions within the brain. This set of studies investigated three questions that are central to the understanding of sound-symbolic crossmodal correspondences. First, we question which physical parameters in visual and auditory stimulus space elicit sound symbolic crossmodal correspondences. Second, we question whether the neural underpinnings of the perception of sound-symbolic crossmodal correspondences suggest that the linguistic advantages of sound symbolism are deeply ingrained in our sensory systems. Third, we investigate how these physical parameters and neural correlates are modulated by individual variation in perceptual capabilities. Using the mappings between auditory pseudowords (e.g. ‘loh-moh’) andvisual shapes (e.g. a blob), these three ideas are investigated in typical individuals and one synesthete with psychophysical and functional neuroimaging paradigms. In our first set of studies, we compared the influence of acoustic and visual parameters on previously collected sound-shape judgments of systematically constructed auditory pseudowords and visual shapes. In our second set of studies, we conducted a neuroimaging study designed to systematically determine how auditory, visual, and multisensory neural processing contribute to the perception of sound-symbolic crossmodal correspondences using both univariate and multivariate analysis methods. In our third set of studies, we investigate the behavioral and neural evidence of variation in the perception of sound-symbolic CCs. We first explore the variation present in a group of typical individuals (n = 24) who have all completed the neural and behavioral paradigms from our second set of studies. We then assess one individual with grapheme-color synesthesia in order to explore the validity of the idea that sound-symbolic CCs arise from grounded cognition or pervasive synesthetic associations. Taken together, this research provides insights into how humans produce and interpret meaning in natural language as well as the sensory basis of associations between sound and meaning.
Table of Contents
Chapter 1: Introduction.....................................................................................1
1.1 Arbitrariness in Language.......................................................................1
1.1.1 Sound symbolism......................................................................4
1.2 Auditory-Visual Multisensory Integration...................................................11
1.2.1 Guiding principles of multisensory integration..................................13
1.2.2 The neural architecture of multisensory integration.............................16
1.2.3 Crossmodal correspondences.......................................................25
1.3 Physical Parameters of Sound-symbolic Crossmodal Correspondences................27
1.4 Neural Underpinnings of Sound-symbolic Crossmodal Correspondences...............29
1.5 Synesthesia and Individual Perceptual Differences........................................31
1.6 Univariate and Multivariate Functional Magnetic Resonance Imaging (fMRI)........33
1.6.1 Standard univariate approaches...................................................34
1.6.2 Multivariate analysis approaches...................................................35
1.7 Dissertation Project.............................................................................38
Chapter 2: Acoustic and Visual Stimulus Parameters Driving Sound-symbolic Crossmodal Correspondences.....................................................................................41
2.1 Abstract...........................................................................................41
2.2 Introduction......................................................................................42
2.3 Materials and Methods...........................................................................45
2.3.1 Participants............................................................................45
2.3.2 Stimuli.................................................................................46
2.3.3 Likert-type rating tasks..............................................................47
2.3.4 Data processing and analysis.......................................................48
2.4 Results.............................................................................................57
2.4.1 Comparison of visual and auditory perceptual ratings..........................57
2.4.2 Comparison of visual perceptual ratings to visual shape indices..............58
2.4.3 Comparison of auditory perceptual ratings to acoustic parameters of pseudowords...........................................................................60
2.4.4 Comparison of indices of visual and acoustic stimuli...........................62
2.4.5 Correlations between significant acoustic parameters and auditory perceptual ratings.....................................................................65
2.5 Discussion.........................................................................................68
2.5.1 Sound-symbolic crossmodal correspondences and perception...................68
2.5.2 Sound symbolism in language acquisition.......................................74
2.6 Conclusion........................................................................................76
Chapter 3: Neural Underpinnings of Sound-symbolic Crossmodal Correspondences.......78
3.1 Abstract..........................................................................................78
3.2 Introduction......................................................................................79
3.3 Materials and Methods..........................................................................85
3.3.1 Participants............................................................................85
3.3.2 General.................................................................................85
3.3.3 Unisensory visual and unisensory auditory fMRI tasks........................86
3.3.4 Multisensory fMRI task..............................................................87
3.3.5 Post-scan behavioral testing..........................................................90
3.3.6 Image acquisition.......................................................................91
3.3.7 Image processing and univariate analysis.........................................91
3.3.8 Multivariate analysis..................................................................94
3.4 Results.............................................................................................96
3.4.1 Behavioral.............................................................................96
3.4.2 Imaging................................................................................99
3.5 Discussion.......................................................................................120
3.6 Conclusion.......................................................................................126
Chapter 4: Synesthesia & Individual Differences in Perception of Sound-symbolic Crossmodal Correspondences..................................................................128
4.1 Abstract..........................................................................................128
4.2 Introduction.....................................................................................129
4.2.1 Individual variation in multisensory integration................................129
4.2.2 Synesthesia as an extreme individual perceptual difference..................131
4.3 Materials and Methods..........................................................................136
4.3.1 Typical individuals...................................................................136
4.3.2 Synesthete.............................................................................137
4.4 Results...........................................................................................137
4.4.1 Typical variation in perception from behavioral analysis.....................137
4.4.2 Synesthetic variation in perception from behavioral analysis................139
4.4.3 Neural variation across individuals...............................................140
4.5 Discussion.......................................................................................151
4.6 Conclusion.......................................................................................153
Chapter 5: General Conclusions and Discussion.....................................................155
5.1 Summary of Findings.............................................................................155
5.2 Future Directions..................................................................................170
5.3 Applications of This Research..................................................................173
5.4 Conclusions.......................................................................................177
References....................................................................................................179
Appendix 1: Visual Shape Stimuli for Experiments......................................................239
Appendix 2: Auditory Pseudoword Stimuli for Experiments............................................248
About this Dissertation
| School | |
|---|---|
| Department | |
| Subfield / Discipline | |
| Degree | |
| Submission | |
| Language |
|
| Research Field | |
| Keyword | |
| Committee Chair / Thesis Advisor | |
| Committee Members |
Primary PDF
| Thumbnail | Title | Date Uploaded | Actions |
|---|---|---|---|
|
|
The sound of meaning: Physical, perceptual, and neural correlates of sound to shape mapping in natural language () | 2019-04-12 14:14:35 -0400 |
|
Supplemental Files
| Thumbnail | Title | Date Uploaded | Actions |
|---|