Investigating the Neural Bases of Crossmodal Correspondences Open Access
McCormick, Kelly (Spring 2019)
Abstract
Across diverse test populations, people have been found to reliably associate stimulus features in different sensory modalities. Such 'crossmodal correspondences' have been found for a number of audiovisual domains. For example, people associate higher-pitched sounds with higher spatial elevation, compared to lower-pitched sounds which are associated with lower spatial elevation. Despite the pervasiveness of crossmodal correspondences, the neural and cognitive mechanisms underlying these phenomena are currently unknown. The overarching goal of this dissertation research has been to better understand the neural systems that provide a basis for such mappings. Three functional neuroimaging experiments were conducted to investigate the neural basis of three audiovisual correspondences: auditory pitch-visuospatial elevation, pseudoword-object shape, and pitch-object size. To identify systems sensitive to each of these correspondences, we contrasted blood-oxygen-level dependent (BOLD) activity for multimodally congruent and incongruent audiovisual stimulus couplings. In addition, three independent localizer tasks were employed to functionally define neural systems involved in multisensory integration, magnitude, and semantics, all of which have been theorized to play a role in crossmodal mappings.
The results of the pitch-elevation experiment (Chapter 2) did not indicate involvement of the functionally localized systems, although activation overlapped with the semantic control condition, possibly reflecting phonological processing. However, patterning of the congruency-related activity was consistent with a possible basis in multisensory attention. The results of the pseudoword-shape experiment (Chapter 3) provided no evidence for semantic mediation, and limited evidence for processes relating to multisensory integration and magnitude estimation as possible underlying mechanisms. Again countering our predictions, support was found for a relationship between pseudoword-object shape mapping and multisensory attention and/or phonological processing. Behavioral results in the experiment on the pitch-size experiment (Chapter 4) were heterogeneous making it difficult to meaningfully interpret neural activity. Together, these experiments provide new insight into the neural basis of the pitch-elevation and pseudoword-shape correspondences and offer a novel and generalizable approach that may be used in future research. In addition to enriching our understanding of crossmodal correspondences more generally, these findings are important for understanding the mechanisms underlying sound-symbolic mappings in language and how words and sounds are mapped to meaning in the brain.
Table of Contents
Table of Contents
Table of Contents............................................................................................................................. i
List of Tables................................................................................................................................. vii
List of Figures............................................................................................................................. viii
Chapter 1. Introduction.................................................................................................................. 1
Behavioral insights into crossmodal mappings..................................................................... 1
Possible neural bases for cross-sensory mappings................................................................ 4
Intersensory connections or multisensory convergence............................................................. 5
Multisensory integrative systems............................................................................................... 8
Magnitude system.................................................................................................................... 14
Semantic systems..................................................................................................................... 18
Overview and Rationale of the Dissertation........................................................................ 22
Elucidating sound-meaning mappings in language..................................................................... 25
Chapter 2. Crossmodal association of auditory pitch and visuospatial elevation...................... 27
Method.......................................................................................................................................... 32
Participants.............................................................................................................................. 32
Stimuli...................................................................................................................................... 33
Procedure.................................................................................................................................. 34
General.................................................................................................................................... 34
Pitch-elevation fMRI task....................................................................................................... 35
Functional localizer tasks......................................................................................................... 36
Multisensory integration localizer........................................................................................ 37
Magnitude localizer.............................................................................................................. 39
Semantic localizer................................................................................................................ 41
Post-scan behavioral testing: Implicit association of Pitch and Elevation............................ 42
Image acquisition..................................................................................................................... 44
Image processing and analysis.................................................................................................. 45
Results........................................................................................................................................... 47
Behavioral................................................................................................................................ 47
In –scanner tasks................................................................................................................. 47
Localizer tasks..................................................................................................................... 47
Pitch-elevation task............................................................................................................. 47
Post-scan pitch-elevation IAT................................................................................................ 51
Imaging..................................................................................................................................... 56
Localizer tasks......................................................................................................................... 56
Multisensory integration...................................................................................................... 56
Magnitude............................................................................................................................ 57
Semantic.............................................................................................................................. 58
Pitch-elevation task................................................................................................................. 59
Multisensory Congruency effects............................................................................................. 59
Basic congruency effect....................................................................................................... 59
Interaction of immediate trial congruency with previous trial congruency............................ 61
Regions modulated by congruency of immediate trial but not previous trial (i.e. ‘pure congruency effect’). 63
Overlap of congruency effect with localizers........................................................................... 65
Multisensory integration...................................................................................................... 66
Magnitude............................................................................................................................ 66
Semantic.............................................................................................................................. 66
Discussion..................................................................................................................................... 66
‘Pure congruency’ regions modulated by congruency of immediate trial but not previous trial: (CC>II) not overlapping with (CC>II)>(IC>CI)................................................................................................................................................... 67
Regions exhibiting both a congruency effect and interaction effect : (CC>II) overlapping with (CC>II)>(IC>CI) 74
Limitations related to task and congruency effects.................................................................. 76
Localizers................................................................................................................................. 78
Potential basis in semantic processing...................................................................................... 78
Potential basis in phonological processing............................................................................... 81
Potential basis in multisensory integration............................................................................... 82
Potential basis in magnitude processing.................................................................................... 84
Limitations and future directions......................................................................................... 86
Task limitations...................................................................................................................... 87
Localizer limitations................................................................................................................ 88
Analysis limitations................................................................................................................. 88
Theory limitations.................................................................................................................. 90
Conclusion.................................................................................................................................... 90
Chapter 3. Crossmodal association of pseudowords and object shape......................................... 92
Word-Shape mappings: The Bouba-Kiki phenomenon...................................................... 95
Mechanism.............................................................................................................................. 99
Method........................................................................................................................................ 104
Participants............................................................................................................................ 104
Stimuli................................................................................................................................... 105
Procedure................................................................................................................................ 105
General.................................................................................................................................. 105
Pseudoword-shape fMRI task................................................................................................. 106
Functional localizer tasks....................................................................................................... 109
Post-scan behavioral testing: Implicit association of pseudoword and shape........................... 109
Analysis.................................................................................................................................. 109
In-scanner behavioral data analysis........................................................................................ 109
Image processing and analysis................................................................................................ 110
Multisensory congruency analysis.......................................................................................... 110
Functional localizer analyses.................................................................................................. 110
Results......................................................................................................................................... 111
Behavioral.............................................................................................................................. 111
In-scanner tasks..................................................................................................................... 111
Localizer tasks................................................................................................................... 111
Pseudoword-shape task....................................................................................................... 112
Post-scan pseudoword-shape IAT....................................................................................... 115
Imaging................................................................................................................................... 118
Localizer tasks....................................................................................................................... 118
Multisensory synchrony..................................................................................................... 118
Magnitude.......................................................................................................................... 120
Semantic............................................................................................................................ 121
Pseudoword-Shape incongruency............................................................................................ 124
Overlap of incongruency effect with localizers...................................................................... 128
Multisensory Integration.................................................................................................... 128
Magnitude.......................................................................................................................... 128
Semantic............................................................................................................................ 128
Discussion................................................................................................................................... 129
Incongruency effects............................................................................................................. 129
Overlaps with localizers....................................................................................................... 134
Limitations and future directions....................................................................................... 136
Conclusion.................................................................................................................................. 138
Chapter 4. Crossmodal association of auditory pitch and visual object size............................. 140
Method........................................................................................................................................ 144
Participants............................................................................................................................ 144
Stimuli.................................................................................................................................... 144
Procedure................................................................................................................................ 146
Functional localizer tasks....................................................................................................... 146
Pitch-size fMRI task.............................................................................................................. 146
Post-scan behavioral testing: Implicit association of pitch and size........................................ 147
Results......................................................................................................................................... 147
Behavioral.............................................................................................................................. 147
In–scanner tasks.................................................................................................................... 147
Localizer tasks................................................................................................................... 147
Pitch-size task................................................................................................................... 148
Post-scan pitch-size IAT................................................................................................... 152
Imaging................................................................................................................................... 158
Localizer tasks....................................................................................................................... 158
Pitch-size task....................................................................................................................... 158
Overlap between incongruency effect and localizers............................................................... 159
Multisensory integration.................................................................................................... 159
Magnitude.......................................................................................................................... 159
Semantic............................................................................................................................ 159
Discussion................................................................................................................................... 159
Individual differences........................................................................................................... 160
Mechanism............................................................................................................................. 161
Conclusion.................................................................................................................................. 161
Chapter 5. Conclusion................................................................................................................ 163
Limitations and future directions....................................................................................... 166
Conclusion............................................................................................................................. 168
References.................................................................................................................................... 169
About this Dissertation
| School | |
|---|---|
| Department | |
| Degree | |
| Submission | |
| Language |
|
| Research Field | |
| Keyword | |
| Committee Chair / Thesis Advisor | |
| Committee Members |
Primary PDF
| Thumbnail | Title | Date Uploaded | Actions |
|---|---|---|---|
|
|
Investigating the Neural Bases of Crossmodal Correspondences () | 2019-04-06 02:58:24 -0400 |
|
Supplemental Files
| Thumbnail | Title | Date Uploaded | Actions |
|---|