Investigating the Neural Bases of Crossmodal Correspondences Open Access

McCormick, Kelly (Spring 2019)

Permanent URL: https://etd.library.emory.edu/concern/etds/76537237v?locale=en
Published

Abstract

Across diverse test populations, people have been found to reliably associate stimulus features in different sensory modalities. Such 'crossmodal correspondences' have been found for a number of audiovisual domains. For example, people associate higher-pitched sounds with higher spatial elevation, compared to lower-pitched sounds which are associated with lower spatial elevation. Despite the pervasiveness of crossmodal correspondences, the neural and cognitive mechanisms underlying these phenomena are currently unknown. The overarching goal of this dissertation research has been to better understand the neural systems that provide a basis for such mappings. Three functional neuroimaging experiments were conducted to investigate the neural basis of three audiovisual correspondences: auditory pitch-visuospatial elevation, pseudoword-object shape, and pitch-object size. To identify systems sensitive to each of these correspondences, we contrasted blood-oxygen-level dependent (BOLD) activity for multimodally congruent and incongruent audiovisual stimulus couplings. In addition, three independent localizer tasks were employed to functionally define neural systems involved in multisensory integration, magnitude, and semantics, all of which have been theorized to play a role in crossmodal mappings. 

The results of the pitch-elevation experiment (Chapter 2) did not indicate involvement of the functionally localized systems, although activation overlapped with the semantic control condition, possibly reflecting phonological processing. However, patterning of the congruency-related activity was consistent with a possible basis in multisensory attention. The results of the pseudoword-shape experiment (Chapter 3) provided no evidence for semantic mediation, and limited evidence for processes relating to multisensory integration and magnitude estimation as possible underlying mechanisms. Again countering our predictions, support was found for a relationship between pseudoword-object shape mapping and multisensory attention and/or phonological processing. Behavioral results in the experiment on the pitch-size experiment (Chapter 4) were heterogeneous making it difficult to meaningfully interpret neural activity. Together, these experiments provide new insight into the neural basis of the pitch-elevation and pseudoword-shape correspondences and offer a novel and generalizable approach that may be used in future research. In addition to enriching our understanding of crossmodal correspondences more generally, these findings are important for understanding the mechanisms underlying sound-symbolic mappings in language and how words and sounds are mapped to meaning in the brain.  

Table of Contents

Table of Contents

Table of Contents............................................................................................................................. i

List of Tables................................................................................................................................. vii

List of Figures............................................................................................................................. viii

Chapter 1. Introduction.................................................................................................................. 1

Behavioral insights into crossmodal mappings..................................................................... 1

Possible neural bases for cross-sensory mappings................................................................ 4

Intersensory connections or multisensory convergence............................................................. 5

Multisensory integrative systems............................................................................................... 8

Magnitude system.................................................................................................................... 14

Semantic systems..................................................................................................................... 18

Overview and Rationale of the Dissertation........................................................................ 22

Elucidating sound-meaning mappings in language..................................................................... 25

Chapter 2. Crossmodal association of auditory pitch and visuospatial elevation...................... 27

Method.......................................................................................................................................... 32

Participants.............................................................................................................................. 32

Stimuli...................................................................................................................................... 33

Procedure.................................................................................................................................. 34

General.................................................................................................................................... 34

Pitch-elevation fMRI task....................................................................................................... 35

Functional localizer tasks......................................................................................................... 36

Multisensory integration localizer........................................................................................ 37

Magnitude localizer.............................................................................................................. 39

Semantic localizer................................................................................................................ 41

Post-scan behavioral testing: Implicit association of Pitch and Elevation............................ 42

Image acquisition..................................................................................................................... 44

Image processing and analysis.................................................................................................. 45

Results........................................................................................................................................... 47

Behavioral................................................................................................................................ 47

In –scanner tasks................................................................................................................. 47

Localizer tasks..................................................................................................................... 47

Pitch-elevation task............................................................................................................. 47

Post-scan pitch-elevation IAT................................................................................................ 51

Imaging..................................................................................................................................... 56

Localizer tasks......................................................................................................................... 56

Multisensory integration...................................................................................................... 56

Magnitude............................................................................................................................ 57

Semantic.............................................................................................................................. 58

Pitch-elevation task................................................................................................................. 59

Multisensory Congruency effects............................................................................................. 59

Basic congruency effect....................................................................................................... 59

Interaction of immediate trial congruency with previous trial congruency............................ 61

Regions modulated by congruency of immediate trial but not previous trial (i.e. ‘pure congruency effect’).      63

Overlap of congruency effect with localizers........................................................................... 65

Multisensory integration...................................................................................................... 66

Magnitude............................................................................................................................ 66

Semantic.............................................................................................................................. 66

Discussion..................................................................................................................................... 66

‘Pure congruency’ regions modulated by congruency of immediate trial but not previous trial: (CC>II) not overlapping with (CC>II)>(IC>CI)................................................................................................................................................... 67

Regions exhibiting both a congruency effect and interaction effect : (CC>II) overlapping with (CC>II)>(IC>CI)       74

Limitations related to task and congruency effects.................................................................. 76

Localizers................................................................................................................................. 78

Potential basis in semantic processing...................................................................................... 78

Potential basis in phonological processing............................................................................... 81

Potential basis in multisensory integration............................................................................... 82

Potential basis in magnitude processing.................................................................................... 84

Limitations and future directions......................................................................................... 86

Task limitations...................................................................................................................... 87

Localizer limitations................................................................................................................ 88

Analysis limitations................................................................................................................. 88

Theory limitations.................................................................................................................. 90

Conclusion.................................................................................................................................... 90

Chapter 3. Crossmodal association of pseudowords and object shape......................................... 92

Word-Shape mappings: The Bouba-Kiki phenomenon...................................................... 95

Mechanism.............................................................................................................................. 99

Method........................................................................................................................................ 104

Participants............................................................................................................................ 104

Stimuli................................................................................................................................... 105

Procedure................................................................................................................................ 105

General.................................................................................................................................. 105

Pseudoword-shape fMRI task................................................................................................. 106

Functional localizer tasks....................................................................................................... 109

Post-scan behavioral testing: Implicit association of pseudoword and shape........................... 109

Analysis.................................................................................................................................. 109

In-scanner behavioral data analysis........................................................................................ 109

Image processing and analysis................................................................................................ 110

Multisensory congruency analysis.......................................................................................... 110

Functional localizer analyses.................................................................................................. 110

Results......................................................................................................................................... 111

Behavioral.............................................................................................................................. 111

In-scanner tasks..................................................................................................................... 111

Localizer tasks................................................................................................................... 111

Pseudoword-shape task....................................................................................................... 112

Post-scan pseudoword-shape IAT....................................................................................... 115

Imaging................................................................................................................................... 118

Localizer tasks....................................................................................................................... 118

Multisensory synchrony..................................................................................................... 118

Magnitude.......................................................................................................................... 120

Semantic............................................................................................................................ 121

Pseudoword-Shape incongruency............................................................................................ 124

Overlap of incongruency effect with localizers...................................................................... 128

Multisensory Integration.................................................................................................... 128

Magnitude.......................................................................................................................... 128

Semantic............................................................................................................................ 128

Discussion................................................................................................................................... 129

Incongruency effects............................................................................................................. 129

Overlaps with localizers....................................................................................................... 134

Limitations and future directions....................................................................................... 136

Conclusion.................................................................................................................................. 138

Chapter 4. Crossmodal association of auditory pitch and visual object size............................. 140

Method........................................................................................................................................ 144

Participants............................................................................................................................ 144

Stimuli.................................................................................................................................... 144

Procedure................................................................................................................................ 146

Functional localizer tasks....................................................................................................... 146

Pitch-size fMRI task.............................................................................................................. 146

Post-scan behavioral testing: Implicit association of pitch and size........................................ 147

Results......................................................................................................................................... 147

Behavioral.............................................................................................................................. 147

In–scanner tasks.................................................................................................................... 147

Localizer tasks................................................................................................................... 147

Pitch-size task................................................................................................................... 148

Post-scan pitch-size IAT................................................................................................... 152

Imaging................................................................................................................................... 158

Localizer tasks....................................................................................................................... 158

Pitch-size task....................................................................................................................... 158

Overlap between incongruency effect and localizers............................................................... 159

Multisensory integration.................................................................................................... 159

Magnitude.......................................................................................................................... 159

Semantic............................................................................................................................ 159

Discussion................................................................................................................................... 159

Individual differences........................................................................................................... 160

Mechanism............................................................................................................................. 161

Conclusion.................................................................................................................................. 161

Chapter 5. Conclusion................................................................................................................ 163

Limitations and future directions....................................................................................... 166

Conclusion............................................................................................................................. 168

References.................................................................................................................................... 169

About this Dissertation

Rights statement
  • Permission granted by the author to include this thesis or dissertation in this repository. All rights reserved by the author. Please contact the author for information regarding the reproduction and use of this thesis or dissertation.
School
Department
Degree
Submission
Language
  • English
Research Field
Keyword
Committee Chair / Thesis Advisor
Committee Members
Last modified

Primary PDF

Supplemental Files