AI-Assisted Healthcare with Multimodal Structured Knowledge Extraction and Augmented Inference Restricted; Files Only

Abstract

The rapid advancement of artificial intelligence (AI) has unlocked new opportunities for enhancing healthcare. However, the heterogeneity and complexity of healthcare data, spanning scientific literature, clinical texts, medical images, and electronic health records, pose significant challenges in extracting useful knowledge and leveraging AI models effectively for clinical decision-making. This thesis addresses these challenges through two key themes: (i) multimodal structured knowledge extraction, focusing on integrating knowledge from diverse data sources and pre-trained models to enable comprehensive data understanding, and (ii) augmented inference, developing techniques to improve the domain-specific reasoning capabilities and reliability of AI models by incorporating the extracted or external knowledge resources. The proposed methods enhance the breadth of multimodal data understanding and the depth of AI models' capabilities in specialized applications. The effectiveness of the proposed core ideas is demonstrated in various domains, including brain disorder analysis, scientific literature understanding, disease prediction, and biomedical reasoning, paving the way for more personalized, precise, and reliable AI-assisted care delivery.

Table of Contents

1 Introduction

1.1 Background and Motivation

1.2 Challenges

1.2.1 Healthcare Data Can be Complex and Heterogeneous

1.2.2 AI-assisted Healthcare Suffers from Limited Labeled Data

1.2.3 Pre-trained Models Have Wide Knowledge Base but May Not be Adequately Reliable for Healthcare

1.3 Research Contributions

1.3.1 Multimodal Structured Knowledge Extraction

1.3.2 Augmented Inference

1.4 Dissertation Outline

2 Multimodal Neurobiological Data: Brain Connectome Extraction and Inference

2.1 Introduction

2.2 Brain Connectome Extraction and Benchmark

2.2.1 Background: Diverse Modalities of Brain Imaging

2.2.2 Brain Network Extraction from Multimodal Imaging

2.2.3 Open Source Benchmark Platform

2.3 Graph Neural Network Baselines for Brain Network Inference

2.3.1 Node Feature Construction

2.3.2 Message Passing Mechanisms

2.3.3 Attention-Enhanced Message Passing

2.3.4 Pooling Strategies

2.3.5 Experimental Analysis and Insights

2.3.6 Discussion and Extensions

2.4 Interpretable Brain Network Inference

2.4.1 Introduction

2.4.2 Preliminaries

2.4.3 Method

2.4.4 Experiments

2.4.5 Interpretation Analysis

2.4.6 Conclusion

3 Broader Types of Multimodal Data: Structured Knowledge Extraction and Augmented Inference

3.1 Specialized Models for Structured Knowledge Extraction from Textual Data

3.1.1 Introduction

3.1.2 Concept Map-based Document Retrieval

3.1.3 Experiments

3.1.4 Conclusion

3.2 Specialized Models for Structured Knowledge Extraction from Visual Data

3.2.1 Introduction

3.2.2 Related Work

3.2.3 Method

3.2.4 Evaluation

3.2.5 Application

3.2.6 Conclusion, Limitations, and Future Work

3.3 Specialized Models for Structured Knowledge Extraction from Multimodal Data

3.3.1 Introduction

3.3.2 Preliminaries

3.3.3 Patching Visual Modality to Textual-Established Multimodal Information Extraction

3.3.4 Experimental Setup

3.3.5 Experimental Results

3.3.6 Related Work

3.3.7 Conclusion

3.4 Language Foundation Models with Augmented Inference for EHR-based Disease Prediction

3.4.1 Introduction

3.4.2 Related Work

3.4.3 Method

3.4.4 Experimental Settings

3.4.5 Experimental Results

3.4.6 Generated Instructions

3.4.7 Conclusions

3.4.8 Ethical Considerations

3.5 Multimodal Foundation Models with Augmented Inference for Adapting Generic Models to the Healthcare Domain

3.5.1 Introduction

3.5.2 Background

3.5.3 Clinician-Aligned Biomedical Multimodality Instruction Tuning Model

3.5.4 Evaluation Plan and Preliminary Results

3.5.5 Further Plans

3.5.6 Conclusion

4 Conclusion

4.1 Summary of Research Contributions

4.2 Future Work

4.2.1 Discovering Unknown Knowledge from Known Data

4.2.2 Grounding Foundation Model Evaluation and Alignment with Domain Knowledge

4.2.3 Augmenting Reasoning by Human-AI Collaboration

About this Dissertation

Rights statement

• Permission granted by the author to include this thesis or dissertation in this repository. All rights reserved by the author. Please contact the author for information regarding the reproduction and use of this thesis or dissertation.

School	Laney Graduate School
Department	Computer Science and Informatics
Degree	Ph.D.
Submission	Dissertation
Language	English
Research Field	Computer Science
Keyword	Structure Augmented Inference AI for Healthcare Multumodality Artificial Intelligence Structured Knowledge Extraction
Committee Chair / Thesis Advisor	Yang, Carl, Emory University
Committee Members	Wang, Fei, Weill Cornell Medicine, Cornell University Ho, Joyce, Emory University Bromberg, Yana, Emory University

Last modified

Primary PDF

Thumbnail	Title	Date Uploaded	Actions
	File download under embargo until 22 November 2024	2024-04-23 01:09:56 -0400	File download under embargo until 22 November 2024

Supplemental Files

Thumbnail	Title	Date Uploaded	Actions