AI-Assisted Healthcare with Multimodal Structured Knowledge Extraction and Augmented Inference Restricted; Files Only
Cui, Hejie (Spring 2024)
Published
Abstract
The rapid advancement of artificial intelligence (AI) has unlocked new opportunities for enhancing healthcare. However, the heterogeneity and complexity of healthcare data, spanning scientific literature, clinical texts, medical images, and electronic health records, pose significant challenges in extracting useful knowledge and leveraging AI models effectively for clinical decision-making. This thesis addresses these challenges through two key themes: (i) multimodal structured knowledge extraction, focusing on integrating knowledge from diverse data sources and pre-trained models to enable comprehensive data understanding, and (ii) augmented inference, developing techniques to improve the domain-specific reasoning capabilities and reliability of AI models by incorporating the extracted or external knowledge resources. The proposed methods enhance the breadth of multimodal data understanding and the depth of AI models' capabilities in specialized applications. The effectiveness of the proposed core ideas is demonstrated in various domains, including brain disorder analysis, scientific literature understanding, disease prediction, and biomedical reasoning, paving the way for more personalized, precise, and reliable AI-assisted care delivery.
Table of Contents
1 Introduction
1.1 Background and Motivation
1.2 Challenges
1.2.1 Healthcare Data Can be Complex and Heterogeneous
1.2.2 AI-assisted Healthcare Suffers from Limited Labeled Data
1.2.3 Pre-trained Models Have Wide Knowledge Base but May Not be Adequately Reliable for Healthcare
1.3 Research Contributions
1.3.1 Multimodal Structured Knowledge Extraction
1.3.2 Augmented Inference
1.4 Dissertation Outline
2 Multimodal Neurobiological Data: Brain Connectome Extraction and Inference
2.1 Introduction
2.2 Brain Connectome Extraction and Benchmark
2.2.1 Background: Diverse Modalities of Brain Imaging
2.2.2 Brain Network Extraction from Multimodal Imaging
2.2.3 Open Source Benchmark Platform
2.3 Graph Neural Network Baselines for Brain Network Inference
2.3.1 Node Feature Construction
2.3.2 Message Passing Mechanisms
2.3.3 Attention-Enhanced Message Passing
2.3.4 Pooling Strategies
2.3.5 Experimental Analysis and Insights
2.3.6 Discussion and Extensions
2.4 Interpretable Brain Network Inference
2.4.1 Introduction
2.4.2 Preliminaries
2.4.3 Method
2.4.4 Experiments
2.4.5 Interpretation Analysis
2.4.6 Conclusion
3 Broader Types of Multimodal Data: Structured Knowledge Extraction and Augmented Inference
3.1 Specialized Models for Structured Knowledge Extraction from Textual Data
3.1.1 Introduction
3.1.2 Concept Map-based Document Retrieval
3.1.3 Experiments
3.1.4 Conclusion
3.2 Specialized Models for Structured Knowledge Extraction from Visual Data
3.2.1 Introduction
3.2.2 Related Work
3.2.3 Method
3.2.4 Evaluation
3.2.5 Application
3.2.6 Conclusion, Limitations, and Future Work
3.3 Specialized Models for Structured Knowledge Extraction from Multimodal Data
3.3.1 Introduction
3.3.2 Preliminaries
3.3.3 Patching Visual Modality to Textual-Established Multimodal Information Extraction
3.3.4 Experimental Setup
3.3.5 Experimental Results
3.3.6 Related Work
3.3.7 Conclusion
3.4 Language Foundation Models with Augmented Inference for EHR-based Disease Prediction
3.4.1 Introduction
3.4.2 Related Work
3.4.3 Method
3.4.4 Experimental Settings
3.4.5 Experimental Results
3.4.6 Generated Instructions
3.4.7 Conclusions
3.4.8 Ethical Considerations
3.5 Multimodal Foundation Models with Augmented Inference for Adapting Generic Models to the Healthcare Domain
3.5.1 Introduction
3.5.2 Background
3.5.3 Clinician-Aligned Biomedical Multimodality Instruction Tuning Model
3.5.4 Evaluation Plan and Preliminary Results
3.5.5 Further Plans
3.5.6 Conclusion
4 Conclusion
4.1 Summary of Research Contributions
4.2 Future Work
4.2.1 Discovering Unknown Knowledge from Known Data
4.2.2 Grounding Foundation Model Evaluation and Alignment with Domain Knowledge
4.2.3 Augmenting Reasoning by Human-AI Collaboration
About this Dissertation
School | |
---|---|
Department | |
Degree | |
Submission | |
Language |
|
Research Field | |
Palavra-chave | |
Committee Chair / Thesis Advisor | |
Committee Members |
![Preview image embargoed](/assets/No_Preview-c99a369db0c8f31173bf3fdc06f581cabae85282e8a747dd66f99b731b54aec0.png)
Primary PDF
Thumbnail | Title | Date Uploaded | Actions |
---|---|---|---|
![]() |
File download under embargo until 22 November 2024 | 2024-04-23 01:09:56 -0400 | File download under embargo until 22 November 2024 |
Supplemental Files
Thumbnail | Title | Date Uploaded | Actions |
---|