Harnessing Function Space of Machine Learning Models for Interpretability, Generalizability, and Efficiency Open Access
Bai, Guangji (Spring 2025)
Abstract
Modern deep learning has achieved impressive results across a wide range of domains, yet it remains fundamentally constrained by its reliance on parameter-space representations. In deep neural networks, multiple parameter configurations can represent the same function, leading to redundancy, limited interpretability, and poor generalization. This disconnect between parameter space and function space presents a critical bottleneck for building scalable, adaptive, and efficient AI systems — particularly as we pursue the broader goal of Artificial General Intelligence (AGI).
This dissertation proposes a unifying perspective centered on the function space of machine learning models — that is, reasoning about models based on the input-output functions they represent rather than their specific parameter values. By shifting the focus to functional behavior, we uncover new principles for enhancing interpretability, generalizability, and efficiency — three foundational pillars for robust and scalable AI.
For interpretability, we introduce Saliency-Regularized Multi-Task Learning (SRDML), which enforces structured task relationships via input gradients in function space, resulting in more coherent and explainable multi-task models. Building on this, SHARC combines saliency with associative memory replay to improve interpretability and mitigate catastrophic forgetting in continual learning.
For generalizability, we propose Drift-Aware Dynamic Neural Networks (DRAIN), which leverage temporal functional interpolation to handle evolving data distributions and enable robust domain generalization over time. We further extend the function-space perspective to multi-source domain adaptation with Prompt-Based Domain Discrimination (POND), employing prompt-tuning to disentangle invariant and domain-specific behavior across domains.
For efficiency, we develop SparseLLM, a global pruning framework that formulates pruning as a sparse functional optimization problem, enabling significant parameter reduction in large language models (LLMs) while maintaining performance. This direction is further advanced through FedSpaLLM, which incorporates sparsity-aware aggregation and layer-wise sampling in federated learning to address both model redundancy and system heterogeneity in decentralized environments.
Collectively, these contributions lay the foundation for a function-space-based framework for designing machine learning models that are interpretable, generalizable, and efficient — essential traits for the next generation of adaptive and resource-aware AI systems.
Table of Contents
1 Introduction 1
1.1 Research Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.1.1 Interpretable and Efficient Bio-inspired Deep Learning via Neuronal
Assemblies . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.1.2 Interpretation for Dynamic Attributed Graphs via Hierarchical
Attention . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
1.1.3 Explanation-Guided Representation Learning on Geometric Data 7
1.2 Contribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
1.3 Thesis Organization . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
2 Interpretable and Efficient Bio-inspired Deep Learning via Neuronal
Assemblies 12
2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2.2 Biologically-Enhanced Artificial Neuronal Assembly Regularization . 15
2.2.1 Layer-wise Neuron Co-activation Divergence . . . . . . . . . . 15
2.2.2 The First-Order Layer-wise Neuron Correlation . . . . . . . . 17
2.2.3 The Second-Order Layer-wise Neuron Correlation . . . . . . . 20
2.3 Experimental Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
2.3.1 The Interpretable Patterns of BEAN Regularization . . . . . . 23
2.3.2 Learning Sparse and Efficient Networks . . . . . . . . . . . . . 28
2.3.3 Towards few-shot learning from scratch with BEAN regularization 31
2.4 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
3 Interpretation for Dynamic Attributed Graphs via Hierarchical Attention
38
3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
3.2 Related work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
3.2.1 Online Health Communities Analysis . . . . . . . . . . . . . . 43
3.2.2 Dynamic Graph Representation Learning . . . . . . . . . . . . 43
3.2.3 Hierarchical Attention Mechanism . . . . . . . . . . . . . . . . 44
3.2.4 Neural Encoder-Decoder Models . . . . . . . . . . . . . . . . . 45
3.3 Problem Formulation . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
3.3.1 User Forum Activities as a Dynamic Graph . . . . . . . . . . 45
3.3.2 Learning Sequence from Dynamic Graph . . . . . . . . . . . . 47
3.4 Dynamic Graph-To-Sequence Model . . . . . . . . . . . . . . . . . . . 48
3.4.1 The DynGraph2Seq framework . . . . . . . . . . . . . . . . . 48
3.4.2 Dynamic Graph Encoder . . . . . . . . . . . . . . . . . . . . . 49
3.4.3 Sequence Decoder with Dynamic Graph Hierarchical Attention 53
3.5 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
3.5.1 Experimental Settings . . . . . . . . . . . . . . . . . . . . . . 57
3.5.2 Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
3.5.3 Interpretablity Analysis . . . . . . . . . . . . . . . . . . . . . 60
3.5.4 Health Stage Sequence Analysis . . . . . . . . . . . . . . . . . 63
3.6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
4 Explanation-Guided Representation Learning on Geometric Data 66
4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
4.2 EGL on Graph-Structured Data . . . . . . . . . . . . . . . . . . . . . 67
4.2.1 Related work . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
4.2.2 GNES Framework . . . . . . . . . . . . . . . . . . . . . . . . . 73
4.2.3 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
4.3 EGL on Image Data . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
4.3.1 Related work . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
4.3.2 GRADIA Framework . . . . . . . . . . . . . . . . . . . . . . . 97
4.3.3 RES Framework . . . . . . . . . . . . . . . . . . . . . . . . . . 104
4.3.4 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
4.4 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
5 Conclusions and Future Works 123
5.1 Research Tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126
5.1.1 Development of Interpretability Techniques for DNNs . . . . . 126
5.1.2 Explanation-Guided Learning on Graphs . . . . . . . . . . . . 127
5.1.3 Explanation-Guided Learning on Images . . . . . . . . . . . . 128
5.2 Publications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129
5.2.1 Published papers . . . . . . . . . . . . . . . . . . . . . . . . . 129
5.2.2 Submitted and In-preparation papers . . . . . . . . . . . . . . 131
5.3 Future Research Directions . . . . . . . . . . . . . . . . . . . . . . . . 132
5.3.1 Explanation-Guided Learning on Medical Image Analysis . . . 132
5.3.2 Trustworthiness and Fairness of Deep Learning Explanation . 133
5.3.3 Contrastive Explanation-Guided Learning . . . . . . . . . . . 134
5.3.4 Interactive Explanation-Guided Learning pipeline on Continual & Active Learning . . . . . . . . . . . . . . . . . . . . . . . . 134
Appendix A Explanation-Guided Representation Learning on Geometric
Data 136
A.1 Proof of Theorem 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136
A.2 Proof of Lemma 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137
A.3 Human Annotation and Evaluation UI demonstration . . . . . . . . . 138
A.4 Efficient Adaptive Threshold Searching Algorithm . . . . . . . . . . . 139
A.5 Detailed Implementation of the Learnable Imputation Layers . . . . . 139
Bibliography 142
About this Dissertation
| School | |
|---|---|
| Department | |
| Degree | |
| Submission | |
| Language |
|
| Research Field | |
| Keyword | |
| Committee Chair / Thesis Advisor | |
| Committee Members |
Primary PDF
| Thumbnail | Title | Date Uploaded | Actions |
|---|---|---|---|
|
|
Harnessing Function Space of Machine Learning Models for Interpretability, Generalizability, and Efficiency () | 2025-04-15 01:22:57 -0400 |
|
Supplemental Files
| Thumbnail | Title | Date Uploaded | Actions |
|---|