Fast Gaussian Process Solver Open Access

Fu, Zhiqi (Spring 2020)

Permanent URL:


Gaussian Process is a non-parametric, stochastic machine learning technique that has been widely utilized in many fields, such as spatial statistics, geology and computer experiments. However, the application of this powerful technique is limited by its cubic computational complexity. To solve this problem, we studied the intrinsic structures inherent in the kernel matrix and developed linear-complexity solvers to quickly train the Gaussian process model. In this study, we worked on the following aspects:

· Combining stochastic Lanczos Quadrature with density of states, which is a concept developed in quantum physics, to develop a log-determinant solver

· Using the Sherman-Morrison-Woodbury preconditioned conjugate gradient to develop a linear system solver

· Substituting the naive grid search with Bayesian optimization to quickly train the optimal hyper-parameters

After the above steps, we applied it to the synthetic data set and global temperature data to compare our predictions with the true temperature values in order to validate our algorithm. 

Table of Contents

1 Introduction

1.1 The Multivariate Gaussian Distribution

1.2 Gaussian Process

1.3 Motivations

1.4 Kernel Function/Matrix

1.5 Log Marginal Function

1.6 Application of Gaussian Process

1.6.1 Application in Finance/Porfolio

1.6.2 Application in Robotic Techniques

1.6.3 Application in Biosystem

1.6.4 Application in Weather Forecasts

1.7 Difficulties and Contribution

2. Log Determinant Solver

2.1 Matrix Functions

2.2 Stochastic Lanczos Quadrature

2.2.1 Hutchinson's Method

2.2.2 Lanczos Algorithm

2.2.3 Lanczos Quadrature

2.3 Density of States

2.3.1 Limitations of DOS algorithm

2.4 Modified DOS

2.4.1 Truncation of DOS

2.4.2 Comparison

3. Linear System Solver

3.1 Conjugate Gradient(CG) Algorithm

3.1.1 Convergence of CG Algorithm

3.2 Preconditioned Conjugate Gradient

3.3 SMW Preconditioned Conjugate Gradient

3.4 Comparison

4. Hyper-parameter Training

4.1 Naive Grid Search

4.1.1 Synthetic Data Experiment

4.2 Bayesian Optimization

4.3 Comparison

5. Experiment Outline

5.1 Data Description

5.2 Dataset Resize

5.3 Dataset Snapshot

5.4 Result

5.5 Comparison

6. Conclusion

About this Honors Thesis

Rights statement
  • Permission granted by the author to include this thesis or dissertation in this repository. All rights reserved by the author. Please contact the author for information regarding the reproduction and use of this thesis or dissertation.
  • English
Research Field
Committee Chair / Thesis Advisor
Committee Members
Last modified

Primary PDF

Supplemental Files