Improving Sampling and Function Approximation in Machine Learning Methods for Solving Partial Differential Equations Open Access

Abstract

Numerical solutions to partial differential equations (PDEs) remain one of the main focuses in the field of scientific computing. Deep learning and neural network based methods for solving PDEs have gained much attention and popularity in recent years. The nonlinear structures and universal approximation property of neural networks allow for a cheaper approximation of functions in high dimensions compared to many traditional numerical methods. Reformulating PDE problems as optimization tasks also enables straightforward setup and implementation and can sometimes circumvent stability concerns common for classic numerical methods that rely on explicit or semi-explicit time discretization. However low accuracy and convergence difficulty stand as challenges to deep learning based schemes and fine-tuning neural networks can also be computationally expensive at times.

We present some of our findings using machine learning methods for solving certain PDEs. Since low and high dimensional PDEs often require very different numerical methods to solve, we divide our work into two main sections based on the dimensionality of a problem. In the first half we focus on the popular Physics Informed Neural Networks (PINNs) framework, specifically in problems with dimensions less than or equal to three. We present an alternative optimization based algorithm using a B-spline polynomial function approximator and accurate numerical integration with a grid based sampling scheme. With implementation using popular machine learning libraries, our approach serves as a direct substitute for PINNs, and through performance comparison between the two methods over a wide selection of examples, we find that for low dimensional problems, our proposed method can improve both accuracy and reliability when compared to PINNs. In the second half, we focus on a general class of stochastic optimal control (SOC) problems. By leveraging the underlying theory we propose a neural network solver that solves the SOC problem and the corresponding Hamilton–Jacobi–Bellman (HJB) equation simultaneously. Our method utilizes the stochastic Pontryagin maximum principle and is thus unique in the sampling strategy, this combined with modifying the loss function enables us to tackle high-dimensional problems efficiently.

Table of Contents

1 Introduction 1

1.1 Overview of Thesis . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

2 Mathematical Background 7

2.1 Neural Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

2.1.1 Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

2.1.2 Training of a Neural Network . . . . . . . . . . . . . . . . . . 10

2.2 Physics Informed Neural Networks . . . . . . . . . . . . . . . . . . . 11

2.2.1 Neural Network Approximation . . . . . . . . . . . . . . . . . 12

2.2.2 Loss Function . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

2.2.3 Sampling and Training . . . . . . . . . . . . . . . . . . . . . . 14

2.3 Finite Element Method . . . . . . . . . . . . . . . . . . . . . . . . . . 15

2.3.1 Non-Stationary PDEs . . . . . . . . . . . . . . . . . . . . . . 16

2.4 Stochastic Optimal Control and HJB Equations . . . . . . . . . . . . 18

2.4.1 Stochastic Optimal Control Problem . . . . . . . . . . . . . . 18

2.4.2 Stochastic Pontryagin Maximum Principle . . . . . . . . . . . 20

2.4.3 Hamilton-Jacobi-Bellman Equation . . . . . . . . . . . . . . . 22

2.4.4 FBSDE Formulation . . . . . . . . . . . . . . . . . . . . . . . 24

2.4.5 Relation to Deterministic Optimal Control . . . . . . . . . . . 25

3 A Spline Based Alternative Model for PINNs 29

3.1 A Deeper Look into PINNs . . . . . . . . . . . . . . . . . . . . . . . . 30

3.1.1 State of the Art in PINNs . . . . . . . . . . . . . . . . . . . . 31

3.2 Pros and Cons of PINNs . . . . . . . . . . . . . . . . . . . . . . . . . 34

3.3 A Trainable Spline Model for PDEs in Low Dimensions . . . . . . . . 36

3.3.1 Spline Interpolation . . . . . . . . . . . . . . . . . . . . . . . . 37

3.3.2 Spline Model for Higher Dimensions . . . . . . . . . . . . . . . 39

3.3.3 Derivatives and Laplacians . . . . . . . . . . . . . . . . . . . . 40

3.3.4 Sampling and Optimization . . . . . . . . . . . . . . . . . . . 42

3.3.5 Outline of Our Method for Testing . . . . . . . . . . . . . . . 43

3.4 Numerical Experiments for Different PDEs . . . . . . . . . . . . . . . 46

3.4.1 2D Poisson Equation . . . . . . . . . . . . . . . . . . . . . . . 48

3.4.2 3D Poisson Equation . . . . . . . . . . . . . . . . . . . . . . . 51

3.4.3 1D Schrodinger Equation . . . . . . . . . . . . . . . . . . . . . 55

3.4.4 Allen-Cahn equation . . . . . . . . . . . . . . . . . . . . . . . 59

3.4.5 2D Taylor-Green Vortex Problem . . . . . . . . . . . . . . . . 64

3.4.6 Additional Numerical Schemes . . . . . . . . . . . . . . . . . . 68

3.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73

4 Deep Learning Approach for SOC problems and HJB Equations 75

4.1 Neural Network Approximation . . . . . . . . . . . . . . . . . . . . . 75

4.2 Formulation of the Training Problem . . . . . . . . . . . . . . . . . . 77

4.3 Numerical Experiments for SOC Problems and HJB Equations . . . . 81

4.3.1 Implementation Details . . . . . . . . . . . . . . . . . . . . . . 81

4.3.2 2D Trajectory Planning Problem . . . . . . . . . . . . . . . . 82

4.3.3 100-dimensional example . . . . . . . . . . . . . . . . . . . . . 91

4.3.4 12D Quadcopter Path Finding Problem with Nonlinear Dynamics 98

4.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103

5 Conclusions and Future Work 105

Bibliography 107

About this Dissertation

Rights statement

• Permission granted by the author to include this thesis or dissertation in this repository. All rights reserved by the author. Please contact the author for information regarding the reproduction and use of this thesis or dissertation.

School	Laney Graduate School
Department	Mathematics
Degree	Ph.D.
Submission	Dissertation
Language	English
Research Field	Applied Mathematics Mathematics
Keyword	Partial Differential Equations Optimal Control Machine Learning
Committee Chair / Thesis Advisor	Ruthotto, Lars, Emory University
Committee Members	Newman, Elizabeth, Emory University Xi, Yuanzhe, Emory University

Last modified

Primary PDF

Thumbnail	Title	Date Uploaded	Actions
	Improving Sampling and Function Approximation in Machine Learning Methods for Solving Partial Differential Equations ()	2024-06-27 14:22:59 -0400	Press to Select an action Download

Supplemental Files

Thumbnail	Title	Date Uploaded	Actions