Efficient Training of Input Convex Neural Networks Using Variable Projection Open Access

Abstract

Normalizing flows are deep generative models that construct a diffeomorphic mapping between a simple reference distribution and a complex probability distribution. Previous studies have utilized the gradient of input convex neural networks (ICNNs) to represent the diffeomorphism, which guarantees invertibility for any network weights. Moreover, the resulting mapping constitutes a unique optimal transformation that minimizes transportation costs, as dictated by optimal transport theory.Training ICNNs presents a challenging non-convex problem, except for weights of the last layer. To address this issue, this thesis proposes a training approach for ICNNs using variable projection (VarPro). The proposed method takes advantage of the affine mapping in the last layer of ICNNs, which preserves convexity of the network. Empirical results based on a two-dimensional synthetic dataset demonstrate that VarPro achieves a lower test loss and requires fewer gradient evaluations compared to the mini-batch gradient descent method Adam.

Table of Contents

1 Introduction...................... 1

1.1 Contribution&Outline...................... 3

2 Background......................5

2.1 ConvexPotentialFlows...................... 6

2.2 VariableProjection........................ 10

3 Efficient Method in Training ICNNs......................12

3.1 SampleAverageApproximation ................. 13

3.2 TrainingICNNswithVariableProjection . . . . . . . . . . . . 14

4 Numerical Experiments......................19

4.1 ExperimentwithMoonsDataset................. 20

5 Discussion......................23

6 Conclusion......................25 

About this Honors Thesis

Rights statement

• Permission granted by the author to include this thesis or dissertation in this repository. All rights reserved by the author. Please contact the author for information regarding the reproduction and use of this thesis or dissertation.

School	Emory College
Department	Mathematics
Degree	B.S.
Submission	Honors Thesis
Language	English
Research Field	Statistics Mathematics
Keyword	Neural Network Generative Modeling Variable Projection Numerical Optimization
Committee Chair / Thesis Advisor	Lars Ruthotto, Emory University
Committee Members	Robert Roth, Emory University Ruoxuan Xiong, Emory University

Last modified

Primary PDF

Thumbnail	Title	Date Uploaded	Actions
	Efficient Training of Input Convex Neural Networks Using Variable Projection ()	2023-04-10 15:27:11 -0400	Press to Select an action Download

Supplemental Files

Thumbnail	Title	Date Uploaded	Actions