Reinforcement Learning with Manifold Optimization Open Access

Abstract

Reinforcement learning is a rapidly advancing field which in recent years has expanded out of its niche origins to become one of the most potent paradigms available within machine learning. The explosion of deep learning in the past decade has only furthered this growth, leading an acceleration in the development of state-of-the-art RL methods for highly complex use cases. Simultaneously, the study of information geometry has expanded the understanding of the geometric spaces traversed by RL algorithms as the field matures. The first part of this work provides an introduction to reinforcement learning from its first principles, followed by an exploration of two categories of differential/information geometric methods applied to RL. The experiments in this work aim to broaden the understanding of this manifold structure and draw intuitive conclusions regarding the utility of such techniques for improving RL methods.

Table of Contents

1 Introduction

2 Background

2.1 Introduction to Reinforcement Learning

2.1.1 History and examples

2.1.2 Markov decision processes

2.2 Methods and algorithms

2.2.1 Classifications

2.2.2 Dynamic programming

2.2.3 Temporal-difference methods

2.2.4 Function approximation

2.2.5 Policy-gradient methods

3 Manifold-optimized RL

3.1 Motivation and previous work

3.2 Manifold-restricted methods

3.2.1 Orthogonal Procrustes

3.2.2 Curvilinear descent on Stiefel manifold

3.2.3 Experiments and Results

3.3 Manifold-regularized/preconditioned methods

3.3.1 Motivation

3.3.2 Geometry of statistical manifolds

3.3.3 Natural policy-gradient

3.3.4 Empirical Fisher information matrix

3.3.5 Approximate-Fisher natural policy-gradient

3.3.6 Experiments and Results

A Appendix

Bibliography

About this Honors Thesis

Rights statement

• Permission granted by the author to include this thesis or dissertation in this repository. All rights reserved by the author. Please contact the author for information regarding the reproduction and use of this thesis or dissertation.

School	Emory College
Department	Mathematics
Degree	B.S.
Submission	Honors Thesis
Language	English
Research Field	Applied Mathematics Mathematics Artificial Intelligence
Keyword	Information Geometry Optimization Manifold Differential Geometry Reinforcement Learning Machine Learning
Committee Chair / Thesis Advisor	Elizabeth Newman, Emory University
Committee Members	Lars Ruthotto, Emory University Matthias Chung, Emory University

Last modified

Primary PDF

Thumbnail	Title	Date Uploaded	Actions
	Reinforcement Learning with Manifold Optimization ()	2024-04-22 17:27:25 -0400	Press to Select an action Download

Supplemental Files

Thumbnail	Title	Date Uploaded	Actions