The dynamics of matrix momentum

Magnus Rattray; David Saad

doi:10.1007/978-1-4471-1599-1_24

The dynamics of matrix momentum

Magnus Rattray, David Saad

Research output: Chapter in Book/Published conference output › Chapter

Abstract

We analyse the matrix momentum algorithm, which provides an efficient approximation to on-line Newton's method, by extending a recent statistical mechanics framework to include second order algorithms. We study the efficacy of this method when the Hessian is available and also consider a practical implementation which uses a single example estimate of the Hessian. The method is shown to provide excellent asymptotic performance, although the single example implementation is sensitive to the choice of training parameters. We conjecture that matrix momentum could provide efficient matrix inversion for other second order algorithms.

Original language	English
Title of host publication	Proceedings of the 8th International Conference on Artificial Neural Networks
Editors	Lars F. Niklasson, Mikael B. Boden, Tom Ziemke
Publisher	Springer
Pages	183-188
Number of pages	6
Volume	1
ISBN (Print)	3540762639
DOIs	https://doi.org/10.1007/978-1-4471-1599-1_24
Publication status	Published - 1 Sept 1998

Bibliographical note

The original publication is available at www.springerlink.com

Keywords

matrix momentum
statistical mechanics
asymptotic performance
matrix inversion
Hessian

Access to Document

10.1007/978-1-4471-1599-1_24

The Dynamics of Matrix Momentum
The original publication is available at www.springerlink.com
Accepted author manuscript, 134 KB

Cite this

@inbook{1ee3026ee8174fa1aa4dd32bb36e5335,

title = "The dynamics of matrix momentum",

abstract = "We analyse the matrix momentum algorithm, which provides an efficient approximation to on-line Newton's method, by extending a recent statistical mechanics framework to include second order algorithms. We study the efficacy of this method when the Hessian is available and also consider a practical implementation which uses a single example estimate of the Hessian. The method is shown to provide excellent asymptotic performance, although the single example implementation is sensitive to the choice of training parameters. We conjecture that matrix momentum could provide efficient matrix inversion for other second order algorithms.",

keywords = "matrix momentum, statistical mechanics, asymptotic performance, matrix inversion, Hessian",

author = "Magnus Rattray and David Saad",

note = "The original publication is available at www.springerlink.com",

year = "1998",

month = sep,

day = "1",

doi = "10.1007/978-1-4471-1599-1_24",

language = "English",

isbn = "3540762639",

volume = "1",

pages = "183--188",

editor = "Niklasson, {Lars F.} and Boden, {Mikael B.} and Tom Ziemke",

booktitle = "Proceedings of the 8th International Conference on Artificial Neural Networks",

publisher = "Springer",

address = "Germany",

}

TY - CHAP

T1 - The dynamics of matrix momentum

AU - Rattray, Magnus

AU - Saad, David

N1 - The original publication is available at www.springerlink.com

PY - 1998/9/1

Y1 - 1998/9/1

N2 - We analyse the matrix momentum algorithm, which provides an efficient approximation to on-line Newton's method, by extending a recent statistical mechanics framework to include second order algorithms. We study the efficacy of this method when the Hessian is available and also consider a practical implementation which uses a single example estimate of the Hessian. The method is shown to provide excellent asymptotic performance, although the single example implementation is sensitive to the choice of training parameters. We conjecture that matrix momentum could provide efficient matrix inversion for other second order algorithms.

AB - We analyse the matrix momentum algorithm, which provides an efficient approximation to on-line Newton's method, by extending a recent statistical mechanics framework to include second order algorithms. We study the efficacy of this method when the Hessian is available and also consider a practical implementation which uses a single example estimate of the Hessian. The method is shown to provide excellent asymptotic performance, although the single example implementation is sensitive to the choice of training parameters. We conjecture that matrix momentum could provide efficient matrix inversion for other second order algorithms.

KW - matrix momentum

KW - statistical mechanics

KW - asymptotic performance

KW - matrix inversion

KW - Hessian

UR - https://link.springer.com/chapter/10.1007/978-1-4471-1599-1_24

U2 - 10.1007/978-1-4471-1599-1_24

DO - 10.1007/978-1-4471-1599-1_24

M3 - Chapter

SN - 3540762639

VL - 1

SP - 183

EP - 188

BT - Proceedings of the 8th International Conference on Artificial Neural Networks

A2 - Niklasson, Lars F.

A2 - Boden, Mikael B.

A2 - Ziemke, Tom

PB - Springer

ER -

The dynamics of matrix momentum

Abstract

Bibliographical note

Keywords

Access to Document

Other files and links

Fingerprint

Cite this