The theory of on-line learning: a statistical physics
approach

David Saad

The theory of on-line learning: a statistical physics approach

Research output: Chapter in Book/Published conference output › Chapter

Abstract

In this paper we review recent theoretical approaches for analysing the dynamics of on-line learning in multilayer neural networks using methods adopted from statistical physics. The analysis is based on monitoring a set of macroscopic variables from which the generalisation error can be calculated. A closed set of dynamical equations for the macroscopic variables is derived analytically and solved numerically. The theoretical framework is then employed for defining optimal learning parameters and for analysing the incorporation of second order information into the learning process using natural gradient descent and matrix-momentum based methods. We will also briefly explain an extension of the original framework for analysing the case where training examples are sampled with repetition.

Original language	English
Title of host publication	Exploratory Data Analysis in Empirical Research: Proceedings of the 25th Annual Conference of the Gesellschaft für Klassifikation e.V., University of Munich, March 14-16, 2001
Editors	Manfred Schwaiger, Otto Opit
Publisher	Springer
Pages	300-309
Number of pages	10
ISBN (Print)	9783540441830
Publication status	Published - 2003
Event	Studies in Classification, Data Analysis and Knowledge Organization - Duration: 1 Jan 2003 → 1 Jan 2003

Publication series

Name	Studies in Classification, Data Analysis, and Knowledge Organization
Publisher	Springer-Verlag

Other

Other	Studies in Classification, Data Analysis and Knowledge Organization
Period	1/01/03 → 1/01/03

Bibliographical note

The original publication is available at www.springerlink.com

Keywords

on-line learning
neural networks
statistical physics
natural gradient descent
matrix-momentum
repetition

Access to Document

NCRG_2001_010.pdf
The original publication is available at www.springerlink.com
Accepted author manuscript, 192 KB

Cite this

Saad, D. (2003). The theory of on-line learning: a statistical physics approach. In M. Schwaiger, & O. Opit (Eds.), Exploratory Data Analysis in Empirical Research: Proceedings of the 25th Annual Conference of the Gesellschaft für Klassifikation e.V., University of Munich, March 14-16, 2001 (pp. 300-309). (Studies in Classification, Data Analysis, and Knowledge Organization). Springer.

Saad, David. / The theory of on-line learning: a statistical physics approach. Exploratory Data Analysis in Empirical Research: Proceedings of the 25th Annual Conference of the Gesellschaft für Klassifikation e.V., University of Munich, March 14-16, 2001. editor / Manfred Schwaiger ; Otto Opit. Springer, 2003. pp. 300-309 (Studies in Classification, Data Analysis, and Knowledge Organization).

@inbook{dad66fbb527d4538b1a4f5ad520e232e,

title = "The theory of on-line learning: a statistical physics approach",

abstract = "In this paper we review recent theoretical approaches for analysing the dynamics of on-line learning in multilayer neural networks using methods adopted from statistical physics. The analysis is based on monitoring a set of macroscopic variables from which the generalisation error can be calculated. A closed set of dynamical equations for the macroscopic variables is derived analytically and solved numerically. The theoretical framework is then employed for defining optimal learning parameters and for analysing the incorporation of second order information into the learning process using natural gradient descent and matrix-momentum based methods. We will also briefly explain an extension of the original framework for analysing the case where training examples are sampled with repetition.",

keywords = "on-line learning, neural networks, statistical physics, natural gradient descent, matrix-momentum, repetition",

author = "David Saad",

note = "The original publication is available at www.springerlink.com; Studies in Classification, Data Analysis and Knowledge Organization ; Conference date: 01-01-2003 Through 01-01-2003",

year = "2003",

language = "English",

isbn = "9783540441830",

series = "Studies in Classification, Data Analysis, and Knowledge Organization",

publisher = "Springer",

pages = "300--309",

editor = "Manfred Schwaiger and Otto Opit",

booktitle = "Exploratory Data Analysis in Empirical Research: Proceedings of the 25th Annual Conference of the Gesellschaft f{\"u}r Klassifikation e.V., University of Munich, March 14-16, 2001",

address = "Germany",

}

Saad, D 2003, The theory of on-line learning: a statistical physics approach. in M Schwaiger & O Opit (eds), Exploratory Data Analysis in Empirical Research: Proceedings of the 25th Annual Conference of the Gesellschaft für Klassifikation e.V., University of Munich, March 14-16, 2001. Studies in Classification, Data Analysis, and Knowledge Organization, Springer, pp. 300-309, Studies in Classification, Data Analysis and Knowledge Organization, 1/01/03.

The theory of on-line learning: a statistical physics approach. / Saad, David.
Exploratory Data Analysis in Empirical Research: Proceedings of the 25th Annual Conference of the Gesellschaft für Klassifikation e.V., University of Munich, March 14-16, 2001. ed. / Manfred Schwaiger; Otto Opit. Springer, 2003. p. 300-309 (Studies in Classification, Data Analysis, and Knowledge Organization).

Research output: Chapter in Book/Published conference output › Chapter

TY - CHAP

T1 - The theory of on-line learning: a statistical physics approach

AU - Saad, David

N1 - The original publication is available at www.springerlink.com

PY - 2003

Y1 - 2003

N2 - In this paper we review recent theoretical approaches for analysing the dynamics of on-line learning in multilayer neural networks using methods adopted from statistical physics. The analysis is based on monitoring a set of macroscopic variables from which the generalisation error can be calculated. A closed set of dynamical equations for the macroscopic variables is derived analytically and solved numerically. The theoretical framework is then employed for defining optimal learning parameters and for analysing the incorporation of second order information into the learning process using natural gradient descent and matrix-momentum based methods. We will also briefly explain an extension of the original framework for analysing the case where training examples are sampled with repetition.

AB - In this paper we review recent theoretical approaches for analysing the dynamics of on-line learning in multilayer neural networks using methods adopted from statistical physics. The analysis is based on monitoring a set of macroscopic variables from which the generalisation error can be calculated. A closed set of dynamical equations for the macroscopic variables is derived analytically and solved numerically. The theoretical framework is then employed for defining optimal learning parameters and for analysing the incorporation of second order information into the learning process using natural gradient descent and matrix-momentum based methods. We will also briefly explain an extension of the original framework for analysing the case where training examples are sampled with repetition.

KW - on-line learning

KW - neural networks

KW - statistical physics

KW - natural gradient descent

KW - matrix-momentum

KW - repetition

UR - http://www.springer.com/computer/security+and+cryptology/book/978-3-540-44183-0?detailsPage=toc

M3 - Chapter

SN - 9783540441830

T3 - Studies in Classification, Data Analysis, and Knowledge Organization

SP - 300

EP - 309

BT - Exploratory Data Analysis in Empirical Research: Proceedings of the 25th Annual Conference of the Gesellschaft für Klassifikation e.V., University of Munich, March 14-16, 2001

A2 - Schwaiger, Manfred

A2 - Opit, Otto

PB - Springer

T2 - Studies in Classification, Data Analysis and Knowledge Organization

Y2 - 1 January 2003 through 1 January 2003

ER -

Saad D. The theory of on-line learning: a statistical physics approach. In Schwaiger M, Opit O, editors, Exploratory Data Analysis in Empirical Research: Proceedings of the 25th Annual Conference of the Gesellschaft für Klassifikation e.V., University of Munich, March 14-16, 2001. Springer. 2003. p. 300-309. (Studies in Classification, Data Analysis, and Knowledge Organization).