Natural gradient matrix momentum

Silvia Scarpetta, Magnus Rattray, David Saad

    Research output: Chapter in Book/Published conference outputChapter

    Abstract

    Natural gradient learning is an efficient and principled method for improving on-line learning. In practical applications there will be an increased cost required in estimating and inverting the Fisher information matrix. We propose to use the matrix momentum algorithm in order to carry out efficient inversion and study the efficacy of a single step estimation of the Fisher information matrix. We analyse the proposed algorithm in a two-layer network, using a statistical mechanics framework which allows us to describe analytically the learning dynamics, and compare performance with true natural gradient learning and standard gradient descent.
    Original languageEnglish
    Title of host publicationNinth International Conference on Artificial Neural Networks, ICANN 99
    Place of PublicationEdinburgh UK
    PublisherIEEE
    Pages43-48
    Number of pages6
    Volume1
    Publication statusPublished - 7 Sept 1999

    Publication series

    NameConference Publication
    PublisherInstitute of Electrical and Electronics Engineers (IEEE)
    Volume470

    Bibliographical note

    ©1999 IEEE. Personal use of this material is permitted. However, permission to reprint/republish this material for advertising or promotional purposes or for creating new collective works for resale or redistribution to servers or lists, or to reuse any copyrighted component of this work in other works must be obtained from the IEEE.

    Keywords

    • Natural gradient learning
    • on-line learning
    • Fisher information matrix
    • matrix momentum algorithm
    • two-layer network

    Fingerprint

    Dive into the research topics of 'Natural gradient matrix momentum'. Together they form a unique fingerprint.

    Cite this