Finite-size effects in on-line learning of multilayer neural networks

David Barber, David Saad, Peter Sollich

    Research output: Contribution to journalArticlepeer-review


    We complement recent advances in thermodynamic limit analyses of mean on-line gradient descent learning dynamics in multi-layer networks by calculating fluctuations possessed by finite dimensional systems. Fluctuations from the mean dynamics are largest at the onset of specialisation as student hidden unit weight vectors begin to imitate specific teacher vectors, increasing with the degree of symmetry of the initial conditions. In light of this, we include a term to stimulate asymmetry in the learning process, which typically also leads to a significant decrease in training time.
    Original languageEnglish
    Pages (from-to)151-156
    Number of pages6
    JournalEurophysics Letters
    Issue number2
    Publication statusPublished - Apr 1996

    Bibliographical note

    Copyright of EDP Sciences


    • probability theory
    • stochastic processes
    • and statistics


    Dive into the research topics of 'Finite-size effects in on-line learning of multilayer neural networks'. Together they form a unique fingerprint.

    Cite this