Large deviation analysis of function sensitivity in random deep neural networks

Bo Li; David Saad

doi:10.1088/1751-8121/ab6a6f

Large deviation analysis of function sensitivity in random deep neural networks

Bo Li, David Saad

College of Engineering and Physical Sciences

Research output: Contribution to journal › Article › peer-review

Abstract

Mean field theory has been successfully used to analyze deep neural networks (DNN) in the infinite size limit. Given the finite size of realistic DNN, we utilize the large deviation theory and path integral analysis to study the deviation of functions represented by DNN from their typical mean field solutions. The parameter perturbations investigated include weight sparsification (dilution) and binarization, which are commonly used in model simplification, for both ReLU and sign activation functions. We find that random networks with ReLU activation are more robust to parameter perturbations with respect to their counterparts with sign activation, which arguably is reflected in the simplicity of the functions they generate.

Original language	English
Article number	104002
Journal	Journal of Physics A: Mathematical and Theoretical
Volume	53
Issue number	10
Early online date	10 Jan 2020
DOIs	https://doi.org/10.1088/1751-8121/ab6a6f
Publication status	Published - 20 Feb 2020

Bibliographical note

© 2020 The Author(s). Published by IOP Publishing Ltd. Original content from this work may be used under the terms of the Creative
Commons Attribution 4.0 licence. Any further distribution of this work must maintain
attribution to the author(s) and the title of the work, journal citation and DOI.

Keywords

deep neural networks
function sensitivity
large deviation theory
path integral

Access to Document

10.1088/1751-8121/ab6a6fLicence: CC BY 3.0

Large Deviation Analysis
© 2019 The Authors
Submitted manuscript, 815 KB
Large deviation analysis of function sensitivity in random deep neural networks
As the Version of Record of this article is going to be / has been published on a gold open access basis under a CC BY 3.0 licence, this Accepted Manuscript is available for reuse under a CC BY 3.0 licence immediately
Accepted author manuscript, 1.02 MBLicence: CC BY 3.0
Li_2020_J._Phys._A__Math._Theor._53_104002
© 2020 The Author(s). Published by IOP Publishing Ltd. Original content from this work may be used under the terms of the Creative Commons Attribution 4.0 licence. Any further distribution of this work must maintain attribution to the author(s) and the title of the work, journal citation and DOI.
Final published version, 1.98 MBLicence: CC BY 3.0

https://arxiv.org/abs/1910.05769

Cite this

@article{bd435a2339ec479b8edc510eae3a90d4,

title = "Large deviation analysis of function sensitivity in random deep neural networks",

abstract = "Mean field theory has been successfully used to analyze deep neural networks (DNN) in the infinite size limit. Given the finite size of realistic DNN, we utilize the large deviation theory and path integral analysis to study the deviation of functions represented by DNN from their typical mean field solutions. The parameter perturbations investigated include weight sparsification (dilution) and binarization, which are commonly used in model simplification, for both ReLU and sign activation functions. We find that random networks with ReLU activation are more robust to parameter perturbations with respect to their counterparts with sign activation, which arguably is reflected in the simplicity of the functions they generate.",

keywords = "deep neural networks, function sensitivity, large deviation theory, path integral",

author = "Bo Li and David Saad",

note = "{\textcopyright} 2020 The Author(s). Published by IOP Publishing Ltd. Original content from this work may be used under the terms of the Creative Commons Attribution 4.0 licence. Any further distribution of this work must maintain attribution to the author(s) and the title of the work, journal citation and DOI.",

year = "2020",

month = feb,

day = "20",

doi = "10.1088/1751-8121/ab6a6f",

language = "English",

volume = "53",

journal = "Journal of Physics A: Mathematical and Theoretical",

issn = "1751-8113",

publisher = "IOP Publishing Ltd.",

number = "10",

}

TY - JOUR

T1 - Large deviation analysis of function sensitivity in random deep neural networks

AU - Li, Bo

AU - Saad, David

N1 - © 2020 The Author(s). Published by IOP Publishing Ltd. Original content from this work may be used under the terms of the Creative Commons Attribution 4.0 licence. Any further distribution of this work must maintain attribution to the author(s) and the title of the work, journal citation and DOI.

PY - 2020/2/20

Y1 - 2020/2/20

N2 - Mean field theory has been successfully used to analyze deep neural networks (DNN) in the infinite size limit. Given the finite size of realistic DNN, we utilize the large deviation theory and path integral analysis to study the deviation of functions represented by DNN from their typical mean field solutions. The parameter perturbations investigated include weight sparsification (dilution) and binarization, which are commonly used in model simplification, for both ReLU and sign activation functions. We find that random networks with ReLU activation are more robust to parameter perturbations with respect to their counterparts with sign activation, which arguably is reflected in the simplicity of the functions they generate.

AB - Mean field theory has been successfully used to analyze deep neural networks (DNN) in the infinite size limit. Given the finite size of realistic DNN, we utilize the large deviation theory and path integral analysis to study the deviation of functions represented by DNN from their typical mean field solutions. The parameter perturbations investigated include weight sparsification (dilution) and binarization, which are commonly used in model simplification, for both ReLU and sign activation functions. We find that random networks with ReLU activation are more robust to parameter perturbations with respect to their counterparts with sign activation, which arguably is reflected in the simplicity of the functions they generate.

KW - deep neural networks

KW - function sensitivity

KW - large deviation theory

KW - path integral

UR - https://doi.org/10.1088/1751-8121/ab6a6f

UR - http://www.scopus.com/inward/record.url?scp=85081296259&partnerID=8YFLogxK

U2 - 10.1088/1751-8121/ab6a6f

DO - 10.1088/1751-8121/ab6a6f

M3 - Article

SN - 1751-8113

VL - 53

JO - Journal of Physics A: Mathematical and Theoretical

JF - Journal of Physics A: Mathematical and Theoretical

IS - 10

M1 - 104002

ER -

Large deviation analysis of function sensitivity in random deep neural networks

Abstract

Bibliographical note

Keywords

Access to Document

Other files and links

Fingerprint

Cite this