[2007.08093] Data-driven effective model shows a liquid-like deep learningopen searchopen navigation menucontact arXivsubscribe to arXiv mailings

Geometric structure of an optimization landscape is argued to be fundamentally important to support the success of deep learning. However, recent research efforts focused on either of toy random models with unrealistic assumptions and numerical evidences about different shapes of the optimization landscape, thereby lacking a unified view about the nature of the landscape. Here, we propose a statistical mechanics framework by directly building a least structured model of the high-dimensional weight space, considering realistic structured data, stochastic gradient descent algorithms, and the computational depth of the network parametrized by weight parameters. We also consider whether the number of network parameters outnumbers the number of supplied training data, namely, over- or under-parametrization. Our least structured model predicts that the weight spaces of the under-parametrization and over-parameterization cases belong to the same class. These weight spaces are well-connected w

3 mentions: @tripdancer0916@villageofsound
Keywords: deep learning
Date: 2020/11/19 09:52

Referring Tweets

@tripdancer0916 DNNのシナプス重み空間におけるlandscapeの解析にはこれまでランダム行列のようなtoy modelか数値実験ベースのものしかなかったが、シナプスを確率分布として学習する枠組みを用いることで「タスクが解けるDNN」をイジングモデルとして統計力学的に解析することに成功した。 t.co/XDhJ4DBESF t.co/mlCJVt6mvs

Related Entries

Read more Non-Negative Matrix Factorization - ML Wiki
1 users, 0 mentions 2018/12/05 04:15
Read more Introducing LCA: Loss Change Allocation for Neural Network Training | Uber Engineering BlogIntroduci...
1 users, 0 mentions 2019/09/10 09:00
Read more Uber Has Been Quietly Assembling One of the Most Impressive Open Source Deep Learning Stacks in the ...
1 users, 0 mentions 2020/03/07 17:21
Read more ECCV 2020 digest. The most interesting self-driving… | by Peter Ondruska | Sep, 2020 | Towards Data ...
1 users, 0 mentions 2020/09/11 23:59
Read more SA-GANの実装から見る画像のSelf attention | Shikoan's ML Blog
1 users, 0 mentions 2019/08/29 16:24