[2001.04413] Backward Feature Correction: How Deep Learning Performs Deep Learningcontact arXivarXiv Twitter

How does a 110-layer ResNet learn a high-complexity classifier using relatively few training examples and short training time? We present a theory towards explaining this in terms of $\textit{hierarchical learning}$. We refer hierarchical learning as the learner learns to represent a complicated target function by decomposing it into a sequence of simpler functions to reduce sample and time complexity. This paper formally analyzes how multi-layer neural networks can perform such hierarchical learning efficiently and automatically simply by applying stochastic gradient descent (SGD). On the conceptual side, we present, to the best of our knowledge, the FIRST theory result indicating how very deep neural networks can still be sample and time efficient on certain hierarchical learning tasks, when NO KNOWN non-hierarchical algorithms (such as kernel method, linear regression over feature mappings, tensor decomposition, sparse coding) are efficient. We establish a new principle called "back

10 mentions: @SebastienBubeck@MSFTResearch@bikash2510
Keywords: deep learning
Date: 2020/01/14 21:50

Referring Tweets

@SebastienBubeck Exciting start of the year for theory of #DeepLearning! SGD on neural nets can: 1) simulate any other learning alg w. some poly-time init [Abbe & Sandon t.co/cjQ21TAkzh] 2) learn efficiently hierarchical concept classes [@ZeyuanAllenZhu & Y. Li t.co/TheMl9zYeR]
@MSFTResearch How does deep learning perform DEEP learning? Microsoft and CMU researchers establish a principle called "backward feature correction" and explain how very deep neural networks can actually perform DEEP hierarchical learning efficiently: t.co/9EtkaThXAT @ZeyuanAllenZhu
@bikash2510 Backward Feature Correction: How Deep Learning Performs Deep Learning!!!! Really intresting article about deep learning basic. #Deeplearning #DataScience #DL t.co/x4xQrFCE0A

Related Entries

Read more Deep Learning for NLP Best Practices
97 users, 0 mentions 2018/04/22 03:40
Read more The major advancements in Deep Learning in 2016 | Tryolabs Blog
0 users, 0 mentions 2018/04/22 03:40
Read more Deep Learningを用いた教師なし画像検査の論文調査 GAN/SVM/Autoencoderとか .pdf
0 users, 0 mentions 2018/10/09 06:53
Read more データマイニングコンペティションサイト Kaggle にも Deep Learning ブームがきてるかと思ったのでまとめる - 糞糞糞ネット弁慶
0 users, 0 mentions 2018/10/14 06:16
Read more Chainerで学ぶdeep learning
0 users, 0 mentions 2018/04/22 03:40