[2001.04413] Backward Feature Correction: How Deep Learning Performs Deep Learningcontact arXivarXiv Twitter

How does a 110-layer ResNet learn a high-complexity classifier using relatively few training examples and short training time? We present a theory towards explaining this in terms of $\textit{hierarchical learning}$. We refer hierarchical learning as the learner learns to represent a complicated target function by decomposing it into a sequence of simpler functions to reduce sample and time complexity. This paper formally analyzes how multi-layer neural networks can perform such hierarchical learning efficiently and automatically simply by applying stochastic gradient descent (SGD). On the conceptual side, we present, to the best of our knowledge, the FIRST theory result indicating how very deep neural networks can still be sample and time efficient on certain hierarchical learning tasks, when NO KNOWN non-hierarchical algorithms (such as kernel method, linear regression over feature mappings, tensor decomposition, sparse coding) are efficient. We establish a new principle called "back

10 mentions: @SebastienBubeck@MSFTResearch@bikash2510
Keywords: deep learning
Date: 2020/01/14 21:50

Referring Tweets

@SebastienBubeck Exciting start of the year for theory of #DeepLearning! SGD on neural nets can: 1) simulate any other learning alg w. some poly-time init [Abbe & Sandon t.co/cjQ21TAkzh] 2) learn efficiently hierarchical concept classes [@ZeyuanAllenZhu & Y. Li t.co/TheMl9zYeR]
@MSFTResearch How does deep learning perform DEEP learning? Microsoft and CMU researchers establish a principle called "backward feature correction" and explain how very deep neural networks can actually perform DEEP hierarchical learning efficiently: t.co/9EtkaThXAT @ZeyuanAllenZhu
@bikash2510 Backward Feature Correction: How Deep Learning Performs Deep Learning!!!! Really intresting article about deep learning basic. #Deeplearning #DataScience #DL t.co/x4xQrFCE0A

Related Entries

Read more [1910.13051] ROCKET: Exceptionally fast and accurate time series classification using random convolu...
0 users, 2 mentions 2019/11/13 20:20
Read more GitHub - amygdala/tensorflow-workshop: This repo contains materials for use in a TensorFlow workshop...
0 users, 1 mentions 2020/01/15 09:50
Read more GitHub - econti/faucetml
0 users, 1 mentions 2020/01/22 02:21
Read more GitHub - SE-ML/awesome-seml: A curated list of articles that cover the software engineering best pra...
0 users, 2 mentions 2020/01/29 23:20
Read more Genomics in Machine Learning - Geonomics in Machine Learning
0 users, 2 mentions 2020/01/30 05:11