[1802.09210] A representer theorem for deep neural networksopen searchopen navigation menucontact arXivsubscribe to arXiv mailings

We propose to optimize the activation functions of a deep neural network by adding a corresponding functional regularization to the cost function. We justify the use of a second-order total-variation criterion. This allows us to derive a general representer theorem for deep neural networks that makes a direct connection with splines and sparsity. Specifically, we show that the optimal network configuration can be achieved with activation functions that are nonuniform linear splines with adaptive knots. The bottom line is that the action of each neuron is encoded by a spline whose parameters (including the number of knots) are optimized during the training procedure. The scheme results in a computational structure that is compatible with the existing deep-ReLU, parametric ReLU, APL (adaptive piecewise-linear) and MaxOut architectures. It also suggests novel optimization challenges, while making the link with $\ell_1$ minimization and sparsity-promoting techniques explicit.

1 mentions: @Quasi_quant2010
Date: 2020/10/18 14:21

Referring Tweets

@Quasi_quant2010 最近、kernelやら勉強してるんだが、 RKHSをつかってDNNを考えるとどうなるかと調べていると、A Representer Theorem for Deep Neural Networks( t.co/arUAWiCapP )を発見。で、Reproducing Kernel Hilbert Spaces in Probability and Statistic( t.co/mOqn9T1kki )を読もうと決心した

Related Entries

Read more beta-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework | OpenReview
2 users, 1 mentions 2019/09/25 15:48
Read more [1708.02002] Focal Loss for Dense Object Detectioncontact arXivarXiv Twitter
3 users, 1 mentions 2020/03/08 14:21
Read more [2008.10242] When Inverse Propensity Scoring does not Work: Affine Corrections for Unbiased Learning...
0 users, 1 mentions 2020/09/03 05:21
Read more [1906.08511] From Zero-Shot Learning to Cold-Start Recommendationopen searchopen navigation menucont...
0 users, 1 mentions 2020/09/17 09:52
Read more [1807.00263] Accurate Uncertainties for Deep Learning Using Calibrated Regressionopen searchopen nav...
0 users, 1 mentions 2020/10/03 14:21