[2002.00041] On Implicit Regularization in $β$-VAEscontact arXivarXiv Twitter

While the impact of variational inference (VI) on posterior inference in a fixed generative model is well-characterized, its role in regularizing a learned generative model when used in variational autoencoders (VAEs) is poorly understood. We study the regularizing effects of variational distributions on learning in generative models from two perspectives. First, we analyze the role that the choice of variational family plays in imparting uniqueness to the learned model by restricting the set of optimal generative models. Second, we study the regularization effect of the variational family on the local geometry of the decoding model. This analysis uncovers the regularizer implicit in the $β$-VAE objective, and leads to an approximation consisting of a deterministic autoencoding objective plus analytic regularizers that depend on the Hessian or Jacobian of the decoding model, unifying VAEs with recent heuristics proposed for training regularized autoencoders. We empirically verify these

2 mentions: @studentofml@matt_vowels
Keywords: vae
Date: 2020/02/12 20:21

Referring Tweets

@studentofml Implicit regularization in beta-VAEs (joint work w/ Ben Poole @poolio): t.co/SaLKxDN1sR We study the regularizing effects of variational distributions on learning in generative models from two perspectives -- (1/3)
@matt_vowels @francoisfleuret This just came out and discusses how the combination of choice of prior and variational family can influence the identifiability (and hence semantic disentanglement) t.co/F3oh3L7tRf

Related Entries

Read more VAEに基づくグラフの表現学習 - Sansan Builders Box
0 users, 13 mentions 2020/01/14 02:00
Read more VAE+LSTMで時系列異常検知 - 某大学生の機械学習日記
4 users, 1 mentions 2018/12/25 14:15
Read more From Autoencoder to Beta-VAE
0 users, 7 mentions 2019/04/18 02:18
Read more [DL輪読会]SOM-VAE: Interpretable Discrete Representation Learning on Tim…
1 users, 2 mentions 2019/01/22 06:46
Read more 深層生成モデルを巡る旅(2): VAE - QiitaQiita
7 users, 8 mentions 2019/11/03 14:20