[2102.08850] Contrastive Learning Inverts the Data Generating Process

Contrastive learning has recently seen tremendous success in self-supervised learning. So far, however, it is largely unclear why the learned representations generalize so effectively to a large variety of downstream tasks. We here prove that feedforward models trained with objectives belonging to the commonly used InfoNCE family learn to implicitly invert the underlying generative model of the observed data. While the proofs make certain statistical assumptions about the generative model, we observe empirically that our findings hold even if these assumptions are severely violated. Our theory highlights a fundamental connection between contrastive learning, generative modeling, and nonlinear independent component analysis, thereby furthering our understanding of the learned representations as well as providing a theoretical foundation to derive more effective contrastive losses.

6 mentions: @wielandbr@hillbig@hillbig@Montreal_AI@ShahabBakht
Date: 2021/02/22 00:53

Referring Tweets

@hillbig 対比学習による教師なし表現学習が多くの後続タスクで有効なのは、対比学習がデータの生成過程(一様分布の潜在変数から可逆変換で生成)を逆向きに辿り生成因子を回転を除いて同定できるため。証明に使う仮定が完全に満たされなくても実験的には同定できる場合が多い。t.co/YDhNl0rKBC
@wielandbr Why is contrastive representation learning so useful? Our latest work shows that contrastive learning can invert the data generating process, and it paves the way towards more effective contrastive losses. Paper: t.co/uhVkSTDuP7 Website/Code: t.co/lzaLdT7n6t [1/5] t.co/MZuxpwen27
@hillbig Contrastive learning implicitly learns to invert the underlying generative process and identify the generative factors (up to rotation). This explains why contrastive learning is effective in a wide range of downstream tasks. t.co/YDhNl0rKBC
@ShahabBakht @patrickmineault Contrastive learning might be your friend: t.co/2snPt7G4K8

Related Entries

Read more [1901.06656] Training Neural Networks with Local Error Signals
1 users, 17 mentions 2019/01/23 15:46
Read more [1903.04933] Hierarchical Autoregressive Image Models with Auxiliary Decoders
1 users, 8 mentions 2019/03/15 09:47
Read more [1904.07392] NAS-FPN: Learning Scalable Feature Pyramid Architecture for Object Detection
1 users, 8 mentions 2019/04/18 00:48
Read more [1910.06961] Tiny Video Networks
1 users, 5 mentions 2019/10/16 23:18
Read more [2005.07186] Efficient and Scalable Bayesian Neural Nets with Rank-1 Factorsopen searchopen navigati...
3 users, 7 mentions 2020/05/18 17:21