[2005.10242] Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphereopen searchopen navigation menucontact arXivarXiv Twitter

Contrastive representation learning has been outstandingly successful in practice. In this work, we identify two key properties related to the contrastive loss: (1) alignment (closeness) of features from positive pairs, and (2) uniformity of the induced distribution of the (normalized) features on the hypersphere. We prove that, asymptotically, the contrastive loss optimizes these properties, and analyze their positive effects on downstream tasks. Empirically, we introduce an optimizable metric to quantify each property. Extensive experiments on standard vision and language datasets confirm the strong agreement between both metrics and downstream task performance. Remarkably, directly optimizing for these two metrics leads to representations with comparable or better performance at downstream tasks than contrastive learning. Project Page: https://ssnl.github.io/hypersphere Code: https://github.com/SsnL/align_uniform

4 mentions: @mosko_mule@arxiv_cs_cv_pr@sk_gnkr96
Date: 2020/05/21 14:21

Referring Tweets

@mosko_mule Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere t.co/QYEN18Urzh CLによって似たサンプルは似た特徴を持ち,特徴は超球上に一様に分布する.これを直接損失とすることでも良い表現学習が可能.画像に加え言語でも実験している. t.co/tgozW5FTVz

Related Entries

Read more [2004.00184] A theory of independent mechanisms for extrapolation in generative modelscontact arXiva...
0 users, 2 mentions 2020/04/02 03:51
Read more [2004.00831] Improving 3D Object Detection through Progressive Population Based Augmentationcontact ...
0 users, 3 mentions 2020/04/03 09:51
Read more [2004.02866] There and Back Again: Revisiting Backpropagation Saliency Methodscontact arXivarXiv Twi...
0 users, 6 mentions 2020/04/08 05:21
Read more [2004.03891] Normalizing Flows with Multi-Scale Autoregressive Priorscontact arXivarXiv Twitter
0 users, 3 mentions 2020/04/10 08:21
Read more [2004.12906] Towards causal generative scene models via competition of expertsopen searchopen naviga...
0 users, 3 mentions 2020/05/03 09:51