[2010.06682] Are all negatives created equal in contrastive instance discrimination?open searchopen navigation menucontact arXivsubscribe to arXiv mailings

Self-supervised learning has recently begun to rival supervised learning on computer vision tasks. Many of the recent approaches have been based on contrastive instance discrimination (CID), in which the network is trained to recognize two augmented versions of the same instance (a query and positive) while discriminating against a pool of other instances (negatives). The learned representation is then used on downstream tasks such as image classification. Using methodology from MoCo v2 (Chen et al., 2020), we divided negatives by their difficulty for a given query and studied which difficulty ranges were most important for learning useful representations. We found a minority of negatives -- the hardest 5% -- were both necessary and sufficient for the downstream task to reach nearly full accuracy. Conversely, the easiest 95% of negatives were unnecessary and insufficient. Moreover, the very hardest 0.1% of negatives were unnecessary and sometimes detrimental. Finally, we studied the pr

4 mentions: @arimorcos@arimorcos@jbohnslav@KevinKaichuang
Date: 2020/10/15 14:22

Referring Tweets

@arimorcos Are all negatives created equal in contrastive instance discrimination? In new work led by Tiffany Cai, we show that only the hardest 5% of negatives per query are both necessary and largely sufficient for self-supervised learning. Tweetprint time! t.co/ijyrMb4zGG t.co/dIeR7DptVF
@arimorcos This work was led by Tiffany Cai, a former AI resident at @facebookai, and was done in collaboration with @jefrankle , @davidjschwab, and myself. For more details, see our paper here: t.co/ijyrMb4zGG
@KevinKaichuang Difficult negative examples are both necessary and sufficient for contrastive pretraining. Tiffany Cai @jefrankle @davidjschwab @arimorcos t.co/q99j6MAWGL t.co/rvSRGN4Grw

Related Entries

Read more [2003.03461] Semi-Supervised StyleGAN for Disentanglement Learningcontact arXivarXiv Twitter
0 users, 7 mentions 2020/03/10 03:50
Read more [2004.01793] Self-Supervised Viewpoint Learning From Image Collectionscontact arXivarXiv Twitter
0 users, 6 mentions 2020/04/08 13:05
Read more [2004.05155] Learning to Explore using Active Neural SLAMcontact arXivarXiv Twitter
0 users, 6 mentions 2020/04/13 03:51
Read more [2004.07219] Datasets for Data-Driven Reinforcement Learningopen searchopen navigation menucontact a...
0 users, 4 mentions 2020/04/16 03:51
Read more [2007.06600] Closed-Form Factorization of Latent Semantics in GANsopen searchopen navigation menucon...
0 users, 4 mentions 2020/07/18 23:21