[1906.02940] Selfie: Self-supervised Pretraining for Image Embedding

We introduce a pretraining technique called Selfie, which stands for SELF-supervised Image Embedding. Selfie generalizes the concept of masked language modeling to continuous data, such as images. Given masked-out patches in an input image, our method learns to select the correct patch, among other "distractor" patches sampled from the same image, to fill in the masked location. This classification objective sidesteps the need for predicting exact pixel values of the target patches. The pretraining architecture includes a network of convolutional blocks to process patches followed by an attention pooling network to summarize the content of unmasked patches before predicting masked ones. During finetuning, we reuse the convolutional weights found by pretraining. We evaluate our method on three benchmarks (CIFAR-10, ImageNet 32 x 32, and ImageNet 224 x 224) with varying amounts of labeled data, from 5% to 100% of the training sets. Our pretraining method provides consistent improvements

5 mentions: @lmthang@quantrad@thtrieu_
Date: 2019/06/10 02:18

Referring Tweets

@lmthang Introducing Selfie, an unsupervised pretraining method that extends the concept of masked language model (e.g., BERT) towards continuous data, such as images. Overall challenging, but promising in low-data regime! Joint work with @thtrieu_ @quocleix https://t.co/KYXyL5eHIK https://t.co/A6ogEIFjtS
@thtrieu_ Great paper exploring attention architecture for images! We encounter similar results in our latest work https://t.co/C28anYyYvT. Table 5 shows that keeping first layers conv, while using attention for last layers improve ResNet performance. https://t.co/ZLlHX6DzPi

Related Entries

Read more Neural Text Embeddings for Information Retrieval (WSDM 2017)
5 users, 0 mentions 2018/12/05 22:16
Read more Document Embedding Techniques - Towards Data Science
0 users, 4 mentions 2019/09/09 13:56
Read more [DL輪読会]Learning an Embedding Space for Transferable Robot Skills
0 users, 0 mentions 2018/04/24 10:16
Read more Document Embedding Techniques - Towards Data Science
0 users, 6 mentions 2019/09/12 19:05
Read more Variational Autoencoder in Tensorflow - facial expression low dimensional embedding - Machine learni...
0 users, 0 mentions 2018/04/22 03:40