[2010.02194] Self-training Improves Pre-training for Natural Language Understandingopen searchopen navigation menucontact arXivsubscribe to arXiv mailings

Unsupervised pre-training has led to much recent progress in natural language understanding. In this paper, we study self-training as another way to leverage unlabeled data through semi-supervised learning. To obtain additional data for a specific task, we introduce SentAugment, a data augmentation method which computes task-specific query embeddings from labeled data to retrieve sentences from a bank of billions of unlabeled sentences crawled from the web. Unlike previous semi-supervised methods, our approach does not require in-domain unlabeled data and is therefore more generally applicable. Experiments show that self-training is complementary to strong RoBERTa baselines on a variety of tasks. Our augmentation approach leads to scalable and effective self-training with improvements of up to 2.6% on standard text classification benchmarks. Finally, we also show strong gains on knowledge-distillation and few-shot learning.

2 mentions: @alex_conneau
Date: 2020/10/15 15:52

Referring Tweets

@alex_conneau Happy to share our latest paper: "Self-training Improves Pretraining for Natural Language Understanding" We show that self-training is complementary to strong unsupervised pretraining (RoBERTa) on a variety of tasks. Paper: t.co/Fi1N9UKao7 Code: t.co/SsPSENYw5L t.co/n4IUsYfVGF

Related Entries

Read more [2004.08145] NAIST COVID: Multilingual COVID-19 Twitter and Weibo Datasetopen searchopen navigation ...
0 users, 4 mentions 2020/04/24 12:58
Read more GitHub - lab-ml/labml: Organize PyTorch Experiments
0 users, 1 mentions 2020/06/03 14:21
Read more GitHub - microsoft/MSR-NLP-Projects: This is a list of open-source projects at Microsoft Research NL...
0 users, 2 mentions 2020/09/19 15:52
Read more [2009.10795] Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamicsopen searcho...
0 users, 6 mentions 2020/09/27 14:21
Read more [2004.12744] Augmenting Transformers with KNN-Based Composite Memory for Dialogueopen searchopen nav...
0 users, 1 mentions 2020/10/21 15:52