[1910.14296] LIMIT-BERT : Linguistic Informed Multi-Task BERT

In this paper, we present a Linguistic Informed Multi-Task BERT (LIMIT-BERT) for learning language representations across multiple linguistic tasks by Multi-Task Learning (MTL). LIMIT-BERT includes five key linguistic syntax and semantics tasks: Part-Of-Speech (POS) tags, constituent and dependency syntactic parsing, span and dependency semantic role labeling (SRL). Besides, LIMIT-BERT adopts linguistics mask strategy: Syntactic and Semantic Phrase Masking which mask all of the tokens corresponding to a syntactic/semantic phrase. Different from recent Multi-Task Deep Neural Networks (MT-DNN) (Liu et al., 2019), our LIMIT-BERT is linguistically motivated and learning in a semi-supervised method which provides large amounts of linguistic-task data as same as BERT learning corpus. As a result, LIMIT-BERT not only improves linguistic tasks performance but also benefits from a regularization effect and linguistic information that leads to more general representations to help adapt to new ta

Keywords: bert
Date: 2019/11/07 15:50

Related Entries

Read more RNNで来月の航空会社の乗客数を予測する:TFLearnでLSTMからGRUまで実装しよう - DeepAgex_{1}, x_{2}, x_{3}, ..., x_{n}x_{1}x_{2}y_{1...
0 users, 0 mentions 2018/04/22 03:40
Read more GitHub - jacobgil/pytorch-explain-black-box: PyTorch implementation of Interpretable Explanations of...
0 users, 0 mentions 2018/04/22 03:40
Read more GitHub - Ninebell/GaneratedHandsForReal_TIME: it referenced paper of GANerated Hands for Real-Time 3...
0 users, 0 mentions 2020/01/19 05:20
Read more GitHub - CLUEbenchmark/CLUENER2020: CLUENER2020 中文细粒度命名实体识别 Fine Grained Named Entity Recognition
0 users, 0 mentions 2020/01/20 00:51
Read more League of Deep Learning Superheroes
0 users, 0 mentions 2020/04/20 23:21