GitHub - facebookresearch/XLM: PyTorch original implementation of Cross-lingual Language Model Pretraining.

GitHub - facebookresearch/XLM: PyTorch original implementation of Cross-lingual Language Model Pretraining.

PyTorch original implementation of Cross-lingual Language Model Pretraining. - facebookresearch/XLM

11 mentions: @GuillaumeLample@alex_conneau@ikuyamada@PythonHub@thammegowda@PapersTrending@PapersTrending@PapersTrending
Date: 2019/06/29 20:18

Referring Tweets

@GuillaumeLample If you want to train BERT from scratch in @PyTorch, you can check out our XLM repository! Our English model outperforms the original BERT on all GLUE tasks, although it's trained on the same data and without the next sentence prediction task t.co/zqFAQd7I1q @alex_conneau
@alex_conneau Using the same data (Wiki/TBC) and without the next sentence prediction task, our GLUE submission outperforms BERT on all tasks. Check out our XLM @PyTorch repo to train BERT from scratch in multiple languages. Code and models available at t.co/1An89JVEIh @GuillaumeLample t.co/0beXyIRoBr
@ikuyamada FacebookのXLMも、GLUEタスクでBERTを越えたということで、GitHubページを見てみると、BERT-largeの半分しか層はないが(12層)、BERT-largeよりパラメータ数は多く (BERT-largeの約倍)、何故かと思ったらembeddingの次元がBERT-largeの倍の2048次元になっている t.co/3InkgYvMOp
@thammegowda I like how t.co/jtn1qKyknq is forked out of t.co/DwaetzTgSE After decades of Supervised Machine Translation, Unsupervised MT is stealing the stage (BLEU score on En->Fr WMT test) 2017: ~15 2018: ~27 2019 Jan: ~33 2019 May: ~37 35+ BLEU is *really good👏 next?🤔

Related Entries

Read more 子どもの言語獲得のモデル化とNN Language ModelsNN
0 users, 0 mentions 2018/10/05 03:23
Read more Generalized Language Models
1 users, 24 mentions 2019/02/03 02:18
Read more GitHub - NVIDIA/Megatron-LM: Ongoing research training transformer language models at scale, includi...
0 users, 3 mentions 2019/09/20 03:48