[2011.04006] Long Range Arena: A Benchmark for Efficient Transformersopen searchopen navigation menucontact arXivsubscribe to arXiv mailings

Transformers do not scale very well to long sequence lengths largely because of quadratic self-attention complexity. In the recent months, a wide spectrum of efficient, fast Transformers have been proposed to tackle this problem, more often than not claiming superior or comparable model quality to vanilla Transformer models. To this date, there is no well-established consensus on how to evaluate this class of models. Moreover, inconsistent benchmarking on a wide spectrum of tasks and datasets makes it difficult to assess relative model quality amongst many models. This paper proposes a systematic and unified benchmark, LRA, specifically focused on evaluating model quality under long-context scenarios. Our benchmark is a suite of tasks consisting of sequences ranging from $1K$ to $16K$ tokens, encompassing a wide range of data types and modalities such as text, natural, synthetic images, and mathematical expressions requiring similarity, structural, and visual-spatial reasoning. We syst

3 mentions: @cgarciae88@Maxwell_110@nlpguy_
Keywords: transformer
Date: 2020/11/21 12:53

Referring Tweets

@cgarciae88 Deep Mind recently came out with Long Range Arena, a benchmark suite to pit Efficient Transformers against each other in Long Range Tasks 🏃🏃‍♀️🏃‍♂️. Do we finally have a winner? Paper: t.co/4Dm3NU11tl t.co/NkOKBuxmRU
@Maxwell_110 Google Research と DeepMind が長いシーケンスの入力を伴うタスクのベンチマーク「Long Range Arena」を発表 👇 t.co/l7zuLq7yNI seq-length に応じて二次関数的に大きさが増加してしまう transformer 系モデルの欠点を評価・改善するための活用を想定 github 👇 t.co/Hc3RQaFWht t.co/C7Tdlal1Wx

Related Entries

Read more 2019 Data Science Bowl | Kaggle
0 users, 2 mentions 2020/01/23 02:20
Read more 『Kaggleで勝つデータ分析の技術』の誤植情報を集める - Qiita
0 users, 1 mentions 2020/08/26 23:21
Read more Cornell Birdcall Identification | Kaggle
0 users, 1 mentions 2020/09/18 00:53
Read more Explaining Keras image classifier predictions with Grad-CAM — ELI5 0.9.0 documentation
0 users, 1 mentions 2020/10/17 15:51
Read more Kaggle Meetup: Cell Nucleus Detection - YouTube
0 users, 1 mentions 2020/11/28 00:53