[2002.05829] HULK: An Energy Efficiency Benchmark Platform for Responsible Natural Language Processing

Computation-intensive pretrained models have been taking the lead of many natural language processing benchmarks such as GLUE. However, energy efficiency in the process of model training and inference becomes a critical bottleneck. We introduce HULK, a multi-task energy efficiency benchmarking platform for responsible natural language processing. With HULK, we compare pretrained models' energy efficiency from the perspectives of time and cost. Baseline benchmarking results are provided for further analysis. The fine-tuning efficiency of different pretrained models can differ a lot among different tasks and fewer parameter number does not necessarily imply better efficiency. We analyzed such phenomenon and demonstrate the method of comparing the multi-task efficiency of pretrained models. Our platform is available at https://sites.engineering.ucsb.edu/~xiyou/hulk/.

2 mentions: @WilliamWangNLP@saqibali_ca
Date: 2021/02/20 21:51

Referring Tweets

@WilliamWangNLP Congratulations to Xiyou Zhou and colleagues on the acceptance of Hulk benchmark to #EACL2021 demo! HULK: An Energy Efficiency Benchmark Platform for Responsible Natural Language Processing t.co/hr1cmRbUqq #NLProc t.co/bfsEvCe5UI

Related Entries

Read more [2012.05322] Deep Learning Segmentation of Complex Features in Atomic-Resolution Phase Contrast Tran...
0 users, 3 mentions 2020/12/11 02:21
Read more [2012.05015] Fusion of rain radar images and wind forecasts in a deep learning model applied to rain...
0 users, 2 mentions 2020/12/11 03:51
Read more [2012.11799] Enforcing exact physics in scientific machine learning: a data-driven exterior calculus...
0 users, 2 mentions 2021/01/27 14:21
Read more Teaching - The Stanford Natural Language Processing Group
0 users, 2 mentions 2021/02/24 23:22
Read more [1911.06612] Position Paper: Towards Transparent Machine Learning
0 users, 2 mentions 2021/02/28 15:52