[2203.07472] Uncertainty Estimation for Language Reward Models

Language models can learn a range of capabilities from unsupervised training on text corpora. However, to solve a particular problem (such as text summarization) it is typically necessary to fine-tune them on a task-specific dataset. It is often easier for humans to choose between options than to provide labeled data, and prior work has achieved state-of-the-art performance by training a reward model from such preference comparisons. However, collecting a large preference comparison dataset is still expensive -- and the learned reward models are unreliable out-of-distribution. We seek to address these problems via uncertainty estimation, which can improve sample efficiency and robustness using active learning and risk-averse reinforcement learning (RL). Specifically, we use bootstrap aggregating (bagging) to train an ensemble of reward models differing in the initialization of their final layer. Ensembles have proved successful in prior applications of active learning, but we find that

1 mentions: @ARGleave
Date:

Referring Tweets

@ARGleave
@ARGleave Fine-tuning language models from human feedback can work great but is expensive: prior work summarizing text used >90k comparisons taking ~4 years of labor! In t.co/PMwlpGSEku we investigate whether active learning can improve sample efficiency. With @geoffreyirving (1/6) t.co/gsDxh8l9cz

Related Entries

[2102.00554] Sparsity in Deep Learning: Pruning and growth for efficient inference and training in n...
Read more [2102.00554] Sparsity in Deep Learning: Pruning and growth for efficient inference and training in n...
0 users, 10 mentions 2021/02/07 05:21
[2101.09978] GUIGAN: Learning to Generate GUI Designs Using Generative Adversarial Networks
Read more [2101.09978] GUIGAN: Learning to Generate GUI Designs Using Generative Adversarial Networks
0 users, 3 mentions 2021/02/12 11:21
[2012.06908] The Lottery Tickets Hypothesis for Supervised and Self-supervised Pre-training in Compu...
Read more [2012.06908] The Lottery Tickets Hypothesis for Supervised and Self-supervised Pre-training in Compu...
0 users, 5 mentions 2021/07/01 12:17
SNN Workshop 2021
Read more SNN Workshop 2021
0 users, 2 mentions 2021/07/07 16:39
[2202.04200] MaskGIT: Masked Generative Image Transformer
Read more [2202.04200] MaskGIT: Masked Generative Image Transformer
0 users, 7 mentions 2022/02/10 03:09

ML-Newsについて

機械学習の技術に関する情報は流速も早いし、分野も多様でキャッチアップが大変です。Twitterで機械学習用のリストを作っても、普段は機械学習以外の話題が多く流れており、効率的に情報収集するのは困難です。

ML-NewsはSNSを情報源とした機械学習に特化したニュースサイトです。機械学習に関する論文ブログライブラリコンペティション発表資料勉強会などの最新の情報を効率的に収集できます。

機械学習を応用した自然言語処理、画像認識、情報検索などの分野の情報や機械学習で必要になるデータ基盤やMLOpsの話題もカバーしています。
安定したサイト運営のためにGitHub sponsorを募集しています。

お知らせ

  • 2021/12/31: デザインを刷新しました
  • 2021/04/08: 日本語Kaggleのカテゴリを新設しました