[2110.05719] Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations

Majority voting and averaging are common approaches employed to resolve annotator disagreements and derive single ground truth labels from multiple annotations. However, annotators may systematically disagree with one another, often reflecting their individual biases and values, especially in the case of subjective tasks such as detecting affect, aggression, and hate speech. Annotator disagreements may capture important nuances in such tasks that are often ignored while aggregating annotations to a single ground truth. In order to address this, we investigate the efficacy of multi-annotator models. In particular, our multi-task based approach treats predicting each annotators' judgements as separate subtasks, while sharing a common learned representation of the task. We show that this approach yields same or better performance than aggregating labels in the data prior to training across seven different binary classification tasks. Our approach also provides a way to estimate uncertaint

2 mentions: @math_rachel@math_rachel
Date:

Referring Tweets

@math_rachel
@math_rachel - Annotator disagreements may capture important nuances ignored by a single ground truth - A multi-task based approach yields same or better performance than aggregating labels prior to training Interesting paper: t.co/lQoFjUtmbS @aidaa @blahtino @vinodkpg h/t @rajiinio t.co/UHT7Kx73ib
@math_rachel
@math_rachel "Preserving different annotators’ perspectives until prediction step provides better flexibility for downstream applications." 4/ t.co/lQoFjUtmbS

Related Entries

[2002.06746] Learning Individually Fair Classifier with Path-Specific Causal-Effect Constraint
Read more [2002.06746] Learning Individually Fair Classifier with Path-Specific Causal-Effect Constraint
0 users, 1 mentions 2021/12/29 09:09
単位正方形の射影変換の変換係数 - Speaker Deck
Read more 単位正方形の射影変換の変換係数 - Speaker Deck
0 users, 0 mentions 2021/11/11 06:09
GitHub - microsoft/dowhy: DoWhy is a Python library for causal inference that supports explicit mode...
Read more GitHub - microsoft/dowhy: DoWhy is a Python library for causal inference that supports explicit mode...
11 users, 1 mentions 2021/11/29 06:08
Welcome to the BBOB workshop series! — The BBOB workshop series 2.5 documentation
Read more Welcome to the BBOB workshop series! — The BBOB workshop series 2.5 documentation
0 users, 1 mentions 2021/12/29 07:37
フリーランスのデータアナリストを1年やってみて - Data Analystのメモ帳
Read more フリーランスのデータアナリストを1年やってみて - Data Analystのメモ帳
0 users, 1 mentions 2021/12/31 09:14

ML-Newsについて

機械学習の技術に関する情報は流速も早いし、分野も多様でキャッチアップが大変です。Twitterで機械学習用のリストを作っても、普段は機械学習以外の話題が多く流れており、効率的に情報収集するのは困難です。

ML-NewsはSNSを情報源とした機械学習に特化したニュースサイトです。機械学習に関する論文ブログライブラリコンペティション発表資料勉強会などの最新の情報を効率的に収集できます。

機械学習を応用した自然言語処理、画像認識、情報検索などの分野の情報や機械学習で必要になるデータ基盤やMLOpsの話題もカバーしています。
安定したサイト運営のためにGitHub sponsorを募集しています。

お知らせ

  • 2021/12/31: デザインを刷新しました
  • 2021/04/08: 日本語Kaggleのカテゴリを新設しました