[1912.08777] PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization

Recent work pre-training Transformers with self-supervised objectives on large text corpora has shown great success when fine-tuned on downstream NLP tasks including text summarization. However, pre-training objectives tailored for abstractive text summarization have not been explored. Furthermore there is a lack of systematic evaluation across diverse domains. In this work, we propose pre-training large Transformer-based encoder-decoder models on massive text corpora with a new self-supervised objective. In PEGASUS, important sentences are removed/masked from an input document and are generated together as one output sequence from the remaining sentences, similar to an extractive summary. We evaluated our best PEGASUS model on 12 downstream summarization tasks spanning news, science, stories, instructions, emails, patents, and legislative bills. Experiments demonstrate it achieves state-of-the-art performance on all 12 downstream datasets measured by ROUGE scores. Our model also shows

1 mentions: @Maxwell_110
Keywords: 要約
Date:

Referring Tweets

@Maxwell_110
@Maxwell_110 PEGASUS 📝 t.co/w4Ae4X519s PEGASUS は Transformer-base の文書要約モデル Google Workspace 上の Google Doc の自動要約モデルの蒸留用教師にも使用されている (t.co/gKsHTQCuQo) 「文章単位のマスク部」を予測するという,要約タスクに似た事前学習を採用し要約の精度を向上 t.co/i7qsEU3kBG

Related Entries

RSNA-MICCAI Brain Tumor Radiogenomic Classification | Kaggle
Read more RSNA-MICCAI Brain Tumor Radiogenomic Classification | Kaggle
0 users, 1 mentions 2021/07/14 00:12
UVO - Workshop and Challenge
Read more UVO - Workshop and Challenge
0 users, 1 mentions 2021/08/25 22:37
231 - Semantic Segmentation of BraTS2020 - Part 0 - Introduction (and plan) - YouTube
Read more 231 - Semantic Segmentation of BraTS2020 - Part 0 - Introduction (and plan) - YouTube
0 users, 1 mentions 2021/09/04 00:09
[1908.10063] FinBERT: Financial Sentiment Analysis with Pre-trained Language Models
Read more [1908.10063] FinBERT: Financial Sentiment Analysis with Pre-trained Language Models
0 users, 1 mentions 2021/10/05 22:37
[2108.07002] Change is Everywhere: Single-Temporal Supervised Object Change Detection in Remote Sens...
Read more [2108.07002] Change is Everywhere: Single-Temporal Supervised Object Change Detection in Remote Sens...
0 users, 1 mentions 2021/11/18 22:37
[1604.02532] T-CNN: Tubelets with Convolutional Neural Networks for Object Detection from Videos
Read more [1604.02532] T-CNN: Tubelets with Convolutional Neural Networks for Object Detection from Videos
0 users, 1 mentions 2021/12/15 22:37

ML-Newsについて

機械学習の技術に関する情報は流速も早いし、分野も多様でキャッチアップが大変です。Twitterで機械学習用のリストを作っても、普段は機械学習以外の話題が多く流れており、効率的に情報収集するのは困難です。

ML-NewsはSNSを情報源とした機械学習に特化したニュースサイトです。機械学習に関する論文、ブログ、ライブラリ、コンペティション、発表資料、勉強会などの最新の情報を効率的に収集できます。

機械学習を応用した自然言語処理、画像認識、情報検索などの分野の情報や機械学習で必要になるデータ基盤やMLOpsの話題もカバーしています。
安定したサイト運営のためにGitHub sponsorを募集しています。

お知らせ

  • 2021/12/31: デザインを刷新しました
  • 2021/04/08: 日本語とKaggleのカテゴリを新設しました