[1904.12166] HELP: A Dataset for Identifying Shortcomings of Neural Models in Monotonicity Reasoning

Large crowdsourced datasets are widely used for training and evaluating neural models on natural language inference (NLI). Despite these efforts, neural models have a hard time capturing logical inferences, including those licensed by phrase replacements, so-called monotonicity reasoning. Since no large dataset has been developed for monotonicity reasoning, it is still unclear whether the main obstacle is the size of datasets or the model architectures themselves. To investigate this issue, we introduce a new dataset, called HELP, for handling entailments with lexical and logical phenomena. We add it to training data for the state-of-the-art neural models and evaluate them on test sets for monotonicity phenomena. The results showed that our data augmentation improved the overall accuracy. We also find that the improvement is better on monotonicity inferences with lexical replacements than on downward inferences with disjunction and modification. This suggests that some types of inferen

Keywords: dataset
Date: 2019/06/05 05:15

Related Entries

Read more Dataset for Semantic Urban Scene Understanding
0 users, 0 mentions 2018/10/12 14:56
Read more GitHub - muhaochen/bilingual_dictionaries: This repository contains the source code and links to som...
0 users, 1 mentions 2019/09/08 09:47
Read more On Building an Instagram Street Art Dataset and Detection Model
0 users, 38 mentions 2019/01/29 20:59
Read more GitHub - allenai/PeerRead: Data and code for Kang et al., NAACL 2018's paper titled "A Dataset of Pe...
0 users, 0 mentions 2018/05/01 07:07
Read more COCO-Text: Dataset for Text Detection and Recognition | SE(3) Computer Vision Group at Cornell Tech
2 users, 1 mentions 2019/04/01 14:17