Evaluating Models' Local Decision Boundaries via Contrast Sets
Authors:
Matt Gardner,
Yoav Artzi,
Victoria Basmova,
Jonathan Berant,
Ben Bogin,
Sihao Chen,
Pradeep Dasigi,
Dheeru Dua,
Yanai Elazar,
Ananth Gottumukkala,
Nitish Gupta,
Hanna Hajishirzi,
Gabriel Ilharco,
Daniel Khashabi,
Kevin Lin,
Jiangming Liu,
Nelson F. Liu,
Phoebe Mulcaire,
Qiang Ning,
Sameer Singh,
Noah A. Smith,
Sanjay Subramanian,
Reut Tsarfaty,
Eric Wallace,
Ally Zhang
, et al. (1 additional authors not shown)
Abstract:
Standard test sets for supervised learning evaluate in-distribution generalization. Unfortunately, when a dataset has systematic gaps (e.g., annotation artifacts), these evaluations are misleading: a model can learn simple decision rules that perform well on the test set but do not capture a dataset's intended capabilities. We propose a new annotation paradigm for NLP that helps to close systemati…
▽ More
Standard test sets for supervised learning evaluate in-distribution generalization. Unfortunately, when a dataset has systematic gaps (e.g., annotation artifacts), these evaluations are misleading: a model can learn simple decision rules that perform well on the test set but do not capture a dataset's intended capabilities. We propose a new annotation paradigm for NLP that helps to close systematic gaps in the test data. In particular, after a dataset is constructed, we recommend that the dataset authors manually perturb the test instances in small but meaningful ways that (typically) change the gold label, creating contrast sets. Contrast sets provide a local view of a model's decision boundary, which can be used to more accurately evaluate a model's true linguistic capabilities. We demonstrate the efficacy of contrast sets by creating them for 10 diverse NLP datasets (e.g., DROP reading comprehension, UD parsing, IMDb sentiment analysis). Although our contrast sets are not explicitly adversarial, model performance is significantly lower on them than on the original test sets---up to 25\% in some cases. We release our contrast sets as new evaluation benchmarks and encourage future dataset construction efforts to follow similar annotation processes.
△ Less
Submitted 1 October, 2020; v1 submitted 6 April, 2020;
originally announced April 2020.
ORB: An Open Reading Benchmark for Comprehensive Evaluation of Machine Reading Comprehension
Authors:
Dheeru Dua,
Ananth Gottumukkala,
Alon Talmor,
Sameer Singh,
Matt Gardner
Abstract:
Reading comprehension is one of the crucial tasks for furthering research in natural language understanding. A lot of diverse reading comprehension datasets have recently been introduced to study various phenomena in natural language, ranging from simple paraphrase matching and entity ty** to entity tracking and understanding the implications of the context. Given the availability of many such d…
▽ More
Reading comprehension is one of the crucial tasks for furthering research in natural language understanding. A lot of diverse reading comprehension datasets have recently been introduced to study various phenomena in natural language, ranging from simple paraphrase matching and entity ty** to entity tracking and understanding the implications of the context. Given the availability of many such datasets, comprehensive and reliable evaluation is tedious and time-consuming for researchers working on this problem. We present an evaluation server, ORB, that reports performance on seven diverse reading comprehension datasets, encouraging and facilitating testing a single model's capability in understanding a wide variety of reading phenomena. The evaluation server places no restrictions on how models are trained, so it is a suitable test bed for exploring training paradigms and representation learning for general reading facility. As more suitable datasets are released, they will be added to the evaluation server. We also collect and include synthetic augmentations for these datasets, testing how well models can handle out-of-domain questions.
△ Less
Submitted 29 December, 2019;
originally announced December 2019.
