For natural language understanding (NLU) technology to be maximally useful, it must be able to process language in a way that is not exclusively tailored to a specific task, genre, or dataset. In pursuit of this objective, we introduce the General Language Understanding Evaluation (GLUE) benchmark, a collection of tools for evaluating and analyzing the performance of models across a diverse set of existing NLU tasks. By including tasks with limited training data, GLUE is designed to favor and encourage models that share general linguistic knowledge across tasks. GLUE also includes a hand-crafted diagnostic test suite that enables detailed linguistic analysis of models. We evaluate baselines based on current methods for transfer and representation learning and find that multi-task training on all our tasks yields better results than training a separate model for each task. However, the low absolute performance of our best model indicates the need for improved general NLU systems. son. 2013. One billion word benchmark for measuring progress in statistical language modeling. arXiv preprint 1312.3005.
NLI is also included in the GLUE benchmark for evaluating general language understanding REF .
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
ArXiv
ArXiv
Computer Science