C o DA 21: Evaluating Language Understanding Capabilities of NLP Models With Context-Definition Alignment (2022)

Abstract: Pretrained language models (PLMs) have achieved superhuman performance on many benchmarks, creating a need for harder tasks. We introduce CoDA21 (Context Definition Alignment), a challenging benchmark that measures natural language understanding (NLU) capabilities of PLMs: Given a definition and a context each for k words, but not the words themselves, the task is to align the k definitions with the k contexts. CoDA21 requires a deep understanding of contexts and definitions, including complex inference and world knowledge. We find that there is a large gap between human and PLM performance, suggesting that CoDA21 measures an aspect of NLU that is not sufficiently covered in existing benchmarks.

## introduction increasing computational power along with the design and development of large and sophisticated models that can take advantage of enormous corpora has drastically advanced nlp. for many tasks, finetuning pretrained transformer-based language models @xcite @xcite has improved the state of the art considerably. language models acquire knowledge during pretraining that is utilized during task-specific finetuning. on benchmarks that were introduced to encourage development of models that do well on a diverse set of nlu tasks (e.g., glue @xcite and superglue @xcite ), these models now achieve superhuman performance @xcite . the pretrain-then-finetune approach usually requires a great amount of labeled data, which is often not available or expensive to obtain, and results in specialized models that can perform well only on a single task. recently, it was shown that generative language models can be applied to many tasks figure 1 : the coda21 task is to find the correct alignment between contexts and definitions: c1-d4, c2-d1, c3-d2, c4-d3. the target words (c1:"dust", c2:"soil", c3:"marble", c4:"feathers"; not provided to the model) are replaced with a @xmath0. we formulate the alignment task as a text prediction task and evaluate, without finetuning, three plms on coda21: bert @xcite , roberta @xcite and gpt-2 @xcite . poor performance of the plms and a large gap between human and plm performance suggest that coda21 is an important benchmark for designing models with better nlu capabilities. ## results human performance on coda21. we asked two nlp phd students 8 to solve the task on s20, a random sample of size 20 from the noun part of coda21-clean-easy. to investigate the effect of the made-up word m, we experiment with several other words on the noun part of coda21-clean-easy. specifically, we investigate another nonce word "opyatzel", a single letter "x" and two frequent words "orange" and "cloud". show significant variability in performance, and perform comparably poor for all words tried. gpt2 versions, which perform considerably better than mlms on coda21, perform similarly for the two nonce words and single letter "x", which do not have a strong meaning. their performance drops significantly when the two frequent words are used as the made-up word, due to the effect of prior knowledge models have about these words. to investigate the effect of the pattern, we compared our pattern q(c, m) with two alternative patterns by evaluating gpt-2 xl on the noun part of coda21-clean-easy. patterns and the evaluation results are shown in to get a better sense of why the task is hard for plms, we give an example, from the coda21 subtask in figure 1 (also figure 2 and ## related work there are many datasets @xcite @xcite for evaluating language understanding of models. many adopt a text prediction setup: lambada (paperno 5 . match scores shown in bold correspond the contextdefinition pairs that are in the predicted alignment by the model that yields maximum total match score. et al., 2016) evaluates the understanding of discourse context, storycloze @xcite evaluates commonsense knowledge and so does hellaswag @xcite , but examples were adversarially mined. lama @xcite tests the factual knowledge contained in plms. in contrast to this prior work, coda21 goes beyond prediction by requiring the matching of pieces of text. wic (pilehvar and camacho-collados, 2019) is also based on matching, but coda21 is more complex (multiple contexts/definitions as opposed to a single binary match decision) and is not restricted to ambiguous words. wnlampro (schick and schütze, 2020) evaluates knowledge of subordinate relationships between words, and wdlampro (senel and schütze, 2021) understanding of words using dictionary definitions. again, matching multiple pieces of text with each other is much harder and therefore a promising task for benchmarking nlu. ## conclusion we introduced coda21, a new challenging benchmark that tests natural language understanding capabilities of plms. performing well on coda21 requires detailed understanding of contexts, performing complex inference and having world knowledge, which are crucial skills for nlp. all models we investigated perform clearly worse than humans, indicating a lack of these skills in the current state of the art in nlp. coda21 therefore is a promising benchmark for guiding the development of models with stronger nlu competence.