S em E val-2022 Task 3: P re TENS -Evaluating Neural Networks on Presuppositional Semantic Knowledge (2022)

Abstract: We report the results of the SemEval 2022 Task 3, PreTENS, on evaluation the acceptability of simple sentences containing constructions whose two arguments are presupposed to be or not to be in an ordered taxonomic relation. The task featured two sub-tasks articulated as: (i) binary prediction task and (ii) regression task, predicting the acceptability in a continuous scale. The sentences were artificially generated in three languages (English, Italian and French). 21 systems, with 8 system papers were submitted for the task, all based on various types of fine-tuned transformer systems, often with ensemble methods and various data augmentation techniques. The best systems reached an F1-macro score of 94.49 (sub-task1) and a Spearman correlation coefficient of 0.80 (sub-task2), with interesting variations in specific constructions and/or languages.

## introduction a growing body of literature on the cognitive side of computational linguistics has tried to probe the ability of language models to recognize deviant linguistic structures. recognizing linguistic illformedness requires some degree of metalinguistic awareness in adult humans (i.e. the ability to say not just that there is 'something wrong' in a sentence, but also where the problem lies or how the sentence could be improved), and it is not clear whether and to what extent artificial systems can induce this type of knowledge purely from exposure to raw linguistic data @xcite . most of the previous investigations on this topic have focused on phenomena that are purely syntactic @xcite ; dislocated arguments with island effects, @xcite ; @xcite ; chowdhury and zamparelli (2018), clause embedding, @xcite or at the syntax/semantics interface (negative-polarity items, jumelet and hupkes 2018; argument structure, quantifier restrictions, @xcite , mostly using lstm architectures (but see @xcite ). the fact that many purely semantic effects result in the (non) availability of certain readings (e.g. the scope of a quantifier over a higher negation, in "i didn't see some people") makes it of course harder to translated them into computationally testable paradigms. the task we describe in this paper focuses on an area of purely semantic competence that, to the best of our knowledge, is unexplored in the computational literature, and one which gives rise to a robust intuition of deviance, triggered by the failure of a type of presupposition: the requirement for two nominal arguments to be (or not be) in an (ordered) taxonomic relation. these presuppositions are introduced by a wide variety of constructions, such as comparatives (1a), coordinations, verbs like prefer, modifiers headed by type or except (1b) etc. (see the task requires world knowledge, commonsense knowledge (in the sense discussed in @xcite , but also linguistic knowledge, to catch the requirement of the specific presuppositioninducing constructions. in this respect, the present task is closer in spirit to semeval-2020 task 4, subtask a, on the validation of sentences for commonsense @xcite , than to semeval 2016 task 3, where participants had to extract and identify the taxonomic relationships between two terms @xcite . besides nli practitioners, the task could be relevant for researchers interested in the potential of nns as cognitive/linguistic models (see e.g. @xcite . we believe that it is also a potentially useful addition to the toolbox of probes used to understand the inner working of current language models. ## participating systems we received six system description papers for both sub-tasks, plus two papers by teams that participated only in sub-task1, for a total of eight papers. as it can be observed in the following summaries of the approaches proposed, there were several points of methodological similarity, but also interesting differences. many teams experimented data augmentation techniques devised to overcome the limited amount of training data, in particular for the solution of sub-task2. these techniques ranged from the use of external resources to the generation of new sentences @xcite , @xcite and van den berg et al. ( 2022 )), to the automatic translations across the three languages considered @xcite and @xcite ), to mapping the likert scale results to the binary values. this was the strategy used by the first two teams classified @xcite and @xcite ) according to the global scores. as we can see by the description of the participating systems (see section 4.1), the majority of teams chose monolingual instead of multilingual models, especially in the resolution of sub-task1. the exceptions are represented by @xcite , who obtained the second position in the global ranking of both sub-tasks, by @xcite , and by @xcite who (in the resolution of subtask 2 only), used the multilingual version of the universal sentence encoder, since it yielded better performance than the monolingual one. interestingly enough, the top-ranked team in both sub-tasks @xcite ) found that, for all languages, the monolingual deberta-v3 models always outperformed the multilingual version. a further approach shared by the participating teams is represented by the adoption of ensemble methods. two main ensemble strategies were suggested. in the first one, the training data used to fine-tune the adopted model was split, obtaining different models, each with its training and validation sets. this was the case with the second-ranked system @xcite and with @xcite , but only in sub-task2. a second main approach used the fusion of the acceptability scores predicted by two different models. as described in the following subsection, @xcite combined the scores predicted by xlm-roberta @xcite and mbert @xcite , while @xcite merged the predictions made by ernie-m and deberta-v3 (only in sub-task1). ## results and discussion almost all teams submitted their runs for the three languages considered. tables 5 and 6 show, for each sub-task, the top submissions received from each team, along with the baseline scores. team names marked with ⋆ represent teams that have submitted system description papers. the rank column reports the position of the team in the ranking for global and language-specific scores. task 1 evaluation: to better understand the performances per construction of the models submitted, we report the average f1-macro (±std) of the top-3 submissions per language, in our results show that english -the most resource-rich language in terms of computational models and data -outperforms the italian and french models for correctly predicting presuppositions in these constructions. however, the english model performed below french in the un-like construction (e.g. "unlike trees, {*oaks / animals} are often mentioned in this text", presupp. a1̸ > <a2), and does quite poorly in generally ("i like oaks, and more generally {trees / *animals}", presupp. @xmath0. note that neither constructions were present in the training set. italian is aligned with english, while interestingly the french model seems to be capable of more accurate generalizations in both cases. task 2 evaluation: to gain a better understanding of the models generalization abilities in this task, we computed the root mean squared error (rmse) between the gold value and the average predicted value by the first three teams classified. this data is shown in figure 2 for each construction and for each language, along with the average value across languages. as it can be seen, the resulting picture contrasts substantially with that of sub-task1. the type and particular constructions, among the worst in the first sub-task, have the lowest error in sub-task2. the second task sees a substantial drop of andtoo (now the worst case) and comparatives (one of the best constructions in sub-task1). we also observe distinctions between languages across the two tasks. in particular, while the average performance for english across constructions is the highest in sub-task1, the french models obtained on average the best results in sub-task2. the success at predicting the presuppositional knowledge triggered by the same construction changes in the two sub-tasks (which are often demanded to different models in the various teams). for example, the french models are the best at classifying the acceptability label for the unlike construction but are the worst in predicting the human score for the same construction. conversely, comparatives turn out to be among the easiest constructions for the english models in sub-task1 but are the most mispredicted english type in sub-task2. quite interestingly, the presence of a construction in training doesn't always guarantee better performances. two notable examples are represented by drather and particular. despite being absent in the training set of the corresponding tasks, the first obtains the highest f1 score (for english and on average) and the second is among the top-predicted constructions in sub-task2. we leave a more thorough analysis of the systematicity of these trends to future work, where we will also consider the linguistic variants for each construction and the semantic categories of the nominal arguments involved. ## conclusion the semeval2022 task3 -pretens offered two sub-tasks aiming to investigate the effectiveness of computational models to detect a certain type of presuppositional failures induced by specific constructions. the task attracted a total of 21 teams, from both academia and industry. the findings showcases the power and ubiquity of large selfsupervised pre-trained models in mono-or multi-lingual settings. despite this apparent uniformity, the participants chose to use and combine the models in very different and creative ways, giving rise to a range of scores from 70.54% to 94.49%. the outcomes of the task highlights the ability of these transformer models to generalize to new/unknown construction in the test sets, but also the presence of intriguing differences in specific constructions and languages (e.g. in the binary task a1 and more generally a2 reaches a 75.2 f-measure in french but a 46.7 in italian). also worth further investigation is the lower correlation between the binary judgments and the human ratings in english -probably reflected in the .05 drop seen in the sub-task2 global score for this language, compared to french. the success of the task indicates a growing interest towards research on prediction models that can incorporate world knowledge and common sense, along with an understanding of the linguistic properties that condition the outcomes. we hope that this trend will continue and the pretens data will help researchers to probe future models for this ability. with this spirit, we make the dataset public.