BLM -s/l E : A structured dataset of E nglish spray-load verb alternations for testing generalization in LLM s (2023)

Abstract: Current NLP models appear to be achieving performance comparable to human capabilities on well-established benchmarks. New benchmarks are now necessary to test deeper layers of understanding of natural languages by these models. Blackbird’s Language Matrices are a recently developed framework that draws inspiration from tests of human analytic intelligence. The BLM task has revealed that successful performances in previously studied linguistic problems do not yet stem from a deep understanding of the generative factors that define these problems. In this study, we define a new BLM task for predicate-argument structure, and develop a structured dataset for its investigation, concentrating on the spray-load verb alternations in English, as a case study. The context sentences include one alternant from the spray-load alternation and the target sentence is the other alternant, to be chosen among a minimally contrastive and adversarial set of answers. We describe the generation process of the dataset and the reasoning behind the generating rules. The dataset aims to facilitate investigations into how verb information is encoded in sentence embeddings and how models generalize to the complex properties of argument structures. Benchmarking experiments conducted on the dataset and qualitative error analysis on the answer set reveal the inherent challenges associated with the problem even for current high-performing representations.

## introduction in recent years, fuelled by the rise of deep learning methods, nlp models have undergone a remarkable transformation, becoming exceptionally powerful and achieving performance comparable to human capabilities on well-established benchmarks @xcite . ## blm-s/le generation process task-solvers in rpm identify patterns based on the available information to identify the missing piece that completes the matrix @xcite . similarly, blms correspond to a problems where only one answer satisfies the constraints defined by the given contexts. in a blm problem, a linguistic phenomenon is presented as an incomplete sequence of sentences (context), deliberately designed to follow given linguistic rules and given linguistic properties. context 1 alternate 1 2 passivized theme with overt agent 3 passivized theme with overt agent and loc 4 passivized theme with overt loc 5 passivized loc with overt agent 6 passivized loc with overt agent and theme 7 passivized loc with overt theme ? alternate 2 the task is to identify the missing element that continues the context pattern based on the observed relationships. the missing element is the only correct answer that represents the solution to the context pattern, among many elements in a carefully curated answer set consisting of minimally differing sentences @xcite . ## creating the dataset to instantiate the templates, we follow a pipeline process comprising several steps. 1. find relevant verbs in naturally occurring seed sentences. each step is discussed in the following paragraphs. figure 5 illustrates the workflow. natural instances of class verbs dedicated rich repositories grouping verbs into semantically coherent classes based on shared syntactic behavior exist @xcite . we retrieved the lexical verbs belonging to the same class as spray foot_5 and the verb load from verbnet @xcite to identify a set of 30 verbs that undergo the spray/load alternation. 3 the data generation process begins by selecting a naturally occurring example for each verb in the class. to accomplish this, we use the spike platform foot_7 @xcite , which offers a collection of english corpora from various text genres (encyclopedic entries, internet reviews, among others). we favoured extracting natural contexts, and we prioritized examples obtained from the amazon reviews subcorpus. we performed the search of different inflected forms of the verbs and analysed manually the naturally occurring examples in the output. for example, the search for different inflected forms of the verb spritz allowed us to retrieve a sentence like i spritz the fish with low-fat oil (amazon customer; a19vw07iuhnv1r). we removed non-target examples (e.g. those cases in which the pp does not represent an argument of the verb, but it is embedded in the nominal phrase, as in this bulb scatters dim light with a green hue around; id: amg enthusiast; ai9jtx5zx0ojg). care was also taken to vary the types of sentences as much as possible (e.g. imperatives, relatives) and pronominal entities (e.g., to counter the fact that a good portion of amazon reviews involving these verbs are indeed in first person singular). ## experiment we present two baselines to investigate the challenge of detecting the underlying patterns of complex syntactic-semantic mapping in transformerbased sentence embeddings @xcite . our blms for the spray/load alternation are a case study of this kind of problem. to provide a benchmark for the task, we choose architectures that are simple, but effective -a threelayer feed-forward neural network, and a threelayer convolutional neural network -as proven by their numerous applications in nlp. they are described in detail in section 4.2. ## related work the current paper does not have a direct comparison, since it is the first proposal of a dataset for a verb alternation adopting a blm scheme. it is directly connected to the works on blm on agreement, studies on verb classification and syntactic rules and, partially, on structured datasets adopted in computer vision @xcite . blm for agreement the proposed structured dataset enriches the typology of linguistic phenomena that can be tested with blms. previous studies focused on investigating agreement patterns between the subject and verb in french @xcite . one of the difficulties of the experiments was provided by the increasing number of attractors between the subject and the verb, which could potentially interfere with the detection of the agreement relation between them. another difficulty was that the agreement patterns followed themselves a global pattern in the matrix that had to be identified. two exploratory baselines showed that despite the simplicity of the phenomenon, deep learning system do not manage to identify the underlying rules. verb alternations previous work has focussed on understanding the automatic learning of verb alternations in terms of syntactic and semantic properties of the verbs and their argument structures (allowing, or disallowing verb alternation) (see majewska and korhonen 2023 and reference therein). these properties have been explored in relation to their representation in large language models @xcite , encompassing various dimensions of performance across different models. in particular, @xcite suggest that llms with contextual embeddings encode lin-guistic information on verb alternation classes, at both the word and sentence levels. specifically, for the spray/load alternation, @xcite probed whether a verb entered this class by building two binary classifiers. the classifier predicts whether a verb can participate in both syntactic frames of the alternation (frame intended as syntactic environment, see also @xcite ). the binary classification task -done based on token embeddings -makes the targeted phenomenon explicit, but does not provide insights into how this linguistic property of verbs is encoded, or how it interacts with the verb's arguments. ## conclusion in this paper, we have introduced blm-s/le, a blm-structured dataset for the english spray/load verb alternation. this is a complex case study for the larger problem of predicate-argument structure. the dataset comprises two groups of templates. each group presents one of the alternants, while the other alternant needs to be inferred based on the information provided by the context of the blm. the answer set is designed to exhibit minimal superficial variations from the correct answer but significant differences in the underlying structure. through error analysis, we gain a better understanding of the types of information that baseline models manage to learn, and what aspects instead will require specifically developed models and architectures. our findings indicate that, on the one hand, curated and structured datasets, such as the one presented in this study, have the potential to lead to articulated understanding of learning. on the other hand, we conducted experiments using two different architectures, and the results highlight the usefulness of the task, whose solution requires detecting underlying linguistic properties and rules. overall, these findings provide further evidence supporting the use of our blms as an innovative benchmark to reach better qualitative understanding of learning in current neural network models.