Do Language Models discriminate between relatives and pseudorelatives? (2023)

Abstract: Large Language Models (LLMs) are often evaluated against massive benchmarks based on general-purpose tasks, which, despite being useful for concrete applications, tell us very little about the capacity of LLMs to learn specific and challenging aspects of the grammar. Here, we evaluate whether LLMs learn to identify a particular structure attested in Romance (and French in particular), called the pseudorelative. This structure, which is often surface-similar to a relative clause, is linked to robust syntactic and semantic restrictions. We present a series of experiments to test if LLMs pretrained on massive yet general corpora, manage to learn those various restrictions. Our results suggest that LLMs learn some but not all of these properties, but crucially fail at recognizing the most specific of them: cliticization.

## preliminary corpus study we run a corpus study to verify the claim that llms are mostly exposed to structurally ambiguous sentences such as (6). we start with simple exact google queries following the patterns in (7), where v denotes one of the verbs listed in the number of hits for these queries are gathered in to confirm this intuition, we matched a series of regular expressions 1 against a subset of the french oscar corpus @xcite @xcite , used to train models such as camembert @xcite . the results shown in ## experiment 1 adapting a recent psycholinguistic experiment @xcite , we test if 8 llms trained on general french corpora (see tab. 3 rows 1-8), learned the association between properties (3)-(4), pertaining to the type of the embedding verb and tense anaphoricity. the expected effects are: a 1 the regular expressions were refined from the templates in (7) to include all possible subject pronouns and allowed up to 3 unspecified words in the wildcard (*). this restricts the search space for ambiguous relative constructions of the from of (6) but ensures that other constructions (such as an unambiguous relative clause located "far away" from the perception verb) are not matched by accident. it also allows to speed-up the search. consequently, the matches and the proportions gathered respectively in the second and last columns of id model lang. reference 1 flaubert_base_uncased fr le et al. (2020) 2 camembert-base fr martin et al. (2020) 3 gpt2-base-french fr (cla) 4 gpt2-wechsel-french fr minixhofer et al. (2022) 5 bert-base-multilingual-cased multi devlin et al. (2018) 6 xlm-roberta-base multi conneau et al. (2019) 7 xlm-roberta-large multi conneau et al. (2019) 8 xlm-mlm-17-1280 multi lample and conneau (2019) 9 bert-large-cased en devlin et al. (2018) 10 gpt2-large en radford et al. (2019) 11 xlnet-large-cased en yang et al. (2019) building on @xcite , our proxy for grammaticality was taken to be the logprobability assigned to a given sentence by the llm (see equations below). it was computed using the minicons library @xcite . id best aic? verb_type tense interaction 1 n . ✗ n.s. . ✓ 2 n . ✓ ** ✓ n.s. 3 y n.s. ** ✓ * ✓ 4 y n.s. ** ✓ . ✓ 5 n n.s. ** ✓ n.s. 6 y n.s. ** ✓ . ✓ 7 y n.s. ** ✓ * ✓ 8 n ** ✓ n.s. n.s. english models (cf tab. 3, rows 9-11) tested on english equivalents of the stimuli exemplified in (8), did not exhibit similar effects -consistent with english not allowing pseudorelatives. plots of the distributions of grammaticality scores obtained with xlm-roberta-large (model 7) in both languages are given in figure 1 . ## experiment 2 we test the same llms on 4800 semi-automatically generated sentences following the template in ( 9 ) and differing in (1) head noun cliticization; (2) the gap's position (subject/object) and (3) the matrix verb's type (perception vs. attitude/action). id best aic? verb type tense interaction 5 y ** ✓ * ✗ * ✗ 6 n . ✓ . ✗ . ✗ 7 n n.s. * ✗ n.s. 8 n ** ✓ n.s. n.s. 9 n n.s. n.s. . ✓ 10 n ** ✓ * ✓ n.s. 11 n n.s. n.s. n.s. given this design, we expect an overall preference for matrix perception verbs, subject gaps and non-cliticized constructions, but also a positive interaction between perception verbs and clitics, perception verbs and subject gaps, clitics and subject gaps, and all three variables together. as tab. sentence clitic? gap verb_type (10a) n s perception (10b) n o perception (10c) y s perception (10d) y o perception (10e) n s attitude (10f) n o attitude (10g) y s attitude (10h) y o attitude the best performing model for this experiment appears to be a french-only gpt-2 model (model 3) -which was also among the best models for exp. 1. grammaticality scores corresponding to this model are plotted in fig. 4 . 5 experiment 3 we finally test property (5) on 4 bert-like lms fine-tuned to perform natural language inference (see id model lang. reference 12 camembert-base-xnli fr (doy) 13 xlm-roberta-largexnli-finetuned-mnli multi (ozs) 14 mdeberta-v3base-mnli-xnli multi (laurer et al., 2022) 15 mdeberta-v3-base-xnlimultilingual-nli-2mil7 multi (laurer et al., 2022) given a negated matrix perception verb embedding a clause c either as an infinitive or as a (pseudo)relative, with or without cliticization of its subject (2×2 design, see ( 11 )), we measure how likely llms are to infer the truth of c ("target inference", ti). we expect the ti to be overall stronger when the embedded clause is a relative as opposed to an infinitive, whether of not the head noun is cliticized. the effect of cliticization in the case of a structure embedding a relative is a little bit less clear: in the absence of cliticization the clause is ambiguous between a pr and a rc, and it is reasonable to think that both parses encourage a ti. assuming that the rc parse imposes a somewhat stronger ti than a pr parse, then we might expect sentences like (12b), which are unambiguously prs due to cliticization, to lead to a slightly weaker ti than sentences like (12a) which allow a rc parse. in other words, we expect non-cliticized sentences embedding an rc to yield the strongest ti. linear mixed-effect modeling reveals that embedded relative constructions systematically lead to a stronger target inference as opposed to infinitives (cf. id best aic? embedded clause (rc) clitic rc/clitic interaction 12 y ** (+) ** (-) ** (-) 13 y ** (+) ** (-) ** (+) 14 y ** (+) ** (-) ** (+) 15 y ** (+) ** (-) ** (-) ## discussion and outlook in this work, we investigated a structure (the pseudorelative) with two interesting distributional properties: (1) it can be ambiguous with a relative clause when the head noun is not cliticized; (2) the disambiguating (cliticized) structure is less frequent in corpora by several orders of magnitude. we think that the conjunction of these two properties makes learning the specific syntactic and semantic properties of prs particularly challenging, even for models trained on large amount of data. the experiments we run show that llms capture certain properties of prs, pertaining to acceptable filler-gap dependencies, matrix verbs, and tense combinations. interestingly, 3 /4 multilingual models exposed to both french (a pr-language) and english (devoid of prs) in exp. 1 managed to contrast the two languages. yet, the property that is perhaps the most specific to pseudorelatives, cliticization, does not seem to influence sentence probability scores in exp. 2, and inference patterns in exp. 3. this raises the question whether llms really get the specificity of the pseudorelative as a syntactic construction (exp. 2) with a specific semantics (exp. 3); or whether they simply recycle general processing heuristics applicable to other structures (e.g. standard rcs). such heuristics may involve a preference for shorter dependencies (subject-gaps) across the board; or learning a statistical correlation between the use of perception verbs and the agentive structure of the perceived event. future work may involve investigating other languages allowing the pseudorelative, but also refining the current design by looking at the influence of the different perception verbs. we think this might be particularly relevant given the rather large frequency differences between these verbs in actual corpora (cf. tables 1 and 2 ), and the potential imbalance between ambiguous vs. unambiguous pr-structures for each of those verbs.