Dependency resolution at the syntax-semantics interface: psycholinguistic and computational insights on control dependencies (2023)

Abstract: Using psycholinguistic and computational experiments we compare the ability of humans and several pre-trained masked language models to correctly identify control dependencies in Spanish sentences such as ‘José le prometió/ordenó a María ser ordenado/a’ (‘Joseph promised/ordered Mary to be tidy’). These structures underlie complex anaphoric and agreement relations at the interface of syntax and semantics, allowing us to study lexically-guided antecedent retrieval processes. Our results show that while humans correctly identify the (un)acceptability of the strings, language models often fail to identify the correct antecedent in non-adjacent dependencies, showing their reliance on linearity. Additional experiments on Galician reinforce these conclusions. Our findings are equally valuable for the evaluation of language models’ ability to capture linguistic generalizations, as well as for psycholinguistic theories of anaphor resolution.

## introduction treating pre-trained language models (lms) as psycholinguistic subjects via the behavioral evaluation of their probability distributions has proven to be a very useful strategy to study to which extent they are able to generalize grammatical information from raw text @xcite . a common method consists of comparing model probabilities for grammatical and ungrammatical sentences (e.g., "the key to the cabinets is|*are on the table"). these experiments often concentrate on syntactic phenomena that are instantiated with surface strings that provide unequivocal information about the elements that enter the dependency, e.g. agreement morphology @xcite ). yet, we know less about the ability of lms to coordinate syntactic and semantic information during the resolution of dependencies whose elements are not overtly signaled by morphosyntactic cues in the input. such is the case of control structures like those in (1). these superficially simple constructions underlie complex lexically-guided antecedent retrieval processes, and they represent an interesting candidate to study dependency resolution at the syntax-semantics interface. at the infinitive verb ser in (1), it is crucial to interpret its implicit subject. in other words, who is tidy? the term control reflects the idea that the interpretation of the implicit subject is controlled by, or is determined by, another referent @xcite . this type of control dependencies entail interpreting an anaphoric relation between the implicit subject of the embedded clause and one of the nps in the main clause (known as controller or antecedent). crucially, this interpretive relation is guided by specific lexicosemantic properties of the main clause predicates @xcite . in (1a), the correct antecedent is juan (the main clause subject) because promise has subject control properties. in (1b), the correct antedecent is maría (the main clause object), because order has object control properties. retrieving the correct antecedent is essential in order to build an accurate representation of the message and to compute the agreement dependency that is established between the controller and the adjective tidy. consequently, the resolution of these dependencies entails coordinating information about the lexico-semantic properties of control predicates, co-reference, and agreement morphology, and provides a great context for probing lms' grammatical abilities beyond morphosyntax. in this work, we take advantage of the rich agreement properties of two romance languages (spanish and galician) in order to examine humans' and language models' ability to correctly identify control dependencies. to do so, we have carefully created an experimental design via the manipulation of the gender of the nps (feminine/masculine), the type of control verb (subject/object control), and the gender of the embedded adjective. this design will allow us to test whether humans and lms identify or produce agreement violations at the adjective, which is used as a proxy for the accuracy of antecedent retrieval processes. furthermore, this design will allow us to test for the presence of interference effects of non-controlling nps (referred to as distractors) when they match or mismatch in gender with the embedded adjective. we created several datasets that have been used for a human acceptability judgement task (experiment 1), a lm acceptability task (experiment 2), and a lm prediction task (experiment 3). for experiments 2 and 3, we tested the most prominent monolingual and multilingual masked lms based on transformers for spanish, and provide additional translated datasets and results from the same computational experiments carried out with galician lms in order to confirm the cross-linguistic robustness of our findings. our results show that while humans correctly identify the acceptability of the strings regardless of the configuration of the nps, language models often fail to correctly identify the relevant antecedent in subject control dependencies, showing their reliance on linear relations rather than linguistic information, something which is observed in their below-chance accuracy for discontinuous dependencies. the main contributions of our paper are: (i) the release of wide-covering and highly controlled datasets to evaluate control structures in spanish and galician, (ii) a psycholinguistic evaluation of humans' performance, a computational evaluation of monolingual and multilingual lms' performance, and a careful comparison between humans and lms; (iii) a demonstration of the limitations of lms to capture grammatical information thanks to the adversarial example of control constructions. ## related work targeted evaluation of lms: targeted evaluations of lms focusing on different syntactic phenomena have found evidence suggesting that these models may generalize syntactic information from raw text @xcite @xcite . in this regard, the subject-verb (number) agreement task is one of the most used adversarial examples for these evaluations, although @xcite introduced further experiments dealing with other syntactic phenomena in english (such as negative polarity items or reflexive anaphora). these types of datasets have been extended and adapted to different languages @xcite @xcite and incorporated into online evaluation platforms @xcite . in these experiments, the overall performance of large pre-trained lms is found to be comparable to that of human subjects @xcite @xcite , except for long-distance dependencies with distracting nouns between the elements of the target dependencies @xcite , where lms often fail to identify the target dependency relation. besides, recent work found that lms' perplexity is not always correlated to their syntactic generalization abilities @xcite , nor with human reading times @xcite . other complex structures that seem difficult to interpret by lms are nested constructions, which may require recursive abilities to be solved. recent studies on italian and english have found that, although both recurrent and transformer neural networks achieve near-perfect performance on short embedded dependencies, their performance drops to below-chance levels on slightly longer dependencies, unlike humans @xcite . lampinen (2022), however, questions these comparisons between humans and lms, as the former receive guidance before the experiments, while lms are evaluated on zero-shot scenarios, and their performance improves with few-shot prompts. despite the fact that most of the work evaluating the linguistic capabilities of lms has been carried out in english, there exist some experiments that have focused on spanish and galician lms showing that the lms tested in this work perform very well in the context of different linguistic dependencies, including simple and complex agreement dependencies with distractors. recent studies in both spanish and galician show that models' performance for these dependencies (which rely on morphosyntactic information) are similar to those in english (with expected variations across models). for instance, @xcite found that monolingual and multilingual models achieve even better performance in agreement resolution in span-ish than bert in english. for galician, several experiments showed that the monolingual bert models can generalize morphosyntactic agreement (number and gender) on complex subject-verb and subject-predicative adjective dependencies @xcite , and that this information is learned relatively early in the training process (de dios-flores and @xcite . the syntactic strengths observed in these models establish a baseline performance against which we can examine the results obtained for control dependencies. concerning control constructions, studies exploring lms' abilities to solve these complex relations are very scarce. in a recent paper, @xcite trained supervised models that take advantage of contextualized representations extracted from bert, and evaluate them at capturing control verb nesting and verb raising in dutch. the results suggest that transformer lms do not adequately capture the target relations, although finetuning the pre-trained models in one-shot learning scenarios improves the performance of the probes. more similar to our study, an initial approximation by @xcite evaluated gpt-2 on object and subject control verbs, using number agreement with an embedded reflexive pronoun to track dependency resolution. their findings suggest generative lms are unable to differentiate between these two types of constructions. however, their manipulations were very limited in scope, as they only used 5 noun phrases, and 3 control verbs. pycholinguistics and control dependencies: even though control constructions have been at the center of linguistic theorizing over the past decades, their theoretical interest has not translated into an equivalent amount of experimental research in the psycholinguistics literature. the key question, though, is whether (and how) control information is used in parsing. some early works have argued that control information was not used during initial parsing stages due to its lexico-semantic nature (e.g., @xcite . nonetheless, these works barely looked at the contrast between lexically induced subject and object control relations. in this regard, more recent eye-tracking investigations have produced results that could be interpreted as evidence that lexical control information is used from early parsing stages (e.g. de dios-flores, 2021; @xcite while they also suggest that object control dependencies seem to be solved faster due to their linear proximity. yet, to our knowledge, no previous work provided acceptability judgements contrasting subject and object control dependencies with distractors, which is a highly informative measurement to establish the grammatical and psycholinguistic status of such constructions. ## the present work the present work takes control dependencies as an adversarial case to test lms' ability to generalize grammatical information at the syntax-semantics interface (experiments 2 and 3). given the complexity of these constructions, and the lack of psycholinguistic evidence, we go one step further and start by evaluating humans' grammaticality perception (experiment 1), not only to obtain a grammatical verification of the acceptability status of such innovative experimental materials and to be able to directly compare humans' and lms' performance, but also to contribute to the scarce psycholinguistic evidence on the processing of control. the datasets, code, and results from all the experiments are freely available. foot_0 ## experiment 1: human acceptability the primary goal of this acceptability task is to determine whether native speakers of spanish are able to detect agreement violations that do not conform with the control properties of main predicates. this is, to our knowledge, the first experimental investigation on control of its kind, and we believe it is essential to corroborate native speakers' offline sensitivity to the different control manipulations that will be then put to the test with artificial lms. it will be of particular importance to elucidate whether comprehenders are able to correctly distinguish the acceptability of the strings regardless of the type of control (subject or object) and the presence of a gender matching or mismatching distractor. ## experiment 2: lm acceptability this experiment aims at observing whether the probabilities of the language models are similar 207 to those of humans. that is, whether lms assign lower surprisal to grammatical than to ungrammatical sentences regardless of the presence of a matching or mismatching distractor. for this purpose, we use the exact same dataset as in experiment 1. foot_1 ## experiment 3: lm masked prediction this experiment aims at further exploring the behavior of lms using the masked prediction task. in contrast with experiment 2, where we compute the surprisal for the same adjective in a given (grammatical or ungrammatical) sentence, our objective here is to test whether lms predict grammatically compatible adjectives in subject and object control sentences regardless of the presence of a matching or mismatching distractor. in experiment 2 the adjective's gender was kept constant across experimental conditions, and hence, we could not assess lms' preferences for the masculine or feminine version. by contrast, here we test if lms predict grammatically compatible adjectives in subject and object control sentences by directly comparing the probabilities of a given adjective in its masculine or feminine form, something which provides us with more comprehensive information in this respect. furthermore, evaluating model accuracy rather than surprisal values will also allow us to assess and compare the performance across models.