Investigating representations of verb bias in neural language models (2020)

Abstract: Languages typically provide more than one grammatical construction to express certain types of messages. A speaker’s choice of construction is known to depend on multiple factors, including the choice of main verb – a phenomenon known as verb bias. Here we introduce DAIS, a large benchmark dataset containing 50K human judgments for 5K distinct sentence pairs in the English dative alternation. This dataset includes 200 unique verbs and systematically varies the definiteness and length of arguments. We use this dataset, as well as an existing corpus of naturally occurring data, to evaluate how well recent neural language models capture human preferences. Results show that larger models perform better than smaller models, and transformer architectures (e.g. GPT-2) tend to out-perform recurrent architectures (e.g. LSTMs) even under comparable parameter and training settings. Additional analyses of internal feature representations suggest that transformers may better integrate specific lexical information with grammatical constructions.

## introduction when we use language, we are often faced with a choice between several possible ways of expressing the same message. for example, in english, to express an event of intended or actual transfer between two animate entities, one option is the double-object (do) construction, in which two noun phrases follow the verb. alternatively, the same content can be expressed using the prepositional dative (po) construction. ## related work several recent studies have investigated how neural language models represent the dative alternation. @xcite constructed a corpus of verbs in common alternations, including the dative, and showed that a degree of information about acceptability is decodable directly from embeddings of the verb. however, acceptability was not based on empirical data and verb bias was treated as a binary variable, preventing an analysis of gradient effects. @xcite found that do constructions are separable from non-do constructions in high-dimensional sentence embeddings (including bert), but did not investigate verb bias. @xcite confirmed that recurrent neural networks (rnns) show human-like sensitivity to several other important aspects of gradience in dative alternations, including the length and definiteness of arguments. however, they included only 16 verbs, all considered "alternating." additionally, in these studies, a limited range of neural models were considered, leaving it unclear exactly how predictions may depend on architectural choices, model size, and training regime. ## the dais dataset the dais dataset contains 50,136 human preference judgments for 5,000 sentence pairs, constructed as follows. first, to obtain a large and heterogeneous set of verbs, we collected the 100 most frequent verbs influentially classified by @xcite as alternating (i.e. acceptably appearing in both po and do constructions), as well as the 100 most frequent verbs classified as "non-alternating" (appearing only in the po construction). this set contains most of the verbs examined in prior corpus-based analyses (see sec. 5). for each verb, we generated do and po sentences across 5 different conditions, manipulating the length and definiteness of the recipient argument (see ex. 3). finally, to obtain a range of distinct items in each condition, we created 5 plausible theme arguments for each verb, including 2 definite and 3 indefinite, for a total of 5,000 alternation pairs. we collected judgments from 1011 participants on amazon mechanical turk. each participant was shown 50 dative alternation pairs (do vs. po) using unique verbs, balanced across the possible recipient and theme conditions. on each trial, participants used a continuous slider to indicate the strength of their preference for the do or the po, with the midpoint used to indicate they were "about the same" (see appendix a for details) 1 . ## analysis on natural corpus while there are many advantages of human judgment datasets like dais -including the ability to include a wider range of infrequently observed verbs and to control for potential confoundsthere are also distinct advantages of corpus data. we thus conducted a further evaluation using do and po utterances extracted from the switchboard corpus by @xcite . instead of testing how well each model was able to predict continuous judgments, we now ask how well each model is able to categorically predict whether the do vs. po was naturally produced by a corpus speaker. for each do or po utterance in the corpus, we used the extracted verb, theme, and recipient to generate the alternating sentence, pairing the attested example with its hypothetical alternation. subjects were chosen from a list of names, as in the dais dataset. after removing incoherent sentences, we obtained 2,206 pairs of sentences in total. for each model, we calculated the likelihood ratio of the po vs. do construction for all corpus examples. following @xcite , we then constructed a classifier to predict which construction was actually produced by fitting a decision threshold for likelihood ratios. the accuracy achieved by each model is shown in ## conclusions in natural languages, speakers routinely select one alternative over others to express their intended message. these choices are sensitive to many interacting factors, including the choice of the main verb and the length and definiteness of arguments. our new dataset, dais, not only offers a higherresolution window into the richness of human preferences, it also provides a newly powerful benchmark for evaluating and understanding the corresponding sensitivity of language models. we found that transformer architectures corresponded especially well with human verb bias judgments. further work is needed to more precisely determine the source of the architectural differences we observed. one possibility is that the transformer's self-attention mechanism and layer-wise organization improves its ability to represent lexicallyspecific structures. however, it is also possible that differences are attributable to training data. another line of future research is to compare the incremental predictions of neural models to finergrained eye-tracking evidence during sentence processing of double-object sentences (e.g. @xcite . as neural language models become more complex, subtler phenomena like verb bias may yield new insights into how lexical and grammatical representations are jointly learned and successfully integrated for language understanding. @xcite . we also included random intercepts and slopes for each human participant. to evaluate the effect of information structure in section 4.1, our mixed-effects model included fixed effects for recipient length, recipient definiteness, and theme definiteness. we included random intercepts and effects of recipient length and definiteness for each participant and verb to control for clustered variance at these levels. see fig. s1 for the full pattern of results, split by "alternating" and "non-alternating" verbs. complete regression results are shown in tables s3 and s4 .