Do LLM s learn a true syntactic universal? (2024)

Abstract: Do large multilingual language models learn language universals? We consider a candidate universal much-discussed in the linguistics literature, the Final-over-Final Condition (Sheehan et al., 2017b). This Condition is syntactic in the sense that it can only be stated by reference to abstract sentence properties such as nested phrases and head direction. A study of typologically diverse “mixed head direction” languages confirms that the Condition holds in corpora. But in a targeted syntactic evaluation, Gemini Pro only seems to respect the Condition in German, Russian, Hungarian and Serbian. These relatively high-resource languages contrast with Basque, where Gemini Pro does not seem to have learned the Condition at all. This result suggests that modern language models may need additional sources of bias in order to become truly human-like, within a developmentally-realistic budget of training data.

## introduction the question of whether large language models (llms) display human-level competence has provoked lively discussion. some commentators' minds are made up: "llms have already demonstrated that human-like grammatical language can be acquired without the need for a built-in grammar" @xcite . others are more guarded: "nearly all studies have reported that [deep neural networks'] behavior deviated from the idealized syntactic competence that a linguist might postulate" @xcite . this controversial question is important both for continued progress in ai systems, as well as for debates over inborn biases that might be necessary to achieve human-level language within a developmentally-plausible number of training examples @xcite . typological generalizations are a key battleground in such debates (see e.g. van der hulst, 2023, chapter 7). where such generalizations qualify as language universals, they could potentially underwrite a strong nativist argument for inborn biases (e.g. @xcite . but this could only happen if competing explanations, based upon general-purpose learning rules, were ruled out. can llms rebut nativism by accounting for true language universals without universal grammar? that is the question this paper takes up. our answer is negative. we find that modern llms do learn a candidate syntactic universal if given superhuman amounts of text. but in the basque language, where data sizes are likely closer to human level, these same models do not learn the universal. this outcome leaves nativism still standing, because the attained level of performance does not meet the criteria of universality or developmental plausibility. section 2 begins by introducing a candidate language universal in the domain of syntactic structure called the final-over-final condition @xcite @xcite . section 3 reports a corpus study of this universal in six "mixed head direction" languages where it conceivably could be violated. the corpus study confirms that the universal indeed holds in these languages, one of which is basque. section 4 goes on to a targeted syntactic evaluation of palm @xcite and gemini pro @xcite . gemini is, at the time of writing of this paper, the most advanced of google's large language models. it surpassed gpt-4 on the lm-sys.org leaderboard on january 26th 2024. palm is an earlier model. both learned the condition in all languages except basque. section 5 discusses this gap between human and model performance, noting that neither data size nor parameter count matter, beyond some threshold point. we contribute the first large-scale corpus study of this universal as well as a novel evaluation that speaks to human language in general. ## the final-over-final condition the final-over-final condition (henceforth: fofc) is "either a very strong tendency among the languages of the world or actually a language universal" @xcite , page 2). here we do not take fofc as a given fact, but as a hypothesis in need of testing. fofc depends on the notion of head direction. larson's 2010 textbook introduces this notion with two examples, reproduced below from page 351. t.-top deru depart daroo. may b. taroo-wa t.-top hanako-o h.-acc tazuneru visit daroo. may c. taroo-wa t.-top ringo-o apple-acc hanako-ni h.-dat ageru give daroo. may english, as exemplified in (1), contrasts with japanese in example (2) along the dimension of head direction. the three sub-examples for each language show that there exists a variable-length unit -the verb phrase (vp) -that may contain zero, one or two non-head phrases called (syntactic) complements. in english the verb precedes its complements, whereas in japanese it follows them. this situation is summarized by saying english has head-initial vps, whereas in japanese vps are headfinal. various criteria have been proposed to differentiate heads from non-heads in a phrase. heads typically determine the part of speech or morphological form of their dependent (see @xcite . on the basis of these examples, larson goes on to remark upon another grammatical category, t. his observation rests on a common categorization of the english modal "may" and the japanese future-possibility word "daroo" as t, a category that includes tense and other auxiliary verbs. analogous to vp, tense phrases tp are head-initial in english but head-final in japanese. this basic idea of viewing phrasal head direction as a language-specific parameter setting is introduced on pages 73-75 of @xcite . @xcite and @xcite 3) survey subsequent developments of this basic idea. the fofc, then, is a very general condition on headed phrase structure trees of depth two. at this depth there are four possible configurations, shown in figure 1 . two of these configurations, 1a and 1b are "harmonic" in the sense that both superphrase and subphrase follow the same head direction. these structures fit well the facts of english and japanese, respectively. the other configurations are "disharmonic" such that the head direction of the superphrase is different from that of the subphrase. these phrase structures are useful for analyzing languages like finnish, where a vp may be head-final in the context of a focused complementizer @xcite . although there may be a tendency across time for languages to change in the direction of more consistent, harmonic word orders (see e.g. @xcite , mixed word order is by no means rare. the fofc claim is that one of the two possible disharmonic configurations is universally banned in human language. in figure 1 the outlawed structure is the grayed-out cell, 1d. this strong claim foot_0 is tempered somewhat by the requirement that β be an extended projection of α in the sense of grimshaw (2005, chapter 1). the basic idea, which has enjoyed wide currency since the early 1990s, is to group phrases that are headed by function words together with their main content word, typically a noun or verb. this restriction to extended projections tightens the fofc so that it applies solely within a given domain i.e. nominal or verbal. @xcite grapples further with the sorts of derived structures to which the fofc applies. perhaps the strongest evidence for the fofc comes from the case where α = v and β = t. here, the fofc outlaws a head-final tp over a head-initial vp. as @xcite discuss, evidence from germanic languages seems to bear this out. these languages attest mixed headedness; in fact virtually every possible combination of auxiliaries, verbs and their objects is attested. what is not attested in the literature on comparative germanic is a tensed auxiliary verb following a verb-object combination. this would be analyzed as a head-final tp with a head-initial vp daughter, exactly what the fofc rules out. section 3 considers this case in quantitative detail. a. ## corpus study the corpus study asks whether the fofc is true or not; section 4 follows up by asking whether a modern neural language model can distinguish fofcrespecting from fofc-flouting sentences. this first question is addressed by asking: does a contingency table for nested combinations of aux and v look the way the fofc predicts it should look? referring to figure 1 , one asks: is the number in cell 1d lower than expected? the corpus study thus seeks "negative" evidence for the grayed out configuration @xcite . this methodology closely follows @xcite and @xcite in applying the χ 2 test, a standard procedure in corpus analysis (for a introduction, see brezina 2018, §4.3 or stefanowitsch 2020, §6.3). the present study is more detailed than previous fofc surveys, such as bazalgette (2012, cited on page 91 of @xcite ) insofar as it is based upon counts of individual sentences rather than descriptive reports about entire languages. as section 2 emphasized, the key claim of the fofc has to do with disharmonic word orders. it does not rule out any structures in consistently head-initial languages like english and indonesian, or consistently head-final languages like japanese and turkish. for this reason, the corpus study focuses on languages that have been described (however superficially) as displaying mixed-headedness. the central comparison is between the permitted initial-over-final order (1c) and the outlawed finalover-initial order (1d). counts in the other cells are only needed to establish the expected value in cell 1d. we consider languages collected in the c4/multilingual dataset @xcite , except for german. in german for idiosyncratic reasons we analyzed a dump from wikipedia. the dump happened near the end of 2020 and was processed using wikiextractor @xcite . all languages were analyzed up through the stage of dependency parsing with the stanza toolkit @xcite . in serbian, we applied a version of stanza that is optimized for south slavic languages @xcite . to make parsing more manageable, we excluded outputs from the stanza sentence-breaker that exceeded 40 words in length. there is debate among dependency grammarians over the status of function words. the well-known universal dependencies (ud) scheme views function words as dependents of content words, on the assumption that content words will prove more helpful in semantic analysis @xcite . this decision conflicts with the view of tp as an extended projection of v -a background assumption in the argument for the fofc from comparative germanic. to reverse it, we convert the ud dependency graphs obtained from freelyavailable stanza models into surface ud graphs @xcite to view the converted sud dependency graphs as x-bar trees, as envisaged in the fofc. the ud/sud part of speech tag aux corresponds best to category t as used in generative grammar. the results in each case take the form of a contingency table. for the sake of space, we discuss only hungarian, basque, german and serbian, summarizing results from the other languages in the counts in parser error inevitably plays a role in a largescale study such as this. in an effort to reduce parser error we performed a small study of hun-garian sentences of 12 words or less. this yielded the same pattern, with 34 examples in the fofcviolating cell. professor tibor laczkó, an expert on hungarian syntax who is also a native speaker, examined these examples and determined that none of them are true fofc violations. the most wellattested error types were: we applied stanza to all the basque text in c4/multilingual; these results are shown in the paucity of verb-initial examples in embedded clause renders the χ 2 test inapplicable. this paucity is consistent with the usual characterization of german as underlyingly verb-final (for a textbook treatment see e.g. @xcite . professor vera lee-schoenfeld, an expert on german syntax and a native speaker, manually examined the thirteen examples in the fofc-outlawed configuration and determined that none were true fofc violations. the three main types of error were: 1. mistagging a prenominal adjective as v 2. mistagging a participle at the end of a reduced relative clause, or a finite verb at the end of a full relative clause as v 3. mistagging a noun or part of a compound noun as v one of the more notable verb-initial examples came from a sacred song that is cited in wikipedia; the song itself is dated 1529 but it may have been written earlier. this underlines the point that modern german does not attest head-initial vps in embedded clause. for this reason, and given the lack of controversy surrounding the fofc in german (see last paragraph of section 2) we did not proceed with further parsing or analysis of this language. two additional constraints were applied to the slavic languages, russian and serbian, in order to rule out vp fronting as detailed in appendix c. in serbian the same pattern manifests itself, shown below in the fofc could have been disconfirmed by finding roughly equal numbers of structures like 1c and 1d, but that was not what we observed. there are far fewer instances of the configuration that the fofc bans, 1d, than would be expected by chance alone. when small samples of these spurious matches are examined by experts, none of them turn out to be actual violations. from this we tentatively conclude that the fofc holds, and proceed to ask whether modern neural language models can learn this constraint. ## evaluating llms on the fofc the targeted syntactic analysis reported in this section asks whether a language model can what a human can do. what humans can do is illustrated by the combination of german judgments in 3 on page 6. this pattern, cited by @xcite , shows the influence of the fofc. consider first the preliminary example (3a). this example shows that verb-object order is possible in colloquial german; it is possible in a vp-topicalized main clause context. in this kind of derived structure, the aux is not an extended projection of vp. for this reason, we sought to exclude such structures in the corpus studies of german, serbian and russian. however in embedded clauses where verb second movement does not apply, subject-verb-object-aux order leads to unacceptability as shown in example (3b). this is the fofc effect. the example can be saved (3c) by extraposing the object "mit ihr." the fact that (3b) is unacceptable whereas (3c) is accepted by competent german speakers may be understood as reflecting the influence of the fofc. the targeted syntactic analysis evaluates whether gemini pro and palm show an influence of the fofc along the lines of the human judgments in figure 2 by scoring minimal pairs (see e.g. @xcite ; marvin (3) a. [ gesprochen spoke [ mit to ihr her ] ] vp hat aux has er he nicht not mehr more 'as for speaking with her, he no longer did that' b. * dass that er he nicht not mehr more [ gesprochen spoken mit with ihr her ] hat has c. dass that er he nicht not mehr more [ gesprochen spoken hat has ] mit with ihr her figure 2 : effect of the fofc on german acceptability @xcite . s v o aux word order in (3b) leads to unacceptability (marked with asterisk) and must be repaired, for instance by extraposing the object outside of vp to the end of the sentence as shown in (3c). and @xcite @xcite . these minimal pairs are created by transforming attested examples from c4/multilingual as shown in figure 3 . the transformations reorder dependency subtrees so as to create strings that exemplify disharmonic word orders. such reordering preserves the length, lexical content and (parser-inferred) grammatical relations of the example. indeed its impact on acceptability need not be disastrous. for instance, changing from verb-initial to verb-final order (i.e. 1a to 1c in the sense of figures 1 and 3 ) transforms an example from the english web treebank, "google finally had an analyst day" into "google finally an analyst day had." in the estimation of one author, this transformed sentence sounds a bit archaic but is not total word salad. we measure the level of respect for the fofc by subtracting the log-probability of 1c-type examples (fofc-respecting) minus the log-probability of 1d-type (fofc-disrespecting) examples. the variability in bell-shaped distributions shown in this figure 4 could reflect errors parsing the untransformed example, information structure factors like focus, or register variation of an essentially sociolinguistic nature. by examining a large sample of matched examples, the analysis seeks to average out this noise and measure the degree of respect the fofc, per se. ## results and discussion the average penalty values in figure 4 can be interpreted as acceptability differences. in hungarian, russian, serbian and german the distribution is centered above zero for both models. this suggests that gemini pro and palm have learned the fofc in those languages. however basque is clearly different. in basque, gemini pro assigns rather more probability to fofc-violating sentences than to fofc-compliant ones. palm is only slightly better in terms of mean value for basque, but most of the distribution's penalties are still in the negative region. the corpus study from section 3 suggest that the fofc does hold in basque, at least in c4/multilingual. these findings imply that gemini pro and palm have not reliably learned the final-over-final condition in basque. there are at least two possible explanations for this phenomenon: model size (i.e. parameter count) and data size (i.e. example count). ## conclusion in six key languages where it could have been disconfirmed, corpus analysis instead lends support to the fofc, a candidate universal. a targeted syntactic analysis with gemini pro and palm, modern multilingual language models, indicates that the fofc was only learned in cases where the model is exposed to developmentally-implausible amounts of training data. in basque, where the training data size was probably closer to developmental plausibility, gemini pro and palm fail to reliably learn the constraint. so while predicting successor words can ultimately serve to learn a candidate universal, at human-scale the stimulus seems to be too poor. llms have learned the fofc in some subset of languages, but they have not learned that it is a universal constraint, applicable to all human languages. our ablation study with palm suggests that increasing model size helps, but not enough to change the pattern. to achieve human-level performance with language universals, it may be necessary to build in some form of inductive bias.