A Discerning Several Thousand Judgments: GPT -3 Rates the Article + Adjective + Numeral + Noun Construction (2023)

Abstract: Knowledge of syntax includes knowledge of rare, idiosyncratic constructions. LLMs must overcome frequency biases in order to master such constructions. In this study, I prompt GPT-3 to give acceptability judgments on the English-language Article + Adjective + Numeral + Noun construction (e.g., “a lovely five days”). I validate the prompt using the CoLA corpus of acceptability judgments and then zero in on the AANN construction. I compare GPT- 3’s judgments to crowdsourced human judgments on a subset of sentences. GPT-3’s judgments are broadly similar to human judgments and generally align with proposed constraints in the literature but, in some cases, GPT-3’s judgments and human judgments diverge from the literature and from each other.

## introduction consider the english article + adjective + numeral + noun (aann) construction: " a beautiful 228 pages @xcite " or "the president has had a terrible five weeks [coca]". usually cardinal numbers precede the adjective ("five terrible weeks"), but here the adjective precedes the numeral. more strangely, the normally singular article "a" in this case is followed by a plural noun phrase. 1 an eclectic dozen or so papers have been written on the construction, many focused on elucidating relevant semantic and syntactic constraints @xcite @xcite @xcite @xcite . the presence of the modifier is crucial: "a 228 pages" is unacceptable. the type of modifier is also crucial: "a pink 228 pages" is odd because color words are "stubbornly distributive" @xcite and thus cannot refer to a set of items as a whole: the nominal phrase needs to function as a unit @xcite . these idiosyncratic constraints are typical of constructions @xcite . prior work on the aann construction has focused on characterizing the semantic and syntactic constraints on the construction and proposing analyses in various frameworks. for instance, @xcite focuses on how the construction coerces the phrase into a singular noun phrase; @xcite proposes treating it as akin to a partitive, and dalrymple and @xcite give an lfg analysis. the same properties that make aann interesting from the perspective of human language use-its low frequency but high sensitivity to constraints-also make it interesting from the perspective of llms and what they learn about linguistic structure. indeed, much work on llm syntactic competence has centered on ubiquitous abstract features of grammar like subject-verb number agreement (e.g., @xcite , part of speech @xcite , and syntactic dependencies (e.g., @xcite . that said, there has also been a recent spate of research on construction-grammar-inspired approaches in nlp, including studies showing that llms have access to construction information @xcite @xcite and capture verb argument construction biases @xcite as well as fine-grained lexical semantic information @xcite . moreover, sentences with similar constructions cluster in embedding space @xcite . these works are valuable because better understanding how llms handle constructions could help us better understand what llms learn about linguistic structure @xcite and could also inform us as to what can be learned about language from primary data @xcite . in our case, for an llm to get the aann construction right, a number of statistical regularities must be eschewed: "a" cannot be treated as a singular marker since the noun is plural, the normal ordering of the number and adjective must be reversed, and normal verb number agreement rules must in some cases be suspended. understanding whether these heuristics (which work well for the vast majority of text) can be overcome can guide us towards future work understanding how they are overcome. here, i ask what gpt-3 text-davinci-002 (now often classed as an instance of gpt-3.5) learns about the aann construction by testing its sensitivity to several constraints proposed in the literature. in doing so, i treat the llm as a linguistic test subject @xcite @xcite @xcite . i use a custom prompt to elicit quantitative grammaticality judgments @xcite ) from gpt-3, and show that the prompt performs well on cola, a data set of binary acceptability judgments on a carefully constructed 10,657 english sentences @xcite . i then unleash it on the aann construction and compare gpt-3 to human ratings. ## methods attaining acceptability judgments from language models is not straightforward. simply comparing sentence probabilities is difficult because they are dependent on the individual lexical items, as well as sentence length @xcite . i rely on the prompting paradigm to elicit acceptability figure 2 : the prompt used for attaining grammaticality judgments. the target sentence is inserted, and then gpt-3 is asked to generate one token (overwhelmingly likely to be "good" or "bad"). that token's probability is taken as a rating. to validate the measure, i tested the final prompt on the cola dev set. it attained accuracy of 84%, with a matthew's correlation coefficient of 0.63. this is worse than, but in the ballpark of, human inter-annotator agreement on cola which is 86% and .697, respectively @xcite . it is also comparable to top performers on the glue leaderboard for the cola sub-task. from these templates and candidate words, i generate semantically plausible sentences (meaning that, throughout the experiments, i only use humanappropriate sentence templates with human nouns and object-appropriate ones with object nouns). depending on the specific question in each experiment, i run controlled subsets of these sentences (or their degenerate variants as in experiment 1) through gpt-3 to obtain acceptability judgments, getting the probability of "good" or "bad" as the next word in the continuation. i also use amazon's mechanical turk to obtain human judgments on a subset of the test sentences. i asked raters to rate sentences on a rating bar scale from 1-10. for experiments 1 and 3, each rater rated 3 critical sentences. they rated 18 critical sentences in experiment 2 since there were many more conditions to test in that experiment. to guard against raters becoming inured to the construction, half or more items were fillers not involving aann. to maintain similar calibration between humans and gpt-3, these fillers always included all example judgments used in the gpt-3 prompt. i excluded participants who did the survey more than once, who did not rate the good filler items at least 1 point higher than the bad, or who did not have a us ip address. i obtained annotations for only a sample of sentences rated by gpt-3. when applicable, analyses focus on the union of sentences rated by both humans and gpt-3. ## exp. 1: aann fundamentals first, i tested the basics of the aann construction as laid out in the literature by having gpt-3 and human raters rate the aann construction (e.g., "a beautiful five days") vs. the default ("five beauti-ful days") vs. a select four degenerate conditions: one with the order of the numeral and adjective switched, one with no modifier, one with a singular noun, and one with no article "a". to generate examples in each condition, i crossed 3 temporal nouns (days, weeks, months), with a low numeral (three), one of 14 appropriate adjectives, and one of 3 templates. for the resulting sentences, i attained human ratings from mturk (after exclusions, 126 raters rating 3 sentences each, for 378 total ratings) and gpt-3 and focus on that subset for analysis. figure 1 shows results for this experiment. although gpt-3 uses a wider range of the scale, both rate the aann construction as just as good as the default and give lower ratings to the 4 degenerate versions. humans rate the 4 degenerate constructions as about equally bad (all between .46 and .53), whereas gpt-3 rates the versions with swapped adjective/numeral order and a missing article as significantly better than the versions with a missing plural and a missing modifier. running a mixed effect regression @xcite comparing the aann construction to the "default" construction and the degenerate alternatives (treating the default construction as the baseline, with random intercepts for adjective class, adjective, and template; and, for humans, for rater), both gpt-3 and humans show no significant difference in rating between aann and default (both @xmath0.05), but do show a significant difference between aann and all 4 degenerate conditions (all @xmath1.0001). see appendix c for regression details. overall, i conclude that humans and gpt-3 "get" the aann construction, even though they differ in the relative ratings of the bad variants. ## exp. 2: adjectives and nouns in this experiment, i focus on only the aann construction and parametrically vary the kinds of adjectives (quantitative, ambiguously quantitative/qualitative, qualitative, human-referring, color adjective, stubbornly distributive; see i test whether gpt-3 and human raters agree with the attested claims that (a) more measure-like nouns (e.g., temporal nouns, distance nouns, and unit-like nouns) are more acceptable in aann, (b) qualitative adjectives are acceptable in only some aann cases, and (c) stubbornly distributive adjectives (including color words) are not acceptable in the aann construction. to assess significance, i predicted acceptability separately for humans and gpt-3 in a mixed effect regression, using adjective class, noun class, and their interaction as predictors and with random effects for adjective, noun, numeral, and template. i treated qualitative adjectives with temporal nouns as the baseline. for both humans and gpt-3, there are significant differences in how nouns interact with adjectives (figure 3 ). for temporal, distance, and unit-like nouns, all adjective types show high acceptability (although the qualitative adjectives are rated lowest). for art and object nouns, qualitative nouns score significantly worse (@xmath2.01 for both humans and gpt-3) than ambiguous or quantitative nouns (an observation consistent with the literature). as predicted, colors and other stubbornly distributive adjectives (which can be tested only for humans and objects) show the lowest acceptability (significantly lower, @xmath3.01, compared to qualitative adjectives, for both human and gpt-3 ratings). see appendix d for regression details. ## exp. 3: adjective order it is claimed that, in aann, qualitative adjectives must appear before quantitative ones in order to be acceptable @xcite : "the family spent a beautiful mere five days in london." is preferred over "the family spent a mere beautiful five days in london.". to compare whether there is an effect of adjective ordering (putting the qualitative adjective before the quantitative one or vice versa), i ran an experiment crossing 3 templates; 5 adjectives (astonishing, impressive, beautiful, hideous, ugly); the noun "days"; and the numeral "three" or "five". i ran each sentence in two conditions (quantitative adjective first or qualitative adjective first). for instance, i compared: "the family spent a beautiful mere five days in london." to "the family spent a mere beautiful five days in london." this left 60 sentences total, rated by 99 raters (each sentence rated between 18 and 36 times; 1,782 ratings total). gpt-3 significantly prefers the order dispreferred in the literature (quantitative first, as in "a mere beautiful five days"; β = .04, @xmath4.01). humans (@xmath5, under the same analysis (but with a random intercept for rater): β = -.513, @xmath6.05. thus, the attested claim does not replicate. but the ratings for these sentences are relatively low overall, so we should remain open to the possibility that there are better examples of the double-adjective constructions than the ones tested. see appendix e for regression details. ## exp. 4: verb agreement the aann construction also challenges number agreement. aann subjects sometimes take singular verbs (when the noun phrase would be singular anyway, as in "a mere fifty cents for a cup of coffee sounds/*sound reasonable to me!"); sometimes plural ("a delicious four courses *was/were served in the main dining room."), and sometimes either ("a healthy two runs weekly was/were prescribed by the doctor." see @xcite . i tested agreement by comparing phrases which differed only in the verb number (e.g., comparing "a beautiful five days is..." vs. "a beautiful five days are...."). sampling a subset of noun classes (art, distance, human nouns, objects, unitlike nouns, and temporal nouns), i generated a total of 280 sentences (each appearing with a singular or plural verb for a total of 560 sentences) and attained judgments from gpt-3. as shown in figure 5 , these results replicated in detail several attested judgments in the literature: art nouns, human nouns, unit-like nouns, and object nouns all prefer the plural for the aann construction (art and humans almost categorically so). distance nouns and temporal nouns prefer the singular (although temporal nouns can also take the plural). see appendix f for details. ## conclusion this work shows that gpt-3 can recognize and use the form of the aann construction in a relatively (but not perfectly) human-like way, matching judgments across a variety of conditions, which is not the same thing as showing that it understands the meaning or function of the construction @xcite . in future work, we should study not just the form of the construction but its construal @xcite . this may be particularly relevant since @xcite , which studies the "xer the yer" construction (e.g., "the better the criticism, the better the science" [coca]), show that llms can recognize the construction but fail at tests of understanding its meaning. that said, gpt-3's performance on the aann construction demands a significant amount of constructional knowledge and involves overriding major widespread "rules" of grammar (e.g., that the article a signals a singular noun). future work exploring how llms override those heuristics, perhaps using causal intervention techniques @xcite could illuminate their syntactic processing mechanisms.