Abstraction not Memory: BERT and the E nglish Article System (2022)

Abstract: Article prediction is a task that has long defied accurate linguistic description. As such, this task is ideally suited to evaluate models on their ability to emulate native-speaker intuition. To this end, we compare the performance of native English speakers and pre-trained models on the task of article prediction set up as a three way choice (a/an, the, zero). Our experiments with BERT show that BERT outperforms humans on this task across all articles. In particular, BERT is far superior to humans at detecting the zero article, possibly because we insert them using rules that the deep neural model can easily pick up. More interestingly, we find that BERT tends to agree more with annotators than with the corpus when inter-annotator agreement is high but switches to agreeing more with the corpus as inter-annotator agreement drops. We contend that this alignment with annotators, despite being trained on the corpus, suggests that BERT is not memorising article use, but captures a high level generalisation of article use akin to human intuition.

## introduction and motivation pre-trained models, such as bert @xcite , roberta @xcite and more recently t5 @xcite , are the state of the art across several tasks in computational linguistics. in addition, transformer-based models are known to have access to information as varied as part of speech information @xcite , parse trees @xcite , the nlp pipeline @xcite , and constructional information @xcite . these models tend to perform so well that, on certain tasks, they outperform human baselines @xcite . in this work, we investigate how well language models, specifically bert large, perform on the linguistically interesting task of article prediction. english article prediction, further discussed in section 2, is a phenomenon that native speakers of english find almost trivial. at the same time, linguists find it particularly difficult to formulate the rules that would govern article usage: article use cannot be captured by local co-occurrence but is dependent on the wider context and often there is no one "right" article, but multiple options are possible, albeit with slight differences in the meaning conveyed. grammar correction systems prior to bert struggled to reach acceptable levels of performance on article selection (detailed in section 2). as we will show, bert shows performance on this task that is superior to that of humans. given this, it is interesting to investigate how bert attains this level of accuracy and what the implications are for the system: does bert manage to go beyond the local vicinity into the larger context to track the referent? the current study compares the performance of transformer-based pre-trained models and humans in an attempt to explore how language models handle an, in essence, creative task, with an emphasis on how model performance changes with inter-annotator agreement. we also explicitly incorporate the plural indefinite or zero (ø) article (detailed in section 2) as in the sentence there are ø merchant bankers who find it convenient to stir up ø apprehension with a view to drumming up ø business for themselves. the flexibility that is inherent in article usage requires us to explore methods of evaluation that do not rely solely on accuracy. while the shortcomings of relying too heavily on accuracy based metrics have been highlighted in prior work (see section 3), these difficulties are accentuated by the presence of flexibility. clearly, there is little need to require a model to output one specific class if people are comfortable with multiple options. as such, we evaluate performance based on the matthews correlation coefficient between human annotators and model outputs at each different level of inter-annotator agreement. to this end, this works aims to answer the following questions: a) how well do language models perform on a task that humans rely on intuition rather than deliberate reasoning, specifically article prediction, and b) how does this performance vary with increased flexibility in the article that can be used, as measured by inter-annotator agreement. so as to ensure reproducibility and to aid future research in this direction, we make our scripts freely available and our dataset, built from the british national corpus (bnc) @xcite , available under the required licence foot_0 . further details on the bnc are presented in appendix a. ## the english article system there are three articles in english: a) the definite article, the, b) the indefinite article, a/an, and c) the absence of an article or the zero (ø) article @xcite . there have been several sets of guidelines for the use of articles starting with the early works by @xcite ; @xcite . the most general ones rely on a few parameters only, such as hearer knowledge (whether the interlocutor can be considered to be able to identify the referent) and referent specificity (whether a specific referent is identified), augmented with number and countability, while the more specific ones offer numerous semantic types and subtypes, bordering on the idiosyncratic; see work by @xcite for an overview. although none of these variables, individually or in conjunction, can accurately predict article usage, recent work on the classification of a large, manually annotated sample has found that a hierarchical ordering of these same parameters, with hearer knowledge at the top, predicts article usage correctly in 93 percent of all cases that allow variation (about 15% of all instances can be considered a set phrase in that only one article can be used, e.g., "one at a time" @xcite . however, deciding whether the interlocutor can be considered as able to identify the referent involves world knowledge, including cultural knowledge; although both sheffield and birmingham are home to many universities, when we refer to the university of sheffield/birmingham we have one particular one in mind, which our interlocutor only knows if they are familiar with the local landscape. in addition, article usage appears to be a matter of what cognitive linguists would call construal, or the freedom to present a situation linguistically in different ways. analysing 3 alternative forced-choice data from 181 native speakers of english who were asked to insert articles that had been removed from longer (200-300 words) texts, @xcite relied on entropy to quantify the restrictiveness of the context and to identify types of contexts in which choice is allowed versus inhibited. they found that some contextual properties, such as referent specificity, are rather restrictive, leaving the speaker with little choice in terms of which article to use while other contextual properties, such as hearer knowledge, are such that several articles are possible, albeit with slightly different semantic implications in other words, only in situations where the referent is specific do native speakers tend to converge on the same article. the english article system thus finds itself in the awkward position of its strongest predictor being open to interpretation. the freedom regarding the interpretation of the top predictor, and the semantic differences it entails, is possibly why second language learners whose first language does not include an article system find the article system notoriously difficult to master. the same can be expected to apply to computational systems who tend to struggle to capture fine-grained meaning nuances, even though they have acquired world knowledge. ## related work automatic article prediction has been the focus of study for several decades starting with rule based systems, aimed at improving machine translation @xcite . subsequent machine learning models for article prediction included work by @xcite , who use decision trees and @xcite , who use a maximum entropy classifier to select among a/an, the, or the zero article. article prediction was then clubbed with similar phenomena, such as prepositions and noun numbers, to be included as part of shared tasks on grammatical error correction at @xcite and @xcite . these shared tasks, and their associated datasets, signifi-cantly increased interest in article prediction albeit as part of the broader problem of grammatical error correction. more recent methods, such as work by @xcite , make use of advances in neural machine translation for grammatical error correction. for an up-to-date and extensive handling of grammatical error correction, including article prediction, we direct readers to the tutorial by @xcite . of relevance to the second question we aim to answer, that of how annotator agreement affects model performance, is the work by @xcite , who study the various factors that influence the level of human agreement. additionally, ribeiro et al. (2020) show that state-of-the-art models are better evaluated using a checklist as opposed to traditional metrics, a notion that we supplement in our experimental procedure (section 4). ## methodology as mentioned in section 1, our goal is to understand how language models do on the task of article prediction and how their performance varies with inter-annotator agreement. our overall methodology for answering these questions involved the following steps: these results are presented in section 5 along with an analysis. the following sections detail the steps listed above. ## empirical evaluation and discussion the results presented in this section were obtained by evaluating bert l on the same gap filling exercise that was presented to humans. bert l was fine-tuned 5 times on 150,000 training examples and evaluated on a development set which, like the training set, was extracted from the corpus and not human annotated. the training data used consisted of 150,000 examples, of which about 135,000 were "the", 60,000 "a" and 146,000 "zero". the development set consisted of 30,000 examples, of which about 25,000 were "the", 12,000 were "'a" and 25,000 were "zero". the best performing run on this development set was used for the human annotated test set. of the 2,500 examples picked for manual annotation, 2,383 were annotated by the required five annotators and this subset was used for evaluation. this evaluation set consists of about 1200 sentences that were annotated by the majority of annotators with "the", 500 with "a", and about 550 with "zero". a further 108 sentences had multiple labels receiving the same number of votes and were thus tied. the complete evaluation set consists of about 150,000 tokens. tables 2 and 3 present the phi coefficients (matthews correlation coefficient) between four human annotators (4 human), different models, a fifth human used as a control (control) and the corpus. across all data, bert l has a higher correlation with the corpus (bert l vs corpus) than do the four human annotators (corpus vs 4 human) across all articles. while this can be ascribed to the fact that bert was fine-tuned on a fairly large training set of 150,000 examples, bert large also has a higher correlation with the four annotators (bert l vs 4 human) than they do with the corpus (4 human vs corpus) across all but one of the articles on which it misses out by an insignificant margin. although bert has a high correlation with the corpus across all data, a fine-grained analysis based on the possible level of flexibility in article use, as determined by inter-annotator agreement ( bert also has a significantly higher correlation with the corpus on the null article than do either the four human annotators or the fifth control annotator except in the case where there is complete agreement between the four annotators (4 agree). we believe that this is a result of the fact that we insert the null article using a fixed set of rules that deep neural models can easily pick up. human annotators, on the other hand, seem to find it harder to identify this addition to the article system, except in the more obvious cases. ## conclusions and future work in this work, we aimed to study the capabilities of pre-trained language models, specifically bert, on the linguistically relevant task of article prediction that native speakers are intuitively good at but linguists have been unable to formalise adequately, while focusing on how these abilities change with the increased flexibility in article use. our results show that bert has a very high correlation with human annotators when there is least flexibility as measured by inter-annotator agreement, but switches to agreeing with the corpus when there is flexibility in article use. these results, we contend, point to bert having access to a high level generalised representation of article use distinct from memorisation. we intend to focus future work on better understanding the specifics of this high level representation of article use contained within bert. also, the current study is limited in the languages explored and we intend to address this limitation by studying similar intuitive phenomena that evade linguistic description on languages other than english; an example would be aspect in slavonic languages. finally, we intend to extend our analysis by comparing bert's output 'confidence' with annotator agreement, similar to methods presented by @xcite . the bnc is available under the bnc user licence foot_3 and given that we build our dataset from the bnc, access to our dataset is subject to access to the bnc.