Detecting Syntactic Change with Pre-trained Transformer Models (2023)

Abstract: We investigate the ability of Transformer-based language models to find syntactic differences between the English of the early 1800s and that of the late 1900s. First, we show that a fine-tuned BERT model can distinguish between text from these two periods using syntactic information only; to show this, we employ a strategy to hide semantic information from the text. Second, we make further use of fine-tuned BERT models to identify specific instances of syntactic change and specific words for which a new part of speech was introduced. To do this, we employ an automatic part-of-speech (POS) tagger and use it to train corpora-specific taggers based only on BERT representations pretrained on different corpora. Notably, our methods of identifying specific candidates for syntactic change avoid using any automatic POS tagger on old text, where its performance may be unreliable; instead, our methods only use untagged old text together with tagged modern text. We examine samples and distributional properties of the model output to validate automatically identified cases of syntactic change. Finally, we use our techniques to confirm the historical rise of the progressive construction, a known example of syntactic change.

## introduction many researchers have taken advantage of the availability of pre-trained language models to study semantic change (e.g., @xcite @xcite , but computational studies of syntactic change are more sporadic, relying on specially trained models @xcite . in this paper, we investigate the extent to which bert @xcite is able to distinguish between the syntax of english from the early 1800s and that of the late 1900s. to do so, we use the british hansard corpus, which consists of a digitized version of the u.k. parliamentary debates starting in the early 1800s and spanning across two centuries. we isolate all of the parliamentary debates from the 1830s and the 1990s. 1 these debates can be found on the u.k. parliament website's hansard archive foot_1 . we fine-tune bert-base-uncased, which has around 110 million parameters and was pre-trained on approximately 3.3 billion words. when training or testing bert on a sentence, we first tokenize it using the original bert-base-uncased tokenizer, as implemented by hugging face @xcite . we focus on two questions: first, to what extent can bert distinguish text from different time periods on the basis of syntactic features alone? second, can we use bert to help identify words or sentences whose analysis reflects syntactic change in an unsupervised way? in §2, we find evidence that bert can be used to detect syntactic differences between the english of the 1830s and that of the 1990s. specifically, even after we control for confounders, a fine-tuned bert model can still distinguish between these two periods even when the semantic information is hidden. in §3, we describe how we train classifiers that can assign pos tags to words based on a 1830s and a 1990s understanding of english, respectively. our main contribution comes in §4, where we use these classifiers to identify words whose analysis reflects syntactic change. our method can automatically find words whose part of speech has changed. moreover, our method avoids running a pos tagger on old text, since automatic tagging may be unreliable in that scenario. we emphasize that our change-detection techniques work with no manual annotation and identify examples of language change without any manual filtering; see appendix b for a list of the top automatically detected words whose part of speech has changed. to the best of our knowledge, such unsupervised syntactic change detection has not been previously done. finally, we confirm a known change (an increase in the frequency of progressives) using our classifiers. we show that a bert model fine-tuned on text from the 1830s displays a poorer understanding of the progressive construction compared to a similar model trained on text from the 1990s. 3 2 detecting syntactic change while ## training classifiers we use bert to identify specific instances of change. to do so, we train two pos taggers, one based on text from the 1830s and the other based on text from the 1990s; in the subsequent sections, we will use these taggers and their errors to identify words or sentences that display syntactic change. in this section, we will focus on describing how these taggers are trained and their performance. crucially, we will train these taggers without ever tagging text from the 1830s; all pos tags are only ever associated with text from the 1990s. we start by fine-tuning two instances of bertbase-uncased: one on text from the 1990s and the other on text from the 1830s. this is done via masked-language modeling in pytorch @xcite , where we process one sentence at a time with 15% of the tokens masked, and the model is trained to predict the masked tokens. we use each fine-tuned model to obtain bert token representations for 1990s text and then average the token representations for all the wordpiece tokens within each word to obtain a representation of a word. we do this twice: once with the model finetuned on text from the 1830s and once with the model fine-tuned on text from the 1990s (yielding two different embeddings of each word). however, even though we have models trained on two time periods, we use both models only to embed words from the 1990s and not any words from the 1830s. at the end of this process, each word from the 1990s has two different bert embeddings. next, using pos tags obtained from stanford corenlp for the 1990s portion of the british hansard, we train a 2-layer perceptron (with one hidden layer of size 50) foot_4 to assign a penn treebank tag foot_5 to each bert representation of each word. an input to this 2-layer perceptron is a bert embedding of a word (extracted as described above), and the output is a pos tag. this classification task ignores punctuation and has 21 classes: we merge the 'nn', 'nnp', 'nns', and 'nnps' tags into one single tag called 'noun', after which we take the 20 most common tags ('noun', 'in', 'dt', 'jj', 'vb', 'prp', 'rb', 'vbn', 'vbz', 'vbp', 'cc', 'md', 'to', 'vbd', 'cd', 'vbg', 'prp$', 'wdt', 'wp', 'wrb') and then group the remaining pos tags into a catch-all class called 'other'. we actually train two such perceptrons separately: one for the 1830s bert embeddings (of words from the 1990s only) and one for the 1990s bert embeddings (also only of words from the 1990s). note that the training and test data for this procedure consist only of sentences from the 1990s; this is done because the pos tags obtained from stanford corenlp are likely to be more accurate on our modern data. using the representations from the final layer of bert (when the classifier has access to 50,000 training sentences, containing 815,578 words, and when the test set has 50,000 sentences, containing 816,428 words), the accuracy obtained by the 1990s model is 97.6%, and the accuracy of the 1830s model is 97.2%. we repeat the above analysis using pos tags from trankit @xcite instead of stanford corenlp because trankit is a state-of-theart parser and tagger. 7 by using trankit tags to train and test the above classifiers based on bert representations, the accuracy obtained by the 1990s model was 98.4% (instead of 97.6% using corenlp tags) and the accuracy obtained by the 1830s model was 98.0% (instead of 97.2% using corenlp tags). the higher accuracy means that the task became easier when predicting trankit tags instead of corenlp tags; since trankit tags are also more accurate, it suggests that our pipeline is working as expected. in §4 and later, we will use these pipelines (and the differences between their predictions) to iden-tify specific instances of syntactic change between the 1830s and the 1990s. ## identifying words that underwent change we now attempt to use the pos taggers trained above to identify words that underwent change. we employ two methods to do this: first, we find the instances for which the two trained models disagree on a pos tag (indicating that the model familiar with text from the 1830s would tag the word differently compared to the one familiar with text from the 1990s). the second method involves masking out the target word to see if the 1830s model predicts the tag correctly on the masked word. if the masked word is tagged correctly but the unmasked word is tagged incorrectly, it suggests that the 1830s model can correctly parse the rest of the sentence but is specifically confused by the target word; when this happens, it indicates that the target word may have undergone change. this latter strategy is depicted in figure 1 . ## confusion scores at the sentence level in this next experiment, we try to discern the extent to which masking out a word in a sentence makes the sentence more difficult for a fine-tuned bert model to parse. the idea is that, for a typical word, once the word is masked out the model has less information about the sentence, and its syntactic understanding should therefore decrease; on the other hand, for a word that has an atypical or new usage, the model may end up being more confused after seeing the word than it is when the word is masked out. in the latter situation, we would expect the model's syntactic understanding of the sentence to improve if the word is masked out because, from the model's point of view, the word is misleading. a similar idea was used in §4.2 at the word level; in this section we will look at the accuracy of the tags for the whole sentence before and after a single word is masked. for example, consider the following sentence: "opposition members think that is how peace processes work." a model trained on english from the 1830s is likely to think that the word "work" is a noun rather than a verb. this could lead such a model to consider an incorrect way to parse this sentence, where the subject "peace" is doing the action of processing the object "work". in this case, if the word "work" is masked out, the sentence will be more likely to be parsed correctly. as a proxy for the success of the parsing of the sentence, we feed in the sentence to the classifiers from §3 to get predicted tags for all the words in the sentence. we can use these classifiers to get probabilistic predictions, and in particular, we can get a probability assigned to the correct tag for each word in the sentence (using trankit tags as ground truth). we then multiply together all these probabilities to get a measure of the overall probability the model assigns to the correct tags of the sentence. we can repeat this for both the 1830s and 1990s classifiers, and we can also repeat this when a specific word is either masked or unmasked. we say a model is "confused" if the product of probabilities assigned to the correct (trankit) tags is higher when the word is masked than when it is unmasked. for instance, in the above example sentence "opposition members think that is how peace processes work," the 1830s model may assign a wrong tag to the word "processes" when the word "work" is unmasked, but it might correctly parse the sentence if the word "work" is masked (in the latter case, it will assume the masked word is an unknown verb, while in the former case it will try to parse the sentence with a noun in that position if it believes that "work" is a noun). in other words, if the 1830s model is confused by a word, it could be because this word underwent change from the 1830s to the 1990s. we create a "confusion score" by taking the logarithm of the ratio of the two aforementioned products (i.e. the product of probabilities assigned to the other words when the selected word is masked, and the corresponding product when the selected word is unmasked). we can get such a confusion score for any word in any sentence, and for either the 1830s model or the 1990s model. we focus on sentences from the 1990s that were not part of the training data. since calculating confusion scores is resource-intensive, we focus on a pre-selected set of word instances, such as those identified as likely having undergone syntactic change in §4. ## detecting the rise of the progressive in this section, we check the extent to which our techniques can identify syntactic change that was previously known to exist. the frequency of the progressive has been increasing over time (e.g. @xcite . an example of a progressive is "he is reading" (as opposed to "he reads"); the former has a form of "to be" followed by a verb that ends in "-ing". we identify progressive constructions using the dependency parse obtained from trankit @xcite . we then mask out the present participle if it has the tag "vbg" and if the previous word is an inflection of the verb "to be". (this means that we mask out the tokens constituting the "-ing" word as well as the "-ing" suffix itself.) next, we use both the 1990s model and the 1830s model to predict the pos tags of all the words in these masked sentences. we compare these predictions to each other and also to the predictions that both models give for the unmasked versions of these sentences. finally, we take randomly chosen words in randomly chosen sentences as a control group and repeat the process for this group. we summarize the results in two figures. first, figure 3 shows the confusion scores for the 1830s model on the progressive sentences compared to random sentences. as the figure shows, the 1830s model is negatively confused when we mask out progressives, which means that it gets the prediction right more often on the unmasked version of these sentences than it does on the masked versions. this makes sense, as it corresponds to the insight that the 1830s model has a difficult time correctly tagging the masked progressives. figure 4 : a histogram for the logarithm of the ratios between the products of probabilities assigned to the true tags by the 1990s model and the 1830s model. we compare this for progressive sentences and random sentences. in all cases, both models were given the masked version of the sentences. figure 4 compares the predictions of the 1990s model to those of the 1830s model for the masked sentences. it shows that the 1990s model predicts the tags of the progressive sentences better than the 1830s model does, and the gap between their predictions is larger than it is for random sentences. this is evidence that the 1830s model is unfamiliar with progressive constructions. ## related work it has been claimed that the performance of transformers on downstream tasks is insensitive to word order @xcite @xcite @xcite , which is consistent with our observation that bert fine-tuned on shuffled text can distinguish between two time periods with high accuracy; however, our experiments indicate bert's accuracy still improves when it has access to word order, consistent with @xcite , who argue that word order sometimes matters by considering non-prototypical cases where meanings and lexical expectations do not match. a sizable literature considers syntactic probes for bert embeddings (e.g. @xcite @xcite , but to our knowledge none of the works that probe syntax consider transformers fine-tuned on different time periods. syntactic change detection has been understudied in the computational linguistics and nlp literature. one work on this topic is @xcite , in which an lstm achieved higher accuracy compared to a model consisting of feedforward networks on the task of predicting the year of composition of sentences. however, in addition to predicting the time period of text, we also identify specific instances of syntactic change. computational methods for the study of syntactic change also appear in @xcite ; this work examined possible causes of the decline in the rate of passivization over time. another related work is @xcite , which found dependency distance to be decreasing over time (between the years 1790 and 2017). many studies have used bert to detect semantic change, including @xcite , @xcite . in addition, @xcite fine-tune historical bert models on a corpus of 19 th -century english books in order to tackle the task of time-sensitive targeted sense disambiguation and find that "bert works better when it is fine-tuned on data contemporaneous to the target period". ## conclusion we investigated the extent to which fine-tuned bert models can distinguish between the syntax of 1830s english and 1990s english. our approach used masking techniques, and we repeatedly compared bert models on masked and unmasked text. first, in §2, we hid semantic information by replacing each word with its pos tag (and representing that tag as a common english word that exemplifies that tag). we fine-tuned bert on such syntax-only data to see if it could tell apart text from the 1830s and text from the 1990s. we found evidence that bert can be used to detect syntactic differences between these two time periods. in the later sections, we masked out individual words to see whether a pos tagger based on bert embeddings (fine-tuned on either 1830s or 1990s) can still tag correctly when the word is masked. we filtered for a specific type of error, where the 1830s model predicts the correct tag when the word is masked but not when it is unmasked, and where the 1990s model is correct even when the word is unmasked. this procedure identified candidates for syntactic change, examples of which we provide in appendix b. we were also able to confirm the rise of the progressive construction. an important point to highlight is that our identification of syntactic change relied on an automatic pos tagger only for the 1990s text; we avoided running any automatic tagging software on older english data. in fact, from §3 onwards, all we did with the older 1830s data was to fine-tune bert on it in an unsupervised fashion. despite this restriction, we were able to automatically identify words whose part of speech had undergone change between the two time periods.