A Cross-model Study on Learning R omanian Parts of Speech with Transformer Models (2024)

Abstract: This paper will attempt to determine experimentally if POS tagging of unseen words produces comparable performance, in terms of accuracy, as for words that were rarely seen in the training set (i.e. frequency less than 5), or more frequently seen (i.e. frequency greater than 10). To compare accuracies objectively, we will use the odds ratio statistic and its confidence interval testing to show that odds of being correct on unseen words are close to odds of being correct on rarely seen words. For the training of the POS taggers, we use different Romanian BERT models that are freely available on HuggingFace.

## introduction transformer models @xcite and deep learning have changed the face of natural language processing (nlp) domain, with a huge number of papers reporting superior performances of any conceivable task of nlp, including machine translation, question answering (which is now handled almost flawlessly by generative large language models), and language analysis (pos tagging, dependency parsing, word sense disambiguation, etc.) transformer models are very good at any nlp task, provided they are pre-trained on very large corpora (billions of words) at supervised tasks such as masked language modeling or next sentence prediction @xcite and then, fine-tuned to the task at hand, e.g. pos tagging. central to the transformer models' remarkable ability to learn syntagmatic information about words is the attention mechanism @xcite , which encodes co-occurrence information in a large window of tokens (typically 512 tokens) for a large vocabulary of tokens (typically 50k tokens). comparatively, the number of papers dealing with the subject of how the transformer model is learning a language (or multiple languages at once), which presumably makes them so good at any language processing task, is very small with respect to the number of papers presenting extensions of the model, accuracy improvements, applications, and so on. with this paper, we want to contribute to the set of papers taping into the learning mechanisms of the transformer models, and we present a study on if and how the bert models (a type of transformer models) learn the grammatical categories (e.g. noun, verb, article, determiner, etc.) of a word in its context (i.e. pos tagging with a smaller tagset). we focus on romanian, and we use romanian-specific bert models for the job. we will try to experimentally prove that bert models have about the same accuracy on unseen (during training) words as on words that were rarely seen. furthermore, the accuracy of frequently seen words is not that much higher than the accuracy of unseen words. to quantify these comparisons, we will use the odds ratio statistic. ## related work experiments on how pos taggers work on words not seen during training were performed more than 20 years ago, at the time when pos taggers were actively developed using e.g. hidden markov models. an example in this regard is the work by @xcite , which addresses the pos tagging of unseen words with enhanced hmms. at that time, the best tagging accuracy on these words was about 66% for english. kim and smolensky (2021) investigated the ability of pre-trained transformer models (i.e. @xcite to perform grammatical category-based generalization of novel words, after being finetuned on limited contexts (without categorization-specific training). inspired by an experimental design in which infants were familiarized to contexts containing novel words and then tested with new sentences that either obeyed or violated category-based cooccurrence restrictions, the authors assumed that a masked language model's (mlm) ability to assign a higher probability to a word in a novel context that obeys the co-occurrence restriction for that category (over a word that does not) means the mlm makes a valid grammatical category inference about a novel word. the two-step method involves finetuning the mlm on two signal contexts that unambiguously mark the novel words (𝑤 1 and 𝑤 2 ) grammatical categories and testing the fine-tuned model by comparing the probabilities of 𝑤 1 and 𝑤 2 on multiple test contexts (higher probability to the new word in the correct test context meaning accurate category inference). the signal and test contexts are based on mnli corpus @xcite , that had different sources from the model's pre-training data. a finetuning set contains two signal contexts with one unseen word each (𝑤 1 and 𝑤 2 ) and 400 test contexts, 200 for each grammatical category (mnli-sampled sentences in wich words with grammatical categories of interest are masked out). six english datasets that test for the binary classification between the four open-class grammatical categories (noun, verb, adjective and adverb) were constructed. to use "unknown" words and make the bertlarge model to "forget" learned words, random weights were used for the unknown words' embeddings. the bert-large model was finetuned for 70 epochs and accuracy was tested at a significance level of @xmath0. conclusions were as follows: an action is telic if it has an end point and atelic otherwise. durative verbs describe an action, while stative verbs describe states. the authors work with english and french, and in one experiment, they fine-tune the transformer models for binary sequence classification of telicity and duration (separately), and of testing their accuracy on predicting these features. for fine-tuning they use a set of sentences annotated for these features. in another experiment, no fine-tuning was performed. instead, a logistic regression to the contextual embeddings of each layer is applied, as provided by the pre-trained models. contextual word embeddings for the annotated verbs are extracted from each layer of the transformer model and a logistic regression model is trained to classify telicity and duration, to understand how much such information was learned by each layer. for classifying telicity, all systems obtain an accuracy above 80% and it improved when training the models with the extra information of verb position in the sentence. bert (both base and large) had the best results. for classifying duration, the results are even better (higher than 93%), despite using a smaller dataset. no improvement could be noticed when 1 https://huggingface.co/readerbench 2 https://github.com/universaldependencies/ud_romanianrrt training the models with the extra information of verb position in the sentence. bert was also the best performing. an error analysis showed that conflicting characteristics of the linguistic context prevent the correct analysis: e.g., sentences where the verb or the verbal phrase would be considered (a)telic, but part of the context defines the temporal aspect of the sentence in the opposite way. for french, the results are not as good as for english, probably because of the characteristics of the french verbal system. contextual embeddings proved to be an efficient way to encode the aspectual information of a verb and its interaction with its context, and this knowledge is probably already learned in the pretraining process. ## robert models robert 1 is a romanian-only, pre-trained bert model. @xcite developed this model to address the gap in pre-trained language models for languages other than english. the model was designed similarly to bert with small, base, and large variants, having the same number of layers, hidden params, and attention heads. the training time in hours for each model was 28, 77, and 255, respectively, training for 40 epochs on a v3-8 tpu on two supervised tasks: masked language modeling and next sentence prediction. the romanian dataset that was used for training was comprised of 3 sources, totaling 2.07b words. without dwelling into details, the robert models outperform the competition in several tasks, namely mbert @xcite , @xcite , and the only true romanian bert model at that time, bert-base-ro @xcite . ## conclusions we have presented evidence that properly trained bert models exhibit learning words' grammatical categories, especially when the words were not seen during the training process. we drew this conclusion by measuring the odds ratio of pos tagging accuracy when the frequency of the test words (in the train set) is greater than 0 vs. when this frequency is 0. thus, models robert-small and robert-large show an odds ratio that is less than 2 for the accuracy of tagging frequent words vs. tagging unseen words. we could not say that model corola bert exhibits the same behavior due to its insufficient pre-training for its large vocabulary. the model robert-base shows a different behavior with respect to accuracy odds ratio vs. test word frequency: while the odds ratio of pos tagging accuracy is below 2 when comparing rare words (@xmath1, as in the case of the other two sibling models, when the frequency increases (e.g. @xmath2, the odds ratio settles at a bit over 3 (twice as much when compared to the other two models). while it is expected that the pos tagging accuracy increases with the test word frequency (in the train set), as more contexts of those words were seen during training, robert-base does much better than the proceedings of clib 2024 other two sibling models when test words were seen during training. this hypothesis is supported by the top pos tagging accuracy of robert-base compared to any other tested model (see tables 4 and 5 ). one possible explanation for this situation is that robert-base has the best number of parameters (not too few, nor too many) for our pos tagging task and this enables its accuracy odds ratio curve to increase more sharply than siblings' curves, but not that sharply as the odds ratio curve of corola bert which indicates more of an overfit of the training data than a good performance.