On the Acquisition of W ord N et Relations in P ortuguese from Pretrained Masked Language Models (2023)

Abstract: This paper studies the application of pretrained BERT in the acquisition of synonyms, antonyms, hypernyms and hyponyms in Portuguese. Masked patterns indicating those relations were compiled with the help of a service for validating semantic relations, and then used for prompting three pretrained BERT models, one multilingual and two for Portuguese (base and large). Predictions for the masks were evaluated in two different test sets. Results achieved by the monolingual models are interesting enough for considering these models as a source for enriching wordnets, especially when predicting hypernyms of nouns. Previously reported performances on prediction were improved with new patterns and with the large model. When it comes to selecting the related word from a set of four options, performance is even better, but not enough for outperforming the selection of the most similar word, as computed with static word embeddings.

## introduction as it happens for many other tasks in the domain of natural language processing (nlp), transformerbased language models have been explored in the acquisition of semantic relations, towards their application in the creation or enrichment of knowledge bases, or on their direct usage as knowledge bases @xcite . more precisely, having in mind that a typical application of language models is text completion, transformer-based models have been used for completing lexical patterns, in what can be seen as a shortcut to earlier research on the acquisition of relations from textual corpora (e.g., @xcite ). if the focus are lexico-semantic relations, such an approach can be useful for enriching wordnets @xcite . in this study, we build on previous efforts, specifically those targeting the portuguese language @xcite , and evaluate the acquisition of synonymy, antonymy, and hypernymy-hyponymy from bert models, namely the base and large versions of bert pretrained exclusively for portuguese @xcite , and the multilingual bert. evaluation is made on two test sets, both covering different variations of the target relations, and starting with source words, but with different goals: in b 2 sg @xcite , a related word has to be selected from four options; in tales @xcite , one related word has to be predicted. since the approach is not just dependent on the models, several patterns were handcrafted for each target relation, building on previous work, but also on the adaptation of patterns used in the scope of varra @xcite , a service for searching for and validating instances of lexico-semantic relations by resorting to portuguese corpora. after fixing the first argument of each instance as the source word, patterns were used to prompt the bert models, results were evaluated in the test sets, and conclusions were drawn. performance with the multilingual model was poor, and the large model is generally the best option. when selecting the correct candidate in b 2 sg, results are positive, but end up being outperformed by simply selecting the option that maximises similarity, computed in a model fine-tuned for computing semantic similarity or in static word embeddings. predicting the related words is more challenging. nevertheless, top performances are achieved when predicting hypernyms and results can still be useful for suggesting new relation instances to wordnets. moreover, using the large version of the model and including the varra patterns contributed to improvements in previously reported performance in tales. in the remainder of the paper, section 2 overviews related work on the automatic acquisition of semantic relations from text and language models; section 3 describes the adopted approach in more detail, focusing on the patterns, the test sets and the models; section 4 reports on the best patterns for each relation and test set, together with their performance; section 5 summarises the main conclusions and future directions of this work. ## related work the enrichment of wordnets with relations extracted automatically from corpora has a long tradition, following the work of @xcite , where a set of lexico-syntactic patterns denoting hyponymy was presented and applied to the acquisition of relation instances. to minimise human intervention, hyponymy patterns were learned automatically with distant supervision @xcite , and patterns for other relations were learned and ranked with weak @xcite , in both cases using seed examples from princeton wordnet @xcite . on relation extraction from portuguese text (de @xcite , only a minority is focused on lexico-semantic relations. these include rule-based approaches for acquiring hyponymy (de @xcite and partof @xcite relations from corpora; as well as other relations from dictionary definitions @xcite . a more recent alternative is to acquire relations from distributional models, such as word embeddings. even if relations are not explicit, analogies @xcite have been computed for a broad range of syntactic and semantic relations. besides the unsupervised discovery of hypernymy instances @xcite , the performance of simple analogy was improved by learning to compute related words from multiple examples @xcite , more specifically, from the bats test set, which covers synonymy, antonymy and hypernymy, among other syntactic and encyclopaedic relations. the previous were also applied to portuguese word embeddings, when used to solve lexico-semantic analogies in tales @xcite , a test of with the same format as bats @xcite . despite the low accuracy, among the predictions there are useful suggestions that may be manually added to wordnets, as it happened with openwordnet-pt @xcite . but the current paradigm in nlp are transformerbased models, like bert @xcite or gpt @xcite , and there has also been work on using them as knowledge bases @xcite . even if they are not ready for explicitly retrieving semantic relations, using the right prompts can result in the acquisition of related words, in what can be seen as a shortcut for earlier corpora-based approaches, i.e., these models are pre-trained in large collections of text and are good at filling blanks @xcite , completing sentences @xcite , or computing their likelihood @xcite . among other efforts, pretrained bert has been assessed for the presence of relational knowledge using discrete prompts @xcite ; for relation induction @xcite , starting with a small number of patterns and seeds; or for classifying semantic relations based on attention weights @xcite . some researchers conclude that the prompting approach suits better some relations (e.g., hypernymy) than others @xcite , while others have shown that bert is not very good at predicting hyponymy relations inherited through transitivity @xcite . for portuguese, recent work exploited bert for detecting hyponymy pairs (paes, 2021), ranking automatically extracted relation instances (gonçalo oliveira, 2022), or acquiring new instances (gonçalo oliveira, 2022). ## approach gonçalo oliveira (2022) proposed the acquisition of lexico-semantic relations from bertimbau @xcite , a bert model pre-trained for portuguese, using prompts that indicated the target relations. since bert is pretrained on masked language modelling in a large corpus, the pretrained version should be enough for acquiring lexico-semantic relations. some considerations were made on setting the prompts and results were evaluated in the tales (gonçalo oliveira et al., 2020) test of lexico-semantic analogies. however, results were limited to using bertimbau-base and to an initial set of handcrafted patterns. here, we augment the previous work by considering a second dataset, b 2 sg, other bert models, and additional patterns adapted from varra @xcite , which lead to improvements on performance. moreover, we discuss synonymy in more detail. ## results this section reports on the best patterns for each test and relation, and discusses the achieved evaluation scores. for each test, scores are also compared with alternative approaches. ## conclusion this paper reports on the experimentation of bert models for portuguese for answering relation tests, by prompting them with patterns that indicate synonymy, antonymy, hypernymy and hyponymy relations. our first conclusion was that monolingual models perform substantially better than a multilingual model. second, when it comes to selecting the related word from a limited set of options, the proposed approach performs ok, even if better for relations between nouns than between verbs. however, this turns out not being so useful, because it is outperformed by simply selecting the most similar word, as computed in a fine-tuned bert or in static word embeddings. third, this approach can be used for predicting related words, in this case, better for noun hypernyms, as in previous work for english @xcite . we also note the positive impact of using bert-large and of including the patterns of a relation validation service, which enabled the improvement of previously reported results in the same dataset. at the same time, there is still much room for improvement, and performances achieved suggest that it might be risky to create or enrich a knowledge base in a completely automatic fashion. yet, given that the reported evaluation ends up being limited by the contents of the test sets, in the future, it could be interesting to test how far one could go by adopting this approach for the creation of a knowledge base completely from scratch. additional conclusions could be taken from manually evaluating a sample of extracted instances. we should, nevertheless, look at bert as an alternative source of knowledge, capable of providing suggestions for enriching knowledge bases, even if they need to be manually-validated before actual inclusion. this would be similar to what happened in the enrichment of openwordnet-pt @xcite , with suggestions computed from static word embeddings. bouraoui et al., 2020). in any case, having in mind reproducibility and future improvements, the list of patterns was made available for anyone willing to use it.