Using W ikidata for Enhancing Compositionality in Pretrained Language Models (2023)

Abstract: One of the many advantages of pre-trained language models (PLMs) such as BERT and RoBERTa is their flexibility and contextual nature. These features give PLMs strong capabilities for representing lexical semantics. However, PLMs seem incapable of capturing high-level semantics in terms of compositionally. We show that when augmented with the relevant semantic knowledge, PMLs learn to capture a higher degree of lexical compositionality. We annotate a large dataset from Wikidata highlighting a type of semantic inference that is easy for humans to understand but difficult for PLMs, like the correlation between age and date of birth. We use this resource for finetuning DistilBERT, BERT large and RoBERTa. Our results show that the performance of PLMs against the test data continuously improves when augmented with such a rich resource. Our results are corroborated by a consistent improvement over most GLUE benchmark natural language understanding tasks.

## introduction given their recent success in various natural language processing (nlp) tasks, there has been increasing work on understanding the abilities of pretrained language models (plms) beyond what they can memorize. having been trained on billions of words, bert @xcite has shown impressive language representation abilities. however, there has not been much work on the degree of knowledge that bert could infer about different topics from just the lexical information that they are trained on. therefore, there has been a growing interest in probing plms on all kinds of linguistic, syntactic and semantic features @xcite @xcite @xcite @xcite @xcite . figure 1 shows a few examples of high-level semantics relating to compositionality. for instance, when asked questions such as "what's higher mt. everest or mt. fuji?´´or "how tall is bill clinton?´´, a person would most likely, and naturally think about altitude and height respectively, to accurately answer this question. when it comes to reasoning and inferences between semantic attributes (net worth) and their values (rich), humans can systematically infer between these concepts. the closer semantics in nlp that fits this case is compositional semantics since we investigate how different words in a sentence are linked to other words i.e. net worth being linked to wealth, and altitude is linked to the height of a mountain. in this paper, we create a large dataset from wikidata @xcite , where each sentence contains two words that are semantically related. we then fine-tune three pre-trained language models, namely bert @xcite , roberta @xcite and distilbert @xcite , using this data. we create test data that has the same style as the training data, but with different objects and inferences. we obtained a remarkable boost in the quality on the test data. furthermore, we also report a consistent improvement over the glue benchmark for natural language understanding @xcite . our main contributions are: • a large dataset containing high-level semantics inferences, ## probing pre-trained language models using probes has become a common way to investigate the knowledge encoded in transformer-based @xcite pre-trained language models such as bert. these investigations have varied from linguistic features to include commonsense knowledge and social biases that plms might have learned during the training. @xcite used question answering to show that plms fail at rational reasoning when it comes to capturing the numerical commonsense. more work has focused on studying different linguistic features and the level of linguistic competence in different plms @xcite @xcite by making use of fine-tuning and sentence-level semantics. probes were also used to identify social toxicity and bias towards different interest groups as we as gender bias @xcite . other probing experiments have been proposed to study the drawbacks of plms in areas such as the biomedical domain @xcite , syntax @xcite , semantic and syntactic sentence structures @xcite @xcite , linguistics @xcite @xcite and commonsense knowledge @xcite @xcite . when it comes to language understanding, @xcite showed that when it comes to polysemy, bert creates closed semantic regions that are not clearly distinguishable from each other. another finding relating to semantics @xcite conveys that, unlike syntax, semantics and general world knowledge are not inherently learned, and thus not brought to the surface by the representations obtained from pre-trained language models. ## data creation we use the knowledge graph extracted from wikidata to construct the dataset. wikidata @xcite ) is a collaborative knowledge base, containing triples (entity id, property id, value id) that define a type of relation holding between an entity and a value. wikidata also contains labels and aliases for the properties, entities, and values, which makes it the perfect resource for extracting similar objects that are likely to have similar values. we then investigate the ability of plms to capture the semantic relationship between the attribute-value pairs and further fine-tune plms to capture this relation effectively. 1 algorithm 1: creating fine-tuning data from wikidata objects. result: fine-tuning dataset fine-tuning-data=; while keyword in (f ood, f urniture, city, tool) do @xmath0, end while @xmath1, @xmath2, i + + do bert -sent(i)= bert prediction on sentence i, extract all subclasses of keyword from wikidata, if bert-sent(prediction) == accurate-prediction then fine-tuning-@xmath3 in the knowledge graph, we focused on entities that were labeled food, furniture, city and tool, with nutritious-healthy, wider-width, rainfall-humidity and longer-length as entity-value pair respectively. food is selected as the key because food items exhibit the attribute of nutrition, and thus comparing the subclasses of food, in terms of their nutrition can enable us to compare which food item is more healthy. for city, different cities have different rainfall and thus comparing the rainfall between different subclasses and instances of city can enable us to compare which city has more humidity. we applied the same analysis to furniture and tool. in order to capture the semantic relationship between the attribute-value pairs, we create a dataset from the sentences where the value in the attributevalue relationship had been accurately predicted by bert. the subclasses and instances of the keys food, furniture, city and tool were extracted from the knowledge graph and then used in combination with each other to create sentences of the form "which is @xcite , and thus has more [value], [object 1] or [object 2]´´where the objects represent the words used for comparing the attributevalue pair. for example, to analyze the ability of plms to capture the semantic relationship between wider(attribute) and width(value), we consider bed(object 1) and chair(object 2) to be the chosen subclass combinations of the key furniture. therefore, the sentence "which is wider, and thus has more width, bed or chair?´´, is constructed with width(value) being masked. our final dataset contains around 8,000 finetuning samples, using five distinct attribute-value pairs. we divided our data into three categories, a dataset containing: (1) one-word objects, such as chairs, and couscous; (2) one-word objects and twoword compounds, such as folding chairs and bean sprout; and (3) all possibilities, including threeword compounds, such as aged cheddar cheese and slip joint plier. the purpose is to check how compound words affect the accuracy of the fine-tuned model, or in other words, does it matter to the plm whether a noun is a compound or not? ## fine-tuning plms for high-level semantics we used huggingface @xcite for fine-tuning bert @xcite , roberta @xcite and distilbert @xcite foot_1 . for the fine-tuning, 15% of the tokens were masked randomly and the plms are fine-tuned with a masked language model objective by minimizing the loss based on the gold standard. the fine-tuned model is then evaluated on the test dataset, which consists of 50 different sentences with different semantic relationships. ## experimental setup test data when finetuning the plms, one of the most challenging tasks is to prove that model could learn from the finetuning and is not just overfitting to the specific task. for that reason, we are testing on two different datasets: a wikidata-based test set and the glue benchmark for natural language understanding @xcite . bert-based models have significantly increased state-of-theart over the glue benchmark, and most of the best scoring models for this benchmark include or elaborate on bert. we train our model on five topics, with different objects, but we test on 50 different attribute-value pairs. in order to show a certain generalization over the training data, we made sure that no attributevalue pair from the training is part of the test data. the masked word is then predicted by different plms. the accuracy of the top one and top five predictions is calculated. we purposefully diversify our test set from our training set to show that the improvement is not mere memorization. our test data contains different objects such the eiffel tower or burj khalifa, which are both instances figure 3 : bert cannot predict the correct predictions when it comes to the mountain context. after finetuning, the predictions are more relevant to the context, even though altitude was not part of the fine-tuning data. results ## conclusion in this paper, we investigate how plms capture a very specific type of compositionality between different concepts. we also finetune two different plms on five different attribute-value pairs and test the model on 50 annotated themes. the training data and the test data have different topics and wording. additionally, we purposefully limited the fine-tuning data for the scope of this short paper, since we did not want to make plms memorize all possible concepts. our results show that, by having a resource that contains a basic level of lexical compositionality, we indeed help improve plms accuracy. however, we also show that there is more improvement in the glue tasks that are more semantically sensitive.