Probing Pre-trained Language Models for Semantic Attributes and their Values (2021)

Abstract: Pretrained language models (PTLMs) yield state-of-the-art performance on many natural language processing tasks, including syntax, semantics and commonsense. In this paper, we focus on identifying to what extent do PTLMs capture semantic attributes and their values, e.g., the correlation between rich and high net worth. We use PTLMs to predict masked tokens using patterns and lists of items from Wikidata in order to verify how likely PTLMs encode semantic attributes along with their values. Such inferences based on semantics are intuitive for humans as part of our language understanding. Since PTLMs are trained on large amount of Wikipedia data we would assume that they can generate similar predictions, yet our findings reveal that PTLMs are still much worse than humans on this task. We show evidence and analysis explaining how to exploit our methodology to integrate better context and semantics into PTLMs using knowledge bases.

## introduction given the ability of pretrained language models (ptlms), such as bert @xcite , to create useful text representations, they have become the standard choice when building nlp applications @xcite @xcite . however, there has recently been a rising amount of research that uses probes to understand the level of linguistics ptlms encode. different probing experiments have been proposed to study the drawbacks of ptlms in areas such as the biomedical domain @xcite , syntax @xcite , semantic and syntactic sentence structures @xcite , prenomial anaphora @xcite , linguistics @xcite @xcite and commonsense knowledge @xcite @xcite . figure 1 : in framenet, adjectives are lexical units that evoke other frames: "rich" is a lexical unit that evokes the frame wealthiness, while "old" evokes age. the relation between rich and high net worth could be defined as a value-attribute pair, where rich is the value of an expression that represents an attribute: wealthiness/net worth. lu stands for lexical unit, fn1_sent represents and finished_initial refer to which framenet version the lexical unit is from. in this paper, we expand on this line of research by probing ptlms to investigate if they cover semantic attributes and their values. the closest work to ours has been proposed by @xcite , where they investigate if ptlms capture checklists such as: red, green and yellow. in contrast to them, we focus on finding out whether pretrained language models capture the correlation between semantic attributes and their values. an example of semantic attributes and their values is the relation that exists between old, age and date of birth, or the relation between rich, wealth and net worth. looking up rich on framenet @xcite would not result in a frame by itself, but would evoke the semantic frame wealthiness (figure 1 ). in wordnet @xcite these associations are called an attributevalue relation, where the attribute is a noun for which adjectives express values. for instance, the patterns a, [mask] and b. what's [value] or [mask], a or b? what's [value] and [mask], a or b? a is [mask], thus they have a [value] a is [value], thus it has more [mask] per m 2 . to know which is [value], a or b, you need [mask]. what's [value], thus has a higher [mask], a or b ? we need the [mask] to know what's [value], a or b. we need the [value] to know who is [mask] a or b a is famous for [value], thus it has more [mask]. knowledge bases (kbs), such as wikidata @xcite , constitute a valuable resource for collecting attributes and their values. in general, kbs have been shown to help improve multiple nlp application as they contain structured information @xcite @xcite . as matter of fact, it is fairly simple to answer factoid questions such as "how old is joe biden?" using wikidata, by simply looking up his date of birth on wikidata. an important step to make this happen is to match old and date of birth to each other. similarly, to check "how rich is jeff bezos?", we only need to extract his net worth from wikidata, and identify that rich and net worth are related to each other. however, a simple task like this one requires us to have tools that can identify the relation between the attribute net worth and its qualitative value rich. despite this task being straightforward to perform manually, it is not yet solved automatically due to linguistic challenges. in the previous example, rich and high net worth are not synonyms, and do not share the same part-of-speech tag since the former is an adjective and the latter is a noun. however, they tend to appear together in text: net worth occurs 34 times in the wikipedia page of jeff bezos, while rich/-er/-est appears 35 times. in this paper, we conduct a case study to identify to what extent do language models that are pretrained on massive amounts of data learn these attribute-value correlations. in other words, we ask the question: "do ptlms understand that a given value is associated with a specific attribute?". we assume that ptlms should learn these relations given that they are trained on everyday's online data, which contains a wide amount of attributevalue pairs. our goal is to see our work aspire future works by: a) enabling more efficient comparative qa and factoid qa such as "q: who's older obama or trump", b) improving ptlms' abilities to faithfully capture attribute-value relations, and c) a step towards finding resources to fine-tune ptlms towards semantic objectives. ## methodology in this paper, we aim at showing to what degree do ptlms contain abstract semantically-based relations like attributes and their values. for instance, when asking "which is denser, new york or hong kong?", humans automatically link density in this specific case to urban cities and population. in order to show what ptlms are able to understand when it comes to attribute-value pairs, we start by defining and collecting data from the predefined set of patterns shown in ## results we evaluate the predictions from the ptlms manually using heuristics based on the relatedness between the predicted attribute-value pairs foot_1 . if an entity-value pair are semantically related, we give a score of 1, else we give a score of 0. for instance, in masked sentence prediction 1. what's heavier or [mask], gold or silver ? lighter, softer, thinner, heavier, light 2. what's heavier and [mask], gold or silver ? lighter, softer, stronger, heavier, brighter 3. what's heavier, and thus has a higher [mask], gold or silver? figure 3 : comparing bert and roberta to human predictions. the human prediction was collected from only one person, the accuracy was high since this kind of task is simple. we excluded xlnet from this evaluation, since its predictions were completely random. the sake of comparison. we note from the figure that the human prediction outperforms the ones from bert and roberta by a very large margin. however, the predictions from roberta outperform bert. to dig deeper into the model's predictions, we looked into concrete examples ( ## discussion and future work we consistently observed throughout multiple examples that ptlms are vulnerable to slight changes such as the name of the city or the name of the element that we are targeting. figure 2 shows ten randomly selected cities from the input sentence: what we found interesting in our case study, is that changing only the city, changes the distribution of the predicted words, but in what seems to be a random fashion. we note from the figure that only two cities, namely freetown and asunción, trigger predictions related to inhabitants and population. while we initially conjectured that freetown has also 'town' in its name, we were surprised that this did not apply for panama city. we argue that even though bert is trained on wikipedia content, the huge amount of data and the biased distribution of other linguistic phenomena makes it difficult to capture attribute-value rela-tions. for that reason, it could be possible to train a robust semantically-aware ptlm by fine-tuning the current ptlms to framenet content, and a first step would be to integrate wikidata entities and their values according to the semantic frames of each entity and every word that is evoked by a value. one advantage of ptlms is their capability to perform well on specific tasks and domains that were not part of their training regime via fine-tuning, i.e. the retraining of a pretrained model with domain-specific examples. we argue that for entities and their values, resources such as framenet and wordnet, while paired with massive resources such as wikidata could be used to fine-tune ptlms towards more semantically-based objectives, as a complementary work to ernie @xcite , which showed that fine-tuning ptlms towards knowledge graphs helps enhancing language representation with external knowledge. ## conclusion we demonstrated that ptlms are unable to capture semantic similarity between different words that refer to the same concepts. while ptlms have been shown to improve the quality of many tasks and are not easy to train, our probing experiments show that an improvement is necessary. all the examples we extracted from wikidata show that by enabling ptlms to capture more semantically-based information by fine-tuning towards more semanticallybased objectives like the ones found in framenet. all our examples are extracted from wikidata to show that, resources such as wikidata are rich and could be used as a resource for fine-tuning bert towards more high-level semantics.