Bridging Perception, Memory, and Inference through Semantic Relations (2021)

Abstract: There is a growing consensus that surface form alone does not enable models to learn meaning and gain language understanding. This warrants an interest in hybrid systems that combine the strengths of neural and symbolic methods. We favour triadic systems consisting of neural networks, knowledge bases, and inference engines. The network provides perception, that is, the interface between the system and its environment. The knowledge base provides explicit memory and thus immediate access to established facts. Finally, inference capabilities are provided by the inference engine which reflects on the perception, supported by memory, to reason and discover new facts. In this work, we probe six popular language models for semantic relations and outline a future line of research to study how the constituent subsystems can be jointly realised and integrated.

## introduction recent works @xcite postulate that it is impossible to learn meaning from surface form alone, and express concerns about what is perceived as an over-reliance on large-scale pretrained neural networks. this line of thought supports the interest in hybrid systems that amalgamate elements from complementary learning paradigms (see, e.g., @xcite @xcite ). in @xcite , we argue that this calls for an explicit distinction to be made between the faculties of perception, memory, and inference. we therefore promote the development of systems that consist of subsystems with responsibilities corresponding to the three faculties. such future systems would thus consist of a perception component realised by a neural network, a component that provides explicit memory in the form of a knowledge base, and a third one performing symbolic inference, that is, rule-based reasoning. we suggest to study how the subsystems can be aligned so for a seamless information flow between them. we view it as particularly important that (i) the network and the knowledge base together yield a consistent treatment of semantic relations and (ii) training takes the knowledge base into account, so that the resulting embeddings are consistent with established facts. our conceptual discussion is complemented by a preliminary empirical evaluation of six popular english language models, which we subject to linear probes to test their abilities to capture central semantic relations. after a brief discussion of related work in section 2, section 3 discusses the role of semantic relations in the context of our envisioned triad system while section 4 and 5 of this paper complement our conceptual discussion with a preliminary empirical evaluation of the chances to achieve (i) by probing six popular language models with respect to a semantic relation learning task. ## related work there is a rapidly growing literature on relation extraction and hybrid systems. @xcite observe that language models such as bert @xcite and @xcite are imprinted with large amounts of common sense and factual knowledge during training. if this information can be reliably extracted then, they argue, word embeddings could find a new use as knowledge bases. to test the practicality of this approach, they consider a knowledge extraction task where a language model is given a sentence containing a subject word x and a relation r, but where the object word y has been removed, and the model should guess the missing y (i.e., rank the vocabu-lary words) based on the fact that x and y are in the relation r. the sentences are generated based on manually constructed templates, one per relation. for example, to the relation birth-place, they use the template " subject was born in blank " and instantiate it to "dante was born in blank ". the most important baselines are two variations of the relation extraction model by @xcite . key findings are that language models appear to be better at learning one-to-one relations, whereas the relation extraction models are better at picking out many-to-many relations. petroni et al. also find that the choice of template has an impact on the performance of the language models, and point this out as an item for future work. @xcite pick up this thread and propose a method for extracting good template sentences from bert, and using these to fine-tune bert so as to improve its performance on relation extraction. for a target binary relation r (represented as a set of ordered pairs) and a sample of pairs r ⊆ r, they filter the training data for sentences expressing that x and y, with (x, y) ∈ r, have the relation r, and which would still be natural if x and y where simultaneously replaced by some other (x , y ) ∈ r. finally, they fine-tune a language model to predict, from an instantiation of one of the remaining sentences with a pair (x , y ), whether (x , y ) ∈ r. the most relevant aspect of this work for the present effort is the evaluation of the bigger analogy test set (also known as bats) which contains 40 relations with 50 instances per relation @xcite . @xcite report a mixed performance on the type of semantic relations considered here, namely hypernyms and hyponyms. additional methods for choosing template sentences are proposed by @xcite who, similar to @xcite , mine the training data for suitable sentences. a dependency analysis on candidate sentences makes it possible to extract a larger variety of phrases that express the desired relationship than @xcite can. the authors also generate candidate sentences by paraphrasing. in short, they find that both mined and paraphrasing have their usages, and that combinations of template types, e.g., manually constructed and mined, often perform well. @xcite that bert contains factual knowledge derived from the training data. the authors believe that in may cases, bert simply exploits superficial similarities and general patterns to guess what is most likely. for example, from the fact that a person has a typically french surname, bert could guess that that person is actually french without having learned the nationality of the particular person. to expose this weakness, @xcite remove what they believe are easily guessed pairs of subjects and objects from the data set of @xcite . they also provide a modified version of bert, e-bert, in which the embeddings of entities mentioned in wikipedia have been replaced by a symbolic entity embedding. they find that e-bert outperforms both bert and ernie on the trimmed data set, but also that a combination e-bert and bert (taking the average of or concatenating the embeddings) give higher accuracy than either on its own. ## the role of semantic relations as the brief account given in the previous section shows, there is a solid body of work on the extraction of relations from language models (see section 2), to derive facts such as that the birth place of olga tokarczuk is sulechów, poland, and that the capital of bolivia is la paz. looking to knowledge bases, it is natural to view them as graphs, where nodes represent objects and properties, and edges represent semantic relations. finally, for fish vertebrate craniate centralised brain starfish hypernym hyponym synonym meronym meronym / figure 1 : in this work we focus on recovering synonyms, hypernyms, hyponyms, and meronyms from natural language models via probing to understand the prerequisites of integration with knowledge bases. ## empirical study: method to gain some initial insight into how well stateof-the-art pretrained contextual embeddings handle lexico-semantic relations, we conducted experiments on word embeddings generated by al-bert @xcite , roberta @xcite , bert @xcite , and gpt-2 @xcite . we also included word2vec @xcite and glove (pennington et al., 2014) models in our experiments, for comparison. these are all self-supervised learning algorithms, based on neural networks and built to translate words into vector representations. bert and gpt-2 are transformer models, each having 12 encoder layers. roberta is a retraining of bert on a larger data set, while albert is an extension of bert that has a higher data throughput with 10x fewer parameters, and thus scales better. in contrast to the works discussed in section 2, we do not extract relations from the embeddings by means of linguistic templates. rather, we view sentence extraction as an instance of probing @xcite @xcite , a diagnostic method to reveal what aspects of the input the embedding actually encodes. probing tasks should ideally be agnostic as to the underlying encoder architecture, so that results are transferable between embeddings @xcite . random control tasks @xcite are implemented, see discussion in section 5. in our experiments, we considered the following probing task: given a pair of word vectors, we ask whether the encoded words are in relation r. this avoids the optimisation problem linked to the choice of template seen in @xcite . all experiments are on the english language, and the data set used in our experiments was obtained from wordnet as follows. we first built a vocabulary v by taking the 5 000 most common nouns in the brown corpus @xcite and removing those not found in wordnet @xcite . this resulted in a vocabulary of 3497 words. for each word w in the vocabulary v and target relation @xmath0, meronym, synonym} we then picked words v and v in v such that we formulate a classification task for each relation r, and probe each of the investigated models for their ability to capture each relation in their respective embeddings. each classification task is based on 1 712, 306, 2 740, 1 630 samples for synonyms, meronyms, hypernyms, and hyponyms respectively. we use a linear classifier probe as these better reflect the availablity of the information probed for, as shown in @xcite . from (w, v, v ), positive (w, v) and negative (w, v ) examples are drawn with equal probability, labeled either 0 or 1, to represent if the tuple represents a negative or a positive pair. the binary labels are given together with either (w, v) or (w, v ) as input to the probe by concatenating both word embeddings. we train the probe for 10 epochs using 5-fold cross validation, using softmax activation, dropout of 0.2 to prevent memorising samples, and cross-entropy loss with the adam optimizer using a @xmath1.001. we average the results over 5 runs. the experiment is implemented with pytorch for cpu and uses the huggingface @xcite library for all pretrained transformers, and the gensim @xcite library for word2vec and glove. the experiments completed within 1 hour on an intel i7-based linux laptop with 32gb ram. the code is available on github 1 . ## results and discussion various observations can be made by comparing the results for the individual embeddings. particularly worthwhile noting is the fact that glove and word2vec performs on par or better than the contextual embeddings, except for the case of hyponyms. this behaviour was seen with 5 and 20 training epochs as well. the relatively strong performance of the pretransformer solutions may not be surprising as far as synonyms are concerned, since their construction builds around aligning words found in the same context. however, we would not have expected similar results for hypernyms and even lesser so for meronyms. we note that albert does not accessibly encode any of the relations, resulting in random guesses. this could be because albert is trained using tenfold fewer parameters to produce much smaller embeddings, and might have less room for this type of information. since al-bert is comparable in performance to, e.g., bert on many data sets and other metrics, this needs further investigation to see to what extent these relations are present in the data sets. the complexity of the probe could also be the culprit, as an embedding with lower dimensionality poses a more difficult task for a probe with limited capabilities of separating intertwined concepts. these results do not mirror those of @xcite , which indicates that the relations studied here could receive more attention in future evaluations of language embeddings. roberta seems to generally outper-form the other transformers, especially on the hyponyms, taking into account that not all results are statistically significant. hypo-/hypernym relations usually follows a tree hierarchy, with hypernyms directed towards the root. this gives a decreasing number of hypernyms, for example, fish has six hypernyms but 39 hyponyms in wordnet, and it is likely that less common words will be chosen as a positive example for hyponyms. weighting the words according to frequency could show different results, but filtering words based on the data the models are trained on is counterproductive to the purpose of these probes. roberta is better able to capture synonyms, which could be an effect of the much larger dataset used in training compared to the other bert-models leading to more of the less common examples of hyponyms being seen more. one hypothesis on why gpt-2 also shows poor performance is that wikipedia is removed from the training data. the proposition is that many wikipedia articles explicitly outlines hyponym relations, e.g. in "the cat is a [domestic species of small carnivorous] mammal" foot_0 . summarising the results, the fact remains that according to our probes no model covers the relations reliably. if this observation is confirmed by further experiments, it supports the case for a combination of neural networks, traditional relational knowledge bases, and inference engines. with this architecture, established facts could be retrieved from the knowledge base and complemented by less certain facts deduced by the network to cover up for missing information without causing inconsistencies. the results also indicate that a significant threshold should be applied for transferring relational knowledge derived from an embedding to a knowledge base, if this should be done at all, to avoid large error propagation. this is especially important if the "facts" in the knowledge base are considered to be absolute truths rather than tentative findings. in conclusion, the reliability of the probe could improve with evaluation sets from relations found in knowledge bases, and a correlational study between probing accuracy and downstream nlp tasks could further support the usefulness of studying these relations.