Can LM s Store and Retrieve 1-to-N Relational Knowledge? (2023)

Abstract: It has been suggested that pretrained language models can be viewed as knowledge bases. One of the prerequisites for using language models as knowledge bases is how accurately they can store and retrieve world knowledge. It is already revealed that language models can store much 1-to-1 relational knowledge, such as ”country and its capital,” with high memorization accuracy. On the other hand, world knowledge includes not only 1-to-1 but also 1-to-N relational knowledge, such as ”parent and children.”However, it is not clear how accurately language models can handle 1-to-N relational knowledge. To investigate language models’ abilities toward 1-to-N relational knowledge, we start by designing the problem settings. Specifically, we organize the character of 1-to-N relational knowledge and define two essential skills: (i) memorizing multiple objects individually and (ii) retrieving multiple stored objects without excesses or deficiencies at once. We inspect LMs’ ability to handle 1-to-N relational knowledge on the controlled synthesized data. As a result, we report that it is possible to memorize multiple objects with high accuracy, but generalizing the retrieval ability (expressly, enumeration) is challenging.

## introduction as a result of their pretraining on large amounts of text, language models (lms) store certain world knowledge facts, such as "paris is the capital of france", in their parameters and can retrieve that knowledge when given a suitable prompt. since the ability to store and retrieve knowledge is also a key functionality of knowledge bases (kbs; @xcite , prior work has proposed to view language models as knowledge bases @xcite . quantitative evaluation of world knowledge in lms has focused on 1-to-1 relational knowledge involving two entities, such as a country and its capital @xcite @xcite , ## experiments 6.1 element-valued supervision in the first experiment, we investigated the ability to memorize multiple objects using element-valued supervision. here, we tested whether the lms could correctly store n objects associated with a single subject. specifically, as shown in figure 2 , the learning process of having one object generated for each input sentence, such as "{subject} has a child @xmath0." or "{subject} directed a film @xmath1." was performed for all objects. thus, the learning setup is such that there are as many target sentences as objects for each input sentence. ## conclusion we addressed handling 1-to-n relational knowledge by a generative approach using the sequenceto-sequence model. since little work has been done on 1-to-n relational knowledge in previous studies, we started by organizing the properties of 1-to-n relational knowledge and setting up the capabilities considered necessary for lms based on these properties. specifically, we defined two essential capabilities: "memory of discretely appearing multiple objects" and "enumeration of objects based on memory." then, we developed training schemes based on these perspectives. we used element-valued supervision and beam search for the former to memorize and evaluate multiple objects. we found that nearly 90% of the objects could be memorized, although we observed a tendency for memory omissions to occur as the number of objects increased. however, we also confirmed that it is challenging to achieve 100% perfect memory. for the latter, we attempted to generalize "enuthe results showed that learning more data improved the generalization performance for acquiring enumeration ability. however, we also observed the lm's behavior, which aligns with human intuition: the more objects increase, the more difficult it becomes to enumerate all of them correctly. notably, the generalization performance for 1-to-2 relational knowledge was only about 50% for the test set, and for 1-to-4 relational knowledge, only about 10% generalization performance at most. for our next steps, we are considering the following approach. the training setup of the current element-valued supervision is characterized by multiple target sentences for one input sentence, which is incompatible with the model's learning algorithm. therefore, we would like to test a memorizing method using ordinal numerals such as first and second to distinguish each template for n objects. we would also like to investigate this memorization method's effect on the generalization performance of enumeration. as for enumeration, which has been difficult to generalize, we would like to examine effective means of improving performance for a small number of objects. specifically, we are considering adjusting the hyperparameters for text generation and verifying whether errors in enumerating will be reduced. after that, we would like to explore learning methods to enumerate n objects without needing hyperparameters adjustment in stages. introducing our 1-to-n problem setting into the lms-as-kbs paradigm opens up many more intriguing challenges. while we investigated this setting under a controlled condition with a uniform frequency of object appearance, the frequency of each of the n objects in a corpus is likely to vary in reality. furthermore, there may be multiple paraphrases expressing the same relation. for example, in our study, we only considered the phrase "{subject} has a child named {object}." but there are other phrases such as "{subject}'s child is {object}." or "{object} is a daughter of {subject}." as a primary avenue for future research, we will explore whether lms can handle 1-to-n relational knowledge effectively under these more complex conditions.