VER : Unifying Verbalizing Entities and Relations (2023)

Abstract: Entities and relationships between entities are vital in the real world. Essentially, we understand the world by understanding entities and relations. For instance, to understand a field, e.g., computer science, we need to understand the relevant concepts, e.g., machine learning, and the relationships between concepts, e.g., machine learning and artificial intelligence. To understand a person, we should first know who he/she is and how he/she is related to others. To understand entities and relations, humans may refer to natural language descriptions. For instance, when learning a new scientific term, people usually start by reading its definition in dictionaries or encyclopedias. To know the relationship between two entities, humans tend to create a sentence to connect them. In this paper, we propose VER: a unified model for Verbalizing Entities and Relations. Specifically, we attempt to build a system that takes any entity or entity set as input and generates a sentence to represent entities and relations. Extensive experiments demonstrate that our model can generate high-quality sentences describing entities and entity relationships and facilitate various tasks on entities and relations, including definition modeling, relation modeling, and generative commonsense reasoning.

## introduction what is x? what is the relationship between x and y? we come up with these questions almost every day. when we come across a new term, e.g., twin prime, we usually refer to its definition to understand it, i.e., "a twin prime is a prime number that is either 2 less or 2 more than another prime number". to express the understanding about relationship between entities (e.g., carbon dioxide and water), we create a sentence to represent their relationship: "carbon dioxide is soluble in water". basically, we understand entities and relations by "verbalizing" them. verbalizing entities and relations also tests our knowledge about entities and relations. literally, by verbalizing entities and relations, we understand the world. similarly, do machines have the ability to verbalize entities and relations? can machines learn about entities and relations from verbalizing them? the answer is "yes". recent studies show that by giving the surface name of an entity (and its context), models (after training) can generate coherent sentences to represent it, i.e., definition modeling @xcite @xcite @xcite , and by giving the surface names of a pair of entities, machines can generate coherent sentences describing their relationships, i.e., (open) relation modeling @xcite . however, verbalizing entities requires understanding relationships between entities, and verbalizing entity relationships requires understanding entities themselves, while existing works deal with entity and relation verbalization separately, ignoring the connections between them. besides, recent works @xcite @xcite have shown that language models pre-trained with self-supervised objectives can equip the model with a significant amount of knowledge @xcite and achieve substantial gains after fine-tuning on a specific task. can we continually pre-train the models with pretraining objectives on entities and relations to enhance their ability on verbalizing entities and relations? in this way, the model can be easier and better adapted to specific tasks on entities and relations and even be used without additional training. therefore, we aim to solve entity and relation verbalization in a unified form and pre-train a model for entity and relation understanding. essentially, definition modeling and relation modeling can be unified as an "entity(s) → sentence" task, i.e., given a set of entities, generating a sentence describing the entities and their relationships. when the size of the set is 1, it is equivalent to definition modeling, and when the size of the set is 2, it is equivalent to relation modeling. by defining the task in this form, we can even model more complex relationships among entities since entity relationships can go beyond pairwise @xcite , named hyper-relation modeling, e.g., {carbon dioxide, water, arbonic acid} → "carbon dioxide reacts with water to produce arbonic acid". based on this, we propose ver (pronunciation: /v3:/): a unified model for verbalizing entities and relations (figure 1 ). specifically, we pre-train models by forming a self-supervised text reconstruction task: given an entity or a set of entities, reconstruct the original sentences (e.g., a definition or a relation description) containing them in the training corpus. in this way, the models acquire knowledge about entities and relations and learn to connect entities to a meaningful coherent sentence. since implicit knowledge stored in the parameters of the models may not be sufficient for generating meaningful sentences for the target entities, we also study ver in the retrievalenhanced setting @xcite @xcite , named rever, i.e., retrival-enhanced ver, by pre-training models augmented with sentences contained the target entities in the pre-training corpus. throughout the remainder of this paper, we use "[re]ver" to denote both ver and rever. experiments on six datasets demonstrate the superiority of our model in verbalizing entities and relations. especially in low-resource settings, [re]ver can achieve significantly better results than baselines on definition modeling, relation modeling, and generative commonsense reasoning. in addition, the performance of [re]ver without additional training is impressive, making itself a potential knowledge source of entities and relations, which may benefit tasks on entities and relations such as entity typing @xcite , relation extraction @xcite , and knowledge graph completion @xcite . ## background and formulations definition modeling. definition modeling aims to generate definitions of entities, which can be formulated as a conditioned sequence generation task. for instance, given twin prime, the expected output is the definition of twin prime: "a twin prime is a prime number that is either 2 less or 2 more than another prime number". we follow the standard sequence-to-sequence formulation in @xcite ; @xcite : given entity x, the probability of the generated definition @xmath0, . . . , s m ] is computed auto-regressively: hyper-relation modeling (unified form). previous works mainly focus on verbalizing single entities or entity pairs. however, in the real world, relationships between entities can be more complex -beyond pairwise, named "hyper" relationships @xcite @xcite . for example, "carbon dioxide reacts with water to produce carbonic acid". here, there are tuplewise relationships among carbon dioxide, water, and carbonic acid. verbalization of hyper relationships was initially investigated in common-gen @xcite but was limited to commonsense concepts, and their outputs are simple short sentences describing everyday scenarios containing the given concepts. we attempt to model and verbalize more general complex "hyper" relationships among entities and find a unified framework to combine single entities (1 entity), pairwise relationships (2 entities), and "hyper" relationships (≥3 entities). combining with definition modeling and relation modeling, we adopt the following unified form: ## [re]ver : verbalizing entities and relations to verbalize an entity, we are likely to connect it to other entities, which requires knowledge about entity relationships. to understand entity relationships, we need to know about entities first. based on this, we attempt to verbalize entities and relations in a unified form and propose [re]ver : a unified model for verbalizing entities and relations. we first create a large dataset with the formulation in eq. ( 3 ), and pretrain a model on this dataset, which equips the model with a significant amount of knowledge about entities and relations and enables the model to generate coherent and meaningful sentences connecting the entities. the model can be further finetuned on specific datasets, e.g., definition modeling, relation modeling, and generative commonsense reasoning, to achieve better performance on specific tasks. ## experiments in this section, we evaluate [re]ver on definition modeling, relation modeling, and hyper-relation modeling in three settings: 1) fine-tune the model on the full task-specific training data; 2) fine-tune the model in low-resource settings; 3) directly use the model without fine-tuning. the main goal of the experiments is to verify whether the continual pre-training step can enhance models' ability on verbalizing entities and relations with/without external knowledge.