Mind the Labels: Describing Relations in Knowledge Graphs With Pretrained Models (2023)

Abstract: Pretrained language models (PLMs) for data-to-text (D2T) generation can use human-readable data labels such as column headings, keys, or relation names to generalize to out-of-domain examples. However, the models are well-known in producing semantically inaccurate outputs if these labels are ambiguous or incomplete, which is often the case in D2T datasets. In this paper, we expose this issue on the task of descibing a relation between two entities. For our experiments, we collect a novel dataset for verbalizing a diverse set of 1,522 unique relations from three large-scale knowledge graphs (Wikidata, DBPedia, YAGO). We find that although PLMs for D2T generation expectedly fail on unclear cases, models trained with a large variety of relation labels are surprisingly robust in verbalizing novel, unseen relations. We argue that using data with a diverse set of clear and meaningful labels is key to training D2T generation systems capable of generalizing to novel domains.

## introduction d2t generation systems need to accurately capture the semantics of relations between values in the data. however, the data labels such as relation names @xcite , table headings @xcite , or meaning representation keys @xcite ) may provide only superficial or-if the labels are abbreviations, such as in the rotowire dataset @xcite -no usable hints about the data semantics. learning how to properly describe the data is thus a challenge for d2t systems, typically requiring in-domain training data of sufficient quality and quantity @xcite . plms such as bart @xcite or t5 @xcite can quickly adapt to new domains and exhibit robustness to out-of-domain inputs. however, the plms for d2t generation are still limited by the expressivity of the data labels. consider figure 1 (a): the model can use its representation of "godparent" to understand there is a "is-a-godparent-of" relation between the entities, but it has to infer (or guess) who is the godparent of whom. even in the less ambiguous cases (b) and (c), the model still has to correctly capture the intended semantics of the relation (e.g. "occupant" meaning "home team"). in this paper, we investigate to what extent plms are able to use arbitrary labels describing relations between entities. a suitable testing ground is the task of describing (i.e., verbalizing) individual triples in a knowledge graph (kg), which can be considered a trivial case of graph-to-text (g2t) generation @xcite . in this task, there is a wide range of lexical choices for the relation label (see current human-annotated datasets for d2t generation contain only a small number of relations and rarely contain any unseen relations in the test x is available on y. x runs on y. country x was born in y. x is located in y. parent x is the parent of y. y is the parent of x. chembl x has an id y in the chembl database. using the rel2text dataset, we evalute the ability of plms to verbalize relations which were not present in the training set. we consider both models finetuned on other relations in our dataset and models finetuned on datasets from a related domain. we also experiment with scenarios involving few-shot finetuning, training on masked labels, and extending the labels with descriptions ( §4, 5). we find that the plms are quite robust in verbalizing a diverse set of relations based on their label (achieving ~90% of overall entailment probability). we show that semantically unfaithful model outputs are often caused by incomplete, ambiguous, or noisy input data. somewhat suprisingly, we also show that longer relation descriptions do not provide substantial improvements over using short labels. however, even for data using short relation labels, the model trained on verbalizing relations can achieve results comparable to verbalizing relations using manual templates in two downstream tasks ( §6). the contributions of our work are as follows: ## related work earlier works in natural language generation from kgs exploited domain-specific ontologies for rulebased systems @xcite @xcite . with the advance of plms, structure-aware modeling and task-specific pretraining has lead to remarkable progress on d2t benchmarks such as webnlg @xcite , agenda @xcite , or e2e @xcite , indicated via both automatic and human evaluation metrics @xcite @xcite . @xcite used a multi-step approach with semantic filtering and distant supervision for verbalizing the english wikidata, covering the wide range of relations present in the kg. the authors use the approach to generate the kelm corpus -an automatically cleaned corpus with synthetic (model-generated) verbalizations of wikidata triplesets. we use the kelm corpus to investigate how models trained on large-scale synthetic data differ from models trained on a small-scale human-annotated dataset (cf. §4). other works have tried incorporating descriptions of data labels in the model inputs. in one of the experiments, @xcite use descriptions of relations from wikidata instead of their labels for relation embeddings, concluding that it results in worse performance on downstream tasks. conversely, @xcite and @xcite improve the performance of their systems by including schema descriptions on the input for the dialogue state tracking and dialogue response generation systems. there has also been a research interest in verbalizing single triples as a stand-alone preprocessing step for nlp tasks. the step has been shown to improve the generalization ability of downstream models for data-to-text generation @xcite @xcite and response generation in dialogue systems @xcite . this step can also serve for making the input similar to the format used during pretraining, e.g. for natural language inference (nli) models @xcite @xcite . the above works employ a variety of methods to convert triples to text, ranging from simple templates and rule-based systems to prompting large plms. however, none of these works investigate how plms behave when presented with novel relations. in a work concurrent to ours, @xcite investigate the aspects of generalization performance of plms on the dart dataset foot_0 @xcite . they compare prompt-based and finetuning-based approaches to d2t generation, focusing on the ability of models to perform on difficult examples. in contrast, we focus on finetuned encoder-decoder models, which were shown in @xcite to be more efficient for d2t generation, and we evaluate the models on clean and manually curated data. ## data for our experiments, we need data with diverse labels and their human verbalizations. in this section, we describe how we gather rdf foot_1 triples from large-scale kgs ( §3.1) and collect their verbalization through crowdsourcing ( §3.2, 3.3). ## analysis and evaluation in our analysis, we are interested in the following research questions: to answer these questions, we divide our rel2text dataset into a training and test splits (see §4.1 for details). we then use the rel2text test set to evaluate a finetuned bart model @xcite , a pretrained encoder-decoder transformer, which is used as a backbone of many recent data-to-text models @xcite @xcite . 8 to answer rq1, we compare the performance of bart finetuned on the rel2text training set with bart finetuned on two qualitatively different d2t datasets -webnlg and kelm. using rel2text only, we then prepare various setups for answering rq2, rq3, and rq4 (details in §4.2). we analyze the outputs of the models both automatically ( §4.3) and manually ( §4.4). ## discussion based on our experiments, we can conclude that plms are indeed able to verbalize novel relations (rq1). however, there is a caveat: if the relation label is ambiguous or when the cues about the relation are limited (rq3), the model will resolve to guessing and the semantic accuracy of the output descriptions may drop. a takeaway for datasets which do not follow standard naming conventions, such as the rotowire dataset with basketball summaries @xcite which uses abbreviations for column headers (e.g. fg3a stands for "the number of shots the player attempted beyond the arc"), is that rephrasing the labels to natural language may increase the robustness of d2t systems applied on these datasets. we have focused on finetuned plms, which in our case require at least several hundreds of examples to produce satisfactory results (rq2). however, recent research suggests prompting large plms capable of in-context learning (brown et al., 2020) may help to bring down the number of examples required close to zero @xcite @xcite @xcite . in this case, the models do not have a possibility to learn the correct verbalizations from the training data, which will probably make using clear and unambiguous labels even more important: an issue to investigate in future work. we showed that improving the outputs using longer relation descriptions is not straightforward (rq4). to achieve more notable improvements, it may be necessary to combine a more detailed specification regarding the relation direction, type, acceptable values, etc., together with a model able to reason about this specification. a promising research in this direction could be using chain-ofthought reasoning, so far applied for tasks such as open-domain question answering or solving math word problems @xcite @xcite . the remaining open question is how to handle input data with noisy labels. we suggest that detecting these cases and fixing them prior to generation (for example with knowledge-augmented systems or a human-in-the-loop setup) could help to improve the robustness of d2t systems in real-world scenarios. ## conclusion we analyzed the abilities of plms to verbalize unseen relations in kgs using the relation labels. based on our findings, we believe that having expressive and unambiguous data labels is a good starting point for adapting d2t systems to new domains. for the analysis, we collected the rel2text dataset, which can help to replace the hand-crafted templates on downstream tasks. future work may investigate how our findings generalize to prompt-based few-shot or zero-shot d2t generation with large plms.