Unsupervised Graph-Text Mutual Conversion with a Unified Pretrained Language Model (2023)

Abstract: Graph-to-text (G2T) generation and text-to-graph (T2G) triple extraction are two essential tasks for knowledge graphs. Existing unsupervised approaches become suitable candidates for jointly learning the two tasks due to their avoidance of using graph-text parallel data. However, they adopt multiple complex modules and still require entity information or relation type for training. To this end, we propose INFINITY, a simple yet effective unsupervised method with a unified pretrained language model that does not introduce external annotation tools or additional parallel information. It achieves fully unsupervised graph-text mutual conversion for the first time. Specifically, INFINITY treats both G2T and T2G as a bidirectional sequence generation task by fine-tuning only one pretrained seq2seq model. A novel back-translation-based framework is then designed to generate synthetic parallel data automatically. Besides, we investigate the impact of graph linearization and introduce the structure-aware fine-tuning strategy to alleviate possible performance deterioration via retaining structural information in graph sequences. As a fully unsupervised framework, INFINITY is empirically verified to outperform state-of-the-art baselines for G2T and T2G tasks. Additionally, we also devise a new training setting called cross learning for low-resource unsupervised information extraction.

## introduction graph-to-text (g2t) generation and text-to-graph (t2g) triple extraction are two mutually inverse tasks that are crucial to the domain of knowledge graphs (kgs). g2t verbalizes the structural information in kg with descriptive texts, which has attracted much attention to expand the application scope of kg, such as kg-based dialogue and q&a system @xcite . as a primary task of information extraction, t2g aims to extract triples from text, the typical subtasks @xcite of which include named entity recognition (ner) and relation extraction @xcite . figure 1 illustrates a training pair sample containing part of a knowledge graph and its corresponding text. g2t and t2g have been intensively studied respectively, mainly treated as two kinds of independent problems in a supervised way. due to the success of pretrained language models (plms) @xcite , mainstream supervised methods have achieved considerable performance with fine-tuning or prompt learning paradigm @xcite @xcite . however, these supervised methods require annotated data. inspired by unsupervised machine translation approaches @xcite , recent work attempts to explore low-resource alternatives that avoid the requirement of graph-text pairs with unsupervised joint learning @xcite . as illustrated in figure 2 , unsupervised methods consist of g2t modules and t2g modules with different parameters, which are trained jointly in an iterative manner through the two steps of back-translation: the generation step and training step. the outputs of the generation step for the current modules serve as the supervised training signals for the other modules in the next iteration. such an interactive and coupling process imperceptibly produces a lot of synthetic parallel data that is helpful to low-resource training. in this paper, we are thus motivated to focus on unsupervised learning of both g2t and t2g tasks in a joint framework. as shown in figure 2 , unsupervised models share two major issues in order to be jointly trained. first, recent state-of-the-art models usually simplify the t2g task into relation classification with given entities @xcite . as a result, the text corpus has to seek external information extraction tools for the acquisition of entity annotations. second, existing research branches on either g2t or t2g separately implement the two tasks using different neural modules, i.e., g2t modules and t2g modules, which contain numerous parameters that make it challenging to train and share information with each other @xcite . to tackle the above issues, we design a novel back-translation-based framework called infin-ity that integrates g2t and t2g tasks under the unsupervised setting. note that we name our framework as infinity since the overall architecture of the interaction between g2t and t2g resembles the shape of ∞ (figure 3 ). we first investigate the power of seq2seq-based plms for g2t and t2g and propose to regard graph-text mutual conversion as two sequence generation tasks, where we manage to ensure the simultaneous generation of continuous synthetic pairs of graph-text sequences in a unified plm-based module with the back-translation technique. in this way, infin-ity requires no additional neural networks beyond the plm. considering that linearizing graphs into sequences may cause possible performance deterioration, we equip infinity with structure-aware strategy. specifically, we adopt the reward augmented maximum likelihood @xcite for training losses to retain the order and structural information in the original dataset during the finetuning process. in contrast to prior unsupervised work @xcite , in-finity is entirely bootstrapped without the assistance from manual or automatic annotation tools. we perform extensive experiments on two datasets: webnlg @xcite and gen-wiki @xcite , both of which belong to the very few benchmarks that can evaluate g2t and t2g jointly. the results show the superiority of infinity over existing methods. in addition, we also propose a newly designed training setting called cross learning, which makes it possible to train on large-scale datasets without parallel data. thanks to its simplicity and efficiency, infinity can be quickly deployed on various scenarios for application. this work presents the following contributions: regarding t2g, it aims to extract entities and relations (triples) from texts, which is a basic task in the domain of natural language processing and usually handled as a classification (tagging) problem to label roles for different tokens @xcite . apart from these approaches, there emerge some triplet-generation models. for instance, copyre @xcite uses the idea of copy mechanism @xcite for triple extraction. cpc @xcite designs various positive and negative samples for direct graph sequence generation under a supervised setting. ## method this section presents the proposed method infin-ity. we first define the tasks and notations. then we describe the framework and training details in the following parts. ## experiments this section conducts a series of experiments to evaluate the performance of infinity. we first introduce the datasets and baselines, then we provide the comparison results. further, we implement extensive analytical experiments. at last, we show how cross learning works.