C ycle GT : Unsupervised Graph-to-Text and Text-to-Graph Generation via Cycle Training (2020)

Abstract: Two important tasks at the intersection of knowledge graphs and natural language processing are graph-to-text (G2T) and text-tograph (T2G) conversion. Due to the difficulty and high cost of data collection, the supervised data available in the two fields are usually on the magnitude of tens of thousands, for example, 18K in the WebNLG 2017 dataset after preprocessing, which is far fewer than the millions of data for other tasks such as machine translation. Consequently, deep learning models for G2T and T2G suffer largely from scarce training data. We present CycleGT, an unsupervised training method that can bootstrap from fully non-parallel graph and text data, and iteratively back translate between the two forms. Experiments on WebNLG datasets show that our unsupervised model trained on the same number of data achieves performance on par with several fully supervised models. Further experiments on the non-parallel GenWiki dataset verify that our method performs the best among unsupervised baselines. This validates our framework as an effective approach to overcome the data scarcity problem in the fields of G2T and T2G.

## introduction knowledge graphs are a popular form of knowledge representation and central to many critical natural language processing (nlp) applications. one of the most important tasks, graph-to-text (g2t), aims to produce descriptive text that verbalizes the graphical data. for example, the knowledge graph triplet "(allen forest, genre, hip hop), @xcite )" can be verbalized as "allen forest, a hip hop musician, was born in 1981." allen forest, a hip hop musician, was born in the year 1981. the music genre hip hop gets its origins from disco and funk music, and it is also which drum and bass is derived from. this has wide real-world applications, for instance, when a digital assistant needs to translate some structured information (e.g., the properties of the restaurant) to the human user. another important task, text-to-graph (t2g), is to extract structures in the form of knowledge graphs from the text, so that all entities become nodes, and the relationships among entities form edges. this can serve many downstream tasks, such as information retrieval and reasoning. the two tasks can be seen as a dual problem, as shown in figure 1 . however, most previous work has treated g2t and t2g as two separate supervised learning problems, for which the data annotation is very expensive. therefore, both fields face the challenge of scarce parallel data. all current datasets are of a much smaller size than what is required to train the model to human-level performance. for example, the benchmark dataset webnlg 2017 only has 18k text-graph pairs for training (after pre-processing by @xcite ), which is several magnitudes fewer than the millions of data for neural machine translation (nmt) systems, such as the 4.4m paired sentences in the wmt14 dataset for nmt @xcite . as a result, most previous g2t and t2g models, which have to be trained on small datasets, display limited performance on both tasks @xcite @xcite . to circumvent the limitations of scarce supervised data, we formulate both tasks in a cycle training framework, and also in the unsupervised manner with fully non-parallel datasets (as shown in figure 1 ). the technical difficulty lies in the different modality of text and graphs, which can be intractable in a joint learning setting. we contribute an effective learning framework, cyclegt, which is iteratively trained with two cycle losses. we first validate cyclegt on widely used webnlg datasets, both webnlg 2017 @xcite and webnlg+ 2020 @xcite which is a more challenging extension of the 2017 dataset. here cyclegt achieves performance that is comparable to many supervised baselines. additionally, we also evaluate our model on the newly released genwiki dataset @xcite with 1.3m non-parallel text and knowledge graphs. our model is the best out of all unsupervised baselines, improving by +11.47 bleu over the best baseline on genwiki fine data and +6.26 bleu on genwiki full . the surprisingly high performance of cyclegt indicates that it is an effective approach to address the data scarcity problem in the fields of both g2t and t2g. consequently, cyclegt can help pave the way for scalable, unsupervised learning, and benefit future research in both fields. ## formulation and notations for unsupervised graph-to-text and text-to-graph generation, we assume two non-parallel datasets: the constraint is that the graphs and text contain the same distribution of latent content z, but are different forms of surface realizations. their marginal log-likelihood can be formulated with the shared latent content z: our goal is to train two models in an unsupervised manner: g2t that generates text based on the graph, and t2g that produces the graph based on text. in this vein, there is an implicit assumption that a one-to-one mapping exists, at least approximately, between text and graphs. denote the parameters of g2t as θ, and parameters of t2g as ϕ. now suppose there exists an unseen ground truth distribution d pair (e.g., as in the test set), where each paired text and graph (t, g) share the same content. then the ideal objective is to maximize the log-likelihood of θ and ϕ over the text and graph pairs (t, g) ∼ d pair : the major challenge of our task is that the ground truth distribution d pair is not available as training data. instead, only the text corpus d t and graph dataset d g are observed separately without alignment. so our solution is to approximate the learning objective in eq. (3) through non-parallel text and graph data. the resulting method we derive, namely cyclegt, will be introduced next. note that we have also recently adopted the underlying cyclegt architecture in the context of developing an alternative conditional variational autoencoder model for handling non-bijective mappings @xcite . ## cyclegt development in this section, we will first introduce the g2t and t2g components in section 3.1 and 3.2, respectively, and then discuss the iterative back translation training strategy of cyclegt in section 3.3. ## related work we will first give an overview of the fields of t2g and t2g separately, and then introduce the cycle learning. data-to-text generation as a classic problem in text generation @xcite , data-to-text generation aims to automatically produce text from structured data @xcite . due to the expensive collection, all the data-to-text datasets are very small, such as the 5k air travel dataset (ratnaparkhi, 2000), 22k weathergov @xcite , 7k robocup @xcite , and 5k rotowire on basketball games @xcite . as for the methodology, tra-ditional approaches adopt a pipeline system @xcite of content planning, sentence planning, and surface realization. recent advances in neural networks give birth to end-toend systems @xcite @xcite that does not use explicit planning but directly an encoder-decoder architecture @xcite . relation extraction relation extraction (re) is the core problem in text-to-graph conversion, as its former step, entity recognition, have off-theshelf tools with good performance @xcite @xcite @xcite . re aims to classify the relation of entities given a shared textual context. conventional approaches hand-crafted lexical and syntactic features @xcite . with the recent advancement of deep neural networks, many models based on cnn @xcite @xcite , rnn @xcite @xcite , and bert @xcite achieve high performance in many datasets. however, constrained by the small datasets of only several hundred or several thousand data points @xcite @xcite , recent research shifts to distant supervision than model innovation @xcite @xcite . cycle training the concept of leveraging the transitivity of two functions inverse to each other has been widely observed on a variety of tasks. in computer vision, the forward-backward consistency has been used since last decade @xcite , and training on cycle consistency has recently been extensively applied on image style transfer @xcite . in language, back translation @xcite @xcite and dual learning @xcite have also been an active area of research centered on umt. similar techniques can also be seen on tasks such as language style transfer @xcite . finally, cycle training has recently been applied in the context of text-graph conversion @xcite ; we compare against this approach in