D ual TKB : A D ual L earning B ridge between T ext and K nowledge B ase (2020)

Abstract: In this work, we present a dual learning approach for unsupervised text to path and path to text transfers in Commonsense Knowledge Bases (KBs). We investigate the impact of weak supervision by creating a weakly supervised dataset and show that even a slight amount of supervision can significantly improve the model performance and enable better-quality transfers. We examine different model architectures, and evaluation metrics, proposing a novel Commonsense KB completion metric tailored for generative models. Extensive experimental results show that the proposed method compares very favorably to the existing baselines. This approach is a viable step towards a more advanced system for automatic KB construction/expansion and the reverse operation of KB conversion to coherent textual descriptions.

## introduction the automatic construction of knowledge bases (kbs) from text and the reverse operation of sentence generation from kbs are dual tasks that are both active research topics. the first task of automatic kb construction remains a significant challenge due to the difficulty of detecting parts of text representing meaningful facts and summarizing them in a systematic form. a simpler sub-task of kb completion, i.e., extending or filling-in missing nodes or edges, has also attracted the attention of the research community. for both tasks, the system needs to generate new or complete existing graph entities coherently, possibly matching to the already existing graph structure. the dual task of decoding the information from kb back to text is a valuable functionality. this enables knowledge transfer from potentially large complex in this work, we address the problem of kb construction/completion and the reverse task of kb decoding, but aim at a simpler objective: transferring a single sentence to a path, and generating text from a single kb path as its dual task. in terms of data, our focus will be on commonsense kbs, derived from sets of commonsense facts expressed in natural language sentences @xcite @xcite . they are represented as graphs where each edge is expressed as a tuple (e h , r, e t ) with head and tail nodes e h and e t composed of free-form text, connected with a relationship operator r; see conceptnet from @xcite or atomic in @xcite for recent and commonly used examples of commonsense kbs. we observe that to train such transfer model, an additional challenge comes from the lack of datasets with parallel text and kb facts, i.e., where text sentences and kbs edges are explicitly paired/labeled from one to another. however, there exist many datasets for each individual transfer domain. therefore, successful approaches transferring text to kb and kb to text must be able to operate in unsupervised or (at best) weaklysupervised settings. we address this challenge by proposing a model trained under dual learning of translation/transfer from text to kb and from kb to text, we name dualtkb. this is similar in philosophy to dual learning in neural machine translation @xcite , or unsupervised style transfer @xcite @xcite . we design our model to be trained in completely unsupervised settings. however, we observed that even a slight supervision significantly boosts model performance and enables better-quality transfers. therefore, we also describe a simple heuristic methodology to create weakly-supervised datasets given a text corpus and a commonsense kb. we must emphasize that our proposed dual learning method is not limited to commonsense kbs and can generalize to other domains/types of kbs such as biomedical kbs. commonsense kbs, and particularly conceptnet, are good starting points due to the nature of their composition. since conceptnet was partly extracted from free-form text originating from the open mind common sense (omcs) list of commonsense fact sentences, its nodes are often composed of parts of sentences from omcs. this allowed us to first explore whether the proposed method worked at all before evaluating a semi-supervised approach by creating a weak supervision from a mapping between conceptnet triples and the original omcs sentences. while kbs are often dense with short named entity descriptions for nodes, many nodes for commonsense kbs are parts of sentences, making them inherently sparse which impacts their performance as empirically studied by @xcite . the evaluation of this type of transfer models is a challenge in itself. for this purpose, we selected a set of metrics to examine different facets of the system using our created weakly-supervised dataset. since repurposing generative models for conventional kb completion evaluation is difficult @xcite , we propose a new commonsense kb completion evaluation task for generative models. it is close in spirit to conventional kb completion, but comes with its own set of challenges. moreover, we employ the graph edit distance (ged) to examine the quality of the generated graph as a whole. for text generation, we rely on traditional nlp metrics such as bleu and rouge. following is a list of highlights of our paper contributions: (1) propose a dual learning bridge between text and commonsense kb. implement approach as unsupervised text-to-path and path-totext transfers; (2) construct a weakly-supervised dataset, and explore weak-supervision training. (3) define a novel commonsense kb completion metric tailored for generative models. (4) investigate successfully multiple model architectures. finally, in fig. 1 and fig. 2 we present a few examples generated by our proposed model. fig. 1 is a text to kb translation. each sentence below the graph is independently transferred to a path, consisting of one edge tuple (e h , r, e t ). the whole path tuple is generated at once, with e h , r, and e t taking a free-form not restricted to any predefined sets of entities or relations. this contrasts with many existing works operating on a limited discrete set of already-defined edges in a dense conventional kb. once all the sentences are transferred, we observe that this set of generated edges forms a connected structure, implicitly merging some nodes to form a prototype of a knowledge graph (kg). fig. 2 shows the transfer from kb to text. each path in the graph is converted to a sentence. there is overall diversity in the generated sentences styles. moreover, the samples show that the generation process is more sophisticated than just a trivial path flattening (i.e., merging text from all edge parts followed by minimal edits). therefore, the proposed approach can eventually become a part of a more sophisticated system converting graphs to a coherent textual story and vice versa. additional examples are presented in appendix a.4. ## dual learning in this work we propose to use a dual learning approach to build a model performing two distinct but complementary generative tasks described in fig. 3 , and using notations from tab. 1: task 1 (text path): given a sentence x a , generate a path x ab with well-formed entities and relation, that can either belong to an already constructed kb, or extend it in a factually meaningful way. this conditional generation is framed as a translation task referred to as t ab where x @xmath0. task 2 (path text): given a kb path x b , generate a descriptive sentence x a , coherently merging entities and relation from the path. this conditional generation is a translation task referred to as t ba , where x @xmath1. from the above two tasks follows the definition of back-translation tasks t bab (path text path) and t aba (text path text). reconstruction tasks t aa and t bb are trivially defined as generating the same text/path from itself. in an unsupervised setting, where sentences x a and paths x b are not paired, the reconstruction tasks t aa , t bb and back-translation tasks t aba , t bab are the back-translation tasks define the so-called cycle/consistency losses, which implicitly control the quality of the first transfers (t ab , t ba ), by checking the reconstruction after the second transfers (t aba , t bab ). by themselves, these cycle losses are effective in training good transfer model. however, as we show in section 5, even a small weak supervision (by pairing some x a to x b ) can significantly improve our model performance. as shown in fig. 3 , our model uses an encoderdecoder architecture with a common encoder and two specialized decoders (a, generating sentences x a , and b, generating paths x b ). the reason for a single encoder is to force the embeddings of path and text to lie in the same space, encoding a fact regardless of modality. it then becomes the job of each specific decoder to recover either one of the modalities. note that the decoders share the same architecture, but not their parameters. ## model given a dataset x of paths and sentences, let x k t be its k-th random sample of type t, for @xmath2, . . . , n and type @xmath3, b}, where x a is a sentence and x b is a path in the kg. given the input x k t to the model, the corresponding generated output will be denoted as x k tt , where tt is the transfer direction, i.e., @xmath4, ab, bb, ba}. for example, given path the back-translations of a sample from a type t back to itself, since @xmath5. given input sentence a model with perfect generation and back-translation would yield x k @xmath6. note that to reduce clutter, we drop the dataset index k from the notations. ## experimental setup in this section, we provide details about model architecture, training procedure, and data processing. ## results and discussions in this section we discuss in detail our dataset construction, evaluate the proposed models, and compare performances to existing baselines. ## conclusion in this paper we proposed to use a dual learning bridge between text and commonsense kb. in this approach, a generative model is trained to transfer a sentence to a path and back. assembling paths together results in a graph showing the presence of inherent structure, while generated sentences exhibit coherent and relevant semantics. for evaluation, we proposed a novel commonsense kb completion task tailored to generative models. although our model is designed to work in unsupervised settings, we investigated the impact of weak-supervision by creating a weakly-supervised dataset and showed that even a slight amount of weak-supervision improves significantly model performance. the current work is one step towards the overarching goal of kb construction/completion and generation of human-readable text from kbs. future work can focus on expanding the capabilities to generating whole paragraphs of text from graphs in kb, as well as converting large parts of text into coherent graph structures. for all models. for results in tab. 6, 4 distinct seeds were used to show performance based on the average of the 4 individual model performances (standard deviation is also provided). mrr and hits metrics are defined in details in @xcite -code associated with the paper provides implementations in python. we reimplemented them for better integration in our own codebase. bleu metrics were used from nltk implementations. when running multiple supervision ratio for weak-supervision, hyper-parameters were all fixed to the values provided above, only the supervision ratio was changed. a.2 aa, aba: reconstruction and back-translation tasks tab. 5 show results for our models gru-gru, gru-bert, and trans-trans for bleu2, bleu3, rouge, and f1 bert score for both aa and aba tasks. aa is an easier task with excellent while aba is more difficult. gru-gru and trans-trans excel at both aa and aba.