WAE _ RN : Integrating W asserstein Autoencoder and Relational Network for Text Sequence (2020)

Abstract: One challenge in Natural Language Processing (NLP) area is to learn semantic representation in different contexts. Recent works on pre-trained language model have received great attentions and have been proven as an effective technique. In spite of the success of pre-trained language model in many NLP tasks, the learned text representation only contains the correlation among the words in the sentence itself and ignores the implicit relationship between arbitrary tokens in the sequence. To address this problem, we focus on how to make our model effectively learn word representations that contain the relational information between any tokens of text sequences. In this paper, we propose to integrate the relational network(RN) into a Wasserstein autoencoder(WAE). Specifically, WAE and RN are used to better keep the semantic structurse and capture the relational information, respectively. Extensive experiments demonstrate that our proposed model achieves significant improvements over the traditional Seq2Seq baselines.

## introduction sequence problems are common in daily life that involves dna sequencing in bioinformatics, time series prediction in information science, and so on. nlp tasks, such as word segmentation, named entity recognition(ner), machine translation(mt), etc, are actually text sequence problems. for text sequence tasks, it is required to predict or generate target sequences based on the understanding of input source sequence, so it plays a pivotal role in nlp to deeply understand the generic knowledge representation in different context. to learn the features of input sequences, probabilistic graphical models, such as hidden markov models(hmm) and conditional random field (crf), can use manually defined feature functions to transform raw data into features, but the quality of the feature functions directly determines the quality of the data presentation. because deep learning can automatically learn the useful and highly abstract features of the data via artificial neural network(ann), many researchers devoted themselves to using neural networks(nns) to obtain low dimensional distributed representations of input data, especially in language modeling, using autoencoder(ae) @xcite to retain the text sequence semantic information in different context has shown promising results. these language models are pretrained on large-scale corpus and complex models to obtain the data representation which contains global information and has strong generalization ability, then the latent representation can be adapted to several contexts by fine-tuning them on various tasks. however, these models simply make use of word order information or position information and ignore the implicit relationship between arbitrary tokens in the sequence, resulting in learning inadequately hidden feature representations and obtaining only superficial semantic representation. more recently, studies on attention @xcite and selfattention @xcite ) mechanism demonstrate that it can effectively improve the performance of several nlp tasks by exchanging information between sentences. however, it only calculates the contribution between vectors by means of weighted sum without exploring and taking advantage of the implicit structural relationships among tokens. in this work, we propose add relational networks(rn) @xcite to the wasserstein au-toencoder(wae)(kingma and welling, 2014) on the basis of the seq2seq architecture to collect the complex relationship between objects and retain the semantic structure in sentences. specifically, to keep the relational information and structural knowledge we add rn layer to encoder since rn integrates the relational reasoning structure that can constrain the functional form of neural network and capture the core common attributes of relational reasoning. to better capture the complex relationships and preserve the semantic structure we use wae as our encoder because wae maps input sequences into the wasserstein space that allows various other metric spaces to be embedded in it while preserving their original distance measurements. the main contributions of our work can be summarized as follows: ## preliminary since the purpose of the proposed method is to better obtain the semantic representation of text sequences, we will focus on the following two issues. ## experiments in this section, we aim to investigate our model's performance over ner and mt, where ner belongs to the problem of sequence prediction and mt belongs to the problem of sequence generation. we first present our experimental set up, then compare our method to other baseline systems, finally we give some analyses about our method. ## conclusion this paper presents a wae -rn model for text sequence tasks, which aims at learning word representatiosn containing structured knowledge. to be specific, to preserve the semantic structure between objects, we propose use wae as the model's encoder. to capture the core common attributes of relational reasoning, we introduce rn. both of which combine well to learn the generic representation that contains relational information. experimental results on mt and ner tasks demonstrate that the proposed model leads to significant improvements. in the future, we plan to extend the general representation to transfer