Learning distributed sentence vectors with bi-directional 3 D convolutions (2020)

Abstract: We propose to learn distributed sentence representation using text’s visual features as input. Different from the existing methods that render the words or characters of a sentence into images separately, we further fold these images into a 3-dimensional sentence tensor. Then, multiple 3-dimensional convolutions with different lengths (the third dimension) are applied to the sentence tensor, which act as bi-gram, tri-gram, quad-gram, and even five-gram detectors jointly. Similar to the Bi-LSTM, these n-gram detectors learn both forward and backward distributional semantic knowledge from the sentence tensor. That is, the proposed model using bi-directional convolutions to learn text embedding according to the semantic order of words. The feature maps from the two directions are concatenated for final sentence embedding learning. Our model involves only a single-layer of convolution which makes it easy and fast to train. Finally, we evaluate the sentence embeddings on several downstream Natural Language Processing (NLP) tasks, which demonstrate a surprisingly excellent performance of the proposed model.

## introduction mapping documents or sentences to vectors @xcite is the foundation of various natural language processing (nlp) tasks, such as text classification @xcite , paraphrase detection @xcite , natural language inference @xcite , question answering @xcite , etc. the most straightforward approach to sentence representation uses the bag-of-words model that represents a sentence as a bag of its constituent words, disregarding grammar and even the word order but keeping multiplicity. another similar approach is called glove @xcite , which takes the average of word vectors of the constituent words of a sentence. these approaches are typically efficient to train, while they ignore the sequential characteristic of the text. to account for the word order, skip-thought @xcite learns sentence representation in an unsupervised way inspired by the skip-gram. it aims to predict the neighboring sentences or phrases for a given sentence. however, the training process of the skip-thought is very slow, which motivates the fastsent @xcite to speed up the training by representing a sentence as a sum of its constituent word vectors. although fastsent is faster than skip-thought in training, it sacrifices the order of words in a sentence, which is important in language models, such as the n-gram feature. for example, @xcite utilize the bi-gram and even tri-gram to train their embedding model. as discussed above, most existing works of sentence representation require pre-trained word vectors as the input or initialization. sentence representation is taken as a downstream task of word representation. however, when human beings read a sentence or an article, their eyes in fact receive a series of text images which are then passed to the brain for recognition and understanding. hence, a natural way of word representation is to use visual shapes of the words or characters as features directly @xcite @xcite @xcite . for example, @xcite and @xcite take chinese and japanese characters as images and apply a subsequent convolutional autoencoder to take those images as input and then output lowdimensional character embeddings. @xcite extract both the forward and backward n-gram features from the text's pixel embedding. we propose to learn the sentence embeddings using the non-pretrained word images but fold them into sentence images as input. our model utilizes multiple bi-gram, tri-gram, quat-gram even five-gram embeddings of both forward and backward orders of words. current research in nlp tends to use deep and complex models, which make the performance compromise to the model complexity. however, the proposed model has a lightweight structure as shown in figure 1 . in detail, we render words or characters of a sentence into images and then fold them into a 3-dimensional sentence tensor @xmath0, where w ×h is the size of the word or character image and l is the length of the sentence. each slice x @xmath1. furthermore, we propose to fully exploit the language feature (i.e., the word order in a sentence) with two distinctive strategies: (1) extracting the multiple n-gram features with several 3-dimensional convolutional kernels of different sizes (n is the number of words covered by the kernel); (2) learning both the forward and backward semantic information from the sentence with bi-direcitonal convolutions, as shown by figure 1 . we name the proposed model as 3d-convlm (3-dimensional convolutional language model). we choose multiple n to integrate multiple n-gram convolutional kernels. taking the demo in figure 1 as an example, we use bi-gram, trigram, and quad-gram information together. moreover, these n-gram information are constructed from both the normal text order and the reverse order through bi-directional convolutions. a subsequent 1dimensional max-over-time pooling is applied to each channel of this 2-dimensional feature map output by each n-gram model with different channels. after pooling, feature maps from each n-gram model of two directions are concatenated as the output feature of the convolutional layer. finally, three fully connected (fc) layers are used for conducting text embedding learning. the contributions of our work are three-fold: the proposed 3d-convlm extracts and integrates multiple n-gram features during forward and backward convolutions, which further increases the flexibility of the input of text information. we evaluate 3d-convlm on text classification and sentence matching, and study the difference between traditional chinese and simplified chinese under the proposed framework. ## conclusion visual embedding of the text has been studied extensively in recent years. cnns can deliver a competitive performance in comparison with lstm on text data analysis. we propose a bi-directional cnn to learn text embedding according to the semantic order of sentences. in our model, visual signals of each character can be extracted by multiple n-grams in both the normal order and reversed order. we conduct text classification and sentence matching on several datasets to evaluate the performance of our model. within the proposed framework, we also study the difference between the simplified chinese and traditional chinese.