Learning to Organize a Bag of Words into Sentences with Neural Networks: An Empirical Study (2021)

Abstract: Sequential information, a.k.a., orders, is assumed to be essential for processing a sequence with recurrent neural network or convolutional neural network based encoders. However, is it possible to encode natural languages without orders? Given a bag of words from a disordered sentence, humans may still be able to understand what those words mean by reordering or reconstructing them. Inspired by such an intuition, in this paper, we perform a study to investigate how “order” information takes effects in natural language learning. By running comprehensive comparisons, we quantitatively compare the ability of several representative neural models to organize sentences from a bag of words under three typical scenarios, and summarize some empirical findings and challenges, which can shed light on future research on this line of work.

## introduction though significant progress has been made, it is still mysterious how humans are able to understand, organize, and generate natural languages. in the field of natural language processing, many efforts have been made to enhance computational models. recently, recurrent neural networks @xcite and encoder-decoder architectures @xcite with long short-term memory @xcite and gated recurrent unit @xcite have demonstrated state-of-the-art performance in sequence modeling and generation. nowadays, the encoder-decoder architectures have become a widely used approach for sequenceto-sequence tasks such as machine translation @xcite , text summarization @xcite and dialogue generation @xcite . such models generally encode the input sequence into a vector representation using recurrent neural networks (rnns) @xcite ., * corresponding authors: dongyan zhao and rui yan. take a look at an example of word rearrange quizzes for language learners foot_0 . given a bag of words from a disordered sentence {the dog james talking sat next to himself to .}, most people can still read with little effort, though disagreement might exist on subtle details as to whether it is the man or the dog that is seated. inspired by this, it is interesting to explore how and to what extent we can encode natural languages without considering order information. from a computational perspective, we ask: can we construct an algorithm that is capable of reading a bag of words as robustly as humans do? our task is to predict the original sentence given a bag of words without orders extracted from a random sentence. this orderless setting is important to characterize the human instinct for understanding languages. the answer to this question also provides insights into many important practical problems: in abstractive text summarization, the summary can be generated according to a bag of extracted key words @xcite ; in statistical machine translation, we need to reorder the words or phrases in the target language to get a natural and fluent ## related work pooling is a basic approach to encode sets (or bags), and has been widely used for many tasks, such as 3d shape recognition @xcite , fewshot image classification @xcite . besides, several studies have explored the capability of attention mechanisms in modeling sets (or bags). vinyals et al. (2015) proposed to encode a set with multi-hop attention operations. ilse et al. (2018) proposed to use attention-based weighted sum-pooling for multiple instance learning. similarly, yang et al. (2020) proposed an attentionbased algorithm to aggregate a deep feature set for multi-view 3d reconstruction. as a new approach to modeling a text sequence, self-attention has been successfully used in many nlp tasks, such as machine translation @xcite ), text summarization @xcite and machine reading comprehension @xcite . however, most studies about selfattention focus on sequence modeling, which ignores the positional invariance of the attention mechanism itself. in perticular, @xcite utilized self-attention to model interactions between the objects in a video, and employed pooling to obtain aggregated features. on this basis of the transformer architecture, @xcite presented an set transformer designed to model interactions among elements in the input set. without considering missing words or noisy words, our task devolves into word ordering problem, which is a fundamental task in natural language generation. previous, researchers usually employed n-gram based language models (de @xcite , syntacticbased language models @xcite or combined models @xcite to solve this problem. more recently, @xcite proposed a bag-to-sequence model, where the decoder rnn directly attended to the word embeddings. however, all these methods aim at finding the best permutation of a bag of words based on language models, and do not consider how to encode a bag of words. ## problem formulation given a bag of words @xmath0, x 2 , • • • , x m } which consists of m tokens, our model will generate a sentence @xmath1, y 2 , • • • , y n }, where n is the length of target sentence. in the normal scenario, the words of x come from a disordered sentence and are the same as y . while in other two scenarios, the condition no longer holds. to be specific, x contains some noisy words that do not appear in y for noise scenario, and x lacks some words that should appear in generated sequence for the missing scenario. we can model this using the conditional probability p (y |x) and decompose it with the chain rule. in our scenario, the source input is a bag of words or even with noisy or missing words and the output is a sentence. ## bag-to-sequence models in this paper, we employ encoder-decoder frameworks to address the bag-to-sentence problem. particularly, the encoder is responsible for learning an order-invariant context representation for the input bag, and the decoder produces the target sentence conditioned on a bag of input words. ## experiments we evaluate our method quantitatively and qualitatively on a large dataset for three typical sentence organization scenarios described in ## conclusions in this paper, we present an empirical study to investigate the ability of neural models to organize sentences from a bag of words under three typical scenarios. we conclude our discussion with the following findings: