Role of Context in Unsupervised Sentence Representation Learning: the Case of Dialog Act Modeling (2023)

Abstract: Unsupervised learning of word representations involves capturing the contextual information surrounding word occurrences, which can be grounded in the observation that word form is largely disconnected from word meaning. While there are fewer reasons to believe that the same holds for sentences, learning through context has been carried over to learning representations of word sequences. However, this work pays minimal to no attention to the role of context in inferring sentence representations. In this article, we present a dialog act tag probing task designed to explicitly compare content-, and context-oriented sentence representations inferred on utterances of telephone conversations (SwDA). Our results suggest that there is no clear benefit of context-based sentence representations over content-based sentence representations. However, there is a very clear benefit of increasing the dimensionality of the sentence vectors in nearly all approaches.

## introduction unsupervised methods for constructing word representations vary widely in their implementation @xcite @xcite , but share the core property that they try to capture the contextual information surrounding word occurrences. this can be grounded in the observation that word form is largely disconnected from word meaning @xcite and that, therefore, trying to extract meaning from form directly would be pointless. however, there are fewer reasons to believe that representing the meaning of a fully composed sentence requires the description of its context in the way that a word does. still, much effort has been devoted to developing and evaluating context-oriented methods for sentence representation @xcite @xcite . interestingly, the theoretical justification for the approach -roughly paraphrased-is that it works well for words @xcite . in the present article, we examine whether a contextual signal is of any significant benefit to general-purpose sentence representation. to accommodate this goal, we infer two sets of utterance representations from telephone conversation transcripts, one by a model trained to encode the transcripts' content, and one by a model trained to encode the transcripts context in a skip-thoughtlike fashion. we evaluate the learned representations with a linear probe classifying the dialog act tags. below, we discuss these choices more extensively. the reason for the linear probing technique is to avoid any further representation learning in the downstream task training, such that the resulting accuracy numbers solely reflect the unsupervised representations. this technique is commonly used to assess to what extent sentence representations exhibit certain linguistic properties, such as part-ofspeech or syntax @xcite @xcite . we chose the switchboard dialog act corpus (swda) @xcite corpus for the following properties: (1) its size allows for inferring representations with both traditional, statistical learning methods and modern neural network based methods; (2) at the same time, the corpus is fairly narrow-domain, enabling small models to converge to meaningful representations foot_0 ; (3) it contains utterance labels that we consider suitable for a probing task with respect to contextuality: while a dialog act tag unquestionably depends on the utterance contents, the surrounding utterances likely provide a good signal about its label too. see example utterances from the corpus in using context to enhance dialog act predictions is not a novel idea @xcite @xcite . to ascertain the usefulness of context in this task, we checked for the reduction in entropy of a tag in a "unigram" setting vs. conditioned on a previous tag, a "bigram" setting. the results were h(currenttag) = 2.1bit and h(currenttag|previoustag) = 1.86bit, meaning that such a context-enhanced baseline would, at least, increase the chance-level accuracy. for the content-oriented representation variants, the representations of utterances were inferred via classic topic modeling methods, word vector averaging, or an lstm-based auto-encoder (see section 3 for details). the resulting sentence vectors were then used as input for a skip-thought-like training regime, where the goal was to learn their context-oriented version by projecting them such that they are predictive of surrounding utterances. additionally, we trained a version where the utterance encoding and projection are trained jointly, with an lstm-based encoder of identical proportions with the one used for auto-encoding. ## related work in the literature review, we summarize some of the most important work in sentence representation and relate it to the issues outlined in previous paragraphs. before the deep learning era of nlp, textprocessing methods relied mainly on lexical features of sentences and were invariant to word order. using word frequency transformations, e.g., tf-idf @xcite , has proven effective to base the representations on more semantically laden words. techniques such as singular value decomposition (svd) @xcite , or topic modeling @xcite @xcite were used to transform the sparse word-count matrices into dense vectors. with the subsequent success of simple neural networks such as word2vec @xcite or glove @xcite at representing words and phrases, a seemingly naive method of averaging these word vectors was shown to produce useful sentence/paragraph/document representations @xcite . with further success of neural networks in text processing, the research on the topic grew markedly. among the unsupervised methods, recursive auto-encoders (raes) @xcite were trained to reconstruct sentences via recursively encoding nodes in a dependency parse tree and performed well at a paraphrasing task @xcite . tree-lstms @xcite later succeeded the raes in the same recursive set up but using an lstm encoder @xcite , and evaluated on predicting semantic relatedness between two sentences @xcite and classifying sentiment @xcite . the current state-ofthe-art unsupervised, general-purpose sentence representation systems are based on pre-trained transformer language models @xcite and fine-tuned via contrastive learning (gutmann and hyvärinen, 2012; chen et al., 2020), i.e., pulling representations of alternative sentence formulations closer together and those of unrelated, random sentence pairs apart @xcite . the success of skip-gram-based word representations inspired researchers to transfer the idea to sentence representation. skip-thought @xcite is a large, lstm-based @xcite , sentence encoder that was trained to produce representations predictive of surrounding sentences. quick-thought (logeswaran and lee, 2018) is a similar system that differs in that it casts the problem as contrastive learning @xcite . the universal sentence encoder (use) is a transformer-based @xcite system introduced by @xcite that uses the skip-thought as one of the tasks in a multi-task learning set-up (including some supervised tasks). the task of deciding whether two sentences are consecutive or not (nsp task) was also used in training of bert @xcite . the latest dialog processing sys-tems, built on top of transformer language models @xcite , typically learn to predict contextual utterrances within the contrastive learning framework and achieve superior performance on dialog tasks compared to universal representations @xcite @xcite . while sentence contextuality is used in all of the above-mentioned studies, there is little to no focus on assessing the necessity for using context to derive sentence representations. an ablation of the nsp task is presented by @xcite , suggesting a slight performance benefit over a version where the task is omitted. however, @xcite only evaluate the fine-tuning pipeline, while the raw bert representations were shown to perform poorly overall @xcite . further, results by @xcite are contradictory and do not credit a performance improvement to the nsp task. lastly, @xcite additionally compare the representations resulting from dropout augmentations @xcite to those of consecutive utterrances and found a significant benefit in favor of the latter augmentations. their results are perplexing, however, since the dropout version they implemented performed even worse than that of simcse @xcite , which also uses dropout augmentation, and should in fact perform worse, not being tailored for dialog. using the natural language inference (nli) task has been effective in several general sentence representation learning systems @xcite @xcite . however, along with machine translation models, these systems fall outside the category of unsupervised sentence representations. ## method we trained several variants of content-based representations, their context-based counterparts, and an end-to-end, skip-thought like system, trained to encode and predict sentences. we evaluated the following variants of contentbased sentence representation 2 : @xcite 2 entries marked with ⋆ use the gensim implementation. we also varied the final dimension of the sentence vector, namely @xmath4, 12, 25}, either by setting the number_of_topics parameter, setting the auto-encoder bottleneck size, or truncating glove vectors. the auto-encoder was based on a 2-layer, bidirectional long short-term memory (lstm) neural network with hidden size of d @xmath5, dropout of @xmath6.1. the variant 5 is one where we randomly assigned a feature vector from an 8dimensional raster to each vocabulary entry (a procedure by which we want to emulate the usage of 1-hot vector encoding, but with dense, real-valued vectors), and created the final sentence representation by averaging corresponding word vectors. to create the context-based representations, we devised a learning regimen for 2 linear layers for each variant. one layer was used to project the sentence content vector to the context space (encode), and the other layer was used to predict the content vectors of surrounding sentences from the context vector (decode). finally, we trained a skip-thought-like model, where each sentence was represented by a sequence of word vectors (obtained via the same procedure as in variant 5), optionally truncated and padded to 20 tokens, processed via an lstm (identical with the auto-encoder in variant 4), and projected such that, conditioned on the resulting vector, a decoder-lstm reconstructs the surrounding sentences well via teacher forcing. in all of the context-based variants, we varied the context size between 1 and 2 sentences, on each side. ## results the classification accuracy results are presented in for lsa, each algorithm achieved the highest score with d @xmath8. there were no big differences and no discerning patterns in the performance of the contextually projected variants. the only case where the training regime of predicting two neighboring sentences on each side scored the best was with the end-to-end, skip-thought like variant, which also achieved the overall highest score. ## discussion all tested models performed well over the majority baseline, including the averaged random word vectors which exceeded the majority baseline by a large margin when coupled with high dimensionality. one reason for this might be that, when averaging word vectors of higher dimension, there is a higher chance that some important lexical features will remain separable from the noise (also applicable to glove). only in case of lsa and auto-encoder, the effect of dimensionality was small. for the latter, using a variational or contrastive learning set-up might allow for better scaling of performance with capacity. based on the patterns observed here, increasing the dimensionality increases performance on the probe, with or without context. we did not find either of these aspects discussed in the mainstream related work @xcite . the question of how far increasing the dimensionality helps performance should be explored in future work. the presented results on the dialog act tag probing task are in line with related literature in that sequence/document representations with word embeddings and classical topic modeling approaches are a tough-to-beat baseline for many neural networks @xcite . our results suggest that transforming existing, content-oriented representations to context-oriented ones provides minimal to no meaningful improvement in representation. an improvement of small, but perhaps meaningful magnitude is one from the best performing auto-encoder to skip-thought variant ( 3%), suggesting that some benefit in this training scheme may be found for sentences, too. however, our experimental design limits us in drawing conclusions about what the skip-thought set-up benefited from specifically: the more general sentence representations may simply be an artifact of the more difficult learning task (predicting neighboring sentences), as opposed to being a result of form disambiguation via context, the way it works with words. future work should employ methodology that separates the contribution of the competing sources of signal more clearly. when compared to the obvious benefits of context on learning word representations, it is striking that the benefits for learning sentence representations appear to be very small at best. this observation is suggestive of a more serious blind-spot in nlp: because context-based representations work well for words, many researchers assume that the same holds for any level of analysis. this is akin to claiming arbitrariness of the sign at any level of language processing. ## conclusion we presented evidence that explicitly examines the differences between content-, and context-oriented unsupervised sentence representations. while results suggest that dimensionality, rather than context, is the determining factor in building good sentence representations, end-to-end approaches based on reconstructing surrounding sentences are a promising way forward.