Unsupervised Contextualized Document Representation (2021)

Abstract: Several NLP tasks need the effective repre-sentation of text documents.Arora et al.,2017 demonstrate that simple weighted aver-aging of word vectors frequently outperformsneural models. SCDV (Mekala et al., 2017)further extends this from sentences to docu-ments by employing soft and sparse cluster-ing over pre-computed word vectors. How-ever, both techniques ignore the polysemyand contextual character of words. In thispaper, we address this issue by proposingSCDV+BERT(ctxd), a simple and effective un-supervised representation that combines con-textualized BERT (Devlin et al., 2019) basedword embedding for word sense disambigua-tion with SCDV soft clustering approach. Weshow that our embeddings outperform origi-nal SCDV, pre-train BERT, and several otherbaselines on many classification datasets. Wealso demonstrate our embeddings effective-ness on other tasks, such as concept match-ing and sentence similarity. In addition,we show that SCDV+BERT(ctxd) outperformsfine-tune BERT and different embedding ap-proaches in scenarios with limited data andonly few shots examples.

## introduction the semantics of a document is highly dependent on the constituent words, and words can have different meaning in different contexts. approaches such as @xcite @xcite go beyond words to capture the semantic meaning of sentences but are restricted to perceiving the meaning of a single sentence, thus reducing its expressive power. a simple weighted average of the individual word embeddings doesn't account for word ordering and long-distance semantic relationships. @xcite proposed clustering-based technique with tf-idf weighting to form sparse composite document vector, thus extending the simple averaging approach beyond a single sentence. recently, @xcite introduced scdv-ms, which highlights how multisense word embedding resolves cluster disambiguation, which improves embedding performance, further enhancing scdv. @xcite additionally demonstrates that a sparsity constraint in clustering can be advantageous. modern contextualized representations such as bert @xcite can capture the exact meaning of a word based on the surrounding context, which can automatically disambiguate the meaning of words in a corpus-based on its interpretations. previous approaches for document representation, as discussed above, ignore these contextualized representation benefits. therefore, in this work, we propose a new approach (scdv+bert(ctxd), which leverages a combination of clustering techniques for word-sense disambiguation and combines it with the expressibility of scdv partition averaging method for creating better document representation. we contextualized the corpus by using the word sense disambiguation power contextualized pre-train bert embedding. contextual embedding from pre-train bert disambiguate the occurrence of the same word with different context (meaning). we then use the scdv partition averaging algorithm to convert this contextualized corpus into document embeddings using contextual pre-train bert word embeddings as static word vectors. we show that our unsupervised embeddings scdv+bert(ctxd) outperform existing techniques on several classification datasets in supervised, semi-supervised, and few-shot settings. we also demonstrate performance improvement in nonclassification tasks such as concept matching and sentence similarity using our representation. the datasets along with the associated scripts, can be located at https://github.com/vgupta123/ contextualize_scdv . ## proposed algorithm our algorithm is similar to scdv algorithm, but uses word sense disambiguated contextualize word vectors obtained from pre-trained bert embedding as static word vectors. below are details of the primary steps involved. corpus contextualization (wsd): our objective here is to disambiguate different occurrences of a word in a corpus document with its separate interpretation. for example, the word 'bank' in "he went to 'bank' for withdrawing money" and "i went to a river 'bank' during summer holiday" has different meanings based on it's used context. given a word w and all its occurrence in the corpus documents as w 1 , w 2 , . . . , w n , for each w i , we find its contextualized embedding representation b w i using transformer based pre-trained language model such as bert @xcite . taking inspiration from mekala et al., 2020, we treat the word disambiguation problem as a local clustering problem of contextualizing word vectors. for our case, we also cluster the contextualize word embedding b w i , obtained via the pre-train bert model from the last step. we use the efficient kmeans algorithm to cluster all b w i into k clusters, where k represents the total possible interpretations of word w in all the documents of the corpus. 1 we use the cosine distance between the contextualize word representation as our clustering metric. 2 the value of k i.e., cluster numbers, is decided using a similarity threshold (τ ), which is a hyperparameter and a dataset property, usually set using the heuristic described in @xcite . let c w 1 , c w 2 , . . . , c w k be the k cluster centroids obtained after the k-means clustering for the word w. we treat these k centroid representation as our 1 k-means is computationally more efficient than other clustering approaches. any other suitable clustering algorithm also works. document representation (scdv): similar to the scdv, we use gaussian mixture model to cluster our final sense-disambiguated word vectors (obtain from earlier step) into k partition of the words in the corpus. 3 for each contextualized word vector w i of word @xmath0, we created k different word-cluster vectors of d dimensions (wc v io ) by weighting word's embedding with sparse probability distribution for the o th cluster, i.e., p(c o |w i ). then, we concatenate the k word-cluster vectors (wc v io ) and weight them with their inverse document frequency (idf(w i )) to form a contextualized word-topic vector (w tv i ). the dimension of wordtopic vector (w tv i ) is k × d. to obtain the final document embedding d v dn , we computed the average of the contextualized word-topic vectors w tv i from the words and it's context as appearing in that document d n . for more details on scdv, refer to algorithm 1 in @xcite . figure 1 shows flowchart for our document embedding approach (scdv+bert(cxtd)). ## experimental results we perform experiments with multi-class classification with data restriction settings, concept matching problem and sentence similarity task. ## comparison with related works the closest work to our paper is scdv by mekala et al., 2017 that extend bowv by @xcite using an overlapping clustering technique and direct idf weighting of word vectors. recently gupta et al., 2020a extended scdv to scdv-ms via utilising the multi-sense embeddings obtained via using adagram @xcite over word-vectors @xcite . our idea is similar to scdv-ms; however, it utilises pre-train bert contextual embedding as word embedding, a.k.a a more robust sense disambiguated aware embedding @xcite . thus, our approach (scdv + bert(ctxd)) uses contextual word vectors as the foundational block for document representation. ## conclusion and future work in this paper, we enhance sparse document representation (scdv) with pre-trained bert contextualization and propose scdv+bert(ctxd). we showed that one could effectively utilize the bert contextualization for word-sense disambiguation. our approach outperforms other unsupervised approaches in the full data regime. our approach is also very successful for low data regime, outperforming the standard model with roughly half the training data required and few shot settings, where fine-tuning of model fails.