Are the Best Multilingual Document Embeddings simply Based on Sentence Embeddings? (2023)

Abstract: Dense vector representations for textual data are crucial in modern NLP. Word embeddings and sentence embeddings estimated from raw texts are key in achieving state-of-the-art resultsin various tasks requiring semantic understanding. However, obtaining embeddings at the document level is challenging due to computational requirements and lack of appropriate data. Instead, most approaches fall back on computing document embeddings based on sentence representations. Although there exist architectures and models to encode documents fully, they are in general limited to English and few other high-resourced languages. In this work, we provide a systematic comparison of methods to produce document-level representations from sentences based on LASER, LaBSE, and Sentence BERT pre-trained multilingual models. We compare input token number truncation, sentence averaging as well as some simple windowing and in some cases new augmented and learnable approaches, on 3 multi- and cross-lingual tasks in 8 languages belonging to 3 different language families. Our task-based extrinsic evaluations show that, independently of the language, a clever combination of sentence embeddings is usually better than encoding the full document as a single unit, even when this is possible. We demonstrate that while a simple sentence average results in a strong baseline for classification tasks, more complex combinations are necessary for semantic tasks

## introduction semantic representations, especially embeddings, are crucial for natural language processing (nlp). in fact, the field has exploded since the success of dense word embeddings @xcite . for some tasks like finding semantic or syntactic relations among words, high quality word embeddings are enough. other tasks, like question classification or paraphrase detection, benefit from sentence embeddings. finally, lots of tasks deal with documents: summarisation, document classification, question answering, etc. document representations are difficult to be learned, especially multilingually, given the amount of available training data and the length of each training instance. for these reasons, document embeddings usually resort to sentence embeddings. since some of the state-of-the-art techniques for language modelling and sentence embeddings are based on selfattention architectures such as bert @xcite , and self-attention scales quadratically with the input length, one cannot afford arbitrarily long inputs. training is usually constrained to input fragments up to 512 tokens (subunits). this limit goes well beyond an average sentence length and can cover several paragraphs. however, full documents can be significantly longer. the average length of a wikipedia article in english is 647 words (not subunits) for example, foot_1 and the average for two of the tasks that we consider in this work, document alignment and icd code classification, is around 800 words, with documents up to 40k words. in order to be able to process long inputs, more efficient architectures such as linformer @xcite , big bird @xcite or longformer @xcite implement sparse attention mechanisms that scale linearly instead of quadratically. these architectures accept at least 4096 input tokens. with this length, one can embed most wikipedia articles, news articles, medical records, etc. these architectures are available as pre-trained models in english 3 and can be finetuned for nlp tasks such as document classification, question answering or summarisation. how-ever, multilingual or non-english versions are rare. for most languages, it is not just a matter of training a model from scratch, but the amount of documents is just not enough to train high quality models. laser @xcite , sentence bert @xcite and labse @xcite are representative and state-of-the-art models which largely adapt language models to be used as task-independent sentence representations. these models are available as pre-trained models and, contrary to the long sequence models introduced before, they are multilingual. laser, which is not transformer-based, allows longer inputs. these observations explain why the two main approaches to obtain multilingual (or non-english) document embeddings are simply (i) truncating the input to 512 tokens and feeding it into a sentencelevel encoder or (ii) splitting the document in shorter fragments and then combine their embeddings. there are few works that do a systematic comparison among methods. @xcite perform a systematic study for document classification in english and found that the most sophisticated models such as longformer do not always improve on a baseline that truncates the input to fit it into a fine-tuned bert. the results mostly depend on how the information is distributed along a document and therefore varies from dataset to dataset. in this work we explore multilingual documentlevel embeddings in three tasks in detail: document alignment, a bilingual semantic task; icd code (multi-label) classification in 2 languages; and cross-lingual document classification in 8 languages. we compare input token number truncation, sentence averaging as well as some simple windowing and in some cases new augmented and learnable approaches. our results show that a simple sentence average is a very strong baseline, even better than considering the whole document as a single unit, but that positional information is needed when the distribution of information across a document is not uniform. ## related work word embeddings have been exceptionally successful in many nlp applications @xcite @xcite . subsequent works developed methods to learn continuous vector representations for longer sequences such as sentences or even documents. skip-thought embeddings @xcite train an encoderdecoder architecture to predict surrounding sentences. @xcite showed that the task on which sentence representations are learnt significantly impacts their quality. infersent @xcite , a siamese bilstm network with max pooling, and universal sentence encoder @xcite , a transformer-based network, are trained over the snli dataset which is suitable for learning semantic representations @xcite . these methods primarily work on a single language but as multilingual representations have attracted more interest, sentence-level embeddings have been extended to obtain a wider language coverage. @xcite (laser) learn joint multilingual sentence representations for 93 languages based on a single bilstm encoder with a shared bpe vocabulary trained on publicly available parallel corpora. however, this architecture was shown to underperform in highresource scenarios @xcite . laser is especially interesting for our work as, being lstmbased, it does not have the 512-length constraint. li and mak (2020) introduce t-laser, which is a version of laser that uses a transformer encoder in place of the original bidirectional lstm. however, this model was tested only on the multilingual document classification (mldoc) corpus @xcite , which does not have significantly long documents. similarly, @xcite (sbert in the following) extended a transformer-encoder architecture, bert, by using a siamese network with cosine similarity for contrastive learning in order to derive semantically meaningful sentence representations. more recently, @xcite (labse) explored crosslingual sentence embeddings with bert by introducing a pre-trained multilingual language model component and show that on several benchmarks, their method outperforms many state-of-the-art embeddings such as laser. while sentence-level representations have been widely explored in literature, document-level representations are less well-explored. the earliest approaches in learning document-level vector representations included an extension of the word2vec algorithm named doc2vec @xcite with two variants proposed, a bagof-words and a skip-gram based model. however, while these methods worked well at the word-level, the document-level counterpart led to issues in scaling due to large vocabulary sizes @xcite . due to these limitations, further works have attempted to improve the computational bottlenecks involved with training on long sequences such as documents. linformer @xcite is a transformer-based architecture with linear complexity due to a sparse self-attention mechanism making it significantly more memory-and timeefficient in comparison with the original transformer @xcite . works such as big bird @xcite and longformer @xcite introduced a sparse attention mechanism and localised global attention respectively. bigbird is able to handle sequences of up to 4,096 tokens and longformer scales linearly with the sequence length, with experiments on sequences of length upto 32,256. to the best of our knowledge, to date not much has been done to extend them beyond english. @xcite and @xcite made available chinese and spanish longformer models, respectively, while @xcite trained a multilingual version starting from a roberta checkpoint and not from scratch. we use longformer as a comparison system in our experiments but we do not consider the multilingual model given that multilinguality was achieved by finetuning on question answering data and we do not explore this task. ## sentence embeddings we use three multilingual sentence-level embedding models that cover different languages, architectures and learning objectives: laser @xcite uses max-pooling over the output of a stacked bilstm-encoder. the encoder is extracted from an encoder-decoder machine translation setup trained on parallel corpora over 93 languages. since it is not based on transformers but on lstms, the maximum number of input tokens can in principle be arbitrary and is set to 12,000. @xcite train a multilingual bert-like model with a masked lm and translation lm objective functions. a dual-encoder transformer is initialised with the model and fine-tuned on a translation ranking task. the final model covers 109 languages. the maximum number of input tokens is 512. ## document embeddings we divide our approaches to build document embeddings into three families: in (i) document excerpts, we feed token sequences as they are directly into laser, labse and sbert to obtain a document-level representation, in (ii) sentence weighting schemes, we divide documents into sentences represented using base sentence embeddings and then explore different combination and weight strategies to obtain document embeddings, in (iii) windowing approaches, we study different distributions to learn document-level positional and semantic information. ## results and discussion thompson and koehn (2020) empirically obtained the best trade-off between accuracy and inference time when using pca-reduced sentence embeddings of 128 dimensions in the bilingual document alignment task. we performed equivalent experiments with 128 and 256 dimensions for selected configurations in the three tasks and confirmed the trend. as we obtained no major gains in using more dimensions, we report all the results for the three tasks with 128-dimensional sentence embeddings. we report confidence intervals at 95% confidence level using bootstrap resampling with 1000 samples for document alignment, 500 samples for icd code classification and 1000 samples for document classification. bilingual document alignment quality ranges from 65% to 96% recall depending on the document embedding method. the other dimension of the study, the particulars of sentence embeddings, is less important to the recall. laser, labse and sbert achieve similar results. as we are working with french and english documents, both languages being highresource, all base sentence embeddings are highquality and therefore they do not impact the final model strongly in a consistent way. multi-label icd code classification shows the same trend with respect to different sentence embeddings as above for german and spanish, with a slight preference towards labse embeddings. cross-lingual document classification data allows us to test the embedding methods on 8 languages ( accuracies in our experiments show that specific base sentence embedding models (laser, labse, sbert) do not impact the performance of the document-level embeddings much. we observe similar performance amongst them across all experiments. however, it is to be noted that we experiment with languages that while being morphologically distinct, are well resourced and covered by the three base sentence-embedding models. it would be interesting to explore how models behave when embeddings have a lower quality. for this, one would need to create evaluation datasets at the document level for low-resourced languages but this is out of the scope of this work. we observed that a simple sentence average is a very strong pooling strategy, specially for classification tasks. positional and contextual information is more important than semantic information for the final performance as exemplified by the fact that pert-based weightings perform better than tf-idf's in all the tasks. when combining both, positional and semantic information, we do not observe statistically significant improvements with respect to only including positional information. for the classification tasks which include a learnable layer, we extend tk-pert to att-pert (and the semantic counterparts) and include global trainable attention on the positional information. this global attention is beneficial in all the cases. the type of document is also relevant to chose the best method. long documents might have the most crucial information stored in different parts. for instance, webpages have a majority of their information in the first half of the document as we observed in the document alignment task. in this case, the positional information significantly outperforms any model that does not take it into account.