Discrete Cosine Transform as Universal Sentence Encoder (2021)

Abstract: Modern sentence encoders are used to generate dense vector representations that capture the underlying linguistic characteristics for a sequence of words, including phrases, sentences, or paragraphs. These kinds of representations are ideal for training a classifier for an end task such as sentiment analysis, question answering and text classification. Different models have been proposed to efficiently generate general purpose sentence representations to be used in pretraining protocols. While averaging is the most commonly used efficient sentence encoder, Discrete Cosine Transform (DCT) was recently proposed as an alternative that captures the underlying syntactic characteristics of a given text without compromising practical efficiency compared to averaging. However, as with most other sentence encoders, the DCT sentence encoder was only evaluated in English. To this end, we utilize DCT encoder to generate universal sentence representation for different languages such as German, French, Spanish and Russian. The experimental results clearly show the superior effectiveness of DCT encoding in which consistent performance improvements are achieved over strong baselines on multiple standardized datasets

## introduction recently, a number of sentence encoding representations have been developed to accommodate the need of sentence-level understanding; some of these models are discussed in @xcite @xcite , yet most of these representations have focused on english only. to generate sentence representations in different languages, the most obvious solution is to train monolingual sentence encoders for each language. however, training a heavily parameterized mono-lingual sentence encoder for every language is inefficient and computationally expensive, let alone the impact on the environment. thus, utilizing a non-parameterized model with ready-to-use word embeddings is an efficient alternative to generate sentence representations in various languages. a number of non-parameterized models have been proposed to derive sentence representations from pre-trained word embeddings @xcite @xcite . however, most of these models, including averaging, disregard structure information, which is an important aspect of any given language. recently, @xcite proposed a structure-sensitive sentence encoder, which utilizes discrete cosine transform (dct) as an efficient alternative to averaging. the authors show that this approach is versatile and scalable because it relies only on word embeddings, which can be easily obtained from large unlabeled data. hence, in principle, this approach can be adapted to different languages. furthermore, having an efficient, readyto-use language-independent sentence encoder can enable knowledge transfer between different languages in cross-lingual settings, empowering the development of efficient and performant nlp models for low-resource languages. in this paper, we empirically investigate the generality of dct representations across languages as both a single language model and a cross-lingual model in order to assess the effectiveness of dct across different languages. ## dct as sentence encoder in signal processing domain dct is used to decompose signal into component frequencies revealing dynamics that make up the signal and transitions within @xcite . recently, dct has been adopted as a way to compress textual information @xcite . a key observation in nlp is that word vectors obey laws of algebra king -man + @xmath0.) queen @xcite . thus, given word embeddings, cast a sentence as a multidimensional signal over time, in which dct is used to summarize the general feature patterns in word sequences and compress them into fixed-length vectors @xcite . mathematically, dct is an invertible function that maps an input sequence of n real numbers to the coefficients of n orthogonal cosine basis functions of increasing frequencies @xcite . the dct components are arranged in order of significance. the first coefficient (c[0]) represents the sum of the input sequence normalized by the square length, which is proportional to the average of the sequence @xcite . the lower-order coefficients represent lower signal frequencies which correspond to the overall patterns in the sequence. for example, dct is used for compression by preserving only the coefficients with large magnitudes. these coefficients can be used to reconstruct the original sequence exactly using the inverse transform @xcite . in nlp, @xcite applied dct at the word level to reduce the dimensionality of the embeddings size, while @xcite applied it along the sentence length as a way to compress each feature in the embedding space independently. in both implementations, the top coefficients are concatenated to generate the final representation for a sentence. as shown in @xcite , applying dct along the features in the embeddings space renders representations that yield better results. also, zhu and de melo (2020) noted that similar to vector averaging the dct model proposed by @xcite yields better overall performance compared to more complex encoders, thus, in this work, we adopt their implementation to extract sentencelevel representations. specifically, given a sentence matrix n × d, a sequence of dct coefficients c @xcite . finally, a fixed-length sentence vector of size kd is generated by concatenating the first @xcite . k dct coefficients, which we refer to as c[0 : k]. foot_0 ## conclusion in this paper, we extended the application of dct encoder to multi-and cross-lingual settings. experimental results across different languages showed that similar to english using dct outperform the vector averaging. we further presented a sentencelevel-based approach for cross-lingual mapping without any additional training parameters. in this context, the dct embedding is used to generate sentence representations, which are then used in the alignment process. moreover, we have shown that incorporating structural information encoded in the lower-order coefficients yields significant performance gains compared to the avg in sentence translation retrieval. s toldova, o lyashevskaya, a bonch-osmolovskaya, and m ionov. 2015. evaluation for morphologically rich language: russian nlp. in proceedings on the international conference on artificial intelligence (icai), page 300. the steering committee of the world congress in computer science, computer . . . . andrew b watson. 1994. image compression using the discrete cosine transform. mathematica journal, 4(1):81. ziyi yang, chenguang zhu, and weizhu chen. 2019. parameter-free sentence embedding via orthogonal basis. in proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (emnlp-ijcnlp), pages 638-648, hong kong, china. association for computational linguistics. biao zhang, deyi xiong, and jinsong su. 2018. accelerating neural transformer via an average attention network. in proceedings of the 56th annual meeting of the association for computational linguistics (volume 1: long papers), pages 1789-1798, melbourne, australia. association for computational linguistics. xunjie zhu and gerard de melo. 2020. sentence analogies: linguistic regularities in sentence embeddings. in proceedings of the 28th international conference on computational linguistics, pages 3389-3400, barcelona, spain (online). international committee on computational linguistics.