Semantic Compression for Word and Sentence Embeddings using Discrete Wavelet Transform (2024)

Abstract: Wavelet transforms, a powerful mathematical tool, have been widely used in different domains, including Signal and Image processing, to unravel intricate patterns, enhance data representation, and extract meaningful features from data. Tangible results from their application suggest that Wavelet transforms can be applied to NLP capturing a variety of linguistic and semantic properties.In this paper, we empirically leverage the application of Discrete Wavelet Transforms (DWT) to word and sentence embeddings. We aim to showcase the capabilities of DWT in analyzing embedding representations at different levels of resolution and compressing them while maintaining their overall quality.We assess the effectiveness of DWT embeddings on semantic similarity tasks to show how DWT can be used to consolidate important semantic information in an embedding vector. We show the efficacy of the proposed paradigm using different embedding models, including large language models, on downstream tasks. Our results show that DWT can reduce the dimensionality of embeddings by 50-93% with almost no change in performance for semantic similarity tasks, while achieving superior accuracy in most downstream tasks. Our findings pave the way for applying DWT to improve NLP applications.

## introduction embedding models have evolved as a crucial part of any nlp application. typically, they transform text, using different numerical analysis methods, into high-dimensional dense vectors that capture semantic and contextual aspects of the text for subsequent use by various tasks. as these models evolved, their complexity, dimensionality, and quality have all increased simultaneously, with a particular emphasis on quality and minimal attention to dimensionality. typically generated embeddings are fixed in size for all tasks and are proportional to the model size, rendering their use in low resource settings a challenge. accordingly, in this paper, we investigate the adaptation of dwt from signal and image processing to the field of nlp for the analysis and compression of word and sentence embeddings. dwt analyzes and reduces the size of the data by identifying redundant and important information in data. we posit that dwt has the potential to generate compressed embeddings capable of retaining contextual information and semantic relationships between words as well as among sentences. our key contributions: 1) introduce dwt as an effective compression method for compressing embeddings with respect to an underlying task; 2) propose a novel approach leveraging dwt for analyzing semantics in word and sentence embeddings; 3) study the efficacy of dwt embeddings in capturing and retaining semantics by applying them to similarity and downstream tasks using various embeddings. ## motivation the use of spectral analysis methods in image and signal processing is driven by their ability to analyze data in the frequency domain, providing insights and revealing hidden patterns and dynamics not detectable in the spatial domain. methods such as the discrete cosine transform (dct) and fourier transform (ft) have been commonly used. however, these methods offer global frequency representations that ignore the location of frequencies in the original domain and lack the ability to perform multi-resolution analysis. dwt effectively addresses these limitations by examining data change over time or position, i.e. frequency and time localization, and consequently captures varied information across different scales (multiresolution analysis). as a result, dwt achieved superior results in other fields. dwt analyzes data based on their variation, filtering low-varying or highly correlated (low-frequency) and highvarying (high-frequency) components while preserving their spatial domain information. figure1 visualizes the result of applying dwt to an image resulting in low-frequency coefficients (approximations) that capture the overall structure of the data, hence used for compression @xcite , while high-frequency coefficients (details) capture abrupt changes and motifs, which provide information about edges and contours, making them useful for edge detection @xcite and noise filtering @xcite . dwt can be further applied, recursively, to any coefficients of the first level to achieve a second level of the transform that contains more approximation and details coefficients. in the realm of nlp, visualizing a word embedding vector, as shown in figure2, we observe a signal-like structure with a few spikes indicating high variation in feature values, alongside features with minimal variation. this pattern suggests that applying spectral analysis to these vectors and numerical representations can reveal additional patterns or insights that may not be immediately apparent. few attempts for applying spectral methods to nlp including dct (almarwani et al., 2019) and higher-order dynamic mode decomposition (hodmd) (kayal and tsatsaronis, 2019) were found successful. however, we are unaware of any application of applying dwt to word and sentence embedding other than our preliminary application of dwt to enhance dct sentence embedding in @xcite . in this work, we posit that dwt can effectively compress embed-ding representations, and analyze them at different scales to understand and capture different linguistic patterns encoded in these embeddings. this analysis allows dwt to concentrate embedding energy with high compression ratios without significant loss of information. dwt allows for more efficient representation, less storage requirements and reduced computational complexity. additionally, dwts are scalable and can be applied to any embedding model and for any task. accordingly, we believe that dwt, as a spectral transform, holds promise for nlp tasks. ## discrete wavelet transforms dwts are mathematical functions that analyze data, f(t), using a window (a function) known as the mother wavelet (mw) ψ(t), also called wavelets. this window moves over the data sequentially to examine the variations in the data with respect to this window (localized in time). within a given analysis window if surrounding data features exhibit minimal variation, they can be grouped together and represented using approximation coefficients (ca). conversely, if certain features vary significantly, they are represented using detail coefficient (cd). to shift over the data, a mw translates and dilates ψ a,b (t) (as in equation ( 1 )) using a shifting factor, b, and scales with a scaling factor, a, to capture different frequency variations at different data segments in time t. the resulting coefficients are equivalent to a pair of sub-band linear (convolutional) filters @xcite : one low-pass and one high-pass. the output of a filter pair is usually down-sampled by 2 so the combined output is of the same size (dimensionality) as the original input. this filtering + downsampling can be cascaded on multiple levels recursively to analyze data at multiple levels of resolution. note that dwt can be compared to convolutional neural networks (cnns) @xcite in that they both use sliding-window filters and downsampling. the difference is that in cnns, the filters are learned from the training data, while in dwt the filters are designed, not learned. there are many families of mws: haar, symmlets, coiflets, and daubechies, to name a few @xcite . as a proof of concept in this paper, and in the interest of space, we will only be using and reporting on a subset of the mws that yield the best results in our experiments. 1 ## evaluation to evaluate the proposed approach, we first investigate its efficacy in capturing and compressing semantics in semantic similarity tasks. we further evaluate their effectiveness in a number of downstream tasks to evaluate their efficacy extrinsically. we consider different embeddings, baselines, experimental setup and tasks as follows. baselines: for every experiment, we use different baselines, in addition to the base embeddings, to explore the capabilities of dwt. our baselines include: (1) base embeddings (in all experiments); experimental setup: for all sentence embeddings experiments, we use the senteval toolkit @xcite for evaluation. for all downstream tasks, we leverage multi-layer perceptron (mlp) classifiers based on the default setup outlined in senteval. foot_1 6.1 semantic similarity evaluation word and sentence similarity tasks have become the de-facto method for semantic evaluation @xcite . semantic similarity involves measuring the degree of relatedness and similarity between pairs of words or sentences compared against human judgments or similarity scores assigned by human annotators. ## conclusion and discussion in this paper, we explored the effectiveness of applying dwt to word and sentence embeddings to selectively reduce embeddings. our experiments illustrate the potential of dwt to enhance space and computational efficiency without decline in performance, reducing embedding size by 50-75%. dwt reveals new insights about embeddings, and exposes hidden patterns and semantic information that were not apparent in the base embedding space. additionally, the generated dwt embeddings postulate that different sets of coefficients capture different semantic aspects of an embedding. we conclude our study outcomes in the following points: correlation and redundancy: while it has not been proven that features within the same embedding are correlated, the comparable performance achieved by dwt suggests a correlation between the dimensions of an embedding. this challenges the previous assumption that these dimensions are uncorrelated. if no correlation existed, dwt would hardly achieve comparable results to the performance of base embeddings. our results also prove that embeddings from complex models, such as pre-trained language models, contain redundant features and hence compressing them with 75% reduction in dimensionality achieves a comparable performance to the base embeddings with no more than 2% degradation. dimensionality reduction techniques: although the general idea of dwt appears to be similar to other dimensionality reduction methods in the context of decomposition and compression, its overall properties and methodology are different @xcite . dwt outperforms other compression techniques such as pca and dct. dwt compresses the data using localized sub-band frequency components capturing both low and high features-variation. dwt stands out as a method that can be applied universally to diverse datasets and models. pca reduces data to a low dimension space, and de-correlates the data, but is much slower because it needs to compute the eigenvec-tors of the original data @xcite . as for dct, despite being a frequency analysis method, it tend to capture important frequencies indiscriminately, lacking localization and discarding other frequencies and accordingly degrading performance. mother wavelets: mw is a component of dwt that controls feature filtering, with each mw corresponding to a unique pair of filters. we did not explore specific mw applications for each task due to space constraints, but limited ourselves to a predetermined number of mw families such that we maintain experimental consistency while empirically investigating the effectiveness of dwt. our experiments indicate that coiflets wavelets generally perform well across tasks. it is important to note that mw filtering is controlled by a scaling factor, which adjusts the level of variation considered in an embedding. for example, a coiflet mw with scale 4 focuses on nearby features and their fine variations, whereas scale 17 emphasizes broader approximations, omitting fine details. since the choice of the best mw varies from base embedding to base embedding, the selection and optimization of mw requires further study beyond this paper's scope. as a result, we leave further exploration of mw details and selection for future research. coefficients selection: much as in image processing, different coefficients tend to capture distinct aspects of the data. therefore, it is essential to select the set of coefficients that are most suitable for the base embedding being used in a given task. some tasks may benefit more from approximations over details, while others may see improved performance with nuanced information, and yet others may use both equally. however, it can be concluded that if minimizing the embedding size is a significant consideration, utilizing only level-1 coefficients (ca or cd) with a 50% reduction in size results in comparable performance to the baselines across nearly all tasks. moreover, approximation coefficients consistently maintain comparable performance across multiple levels of dwt, as demonstrated in the 4-level analysis, with the exception of trec. this consistency suggests that these coefficients retain relevant information about the original embeddings. nevertheless, a detailed study and analysis of coefficient selection fall beyond the scope of this paper and are planned for future work. dwt for compression: dwt offers a powerful method for compressing embeddings by exploit-ing correlations and reducing redundancy. the performance of dwt in compressing embeddings strongly suggests that approximation coefficients can be effectively utilized for compression, akin to their role in image compression, where they typically capture essential features of the data. in the compression process, approximation coefficients not only reduce the embedding dimensionality but also reduce noise while retaining relevant features. by reducing noise, the performance of the model improves as it casts down on the error rate thus improving the accuracy of these results. selecting the appropriate compression level for a given task is a hyperparameter that depends on various factors, including space complexity, resources allocated for a particular task and the trade-off between effectiveness and compression ratio.