Discovering Universal Geometry in Embeddings with ICA (2023)

Abstract: This study utilizes Independent Component Analysis (ICA) to unveil a consistent semantic structure within embeddings of words or images. Our approach extracts independent semantic components from the embeddings of a pre-trained model by leveraging anisotropic information that remains after the whitening process in Principal Component Analysis (PCA). We demonstrate that each embedding can be expressed as a composition of a few intrinsic interpretable axes and that these semantic axes remain consistent across different languages, algorithms, and modalities. The discovery of a universal semantic structure in the geometric patterns of embeddings enhances our understanding of the representations in embeddings.

## introduction embeddings play a fundamental role in representing meaning. however, there are still many aspects of embeddings that are not fully understood. for instance, issues such as the dimensionality of embeddings, their interpretability, and the universal properties shared by embeddings trained with different algorithms or in different modalities pose important challenges in practical applications. discussions and research have explored the topics of low-dimensionality and interpretability of embeddings @xcite . proposals have been made for learning and post-processing methods that incorporate constraints, aiming to achieve sparse embeddings or acquire semantic axes. additionally, research has focused on aligning embeddings trained in different languages through various transformations. however, in contrast to this ongoing trend, our specific focus lies on the intrinsic independence present within embeddings. in this research, we post-process embeddings using independent component analysis (ica), providing a new perspective on these issues @xcite . there are limited studies that have applied ica to a set of word embeddings, with only a few exceptions @xcite @xcite . there has also been a study that applied ica to wordcontext matrices instead of distributed representations @xcite . although it has received less attention in the past, using ica alpca. each embedding is normalized to have a norm of 1 for visualization purposes. the components from the first 100 axes of 500 embeddings are displayed in the upper panels, and the first five axes are magnified in the lower panels. for each axis of english embeddings, the top 5 words were chosen based on their component values, and their translated words were used for other languages. correlation coefficients between transformed axes were utilized to establish axis correspondences and carry out axis permutations; english served as the reference language to align with the other languages when matching the axes, and the aligned axes were rearranged in descending order according to the average correlation coefficient, as detailed in section 4 and appendix c. lows us to extract independent semantic components from a set of word embeddings. by leveraging these components as the embedding axes, we anticipate that each word can be represented as a composition of intrinsic (inherent in the original embeddings) and interpretable (sparse and consistent) axes. our experiment suggests that the number of dimensions needed to represent each word is considerably less than the actual dimensions of the embeddings, enhancing the interpretability. fig. 1 shows an example of independent semantic components that are extracted by ica. each axis has its own meaning, and a word is represented as a combination of a few axes. furthermore, figs. 2a and 3a show that the semantic axes found by ica are almost common across different languages when we applied ica individually to the embeddings of each language. this result is not limited to language differences but also applies when the embedding algorithms or modalities (i.e., word or image) are different. principal component analysis (pca) has traditionally been used to identify significant axes in terms of variance, but it falls short in comparison to ica; the patterns are less clear for pca in figs. 2b and 3b . embeddings are known to be anisotropic @xcite , and their isotropy can be greatly improved by post-processes such as centering the mean, removing the top principal components @xcite , standardization @xcite , or whitening @xcite , which can also lead to improved performance in downstream tasks. while the whitening obtained by pca provides isotropic embeddings regarding the mean and covariance of components, notably, ica has succeeded in discovering distinctive axes by focusing on the anisotropic information left in the third and higher-order moments of whitened embeddings. ## ica: revealing the semantic structure in the geometric patterns of embeddings we analyzed word embeddings using ica. it was discovered that they possess inherent interpretability and sparsity. ica can unveil these properties of embeddings. ## universality across languages this section examines the results of conducting ica on word embeddings, each trained individually from different language corpora. interestingly, the meanings of the axes discovered by ica appear to be the same across all languages. for a detailed description of the experiment, refer to appendix c. ## universality in algorithm and modality we expand the analysis from the previous section to two additional settings. the first setting involves comparing fasttext and bert, while the second setting involves comparing multiple image models and fasttext simultaneously. ## quantitative evaluation we quantitatively evaluated the interpretability (section 6.1) and low-dimensionality (section 6.2) of ica-transformed embeddings comparing with other whitening methods (pca, zca) as well as a well-known rotation method (varimax). these baseline methods are described in appendix e.1. in the monolingual experiments, we used 300dimensional word embeddings trained using the sgns model on the text8 corpus, as outlined in appendix b. furthermore, we assessed the crosslingual performance (section 6.3) of pca and icatransformed embeddings, along with two other supervised baseline methods. ## conclusion we have clarified the universal semantic structure in the geometric patterns of embeddings using ica by leveraging anisotropic distributions remaining in the whitened embeddings. we have verified that the axes defined by ica are interpretable and that embeddings can be effectively represented in lowdimensionality using a few of these components. furthermore, we have discovered that the meanings of these axes are nearly universal across different languages, algorithms, and modalities. our findings are supported not only by visual inspection of the embeddings but also by quantitative evaluation, which confirms the interpretability, lowdimensionality, and universality of the semantic structure. the results of this study provide new insights for pursuing the interpretability of models. specifically, it can lead to the creation of interpretable models and the compression of models.