Exploring Intra and Inter-language Consistency in Embeddings with ICA (2024)

Abstract: Word embeddings represent words as multidimensional real vectors, facilitating data analysis and processing, but are often challenging to interpret. Independent Component Analysis (ICA) creates clearer semantic axes by identifying independent key features. Previous research has shown ICA’s potential to reveal universal semantic axes across languages. However, it lacked verification of the consistency of independent components within and across languages. We investigated the consistency of semantic axes in two ways: both within a single language and across multiple languages. We first probed into intra-language consistency, focusing on the reproducibility of axes by performing ICA multiple times and clustering the outcomes. Then, we statistically examined inter-language consistency by verifying those axes’ correspondences using statistical tests. We newly applied statistical methods to establish a robust framework that ensures the reliability and universality of semantic axes.

## introduction word embedding is a technique that represents words from natural languages as multidimensional real vectors in a space (e.g., euclidean space), making it easier to handle them as data. these embeddings create a continuous representation of words and sentences, facilitating data analysis and processing. however, word embeddings are challenging to interpret because the values vary greatly depending on the training data and the dimension of the embedding space @xcite . for example, it is unclear what the embedding exactly means even if we say the embedding of "argentina" is [0.0088871, -0.02218, . . . ]. in order to cope with the interpretability problem, several approaches were suggested, such as principle component analysis (pca) and independent component analysis (ica, hyvärinen et al., 2001). ica gives a more interpretable representation of semantic axes (i.e., components labeled with high-relation words) over pca @xcite . for example, if an independent component scores high on the words "apple", "banana", and "peach", the semantic axis can be interpreted as the concept of fruits and labeled as [apple banana peach]. @xcite demonstrated that independent component analysis (ica) can break down multilingual word embeddings into interpretable axes, suggesting that certain semantic axes may be universal across languages. however, their study had two main limitations. first, it focused solely on calculating the correlation coefficients for the correspondence of semantic axes between languages. second, it lacked verification of the consistency of independent components within and across languages. in this study, consistency refers to the reliability of independent components that appear in different runs within one language (intralanguage consistency) and the accurate correspondence of semantic axes among multiple languages (inter-language consistency). while @xcite and @xcite have made significant progress in uncovering semantic axes within individual languages, the extent to which these axes are shared across languages remains unclear. our study addresses this gap by quantitatively demonstrating the similarity of semantic axes across languages, providing insights that are difficult to achieve with alignment methods alone. to do this, we first test the reliability of each language's independent components using icasso @xcite , a method based on running the ica algorithm multiple times and clusters the results to ensure consistency. after labeling the components with words at the cluster centers, we analyze the statistical correspondence of these semantic axes. we then use the method by @xcite , originally created for neuroscience, to identify common independent components across different languages. our contribution improves the interpretability and consistency of word embeddings. by ensuring robust independent components within languages through icasso and verifying their correspondences across languages with statistical tests, we provide a more rigorous framework for analyzing semantic structures in multilingual word embeddings. foot_0 2 background interpretability in word embeddings the interpretability problem of word embeddings has been actively discussed. various approaches have been proposed to enhance the interpretability of word embeddings, including aligning embedding spaces @xcite @xcite and applying techniques such as loss function design, sparse overcomplete vectors @xcite . while these methods have shown promise, they primarily focus on ensuring that words with similar meanings are close to each other in the embedding space, without necessarily providing deeper interpretive insights into the semantic structure. ica has emerged as a relatively new method for interpreting word embeddings, showing great potential in explaining semantic axes. @xcite and @xcite applied ica to word embeddings and presented the semantic axes of the words. @xcite conducted both pca and ica on word embeddings, demonstrating that there are intrinsic semantic axes among them. their comparison revealed that ica showed more distinctive axes than pca. ica ica is a method to extract statistically independent components from multivariate data @xcite . let @xmath0, where d is the data dimension and n is the number of observations. ica is based on the assumption that x is represented as while ica is non-deterministic, which could potentially limit its reproducibility, our research addresses this issue by employing icasso. this method clusters multiple runs of ica to ensure the reliability of independent components. this approach mitigates the non-deterministic nature of ica by providing a robust statistical framework, thereby enhancing the interpretability and consistency of the semantic axes derived from ica. our study extends the work of @xcite by applying icasso and using the cluster correspondence method by @xcite to verify these components' intra-language and inter-language consistency. the method ensures that the results of semantic axes are meaningful but reproducible and reliable, making ica a more exciting and valuable method than pca for analyzing the semantic structure of word embeddings across languages. ## experimental settings we conducted the intra-language experiment focusing on the consistency of each language and then conducted the inter-language experiment focusing on the consistency among the languages. to align with @xcite , we used the same fasttext @xcite embeddings obtained by training on 157 different languages. we obtained 300-dimensional embedding vectors of 50000 words for english, japanese, and chinese, respectively, with matrices x 0 , x 1 , x 2 ∈ r 300×50000 . the 50000 words consist of 6903 common words among the three languages selected from the multilingual word dictionary @xcite and 43097 words selected in order of their frequency of occurrence in each language by wordfreq @xcite . we applied icasso implemented by @xcite to fasttext's word embeddings of english, japanese, and chinese with 10 runs, designated 300 as the number of clusters. we then tested the consistency among the components by the method proposed by @xcite by setting the false discovery rate and the false positive rate at 1%. detailed explanations are in appendix b. ## conclusion our study statistically confirmed the consistency of semantic axes within and across languages using ica components. recognizing the inherent instability of ica, we employed icasso to ensure robustness by running multiple iterations and clustering the results. this process resulted in highquality, reproducible semantic axes for english, japanese, and chinese. we then statistically verified inter-language consistency by identifying common semantic axes shared among these languages, supported by rigorous statistical tests. our primary contribution is the innovative use of statistical methods to ensure the reliability and universality of semantic axes. the validation underscores the effectiveness of our approach in achieving consistent and interpretable word embeddings and highlights the potential for improved multilingual natural language processing applications.