Axis Tour: Word Tour Determines the Order of Axes in ICA -transformed Embeddings (2024)

Abstract: Word embedding is one of the most important components in natural language processing, but interpreting high-dimensional embeddings remains a challenging problem. To address this problem, Independent Component Analysis (ICA) is identified as an effective solution. ICA-transformed word embeddings reveal interpretable semantic axes; however, the order of these axes are arbitrary. In this study, we focus on this property and propose a novel method, Axis Tour, which optimizes the order of the axes. Inspired by Word Tour, a one-dimensional word embedding method, we aim to improve the clarity of the word embedding space by maximizing the semantic continuity of the axes. Furthermore, we show through experiments on downstream tasks that Axis Tour yields better or comparable low-dimensional embeddings compared to both PCA and ICA.

## introduction embedding is an important tool in natural language processing, but interpreting high-dimensional embeddings is challenging. to address this, independent component analysis (ica) @xcite offers an effective solution @xcite @xcite . ica-transformed embeddings reveal interpretable semantic axes; however, the order of these axes is arbitrary @xcite . in this study, inspired by a one-dimensional word embedding method, word tour (sato, 2022), which leverages the traveling salesman problem (tsp), we aim to improve the clarity of the word embedding space by maximizing the semantic continuity of the axes. figure 1 shows two sets of two-dimensional projections of word embeddings: one is ordered by axis tour (our proposal), and the other is sorted in descending order of skewness (skewness sort). in axis tour, the top words of the axes are positioned farther from the center, with the meanings of the figure 1 : scatterplots of normalized ica-transformed word embeddings whose axes are ordered by axis tour and skewness sort. in the upper part, axis tour is applied to 300-dimensional glove, with nine consecutive axes arranged counterclockwise. in the lower part, these nine axes are rearranged clockwise in descending order of skewness. the embeddings are projected onto two dimensions along these axes. the top five embeddings on each axis are labeled by their words. each word is assigned the color of the axis with the highest value. in both cases, words that cross the horizontal axes are not displayed. refer to appendix a for more details. we also assume that the consecutive axes in the axis tour embeddings can be considered a subspace with axes that have similar meanings. based on this idea, we project each subspace onto a single dimension for dimensionality reduction. we show through experiments that axis tour yields better or comparable low-dimensional embeddings compared to both pca and ica. ## related work some studies transform embeddings by rotation @xcite from factor analysis @xcite or principal component analysis (pca) @xcite . independent component analysis (ica) @xcite has gained attention for its ability to reveal interpretable semantic axes in the transformed embeddings @xcite @xcite . research on interpreting embeddings by focusing on axes representing opposing concepts (e.g., cold vs. hot, soft vs. hard) is also actively pursued. approaches such as semaxis @xcite , @xcite , and frameaxis @xcite deal with static embeddings, while @xcite and @xcite deal with dynamic embeddings. in particular, section 5.3 provides a comparison of the axis tour embeddings and those from polar. relevant to our study is topographic ica (tica) @xcite . tica relaxes the assumption of statistical independence and assumes higher-order correlations between adjacent axes. then estimates the order of the axes. unlike tica, axis tour is applied to ordinary ica-transformed embeddings and uses the embeddings themselves to measure axis similarity. for more details on tica, refer to appendix f. ## background the pre-trained word embedding matrix is given by @xmath0, . . . , x n ] ⊤ ∈ r n×d , where x is centered (i.e., the mean of each column is zero). here, x @xmath1. ## axis tour this section explains axis tour and the dimensionality reduction method using axis tour. as mentioned in section 3.2, [d] = {1, . . . , d} and p @xcite ) is the set of all permutations of [d]. ## experiments similar to the word tour experiments, we used 300-dimensional glove @xcite with @xmath2,000, and the lkh solver foot_4 @xcite for the optimization of the tsp. for ica, we used fastica (hyvärinen, 1999) from scikit-learn @xcite , setting the iterations to 10,000 and the tolerance to 10 -10 , consistent with @xcite . we computed the axis embeddings 6 v ℓ in (3) with @xmath3. for baselines, we used whitened pca-transformed embeddings 7 , along with two types of whitened ica-transformed embeddings 8 : random order, which randomly flips the sign of the axes in s and randomizes the order of the axes, and skewness sort, which sorts the axes of s in descending order of skewness. see appendix e for additional experiments, including those of other embeddings. ## discussion we confirmed through both qualitative and quantitative experiments that the axis order determined by axis tour exhibits high semantic continuity, and its effectiveness was also validated in the downstream task of dimensionality reduction. there are two advantages to ordering the axes of ica-transformed embeddings using axis tour. first, as shown in fig. 1 , when projecting the embedding space, the scatterplot becomes easier to interpret and more visually accessible because higherranking words on each axis are farther from the origin. second, although the axis tour embeddings t are the same size as the ica-transformed embeddings s, the axis order in axis tour preserves information about the similarities between the axes. an interesting relationship between semantic continuity and skewness is shown in fig. 5 , where the semantic continuity of axis ℓ is measured by the average of two cosines. in axis tour, there is a high correlation between skewness and semantic continuity, while in skewness sort, they are almost uncorrelated. a large skewness indicates that an axis has a distinctive meaning, and in axis tour, this seems to contribute to semantic continuity. ## conclusion in this study, we proposed a novel method, axis tour, which optimizes the order of axes in icatransformed word embeddings. we focused on the fact that the word embeddings reveal interpretable semantic axes while the order of these axes is arbitrary. axis tour aims to improve the clarity of the word embedding space by maximizing the semantic continuity of the axes. additionally, we demonstrated through experiments on downstream tasks that axis tour yields better or comparable lowdimensional embeddings compared to both pca and ica.