Understanding Higher-Order Correlations Among Semantic Components in Embeddings (2024)

Abstract: Independent Component Analysis (ICA) offers interpretable semantic components of embeddings.While ICA theory assumes that embeddings can be linearly decomposed into independent components, real-world data often do not satisfy this assumption. Consequently, non-independencies remain between the estimated components, which ICA cannot eliminate. We quantified these non-independencies using higher-order correlations and demonstrated that when the higher-order correlation between two components is large, it indicates a strong semantic association between them, along with many words sharing common meanings with both components. The entire structure of non-independencies was visualized using a maximum spanning tree of semantic components. These findings provide deeper insights into embeddings through ICA.

## introduction embeddings play an important role in natural language processing, ranging from word embeddings @xcite to internal representations in language models @xcite @xcite . understanding how embeddings represent meaning is crucial for unraveling black box nlp models. independent component analysis (ica) @xcite is an effective method for visualizing and interpreting the geometric structure of embeddings @xcite . just as pca aims to make coordinate axes uncorrelated, ica seeks to transform the coordinate axes into statistically independent components. the resulting axes from ica tend to have sparser component values with a few larger values compared to pca, which increases interpretability as the axes can be seen as specific semantic components (fig. 1 ). however it has been pointed out that the estimated 'independent components' are only approxi-figure 1 : heatmap visualization of 300-dimensional sgns embeddings transformed by pca and ica, with axes sorted by variance and skewness, respectively. each embedding has been normalized to have a norm of 1 for better visual interpretation. for each axis, the top 4 words (frequency n @xmath0. the first 100 axes are displayed in the top panels, and the first 5 axes with the word labels are displayed in the bottom panels. see appendices a, b and g for details. in this study, we aim to further interpret the results of applying ica to embeddings by focusing on the non-independence between 'independent components'. we measure the degree of nonindependence by calculating higher-order correlations between components and find that components with large higher-order correlations can be interpreted as having strong semantic associations. the entire structure is revealed by visualizing the maximum spanning tree of semantic components with higher-order correlations as edge weights. here, r ica is obtained by minimizing the mutual information 3 i(s 1 • • • s d ) = h(s i ) -h(s 1 • • • s d ), which is equivalent to maximizing the non-gaussianity 4 of the distributions of s i @xcite . the normalized ica-transformed embeddings, with each embedding in s rescaled to a norm of 1, offer high interpretability @xcite and are used for visualizations in this paper. figure 2 : scatterplots of normalized word embeddings along the 10th and 20th axes. the axes for pca and ica-transformed embeddings were arranged in descending order of variance and skewness, respectively. in both transformations, the components are uncorrelated. comparison of pca and ica. figure 2 shows that ica can find the 'spiky and interpretable shape' of the embedding distribution (e.g., "biology" and "stars" for the 10th and 20th axes, respectively), but pca cannot. this is because ica determines the coordinate axes toward high non-gaussianity, while pca only considers variance information. higher-order correlation. to quantify nonindependencies, methods like mutual information and hilbert-schmidt independence criterion (hsic) @xcite exist. here we use the higher-order correlation, the simplest measure in terms of computation and formulation. this measure is expressed as follows: here, s is the whitened matrix 5 . this can also be interpreted as the covariance between s 2 i and s 2 j , plus one, as cov(s 2 i , s 2 j ) = e(( if s i and s j are independent of each other, then e(s 2 i s 2 j ) = e(s 2 i )e(s 2 j ) = 1. thus, the deviation of e(s 2 i s 2 j ) from 1 is the degree of dependence between s i and s j . figure 3 : heatmaps of the correlation coefficient e(s i s j ) and the higher-order correlation e(s 2 i s 2 j ) of component pairs (s i , s j ) from ica on 300-dimensional sgns embeddings. see appendix c for details. figure 3 shows that the estimated independent components of the embeddings are uncorrelated but not completely independent, with varying degrees of higher-order correlations across pairs. these e(s 2 i s 2 j ) values provide a useful metric of association, as demonstrated in the following section. ## visualization of non-independence structure in this section, we construct a maximum spanning tree (mst) based on higher-order correlations to visualize the non-independence between estimated independent components. settings. the 300 ica components, originally ordered by skewness with @xmath1, • • • , 299, were resorted in descending order of semantic component consistency scores to prioritize axes that are more easily interpretable as specific semantic components. the semantic component consistency scores were determined by a word intrusion task @xcite . a higher consistency score indicates easier interpretability. details of the scoring methods are provided in appendix e.1. we introduce the notation σ to map the order of consistency scores to the original axis numbers in the skewness order: σ(j) represents the axis number in the skewness sort for the axis with the j-th highest consistency score. then, we consider a weighted complete graph g 150 , with 150 components having high consistency scores s σ(i) (@xmath2, • • • , 149) as nodes. for the edge between the node pair (s i , s j ), figure 5 : subtrees of mst t 150 defined in sec. 5. each node represents an independent component s k (i.e., axis k) estimated by ica. the label of each node is "k : topword(k)", where topword(k) is the word with the largest component value along axis k among words with frequency n @xmath3. the color of the edge between nodes (i, j) represents the magnitude of the e(s 2 i s 2 j ) value between the components, with darker edge colors indicating larger values. interpretation of the mst. the mst t 150 represents a graph structure expressing the nonindependence between estimated independent components. since the edges in t 150 connect component pairs with large higher-order correlations, we can interpret that there is a strong relationship between the components connected by these edges. furthermore, the subtrees of t 150 represent groups of semantically related components, and the components within these groups tend to have similar meanings. results and discussion. figure 5 shows a part of the mst t 150 ; the entire mst t 150 is exhibited in appendix e.2. the colors correspond to the clusters obtained by applying spectral clustering 8 (ng et al., 2001) to t 150 . the weights used for clustering are the higher-order correlations. from 7 we used minimum_spanning_tree implemented in @xcite for the computation of the mst. see appendix e.2 for details. 8 we used spectralclustering implemented in scikit-learn @xcite . the mst, we can infer structures such as connections and groupings of meanings among three or more components 9 . for example, semantically related component pairs such as (2: dna, 10: acid) in the pink cluster and (27: greek, 64: goddess) in the cyan cluster are connected by edges in t 150 . additionally, groups such as {168: license, 13: windows, 56: cpu} in the yellow cluster and {46: spacecraft, 35: aircraft, 47: ship} in the blue cluster form semantic clusters as sets of nodes connected by edges. the components within these groups can be interpreted as having meanings related to "computer" and "vehicle", respectively.