Explaining Word Embeddings via Disentangled Representation (2020)

Abstract: Disentangled representations have attracted increasing attention recently. However, how to transfer the desired properties of disentanglement to word representations is unclear. In this work, we propose to transform typical dense word vectors into disentangled embeddings featuring improved interpretability via encoding polysemous semantics separately. We also found the modular structure of our disentangled word embeddings helps generate more efficient and effective features for natural language processing tasks.

## introduction disentangled representations are known to represent interpretable factors in separated dimensions. this property can potentially help people understand or discover knowledge in the embeddings. in natural language processing (nlp), works of disentangled representations have shown notable impacts on sentence and document-level applications. for example, @xcite and @xcite proposed to disentangle sentiment and semantic of sentences. by manipulating sentiment factors, the machine can rewrite a sentence with different sentiment. @xcite also demonstrated sentence generation while more focusing on syntactic factors such as part-of-speech tags. for document-level applications, @xcite presented a learning algorithm which embeds biomedical abstracts disentangling populations, interventions and outcomes. regarding word-level disentanglement, @xcite proposed mixture of gaussian models which can disentangle meanings of polysemous words into two or three clusters. it has a connection with unsupervised sense representations @xcite which is an active research topic in the community. in this work, we focus on word-level disentanglement and introduce an idea of transforming dense word embeddings such as glove @xcite or word2vec @xcite into disentangled word embeddings (dwe). the main feature of our dwe is that it can be segmented into multiple sub-embeddings or sub-areas as illustrated in figure 1 . in the figure, each sub-area encodes information relevant to one specific topical factor such as animal or location. as an example, we found words similar to "turkey" are "geese", "flock" and "goose" in the animal area, and the similar words turn into "greece", "cyprus" and "ankara" in the location area. we also found our dwe generally satisfies the modularity and compactness properties proposed by @xcite and @xcite which can be a definition of general-purpose disentangled representations. also, our dwe can have the following advantages: ## application: compact features for downstream tasks here we demonstrate an application of the modularity and compactness properties of our dwe. we firstly aim to show the sub-embeddings can directly be informative features and can outperform glove with the same number of dimensions. ## conclusion in this work, we propose a new definition and learning algorithm for obtaining disentangled word representations. as a result, the disentangled word vectors can show higher interpretability and preserve performance on various nlp tasks. we can also see the ambiguous semantics hidden in typical dense word embeddings can be extracted and separately encoded. finally, we showed the disentangled word vectors can help generate compact and effective features for nlp applications. in the future, we would like to investigate whether similar effects can be found from non-distributional or contextualized word embeddings.