Task-oriented Domain-specific Meta-Embedding for Text Classification (2020)

Abstract: Meta-embedding learning, which combines complementary information in different word embeddings, have shown superior performances across different Natural Language Processing tasks. However, domain-specific knowledge is still ignored by existing meta-embedding methods, which results in unstable performances across specific domains. Moreover, the importance of general and domain word embeddings is related to downstream tasks, how to regularize meta-embedding to adapt downstream tasks is an unsolved problem. In this paper, we propose a method to incorporate both domain-specific and task-oriented information into meta-embeddings. We conducted extensive experiments on four text classification datasets and the results show the effectiveness of our proposed method.

## introduction building semantic representations @xcite @xcite of words is a vital procedure in various natural language processing (nlp) tasks. over recent years, many pre-trained word embeddings have emerged, such as pre-trained word2vec @xcite and pre-trained glove @xcite . despite their usefulness, some previous works find that the performance of different pre-trained word embeddings has significant variation for different tasks @xcite . to obtain a stable and better performance, @xcite proposed the meta-embedding learning task that aims to obtain a robust and superior word embedding (i.e., metaembedding) by combining the different pre-trained word embeddings. most previous meta-embedding methods neglect the importance of domain-specific information and use the same embedding for each word in all domain-specific datasets @xcite @xcite . it is beneficial to incorporate domain-specific information into general word embeddings and provide different word representations for different domains, which has been shown to improve the performance in some other tasks @xcite . this leads us to explore how to combine general and domain-specific information in metaembedding learning. intuitively, the importance of the general and domain embeddings depends on a specific domain. for example, in the computer domain, for the domain-specific words (e.g., "mouse"), we should preserve their domain information but discard their general information. on the other hand, some general words (e.g., "we", "people") may not be able to get a high-quality domain embedding due to the insufficient domain data, in this situation, their general word embeddings are preferable. however, most previous metaembedding methods are unsupervised, it is hard to learn which embedding is preferable. we consider that it is necessary to use the supervision from a downstream task to address this limitation. specifically, we focus on text classification (tc) and use the words' category distributions of a tc dataset to guide the meta-embedding learning process. in this paper, we propose a supervised autoencoder method, named task-oriented domainspecific autoencoded meta-embedding (tdaeme), to learn meta-embedding for text classification. tdaeme combines both general and domain word embeddings in a supervised manner, which is implemented by a supervised autoencoder. specifically, tdaeme predicts the words' category distribution. this makes the downstream classifier easier to extract useful information from our task-oriented domain-specific meta-embedding. we evaluate tdaeme on four text classification datasets, the results demonstrate the effectiveness of our method. one contemporary work also uses the supervised autoencoder method for meta-embedding learning (o'neill and bollegala, 2020). however, their motivation and contribution are different from ours. @xcite aim to enhance the meta-embedding with words' similarity information, so they use the similarity score between words as the supervision signal in meta-embedding learning, while we focus on a more specifically task (i.e., text classification), our model uses words' categories information as the supervision signal, which is specifically designed for the classification task. in text classification task, words within the same category should be close to each other in the representation space, using similarity information may make two words in different categories get closer (e.g., "learning" and "education" with high similarity but mainly appear in two different categories "ai" and "sociology" respectively). ## method suppose that we have a word embedding set @xmath0, s 2 , ..., s n } with a vocabulary v ; a labeled text classification dataset x which contains a training set x train and a test set x test , we denote its vocabulary as v x and its categories as c x with |c x | = l. we aim to learn the word task-oriented domain-specific meta-embedding m(w) for each word @xmath1. the architecture of tdaeme is visualized in figure 1 3 ## conclusion in this paper, we propose a meta-embedding learning approach called task-oriented domain-specific autoencoded meta-embedding (tdaeme), which leverages task-oriented supervision to improve the combination of general and domain embeddings. we conducted experiments on four text classification datasets and the results show the effectiveness of our proposed method.