L’optimisation du plongement de mots pour le français : une application de la classification des phrases (Optimization of Word Embeddings for F rench : an Application of Sentence Classification) (2018)

Abstract: Nous proposons trois nouvelles méthodes pour construire et optimiser des plongements de mots pour le français. Nous utilisons les résultats de l’étiquetage morpho-syntaxique, de la détection des expressions multi-mots et de la lemmatisation pour un espace vectoriel continu. Pour l’évaluation, nous utilisons ces vecteurs sur une tâche de classification de phrases et les comparons avec le vecteur du système de base. Nous explorons également l’approche d’adaptation de domaine pour construire des vecteurs. Malgré un petit nombre de vocabulaires et la petite taille du corpus d’apprentissage, les vecteurs spécialisés par domaine obtiennent de meilleures performances que les vecteurs hors domaine.

## introduction word embedding techniques have prevailed in natural language processing (nlp) and have obtained impressive results in several areas @xcite . word embeddings are trained on word cooccurrence in text, and can capture semantic information about words and their meanings. word2vec @xcite and glove @xcite have been proposed to learn a distributed representation for words in a continuous vector space. word embedding vectors with subwords were also presented to improve embedding quality @xcite . there are still many *2vec-style variations. multilingual embedding @xcite or embedding with polysemy @xcite have also been presented as one extension of word embedding techniques. at syntactic level, @xcite proposed dependency-based word embeddings, and @xcite developed even further to encode the syntactic-aware context of entities. as proposed in previous work word embedding techniques have rapidly been used in a large range of applications in nlp over the last years. in this paper, we present three methods for building and optimizing embedding vectors for french. we use pos tagging, multiword expression detection and lemmatization of words to optimize a word embedding task. we extrinsically evaluate the proposed method for embedding vectors by using a sentence classification task. the current work differs from previously proposed opinion mining (or sentiment analysis) (inter alia, @xcite ) where the classification task in previous work is mainly based on the document. for sentence classification, @xcite shows phrase fragments classification using a recursive neural network model. more recently, @xcite proposes the sequential sentence classification task by adding a sentence label prediction layer. the main contribution of this paper is as follows. first, we explore linguistically motivated methods to optimize word embeddings for french. secondly, we propose a new data set for the sentence classification task for french. in addition to embedding optimization and sentence classification, we introduce a domain adaptation approach by selecting the training corpus for embeddings. we find several practical facts in word embeddings including the effects on the size of the training corpus, the size of vocabularies, and their relatedness with the task. ## optimizing word embeddings for french this section describes three linguistically motivated methods for optimizing word embedding vectors for french : pos-aware, mwe-aware and lemma-aware embeddings. instead of directly using surface forms of the word, we first use a pair of word and part of speech (pos) to disambiguate possible polysemy. secondly, since words in multiword expressions (mwes) can give more senses when they are bonded together, mwes are dealt as a single unit. thirdly, we use lemma forms in order to avoid sparsity because of rarely used inflected words. while the current methods have been proposed and used in the bag-of-words model, to our best knowledge there are no previous efforts for word embeddings to optimize the vector space model for french. we note that, while @xcite proposed complex terms for embeddings, he worked on embeddings for english. previously, character @xcite @xcite syllable @xcite , and subword @xcite @xcite @xcite embedding techniques have been proposed to enrich word vectors. ## sentence classification we extrinsically evaluate the proposed optimization methods for word embedding by using the sentence classification task. first, we build sentence classification data, and then present classification results using various embedding settings and corpus sizes. we also explore the domain adaptation approach by using the in-domain corpus to build embedding vectors. ## discussion and conclusion we proposed several methods for optimizing word embeddings for french by using pos tagging, mwe detection and lemmatization. we used these embedding vectors in the sentence classification task, and mwe-aware in which the size of vocabularies are relatively large to other embeddings improved the classification result. to extend current mwe-aware embedding, we can consider formulaic sequences and named entities @xcite in addition to mwes. we leave these extensions for future work. while reported results show a minor improvement, they confirm our intuition that incorporated context information in embeddings either linguistically motivated (mwes) or not (bigram features) is important. as described, "in-domain" information played an important role, which should be well adapted to the proposed task. in-domain embeddings using a small number of vocabularies and the small size of the training corpus (roughly a third of the out-domain corpus) outperformed out-domain embeddings for all cases. the word embedding vectors, preprocessed source text files including the preprocessing script, and the sentence classification data for french are publicly available at https://github.com/jungyeul/taln2018 .