A n Ensemble Method for Producing Word Representations focusing on the G reek Language (2020)

Abstract: In this paper we present a new ensemble method, Continuous Bag-of-Skip-grams (CBOS), that produces high-quality word representations putting emphasis on the Greek language. The CBOS method combines the pioneering approaches for learning word representations: Continuous Bag-of-Words (CBOW) and Continuous Skip-gram. These methods are compared through intrinsic and extrinsic evaluation tasks on three different sources of data: the English Wikipedia corpus, the Greek Wikipedia corpus, and the Greek Web Content corpus. By comparing these methods across different tasks and datasets, it is evident that the CBOS method achieves state-of-the-art performance.

## introduction neural networks have significantly affected natural language processing (nlp) tasks. one of those tasks is representation learning for words, also known as word embeddings that represent words/tokens in a low dimensional hilbert space where similarity computations are feasible and enable machine learning algorithms. the main idea behind word embeddings is the distributional hypothesis @xcite , which states that the meaning of a word can be captured by the context in which it appears. word embeddings are beneficial for most nlp applications increasing the overall performance and capturing different aspects of similarity among words. numerous researches have shown these benefits in sequence tagging @xcite ) and text classification @xcite . recently, @xcite have shown that pretrained word embeddings may be a valuable feature in machine translation, particularly in low-resource scenarios. while living in the nlp era passing from static word representations to dynamic (contextualized) word representations, there are still applications where static word embeddings (word2vec, fasttext, glove) are used, such as various rnns/cnns models. it is also known that in various nlp tasks, using a concatenation of contextaware word embeddings with static word embeddings @xcite achieves better results. we propose a new architecture continuous bag-of-skip-grams (cbos), aiming to combine the benefits from skip-gram and cbow approaches. our model achieves competitively high accuracy across different tasks compared to the aforementioned models. these results lead to an overall increased performance of the word embeddings. in addition, the cbos architecture does not increase the computational cost significantly due to its efficient implementation. thus cbos can be trained on vast amounts of text corpora within a reasonable time. the main contributions of our work are: the rest of the paper is organized as follows: firstly, section 2 is a brief overview of previous work that has been done on word embeddings and nlp in modern greek. section 3 describes the data and tools that were used or produced for the training of our model. in section 4, our proposed cbos model is explained along with its differences to other popular models. section 5 presents the evaluation methods used for comparing models in the experimental setup and section 6 shows the results of the different experiments. finally, in section 7 we provide conclusions based on the results of the experiments. ## data sources and tools in this section, we describe the datasets that were used/produced for this research, along with their sources. furthermore, we present the tools and libraries that were used for the development of word embeddings models. ## evaluation the quality of our new embedding method was thoroughly evaluated on the basis of restricted lexical semantics tasks, such as scoring word similar-ity and linear relationships for analogies (intrinsic evaluation). in order to focus on how performance correlates with downstream nlp tasks, two characteristic tasks are selected: a sequence labelling task for word level and a text classification task for sentence level (extrinsic evaluation). ## conclusions this paper aimed at producing high-quality word embeddings mainly for the greek language, devising a new embedding method, the continuous bag-of-skip-grams (cbos). cbos combines the benefits of the cbow and skip-gram approaches introduced in @xcite . because of its neat implementation, cbos does not increase the computational cost of the training phase. we presented in section 6 that there are particular tasks where cbow outperforms skip-gram and vice versa. since there is no one method that outperforms all others in all tasks, we strongly recommend the use of cbos. it is evident that cbos