Clustering Word Embeddings with Self-Organizing Maps. Application on L a R o S e D a - A Large R omanian Sentiment Data Set (2021)

Abstract: Romanian is one of the understudied languages in computational linguistics, with few resources available for the development of natural language processing tools. In this paper, we introduce LaRoSeDa, a Large Romanian Sentiment Data Set, which is composed of 15,000 positive and negative reviews collected from the largest Romanian e-commerce platform. We employ two sentiment classification methods as baselines for our new data set, one based on low-level features (character n-grams) and one based on high-level features (bag-of-word-embeddings generated by clustering word embeddings with k-means). As an additional contribution, we replace the k-means clustering algorithm with self-organizing maps (SOMs), obtaining better results because the generated clusters of word embeddings are closer to the Zipf’s law distribution, which is known to govern natural language. We also demonstrate the generalization capacity of using SOMs for the clustering of word embeddings on another recently-introduced Romanian data set, for text categorization by topic.

## introduction perhaps one of the most studied tasks in computational linguistics is sentiment classification, a.k.a. opinion mining or polarity classification. the task has been studied across several languages, the most popular being english @xcite @xcite @xcite @xcite @xcite @xcite , chinese @xcite @xcite @xcite , arabic @xcite @xcite @xcite or spanish @xcite @xcite @xcite . however, studying this task for under-studied languages, e.g. romanian, is difficult without access to large data sets. we hereby introduce laroseda, a large romanian sentiment data set, which is freely available for download at https://github.com/ancatache/laroseda for noncommercial use. with a total of 15,000 positive and negative reviews collected from one of the most popular romanian e-commerce websites, to our knowledge, laroseda is the largest data set for romanian polarity classification. we experiment with two baseline methods on our novel data set. the first baseline employs string kernels, an approach based on low-level features (character n-grams), that was found to work well for sentiment analysis across multiple languages, e.g. english @xcite , chinese @xcite and arabic @xcite , requiring no linguistic resources besides a labeled training set of samples. the second baseline employs bag-ofword-embeddings (ionescu and @xcite , an approach based on high-level features (clusters of word embeddings generated by kmeans), that attains good results in various text classification tasks @xcite @xcite , including sentiment analysis. as an additional contribution, we replace the k-means clustering algorithm in the second baseline method with self-organizing maps (soms) @xcite , obtaining better results because the generated clusters of word embeddings are closer to the zipf's law distribution (see figure 2 ), which is known to govern natural language @xcite . to our knowl-edge, we are the first to apply soms to cluster word embeddings, showing performance gains for both word2vec @xcite and romanian bert @xcite embeddings. we also demonstrate the generalization capacity of using soms in the bag-of-word-embeddings on another recently-introduced romanian data set @xcite , for the task of text categorization by topic. in summary, our contribution is twofold: ## related work to date, a small number of works targeting sentiment classification in the romanian language have been published. preceding the sentiment analysis efforts on romanian texts, there are a few studies on subjectivity, that have introduced two corpora built through cross-lingual projections from english to romanian @xcite or through machine translation @xcite . an extensive study conducted by @xcite looks at sentiment and subjectivity from a computational linguistics perspective, in a multilingual setup in which romanian is also included. however, in these initial works, romanian is studied only from a subjectivity perspective, which does not go down to the level of polarity. on our topic (i.e. sentiment analysis in romanian), the first study that we have found describes two word sets tagged with polarity for romanian and russian @xcite . @xcite introduced a sentiment analysis service intended for multiple languages, that also supports romanian. they perform sentiment identification using a list of manually-built triggers which, to our knowledge, is not publicly available. another effort @xcite in creating an opinion lexicon with polarity annotations introduced a collection of 2,521 romanian tourist reviews and an extensive linguistic analysis of the corpus. the data set is not released for public use. similarly, we did not find any public link to roemolex, a lexicon with approximately 11,000 romanian words tagged for emotion and sentiment @xcite . the only romanian data set annotated for sentiment that we have found freely available is rather small, with 1,000 movie reviews manually extracted from several blogs and sites @xcite . with 15,000 reviews, our corpus is much larger. ## data set in order to build laroseda, we collected product reviews from one of the largest e-commerce websites in romania. along with the textual content of each review, we collected the associated star ratings in order to automatically assign labels to the collected text samples. following the same approach used for data sets containing english reviews @xcite @xcite , we assigned positive labels to the reviews rated with four or five stars and negative labels to the reviews rated with one or two stars. however, the star rating might not always reflect the polarity of the text. we thus acknowledge that the automatic labeling process is not optimal, i.e. some labels might be noisy. since automatic labeling based on star ratings is a commonly accepted practice for opinion mining data sets of product reviews, we leave the analysis of noisy labels and manual labeling for future work. we also imitate the data collection approach used for english review data sets @xcite @xcite , selecting a balanced set of romanian reviews. more precisely, laroseda is formed of a total of 15,000 reviews that are perfectly balanced, i.e. half of them (7,500) are positive reviews and the other half (7,500) are negative reviews. in figure 1 , we show the distribution of reviews with respect to the star ratings. we note that most of the negative reviews (5,561) are rated with one star. similarly, most of the positive reviews (6,238) are rated with five stars. hence, the corpus is highly polarized. we divide laroseda into a training set containing 80% of the data samples and a test set containing the remaining 20%. in ## methods string kernels. a simple language-independent and linguistic-theory-neutral approach is to interpret text samples as sequences of characters (strings) and to use character n-grams as features. the number of character n-grams is usually much higher than the number of samples, so representing the text samples as feature vectors may require a lot of space. string kernels provide an efficient way to avoid storing and using the feature vectors (primal form), by representing the data though a kernel matrix (dual form). each component k ij in a kernel matrix represents the similarity between data samples x i and x j . in our experiments, we use the histogram intersection string kernel (hisk) @xcite as the similarity function. for two strings x i and x j over a set of characters s, hisk is defined as follows: bag-of-word-embeddings. following the seminal paper of @xcite introducing word2vec, word embeddings became one of the mainstream approaches in various computational linguistics tasks @xcite @xcite @xcite @xcite @xcite . in order to build the bag-of-word-embeddings (bowe), we first trained word2vec on the collected romanian reviews using the continuous bag-of-words (cbow) model. before training, we transformed all letters to lowercase and removed punctuation. in addition to word2vec, we consider a recently introduced romanian bert model @xcite as an alternative way to produce word embeddings, which is likely to produce much better results, considering the success of bert @xcite in english nlp tasks. instead of averaging the word embeddings to obtain document-level representations @xcite , we follow a different and more effective path suggested by some recent works @xcite @xcite . more specifically, we cluster the word embeddings collected from the entire training set using k-means. for a document d of n words, @xmath0, w 2 , ..., w n ), a word embedding model, be it word2vec or bert, outputs a matrix of n × m components (or a set of n m-dimensional vectors), the m-dimensional vector at index i corresponding to word w i . we apply clustering on the word vectors extracted from all training documents, thus obtaining a set of k clusters. a document d is then represented as a bag-of-word-embeddings (histogram) @xmath1, h 2 , ...., h k ) in which each component h i retains the number of word embeddings from the document d that fall in cluster i, where @xmath2, 2, ..., k}. we note that the size of the bag-of-word-embeddings is equal to the number of clusters k. in the case of bert, we emphasize that, although the embedding vector of a word depends on the context, it is likely that the embedding vectors corresponding to a specific word will fall in the same cluster. hence, bowe is able to cope well with this situation. replacing k-means with soms. quantitative linguistics studies @xcite have pointed out that, given a corpus of text documents, the frequency of any word is inversely proportional to its rank in the frequency table, giving rise to a zipf's law distribution of words in natural language. however, the k-means algorithm tends to ignore the data density, producing equally-sized clusters. we therefore propose to replace the k-means algorithm with an approach that takes into account the density in the word embedding space, producing a set of clusters that follow the zipf's law. we propose to perform clustering using self-organizing maps (soms) @xcite , since these models are known to preserve the topological properties of the input space. indeed, figure 2 shows that soms produce clusters of romanian word embeddings closer to the zipf's law distribution than k-means. it is important to emphasize that k-means can produce clusters of different size, as shown in figure 2. our observation refers only to the fact that the data density is not particularly modeled by the k-means optimization process, while soms are optimized by shifting the neural units following the density of the data (units tend to migrate where the space is more dense). our observation with respect to k-means is also confirmed by other studies. for example, @xcite note that: "even when all other implicit geometric assumptions of kmeans are satisfied, it will fail to learn a correct, or even meaningful, clustering when there are significant differences in cluster density". since natural language involves such significant differences (due to the presence of the zipf's law), we believe that k-means is a sub-optimal choice. ## conclusion in this paper, (i) we introduced laroseda, a large data set for polarity classification of romanian reviews, and (ii) we employed self-organizing maps, a clustering approach that preserves the density of words in the embedding space, resulting in a more effective bag-of-word-embeddings representation. our top accuracy rates on laroseda are 89.54% for the cross-validation procedure and 90.90% on the test set. we note that soms had a significant contribution in attaining these high accuracy rates. we conclude that the combination of hisk and bowe-bert with soms is a strong baseline which should encourage future research in proposing non-trivial models for romanian polarity classification. furthermore, the results obtained on moroco confirm that soms provide better accuracy rates than k-means when it comes to building document-level representations based on clustering word embeddings.