W ordnet-based Evaluation of Large Distributional Models for P olish (2018)

Abstract: The paper presents construction of large scale test datasets for word embeddings on the basis of a very large wordnet. They were next applied for evaluation of word embedding models and used to assess and compare the usefulness of different word embeddings extracted from a very large corpus of Polish. We analysed also and compared several publicly available models described in literature. In addition, several large word embeddings models built on the basis of a very large Polish corpus are presented.

## introduction distributional semantics (ds) is focused on describing semantic associations between words on the basis of their distributional patterns in texts by applying statistical methods. ds methods are used to extract different kinds of the measures of semantic relatedness (msr) from corpora . an msr can cover the whole range of semantic relations from topic or domain based till lexico-semantic relations. for many applications it is desirable to obtain an msr which is close to a measure of semantic similarity (mss), i.e. a measure which assigns the highest values to words associated by linguistic lexico-semantic relations. recently, word embeddings have become one of the best tools of ds. however, word embeddings, e.g. @xcite , are based on predicting a word occurrence in a context (mostly a sequence) of other words. this aspect of cooccurrence prediction in a local context can influence an msr built on the basis of word embeddings. an mss can be an important source of knowledge supporting wordnet development, e.g. @xcite . however, the question is how to evaluate to which extent the given msr resembles an proper mss? experiments with the participation of humans are laborious, costly and the datasets created as a result are of limited size. it is hard to construct an evaluation by application in a way revealing the properties of a potential mss. a large wordnet is built on knowledge originating from humans. it includes directly the knowledge about lexico-semantic relations and offers an opportunity to build large scale, realistic tests. our goal is to construct large scale test datasets for word embeddings on the basis of a large wordnet, apply them for evaluation of word embedding models and next to analyse and com-pare the usefulness of different word embeddings extracted from a very large corpus of polish. finally, we want to publishing word embedding models of known properties built on the basis of a very large corpus of polish. typical datasets used in the intrinsic evaluation are small, e.g. @xcite , ws-353 @xcite and most of the all 10 data sets discussed in @xcite , where only two of them include ≈2000 and ≈3000 word pairs. they were used in many tests, in fact overused. small sizes of these datasets make performing proper evaluation more difficult, e.g. because of the lack of the common partitioning into training, tuning and testing parts. datasets for msr evaluation are often collected during experiments based on testing human judgement in reaction to some prompting signal, which is close to reaction to a stimuli, e.g. @xcite measured the correlation between the reaction times in the context of priming with ranking based on word embeddings. however, this is slightly different situation than analysis of lexical meanings during language utterance interpretation, especially a textual utterance. msr is extracted from a text corpus, and it is more natural to evaluate it against language resources. moreover, @xcite noticed that the distinction between similarity and relatedness is not well defined and consistently expressed in most popular test datasets. @xcite evaluated systematically different ds models, but finally all tests were based on data collected during crowdsourcing experiments using amazon turk. @xcite performed "evaluation focused on data efficiency" with respect to 4 categories, namely: "similarity, analogy, sentence and single word". in the case of similarity, which is most interesting for us, they used only well known data sets for english. for each type of dataset different combinations of preprocessing and classification algorithms were applied. it is worth to notice, that the cost of preparing larger datasets for another language than english is quite sub-stantial. this is one of the reasons that it is hard to find such approaches for other languages, with notable exceptions e.g. @xcite for portuguese. in our case we want to explore the possibility of constructing of large test datasets on the basis of an already existing wordnet. as the primary application we focused on is support for wordnet development, so comparison with data collected in experiments with humans is not necessarily the best solution for us. ## wordnet-based evaluation in many approaches a wordnet was used to generate a wordnet-based measure of semantic similarity that was next used to assess the correlation between it and an msr, e.g. @xcite . it was assumed that similarity rankings generated by the two measures should be similar. however, there are many wordnet-based similarity measures of different properties and some of them depend on additional knowledge like information about the frequency of word senses. thus, the result of the comparison can be different depending on the wordnetbased similarity measure applied and in all cases is not straightforward in interpretation. we want to follow a different approach and to explore two methods that are free of these problems. @xcite proposed a wordnet-based synonymy test (wbst) in which for a question word x an n-tuple is automatically generated: @xmath0, . . . d n , such that one the elements: d i is the correct answer, i.e. it is synonymous with x and belongs to the same synset as x, and all other d @xmath1, i.e. false answers that are not synonymous with x. elements of d and the position of the correct answer are randomly selected. msr is tested by using its values in selecting a possible answer for the problem word x. ## experiments during experiments, we built several word embeddings models from the largest corpus of polish available. next we evaluated them in several tests based on plwordnet 3.1 (i.e. the most contemporary version) and compared with other word embedding models for polish extracted from smaller corpora and published in the web. ## results the results of the tests in tab. 1 illustrate well the differences in the difficulty of the tests: wbst is the simplest one, ewbst the hardest. the difference between ewbst and the two other tests is striking in all experiments. the difficulty of ewbst can be tuned by changing the wordnet-base similarity measure it is based on and the dependency between the similarity measure and the probability distribution of the selection of detractor words. skip-gram model is better than cbow according to wbst and ewbst in most of the cases while in the other cases the differences are small. only in hwbst cbow-ns achieved higher result that can be attributed to a kind of generalisation introduced by the inclusion of hypernyms into correct answers. contrary to the synonymy tests, in the case of wbcr evaluations of the models generated from plwnc-multi presented in tab. 3, we can notice that cbow models are superior in all cases in comparison to skip-gram models. it means that skip-gram models are better in describing differences between word meanings, while cbow enable broader exploration of potential lexico-semantic relations. however, relatively good precision signals that instances of lexicosemantic relations receive higher values. definitely the results of the test are negatively biased by lacking relation instances in plwordnet. this kind of tests and evaluations can be used also as a diagnostic tool to spot these subdomains in a wordnet that are potentially not well enough described by relation links. we can ob-serve also that the application of hierarchical softmax consistently produces better results in all frequency ranges. however, hierarchical softmax should result in better estimation of the representation. we evaluated also word embedding models extracted from plwnc-lem, i.e. a version without folding pns and mwes into single tokens. wbcr tests showed lower performance due to the lack of mwe description. we also plan to apply tests not including mwes in the future in order to investigate the effect of folding more precisely. thus, models based on plwnc-multi offer a unique opportunity of obtaining good distributional description of pns and mwes. quite surprisingly, we can observe in tab. 4 that models built on a smaller corpus of wikipedia behave in a slightly different way in wbcr tests for less frequent words than those constructed on a very large corpus. in tab. 4 skip-gram models express higher recall, fasttext skip-gram with sub-word representation have much higher recall for words with lower frequency. however, this can be an effect of different preprocessing and filtering of the data. nevertheless, all results obtained on the polish wikipedia are worse than those in tab. 3 generated from a very large corpus (including also the wikipedia data). it means that for wbcr tests, that cover a wider spectrum of relations, larger data used result in the improvement of the model. finally, in analogy tests, see ## conclusions we showed that a large comprehensive wordnet can be successfully used as a basis for two different types of msr evaluation methods, namely the family of wordnet-based synonymy tests and wordnet-based cut-off rendering tests. in both types of tests very large datasets can be generated allowing for very intensive testing and high statistical significance of the test results. the datasets are enough large to conveniently partitioned according to the frequency criteria of semantic criteria. in fact the datasets and tests are based on human decisions expressed in the wordnet structure. both tests describe the ability of an msr to be used as a basis for developing a lexico-semantic language resource. wbst-family tests focus on the ability of an msr to distinguish between different lexical meanings, while wbcr is sensitive more to representation of different types of wordnet relations by an msr. as a result both types of tests are quite complementary. moreover, by changing the similarity and context definitions in ewbst we can obtain tests of different difficulty. in the further work, we develop a wordnet-based test that has properties of contextual tests, e.g. which is similar to stanford contextual word similarity dataset (scws) @xcite . we will also expand the presented evaluation to the dataset covering all four pos, namely nouns, adjectives, verbs and adverbs. the constructed word embedding models and evaluation datasets have been published on open licences under the link: https://clarin-pl.eu/dspace/handle/ 11321/446.