BAHP : Benchmark of Assessing Word Embeddings in Historical P ortuguese (2021)

Abstract: High quality distributional models can capture lexical and semantic relations between words. Hence, researchers design various intrinsic tasks to test whether such relations are captured. However, most of the intrinsic tasks are designed for modern languages, and there is a lack of evaluation methods for distributional models of historical corpora. In this paper, we conducted BAHP: a benchmark of assessing word embeddings in Historical Portuguese, which contains four types of tests: analogy, similarity, outlier detection, and coherence. We examined word2vec models generated from two historical Portuguese corpora in these four test sets. The results demonstrate that our test sets are capable of measuring the quality of vector space models and can provide a holistic view of the model’s ability to capture syntactic and semantic information. Furthermore, the methodology for the creation of our test sets can be easily extended to other historical languages.

## introduction distributional semantics assumes that words with similar contexts have similar meanings @xcite , hence, words similar in meaning should be similar in vector representation since word embeddings are learned from context. the success of word embeddings in different tasks in natural language processing verifies such assumptions to some extent. however, it has also been noticed that the quality of word embeddings is critical to different applications since the embeddings are learned in an unsupervised way. thus, for evaluating embedding models, different tasks appear, but most of these test sets are designed for english, and the multilingual versions are usually created by translation. as more historical texts have been digitized, we notice the surge of using computational methods to study historical language. compared to the large amount of digital resources in modern language, researchers normally use very limited resources to generate historical embedding models. however, the quality of embedding models generated from small corpora is not always satisfying @xcite . thus, assessing embedding models are critical to distributional semantics studies in historical languages. for a number of reasons, there is a lack of evaluation datasets for historical word embeddings. first, it is difficult to directly translate english evaluation benchmarks to historical languages because of the lack of translation tools and mismatch of concepts. second, creating intrinsic or extrinsic evaluation tasks specific to older stages of a language demands large amounts of manual annotation and requires related expert knowledge. meanwhile, we notice that prior studies in evaluation benchmark creation typically focus on a single type of test set, and few studies discuss the distinction between different test sets, especially for historical languages and embedding models generated from small datasets. in this paper, we aim to create intrinsic tasks for evaluating historical portuguese embedding models in an efficient way. major methods for assessing word embeddings such as analogy tests and similarity tests are questioned for issues of the appropriateness and informativeness and lower inter-annotator agreement. building a benchmark containing more types of evaluating methods can avoid the bias from a single method. efficiency, measured in terms of human effort and amount of resources needed, is another concern for creating the benchmark. we try to minimize the cost of building test sets of historical languages and make the benchmark reliable at the same time. four types of evaluation sets are adopted. they target different types of relations within word embeddings: syntactic, semantic, and morphological. we then use our benchmark to evaluate word2vec models generated from two historical portuguese corpora. we find that our evaluation datasets can effectively demonstrate the quality of the distributional model. this paper is structured as follows: we review predominant word embedding assessment methods and existing portuguese word embedding evaluation resources in the next section. in section 3, we introduce how we created four types of word embedding assessment datasets. then, the evaluation experiments of word2vec models generated from medieval and classical portuguese corpora are discussed in section 4. we summarize our findings in the last section. ## related work analogy tests were first introduced by @xcite to evaluate the word2vec models. this analogy test also known as google analogy test (gat) which contains 9 types of syntactic relations and 5 types of semantic categories has been widely applied in embeddings assessment. but many researchers also pointed out the disadvantages of analogy test, for example, irrelevant neighborhood structure being captured @xcite and drawbacks in appropriateness and informativeness @xcite . compared to finding the analogy relation within the vector space, word similarity test uses the corelation score between human annotated word pairs and similarity scores from the distributional model to assess the embeddings. based on the different definition of similarity, such test sets could be divided into word similarity like sim999 @xcite , word relatedness like men @xcite , and the mix of these two concepts like sim353 @xcite . the major concern toward word similarity is low human inter-annotator agreement across many datasets @xcite . besides the two instrinsic tasks above, other evaluation methods include concept categorization @xcite , outlier detection tests (camacho-collados and navigli, 2016), and coherence tests @xcite . these tests can make up for the drawbacks of analogy tests and word similarity to some degree. regarding evaluating portuguese distributional models, we see different types of intrinsic tasks in modern portuguese. @xcite translated a set of english intrinsic datasets including an analogy test, word similarity, and concept categorization into portuguese. @xcite used portuguese babelnet to create a toefl-like test set called bsg targeting semantic relations. @xcite created an english 12-8-8 outlier detection set and then translated it into portuguese. analogy test set tales created by oliveira et al. ( 2020 ) is compiled by using existing lexical resources and examines the lexical-semantic relations in portuguese. unlike the resourceful test sets in modern languages, only a few studies developed assessing tasks for historical languages. @xcite designed a task to measure language change in german by annotating semantic relatedness between diachronic usage of words. this method has been adopted by the following studies and expanded language change detection tasks to english, latin and swedish @xcite . @xcite designed analogy tests for medieval spanish. their analogy test includes seven categories of morphological and semantic relations, with more tests focusing on the former. but, these studies focus only on one type of instrinsic tasks (relatedness or analogy). ## model evaluation to have a better understanding of how our test sets fit into the model evaluation, we examined the quality of two distributional models trained on two corpora in this section ## conclusion in this paper, we presented a new benchmark for assessing word embeddings in historical portuguese that includes four intrinsic tasks. 5 the results of our tests reported here show that the cleaner corpus, colonia, led to better results, which confirms the findings of @xcite regarding two spanish corpora. in that study, the medieval corpus (a smaller corpus, with high spelling and morphological variation) was likewise more challenging than the much larger and pre-processed corpus of contemporary spanish. moreover, learning from the creation process of existing intrinsic tasks, we find a more economic way to develop test datasets for assessing embedding models generated from historical languages. by bringing together a few experts in the history of portuguese and a computational linguist, we have provided a feasible method to create multiple reliable intrinsic tasks using raw corpora. unlike previous work that has relied just on one type of task, our benchmark has the advantage of targeting different lexical relations, thus providing a more thorough assessment. in the future, we plan on doing a more thorough comparison of different embedding methods and we hope that this benchmark can be used to test models for historical corpora of other languages.