Fine-tuning Distributional Semantic Models for Closely-Related Languages (2021)

Abstract: In this paper we compare the performance of three models: SGNS (skip-gram negative sampling) and augmented versions of SVD (singular value decomposition) and PPMI (Positive Pointwise Mutual Information) on a word similarity task. We particularly focus on the role of hyperparameter tuning for Hindi based on recommendations made in previous work (on English). Our results show that there are language specific preferences for these hyperparameters. We extend the best settings for Hindi to a set of related languages: Punjabi, Gujarati and Marathi with favourable results. We also find that a suitably tuned SVD model outperforms SGNS for most of our languages and is also more robust in a low-resource setting.

## introduction the development of word embedding models in nlp has led to improved performance on a range of lexical semantic tasks @xcite . the sgns (skip-gram negative sampling) training method has been shown to outperform previously used count-based models such as ppmi (positive point based mutual information) and svd (truncated singular value decomposition). in this paper, we look at the task of word similarity in hindi and related languages punjabi, gujarati and marathi. specifically, we experiment with hyperparameter tuning for sgns, svd and ppmi for hindi and then ask whether the same hyperparameters can be extended and applied to typol related languages. we make use of the hyperparameters formulated in @xcite to tune all three models. we find that a suitably tuned svd model outperforms sgns. this result differs from @xcite which shows that fine-tuned sgns and svd models perform at par. we find that our hindi svd results are better than multilingual fast-text @xcite and the recently released indicnlp suite @xcite . we hypothesize that hyperparameters are sensitive to linguistic properties. if this is true, then we should find similar results in languages that are typologically related to hindi. our results suggest that these hyperparameter settings for hindi can be extended to typologically-related languages. indeed, the results show that adapting the hyperparameters from hindi is more advantageous as compared to the default settings or the settings recommended for english in @xcite . while reasonably large resources and datasets exist for the major indian languages, @xcite have shown that more than half of the indo-aryan languages represented in wikipedia can be classified as having 'poor resource availability'. given these limitations, a model that is able to generate representations that are robust in the face of less data can be advantageous. in our experiments, we take successively smaller corpus slices from our hindi, marathi, gujarati and punjabi corpora in order to test our models' performance. we find that svd is more robust compared to other models in a low-resource setting. @xcite have investigated the effects of data size on distributional semantic models and their results on the robustness of svd correspond with our findings for hindi and related languages. in this paper, we first describe the creation of a new word similarity dataset for hindi, which addresses some of the limitations of existing evaluation datasets. we then discuss the hyperparameter settings suggested in @xcite and describe our results for hindi and related languages on the entire corpus as well as on smaller corpus sizes. we conclude with a summary of our findings. ## word similarity dataset in order to evaluate our distributed semantic models, we carry out an intrinsic evaluation of the models based on word similarity. for the languages in our study, the currently available word similarity datasets include translated versions of english wordsim-353 (ws-353) @xcite . however, we note that wordsim-353 @xcite has been criticized for conflating association and similarity in its word-pair annotation guidelines @xcite . as a consequence, ws-353 measures association rather than similarity. in addition to this, we observe that the hindi version of ws-353 consists of numerous transliterations from english. in order to develop a more robust evaluation dataset for our word experiments, we created hin-rg63, the hindi version for english rg-65 @xcite . this dataset carefully dissociates similarity and relatedness in its annotation guidelines and has been used as a benchmark for semeval tasks @xcite . while we were unable to create rg-65 translations for the other languages included in this paper, we plan to extend the work done for hindi to more languages in the future. two native speakers of hindi, who were also bilinguals provided the translation of words in english rg-65 to hindi. a third translator moderated any disagreements. it is noteworthy that two of the word pairs from english rg-65 did not have any suitable distinct translation and hence were not included in the final dataset. 16 hindi native speakers were presented with the similarity scoring guidelines given in @xcite . the annotators were presented with a practice session consisting of sample word pairs before rating the actual hindi word pairs in the dataset. next, the annotators were asked to score each pair on a scale of 0 to 4. to present more flexibility, scoring with a step of 0.5 was permitted. we computed inter-annotator agreement using pairwise correlation between individual annotators' ratings. pearson and spearman correlation coefficients are used to assess the linear correlation and monotonic relationship respectively. we report an average pairwise pearson correlation of 0.814, and average pairwise spearman correlation of 0.805 for hin rg-63, our version of english rg-65. we make the dataset available for public use. foot_0 3 comparing svd, ppmi and sgns @xcite 's paper compared word embedding models or 'context-predicting models' like sgns with 'context-counting' models like svd on various lexical semantic benchmarks. their results showed the superiority of context-predicting models. in a follow-up to this result, @xcite demonstrated that suitably augmented and tuned pmi (pointwise mutual information) and svd (singular value decomposition) i.e contextcounting models can perform at par with word embedding models. in fact, insights from word embeddings can be used to augment count-based models, resulting in only very small differences in performance. these system design changes formulated as a set of transferable hyperparameters in @xcite were applied either at the pre-processing or post-processing stage, modifying the word vectors generated from these methods. the following section expands upon these hyperparameters ## summary our work shows that careful hyperparameter tuning can go a long way in improving the performance of distributed semantic models. interestingly, we find that the general recommendations in @xcite for hyperparameter settings work well only for english word representations. for hindi, some of their recommendations hold, whereas others do not. moreover, the hyperparameter settings for hindi carry over well to related languages and there is performance improvement compared to the default or english-specific settings. this seems to suggest that language specific differences are playing a role with respect to hyperparameter settings. perhaps the most interesting result is that modified svd is more robust than sgns, both across languages and data sizes. this contrasts with the idea that the differences between the architectures of particular distributional semantic models are trivial so long as they are trained in a similar fashion @xcite . we also note that the problem of building word embeddings for low-resource languages has been addressed using cross-lingual representations @xcite . these rely on alignments between low-resource and better-resourced languages. techniques such as cognate detection have been used to improve these alignments @xcite . newer contextualized word embedding models can make these alignments internally, allowing for cross lingual transfer @xcite . in this paper we have chosen to focus on monolingual word embeddings and leave the exploration of cross-lingual representations for future work.