Unsupervised Cross-lingual Word Embedding Representation for E nglish-isi Z ulu (2023)

Abstract: In this study, we investigate the effectiveness of using cross-lingual word embeddings for zero-shot transfer learning between a language with an abundant resource, English, and a languagewith limited resource, isiZulu. IsiZulu is a part of the South African Nguni language family, which is characterised by complex agglutinating morphology. We use VecMap, an open source tool, to obtain cross-lingual word embeddings. To perform an extrinsic evaluation of the effectiveness of the embeddings, we train a news classifier on labelled English data in order to categorise unlabelled isiZulu data using zero-shot transfer learning. In our study, we found our model to have a weighted average F1-score of 0.34. Our findings demonstrate that VecMap generates modular word embeddings in the cross-lingual space that have an impact on the downstream classifier used for zero-shot transfer learning.

## introduction in the development of egalitarian natural language processing (nlp) systems, cross-lingual word embeddings are gaining prominence. according to distributional theory, words that appear in comparable contexts have semantic commonalities @xcite . as a result, word embeddings have paved the way for nlp technology advancement. the use of word embedding representation has provided satisfactory performance for many nlp applications and associated applications @xcite . word embeddings have been used for feature engineering @xcite and transfer learning purposes @xcite . translation models, sentiment analysis tasks and classification tasks have all benefited. these improvements have been primarily in high-resource languages, such as english, due to the abundance of labelled corpora in these languages. the lack of labelled corpora in low-resourced languages has come at a disadvantage in advancing nlp technologies within this space. many news publications in south africa are in english even though south africa has eleven official languages @xcite . according to the statistics south africa (stats sa) census 2011 results, isizulu was the most spoken home language with 22.7% of the population indicating it as a home language (statistics south africa (stats sa), 2012). isizulu forms part of the many indigenous languages in south africa, and belongs to the nguni family of languages @xcite . the nguni language family is characterised by their agglutinative morphology @xcite . south africa is a multilingual nation where the large majority of the population have a secondary language that they use on top of their primary languages. over the post-apartheid years, the country has seen a population growth of citizens that speak english as a second language. this behaviour has also become prevalent for isizulu, where citizens have added isizulu as a second language @xcite . this emphasizes the importance of advancing nlp work for low-resourced languages in south africa. in this work we attempt to take advantage of a high-resource corpus and a low-resource corpus in order to learn cross-lingual word embeddings for english and isizulu. the use of cross-lingual word embeddings would allow us to perform model transfer learning from a high resource language to a low-resource language. the undertaking for isizulu is especially challenging due to the morphological complexity of the language. hence, we have to a handle words in a manner that can maximise syntactic and semantic representation of both english and isizulu in the cross-lingual space. in this work, we aim to answer the following research questions: additionally, we introduce two new datasets: the remainder of the work is organised as follows: section 2 will discuss the background and prior work. section 3 describes the methodology used for the vecmap library and zero-shot learning classifier. section 4 discusses the experiments and results. finally, concluding remarks and potential next steps for future research are discussed in section 5. ## background & related work the popular methods that have been used to represent tokens as vectors have been bag-of-words (bow), term frequency-inverse document frequency (tf-idf) and pointwise mutual information (pmi). however, these techniques create sparse and large matrices and also create vectors that do not capture semantics @xcite . innovative advancements have led to the emergence of new techniques using neural networks. these methods include the popular word2vec models @xcite and attention based models @xcite . these models have achieved state-of-the-art performance on adaption and transfer learning tasks in nlp @xcite . the disadvantage of monolingual word embeddings is that they struggle with transferring between languages that are dissimilar @xcite . multilingual word embeddings trained on multiple languages have also shown to perform poorly for low-resource languages @xcite . approaches for cross-lingual representation learning have been proposed to address this issue @xcite . there are two types of cross-lingual word embedding methods: mapping and post-hoc approaches @xcite . initial mapping approaches required the use of a bilingual mapping dictionary between the languages of study, but recent improvements have introduced adversarial methods. adversarial approaches presume that the input monolingual spaces are isomorphic and hence reducing the requirement of a bilingual mapping dictionary @xcite . however, it has been shown that adverserial methods perform poorly when the monolingual embeddings are not isomorphic @xcite . downstream nlp tasks have been proven to benefit from cross-lingual word embeddings. for example, cross-lingual word embeddings have increased the performance of machine translation @xcite and part-of-speech (pos) tagging @xcite tasks as a transfer learning strategy. vecmap has been used to create a mapping between english and welsh and the resulting embedding were used to train a zero-shot and fewshot learning welsh sentiment classifier @xcite . in the south african context, vecmap was used to develop a sepedi-setswana cross-lingual word embedding and adopted a semantic assessment method to analyse the similarity between pairs of setswana and sepedi terms @xcite and used in a noisy, multilingual question-answering challenge to train an lstm classification model @xcite . zero-shot transfer learning has not been widely used for south african news classification. marivate et al. (2020) used an annotated corpus of local news items to create a news classifier for setswana and sepedi. instead of using static word embedding as those used by vecmap, using contextual word embeddings from a fine-tuned bert model has achieved impressive results for a zero-shot learning named entity recognition (ner) task for isizulu @xcite . however, this work would require significant computing and has not been applied to news classification. to our knowledge, this is the first work that uses cross-lingual word embeddings generated from vecmap to create a news article classifier from english to isizulu. ## methodology we discuss the methodology for the development of the cross-lingual word embeddings and the downstream classifier for news classification. ## experiments & results in this section of the paper we describe the experiments and outcomes conducted in an effort to answer the research questions. we describe the created cross-lingual word embeddings and the experiments undertaken to develop the news article classifier. ## conclusion in this work we use vecmap to create cross-lingual word embeddings between english and isizulu. we have shown that vecmap generates modular word embeddings in the cross-lingual space due to the monolingual word embeddings generated by fast-text. hence, we generate cross-lingual embeddings that are used to train a classifier that performs good on english news. however, we were unable to transfer the learning on unseen isizulu news articles. in future work, we would like to examine the performance of vecmap using a larger corpus for isizulu. additionally, it would also be advantageous to apply the modularity metric to optimize the hyperparameters of fasttext in order to generate appropriate monolingual embeddings for the task.