How many words does it take to understand a low-resource language? (2025)

Abstract: When developing language technology, researchers have routinely turned to transfer learning to resolve the data scarcity conundrum presented in low-resource languages. As far as we know, this study is the first to evaluate the amount of documentation needed for transfer learning, specifically the smallest vocabulary size needed to create a sentence embedding space. In adopting widely spoken languages as a proxy for low-resource languages, our experiments show that the relationship between a sentence embedding’s vocabulary size and performance is logarithmic with performance leveling at a vocabulary size of 25,000. It should be noted that this relationship cannot be replicated across all languages and this level of documentation does not exist for many low-resource languages. We do observe, however, that performance accelerates at a vocabulary size of \le 1000, a quantity that is present in most low-resource language documentation. These results can aid researchers in understanding whether a low-resource language has enough documentation necessary to support the creation of a sentence embedding and language model.

## introduction more than 43% of the languages spoken in the world are endangered @xcite . due to globalization and neocolonialism, language loss occurs at an accelerated rate @xcite . saving and revitalizing endangered languages has become very important for maintaining cultural diversity @xcite . in times of crisis, these language technologies allow first responders to save lives. for example, the low resource languages for emergent incidents (lorelei) provides situational awareness based on information from any language and supports humanitarian assistance/disaster relief, peacekeeping, and infectious disease response @xcite . working with minimal data-as would be the case with endangered languages-makes it difficult to train natural language models from scratch. for these reasons, transfer learning is a potential method for language models to adapt to endangered languages @xcite @xcite . we focus our research questions on cross-lingual transfer learning for low-resource languages to: by establishing this lower-bound, we can better assess whether a low-resource language has enough documentation to support the creation of a sentence embedding space and language model. ## methodology we analyze sentence embeddings as they are highly important in the creation of language models @xcite . in a survey of cross-lingual transfer learning learning methodologies, we found that @xcite 's methodology to be the simplest. @xcite draws on finnish word embeddings to create embedding spaces and sentiment classifiers for endangered uralic languages. the choice of finnish as the source language is ideal as finnish is part of the same language family as the endangered uralic languages. we proceeded to modify the cross-lingual transfer methodology described in the paper. when performing cross-lingual transfer learning, we select dutch as the "high-resource" source language and english to train a dutch sentiment classifier. to evaluate whether vocabulary size varies by proximity to the high-resource source language figure 1: overview of methodology. using translation lexicons, the dutch embedding space is replicated for each proxy language and aligned using muse. sentiment classifiers are then built from the embedding spaces and finetuned on sts english examples. these classifiers are evaluated on their respective language in mteb. dutch, we select four widely spoken languages as proxies for low-resource target languages: german, turkish, arabic, and mandarin. we test arabic and mandarin separately to determine how replicable cross-lingual transfer is across different languages. adopting high-resource languages as proxies allow us to experiment with varying degrees of language documentation, from the very small to the very large. our methodology is illustrated in figure 1 . we select a classic tokenizer that splits on whitespace and punctuation as an acknowledgment of the reality faced by many low-resource languages: a lack of data to train a more sophisticated tokenizer. with the help of translation lexicons, we replicate dutch word embeddings for each proxy language before aligning all word embeddings. we then create sentence embeddings, each finetuned on english data as done in @xcite . we then evaluate these sentence embeddings by injecting sentence pairs into the sentence embedding space and comparing the model's cosine similarity score with the actual similarity score using the spearman correlation @xcite . language text dutch german turkish arabic mandarin hij stierf dinsdag in osaka. er verstarb am dienstag in osaka. sall gtinii ()saka'da vefat etti. (di ; a »w@,¥ §7<@'w; ## results under the methodology described in section 2, the quality of the translations improve as the vocabulary size of the proxy language grows (see subsection c.2). the relationship between vocabulary size and the performance of the sentence embedding is logarithmic. this is evident in the fact that the greatest increases in performance occur at smaller vocabulary sizes. once the vocabulary size hits 25,000, we begin to see diminishing returns (see figure 3 and figure 4 ). the notable exception is mandarin as increasing the vocabulary size consistently results in poor performance (see figure 4 ). the poor performance in mandarin can be attributed to its prediction of a constant or nearconstant cosine similarity score (see figure 11 ). interestingly, turkish and arabic-two of the languages that are considered linguistically different from the source language dutch-outperformed german, the language figure 2: architecture of sentiment classifier. to determine whether a sentence has a positive or negative connotation, the sentence is processed through a sentence embedding layer, followed by three dense layers, a dropout layer, and a sigmoid activation function. using the procedure discussed in subsection 2.6, we compare our results against @xcite in ## conclusion genetic proximity between the source and target language may not have an impact on how well the target language performs on the sts task. we note that the performance of the target language plateaus at a vocabulary size of 25,000. this may be dependent on morphology as seen in the case of mandarin. based on data from panlex, low-resource languages lack the level of documentation deemed necessary in this study but embedding spaces experience the greatest level of improvement when vocabularies are relatively small. while word embeddings are useful in modeling language, they would not exist without a tokenizer. it can be argued that a tokenizer is just as an important area of research as word embeddings, if not more important; without a tokenizer, a model could not extract the relevant semantic features from text. future research could investigate the minimum amount of data needed to develop this foundational tool in language processing. richard socher, alex perelygin, jean wu, jason chuang, christopher d. manning, andrew ng, and christopher potts. 2013. recursive deep models for semantic compositionality over a sentiment treebank. in proceedings of the 2013 conference on empirical methods in natural language processing, pages 1631-1642, seattle, washington, usa. association for computational linguistics. c. spearman. 1904. the proof and measurement of association between two things. the american journal of psychology, 15(1):72-101. suzanne stevenson and paola merlo. 2022. beyond the benchmarks: toward human-like lexical representations. frontiers in artificial intelligence, 5. stephanie strassel and jennifer tracey. 2016. lorelei language packs: data, tools, and resources for technology development in low resource languages. in proceedings of the tenth international conference on language resources and evaluation (lrec'16), pages 3273-3280, portorož, slovenia. european language resources association (elra). ke tran. 2020. from english to foreign languages: transferring pre-trained language models. preprint, arxiv:2002.07306. torsten zesch. 2010. study of semantic relatedness of words using collaboratively constructed semantic resources. ph.d. thesis, technische universität darmstadt, darmstadt, germany. shiyue zhang, ben frey, and mohit bansal. 2022. how can nlp help revitalize endangered languages? a case study and roadmap for the cherokee language. in proceedings of the 60th annual meeting of the association for computational linguistics (volume 1: long papers), pages 1529-1541, dublin, ireland. association for computational linguistics. xiang zhang, junbo jake zhao, and yann lecun. 2015. character-level convolutional networks for text classification. corr, abs/1509.01626. xingyuan zhao, satoru ozaki, antonios anastasopoulos, graham neubig, and lori levin. 2020. automatic interlinear glossing for under-resourced languages leveraging translations. in proceedings of the 28th international conference on computational linguistics, pages 5397-5408, barcelona, spain (online). international committee on computational linguistics.