Low-resource Bilingual Dialect Lexicon Induction with Large Language Models (2023)

Abstract: Bilingual word lexicons map words in one language to their synonyms in another language. Numerous papers have explored bilingual lexicon induction (BLI) in high-resource scenarios, framing a typical pipeline that consists of two steps: (i) unsupervised bitext mining and (ii) unsupervised word alignment. At the core of those steps are pre-trained large language models (LLMs).In this paper we present the analysis of the BLI pipeline for German and two of its dialects, Bavarian and Alemannic. This setup poses a number of unique challenges, attributed to the scarceness of resources, relatedness of the languages and lack of standardization in the orthography of dialects. We analyze the BLI outputs with respect to word frequency and the pairwise edit distance. Finally, we release an evaluation dataset consisting of manual annotations for 1K bilingual word pairs labeled according to their semantic similarity.

## introduction learning in low-resource settings is one of the key research directions for modern natural language processing (nlp; @xcite . the omnipresent pre-trained language models support high-resource languages by using increasingly large amounts of raw and labeled data. however, data scarcity hinders the training and evaluation of nlp models for less-resourced languages. at the same time, the participation of native speakers of different languages in the world of digital technologies increases the demand for supporting more language varieties. this encourages studies to explore suitable transfer learning and crosslingual techniques. local varieties (dubbed as dialects) may fall under the umbrella of low-resource languages. processing dialects faces unique challenges that should be addressed from a new perspective. large volumes of writing in dialects such as newspapers or fiction are rarely produced and access to conversational data in social media is limited and difficult to reliably collect. besides, dialects are non-standardized, they lack unified spelling rules and exhibit a high degree of variation @xcite . finally, dialects may additionally show a significant rate of code-mixing to standard languages @xcite . the mainstream of cross-lingual transfer research towards low-resource languages, e.g., @xcite , builds upon cross-lingual representations, namely static embeddings @xcite or multilingual pre-trained language models @xcite . as shown by @xcite various factors can influence the performance, including the degree of relatedness to a pre-training language and the script. as there is no winning technique for all languages, it is important to understand how cross-lingual representations act for each particular language or a language family and whether the results in processing standard languages are transferable to its dialects. in this paper, we focus on the ability of crosslingual models to make semantic similarity judgments in german and two of its dialects, namely bavarian (iso 639-3:bar) spoken in south germany, austria, south tyrol, and alemannic (iso 639-3:gsw) spoken in switzerland, swabia, parts of tyrol, liechtenstein, alsace, and italian re-gions. using the available raw data in wikipedia (section 3) we induce two bilingual lexicons, mapping words from bavarian / alemannic to german. to do so, we first mine bitext sentences (section 4) and exploit machine translation aligners next (section 5). the output lexicon exhibits an evident tendency for a german word to be aligned to multiple dialect synonyms due to spelling variations. finally, we manually evaluate the output of each step: we evaluate the semantic similarity in (i) 1,500 bilingual sentence pairs according to the likert scale and (ii) 1,000 bilingual word pairs according to a binary scale. our results demonstrate the discrepancy between natural writing and linguistic dictionaries. to sum up, this paper explores the following research question (rq): how effective are standard pipelines for inducing bilingual lexicons for german dialects, and what factors influence their performance? to answer this question we make the following contributions: (i) we conduct a thorough analysis of cross-lingual models' behavior in two tasks of bitext mining and word alignment for the german language and two of its dialects. (ii) we release the evaluation datasets for bitext mining (1,500 samples each) and for bilingual lexicon induction (1,000 samples which). we make mined bitext dataset and induced bilingual lexicons for the bavarian and alemannic dialects publicly available. (iii) we publish the code to reproduce bitext extraction and word alignment in open access. foot_0 2 related work nlp for german dialects. previous efforts in processing german dialects mainly concentrate on speech processing. bastat comprises the recordings of spoken conversational speech from main areas of spoken german @xcite . @xcite build a parallel corpus of spoken alemannic dialect, in which a sentence in german is matched with spoken and written translations into eight dialects. archimob is a general domain spoken corpus equipped with transcriptions and part-of-speech labeling @xcite . in the domain of written text processing, machine translation techniques have been applied to re-write sentences from dialect to standard german @xcite @xcite . other works tackle sentiment classification @xcite , part-of-speech tagging @xcite and dialect identification tasks @xcite . @xcite have collected a bilingual bavarian-german lexicon using the knowledge of facebook users, while @xcite hire expert native speakers to build a bilingual alemannic-german lexicon. language resources used to collect raw dialect data are wikipedia, social media (grubenmann et al., 2018), regional newspapers, and fiction @xcite . for a more comprehensive review, we refer to the concurrent survey of @xcite . bitext mining. sentence representations are core to mining bitext (dubbed as parallel or comparable datasets) in an unsupervised fashion @xcite . @xcite show that [cls]-pooling with multi-lingual encoders performs reasonably well for the task. most recent studies proposed learning sentence embeddings from encoder-decoder models with a machine translation objective @xcite , by extending a monolingual sentence model to cross-lingual encoding with knowledge distillation @xcite , or from dual encoders with a translation ranking loss @xcite . bitext datasets collected from wikipedia @xcite and the common crawl corpus @xcite serve to train machine translation models @xcite and to improve crosslingual methods for structured prediction @xcite . @xcite show that cross-lingual sentence models scale across unseen languages. we adopt the recent state-of-the-approach of (reimers and gurevych, 2020), which scores sentences embeddings, obtained from cross-lingual embedders, with cosine similarity measure in order to retrieve most similar sentence pairs. ## data wikipedia offers articles written in more than 300 languages. 2 it is recognized that some parts of wikipedia are human-translated @xcite ; examples are shown in ( we used the wikipedia2corpus foot_5 tool to extract raw sentences. the texts were split into sentences and tokenized with the somajo sentence splitter and tokenizer foot_6 @xcite . the sentences were filtered by the 5-to-25 token range. incomplete sentences were removed according to simple heuristics, such as the number of opening and closing brackets or the presence of a bullet point. sentences containing non-german characters (e.g. letters from greek, cyrillic, and hebrew alphabets) were filtered out. the left-hand part of ## bitext mining method. we start from the assumption parallel sentences are most often found on parallel pages, e.g. pages that are inter-linked between wikipedias in two languages. we collect interlingual links between pages in dialect wikipedia and german wikipedia. overall, we found 11k parallel pages for bavarian and 32k parallel pages for alemannic out of 43k and 71k, correspondingly. given two parallel pages split into sentences, we embed each sentence with a language model. for each dialect sentence, we retrieve the nearest neighbors using the cosine similarity. ta-ble 2 provides examples of the found parallel sentences and the corresponding cosine similarity values. we aim to select the best-performing embedding model and the optimal cutoff value. models. we leverage the sentencetransformer toolkit foot_7 (reimers and gurevych, 2020) for bitext mining. the experiments are with the following mono-and multi-lingual encoders and sentence models released as a part of huggingface library foot_8 @xcite ): human evaluation. we sampled two random sets of 1,500 bitext instances with labse similar-ity values in the [0.4; 0.95] range to be manually labeled for semantic similarity and further justifications (see next and appendix for details). we start from the labse model since it is the current state-of-the-art model that has been shown to produce high-quality sentence embeddings @xcite . these embeddings capture both semantic and syntactic information, making them useful for a range of natural language processing tasks, including bitext mining. furthemore, labse was trained on a large-scale multilingual corpus, which makes it more robust and better able to handle variations in language and text structure @xcite . the annotation schema utilized in our study is a five-point likert scale, with a score of 5 indicating equivalence between the dialect sentence and the german sentence, and a score of 1 indicating no relation. annotators were instructed to provide justifications for assigning scores that deviated from 5, by assessing the factual similarity between two given sentences, considering whether one sentence provided more information than the other. additionally, annotators were asked to identify any significant differences in grammatical structure between the two sentences. the annotation instructions are provided in section a. the likert scale is a standardized approach to measuring sentence similarity, providing a more balanced set of response options when compared to binary judgments @xcite . the annotations were carried out by a native german speaker with a linguistic background and significant exposure to dialects. 12 to ensure the quality of annotations, a smaller sample of 200 sentences was labeled by a second annotator, one of the authors, who is fluent in german. the annotators were instructed to abstain from labeling sentence pairs with a likert scale if they lacked a full understanding of the content, if the sentence was written in standard german rather than the dialect, or if the sentence contained a mixture of both. the inter-annotator agreement between the two annotators yielded a score of 0.80/0.78 for exact match and pearson correlation, respectively, for bavarian, and 0.9/0.6 for alemannic. notably, the primary source of confusion between the annotators was in labeling sentence pairs with scores that were in close proximity, specifically (2,3) and (3,4). notably, there were no instances in which the annotators disagreed and assigned opposite scores of 1 and 5. additionally, the annotators were instructed to reject incomplete sentences or those not written in dialect, resulting in the rejection of 83 and 162 sentences, respectively. the remaining 1,417 and 1,338 sentence pairs in bavarian and alemannic were included for further analysis. the results from the human annotation show that the distribution of labels is different for the two dialects: 1,254 sentences were labeled as similar (5) or near similar (4) for bavarian and almost twice as less, 644 -for alemannic. in the 250 alemannic sentences marked with the label 3, the annotator pointed out that bitext sentences differ in minor factual details. sentences in bavarian differ less from their german counterparts, so that fewer than 100 sentences are marked as having differences in minor factual details. there are 250/350 bavarian/alemannic sentences labeled as using different grammatical structures such as active vs passive, imperfect vs perfect. in summary, based on this annotation study, we conclude that the authors of the bavarian wikipedia are more inclined towards literal translation, while the authors of the alemannic wikipedia rely less on translation. model comparison. many of the retrieved sentence pairs have high similarity values. for instance, labse assigns the scores of 0.8 or above to 42% and 24% of the dataset for bavarian and alemannic, correspondingly. overall, the distribution of cosine values tends to be skewed to higher values for all embedders. gbertlarge-sts-v2 shows the least reasonable performance: the average similarity value is 0.98 and the standard deviation is close to 0.01 for both dialects, leaving no discriminative power to se-lect a precise cutoff threshold. this may happen due to over-fitting to semantic similarity tasks. the choice of pooling strategy does not affect the performance of mbert: mbert+[cls] and mbert+[mean] output strongly correlated cosine values (0.81 and 0.82 for bavarian and alemannic). this is not the case for gbert, for which both [cls] and [mean] pooling strategies lead to less correlated results (≈0.5 for both dialects). we include the mbert, gbert, and labse models in our comprehensive comparison of their abilities in bitext mining and bilingual lexicon induction (section 5). the resulting annotated dataset helps to evaluate whether the embedders can judge semantic similarity and assign lower scores to unrelated sentences. at the same time, we may use it to calibrate the cutoff threshold, which distinguishes between similar and unrelated sentence pairs. figure 3 and figure 4 in appendix c show the cosine similarity values, derived with mbert+[cls], gbert+[cls], and labse models, grouped according to likert scale values. although none of these models can divide the data into five groups, there is more evidence that labse better separates unrelated sentences (scores 1, 2) from nearly similar or similar sentences (scores 4, 5) leaving somewhat similar sentences (score 3) in between. after careful consideration, we have chosen to use labse in subsequent bli experiments, setting the cutoff for the cosine similarity of nearly similar sentences to 0.7. ## bilingual lexicon induction method. we use the state-of-the-art awesomealign toolkit 13 @xcite with mbert and gbert as backbone models. awesome-align supports an unsupervised mode, so there is no need to fine-tune the models on the parallel data. the word alignments are extracted from parallel sentences by evaluating the similarity between word representations. awesome-align produces one-to-one alignment by default. when the source dialect sentence uses the perfect tense and the target german sentence uses the preterite tense, in the vast majority of cases, the auxiliary verbs align with the preterite verb. we feed the extracted parallel dialect-german sentences to the aligner. the outputs are word pairs, in which one of the words is written in dialect and the other in german (see next, we use several strategies to evaluate collected word pairs. we excluded word pairs that contained a non-word token, such as a number, typographical symbol, or punctuation mark. previous research has demonstrated that the performance of bli methods is highly dependent on word frequency, with higher frequency source words generally resulting in more accurate translations @xcite . to account for this, and to increase coverage of low-frequency words, we employed a stratified sampling approach for word selection in our evaluation. specifically, we computed the frequency of each dialect word in wikipedia and divided word pairs into four groups based on quartiles of dialect word frequency. from each group, we randomly selected 250 word pairs for further analysis. dictionary-based evaluation. to the best of our knowledge, there are no high-quality bavarian-german or alemannic-german lexicons, that can be easily accessed for computational experiments, so we turn to community-based resources. glosbe foot_13 is a collection of community-maintained dictionaries, including bavarian-german and alemannic-german dictionaries. since glosbe does not provide an api, we manually look up german words and record the suggested translations into dialects. the mismatch between the induced translation and the dictionary-based translation is mainly caused by orthographic variations (see human evaluation. in addition to the dictionary-based evaluation, we also performed a human evaluation of the same word pairs using a binary scale to assess semantic similarity. our aim was to determine whether a german word is a correct translation of a dialect word. the word pairs were presented without the surrounding context, and annotators were given the option to reject a word pair if they did not understand the dialect word. the use of a binary scale was chosen because it simplifies the assessment of semantic similarity and provides a clear indication of whether a word pair is a correct translation or not. the same annotators who participated in the evaluation of the bitext (section 4) were recruited for the task. we provided annotators with guidelines that are detailed in appendix b. to assess the level of agreement between annotators, we included a control sample consisting of 200 word pairs for each dialect. the exact match between annotators was high, with a score of 0.96 for bavarian and 0.85 for alemannic. disagreements between annotators were mainly caused by judgments of overlapping words (turm -kirchturm, steeple -church steeple, in bavarian). the evaluation of bli through human annotation is presented in interestingly, for mid-frequency words, one of the main sources of errors is the alignment of words that may be used in similar contexts but are not synonyms. for example, the word pair "soizsään -mineralquellen" in bavarian, which translates to "salt lakes -mineral springs" in german, was found to be misaligned. overall, the annotation study identified 860 and 774 out of 1,000 baseline. we use supervised muse embeddings @xcite as a baseline for bilingual lexicon induction. we employ fasttext embeddings, pre-trained on mono-lingual wikipedia @xcite , with identical character words as seeds. following the project's guidelines, 15 we set up the dialect embeddings as the bavarian german glosbe (@xmath0, zaum bavarian german muse(@xmath1 ## conclusion and future work the project developed a two-stage pipeline for inducing bilingual lexicons for german and its dialects, bavarian and alemannic. the first stage involved extracting parallel sentences from public data, specifically wikipedia, while the second stage used an alignment tool to induce word pairs from these parallel sentences. both stages relied heavily on pre-trained llms, which were calibrated based on the results of annotation studies that judged the semantic similarity between extracted sentences and induced word pairs. returning to the research question raised, we may conclude that existing llms have a certain capacity for inducing bilingual lexicons. our results have identified two key factors that in-fluence their performance: (i) whether the pretraining included multilingual or dialect data, and (ii) whether the model was trained with a taskspecific objective. our evaluations demonstrate that the german gbert is surpassed in both tasks, indicating that its monolingual pre-training is insufficient to effectively process related dialects. however, the main conundrum that remains is developing linguistic pipelines to process diverse and non-standardized dialect data. the development of dialect-specific tools such as lemmatizers, taggers and phonemizers can help improve the accuracy and consistency of bilingual lexicon induction. future work includes exploring the effect of fine-tuning cross-lingual llms on german and dialect data for bilingual lexicon induction, differentiating between several bavarian/alemannic dialects, and extending the experiments to other german dialects. limitations. while our study provides a comprehensive evaluation of induced bilingual lexicons for the bavarian-german and alemannic-german language pairs, there are some limitations to our approach. these limitations come with the low-resource setup.