Very Low Resource Sentence Alignment: Luhya and S wahili (2022)

Abstract: Language-agnostic sentence embeddings generated by pre-trained models such as LASER and LaBSE are attractive options for mining large datasets to produce parallel corpora for low-resource machine translation. We test LASER and LaBSE in extracting bitext for two related low-resource African languages: Luhya and Swahili. For this work, we created a new parallel set of nearly 8000 Luhya-English sentences which allows a new zero-shot test of LASER and LaBSE. We find that LaBSE significantly outperforms LASER on both languages. Both LASER and LaBSE however perform poorly at zero-shot alignment on Luhya, achieving just 1.5% and 22.0% successful alignments respectively (P@1 score). We fine-tune the embeddings on a small set of parallel Luhya sentences and show significant gains, improving the LaBSE alignment accuracy to 53.3%. Further, restricting the dataset to sentence embedding pairs with cosine similarity above 0.7 yielded alignments with over 85% accuracy.

## introduction sentence alignment is the creation of parallel corpora from monolingual data @xcite . this alignment can be done manually and/or automatically. manual alignment is laborious and costly hence there has been a lot of work on automatic sentence alignment @xcite @xcite . tasks such as building using comparable corpora (bucc) focus on building parallel corpora using neural methods @xcite . essentially, sentences are aligned to the corresponding translation in another language using language agnostic sentence embeddings with the idea that sentences that are translations of each other will be close in the vector space @xcite . these sentence embeddings are generated using pre-trained models such as language agnostic sentence representation (laser) and language agnostic bert sentence embeddings (labse) @xcite @xcite . laser and labse have been used to effectively mine bitext from comparable corpora. as these pre-trained models are effective in mining bitext, we investigate how they would perform on an unseen low-resource language: luhya, marama dialect, as well as swahili. our main contributions are: ## multilingual sentence embeddings in this section, we review laser and labse. ## languages in this section, we provide a brief description of the low-resource languages experimented on: luhya (marama dialect) and swahili. the marama dialect of the luhya language, also known as olumarama, is spoken in western kenya in kakamega and vihiga region with about 43,000 speakers @xcite . the language status of marama is educational as it is used vigorous both verbally and in broadcast media @xcite . on the other hand, swahili is spoken in east and central africa including countries such as kenya, tanzania, drc, parts of uganda & rwanda and has approximately 100 million speakers @xcite . its status is national as several countries use it as their national language @xcite . these two languages are both bantu languages from the niger-congo language family (world atlas of language structures, 2021). being from the same language family, they have the same word order structure, namely sentences follow a subject-verb-object (svo) ordering. they are also both agglutinative. since both laser and labse were trained on swahili, luhya makes a very interesting zero-shot example to see how much information is transferred from the raw embeddings. ## related work with the proliferation of neural embedding techniques, there have been various efforts to align sentences in various low-resource languages. @xcite introduce vecalign which aligns sentences using laser sentence embeddings @xcite similarity as well as dynamic programming approximation. they experiment on low-resource language pairs namely: sinhala-english and nepali-english. they show that the sentences aligned using this method achieve improvement in machine translation models downstream. on the other hand, tien et al. ( 2021 ) proposed kc4align that utilises a multilingual translation system to generate embeddings. the similarity in these embeddings is used to perform paragraph alignment. sentences are aligned where the sentences appear in the paragraph alignment using similarity scores and sentence length ratio. this method was tested on the vietnamese-laos language pair. focusing on african low-resource language, @xcite extracts parallel sentences from wikipedia using multilingual sentences embeddings for swahili among other languages. regarding luhya, there has been work on building luhya datasets. @xcite work focuses on parts of speech tagging for the wanga luhya dialect whereas @xcite focuses on producing parallel datasets for several luhya dialects to english with the help of human translators. this is in contrast to our work which analyses automatic sentence alignment for luhya. ## experiments 6.1 datasets and alignment we experiment on both luhya-english and swahili-english sentence alignment. the luhya-english parallel set was created by aligning sentences from the new testament bible translations. this dataset consists of 7952 parallel sentences in the marama dialect of luhya. this bitext creation was achieved by cleaning, curating and aligning the new testament of the bible in luhya and english. after alignment, the dataset was assessed by three independent native speakers to assess the quality of alignment. this dataset is the only known digital parallel corpus in luhya-english for the marama dialect. see examples of the aligned sentences on the swahili-english dataset was sourced from the sawa dataset which contains approximately 89k parallel sentences from various domains @xcite . we sampled 10k parallel sentences from the bible. this sampling allows comparison of automatic alignment specifically in the religious domain. data cleaning involved getting rid of characters from different text encodings, removing both extra white spaces and verse numbers from all the datasets.