Noisy Parallel Data Alignment (2023)

Abstract: An ongoing challenge in current natural language processing is how its major advancements tend to disproportionately favor resource-rich languages, leaving a significant number of under-resourced languages behind. Due to the lack of resources required to train and evaluate models, most modern language technologies are either nonexistent or unreliable to process endangered, local, and non-standardized languages. Optical character recognition (OCR) is often used to convert endangered language documents into machine-readable data. However, such OCR output is typically noisy, and most word alignment models are not built to work under such noisy conditions. In this work, we study the existing word-level alignment models under noisy settings and aim to make them more robust to noisy data. Our noise simulation and structural biasing method, tested on multiple language pairs, manages to reduce the alignment error rate on a state-of-the-art neural-based alignment model up to 59.6%.

## introduction modern optical character recognition (ocr) software achieves good performance on documents in high-resource standardized languages, producing machine-readable text which can be used for many downstream natural language processing (nlp) tasks and various applications @xcite @xcite . however, attaining the same level of quality for texts in less-resourced local and non-standardized languages remains an open problem @xcite . the promise of ocr is particularly appealing for endangered languages, for which material might exist in non-machine-readable formats, such as physical books or educational materials. digitizing such material could lead to the creation of nlp technologies for such otherwise severely under-served communities @xcite . beyond the primary goal of digitizing printed material in endangered languages, the need for robust alignment tools is wider. the majority of the world's languages are being traditionally oral @xcite , which implies that to obtain textual data at scale one would need to rely on automatic speech recognition (asr), which in turn would produce invariably noisy outputs. it is worth noting that the availability of translations can significantly improve systems beyond machine translation (mt), such as ocr @xcite or asr @xcite . this creates a chicken-and-egg situation: on one hand, ocr and asr can be used to obtain noisy parallel data; on the other hand, having good quality aligned data can improve ocr or asr. in this vein, we focus on the scenario of digitizing texts in a less-resourced language along with their translations (usually high-resource and/or widely spoken) similar to @xcite . digitizing parallel documents can also be beneficial for educational purposes, as one could then create dictionaries through word-and phrase-level alignments, or ground language learning on another language (a learner's either l1 or l2). also, as @xcite showed in recent work, such parallel corpora can be meaningfully used to create mt systems. however, the process that transforms digitized books or dictionaries into parallel sentences for training mt systems requires painstaking manual intervention. in theory, the process could be semi-automated using sentence alignment methods, but in practice, the situation is very different: ocr systems tend to generate very noisy text for endangered languages (alpert-abrams, 2016, inter alia), which in turn leads to poor alignments between two parallel sides. as we show, alignment tools are particularly brittle in the presence of noise. in this work, we take the first step towards solving the above issue. we investigate the relationship between ocr noise and alignment results and build a probabilistic model to simulate ocr errors and create realistic ocr-like synthetic data. we also manually annotate a total of 4,101 gold alignments for an endangered language pair, griko-italian, in order to evaluate our methods in a real-world setting. we leverage structural knowledge and augmented data, greatly reducing the alignment error rate (aer) for all four high-and low-resource language pairs up to 59.6%. ## problem setting our work is a straightforward extension of previous word-level alignment work. given a sequence of words @xmath0, . . . , x n ) in a source language and @xmath1, . . . , y m ) in a target language, the alignment model produces alignment pairs: foot_1 @xmath2, y j ) : x @xmath3, y @xmath4 the difference with previous work is that the starting data will be the output of an ocr pipeline, hence producing noisy parallel data (x * , y * ) instead of "clean" (x, y) ones. the level of noise may vary between the two sides. ## method we create synthetic data that mimic ocr-like noise, that can be used to train/finetune alignment models. our simple yet effective method mainly consists of (i) building probabilistic models based on edit distance measures and capturing real ocr errors; (ii) creating synthetic (noisy) ocr-like data by applying our error-introducing model on clean parallel data; (iii) training or finetuning alignment models on synthetic data. ## languages and datasets we study four language pairs with varying amounts of data availability: english-french, english-german, griko-italian, and ainu-japanese. foot_6 ## experiments in this section, we present multiple experiments and demonstrate that our method results in significant aer reductions. ## analysis and discussion in this section we conduct several analyses to better understand our method. incorporating diagonal bias as shown in degradation of alignment size of synthetic data we conduct quantitative analyses as shown in figure 3 to examine awesome-align with different sizes of english-french synthetic data under both unsupervised and supervised settings. for space economy reasons, here we only discuss the results of the more challenging noisy-noisy test set. note that dramatic degradation of alignment is observed when applying ocr-like noise to clean text (see varying degrees of cer in a real-world scenario, the cer of ocred data is typically un-known due to the absence of clean text. we investigate how different degrees of cer affect alignments by creating several english-french synthetic data with varying degrees of cer, testing them on awesome-align. we elaborate on the process and results in appendix b.1. the main finding is that higher cer leads to greater aer, which is expected. however, we also find that mixing with different degrees of cer generally produces better results than a fixed cer throughout the corpus, suggesting that our augmentation approach could also work on the unknown cer real-world scenario. ## conclusion in this work, we benchmark several popular word alignment models under ocr noisy settings with high-and low-resource language pairs, conducting several studies to investigate the relationship between ocr noise and alignment quality. we propose a simple yet effective approach to create realistic ocr-like synthetic data and make the stateof-the-art neural awesome-align model more robust by leveraging structural bias. our work paves the way for future word-level alignment-related research on underrepresented languages. as part of this paper, we also release a total of 4,101 ground truth word alignment data for griko-italian, which can be a useful resource to investigate word-and sentence-level alignment techniques on practical endangered language scenarios.