WSPA lign: Word Alignment Pre-training via Large-Scale Weakly Supervised Span Prediction (2023)

Abstract: Most existing word alignment methods rely on manual alignment datasets or parallel corpora, which limits their usefulness. Here, to mitigate the dependence on manual data, we broaden the source of supervision by relaxing the requirement for correct, fully-aligned, and parallel sentences. Specifically, we make noisy, partially aligned, and non-parallel paragraphs in this paper. We then use such a large-scale weakly-supervised dataset for word alignment pre-training via span prediction. Extensive experiments with various settings empirically demonstrate that our approach, which is named WSPAlign, is an effective and scalable way to pre-train word aligners without manual data. When fine-tuned on standard benchmarks, WSPAlign has set a new state of the art by improving upon the best supervised baseline by 3.3 6.1 points in F1 and 1.5 6.1 points in AER. Furthermore, WSPAlign also achieves competitive performance compared with the corresponding baselines in few-shot, zero-shot and cross-lingual tests, which demonstrates that WSPAlign is potentially more practical for low-resource languages than existing methods.

## introduction word alignment, which aims to align the corresponding words in parallel texts, is a fundamental natural language processing (nlp) task that was originally developed as an important supporting task for statistical machine translation. while deep end-to-end models have become the mainstream solution for machine translation, word alignment is still of great importance in many nlp scenarios, such as projecting linguistic annotations @xcite and xml markups @xcite , post-editing for detecting problem of under-translation @xcite , and enforcing pre-specified terminology constraints in translation @xcite . besides, word alignment can also improve the crosslingual language pre-training @xcite . however, most existing word alignment methods rely on either manual alignment datasets or parallel corpora for training, which weakens their usefulness because of the limiting accessibility of data. an additional weakness with requiring manual data is the generalization ability because deep models trained on a dataset can fail on other datasets. therefore, these existing approaches are also limited in terms of potential cross-lingual use. on the other hand, recent studies @xcite @xcite @xcite in various fields leverage weak supervision signals in large-scale data available on the web for pre-training, which is a promising alternative to training on manual data. inspired by this, we propose to utilize wikipedia and multi-lingual pre-trained language models (plms) to make large-scale word alignment supervision for pre-training via span prediction. we broaden the source of supervision by relaxing the requirements for correct (manually made), fully-aligned (all words in a sentence pair are annotated), and parallel sentences. specifically, we make noisy (automatically made), partiallyaligned, and non-parallel paragraphs (or monolingual paragraph pairs). we make automatic partial alignment between non-parallel sentences from either co-mentions foot_1 of entities obtained through wikipedia hyperlinks or alignments of common words based on the similarity of contextual word embeddings. we name our method wspalign, which is short for weakly supervised span prediction pretraining for word alignment. with weak supervision, we are potentially able to scale the pretraining data up to millions of paragraph pairs in ## conclusion in this paper, we propose to pre-train word aligners with weakly-supervised signals that can be automatically collected. we broaden the source of supervision by relaxing the requirement for correct, fully-aligned, and parallel sentences. specifically, we make noisy, partially aligned, and non-parallel paragraphs on a large scale. experimental results in this paper show that pre-training with large-scale weakly-supervision can significantly improve existing word alignment methods and make word aligners more practical as well because no manual data is needed. we provide empirical evidence of how much large-scale span prediction pre-training can help word alignment in terms of data accessibility, the number of manual examples used, and crosslingual ability. we hope this paper can contribute to further exploiting practical word alignment techniques with large-scale weak supervision.