Neural semi- M arkov CRF for Monolingual Word Alignment (2021)

Abstract: Monolingual word alignment is important for studying fine-grained editing operations (i.e., deletion, addition, and substitution) in text-to-text generation tasks, such as paraphrase generation, text simplification, neutralizing biased language, etc. In this paper, we present a novel neural semi-Markov CRF alignment model, which unifies word and phrase alignments through variable-length spans. We also create a new benchmark with human annotations that cover four different text genres to evaluate monolingual word alignment models in more realistic settings. Experimental results show that our proposed model outperforms all previous approaches for monolingual word alignment as well as a competitive QA-based baseline, which was previously only applied to bilingual data. Our model demonstrates good generalizability to three out-of-domain datasets and shows great utility in two downstream applications: automatic text simplification and sentence pair classification tasks.

## introduction monolingual word alignment aims to align words or phrases with similar meaning in two sentences that are written in the same language. it is useful for improving the interpretability in natural language understanding tasks, including semantic textual similarity @xcite and question answering @xcite . monolingual word alignment can also support the analysis of human editing operations (figure 1 ) and improve model performance for text-to-text generation tasks, such as text simplification @xcite and neutralizing biased language @xcite . it has also been shown to be helpful for data augmentation and label projection with canadian collaborators, lloyd went on to conduct laboratory simulations of his model. ## related work word alignment has a long history and was first proposed for statistical machine translation. the most representative ones are the ibm models @xcite , which are a sequence of unsupervised models with increased complexity and implemented the giza++ toolkit @xcite . many more works followed, such as fastalign @xcite . @xcite also used a globally normalized log-linear model for discriminative word alignment. @xcite proposed a hidden semi-markov model to handle both continuous and noncontinuous phrase alignment. these statistical methods promoted the development of monolingual word alignment @xcite @xcite . @xcite proposed a crf aligner following @xcite , then extended it to a semi-crf model for phrase-level alignments @xcite . @xcite designed a simple system with heuristic rules based on word similarity and contextual evidence. neural methods have been explored in the past decade primarily for bilingual word alignment. some early attempts @xcite did not match the performance of giza++, but recent transformer-based models started to outperform. @xcite proposed a multi-task framework for machine translation and word alignment, while @xcite designed an alignment layer on top of transformer for machine translation. both can be trained without word alignment annotations but rely on millions of bilingual sentence pairs. as for supervised methods, @xcite extracted representations from the transformerbased mt system, then used convolutional neural network to incorporate neighboring words for alignment. @xcite proposed a span prediction method and formulated bilingual word alignment as a squad-style question answering task, then solved it by fine-tuning multilingual bert. we adapt their method to monolingual word alignment as a new state-of-the-art baseline ( §5.1). some monolingual neural models have different settings from this work. @xcite introduced pointer networks for long, sentence-or clause-level alignments. @xcite utilized constituency parsers for compositional and non-compositional phrase alignments. culkin et al. (2021) considered span alignment for framenet @xcite annotations and treated each span pair as independent prediction. ## neural semi-crf alignment model in this section, we first describe the problem formulation for monolingual word alignment, then present the architecture of our neural semi-crf word alignment model (figure 2 ). ## a multi-genre benchmark for monolingual word alignment in this section, we present the manually annotated multi-genre monolingual word alignment (mul-timwa) benchmark that consists of four datasets of different text genres. as summarized in multimwa-mtref. we create this dataset by annotating 3,998 sentence pairs from the mtreference @xcite , which are human references used in a machine translation task. the original labels in mtreference were annotated by crowd-sourcing workers on amazon mechanical turk following the guideline from @xcite . in an early pilot study, we discovered that these crowd-sourced annotations are noisy and contain many obvious errors. it only gets 73.6/96.3/83.4 for precision/recall/f 1 on a random sample of 100 sentence pairs, when compared to the labels we manually corrected. to address the lack of reliable annotation, we hire two in-house annotators to correct the original labels using goldalign foot_3 @xcite , an annotation tool for monolingual word alignment. both annotators have linguistic background and extensive nlp annotation experience. we provide a three-hour training session to the the annotators, during which they are asked to align 50 sentence pairs and discuss until consensus. following previous work, we calculate the inter-annotator agreement as 84.2 of f 1 score for token-level nonidentical alignments by comparing one annotator's annotation against the other's. the alignments between identical words are usually easy for human annotators. after merging the the labels from both annotators, we create a new split of 2398/800/800 for train/dev/test set. to ensure the quality, an adjudicator further exams the dev and test sets and constructs the final labels. multimwa-newsela. newsela corpus @xcite consists of 1,932 english news articles and their simplified versions written by professional editors. it has been widely used in text simplification research @xcite @xcite . we randomly select 500 complex-simple sentence pairs from the test set of newsela-auto @xcite ), foot_4 which is the newest sentencealigned version of newsela. 214 of these 500 pairs contain sentence splitting. an in-house annotator foot_5 labels the word alignment by correcting the outputs from giza++ @xcite . ## experiments in this section, we present both in-domain and outof-domain evaluations for different word alignment models on our multiwma benchmark. we also provide a detailed error analysis of our neural semi-crf model and an ablation study to analyze the importance of each component. ## downstream applications in this section, we apply our monolingual word aligner to some downstream applications, including both generation and understanding tasks. ## conclusion in this work, we present the first neural semi-crf word alignment model which achieves competitive performance on both in-domain and outof-domain evaluations. we also create a manually annotated multi-genre monolingual word alignment (multimwa) benchmark which is the largest and of higher quality compared to existing datasets.