Neural CRF Model for Sentence Alignment in Text Simplification (2020)

Abstract: The success of a text simplification system heavily depends on the quality and quantity of complex-simple sentence pairs in the training corpus, which are extracted by aligning sentences between parallel articles. To evaluate and improve sentence alignment quality, we create two manually annotated sentence-aligned datasets from two commonly used text simplification corpora, Newsela and Wikipedia. We propose a novel neural CRF alignment model which not only leverages the sequential nature of sentences in parallel documents but also utilizes a neural sentence pair model to capture semantic similarity. Experiments demonstrate that our proposed approach outperforms all the previous work on monolingual sentence alignment task by more than 5 points in F1. We apply our CRF aligner to construct two new text simplification datasets, Newsela-Auto and Wiki-Auto, which are much larger and of better quality compared to the existing datasets. A Transformer-based seq2seq model trained on our datasets establishes a new state-of-the-art for text simplification in both automatic and human evaluation.

## neural crf sentence aligner we propose a neural crf sentence alignment model, which leverages the similar order of content presented in parallel documents and captures editing operations across multiple sentences, such as splitting and elaboration (see figure 1 for an example). to further improve the accuracy, we first align paragraphs based on semantic similarity and vicinity information, and then extract sentence pairs from these aligned paragraphs. in this section, we describe the task setup and our approach. ## constructing alignment datasets to address the lack of reliable sentence alignment for newsela @xcite and wikipedia @xcite , we designed an efficient annotation methodology to first manually align sentences between a few complex and simple article pairs. then, we automatically aligned the rest using our alignment model trained on the human annotated data. we created two sentence-aligned parallel corpora (details in §5), which are the largest to date for text simplification. ## evaluation of sentence alignment in this section, we present experiments that compare our neural sentence alignment against the stateof-the-art approaches on newsela-manual ( §3.1) and wiki-manual ( §3.2) datasets. ## experiments on automatic sentence simplification in this section, we compare different automatic text simplification models trained on our new parallel corpora, newsela-auto and wiki-auto, with their counterparts trained on the existing datasets. we establish a new state-of-the-art for sentence simplification by training a transformer model with initialization from pre-trained bert checkpoints. ## related work text simplification is considered as a text-totext generation task where the system learns how to simplify from complex-simple sentence pairs. there is a long line of research using methods based on hand-crafted rules @xcite , statistical machine translation @xcite @xcite , or neural seq2seq models @xcite @xcite . as the existing datasets were built using lexical similarity metrics, they frequently omit paraphrases and sentence splits. while training on such datasets creates conservative systems that rarely paraphrase, evaluation on these datasets exhibits an unfair preference for deletion-based simplification over paraphrasing. sentence alignment has been widely used to extract complex-simple sentence pairs from parallel articles for training text simplification systems. previous work used surface-level similarity metrics, such as tf-idf cosine similarity @xcite @xcite , jaccard-similarity @xcite , and other lexical features @xcite . then, a greedy @xcite or dynamic programming @xcite algorithm was used to search for the optimal alignment. another related line of research @xcite @xcite @xcite aligns parallel sentences in bilingual corpora for machine translation. ## conclusion in this paper, we proposed a novel neural crf model for sentence alignment, which substantially outperformed the existing approaches. we created two high-quality manually annotated datasets (newsela-manual and wiki-manual) for training and evaluation. using the neural crf sentence aligner, we constructed two largest sentencealigned datasets to date (newsela-auto and wiki-auto) for text simplification. we showed that a bert-initalized transformer trained on our new datasets establishes new state-of-the-art performance for automatic sentence simplification.