TS - ANNO : An Annotation Tool to Build, Annotate and Evaluate Text Simplification Corpora (2022)

Abstract: We introduce TS-ANNO, an open-source web application for manual creation and for evaluation of parallel corpora for text simplification. TS-ANNO can be used for i) sentence–wise alignment, ii) rating alignment pairs (e.g., w.r.t. grammaticality, meaning preservation, ...), iii) annotating alignment pairs w.r.t. simplification transformations (e.g., lexical substitution, sentence splitting, ...), and iv) manual simplification of complex documents. For evaluation, TS-ANNO calculates inter-annotator agreement of alignments (i) and annotations (ii).

## introduction a large number of texts are difficult to understand for many people, e.g., people with low literacy skills, non-native speakers, or people with cognitive disabilities @xcite ). text simplification (ts) aims to make complex texts more accessible by editing their wording and syntax, while preserving the original meaning @xcite . in automatic ts, parallel corpora that align sentences from the original text with corresponding professionally simplified sentences are precious resources for training and evaluating ts systems. currently, however, high-quality corpora of this type are rare and often of comparably small size (e.g., zero hora @xcite or terence & teacher @xcite ). therefore, often resources that were not designed for ts in the first place are used to train ts systems (e.g., simple wikipedia texts @xcite ) @xcite . as text simplification is often performed on sentence-level, two further problems of ts corpora arise: missing sentence-level alignment (e.g., see newsela @xcite ) or error-prone automatic sentence alignment (e.g., see pwkp @xcite ) @xcite . furthermore, ts corpora are provided, if at all, with the alignment, e.g., wikilarge @xcite or wiki-auto @xcite . only a few corpora contain information about the actual types of simplification (simplification transformations, respective grammaticality, lexical complexity etc. of the aligned sentences, etc.) (e.g., see @xcite , simplesew corpus @xcite or terence & teacher corpus @xcite ). collecting such data is difficult but could be useful to analyze the advantages and limitations of ts systems @xcite . to facilitate the creation of manually annotated high-quality ts corpora with n:m alignments of full documents, we developed ts-anno, an opensource, language-independent, all-in-one web application. the web application supports: the main functionalities of the annotation tool are illustrated in figure 1 . a demonstration of the tool and a demo video can be found here https://ts-anno.phil.hhu.de/ . 1 the source code is also available to create an own copy of the annotation tool. in the remainder of the paper, we elaborate technical details and functionalities of ts-anno and exemplify its usage. ## ts-anno: preparing the data this section introduces implementation details, user administration, data uploading and preprocessing options of ts-anno. ## ts-anno: annotation the main functionality of ts-anno is the annotation, which comprises alignment, transformation annotation, rating and manual simplification. overall, the annotation is structured by the corpora to annotate. each corpus contains at least one document pair, which in turn consists of a complex document and a parallel simplified document (except for manual simplification). for detailed instructions on how to annotate and how to use the annotation tool, we refer to the annotation guidelines. 14 the online demo contains pre-aligned senentece pairs of asset @xcite and a few annotated document pairs of alumniportal to illustrate the functionalities of ts-anno. ## use case ts-anno was already tested by aligning german parallel web texts, e.g., the openly licensed, parallel documents of the website "alumniportal 20 https://github.com/huggingface/datasets/blob/master/templates/readme_guide.md deutschland". the documents have been uploaded with the internal web crawler and were automatically pre-processed. two annotators have manually aligned the sentence pairs and annotated them with rewriting transformations and rating aspects. 21 the demonstration version of ts-anno contains a few annotated documents of this corpus, including different alignment types, ratings and rewriting transformations to exemplify the evaluation and download options. overall, the test usage of ts-anno led to some minor improvements, which are already included in this system description. ## related work several aligned ts corpora exist, some with annotated rewriting transformations and some test sets with rated evaluation aspects (see @xcite as an overview). however, most authors have not provided a reusing option for their annotation interfaces or have only focused on one of the ts annotation tasks, e.g., aligning, simplification, rating, or transformation annotation. therefore, we present the related work per ts task. alignment. the most comparable manual alignment annotation frameworks to ts-anno are @xcite ; @xcite ; @xcite , which show also both documents (the original and simplified), in parallel and highlight paragraph blocks. however, in contrast to ts-anno, isa (tiedemann, 2006) supports only 1:1 alignment, and, in the annotation interface of @xcite , sentence borders can be changed as they are highlighted and not tagged as in our tool. simplification. for writing simplifications, several commercial and non-commercial tools, or computer-aided translation software exist to facilitate writing easy to understand texts. some of them offer more writing support than the others: e.g., languagetool for german, 22 friendlyreader for 21 both annotators are german native speakers, trained in linguistics as well as simple languages, have at least a graduation diploma, and were paid for their work with at least the minimum wage of their country of residence. the annotators were provided with the following annotation guidelines and instructions on how to use the annotation tool: https://github.com/rstodden/ts_annotation_tool/blob/- master/annotation_schema/annotationsrichtlinien_ts_annode.pdf . ## limitations so far, ts-anno has been tested only for english, spanish, french, farsi, and german. spacy, which is used for pre-processing, should support many languages, though maybe not always be with the same quality. for languages that spacy does not support, corpora have to be sentence split and tokenized before uploading them. however, it is also possible to exchange spacy with another nlp framework. the annotation tool does not support active learning yet; its current focus is on high-quality manual alignment, rating and annotation of parallel data. @xcite ; alva-manchego et al. (2020b) state that rating is not enough to evaluate ts texts, comprehension tests, measuring reading time and eye fixations are also relevant. unfortunately, ts-anno only supports rating as it is yet the most dominant approach. ## conclusion & further work we presented ts-anno, an open-source, webbased application for the purpose of facilitating the time-consuming process of building high-quality corpora for text simplification or of evaluating quality of existing corpora. the annotation tool combines relevant functionalities for building ts corpora, e.g., crawling parallel web documents, sentence-wise alignment of parallel documents, human assessment of alignment pairs, annotation of transformations applied to get the simplified sentence of the pair and evaluating the data, e.g., via inter-annotator agreement. the human assessment of simplification pairs could help to identify and filter out pairs of sentences with an increase of complexity (rather than a decrease) from the complex to the simple document. furthermore, transformation labels allow characterizing the performed changes. both information could be used to evaluate ts systems and give insights in their "black boxes". in future work, we plan to extend the simplification option by highlighting complex phrases. in addition, it would be interesting to integrate active learning or a neural alignment system, which could suggest possible sentence alignments. it is also planned to add more websites to the crawler option of ts-anno to facilitate creating corpora with texts of other languages or other domains.