S imple NER Sentence Simplification System for GEM 2021 (2021)

Abstract: This paper describes SimpleNER, a model developed for the sentence simplification task at GEM-2021. Our system is a monolingual Seq2Seq Transformer architecture that uses control tokens pre-pended to the data, allowing the model to shape the generated simplifications according to user desired attributes. Additionally, we show that NER-tagging the training data before use helps stabilize the effect of the control tokens and significantly improves the overall performance of the system. We also employ pretrained embeddings to reduce data sparsity and allow the model to produce more generalizable outputs.

## introduction sentence simplification aims at reducing the linguistic complexity of a given text, while preserving all the relevant details of the initial text. this is particularly useful for people with cognitive disabilities @xcite , as well as for second language learners and people with low-literacy levels @xcite . text and sentence simplification also play an important role within nlp. simplification has been utilized as a preprocessing step in larger nlp pipelines, which can greatly aid learning by reducing vocabulary and regularizing of syntax. in our model, we use control tokens to tune a seq2seq transformer model @xcite for sentence simplification. we take character length compression, extent of paraphrase, and lexical & syntactic complexity as attributes to gauge the transformations between complex and simple sentence pairs. we then represent each of these attributes as numerical measures, which are then added to our data. we show that this provides a considerable improvement over as-is transformer approaches. the use of control tokens in seq2seq models for sentence simplification has been explored before @xcite . but this approach has shown to add data sparsity to the system. this is because the model is required to learn the distribution of the various control tokens and the expected outputs across the ranges of each control token. to mitigate this sparsity, we process our data to replace named entities with respective tags using an ner tagger. we show that this reduces the model vocabulary and allows for greater generalization. to further curb the data sparsity, we make use of pre-trained embeddings as initial input embeddings for model training. our code is publicly available here. foot_0 2 background ## results we report the bleu and sari scores on the test and validation splits of the asset & turkcorpus datasets for each of the four models ( simplener shows a better retention of named entities from the source sentence than the baseline model (example 1, we also noted that using ner tagging can hamper certain outputs: while decoding, if the model generates an ner-tag that either has a type or index mismatch with the original ne token, then the tag remains in the output even after ner-untagging (example 3, ## social impact the following is a summary of the response submitted with our output and model card submission to the gem 2021 modelling shared task. ## conclusion in this paper, we report the performance of four seq2seq transformer models on the sentence simplification task of gem 2021 under two tracks: as-set and turkcorpus. we show that individually using pre-trained embeddings and ner-replaced data substantially boosts the performance of a transformer model assisted by control tokens. the ner tagging prevents the model from replacing important nes with low rank tokens also, using pretrained embeddings lets the model access a larger and fine-grained content-word vocabulary for simplification, despite training the model on relatively small data. when put together, the two approaches give rise to a much higher overall performance on the task.