An Investigation into the Effect of Control Tokens on Text Simplification (2022)

Abstract: Recent work on text simplification has focused on the use of control tokens to further the state of the art. However, it is not easy to further improve without an in-depth comprehension of the mechanisms underlying control tokens. One unexplored factor is the tokenisation strategy, which we also explore. In this paper, we (1) reimplemented ACCESS, (2) explored the effects of varying control tokens, (3) tested the influences of different tokenisation strategies, and (4) demonstrated how separate control tokens affect performance. We show variations of performance in the four control tokens separately. We also uncover how the design of control tokens could influence the performance and propose some suggestions for designing control tokens, which also reaches into other controllable text generation tasks.

## introduction text simplification (ts) refers to reducing linguistic complexity at both syntactic and lexical levels without losing the main content @xcite . it is commonly used to increase the readability of documents intended for children @xcite , non-native speakers @xcite and people with dyslexia. the requirements for simplified outcomes may vary among audiences @xcite , for instance, depending on the characteristics of the dataset. the task can be roughly divided into sentence-level simplification @xcite and paragraph-level simplification @xcite . the two types of tasks may have different focuses, and this paper only involves sentence-level simplification. in order to fit the requirements of different user groups, some projects introduced explicit discrete prompts as control tokens to assist the model in learning from datasets and adjusting the simplifications @xcite . by adjusting the value in different control tokens, researchers can manually adjust the characteristics of the output, such as length, syntactic and lexical difficulties, etc. the control tokens are added to the beginning of the complex sentences and represent a relationship between that sentence and the desired input (such as the desired compression ratio). in addition, the numerical value also changes with the demands of the outcome. the format of the control token is: <token_value>, where token is a novel extra-vocabulary token with human interpretable meaning, and value is a numerical value indicating some relationship between the given input and output as shown in figure 1 and appendix a. the design of the control tokens is based on the need for adjustment. multiple control tokens can be applied simultaneously, and four control tokens are used in this project. although the control tokens are manually crafted, how the control tokens change the outcome remains unstudied. to explore the mechanisms of control tokens in simplification, this paper proposes the following: (1) verify the importance of control tokens in section 4.2. (2) reimplement the access @xcite ) used in the current state-of-the-art (sota) in section 3.3. (3) explore the influence of the variation of control tokens in the format in section 4.1. and finally (4) investigate the effects of the tokenization method in section 4.2. ## literature review natural language generation (nlg) is a sub-task in natural language processing. there have been attempts to build an nlg system based on handcrafted rules and to define the problem and features based on knowledge in the last century @xcite . with the development of computation power and the introduction of neural networks, more neural-network-based statistical methods were applied @xcite @xcite . one important change happened with the publishing of the transformer architecture @xcite , which inspired the "pre-train and fine-tune" paradigm. later, due to the new architecture outperforming existing ones in both performance and computation consumption, the transformer architecture and its derivatives occupied a dominant position in the nlg domain @xcite @xcite . as a sub-task of nlg, text simplification can also be regarded as monolingual machine translation @xcite . with the development of sequence-to-sequence machine translation, text simplification also drew more attention @xcite @xcite in recent years, researchers tried to introduce explicit parameters to control the simplified output @xcite @xcite . @xcite introduced four hyper-parameters in the audience-centric sentence simplification (access): the number of characters, levenshtein similarity @xcite , word rank and dependency tree depth, which are used to control the length, similarity, lexical complexity and syntactic complexity respectively. with the help of the parameters, users can modify the generated simplification based on their needs. however, these parameters may be less straightforward for lay users, and @xcite replaced the detailed parameters with simplification grades. in addition, a minor change in these parameters may significantly affect the readability and fluency of output. although the value set that maximises the benchmark scores can be given, it may be of little help to the end-users with specific requirements. further exploration of the effect and proper parameter preferences needs to be made to guide and help lay users adjust these parameters based on their needs. another novel research on the training datasets is multilingual unsupervised sentence simplification (muss) @xcite . they fine-tuned bart @xcite on their mined paraphrases datasets instead of complex-simple parallel corpora and found that with the help of access, the unsupervised model outperformed the other unsupervised text simplification models and became the latest sota. as an extension of access, the authors improved the design of control tokens and changed the tokenization strategy. they showed that performance differences between the two types of datasets might be acceptable only if the mined paraphrase dataset is good enough. training on paraphrase datasets provides more options than training solely on the supervised datasets and there is a nearly unlimited amount of unlabelled data. they also found that the performance of the combination of unsupervised and supervised training is the best, which is very similar to the pre-train and fine-tune paradigm. although multilingual tests were made in the muss, they were delivered separately and had little interference with the aim of this project. thus, there is little need to focus on their research in french and spanish. the metrics also play a vital role in evaluating the performance of models. although current metrics can hardly compete with human evaluations, they can still partially reflect the performance in certain indexes. among the popular metrics, there are reference-based metrics like bilingual evaluation understudy (bleu) @xcite and recall-oriented understudy for gisting evaluation (rouge) @xcite and non-reference-based metrics like flesch-kincaid grade level (fkgl) @xcite . currently, the most popular metric for text simplification is the system output against references and against the input sentence (sari) @xcite . sari is designed especially for text simplification tasks, which evaluates the outputs in aspects of adding, keeping and deleting. although it is found to have some deviation from human judgement, sari is still a valuable metric to evaluate simplicity @xcite . as for the non-reference-based metrics, the bert score is a bert-based metric that evaluates the similarity between input and output by calculating the correlation in the embedding space @xcite . it is found to have a high correlation with human judgement @xcite . by combining the metrics, the performance can be evaluated more comprehensively. ## discussion and future directions one phenomenon found during the optimisation section in the original project is that the score of recommended optimisation is even lower than the default values of control tokens at 0.8. a hypothesis emerged that continuous optimisation is not an ideal option to maximise the score. as shown in the first four rows in figure 2 and as for the optimal value, the most significant variation between single and combined control tokens is in dependencytreedepth ratio. the optimal value in combined control tokens in the joint and separate tokenization method is 0.35 instead of 0.6. although no direct comparison is listed in another interesting finding between sari and bert in this paper is that most bert score for optimal value is around 0.78 to 0.8. however, as shown in figure 3(b) and 3(d), there are more than 1 points that have such value, so the bert score alone cannot be used to evaluate the text simplification results. it may be a necessary but not sufficient condition for a good simplification. since the sari score is not perfect and relies on references, it is important to build non-reference-based metrics to evaluate the model on a different genre of corpora. the bert score may play a role in these new metrics. thus, this guess is worth further verification in future work. in addition to the values, as shown in