Improving Formality Style Transfer with Context-Aware Rule Injection (2021)

Abstract: Models pre-trained on large-scale regular text corpora often do not work well for user-generated data where the language styles differ significantly from the mainstream text. Here we present Context-Aware Rule Injection (CARI), an innovative method for formality style transfer (FST) by injecting multiple rules into an end-to-end BERT-based encoder and decoder model. CARI is able to learn to select optimal rules based on context. The intrinsic evaluation showed that CARI achieved the new highest performance on the FST benchmark dataset. Our extrinsic evaluation showed that CARI can greatly improve the regular pre-trained models’ performance on several tweet sentiment analysis tasks. Our contributions are as follows: 1.We propose a new method, CARI, to integrate rules for pre-trained language models. CARI is context-aware and can trained end-to-end with the downstream NLP applications. 2.We have achieved new state-of-the-art results for FST on the benchmark GYAFC dataset. 3.We are the first to evaluate FST methods with extrinsic evaluation and specifically on sentiment classification tasks. We show that CARI outperformed existing rule-based FST approaches for sentiment classification.

## introduction many user-generated data deviate from standard language in vocabulary, grammar, and language style. for example, abbreviations, phonetic substitutions, hashtags, acronyms, internet language, ellipsis, and spelling errors, etc are common in tweets @xcite @xcite . such irregularity leads to a significant challenge in applying existing language models pre-trained on large-scale corpora dominated with regular vocabulary and grammar. one solution is using formality style transfer (fst) @xcite , which aims to transfer the input text's style from the informal domain to the formal domain. this may improve the downstream nlp applications such as information extraction, text classification and question answering. a common challenge for fst is low resource @xcite @xcite . therefore, approaches that integrate external knowledge, such as rules, have been developed. however, existing work @xcite deploy context-insensitive rule injection methods (ciri). as shown in figure 1 , when we try to use ciri-based fst as the preprocessing for user-generated data in the sentiment classification task, according to the rule detection system, "extro" has two suggested changes "extra" or "extrovert" and "intro" corresponds to either "introduction" or "introvert." the existing ciri-based fst models would arbitrarily choose rules following first come first served (fcfs). as such, the input "always, always they think i an extro, but im a big intro actually" could be translated wrongly as "they always think i am an extra, but actually, i am a big introduction." this leads to the wrong sentiment classification since the fst result completely destroys the original input's semantic meaning. in this work, we propose context-aware rule injection (cari), an end-to-end bert-based encoder and decoder model that is able to learn to select optimal rules based on context. as shown in figure 1 , cari chooses rules based on context. with cari-based fst, pre-trained models can perform better on the downstream natural language processing (nlp) tasks. in this case, cari outputs the correctly translated text "they always think i am an extrovert, but actually, i am a big introvert," which helps the bert-based classification model have the correct sentiment classification. in this study, we performed both intrinsic and extrinsic evaluation of existing fst models and compared them with the cari model. the intrinsic evaluation results showed that cari improved the state-of-the-art results from 72.7 and 77.2 to 74.31 and 78.05, respectively, on two domains of a fst benchmark dataset. for the extrinsic evaluation, we introduced several tweet sentiment analysis tasks. considering that tweet data is typical informal user-generated data, and regular pre-trained models are usually pre-trained on formal english corpora, using fst as a preprocessing step of tweet data is expected to improve the performance of reg- user-generated input: always, always they think i an extro, but im a big intro actually rule detection system @xmath0, always they think i an extro, but im a big intro actually always, always they think i an extra, but im a big intro actually always, always they think i an extra, but i am a big intro actually always, always they think i an extra, but i am a big introduction actually info1: i an extra , but info2: i an extrovert , but info3: , but i am a big info4: a big introduction actually info5: a big introvert actually info1 + info2 + info3 + info4 + info5 encoder-decoder model user-generated input + ciri output user-generated input + cari output input + ciri output: always, always they think i an extro, but im a big intro actually [sep] always, always they think i an extra, but i am a big introduction actually ciri fst result: they always think i am an extra, but actually, i am a big introduction (incorrect fst result which fails the downstream tasks) input + cari output: always, always they think i an extro, but im a big intro actually [sep] i an extra , but [sep] i an extrovert , but [sep] , but i am a big [sep] a big introduction actually [sep] a big introvert actually cari fst result: they always think i am an extrovert, but actually, i am a big introvert (correct fst result which helps the downstream tasks) cari fst result ciri fst result (context window @xmath1.g. tweet classification) ciri output cari output user-generated input ciri fst result cari fst result input input output output incorrect classification correct classification incorrect classification input output figure 1: an example of using context-insensitive rule injection (ciri) and context-aware rule injection (cari) fst models. ciri models are not context aware and therefore select rules arbitrarily and in this case, apply the rules first come first serve (fcfs). the errors introduced ("extra" and "introduction") in the ciri model impact the downstream nlp tasks, and in this case leading to the incorrect sentiment classification. in cari, rules are associated with context and through training, cari can learn to choose the right rules according to the context. this leads to improved fst thereby improves the downstream sentiment classification tasks. we regard measuring such improvement as the extrinsic evaluation. the extrinsic evaluation results showed that using cari model as the prepocessing step improved the performance for both bert and roberta on several downstream tweet sentiment classification tasks. our contributions are as follows: ## related work rule-based formality style transfer in the past few years, style-transfer generation has attracted increasing attention in nlp research. early work transfers between modern english and the shakespeare style with a phrase-based machine translation system @xcite . recently, style transfer has been more recognized as a controllable text generation problem @xcite , where the style may be designated as sentiment @xcite , tense @xcite , or even general syntax @xcite . formality style transfer has been mostly driven by the grammarly's yahoo answers formality corpus (gyafc) @xcite . since it is a parallel corpus, fst usually takes a seq2seq-like approach @xcite . existing research attempts to integrate the rules into the model because the gyafc is low resource. however, rule matching and selection are context insensitive in previous methods @xcite . this paper focuses on developing methods for contextaware rule selection. evaluating style transfer previous work on style transfer @xcite @xcite has repurposed the machine translation metric bleu @xcite and the paraphrase metric pinc @xcite for evaluation. @xcite introduced three evaluation metrics based on cosine similarity, language model and logistic regression. they also introduced human judgments for adequacy, fluency and style @xcite . @xcite evaluated formality, fluency and meaning on the gyafc dataset. recent work on the gyafc dataset @xcite ) mostly used bleu as the evaluation metrics for fst. however, all aforementioned work focused on intrinsic evaluations. our work has in addition evaluated fst extrinsically for downstream nlp applications. lexical normalisation lexical normalisation @xcite is the task of translating non-canonical words into canonical ones. like fst, lexical normalisation can also be used to preprocess user-generated data. the monoise model (van der goot and van noord, 2017) is a state-of-the-art model based on featurebased random forest. the model ranks candidates provided by modules such as a spelling checker (aspell), a n-gram based language model and word embeddings trained on millions of tweets. unlike fst, monoise and other lexical normalisation models can not change data's language style. in this study, we explore the importance of language style transfer for user-generated data by comparing the results of monoise and fst models on tweets nlp downstream tasks. improving language models' performance for user-generated data user-generated data often deviate from standard language. in addition to the formality style transfer, there are some other ways to solve this problem @xcite . finetuning on downstream tasks with a user-generated dataset is most straightforward, but this is not easy for many supervised tasks without a large amount of accurately labeled data. another method is to fine-tune pre-trained models on the target domain corpora @xcite . however, it also requires sizable training data, which could be resource expensive @xcite @xcite . ## approach for the downstream nlp tasks where input is user-generated data, we first used the fst model for preprocessing, and then fine-tuned the pretrained models (bert and roberta) with both the original data d ori and the fst data d f st , which were concatenated with a special token @xcite , forming an input like (d ori [sep ]d f st ). for the formality style transfer task, we use the bert-initialized encoder paired with the bertinitialized decoder @xcite as the seq2seq model. all weights were initialized from a public bert-base checkpoint @xcite . the only variable that was initialized randomly is the encoder-decoder attention. here, we describe cari and several baseline methods of injecting rules into the seq2seq model. ## conclusions in this work, we proposed the context-aware rule injection(cari), an innovative method for formality style transfer (fst) by injecting multiple rules into an end-to-end bert-based encoder and decoder model. the intrinsic evaluation showed our cari method achieved the highest performance with previous metrics on the fst benchmark dataset. besides, we were the first to evaluate fst methods with extrinsic evaluation and specifically on the sentiment classification tasks. the extrinsic evaluation results showed that using the cari-based fst as the preprocessing step outperformed existing rule-based fst approaches. our results showed the rationality of adding such extensive evaluation.