Detecting Text Formality: A Study of Text Classification Approaches (2023)

Abstract: Formality is one of the important characteristics of text documents. The automatic detection of the formality level of a text is potentially beneficial for various natural language processing tasks. Before, two large-scale datasets were introduced for multiple languages featuring formality annotation—GYAFC and X-FORMAL. However, they were primarily used for the training of style transfer models. At the same time, the detection of text formality on its own may also be a useful application. This work proposes the first to our knowledge systematic study of formality detection methods based on statistical, neural-based, and Transformer-based machine learning methods and delivers the best-performing models for public usage. We conducted three types of experiments – monolingual, multilingual, and cross-lingual. The study shows the overcome of Char BiLSTM model over Transformer-based ones for the monolingual and multilingual formality classification task, while Transformer-based classifiers are more stable to cross-lingual knowledge transfer.

## introduction according to @xcite , five different types of text formality are commonly identified in linguistics: frozen style, formal style, consultative style, casual style, and intimate style. the correct use of style is important for fluent human communication and, therefore, for fluent human-to-machine communication and various natural language processing (nlp) systems. the examples of formal and informal samples for english, brazilian portuguese, french, and italian languages are provided in these examples are taken from two only currently available text collections with formality annotation are gyafc @xcite and x-formal @xcite . however, these datasets were primarily introduced for the task of style transfer. in this paper, we propose to look at these data sets from a different angle. even for the evaluation of the results of formality style transfer, we need to calculate style transfer accuracy. while there is ongoing work of developing automatic evaluation metrics for formality style transfer in general @xcite , this work introduces a systematic evaluation of formality style classifiers. in this paper, we aim at closing the gap by proposing a comprehensive computational study of various text categorization approaches. namely, we argue that nlp practitioners will be benefiting from the knowledge of answers to the following questions: ## datasets here, we provide the detailed description of the data-nature of the texts and general datasets' statistics-used for the experiments. ## text classification models following @xcite , we address the formality detection as text classification task. we experiment with several state-of-the-art models optimizing their hyper-parameters. a detailed description of these most successful models is presented below. ## discussions as all the above experiments results showed that none of the models achieved accuracy and f1 scores higher 90.0, we analyzed misclassifications. in appendix a in for example, the char bilstm model trained for the english language misclassified sentence 1 would work for me but both would be even better into formal class. indeed, the whole structure of the sentence and the usage of word would make the text looks like a formal one. we suppose that this text was marked as informal because it is fully written in the upper register. on the other hand, there are many sentences with formal labels without an obvious reason for that. texts like ignore it when people start rumors., i do not want her to die. does not look like to be written in a formal style. on the contrary, the usage of the phrase ignore it seems to be quite informal. also, if we look at misclassification examples of mdistillbert models, we can see examples of obvious violations of formal style. for example, we can observe sentences that are grammatically correct, but the content is toxic (are you serious or just that ignorant?) or refers to some informal ways of entertainment (after watching that, i had to consume alcohol!). that might be that the general topic of these sentences is more closer to the topics usually discussed informally that confuses the model. in addition, we draw attention to the sample which is mostly formal, however, contains informal insertion: i'm grateful, i now comprehend. significantly, er, electrical. such mistakes can be connected with the process of the creation of the gyafc and xformal datasets. the train part consists of informal texts and their formal paraphrases with amazon mechanical turk workers. however, the tune part contains paraphrases from formal into informal styles and vice versa. the annotation process can contain some inaccuracies that may be resulting in fuzzy logic of labels assignment. in addition, another interesting observation might be that for some transformer-based models their multilingual versions yields higher accuracy than monolingual ones. thus, for distilbert, the bets english monolingual accuracy is 80.1, while its multilingual version achieves 86.6 score on english test set. the same observation can be applied for bert model as well. in the end, we can observe quit high results from char bilstm model which outperform in some cases transformer-based models. one of the explanations might be: the usage of slang or unusually modified words in informal style that can be precisely tokenized and embedded with transformerbased encoders, however, can be learned with character-level words' split. ## conclusion in this paper, we presented the first computational study on text categorization models that detect text formality. we based our experiments on two large-scale multilingual datasets-gyafc and x-formal-and tested a vast amount of baselines and state-of-the-art neural models. the best english monolingual results are achieved by transformer-based model-debertalarge. however, other obtained results show the superiority of models based on character representation, such as char bilstm models, over models based on word and bpe representations, including even large pre-trained transformer models. notably for both monolingual and multilingual formality detection for all examined languages, char bilstm model illustrates the best accuracy. our experiments also show that multiple models demonstrate abilities of cross-lingual transfer. while char bilstm showed the best performance in monolingual and multilingual setups, it had a significant drop in the performance while trying to transfer formality knowledge to another language. in this scenario, mdistilbert model demonstrated the best stability to new languages. all code and data allowing reproduce our experiments are available online. 3 we release for a public usage the best transformer-based monolingual foot_4 , multilingual foot_5 , and cross-lingual foot_6 models.