A Question of Style: A Dataset for Analyzing Formality on Different Levels (2023)

Abstract: Accounting for different degrees of formality is crucial for producing contextually appropriate language. To assist NLP applications concerned with this problem and formality analysis in general, we present the first dataset of sentences from a wide range of genres assessed on a continuous informal-formal scale via comparative judgments. It is the first corpus with a comprehensive perspective on German sentence-level formality overall. We compare machine learning models for formality scoring, a task we treat as a regression problem, on our dataset. Finally, we investigate the relation between sentence- and document-level formality and evaluate leveraging sentence-based annotations for assessing formality on documents.

## introduction textual style can be approached from various points of view. we focus on its inherent formality dimension stretching from informal to formal language use. see these two sentences, for example: wir haben gründlich über ein adäquates beispiel nachgedacht. (2) racked our brains about a niiice example... :d haben uns den kopf über ein schööönes beispiel zermartert... :d while both sentences transport the same content, they differ in their degree of formality. (2) is less formal than (1). it may be suitable only for more informal discourse contexts and inappropriate in formal settings. understanding these different nuances of formality is crucial for effective communication. consequently, striking the right tone is relevant not only for humans but also for various nlp applications. may it be machine translation in need to transfer expressions of formality between different languages adequately @xcite , chatbots aiming to produce contextually appropriate language to increase user satisfaction @xcite , or writing assistance systems altering content to be more formal @xcite . hence, intra-lingual formality style transfer, which deals with generating a formal phrase given its informal version and vice versa, has recently also received increased attention (e.g., @xcite . our paper addresses a prerequisite for this task: assessing linguistic formality. rating the transferred style strength is necessary for evaluating formality style transfer models. further, parallel corpora with formal and informal language pairs, often the basis for style transfer, are commonly built by automatically grading and extracting informal sentences first @xcite . for facilitating such formality assessments and analyzing linguistic formality in general, we make the following contributions: ## related work with their continuous formality score based on frequencies of parts of speech, @xcite established a milestone for the definition of formality. @xcite adapted this measure from the document to the sentence level. most approaches targeting the lexical dimension of formality also regarded formality as a continuum @xcite @xcite . to the best of our knowledge, datasets comprising sentences with human formality assessments on a continuous informal-formal scale have not been constructed before. @xcite built an english dataset collecting formality annotations on a 7-point likert scale for sentences from only four sources (compared to the twelve in our dataset). they introduced formality detection as a regression task using features based on analyzing human perceptions of formality for a ridge regression model. other datasets targeting sentence-level formality have binary labels since they primarily serve as parallel data for formality style transfer and contain formal and informal language pairs. they cover english @xcite , brazilian portuguese, french and italian @xcite , and hindi, bengali, kannada and telugu @xcite . work on formality style transfer mainly used classification for measuring style strength and a handful of different classifiers (e.g., @xcite employed a cnn, wang et al. (2019) an lstm, and krishna et al. (2020) transformers). evaluating the style strength as a regression task, rao and tetreault (2018) borrowed the approach from pavlick and tetreault (2016), and briakou et al. (2021b) relied on fine-tuning transformers. for the german language, not yet considered for intra-lingual formality style transfer, two sentence collections with binary formality annotations based on formal and informal direct address exist @xcite . (since these formality levels do not exist in english, they pose a problem for machine translation @xcite .) hence, these datasets target a very constrained view of formality only. focusing on the document level, several works used traditional machine learning models for binary formality classification based on linguistic features. as training data, abu sheikha and inkpen (2010) assumed binary labels for formality from the text genre, and @xcite al. (2011 ) manually annotated emails from the english enron corpus (klimt and yang, 2004) with four formality classes. treating formality assessment on documents as a regression task, chhaya et al. (2018) employed linguistic features for formality scoring on enron emails, which have been rated on a 5-point likert scale, whereas eder et al. (2021) evaluated word formality scoring on emails from the german corpus code alltag (eder et al., 2020) based on continuous formality annotations. all these manually labeled document collections are small in size (∼1k) and built from a single domain only, i.e., emails. none of these works leverages formality-annotated sentences nor fine-tunes transformer models to assess the formality of documents. ## data to build our dataset, we collected 3,000 german (de) sentences from different domains and let crowdworkers assess their formality on a continuous formality scale via comparative annotations. ## formality scoring on sentences we compared different models for predicting formality scores for sentences on our dataset. ## formality scoring on documents documents may assemble an even more diverse range of clues for degrees of formality than sentences. only recently, @xcite proposed extending style transfer to the more complex document level, but manual formality annotations of documents are more expensive to obtain than sentence-level assessments. therefore, this section investigates how single sentences and linguistic properties contribute to the overall document formality. we examine if sentence-level formality annotations are useful for assessing formality on documents. ## conclusion we presented the first dataset of sentences with highly reliable human formality assessments on a continuous informal-formal dimension obtained via best-worst scaling. our dataset comprises 3,000 sentences evenly distributed over twelve different domains to cover the broad spectrum of formality best possible. it is the first for the german language with a comprehensive perspective on sentence-level formality altogether. we evaluated various machine learning models for the regression task of assessing formality on our dataset. we found that a transformer model fine-tuned on an existing german dataset including only sentences of formal and informal address (sie vs. du/du) yielded the worst results. hence, this restricted view on formality is insufficient to capture a more comprehensive concept of formality. cross-lingual settings utilizing transformer-based classifiers pre-trained on huge datasets with formal and informal language pairs not restricted to a particular form of formality performed better. however, a binary categorization of formality strikes as inappropriate since ridge regression models employing simple linguistic features outperformed them. fine-tuning transformers for regression on an english dataset produced similar (for the cross-lingual setting or the english translation of our dataset) or higher (for the german translation of the training data) results. in comparison, a transformer model fine-tuned on our dataset with its broader formality scale outperformed all other settings significantly. expanding the scope to longer texts, a requested future research direction of style transfer @xcite , we investigated the influence of the formality of sentences on a document's formality. we observed that the sentences included in the documents cover a wide spectrum of formality with higher formality scores at the beginning. our results indicate that a transformer model fine-tuned for formality scoring on our sentences generalizes better across text levels than linguistic features and can be used to predict the degree of formality of german and english documents. we anticipate our dataset to facilitate future work on german formality style transfer and formality analysis in general on both the document and the sentence level. it may also be valuable for other languages. our dataset and a tool for analyzing styles with a wide range of linguistic features are available under https://github.com/ee-2/in_ formal_sentences and https://github.com/ ee-2/register .