Acquiring a Formality-Informed Lexical Resource for Style Analysis (2021)

Abstract: To track different levels of formality in written discourse, we introduce a novel type of lexicon for the German language, with entries ordered by their degree of (in)formality. We start with a set of words extracted from traditional lexicographic resources, extend it by sentence-based similarity computations, and let crowdworkers assess the enlarged set of lexical items on a continuous informal-formal scale as a gold standard for evaluation. We submit this lexicon to an intrinsic evaluation related to the best regression models and their effect on predicting formality scores and complement our investigation by an extrinsic evaluation of formality on a German-language email corpus.

## introduction the computational treatment of style in verbal communication has long been dominated by application concerns, e.g., the identification or profiling of authors in forensic linguistics @xcite or the recognition of plagiarism @xcite . this research was conducted assuming that simple lexico-statistic patterns identified by stylometric computations were sufficient to solve authorship and plagiarism assignment problems. despite their undisputed success in those limited fields, these studies scratched only the surface of the notion of 'style' as discussed in linguistic pragmatics @xcite . from the many ways 'style' can be approached from a pragmatics perspective, we here focus on its inherent formality dimension, i.e., the distinction between formal (standard) and informal (colloquial) language use (for a survey, cf. @xcite ), with further extensions directed at the higher level of formal (e.g., elevated style) and the lower level of informal (e.g., vulgar) phrasing. such distinctions of formality levels are crucial for the appropriateness of verbal expressions in a given discourse context. in order to track different levels of formality in written communication, we introduce a novel type of lexicon for the german language, with entries ordered by their degree of (in)formality. 1 we start with a set of words extracted from traditional lexicographic resources, extend it by sentence-based similarity computations, and let crowdworkers assess the enlarged set of lexical items on a continuous informal-formal scale. this workflow is described in section 3. the resulting lexicon comprising words with their respective formality scores subsequently serves as a gold standard for evaluation. in section 4, we submit this lexicon to an intrinsic evaluation related to the best regression models and their effect on predicting formality scores, and complement our investigation by an extrinsic evaluation of formality on a german-language email corpus in section 5. ## related work the relevance of 'style' for nlp is obvious for language output-focused core applications such as language generation @xcite @xcite , machine translation @xcite or proper phrasing in argumentation @xcite . quite recently, the notion of 'formality style transfer' has received increasing attention, which captures the idea to generate a formal sentence given an informal one (et vice versa), while preserving its meaning @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite . many efforts to cope with language style have been spent, however, in application niches, such as author identification or plagiarism detection. most of the methodological contributions developed in this forensic branch are summarized under the label of stylometrics and have recently found their way into nlp analytics to unveil deception @xcite or linguistic aggression @xcite @xcite . the computational analysis of style according to stylometric principles, from its inception, is closely linked with lexical frequency counts. typically, mostly function words (such as articles, pronouns, conjunctions, contractions, common abbreviations, hedging terms, also including punctuation marks) are assembled in small-sized dictionaries, together, if at all, with only a few content words (domainspecific nouns, verbs, adjectives). the frequency distributions resulting from counting these dictionary entries at the document or corpus level are already very beneficial for successfully dealing with disputed authorship problems (mostly for literary texts, but also for the detection of spam, fake news, or other kinds of toxic language) or uncovering plagiarism (mostly in scientific or news documents). similar in spirit, word and sentence length criteria originating from readability metrics (flesch-kincaid, etc.) and several measures of vocabulary richness (e.g., type-token ratios, yule's k and burrow's ∆) were also incorporated into stylometric toolkits @xcite . as a simple extension from these uni-grams, lexical or pseudo-lexical character n-grams (bi-or tri-grams, mostly) were determined and counted, as well. slightly extending this (pseudo-)lexically focused approach by syntactic information, partof-speech n-grams (or part-of-speech frequencies) were also considered to trace the human 'stylome', although lexical factors were found to be more relevant for style analysis than syntactic (pos sequence) patterns @xcite . simple frequency metrics have increasingly been complemented by various forms of lexical association measures (such as information gain, mutual information), and more sophisticated probabilistic models (principal component analysis (pca), latent semantic analysis (lsa), or other types of topic models). comprehensive lists of criteria and metrics are provided by sheikha and inkpen (2010); @xcite . we claim that despite their relevance for applications, such as authorship attribution and plagiarism detection, these mechanisms merely serve as easy to trace proxies for characterizing linguistic style. in our work, we will have a closer look at the style-marking semantic connotation of single lexical items as explicit carriers of linguistic formality as an important facet of language style. a milestone for the formal definition of formality was set up by the pioneering work of @xcite who defined the f-score-close in spirit with the simple lexico-statistic frequency metrics from stylometry-as the percentage difference between deictic (article, pronouns, etc.) and non-deictic parts of speech (nouns, adjectives, etc.) in a document (f ranges between 0 and 100, with higher f indicating higher formality). 2 this document-level perspective was adapted by @xcite to sentence-level formality analysis. a complementary lexical dimension for the formalization of formality was introduced by @xcite . they define the formality score for a word as a real number value in the range 1 to -1, with 1 representing an extremely formal word and -1 an extremely informal one, and assign a formality score to each lexical item based on standard word length, morphology-based features, lexical distribution criteria or association methods (lsa). our work adheres to their way formality is scored in a formality lexicon and manually supplied seed sets are used (as starters), but differs markedly whether the lexicon is considered as a static @xcite or a dynamic resource (as we do; in a later study, @xcite proposed a dynamic acquisition method, as well, by assigning a continuous formality score to single words based on their co-occurrence frequency with a hand-picked seed set of formal, neutral and informal words), and the way how semantic similarity is computed (lsa vs. embeddings). further, we do not induce formality levels for a near-synonym task automatically but rather crowdsource nuances of formality for a relationally unrestricted lexical inventory from human raters. @xcite proposed a model of formality based on an empirical analysis of human formality perceptions. they apply their approach to analyze language use in online debate forums for multiple genres (news, blogs, emails, and community question answering sites). formality assessments are solicited via amazon turk (following the protocol established by @xcite ) using a 7-point likert scale, with labels ranging from -3 (very informal) to 3 (very formal). a ridge regression classifier uses 11 different feature groups-five rarely used ones (among them word2vec embeddings @xcite , parse trees, dependency tuples, and named entities) and six much more common ones (among them lower/upper casing, punctuation, readability scores, pos tags, and length-normalized formality and subjectivity scores)-to determine the formality level of sentences. cross-genre analysis reveals that n-grams and word embeddings perform the best among all tested features (they achieve over 80% of the performance of the full classifier in all cases). this work comes closest to our approach, yet with differences in the way formality is assessed (likert scales vs. best-worst scaling) and lexicon building is dealt with. @xcite employ an acquisition method to score the formality of unseen phrases along the formal-casual dimension from scratch, as described in earlier work by @xcite who use a log ratio metric based on the occurrence of phrases in various style-tagged corpora, in contrast to the embedding-based similarity model we propose. earlier computational models for detecting formality were proposed by sheikha and inkpen (2010); @xcite ; @xcite . the first two perform a binary classification only into formal vs. informal utterances, the third model classifies into four levels of (in)formality, and all of them operate at the document (as opposed to sentence) level. ## intrinsic evaluation of i-forger rather than increasing the size and thus the coverage of lexicons to improve performance on potential applications, we intend to score (unseen) words on the fly. hence, we first evaluate the word scoring model (section 4.1). next, we assess the four main input streams of i-forger (section 4.2) and the extension of the scale regarding formality levels (section 4.3). figure 3 illustrates the schematic workflow for our word scoring procedure, including (and marked in green) the three evaluation tasks. figure 3 : overview of the word scoring workflow; parts to evaluate are marked in green 4.1 regression models for word scoring we adopted various approaches using a seed lexicon, actually, the entries' word embeddings, as training data for regression models to automatically score new lexical items for their formality connotation (see, e.g., @xcite and @xcite for a similar scenario for automatic emotion induction). as input features we decided for fasttext word embeddings @xcite with their own out-of-vocabulary (oov) functionality. we found that they performed better than getting the oov handling from bpemb subword embeddings @xcite , based on byte pair encoding (bpe) @xcite , instead, or solely utilizing pure bpemb embeddings. we evaluated different regression models. besides ridge regression, foot_15 which is linear regression with l 2 regularization during training, we also experimented with densifier @xcite , which learns an orthogonal transformation of the embedding space, and a modified, more robust variant of the latter, densray @xcite . 18 we ran a feed-forward neural network with one hidden layer combined with the boosting algorithm adaboost.r2 (boosted ffnn) as proposed by @xcite . 19 further, we tested neural networks with more than one hidden layer, namely two hidden layers with 256, 128 units (nn 2hidden ) and three hidden layers with 256, 128 and 64 units (nn 3hidden ). 20 ## extrinsic evaluation of i-forger in order to gather evidence for the value of i-forger in combination with the word scoring approach within a realistic use case, we ran experiments with emails, which possess a higher stylistic variability than news concerning their formality spread @xcite . other work related to the formality of emails is typically carried out in the context of communication behavior studies in enterprises, with a focus on determining social factors (social distance, relative power, and the weight of imposition) that affect the sender's choice of formality @xcite or on the affective dimension of email exchanges @xcite in terms of the prediction of frustration of employees from email data. ## conclusion different levels of formality these days find increasing attention, both in methodological approaches and nlp applications. the necessity of choosing a socially appropriate tone is particularly evident in digitally mediated discourse, e.g., formal business or informal private email communication @xcite or social media interaction via reviews, chats, or blogs @xcite . the increasing relevance of conversationally adequate virtual personal assistants @xcite , chatbots @xcite and automatic procedures for smart response generation @xcite requires sensitivity on the generator's side to strike the right tone and avoid the false one. similarly, machine translation poses special problems when expressions of (in)formality have to be adequately transferred between different languages @xcite . progress in monitoring formality levels is a methodological prerequisite for several downstream applications that have to comply with users' habitual expectations or increase user satisfaction, e.g., in commercial interactions (customer service communication) @xcite or medical consultation @xcite . as a methodological contribution, we here propose a lexical approach to computational style analysis based on i-forger, a lexicon whose (3,000) items are scaled on a continuous informal-formal spectrum. we make three new contributions to style analysis: first, a language-independent lexicon acquisition architecture employing sentence embed-dings forms the basis for computing sentence similarity, thus finding formality-sensitive lexical items not contained in the seeds. second, best-worst scaling is used for creating gold standards available for an in-depth intrinsic and extrinsic evaluation of the new lexical resource. finally, i-forger stands out as the first formality-informed lexicon for the german language. this resource is available at https://github.com/ee-2/i-forger . despite our lexical focus, we are aware of the fact that formality is not only lexically expressed. consequently, a lexicon-based approach has to be complemented by methods that account for nonlexicalized varieties of formality. such forms may include syntactic variability, linguistic complexity and readability, as well as correctness of language use regarding orthography, morphology and syntax. for research on formality detection incorporating its syntactic, semantic and discourse facets, cf., e.g., @xcite , @xcite or @xcite . these branches will also be part of our future work. still, a (potentially) large portion of formality assessments is rooted in lexical signals, which we capture by the methodology advanced in this paper.