Does It Capture STEL ? A Modular, Similarity-based Linguistic Style Evaluation Framework (2021)

Abstract: Style is an integral part of natural language. However, evaluation methods for style measures are rare, often task-specific and usually do not control for content. We propose the modular, fine-grained and content-controlled similarity-based STyle EvaLuation framework (STEL) to test the performance of any model that can compare two sentences on style. We illustrate STEL with two general dimensions of style (formal/informal and simple/complex) as well as two specific characteristics of style (contrac’tion and numb3r substitution). We find that BERT-based methods outperform simple versions of commonly used style measures like 3-grams, punctuation frequency and LIWC-based approaches. We invite the addition of further tasks and task instances to STEL and hope to facilitate the improvement of style-sensitive measures.

## introduction natural language is not only about what is said (i.e., content), but also about how it is said (i.e., linguistic style). linguistic style and social context are highly interrelated @xcite . for example, people can accommodate their linguistic style to each other based on social power differences @xcite . furthermore, linguistic style can influence perception, e.g., the persuasiveness of news @xcite or the success of pitches on crowdsourcing platforms @xcite . as a result, style is relevant for natural language understanding, e.g., in author profiling @xcite , abuse detection @xcite or understanding conversational interactions @xcite . additionally, style can be important to address in natural language generation @xcite , including identity modeling in dialogue systems @xcite and style preservation in machine translation @xcite . anchor 1 (a1) and anchor 2 (a2) and the alternative sentences 1 (s1) and 2 (s2) are split along the same style dimension (here: formal/informal). the sentences and anchors are paraphrases of each other. the stel task is to order s1 and s2 to match a1-a2. here, the correct order is s2-s1. there are several general evaluation benchmarks for different linguistic phenomena (e.g., @xcite ) but less emphasis has been put on linguistic style. nevertheless, natural language processing literature shows a variety of approaches for the evaluation of style measuring methods: they have been tested on whether they group texts by the same authors together @xcite , whether they can correctly classify the style for ground truth datasets @xcite and whether 'similar style words' are similarly represented @xcite . however, these evaluation approaches are (i) often application-specific, (ii) rarely used to compare different style methods, (iii) usually do not control for content and (iv) often do not test for fine-grained style differences. these shortcomings (i)-(iv) might be the result of the following challenges for the construction of style evaluation methods: 1. style is a highly ambiguous and elusive term @xcite @xcite . we propose a modular framework where components can be removed or added to fit an application or specific understanding of style. 2. variation in style can be very small. our proposed evaluation framework can be used to test for fine-grained style differences. 3. style is hard to disentangle from content as the two are often correlated (e.g., @xcite ; @xcite ). for example, people might speak more formally in a job interview than in a bar with friends. thus, language models and methods might pick up on spurious content correlations (similar to @xcite ) in a benchmark that does not control for content. to this end, we propose the modular, fine-grained and content-controlled similaritybased style evaluation framework (stel). we demonstrate it for the english language. an example task is shown in figure 1 . the task is to order sentence 1 (s1) and sentence 2 (s2) to match the style order of anchor 1 (a1) and anchor 2 (a2). our stel framework encompasses two general dimensions of style (formal/informal and simple/complex) as well as two specific characteristics of style (contraction and number substitution). by design, the style characteristics are easy to identify. thus, the stel characteristic tasks are easier to solve than the stel dimension tasks. stel contains 815 task instances per dimension and 100 task instances per characteristic (see ## style evaluation framework we introduce the modular, fine-grained, and content-controlled similarity-based style evaluation framework (stel). stel tests a (language) model's ability to capture the style of a sentence. ## task generation we describe the task instances of stel: first, we generate potential task instances ( §4.1). second, we describe problems with the generated instances (i.e., ambiguity in §4.2). third, we filter out the problematic instances via crowd-sourcing ( §4.3). ## evaluation we use our stel framework to test several models and methods that could be expected to capture style information ( §5.1). we describe how the models decide the stel tasks ( §5.2) and discuss their performance on stel ( §5.3). ## limitations and future work our illustrative set of task instances does not cover all possibilities of style variation. future work could extend stel to cover additional style dimensions or more fine-grained task instances using several sources of data. the stel task instances for one style component can contain correlations with unconsidered (style) components. consider the following task instance (shortened for readability): (a1) "fortynine species of pipefish [...] have been recorded.", (a2) "forty-nine type of pipefish [...] have been found", (s1) "patients [...] must have their liver checked for damage and other side effects." and ( s2 ) "[...] patients [...] must be monitored for liver damage and other possible side effects.". (a2) and (s1) are the simpler version of (a1) and (s2) @xcite . additionally, the sentences vary along other aspects: (a2) is missing the punctuation mark and includes a misspelling. (s1) is different in content from ( s2 ) as (s1) is only considering effects on the liver while (s2) also includes other side effects. however, those aspects did not change the label given by the annotators (s2-s1) and should mostly be secondary to the considered style dimension. with stel, language models and methods are tested only on whether they capture clear differences in style when content is approximately the same. when there are also content differences, such models might put more emphasis on content than stylistic aspects. our framework could be extended to allow testing for whether a model prefers style over content (e.g., with a new task format where sentence 1 is closer in content to anchor 1 but closer in style to anchor 2, c.f. figure 1 ). stel could also be extended to test for individual author styles and style variation related to the social or regional background of authors (e.g., different age groups). for example, by including sentence pairs with the same content but written by different authors. current and future dimensions could also be extended by a train/dev/test split to enable training on the task directly. further, stel could be enriched by including longer texts (e.g., paragraphs or documents) as anchor and alternative sentences. ## conclusion style is an integral part of language. however, there are only few benchmarks for linguistic style. in this work, we introduce stel, a modular, content controlled and fine-grained similarity-based style evaluation framework. out of the evaluated language models and methods, the cased bert base model performs the best on stel. simpler sentence features perform close to the random baseline. stel includes two general style dimensions and two specific style characteristics. we hope that this framework will grow to include an even more exhaustive representation of linguistic style and will facilitate the development of improved style(-sensitive) measures.