DOPA METER – A Tool Suite for Metrical Document Profiling and Aggregation (2023)

Abstract: We present DOPA METER, a tool suite for the metrical investigation of written language, that provides diagnostic means for its division into discourse categories, such as registers, genres, and style. The quantitative basis of our system are 120 metrics covering a wide range of lexical, syntactic, and semantic features relevant for language profiling. The scores can be summarized, compared, and aggregated using visualization tools that can be tailored according to the users’ needs. We also showcase an application scenario for DOPA METER.

## introduction the way how we encode contents in natural language utterances gives rise to linguistic divisions into registers, genres, style levels, etc. (for a thorough distinction of these terms, see @xcite ; @xcite ) that follow functional communication requirements, e.g., ease of comprehension or adherence to the wording of social peer groups. the behavioral traits indicating such divisions are manifold and range from simple token frequencies, lexical choice options (synonyms, more specific vs. more general or sublanguage vs. layman terms), via syntactic variations (easy vs. complex sentence constructions) over to pragmatic distinctions (e.g., formal vs. informal language use). many of nlp's most pressing applied research questions (e.g., hate and fake detection, communication biases relating to people's political, religious, racial, personal orientation) are considered to be flagged this way @xcite . in this paper, we address a large variety of such behavioral aspects of language use from a metrical perspective. none of these metrics is new, but their assembly and broad coverage in a coherent tool suite and modular software framework is. we also provide means for summarization, comparison and aggregation of results and their proper visualization. ## related work the tool-based computational analysis of behavioral traits of language use can be divided into three branches of research: (1) readability checkers with language complexity measures incorporating mostly surface-level syntactic and lexicosemantic features of utterances, (2) stylometrics tools with strong emphasis on powerful lexicostatistical metrics, and (3) psychometrics devices with mostly simple frequency-based computations complemented by dictionaries with psychologically typed lexical categories. from the perspective of readability (for a survey, see @xcite ), the delite system (vor der @xcite can be considered as one of the language profiling systems closest to the design goals and feature types of our system. still, its main goal, as a readability checker, is much narrower than ours. delite identifies and highlights passages of text which are difficult to understand (together with reasons why this is the case). to reach this goal, delite comes with a wide range of shallow and deep features to score the readability of documents, which is also at the heart of our work. deep features include, e.g., topological information from dependency trees for syntactic scoring (e.g., center embedding depth, phrasal fan-out ratios) and from semantic networks for semantic scoring (number of readings per lexical entry, number of propositions per sentence, semantic network connectivity). altogether, 48 indicators for readability at the morphological, lexical, syntactic and semantic level can be calculated, averaged per document, and a global document readability score is finally computed by applying a k-nearest neighbor classifier. the system ran on german and english input data, yet has, to the best of our knowledge, never been made publicly accessible. in the field of stylometrics (for a survey, see @xcite ), stylo @xcite has become a de facto standard for the quantitative study of writing style. stylo is an r package equipped with powerful statistical analysis modules for analytics based on frequency measurements of character-and token-based n-grams (pos n-grams etc., not supplied by default, require externally preprocessed input). stylo comes in two flavors. its api allows to configure a complete processing pipeline using traditional r scripting, while it also offers a rich graphical user interface (gui) for non-technical users to run stylometric analyses and interpret their outcome without the need for elaborate programming experience. the seamless integration of various analytical tools under a common programming framework (making use of r's core library but also extending it by various clustering algorithms and machine learning classifiers) and its public accessibility on github foot_0 make stylo a landmark development for stylometric tooling. yet, stylo does not integrate any deeper lexical, syntactic and semantic processing going beyond textual surface computations (such as distance metrics, e.g., burrows's ∆, very popular in the stylometric community). the third stream of work emphasizes human lexical choice patterns in terms of the psychometrics of word use. perhaps its most prominent representative is the linguistic inquiry and word count approach and its associated liwc engine (tausczik and pennebaker, 2010). foot_1 liwc's focus is on a categorically stratified dictionary resource (the current master dictionary comprises 6,400 words, word stems, and selected emoticons) with simple descriptive statistical tools though. liwc reads documents word-by-word, matches each word with its dictionaries and outputs simple frequency-based lexical and pos statistics. overall more than 80 psychologically relevant categories ranging from linguistic ones (such as function vs. content words, parts of speech, tense markers) to psychological ones (such as cognitive, perceptual, and biological processes) are attached to single lexical entries and counted during text analysis. liwc was recently compared and outperformed by the seance system @xcite which makes use of a range of newer, even more specialized dictionaries with a larger number of more expressive psychological categories and variables and a higher coverage of entries. @xcite use a battery of independent systems for their experiments, each one highly specialized for computing different dimensions of readability, such as syntactic complexity (177 indices from the taassc system ( kyle and crossley, 2018) ), lexico-semantic frequency and richness (135 indices from the taales system @xcite ), text cohesion (over 150 indices from the taaco system @xcite ), and sentiment and social cognition scores (20 indices from the seance system @xcite ). hence, roughly 500 individual scores have to be assembled from these stand-alone systems and combined in an umbrella system for result merging. alternatively (not used by @xcite , but playing a prominent role in many recent readability studies), coh-metrix foot_2 @xcite provides a multi-dimensional set of (psycho)linguistic and discourse features (version 3.0 incorporates 108 different indices). recently, @xcite introduced coco, an advanced system with cognitively plausible features, yet its focus is limited on conceptual complexity computation of texts. sentspace @xcite is a sentence-focused analysis engine rather close to the design goals of our work, which also uses a range of cognitively plausible lexical, syntactic and semantic features. however, it lacks classical stylometric and readability indices and is limited to analyses up to the single document level only. in contrast to this work, we aim at crossdocument and cross-corpus analyses for more powerful register, genre and style analyses. despite the remarkable progress that has been made already-the proliferation of surface-level, linguistic and cognitive features under scrutiny, and the growing number of metrics making use of them-we observe a fundamental lack of integration of and abstraction from single counts and scores in these precursors. accordingly, a major goal of our work is to provide reasonable summarization, comparison, and aggregation levels for single metrics so that divisions into registers, genres and styles can be computed on the fly based on the contributions of a wide range of linguistic layers (integrating lexical, syntactic, and semantic features) for complex collections of (multilingual) linguistic data in terms of (sets of) corpora. and supports all spacy compatible language modules. our system is publicly accessible via github. 5 it is based on strict software engineering principles, such as modularity, easy resource maintenance and (re-)configuration (selection and augmentation of metrics and language resources, such as corpora and lexicons). the three-layered architecture of dopa meter is depicted in fig. 1 . it consists of • arbitrarily many text corpora that can also be grouped into collections of corpora which serve as textual input channel (including a preprocessing pipeline), • the feature hub that elicits relevant features from the corpora for use by a large variety of metrics, • and three analytics layers-apart from simple report generation (summaries of metricsderived scores), we offer a comparison mode across documents and corpora, as well as cluster-based aggregation of results. ## dopa meter in action we now illustrate facets of the rich functionality of dopa meter. our scenario features two languages, english and german, and a broad application domain (medicine) with six corpora (collections) from a wide range of genres (see en.clin incorporates public corpora supplied for the i2b2 and n2c2 challenge series, 12 and en.socmed combines english language twitter corpora with biomedical content: bear @xcite , covert @xcite , and bioclaim @xcite . the highest syntactic complexity in terms of parse tree depth is attributed to the german expertlayman corpus (expert statements seem to suffer from 'hard' syntax), with no substantial differences for the remaining corpora. the german social media corpus (in contrast to the english one) is the richest in terms of synonyms, whereas both clinical corpora are semantically poor at that level (adhering to canonical medical terminology-the english one being even poorer than the german one). the medical german wikipedia is in a similar range with german clinical and pubmed documents on that dimension. ## conclusions we introduced dopa meter, a toolkit for quantifying feature distributions at the lexical, syntactic and semantic dimension. we supply 120 metrics for scoring linguistic behavior at these axes. scores can be summarized, compared, and aggregated using flexibly tailorable visualization tools. dopa meter's feature collection reflects one main design goal of our work, namely the integration of as many linguistic levels as possible, thus moving away from much more selective approaches in stylometrics and psychometrics. a second unique feature of our approach is its focus on lucid system architecture for flexible system engineering, i.e., easy maintainability and augmentation by new metrics and language resources (corpora, lexicons) in a coherent all-in-one system design. this contrasts with the proliferation of stylometric extensions spread over lots of local github links lacking further integration, on the one hand, and frozen system packages in the psychometric domain, on the other hand. the source code and its documentation are provided under the open mit licence and our tool can be conveniently expanded and adapted to specific needs. this way, dopa meter may be useful as a metadata generator for documents and text corpora, with facilities for quantitative data description (scoring), comparison and aggregation. such an approach may also pave the way towards an empirically sound way of routinely running nlp data diagnostics @xcite .