Dialect-robust Evaluation of Generated Text (2023)

Abstract: Text generation metrics that are not robust to dialect variation make it impossible to tell how well systems perform for many groups of users, and can even penalize systems for producing text in lower-resource dialects. In this paper, we introduce a suite of methods to assess whether metrics are dialect robust. These methods show that state-of-the-art metrics are not dialect robust: they often prioritize dialect similarity over semantics, preferring outputs that are semantically incorrect over outputs that match the semantics of the reference but contain dialect differences. As a step towards dialect-robust metrics for text generation, we propose NANO, which introduces regional and language information to the metric’s pretraining. NANO significantly improves dialect robustness while preserving the correlation between automated metrics and human ratings. It also enables a more ambitious approach to evaluation, dialect awareness, in which system outputs are scored by both semantic match to the reference and appropriateness in any specified dialect.

## introduction most natural language generation (nlg) evaluation metrics compare a system output against a human-written reference. references are usually drawn from a relatively narrow range of linguistic styles. they often exclude varieties like indian english or iberian portuguese, which are geographical dialects with millions of speakers. as a result, outputs in dialects that are not represented in the reference may score poorly, discouraging the development of systems to meet the needs of these language communities. although contemporary metrics such as comet @xcite ) can be reference-free, they still rely on training data and rater pools that do not cover all dialects of interest, leading to a high number of out-of-domain dialects. the performance of evaluation metrics on these out-of-domain dialects has not been quantified. we define a dialect-robust evaluation metric as one that produces the same score for system outputs that share the same semantics, but are expressed in different dialects. to understand whether current evaluation metrics are dialect-robust, we propose to quantify the dialect robustness at the dialect feature-level and sentence-level. the analyses measure the dialect-sensitivity of evaluation metrics by comparing semantics-preserving dialect edits to perturbations that change the meaning of sentences. through our analyses, we demonstrate that multiple state-of-the-art nlg evaluation metrics are not robust to dialects of mandarin, english, and portuguese. in many cases, system outputs that are perturbed so as to differ semantically from the reference score higher than outputs in which the only change is to the dialect. with the goal of increasing the dialect robustness and without performance degradation on standard benchmarks, we propose a training schema nano. nano is an unsupervised pretraining step to a metric that distills dialect information of the multilingual pretraining dataset into a model, which we demonstrate leads to improved dialect robustness. based on our findings, we lay out research goals toward dialect-inclusive metrics. moving beyond dialect robustness, we formalize the goal of dialect awareness, in which metrics can be applied to any user-specified language and dialect regardless of the language of the reference or source document. ## dialect robustness and awareness dialects can be regarded as linguistic subdivisions that align with communities of speakers, often grouped by geographical or demographic attributes @xcite . a classic example is nation-level varieties, such as brazilian and iberian portuguese. dialects are distinguished from each other by a set of dialect features, which can op-0.69 0.77 0.81 0.83 figure 1 : an illustration of dialect robustness in the context of generation evaluation. we define dialect robustness as evaluation metrics that are expected to have the same output across dialects that share the same semantics. dialect edits (highlighted in yellow) should not lead to a greater degradation of score than edits that change the underlying semantics (highlighted in underline). bleurt-20 in the figure assigns higher score to semantically-perturbed sentences than sentences with dialect features, exposing its vulnerability to dialects. definition 2.1 (dialect robustness). let y (d) and y (d ′ ) be two system outputs that are semantically equivalent but written in different dialects. an evaluation metric m : this definition is strict: it would not apply to any system that produced even small differences in score between semantically equivalent, regionally distinct outputs. for that reason, we propose a relaxed criterion, which compares the change in the metric induced by dialect to changes induced by semantic perturbations: definition 2.2 (ϕ-dialect robustness). let y (d) and y (d ′ ) be two semantically-equivalent system outputs that differ in dialect. let ϕ : @xmath0. an evaluation metric m : informally, a metric is dialect aware if, given a dialect identifier and a pair of semanticallyequivalent texts that vary by dialect, it assigns the highest score to the text in the dialect specified by the identifier. dialect awareness is undefined with respect to inputs that are not semantically equivalent. this means that the definition is agnostic as similarly, a corpus-level quality estimation metric could be defined with mi(yi) = δ(yi, xi) with xi indicating the input, such as the source language or passage to be summarized. for the corpus-level metric to be dialect robust (or ϕ-robust), all mi must be dialect robust (or ϕ-robust). figure 1 illustrates the concepts of dialect robustness and dialect awareness. the top two rows of each panel vary only by dialect; the bottom row shows semantic perturbations of the top row. ϕdialect robustness implies that the top row is scored as more similar to the middle row than to the bottom row. dialect awareness implies that the quality of the surface form in each row should be highest when paired with the correct dialect label. is semantic equivalence realistic? the above definitions presume that it is possible to characterize utterances in different dialects as semantically equivalent. such characterizations have been criticized as lacking a strong foundation for semantic equivalence, outside the limited case in which the dialect differences are purely phonological @xcite . one such criticism is that a pair of utterances might be semantically equivalent for some communicative purposes, but not for others. to avoid the gray area between dialect differences that change semantics and those that do not, we design perturbations that have a small surface-level impact on the original utterance but a strong effect on its meaning, e.g. by negating the main proposition or changing an important semantic argument. this establishes a necessary condition for dialect robustness: if a metric scores such perturbations more highly than dialect pairs, then it is certainly not dialect robust. proving that a metric is dialect robust is more difficult, because it requires constructing more subtle semantic perturbations that are harder to distinguish (even conceptually) from dialect variables. furthermore, from a practical standpoint we cannot evaluate y (d) with respect to all semantic perturbations ỹ ∈ ϕ(y (d) ), but the existence of perturbations for which m(y ## existing metrics to assess the quality of a generated text, most automatic evaluation approaches compare it to a "ground truth" reference, with higher similarity to the reference implying higher-quality output @xcite . similarity can be based on lexical features, as in bleu @xcite and chrf @xcite , or distributed repre-sentations, as in bleurt @xcite ), foot_0 prism @xcite and yisi @xcite . when distributed representations are used, they may be unsupervised @xcite or finetuned on a corpus of human ratings. in addition to these similarity-based metrics, there are also reference-free metrics for quality estimation (e.g., comet-eq; rei et al., 2020), which we discuss in §5.2. existing distributed metrics either use the multilingual representation from pretrained models, or create multilingual training data through various augmentation strategies. however, none of them explicitly accounts for dialectal variation. ## testing dialect robustness in this section, we describe our methodology for assessing dialect robustness. we first introduce two ways to perturb sentences to get two comparable metrics' outputs and then describe the statistical tests we use to aggregate the outputs over a corpus. ## nano we hypothesize that explicitly encoding dialect information while pretraining a model will lead to improved downstream robustness. to test this hypothesis on learned metrics for text generation, we introduce nano, foot_1 a model-agnostic pretraining schema with the goal of improving dialect robustness without performance degradation on downstream metric benchmarks. ## experiments in this section, we demonstrate that existing metrics are not dialect robust by applying our proposed methods and statistical tests to existing corpora in english, portuguese, and mandarin ( §6.1). we ## related work most popular nlp datasets and evaluation metrics do not take dialectal variation into consideration. for example, machine translation systems are usually evaluated by whether they match references in the target language, for which the dialect is generally unspecified @xcite . the subcommunity that has attended most to dialect is the vardial series of workshops, which has featured shared tasks such as dialect classification @xcite , translation between dialects @xcite , and transfer of nlp systems across dialects @xcite . of this prior work, dialect classification is clearly relevant to the criterion of dialect awareness introduced in definition 2.3 (see also @xcite , but our goal is to reward system outputs that match a target dialect rather than to classify the dialect of existing human-written texts. a related line of work has focused on inducing dialect features from corpora @xcite @xcite and on recognizing dialect features in text @xcite . following the feature-based view of dialect, we use cross-dialectal minimal pairs to measure dialect robustness in §4.1. on the more specific topic of dialect-aware evaluation, classic approaches focused on the creation of dialect-specific test sets, e.g. for translation to and from arabic dialects (e.g., @xcite . this idea has been extended to modern multi-task natural language understanding benchmarks by the value project @xcite , which used transformation rules to convert the glue benchmarks @xcite into african-american english. our evaluation in §4.2 builds on the frmt dataset of multi-dialectal translations @xcite to evaluate metrics for dialect robustness. however, in many cases it is not feasible to produce multi-dialect references or test sets. in these cases, dialect-robust and dialect-aware metrics can provide a more efficient solution, particu-larly if these capabilities can be achieved through a pretraining step like nano, which can be transferred to multiple tasks and evaluation settings. our work is additionally motivated by several papers that demonstrate the social impact of the failure to consider dialect variation in language technology. for example, literature shows that the out-of-the-box pos taggers @xcite and language identification and dependency parsing tools @xcite perform poorly on aave texts. other work has demonstrated large racial disparities in the performance of commercial speech recognition systems @xcite . our results contribute to this line of work by showing that metrics for text generation tend to penalize dialectal variants. we view the design of dialect-robust and dialect-aware metrics like nano as a step towards making language technology that works more broadly across dialects. ## conclusion and future work we introduce and formalize the dialect robustness and dialect awareness in the context of generation evaluation. grounded by a suite of statistical tests, we find that existing evaluation methods are not robust to dialects. as a first step toward a solution to this problem, we propose nano as a pretraining strategy. our experiments demonstrate that nano offers a size-efficient way to improve both the dialect robustness, shows the preliminary success towards dialect awareness and improves the metric performance of metrics on wmt benchmark. due to the limited availability of dialect-parallel corpora, our robustness tests are conducted in thousands of examples for mandarin and portuguese and hundreds of examples for english, which is insufficient to capture the full extent of these languages. we encourage future work to develop more resources, including benchmarks and corpora to conduct research on dialects for nlg evaluation. due to this limitation, our work focuses on dialect robustness and only briefly evaluates dialect awareness. future works may extend the details and criteria of the dialect-aware nlg evaluation, and we hope our work can serve as a baseline in this new research direction. our encouraging preliminary results lead us to urge researchers to consider and improve the dialect diversity during pretraining.