Perceptual Quality Dimensions of Machine-Generated Text with a Focus on Machine Translation (2022)

Abstract: The quality of machine-generated text is a complex construct consisting of various aspects and dimensions. We present a study that aims to uncover relevant perceptual quality dimensions for one type of machine-generated text, that is, Machine Translation. We conducted a crowdsourcing survey in the style of a Semantic Differential to collect attribute ratings for German MT outputs. An Exploratory Factor Analysis revealed the underlying perceptual dimensions. As a result, we extracted four factors that operate as relevant dimensions for the Quality of Experience of MT outputs: precision, complexity, grammaticality, and transparency.

## introduction in recent years, automatically generated text has increasingly gained importance, e.g., chatbots, automatic summarizations, or machine translations. although the quality of such texts has greatly improved over time, it has not yet reached human parity @xcite . therefore, the quality of machine-generated text is of ongoing interest to the research community and is further important for gaining acceptance in different applications. the quality of experience (qoe) is defined as "the degree of delight or annoyance of the user of an application or service" @xcite . this means that the qoe is a subjective perception that needs to be quantified in empirical studies @xcite . while there are standardized methods for auditory and visual media, such as itu p.800, p.910, or bt.500, the qoe of text has been mostly disregarded until now. the perceptual quality of machine-generated text is a highly complex construct. many aspects and dimensions play a crucial role; hence, it is the object of investigation of various research areas. we suggest that a multi-dimensional prediction model covering a wide variety of aspects is the best approach to assess the quality of machine-generated text. to the best of our knowledge, no such model exists. therefore, we are developing a prediction model for the quality of german machine-generated text, specifically, machine translation (mt). we aim to create our model based on a combination of linguistic data and automatically extractable factors that can predict the qoe of mt outputs. our first milestone is identifying relevant perceptual quality dimensions, the foundation of our model. we achieved this milestone by conducting a crowdsourcing study in the style of a semantic differential and subsequently extracting the quality dimensions through an exploratory factor analysis. ## related work this section provides an overview of the existing metrics for capturing the performance or quality of mt systems. the first category of metrics is automatic methods, which have the advantage of being fast, low-cost, and reproducible. the most commonly used metrics are bleu @xcite , meteor @xcite , comet @xcite , and prism @xcite . metrics like ter @xcite measure the translation edit rate, and quality estimation methods @xcite can predict the quality without access to the reference translation(s). however, one shared shortcoming of all these automatic metrics is that, as opposed to our approach, they are not based on relevant quality dimensions and thus lack diagnostic power. the second category of metrics is subjective methods for directly measuring quality that are more costly yet more reliable. there are largescale human rankings that are often conducted in international conferences in order to compare the performance and/or quality of several mt systems @xcite . the multidimensional quality metrics (mqm) is a framework for the manual assessment of translation quality @xcite . additionally, test suites have recently regained more importance. a test suite is a challenge set created to systematically analyze the behavior of mt systems in different aspects, e.g., @xcite , @xcite , or @xcite . while the mentioned techniques focus on capturing the performance or quality of mt systems, they cannot sufficiently capture the qoe by users of mt output as qoe is the only technique that is not measured by pre-defined criteria. instead, qoe is based on identifying relevant criteria (i.e., quality dimensions) in a real-world scenario. ## experimental setup we conducted a study to identify relevant dimensions for the quality of machine-generated text, specifically german mt outputs. we did so by utilizing a crowdsourcing survey in which participants had to rate mt outputs. our corpus contained english to german translations from the submissions to the news translation task of the fourth conference on machine translation (wmt19) foot_0 . we chose this data for our corpus as we needed test sentences from several mt systems with varying translation quality. furthermore, the data is freely available for research purposes foot_1 . we extracted a set of translations from six submitted systems that appeared at the top, the middle, and the bottom of the ranking of wmt19 systems @xcite , resulting in a corpus of 11,922 sentences. a linguistic expert created a sub-corpus for the survey, dedicating around 15 hours to carefully extract translations varying in length, quality, and error types. the sub-corpus consists of 45 sentences. 3 the survey was conducted as a semantic differential (sd) @xcite ). an sd is a rating scale that measures a person's attitude towards an entity, here: our test sentences. the participants were asked to rate their perception of the test items on a scale between two polar adjectives, e.g., "grammatical -ungrammatical". all adjective pairs used in the study can be found in we would like to emphasize that while we are using mt as an example text type, the focus of our study lies on the quality of machine-generated text. therefore, we solely work with the mt outputs and do not take the source sentences and concomitant quality aspects into account (as opposed to approaches that focus on the quality of mt). ## multidimensional analysis qoe can be formalized as a multidimensional perceptual space where the defining parameters function as dimensions. it is the aim of the multidimensional analysis to identify those dimensions for the qoe of mt output. ## conclusion and outlook we present a study exploring the relevant quality dimensions for mt outputs. we identified antonym pairs of a semantic differential in a preliminary study and used these attributes to rate 45 german test sentences. we then carried out an exploratory factor analysis that resulted in the extraction of four relevant quality dimensions: precision, complexity, grammaticality, and transparency. according to our study, these are the quality dimensions that are relevant for the qoe, i.e., the subjective perception of a user of a text. our ultimate goal is to develop a prediction model to assess the quality of machine-generated text. we focus on two text types: machine translation and automatic text summarization (ats). our next step is to identify the relevant quality dimensions for ats. to do so, we are currently conducting another crowdsourcing study with an adapted set of adjective pairs. the focus on two different types of machine-generated texts allows us to compare the (potential) differences in the perceptive quality dimensions and enables us to draw generalizations for other text types. simultaneously, we are working on the quantification of the quality dimensions for mt. as the factor analysis conducted in the study at hand is highly complex, we are developing a simplified survey in which we present only one representative antonym pair per dimension. if the result of the follow-up study verifies our current study, we can assume our dimensions to be accurate. further steps will involve correlating automatically extractable text parameters and quality dimensions, and building and testing various prediction models. these efforts should ultimately result in a quality prediction model for mt, ats, and potentially other types of machine-generated text. other potential future work includes analyzing the possible overlap between the four dimensions at hand and other existing quality metrics, e.g., mqm. furthermore, it would be of interest to expand the analysis to other languages, as it might also counteract the weird bias.