Validating Predictive Models Of Evaluative Language For Controllable D ata2 T ext Generation (2023)

Abstract: In data2text generation, tabular data is transformed into a text that expresses information from that source domain. While some text types, such as instructions, demand objective and neutral language without any expressive and evaluative content, many other text types are expected to provide expressions for these kinds of subjective meanings. In controllable, pipelined neural NLG separate learning models, notably regression models, can be used to predict whether some feature deviates sufficiently strongly from an expected value, so that evaluative language would be appropriate for verbalizing this finding. In this paper, we present an empirical study on the comprehension of evaluative adverbs and adjectival modifiers in car reviews, a text type that is characterized by a mixture of factual information with evaluations expressing positive or negative surprise. We show to what extend regression-based decision boundaries for producing evaluative content in controllable data2text NLG match the reader’s expectations that are raised by those evaluative markers. Finally we show that regression values in combination with standard deviation of the technical input data constitute reasonable Boolean thresholds for both positive and negative surprise, which provide the basis for the development of more complex models that also include the scalar base of adverbs and modifiers.

## introduction in controllable data-to-text natural language generation (henceforth data2text), tabular data is transformed into surface text that structures, relates and verbalizes the relevant pieces of information as close as possible to the domain-specific characteristic features and structures of the corresponding text types. the goal of adjusting the output to style, tone and structure of typical texts in the respective domain is achieved by either manually encoding template-based systems, or by fine-tuning large lan-guage models (llms) like bert or gpt-x on a subset of the corpus. an application domain that is especially challenging in regard to data2text nlg are car reviews. in these texts, technical information is seamlessly interwoven with subjective impressions of the test driver and, even more important, with the test driver's assessment of the car's features against his experience with cars of comparable quality and class. although a trustworthy system should not generate information that is not present in the underlying database of car features, it is possible to approximate the domain knowledge needed to reproduce the expert's assessment of the car's technical features. for example, sentence (b) in table (1) from a driving report about the lotus elise sport 2015 cannot be generated from purely technical information about the vehicle. it contains additional, subjective information on the driving experience. if a system for controllable data2text nlg shall be capable of generating sentences of this kind, the respective information must be added or the verbalisation be hard-wired into the system, including a mechanism for deciding when it is suitable. talking about being pressed back into the seat would be inadequate, even quirky, if the car was a compact car with a 60 hp motor. llms may encounter such expressions during training and reproduce it when realizing the features for the target text, but controlling style and content for data2text with an llm is a non-trivial task, which is why most nlg systems in-use do not rely on them (yet). accordingly, in sentence (a) in table (1), where the strong acceleration is contrasted with a comparably low motor power, controllable data2text nlg needs a mechanism for determining when the mismatch between acceleration and power output is large enough to permit the usage of an evaluative adverb like surprisingly and respective modifiers like enormous. again, llms may learn using (a) surprisingly, the sports car has an enormous acceleration from 0 to 60 in under 6.5 seconds, given the comparably low power of 136 hp. (b) when you hit the gas, the first law of motion kicks in and you feel that well-known feeling rise in your stomach when being pressed back into your seat. (c) the reason for this is that the lotus elise has an exceptionally low kerb weight of 1931 pounds. sentence (c) finally gives the reason for the high acceleration given the low hp: an exceptionally low kerb weight of only 1931 pounds. the adverb exceptionally signals the impossibility to predict a value of 6.5 seconds for acceleration when only considering the power output, but when considering the joint effect with a tiny weight, the acceleration is, albeit a rare combination, technically possible. the exemplary excerpt demonstrates the relevance of evaluative expressions on sentence and discourse level. however, although evaluative meaning often implies a subjective estimation why some features or states have a positive or negative stance, these evaluations rely on experiences and expectations that can be inferred from underlying data. we consider evaluative expressions as a key element to generating humanlike, pragmatically rich text and understanding the underlying mechanisms for producing evaluative language in general. this is of special importance when solely being informative is not the fundamental driving force for communication. evaluative adverbs and modifiers, which we focus on, are just one way of realizing a mentionable data mismatch that is present at the early stage of document planning. different nlg systems, either traditional or neural models, may then proceed differently for realizing the respective evaluative content and including it in the generated output. in this paper, we describe an empirical study on the comprehension of evaluative expressions in driving reports, performed with non-experts in order to identify the relation between their evaluative interpretation and the underlying technical data. the study inverts the typical acceptability rating approach of fixed sentences. the participants have to define lower and upper thresholds for numerical features that make the respective phrase acceptable, allowing us to empirically approximate intervals and their match with evaluative scales. we then compare the results with regression-based approaches to data analyses to show whether these models are adequate for dynamically determining decision boundaries of evaluative language in text generation. we use the car domain with its comprehensive technical specifications, but the approach we are presenting should be transferable to any dataoriented generation model and corresponding texts with evaluative content related to these data, such as technical devices and their use, weather reports, or data and reports from sports events, such as football games, as in the rotowire corpus @xcite . ## related research evaluative expressions (and expressive constituents in general) have received much attention in formal semantics and pragmatics, revolving around the questions how these items can be integrated into a compositional framework, and how their meanings interact with truth-conditional and inferred content @xcite . as meaning analyses that are focusing on conditions for the use of these expressions, they can ignore the question of what world knowledge or data their use is based on, which is an indispensable aspect of data2text nlg however @xcite @xcite . evaluative adverbs and modifiers, the linguistic topic of this paper, are also attributed to emotion generation or affective language generation (de @xcite , where evaluative items are generated in order to convey information with a specific stance @xcite , for example communicating exam marks @xcite or tailoring texts to specific users @xcite . the same intention is inherent to car reviews that include driving reports, where the tone of the text is coloured by the author's opinion. experimental studies in this area of research are quite complex due to the multitude of influential factors regarding audience, personality, individual preferences and the level of knowledge or common ground. experimental results are often non-reproducible @xcite , since the target group is a decisive factor. affective language generation has been implemented in both, template-based nlg systems (de @xcite , as well as in neural end2end language models @xcite . regarding controllability, generating such evaluative markers with transformer models and llms in general poses a challenge due to toxicity and fact hallucination @xcite @xcite . first attempts have been made to adapt gpt models to and condition on the intended evaluative stance of the text to be generated @xcite . in recent approaches, end-to-end language models have been re-pipelined in order to enhance controllability of intermediate representations. @xcite propose a re-modularization of neural generation networks, chaining separately trainable and evaluable networks that are specialized for the different tasks of content selection, document planning and surface realisation. they show that these pipelined neural generation models outperform endto-end networks, especially on unseen data, where the latter tend to produce topic-unrelated, incoherent texts and hallucinations. in pipelined data2text generation it is possible to integrate the necessary calculations for determining the adequacy of evaluative content on a polarized scale @xcite with regression and classification @xcite and using large language models such as t5 @xcite for transformation into surface text. ## regression models for predicting expectations the technical data we are using have been extracted from the adac database, the database of europe's largest automobile association. the database contains technical information on vehicles of various types as well as independently written reports on these vehicles from adac's vehicle experts. we use this database due to the comprehensive technical details it provides in combination with the vehicle reports. commercial interests are not present. we have trained different regression models. the first is a standard linear regression model, which we expect to underfit the data since some of the features, e.g. 'power output' and 'acceleration', have exponential rather than a linear relation. the second is a polynomial regression model, which may underfit for the same reasons of mixed relation types between predictors and response. furthermore, we implemented a deep neural network with intermediate, non-linear layers for regression. for these networks, we used adam optimizer and mean absolute error as loss function. the best fitting model is the dnn with an mse of 2.73 and an r2 score of 0.62 for the examples given in figure (1 ). in general, experts in automotive engineering have expectations about certain features of a car, given its technical details such as a certain acceleration given the power output and kerb weight. the images in @xcite show relations between each pair of the features 'acceleration', 'power output' and 'weight' as extracted from the database. the color of each "+" marker represents the feature that is not present on the axes. for example, in subfigure (c) the color encodes the weight, light-blue indicating small numbers and light-brown indicating large numbers. the green dot in each sub-figure marks the data point for the lotus elise which has been described by the example sentences in table ( 1 ). the position of these data points outside of the data cloud is a strong indicator already for using evaluative expressions. however, we require a general approach for deciding whether information is evaluative at all. in figure (1a), there seems to be a linear relationship between power output and weight with strong variation of the power value for instances higher than 2500 kg. due to large variation, regression models agree ever less with each other with increasing power, the polynomial parabola turning down again, the dnn taking a steeper increase upwards. ## empirical study on expectation values raised by evaluative adverbs in order to evaluate the adequacy of the regression models, we need to consider how evaluative adverbs and modifiers are interpreted and whether the regression models correctly capture expectations of readers. for this reason, we conducted a study on the interpretation of these evaluative items and their influence on the expectations about numerical features in driving reports. the study is designed as a webserver application, participants from germany and austria as well as the usa were acquired via prolific. participants were selected w.r.t. their first language, highest education level (at least an academic bachelor degree), possession of a driving licence and ownership of a car. for each study in german and english, 50 participants took part. additionally, 50 further participants took part in a shorter ablation study on modifiers in german. the participants were paid 13 pounds per hour. 20 minutes were scheduled for each participant, but most were significantly faster (8-13 minutes median time). for the ablation study, 12 minutes were scheduled. for the german studies we analyse in this paper, no outliers had to be excluded. we took into account the features 'acceleration', 'mileage', 'maximum speed', 'power output' and 'displacement', the former four because they are the most well-known and intuitive features, the latter representing features that are less intuitively accessible. we used the german counterparts to the evaluative adverbs surprisingly, disappointingly, amazingly and unfortunately, as well as the modifiers good, bad, low, high, slow, fast, average and enormous. with a power output of 200 hp, the car goes from 0 to 60 mph in [. . . ] seconds. 2 amazingly, the car goes from 0 to 60 mph in only [. . . ] seconds with a power output of 200 hp. ## conclusion for controllable generation of evaluative adverbs in data2text nlg, we devised a learning-based, generalizable approach to approximate thresholds for binary decisions on the presence of evaluative lan-guage and validated our findings with an empirical study on the expectations raised by comprehending evaluative adverbs. the elicited data supports the adequacy of the model and shows a surprisingly good match between regression predictions, real data and human estimations. in a pipelined neural generation system, these learning-based models permit determining generation of evaluative language at an early stage in document planning and therefore improve controllability of evaluative content before applying transformers for surface realisation. supplementary materials availability statement: source code of regression models, source code of the web application for the empirical study, empirical data collected during the study as well as source code for analysis of the data including functions for reproducing all graphs and figures in this paper are available online on github: https://github.com/mmlangner/ evaladvindata2textnlg/ . the database containing technical data of cars is proprietary, which is why we are not allowed to distribute it. please contact the authors for information on how to obtain the technical database in the required format from the adac.