Imaginary Numbers! Evaluating Numerical Referring Expressions by Neural End-to-End Surface Realization Systems (2024)

Abstract: Neural end-to-end surface realizers output more fluent texts than classical architectures. However, they tend to suffer from adequacy problems, in particular hallucinations in numerical referring expression generation. This poses a problem to language generation in sensitive domains, as is the case of robot journalism covering COVID-19 and Amazon deforestation. We propose an approach whereby numerical referring expressions are converted from digits to plain word form descriptions prior to being fed to state-of-the-art Large Language Models. We conduct automatic and human evaluations to report the best strategy to numerical superficial realization. Code and data are publicly available.

## introduction the significant advances in deep learning for nlp and its enormous success in other text generation tasks, such as machine translation @xcite . as a result, approaches to surface realization of data-to-text systems have moved from traditional modular pipeline architectures @xcite to end-to-end ones. these systems transform a simple meaning representation into text without any explicit intermediate representations @xcite @xcite . while early neural data-to-text systems required a high amount of parallel training data, current stateof-the-art (sota) architectures, known as large language models (llms) @xcite @xcite , can deliver impressive results with less training, even excelling in zero-short or few-shot settings. with respect to linguistic output, neural end-toend surface realizers appear to generate more fluent text than classical pipeline architectures but are more likely to suffer from (semantic) adequacy problems, in particular, hallucinations @xcite , whereby the system produces text that contains information which is not present in the input representation. a particular hallucination problem that modern approaches seem to struggle with, unlike classical architectures, is numerical referring expression generation @xcite @xcite . for instance, let's hypothesize the case where a surface realizer produces the outcome: "the country registered 458,098 cases of covid-19", whereas the gold-standard reference points to "the country registered 408,098 cases of covid-19". albeit there is only a single-digit difference between both texts (which can be overlooked by popular automatic quality metrics), the difference represents an arithmetic change of 50,000 and may lead readers to make drastic errors given the sensitivity of the context. to the best of our knowledge, this problem has never been investigated in surface realization systems, despite having been addressed in other generation tasks such as text normalization @xcite , question-answering @xcite , and text-tospeech (nikulásdóttir and guðnason, 2019); tasks which also struggle to synthesize texts with numerical referring expressions represented by digits. one approach to circumvent the problem in textto-speech systems is to normalise the input texts by converting numerical referring expressions from digits to plain word form descriptions prior to being fed into the system @xcite . another technique used in referring expression generation (reg) systems is slot-filling or delexicalisation where values like date, number, or constants are represented as a literal @xcite . in the context of end-to-end surface realizers, this study raises two questions: (rq1) how well do state-of-the-art end-to-end surface realizers generate numerical referring expressions? (rq2) are numerical referring expressions better verbalized when represented by digits or text (spell-out form)? to answer these questions, we conducted automatic and human evaluations with three sota llms: gpt-2 @xcite , bart @xcite , t5 @xcite , and their multilingual counterparts. these models were used to verbalize english and brazilian portuguese news about amazon deforestation, fire alerts, and covid-19 cases using four different strategies, which we discuss in section 3. code and data will be publicly available. ## data for training and evaluating the models, we used automatic-generated reports by botsdobem, a group of twitter robot-journalists such as coronareporter foot_1 and damata foot_2 , which publish news in brazilian portuguese and english. for brazilian portuguese, the dataset comprises of i) both daily and monthly reports on deforestation in the legal amazon area of brazil @xcite , ii) daily reports about fires in the brazilian biomes, as well as iii) covid-19 cases in the country @xcite . for english, the dataset comprises of daily and monthly reports on deforestation in the legal amazon. although automatically generated, these texts contain a high number of numerical referring expressions, making them suitable for our goal of evaluating how well neural end-to-end surface realizers generate numerical referring expressions. figure 1 illustrates the structure of instances in both the english and portuguese datasets, which consist of meaning representations starting with a tag representing the report domain, followed by a tag that marks the beginning of the sentence intents (e.g., intents). each intent in the meaning representation follows the intent-attribute-value schema. finally, the tag [history] marks where the verbalization of the previous sentences in the paragraph of the target will be depicted. in the example, the tag [paragraph] means that the target sentence is at the beginning of the paragraph. ## numerical referring expressions to evaluate the effectiveness of a neural end-toend surface realizer in generating numerical expressions, we consider two forms of number representation: digits and word (spell-out) form descriptions. these are assessed in both the meaning representations and the verbalizations, resulting in a total of four distinct strategies: to exemplify, numeric referring expressions strategies area month year input mr no desc 322.91 4 2021 in april 2021, 322.91 sq km of the legal amazon were deforested, according to data from the national institute for space research (inpe) desc src three hundred and twenty-two point nine one four two thousand and twenty-one in april 2021, 322.91 sq km of the legal amazon were deforested, according to data from the national institute for space research (inpe). desc trg 322.91 4 2021 in april two thousand and twenty-one, three hundred and twenty-two point nine one sq km of the legal amazon were deforested, according to data from the national institute for space research (inpe). desc three hundred and twenty-two point nine one four two thousand and twenty-one in april two thousand and twenty-one, three hundred and twenty-two point nine one sq km of the legal amazon were deforested, according to data from the national institute for space research (inpe). the python library foot_3 , num2words, to transform numerical digits into their textual counterparts. this library is effective for both english and brazilian portuguese languages. ## experiments to address our first research question (rq1), we evaluate the performance of three llms in generating numerical references: i) gpt-2, ii) bart, and iii) t5 for english domains. additionally, for portuguese, we fine-tuned gportuguese-2 @xcite , a brazilian portuguese version of gpt-2, as well as mbart-50 @xcite and mt5 @xcite , which are the multilingual versions of bart and t5, respectively. these models were selected due to a more sustainable perspective of llms @xcite and the environmental implications of the new llms, such as chatgpt (openai, 2023) and bard foot_4 . the model training process involved 30 epochs, a learning rate of 1e-5, a batch size of 1, 5 early stops, and a maximum token length of 300. ## results the error rates and bleu scores for each numerical strategy and model for both english and portuguese are presented in error rate full results for english (en) and brazilian portuguese (pt) s lm number named entity word context uncheckable other fluency bleu en pt en pt en pt en pt en pt en pt en pt en pt en pt no desc in english, the average results per strategy indicated that using text to represent numerical references did not yield a positive impact. this is evidenced by the no desc strategy, which resulted in the lowest error rate. however, when examining the results per model, t5(desc source) strategy presented the lowest error rate, followed by bart(desc source) and t5(no desc) strategies. in terms of automatic evaluation, the desc target strategy yielded the highest bleu score with t5 being the best model in this strategy. the kappa coefficient for inter-rater agreement regarding in-correct number error for both languages reached up to 0.90 according to contrary to english, describing portuguese numerical referring expressions in the desc source strategy resulted in the lowest error rate. the model with the fewest errors was gpt2-pt(desc source) strategy. regarding the automatic evaluation, the desc strategy yielded the highest bleu score (0.68) with mt5, being the best model in this strategy for portuguese. it is important to note that brazilian portuguese approaches were evaluated across more domains than their english counterparts due to differences in both datasets. to compare the numerical error rate of models across languages, ## conclusion and limitations finally, we revisit the research questions outlined in section 1: (rq1) a human evaluation was performed to annotate different error categories, such as numerical, named entities, context, word, uncheckable, other, and fluency errors. results depicted across languages, models, and numer-ical strategies show the numerical error rate as the highest among the errors. hence, concerning this research question, there is clear evidence that pure state-of-the-art large language models struggle to generate adequate and faithful numerical referring expressions. (rq2) results demonstrated that the brazilian portuguese approach desc source performs better. however, for english, representing numerical references in spell-out form did not help regardless of whether it was present in the source meaning representation (desc source), in the target text (desc target) or both (desc). as depicted in as evidenced in the results, this study confirms that large language models struggle to generate numerical referring expressions, although t5 has performed better. the proposed strategy to solve the problem did not affect english, although it decreased numerical errors when describing the numbers on the source of portuguese trials. hence this strategy for describing numbers may help in lowresource scenarios. for future work, we plan to extend our experiments to gpt3 and gpt4 foot_5 . however, since these models are neither free, nor reproducible due to limited or no information concerning model size, architecture, training parameters, and data set creation, we will investigate related open-source variations such as bloom foot_6 and gpt-j foot_7 .