Using contextual and cross-lingual word embeddings to improve variety in template-based NLG for automated journalism (2021)

Abstract: In this work, we describe our efforts in improving the variety of language generated from a rule-based NLG system for automated journalism. We present two approaches: one based on inserting completely new words into sentences generated from templates, and another based on replacing words with synonyms. Our initial results from a human evaluation conducted in English indicate that these approaches successfully improve the variety of the language without significantly modifying sentence meaning. We also present variations of the methods applicable to low-resource languages, simulated here using Finnish, where cross-lingual aligned embeddings are harnessed to make use of linguistic resources in a high-resource language. A human evaluation indicates that while proposed methods show potential in the low-resource case, additional work is needed to improve their performance.

## introduction the use of automation to help journalists in news production is of great interest to many newsrooms across the world @xcite . natural language generation (nlg) methods have previously been employed, for example, to produce soccer reports @xcite , financial reports @xcite and weather forecasts @xcite . such 'automated journalism' @xcite or 'news automation' @xcite imposes restrictions on system aspects such as transparency, accuracy, modifiability, transferability and output's fluency @xcite . likely as a consequence of these requirements, news industry applications of nlg have traditionally employed the 'classical' rule-based approaches to nlg, rather than the more recent neural methods increasingly seen in recent academic literature @xcite . a major downside of these rule-based systems, however, is that their output often lacks variety. adding variety by increasing the amount of templates is possible, but this would significantly increase the cost of system creation and limits reuse potential. as users of automated journalism already find the difficulty of reuse limiting @xcite , this is not a sustainable solution. in this paper, we extend a modular rule-based nlg system -used for automated journalism in the domain of statistical news -with a dedicated component for varying the produced language in a controlled manner. the proposed extension enables two methods of inducing further variation: in insertion, new words are introduced into the generated text, whereas in replacement certain words in the original sentence are replaced with synonyms. to accomplish these tasks, we employ a combination of traditional language resources (e.g. synonym dictionaries) as well as recent neural processing models (i.e. word embeddings). these resources complement each other, enabling us to harness the power of statistical nlp tools while retaining control via the classical linguistic resources. we also experiment with using these methods in the context of a low-resource language which lacks linguistic resources such as synonym dictionaries. for this case, we propose to use cross-lingual aligned word embeddings to utilize a high-resource language's resources even within said low-resource language. in the next section, we briefly describe some related previous works and further motivate our approach. section 3 describes our proposed variation induction methods for both the high-resource and the low-resource contexts. sections 4 and 5, respectively, introduce our human evaluation method and the results obtained. sections 6 provides some additional thoughts on these results, while section 7 concludes the paper. ## background natural language generation has been associated with news production from the early years of the field, with some of the earliest industry applications of the nlg methods being in the domain of weather report production @xcite . interest in applying nlg to news production has only increased since, with many media houses experimenting with the technology @xcite . still, adoption of automated journalism methods has been slow. according to news media insiders, rule-based, classical, nlg system such as those described by @xcite , are costly to create and difficult to reuse @xcite . at the same time, even the most recent neural (end-to-end) approaches to nlg are not fit for customer needs as they limit the ability to "customise, configure, and control the content and terminology" @xcite . another major problem is the fact they suffer from a form of overfitting known as 'hallucination', where ungrounded output text is produced. this is catastrophic in automated journalism. concurrently with works on improved neural nlg methods, others have investigated increasingly modular rule-based approaches with the intent of addressing the reusability problem described by @xcite . for example, @xcite describe a modular rule-based system for automated journalism that seeks to separate text domain specific processing from language specific processing to allow for easier transfer of the system to new text domains. while such rule-based approaches produce output that is grammatically and factually correct @xcite , they often suffer from a lack of variety in language. this is especially true for systems that are based on some type of templates, or fragmentary language resources that are combined to form larger segments of text and into which content dependent on system input is embedded. using such templates (or hand-crafted grammars) is costly, especially when a large number is required for varied output. as template (or grammar) production can be costly, automated variation induction methods that could be integrated into rule-based systems are very interesting. one trivial approach to inducing variation would be to employ a synonym dictionary, such as is available in wordnet @xcite , to replace words within the generated text with their synonyms. this approach, however, suffers from some major problems. first, simply looking up all synonyms for all meanings of a token is not feasible due to polysemy and homonymy. at the same time, incorporating knowledge of which semantic meaning of a token is correct in each case significantly slows down template and grammar generation. furthermore, even within a certain semantic meaning, the various (near) synonyms might not be equally suitable for a given context. finally, such linguistic resources are not available for many low-resource languages. an alternative approach, more suited to generation within medium and low-resource languages where there are no available synonym dictionaries, but large text corpora can be collected, would be to use word embeddings (e.g. @xcite @xcite to identify words that are semantically close to the words in the template. this approach, however, suffers from the fact that both synonyms and antonyms of a word reside close to it in the word embedding space. while potential solutions have been proposed (e.g. nguyen et al., 2016), they are not foolproof. ## variety induction algorithms as described above, naïve methods based on either classical linguistic resources or word embeddings alone are not suitable for variation induction. to this end, we are interested in identifying a simple variety induction method that combines the positive sides of both the classical linguistic resources (such as synonym dictionaries) with those of statistical resources such as word embeddings. optimally, the method should also function for a wide variety of languages, including low-resource languages where costly resources such as comprehensive synonym dictionaries are not readily available. in this work, we introduce variety into the generated language using two distinct methods: by introducing completely new words into sentences, and by replacing existing words. we will use the terms insertion and replacement to distinguish between the two approaches, respectively. ## evaluation we have implemented the above algorithms within a multi-lingual (finnish and english) natural language generation system that conducts automated journalism from time-series data provided by eurostat (the statistical office of the european union). the system is derived from the template-based modular architecture presented by @xcite . it produces text describing the most salient factors of the input data in several languages in a technically accurate manner using only a few templates, but the resulting language is very stiff, and the sentences are very alike. this makes the final report very repetitive and thus a good candidate for variety induction. for all of the algorithms described, we utilise the same trilingual bert model: finest bert @xcite . the finest bert model is trained with monolingual corpora for english, finnish and estonian from a mixture of news articles and a general web crawl. in addition to the bert model, the low-resource language variants of the algorithms utilize crosslingual pairwise aligned word embeddings for word 'translations'. we use monolingual fasttext @xcite word embeddings mapped with vecmap @xcite to form the cross-lingual embeddings. pos tagging is done with nltk @xcite and the lexical database used as a synonym dictionary is word-net @xcite . a human evaluation of our methods was conducted following the best practices proposed by @xcite . in the evaluation setting, judges were first presented with three statements about a sentence pair. sentence 1 of the pair was an original sentence, generated by the nlg system without variation induction. sentence 2 of the pair was the same sentence with a variation induction procedure applied. cases where the sentence would remain unchanged, or where no insertion/replacement candidates were identified, were ruled out from the evaluation set. the part of the sentence to be modified was marked in the original sentence and the inserted/replaced word highlighted. the judges were asked to evaluate the following statements on a likert scale ranging from 1 ('strongly disagree') to 4 ('neither agree nor disagree') to 7 ('strongly agree'): q1: sentence 1 is a good quality sentence in the target language. q2: sentence 2 is a good quality sentence in the target language. q3: sentences 1 and 2 have essentially the same meaning. in addition to the two sentences, the judges were presented with two groups of words to examine if using the scores by bert would correctly distinguish suitable words from unsuitable words. group 1 contained the words scored as acceptable by bert while group 2 contained the words ruled out due to a low score. all words in both groups met the criteria of being synonyms (in the case of replacement) or being the correct pos (in the case of insertion). the judges were asked to evaluate the following questions on a 5-point likert scale ranging from 1 ('none of the words') to 3 ('half of the words') to 5 ('all of the words'): q4: how many of the words in word group 1 could be used in the marked place in sentence 1 so that the meaning remains essentially the same? q5: how many of the words in word group 2 could be used in the marked place in sentence 1 so that the meaning remains essentially the same? for the high-resource language results, we gathered 3 judgements each for 100 sentence pairs. the judges were recruited from an online crowdsourcing platform and they received a monetary reward for participating in the study. the judge recruitment was restricted to countries where majority of people are native speakers of english. for the low-resource language results, 21 judges evaluated 20 sentence pairs. the judges were recruited via student mailing lists of university of helsinki in finland and were not compensated monetarily. all but one of the participants in the low-resource evaluation were native speakers of the target language. the final participant self-identified as having a 'working proficiency.' ## results in the high-resource insertion case, the results indicate that inducing variation using the proposed method does not decrease output quality, as both the original sentences' qualities (q1 mean 5.57) and modified sentences' qualities (q2 mean 5.76) were similar. as the sentence meaning also remained largely unchanged (q3 mean 5.54), we interpret this result as a success. the results for q4 and q5 indicate that our filtering method based on a threshold language model score can be improved: results for q4 (mean 3.11 on a 5-point likert scale) indicate that unsuitable words are left unfiltered, while q5 (mean 3.03) indicates that some acceptable words are filtered out. values are the mean evaluation result and the standard deviation (in parentheses). in the context of the statements, sentence 1 is the original, unmodified sentence, while sentence 2 is a sentence with added variety. in the low-resource case insertion, we observe some change in meaning (q3 mean value 4.34) and a slight loss of quality, but even after variety induction the output quality is acceptable (q1 mean 6.43 vs. q2 mean 5.12). interestingly, in the low-resource setting, we observe that the language model is slightly better at distinguishing between suitable and unsuitable candidates (q4 and q5 means 2.53 and 1.46, respectively) than in the high-resource case. we are, at this point, uncertain of the reason behind the difference in the ratios of q4 and q5 answers between the high-resource and the low-resource case. notably, even this 'better' result is far from perfect. we also conducted pos tag specific analyses for both the high-resource and the low-resource insertion cases. in the high-resource case, no major differences were observed between various pos tags. in the low-resource (finnish) case, however, we observed that with some pos tags, such as adverbs, the results are similar to those observed with english. low-resource results for adverbs only are shown in figure 3 . we emphasize that this is the best observed subresult and should be viewed as post-hoc analysis. in the high-resource replacement case, we observe promising results. inducing variation did not negatively affect sentence quality (q1 mean 5.55 vs. q2 mean 5.60) and concurrently retained meaning (q3 mean 5.65). results for q4 and q5 (means 3.39 and 3.21, respectively) indicate that, as above, the filtering method still has room for improvement, with poor quality options passing the filter and high-quality options being filtered out. however, in the low-resource case replacement case, we observe a significant drop in sentence quality after variation induction (q1 mean 6.67 vs q2 mean 3.89), as well as significant change in sentence meaning (q3 mean 3.39). while q5 results are relatively good (mean 1.62), as in very few if any good candidate words are filtered out, q4 results (mean 1.76) indicate some fundamental problem in the candidate generation process: as there are few if any good candidates in either group, it seems that most of the proposed words are unsuitable. -100 -80 -60 -40 -20 0 20 40 60 80 100 percentage meaning remained essentially the same modified sentence is good finnish original sentence is good finnish strongly disagree quite strongly disagree somewhat disagree neither agree nor disagree somewhat agree quite strongly agree strongly agree figure 3 : quality of sentences with low-resource insertion in finnish with english as the high-resource language, and preservation of sentence meaning. results shown for adverbs only, representing the best observed performance across the various parts of speech generated. we emphasize that the graph shows only a subset of the complete results (see ## discussion our high-resource results indicate that the proposed approach is suitable for inducing some light variation into automatically generated language. the use of synonym dictionaries removes the need to manually construct variants into the templates used in the generation, while the use of language models allows for contextual scoring of the proposed variants so that higher quality results are selected. we suspect that a major contributor to the low quality of the modified sentences in the lowresource scenarios was the complex morphology of the finnish language. especially in the case of finnish, the process wherein the original word was grammatically analyzed and the replacement word reinflected into the same form would have likely resulted in cases where the resulting word is technically grammatically feasible in isolation, but not grammatical in the context of the rest of the sentence. our post-hoc investigation also indicates that at least in some cases the resulting reinflected words were outright ungrammatical. in addition, it seems that the language model employed did not successfully distinguish these failure cases from plausible cases, which led to significant amounts of ungrammatical words populating the proposed set of replacement words. our post-hoc analysis further indicates that the methods led to better results when use of compound words was avoided in the finnish templates. we hypothesize that applying the method to a morphologically less complex language might yield significantly better results. at the same time, in the case of low-resource variation induction using insertion, our results indi-cate that some success could be found if the method is applied while restrained to certain pre-screened parts of speech, such as adverbs (see figure 3 ). this further indicates that the performance of the replacement approach might be improved significantly if the morphology issues were corrected. notably, our analysis of the results did not include an in-depth error analysis to determine what parts of the relatively complex procedure fundamentally caused the errors, i.e. were the errors introduced during pos-tagging, language model based scoring, or some other stage. furthermore, we did not rigorously analyze whether the generation errors were semantic or grammatical in nature. as a final note, we emphasise that these results were evaluated on local (sentence) rather than on global (full news report) level. we anticipate that, for example, when inserting a word like 'still' in a sentence (see figure 1 ), the results might differ when evaluating on a global level. ## conclusions in this work, we proposed two approaches, with variations for both high-resource and low-resource languages, for increasing the variety of language in nlg system output in context of news, and presented empirical results obtained by human evaluation. the evaluation suggests that the high-resource variants of our approaches are promising: using them in the context of a case study did create variety, while preserving quality and meaning. the low-resource variants did not perform as well, but we show that there are some positive glimpses in these initial results, and suggest future improvements.