Evaluation of Abstractive Summarisation Models with Machine Translation in Deliberative Processes (2021)

Abstract: We present work on summarising deliberative processes for non-English languages. Unlike commonly studied datasets, such as news articles, this deliberation dataset reflects difficulties of combining multiple narratives, mostly of poor grammatical quality, in a single text. We report an extensive evaluation of a wide range of abstractive summarisation models in combination with an off-the-shelf machine translation model. Texts are translated into English, summarised, and translated back to the original language. We obtain promising results regarding the fluency, consistency and relevance of the summaries produced. Our approach is easy to implement for many languages for production purposes by simply changing the translation model.

## introduction the processes of deliberation and collective intelligence production have evolved radically thanks to the possibility of carrying them out digitally. however, this often results in large amounts of generated content in the deliberations, causing information overload that prevents their potential from being fully realised @xcite @xcite . to address this, we evaluate the potential value of abstractive summarisation models when combined together with a machine translation system in synthesising and filtering information collected through such processes. whereas the current technology of language models is mostly limited to a few languages, which creates a barrier to their more widespread use, our approach can be deployed for many languages just by changing the translation model without the need to generate new, ad-hoc corpora for the task or costly retraining for each new language. the current evaluation is done in a spanish deliberation dataset. we have carried out an evaluation with 6 abstractive summarisation models: bart @xcite , t5 @xcite , bert (presumm -bertsumextabs: liu and lapata, 2019), pg (pointer-generator with coverage penalty) @xcite , copytransformer @xcite , and fastabsrl @xcite . those models are applied in combination with the machine translation system marianmt @xcite using the opus-mt models @xcite . we have evaluated the quality of the summaries for each model and their comparison. early research on the problem of text summarisation in low resourced languages (although not focused on deliberation) @xcite demonstrated the limitations of machine translation systems at that time. recently, @xcite revisited the problem of low quality translations in low resourced languages and successfully demonstrated the possibility of using abstractive summarisation by retraining their model on corpora that have gone through the same machine translation process. in this study, we complete the cycle, translating from the original language to english, summarising, and translating back to the original language, thus avoiding the need for retraining. using other approaches, @xcite studied english-to-chinese summarisation combining an extractive approach with a process of sentence compression that effectively abstracts the results. duan et al. (2019), following shen et al. (2018) , exploited the capability of a resource-rich language summariser in a teacher-student framework that connects it to the target language summariser. ## dataset the evaluation has been carried out with a dataset from deliberative processes in spanish, which was translated into english to carry out the summarisation. the generated summaries were then translated back into spanish for evaluation. thus, the evaluators evaluated summaries of spanish texts. the dataset is available in the madrid city council 'datos abiertos' repository foot_0 , called 'comentarios'. it contains public deliberations in relation to citizen proposals submitted to the participation platform of the city council. the dataset has been selected due to the great success of the participation platform, which has led to 26, 400 proposals and 125, 135 comments being submitted. this is one of the most successful cases of digital participation in the world and is therefore a perfect case study for evaluating the information overload problem in deliberative processes @xcite . each proposal presents a debate space where public comments can be found. forty debates were selected covering different deliberation scenarios in the dataset. these represent three cases: 20 debates with (@xmath0, the most common case of debates with few comments; 15 debates with (20 ≤ @xmath1, for the medium case; and 5 debates with (60 ≤ @xmath2, the large number of comments case. the debates were also selected to cover three different comment scenarios, i.e., from very short to very lengthy comments. in the first scenario from 1, 000 to 5, 000 total characters; in the medium scenario from 3, 000 to 13, 000; and in the large scenario from 10, 000 to 18, 000 characters. for each debate the text to summarise was created by concatenating its comments into a single text. by using debates from all scenarios regarding the number of comments and comment length we ensure that the selection is not biased to a specific scenario of deliberation that could skew our results. examples of the debates can be found in the appendix, illustrating the combination of multiple narratives through the different comments and the poor grammatical quality of the texts. ## abstractive summarisation methodology different models were selected, covering some of the best available summarisers, but also different model architectures: this uses the transformer architecture, but one attention head defines the copy distribution. the 'opennmt transformer' pretrained model 4 has been used. the reported rouge scores of these models @xcite are shown in additional models were also evaluated: adversarial reinforce gan @xcite matching @xcite , joining elmo with a domain fluency model; poda @xcite , denoising autoencoder transformer with a pointer-generator layer; and genparse @xcite , combining sequential word generation with tree-based parsing. our initial qualitative evaluation found that none of them were competitive enough with the selected models. several of these models work at the sentence level, which may impact their relevance in our deliberative case, where texts are composed of multiple authors' comments. the machine translation system used was mar-ianmt (junczys-dowmunt et al., 2018) using its huggingface implementation, with opus-mt models foot_6 developed by the helsinki-nlp group. machine translation was first applied to the original text of the deliberations before applying the summarisers, and then to the summary generated to convert back to the original language (see appendix). thus, even when the summarisation models are trained with english datasets, the full system can be used in deliberations of any language supported by the machine translation system. the opus-mt models used in this work count currently with pre-trained models for 1738 language pairs. it is left for future work to evaluate the effect of the translation model, and to apply it to other languages to determine their quality. the models used here show a good performance (see bleu scores in opusmten; opusmtes) for the languages used. ## evaluation design we developed a protocol for the human evaluation of the summaries generated by the different models, following designs used in previous studies (amplayo and lapata, 2020; @xcite @xcite @xcite . first, the different models were compared regarding their relative overall quality using the best-worst scaling @xcite , shown to be more accurate than a generic individual scoring model, and simultaneously reducing the number of assessments required @xcite . for each debate, 6 different summaries were generated, one for each of the models to be evaluated. these summaries were organised in 9 tuples of 4 elements each, where each summary appeared in 6 of the tuples in random order not allowing the evaluator to identify each model used. in total, considering all the debates, 360 tuples were produced. each of these tuples was evaluated by 5 independent evaluators (native spanish speakers with a minimum education level of a bachelor's degree), producing a total of 1, 800 evaluations. the score for each summary consisted of the percentage of times it was evaluated as best, minus the percentage of times it was evaluated as worst. in addition, a second evaluation was carried out for two summaries in each debate. the models were selected randomly in each case, while ensuring that each model had the same number of evaluations. here, we were interested in whether the models produce results of sufficient quality to be useful to participants in the debate. thus, we we used an absolute rather than a relative score. we asked evaluators to rate the following (definitions were shared with evaluators) on a likert scale from 1 (strongly disagree) to 4 (strongly agree): ## evaluation results the results obtained for the overall comparison between models are shown in the results for the evaluation of the qualitative aspects of each summariser are shown in in this individual evaluation of each model, it can be seen again how bart obtains the best ratings in all four categories evaluated. bert is the second best for the categories of 'informativeness', 'fluency' and 'consistency', while pg jumps to the second position for 'creativity'. t5 is in the third position for the categories 'informativeness' and 'fluency' and pg is the third best for 'consistency'. this confirms the best results of bart and bert, and a close result for t5 for generating informative summaries, but a poorer result for fluency. this may be the reason why the t5 model performed worse in the general overall comparison. bart and bert perform well in terms of 'consistency', with scores close to 3. they perform a bit worse for 'fluency' and 'informativeness', around the middle of the possible rating 2.5. regarding 'creativity', the models have a poor performance, with a score of around 2, meaning that they tend to copy instead of paraphrase. ## conclusions in this study we have evaluated the application of state-of-the-art, abstractive summarisation models to deliberative processes in spanish using an off-the-shelf machine translation model. although we focused on spanish in this study, our proposed pipeline can be easily deployed without additional effort to many other languages. this offers significant benefits for production applications (especially cases dealing with wide ranges of languages) that are rarely available in other approaches that usually need to be tuned for each language. however, the evaluation of the quality for other languages is left for future work. we have done a comparative evaluation of the overall quality of the models, and an evaluation of each model with respect to different qualitative aspects: informativeness, fluency, consistency, and creativity. as a general conclusion, from the models evaluated bart and bert produced the best results, and satisfactory results are obtained in the proposed pipeline for the quality of the summaries. with regard to the most important aspects, the models show a good result for the categories of fluency and consistency, and an average result regarding the informativeness. these results are especially promising considering the complexity and low grammatical fluency and consistency involved in texts typical of deliberative processes. bart and bert are the only models that score above the middle score in each of the three categories, and thus we argue perform sufficiently well to be used in practice.