Constrained Regeneration for Cross-Lingual Query-Focused Extractive Summarization (2022)

Abstract: Query-focused summaries of foreign-language, retrieved documents can help a user understand whether a document is actually relevant to the query term. A standard approach to this problem is to first translate the source documents and then perform extractive summarization to find relevant snippets. However, in a cross-lingual setting, the query term does not necessarily appear in the translations of relevant documents. In this work, we show that constrained machine translation and constrained post-editing can improve human relevance judgments by including a query term in a summary when its translation appears in the source document. We also present several strategies for selecting only certain documents for regeneration which yield further improvements

## introduction query-focused summarization creates an overview of a document which reflects how that document may be relevant to a provided query; such a task is useful for any search engine, such as for news articles or academic papers, where a user may want to search documents by a given query. in this paper, we further narrow the use case to one in which the user seeks a document containing a single specific query term (which may be a multi-word expression). for example, if the query is "dossier", the user is interested in finding information about a specific type of collection of files, as might exist in an intelligence investigation. a summary in the user's language can help them decide if a foreign language document is relevant. our work focuses on query-focused extractive summarization in a cross-lingual setting, where the summaries are generated in a language (here, english) different from the source language of the documents (here, farsi, kazakh, and georgian). because large summarization corpora do not exist in these languages, we follow a translate-thensummarize approach @xcite in which we first apply machine translation (mt) to translate documents into english, a language with abundant summarization corpora, and then summarize the translated document; however, this introduces additional concerns. translating a document once, before a query term is known, can lead to wording choices that are sub-optimal for any particular query term (e.g., if the kazakh for "dossier" were translated as "file", and it may be unclear whether the specific meaning of dossier occurred in the source as opposed to other meanings of "file"). to address this, we present a constrained regeneration framework where we translate a document, summarize it with an extractive summarizer that uses evidence from the source language, and select a sentence to be regenerated under the constraint to include the requested query term if appropriate. our work is implemented within a pipeline that includes cross-lingual information retrieval (clir) followed by summarization of retrieved documents; in the latter step, a summary is generated for a document given a specific query term. based on the intuition that seeing the query term in the summary is a strong signal of relevance to end users, we first present work on three types of constrained regeneration systems: marian-c, a constrained version of marian @xcite ; editor @xcite ; and constrained automatic post-editing @xcite . in initial experimentation, however, we found that these systems often insert the requested query term even in cases when the foreign document did not contain a suitable translation. to address this, we further in-troduce document selection methods to determine when to apply regeneration and thus avoid inserting query terms inappropriately. we perform a human evaluation and show that the combined use of regeneration and document selection improve humans' ability to accurately distinguish relevant and irrelevant non-english documents by their generated english summaries. our approach combines complementary strengths of the three primary modules needed for cross-lingual query-focused summarization: clir excels at discovering cross-lingual mappings at the lexical level, neural mt produces complete sentences that are often very fluent, but sometimes at the expense of adequacy and term preservation, and summarization helps users assess relevance efficiently. the novelty of our approach lies in a tight integration of these components, exploiting clir to detect relevance, and combining summarization and selective regeneration of summary sentences to produce a human-useful summary. our contributions are as follows: ## our models our approach to improve relevance judgments of summaries is based on (1) constrained regeneration models that encourage the inclusion of query terms in document translations, and (2) document and sentence selection models that identify documents where the inclusion of query terms is appropriate. ## experiments for each of our three languages, we draw a random sample of query-document pairs with a high likelihood of being relevant according to our trained clir system for that language. then, for each query-document pair, we generate an extractive summary with no regeneration applied; this is our baseline system. we then apply each applicable constrained regeneration system to each summary independently, generating a new copy of the summary for each regeneration system. we then submit each of these summaries to amazon mechanical turk for human evaluation, the formulation of which is described in detail below in section 5.1. the result of the human evaluation is a relevance score for each summary-that is, for each (q, ⟨d, f ⟩, regeneration system) triple. we can evaluate different regeneration systems at this stage by simply collecting the labels they are assigned and comparing them to the ground truth relevance labels. finally, we apply our document selection methods; we select a subset of documents whose summaries should be regenerated according to our document selection threshold, and we use the collected scores from the regenerated and baseline variants to evaluate the +selection and +omission variants of the regeneration systems. our farsi experiments are done on a random sample of 1000 query-document pairs from the documents returned by clir 7 for the material eval partition, equally split between ground-truth irrelevant and relevant documents. these querydocument pairs are selected such that the summaries the baseline system produced did not contain the query word, indicating an opportunity for regeneration systems to incorporate query terms. we repeat the experiments for kazakh and georgian similarly, using samples of 2000 documents for each language from their dev partitions. ## results the results of our experiments are shown in for the low-resource languages, kazakh and georgian, applying regeneration via cape or marian-c shows consistent and significant improvement over the baseline, with marian-c performing best. editor particularly improves recall over the baseline, but overall the improvements are not statistically significant. we see similar trends as in farsi, where applying any form of regeneration increases recall and yields similar precision when not using document selection, leading to increased f1. when we include document selection as a pipeline step before applying regeneration, however, precision also increases for all systems while retaining an improvement in recall (though not as large); as the vast majority of documents are irrelevant to any given query term, selection results in an overall net increase in f1 for cape and marian-c. editor, which is less aggressive in including its constraints, interacts poorly with document selection in kazakh and yields reduced f1 under this setting. finally, the +omission variant of document selection actually performs best overall because of how much it improves precision, although it does not increase recall as much as the +selection variant. thus we see three variants of our systems (no selection, +selection, +omission) occupying different points on the precision-recall tradeoff in the low-resource setting. we therefore see that for our low-resource languages, adding document selection to our regeneration improves the overall performance because it increases precision; the regeneration systems in these languages tend to take irrelevant documents and make their summaries appear relevant. however, in the case of our high-resource language, the improvements to precision afforded by document selection are minimal and do not balance out its diminished recall. we note that from cross-lingual summarization. prior work on cross-lingual summarization has mostly focused on two paradigms -summarize-then-translate @xcite @xcite and translate-then-summarize @xcite . the summarize-then-translate approach, however, requires a large amount of summarization training data in the source language @xcite , which makes them unsuitable for our setting since the source languages in our setting are low-resource. prior work has shown that translatethen-summarize approaches are prone to error propagation @xcite , and propose methods to produce more fluent summaries. in our setting, having a fluent translation is not sufficient -we also need to have a translation with wording that is appropriate for the given input query. therefore, in our work we focus on an integration of summarization with regeneration to more clearly indicate relevance. ## conclusion we have presented a novel method of cross-lingual query-focused extractive summarization in which we apply regeneration to a generated summary in order to force inclusion of the query term when it appears in the source language document. we demonstrated large, significant improvements over the baseline in all cases through the addition of regeneration, showing increased recall and precision over the baseline. for our noisy low-resource languages, the combination of an aggressive constrained mt system and a document selection filter additionally allows the benefits of including the query term in a relevant summary while avoiding creating new false positives. we experimented with three methods of constrained regeneration, which attempt to re-translate or edit a given sentence to include a given constraint: constrained automatic post-editing (cape), nonautoregressive mt (edi-tor), and our own implementation of autoregressive mt (marian-c). for low resource languages, autoregressive mt consistently performed better, while for farsi, cape was best. we believe this work opens the door to interesting future work experimenting with more complex varieties of document selection; with different, customized kinds of constrained regeneration; and with what types of languages benefit from these and other techniques.