Attacks against Abstractive Text Summarization Models through Lead Bias and Influence Functions (2024)

Abstract: Large Language Models (LLMs) have introduced novel opportunities for text comprehension and generation. Yet, they are vulnerable to adversarial perturbations and data poisoning attacks, particularly in tasks like text classification and translation. However, the adversarial robustness of abstractive text summarization models remains less explored. In this work, we unveil a novel approach by exploiting the inherent lead bias in summarization models, to perform adversarial perturbations. Furthermore, we introduce an innovative application of influence functions, to execute data poisoning, which compromises the model’s integrity. This approach not only shows a skew in the models’ behavior to produce desired outcomes but also shows a new behavioral change, where models under attack tend to generate extractive summaries rather than abstractive summaries.

## introduction in recent years, with the advent of large language models (llms), such as bert @xcite , bart @xcite , t5 @xcite , and gpt @xcite , the field of natural language processing (nlp) has witnessed a monumental transformation. these models have revolutionized the way how machines understand and generate human language, offering capabilities in a wide range of applications from text classification, machine translation, and question-answering to text summarization. in particular, text summarization benefits from llms to consume vast amounts of information and provide concise and coherent summaries. however, llm's susceptibility towards adversarial tactics and poisoning attacks presents a critical vulnerability. attacks mainly involve making subtle modifications to the model's input to produce incorrect or misleading outputs @xcite . to date, studies have shed light on how adversarial inputs can impact models performing the task of text classification and translation @xcite . recent works have started to study the impact of adversarial perturbations on text summarization. for instance, they have shown that minor adversarial perturbations like synonym substitution @xcite or utilizing homoglyphs @xcite can lower the quality of generated summaries. however, existing attack strategies developed for classification or translation tasks are not directly applicable to summarization due to differences in their goals and evaluation metrics. for example, while a change in predicted labels might measure success in classification, summarization requires consideration of more nuanced aspects, such as the quality, coherence, and context of the generated summaries. in addition, to the best of our knowledge, no work has systematically explored adversarial vulnerabilities specific to summarization tasks, especially in leveraging the llms and algorithmic properties. we employ a systematic, large-scale, layered approach that spans different levels ranging from characters, words, sentences, and documents. this comprehensive strategy allows us to explore a wider range of vulnerabilities specific to summarization models. building on these methodological differences, we investigate exploiting lead bias @xcite within llms used for text summarization, which is the tendency of models to overly rely on the initial sentences of a document while generating summaries. we demonstrate how this bias poses a critical vulnerability in how text summarization models process and prioritize content. by embedding various types of adversarial perturbations to these leading sentences, we uncover a significant discrepancy in the model's ability to present essential information accurately. furthermore, poisoning attacks, where the training data is manipulated to degrade the model's performance, have been explored for the tasks of text classification and translation @xcite . however, they are unexplored in the case of text summarization. our work parallels dirty label attacks, a subset of poisoning attacks in which labels are intentionally altered to deceive models. we apply similar principles and implement new types of attacks specific to text summarization, where summaries change to contrastive or include toxic content without changing the training document's actual context or keywords. central to our methodology is the innovative application of influence functions to strategically introduce poisoned data into the training dataset. traditionally, these influence functions are used to assess the impact of a single data point on the overall model's predictions @xcite . leveraging these functions, we identify influential data points in the training dataset whose alteration can result in a modification in the behavior of these models. moreover, we unveil a novel observation: the poisoned models tend to generate extractive summaries instead of abstractive summaries. this behavioral shift signifies not just a vulnerability to data poisoning attacks but also a fundamental alteration in how models process and summarize textual information under adversarial influence. this study examines multi-document text summarization (mdts), which better simulates the information-gathering process in genai systems. these systems typically summarize information from multiple sources to answer user queries on specific topics. it also provides a more practical threat model, where the adversary modifies a few documents from various sources, potentially affecting the summarization outcome. by systematically exposing these vulnerabilities in mdts models, our work aims to motivate and inform future research into developing more secure and robust text summarization models that can maintain their integrity and performance in the face of potential adversarial manipulation. the primary contributions of the work are as follows: comprehensive evaluation of adversarial perturbations: we analyze the response of text summarization models like bart, t5, and pegasus, and the latest chatbots, chatgpt-3.5, claude-sonet, and gemini to adversarial perturbations, ranging from character-level changes to broader manipulations at the word, sentence, and document level. lead bias exploitation analysis: we present the first study to exploit the lead bias in text summarization models for adversar-ial purposes, demonstrating a key vulnerability in model integrity. poisoning attack strategies during model fine-tuning: using influence functions, we identify influential data points to poison training datasets, revealing a skew in the model's behavior and a shift in the model's tendency to generate extractive summaries instead of abstractive summaries when poisoned. our codes are available here: https://github.com/rog11/summary-attack . data poisoning attacks. data poisoning attacks are aimed at integrity of ml models, where attacker intentionally adds examples to training set to manipulate the behavior of the model at test time @xcite . these attacks in literature mainly include label-flipping or dirty label attacks @xcite , where adversaries can manipulate the labels of training data points, to degrade the model's performance. other types of these attacks include backdoor attacks @xcite , which causes models to deviate from expected behavior when a trigger is encountered. ## threat model adversarial perturbations: adversaries can be motivated to perturb text summarization inputs during inference time so that they generate biased or misleading summaries. in this work, we assume the attacker's goal is to successfully implement sentence exclusion attack to fool the summarization model not to use a specific sentence, here the lead sentence. as a consequence of this attack, the model's output may suffer from degradation in quality, i.e., generating incomplete, incoherent, or misleading summaries. for example, recently, summarization has been increasingly proposed to improve fact-checking processes @xcite @xcite . moreover, some practical implementations of fact-checkers by @xcite and @xcite utilize summarization to efficiently process and present fact-checked information to the public. our threat model considers an adversary who strategically implants fabricated news across multiple foreign outlets using an adversarial perturbation attack, ensuring they do not surface in the fact-checker platform's summaries. consequently, misinformation evades debunking and persists in its spread.we also assume a black box setting in which attackers do not have access to model parameters or training data. ## adversarial perturbations with their success on text classification, we examine the robustness of summarization models against adversarial perturbations, which can be in different levels -character, word, sentence, and document. the space of possible modifications at every level is huge @xcite . we show how an attacker, leveraging the biases in summarization models, can implement sentence exclusion attack, which can also result in quality degradation. in mdts, models exhibit a phenomenon known as lead bias, where they disproportionately focus on the initial sentences of a document @xcite . this bias arises due to training patterns where crucial information is typically located at the beginning of multiple documents. additionally, document ordering bias can play a role where models giving more weight to the content of documents presented earlier in the sequence @xcite . we hypothesize that these biases make text summarization models vulnerable to adversarial perturbations. as shown in figure 1 , we implemented eleven attacks, including four attacks using character-level perturbations, three attacks using word-level and sentence-level perturbations, and one attack at the document level. model fine-tuning and bias confirmation: we verify the existence of lead bias in llm-based text summarization models using publicly available pretrained models and multi-document datasets. the models' susceptibility to lead and document ordering biases gives attackers a cue on where to modify the input documents to manipulate the summary. this can reduce the search space and efficiently influence the overall summary. next, we formalize the adversarial perturbations and describe the process of identifying influential tokens. adversarial perturbations formalization: for a set of documents {d 1 , d 2 , ..., d k }, where each d i consists of sentences {s i1 , s i2 , ..., s in }, we specifically target the lead sentences of the first document, d @xmath0, s 12 , ..., s 1m }, with m being a small number, such as 2 or 3. this targeted approach stems from the hypothesis that alterations in the lead sentences of the first document can disproportionately influence the overall summary. identification of important tokens: in character and word level, we employ tf-idf to determine the important words within d lead . instead of applying adversarial perturbations to all the important words in the set, we match the words present in sentences of summary and filter them to apply perturbations. this set of selected words is denoted as w imp . our adversarial strategy involves applying a perturbation function p to w imp . this function p(w) is designed to apply perturbations across characters and words in the set of w imp , encompassing insertions, deletions, or homoglyph, synonym replacements while adhering to the constraint of minimal perturbation. at the sentence level, p(w) is designed to apply perturbations across d lead , encompassing replacement with paraphrases and homoglyphs and re-ordering. at the document level, p(w) is designed to apply perturbations across d 1 by changing the document's location from top to bottom. the application of p(w) to d lead results in a perturbed version, d ′ lead . character swapping, deletion and insertion: these perturbations can simulate common typo errors and input noise that can occur in real-world scenarios. we assess models' ability to correct or accommodate such variations in summarization. replacement with homoglyphs: homoglyphs are visually similar characters/ words that are less noticeable to human readers and can be used for deceptive purposes. we assess models' adversarial robustness when one character or word at a time is replaced with its homoglyph counterpart. word deletion: important words or entities may be missing due to user input errors, censorship, or data corruption. we evaluate the models' ability to handle such missing information. word replacement with synonyms: words can be expressed in multiple ways using synonyms. motivated by the success of synonym replacement in attacking text classification tasks, we test the models' ability to understand contextually equivalent expressions during summarization when one word at a time is replaced with its synonym. sentence/document reordering: the order of sentences and paragraphs helps understand their context. we evaluate the models robustness against such changes in structure by moving one of the sentences in a document from the top to the bottom and placing the top document at the bottom. sentence paraphrasing: models should be able to handle paraphrased expressions while capturing the core meaning. we test the models' ability to summarize effectively while replacing the original sentence with its paraphrased version. ## influence functions for data poisoning the methodology we implemented for data poisoning is similar to dirty label attacks, which have proved to be successful in the case of text classification @xcite . however, these approaches are not directly applicable to text summarization. specifically, text classification tasks involve labels that can be manipulated for a dirty label attack, where incorrect labels are intentionally introduced to degrade model performance. in contrast, text summarization does not rely on such labels, and it involves generating coherent summaries, where a different approach is required for data poisoning. we propose a novel attack strategy tailored to text summarization models, where attackers can employ influence functions to systematically target and modify training data. influence functions allow us to quantify the impact of a single data point on the model's predictions @xcite . by leveraging this information, attackers can identify the most influential training samples and strategically perturb them to manipulate the model's behavior. our proposed approach differs from dirty label attacks in two key aspects. firstly, instead of modifying labels, we focus on perturbing the content of summaries in training instances to either a contrastive or a toxic version. second, we utilize the influence functions to guide the selection of instances to be modified, making sure that the perturbations have a significant impact on the model's predictions. the framework to execute this attack is outlined in figure 2 , with the following components: (1) initial setup: initially, an attacker has access to a benign training dataset, a testing dataset, and a publicly available pre-trained llm. the pre-trained llm can be fine-tuned using this benign dataset and run on the test set to observe its original summarization behavior. (2) utilization of influence functions: to poison a small sample of the training dataset, we utilize the concept of influence functions, which quantify the impact of training data points on the model's predictions @xcite . these functions approximate the effect on the model's predictions or parameters when a data point is either altered or removed entirely @xcite . specifically, the influence function is calculated by taking the dot product of the inverse hessian and the gradient of the loss with respect to the model's parameters, evaluated at the data point of interest @xcite . however, computing the inverse of the hessian matrix could be computationally expensive. we leverage the influence functions, inspired by datainf @xcite with better memory complexity, to determine influential data points for summarization models. (3) generation of poisoned data: for each identified influential sample, we apply the dirty label attack to alter the summaries by creating either a contrastive version or a toxic version. examples of these altered summaries are provided in ## experimental setup this section outlines the methodologies employed to evaluate the robustness of various models against adversarial perturbations and data poisoning. for evaluation, we chose the datasets including multi-news (fabbri et al., 2019) and multi-xscience @xcite , and three state-of-the-art models, including bart @xcite , pega-sus @xcite and t5 @xcite . in addition to baseline models, we evaluate the effectiveness of adversarial perturbations against state-of-the-art chatbots, including gpt-3.5 (openai, 2022), claude-sonet (anthropic, 2024), and gemini @xcite . for details on each dataset, model specifications, and chatbot configurations, please refer to appendix 11.1. evaluation metrics for perturbations: for evaluation, we use the text summarization model to generate summaries from both the original lead part (d lead ) and the perturbed lead part (d ′ lead ). we then compute a metric that checks if the perturbed sentences from d ′ lead are present in the generated summary s. the metric returns a value of 1 if the perturbed sentences are not present in the summary, indicating that the perturbation successfully misled the model; otherwise, it returns 0. the percentage exclusion is calculated as the percentage of document sets where the perturbations successfully led to the exclusion of the perturbed sentences (d robustness quotient: these metrics calculate the change in standard summary quality metrics, such as rouge-1,2, and l @xcite before and after perturbation. rouge measures the overlap of n-grams between the generated summary and the original summary. a small change would indicate that the model can maintain the quality and accuracy of the generated summaries despite the adversarial perturbations. evaluation metrics for data poisoning: as the attacker's main target is to skew the model's behavior, as per the poisoned dataset, we provide the following metrics. sentiment inversion rate: using this metric, we measure the rate at which the sentiment of sentences in the summary is inverted from the source text due to poisoning. a sentiment inversion, identified by the negation or reversal of sentiment from positive to negative or vice versa, is an indication of a successful poisoning attack. to assess the sentiment inversion, initially, we tokenize the sentences in generated summaries and try to match the sentences with their respective sentences in the documents. later, we utilize a robertabased sentiment classifier obtained from huggingface @xcite to classify the sentiment of these sentences into positive, negative and neutral. toxic content detection: this metric assesses the influence of toxic content introduced into training data on the summaries produced by the models. we utilize google's perspective api (api, 2021) to detect toxic elements within these summaries. it is an ai-based tool designed to evaluate text and identify language that may be considered abusive or inappropriate, assigning scores across several attributes: severe toxicity, profanity, sexually explicit, threats, and insults, with each attribute receiving a score from 0 to 1. for our study, we particularly focus on the severe toxicity attribute because it captures the most extreme and harmful forms of toxic language, which can significantly distort the quality and integrity of model-generated summaries. this level of toxicity can also have damaging social implications, making it essential to identify and mitigate in any summarization task. abstractive to extractive: to evaluate the impact of data poisoning on the shift from abstractive to extractive summarization, we calculate the cosine similarity between sentences in the adversarial summary and the original document. for each sentence in the summary, the highest similarity with any sentence from the document is determined. a higher average of these similarity scores across summary sentences suggests a shift from abstractive to extractive summarization. this can be problematic because abstractive summarizers aim to generate concise, coherent, and fluent summaries by paraphrasing the input text. they can capture key ideas and present them in a clear and logical manner. however, extractive summarizers select and attach sentences from the original text without considering the overall flow, resulting in less coherent and disjointed summaries. this shift highlights the importance of monitoring changes in summarization behavior due to data poisoning. ## conclusion this paper presents a comprehensive evaluation of adversarial perturbations affecting text summarization models, such as bart, t5, and pegasus, and the latest chatbots, such as chatgpt-3.5, claude-sonet, and gemini, uncovering significant vulnerabilities. a novel aspect of our work is the exploitation of lead bias, demonstrating that attackers can manipulate outputs by targeting initial text segments. remarkably, introducing adversarial perturbations disrupts the model's usual prioritization of lead sentences, an unintended consequence that serves as compelling evidence of the attack's effectiveness in manipulating model behavior. furthermore, we pioneer the use of influence functions for poisoning attacks, successfully skewing model behavior to produce desired outputs and inducing a shift from abstractive to extractive summaries. by exposing the vulnerabilities of these models, we argue that there is a critical need for more resilient systems for text summarization.