Adversarial Regularization as Stackelberg Game: An Unrolled Optimization Approach (2021)

Abstract: Adversarial regularization has been shown to improve the generalization performance of deep learning models in various natural language processing tasks. Existing works usually formulate the method as a zero-sum game, which is solved by alternating gradient descent/ascent algorithms. Such a formulation treats the adversarial and the defending players equally, which is undesirable because only the defending player contributes to the generalization performance. To address this issue, we propose Stackelberg Adversarial Regularization (SALT), which formulates adversarial regularization as a Stackelberg game. This formulation induces a competition between a leader and a follower, where the follower generates perturbations, and the leader trains the model subject to the perturbations. Different from conventional approaches, in SALT, the leader is in an advantageous position. When the leader moves, it recognizes the strategy of the follower and takes the anticipated follower’s outcomes into consideration. Such a leader’s advantage enables us to improve the model fitting to the unperturbed data. The leader’s strategic information is captured by the Stackelberg gradient, which is obtained using an unrolling algorithm. Our experimental results on a set of machine translation and natural language understanding tasks show that SALT outperforms existing adversarial regularization baselines across all tasks. Our code is publicly available.

## introduction adversarial regularization @xcite has been shown to improve the generalization performance of deep learning models in various natural language processing (nlp) tasks, such as language modeling @xcite , machine translation @xcite , natural language understanding @xcite , and reading comprehen-sion @xcite . however, even though significant progress has been made, the power of adversarial regularization is not fully harnessed. conventional adversarial regularization is formulated as a zero-sum game (a min-max optimization problem), where two players seek to minimize/maximize their utility functions. in this formulation, an adversarial player composes perturbations, and a defending player solves for the model parameters subject to the perturbed inputs. existing algorithms find the equilibrium of this zerosum game using alternating gradient descent/ascent @xcite . for example, in a classification problem, the adversarial player first generates the input perturbations by running projected gradient ascent to maximize a loss function, and then the defending player updates the model using gradient descent, trying to decrease the classification error. notice that in this case, neither of the players know the strategy of its competitor, i.e., the model does not know how the perturbations are generated, and vice versa. in other words, the two players are of the same priority, and either one of them can be advantageous in the game. it is possible that the adversarial player generates over-strong perturbations that hinder generalization of the model. to resolve this issue, we grant the defending player (i.e., the model) a higher priority than the adversarial player by letting the defender recognize its competitor's strategy, such that it is advantageous in the game. consequently, we propose stackelberg adversarial regularization (salt), where we formulate adversarial regularization as a stackelberg game @xcite . the concept arises from economics, where two firms are competing in a market, and one of the them is in the leading position by acknowledging the opponent's strategy. in stackelberg adversarial regularization, a leader solves for the model parameters, and a follower generates input perturbations. the leader procures its advantage by considering what the best response of the follower is, i.e., how will the follower respond after observing the leader's decision. then, the leader minimizes its loss, anticipating the predicted response of the follower. the salt framework identifies the interaction between the leader and the follower by treating the follower's strategy (i.e., the input perturbations) as an operator of the leader's decision (i.e., the model parameters). then we can solve for the model parameters using gradient descent. one caveat is that computing the gradient term, which we call the stackelberg gradient, requires differentiating the interaction operator. to rigorously define this operator, recall that the follower can be approximately solved using gradient ascent. we can treat the perturbations in each iteration as an operator of the model parameters, and the interaction operator is then the composition of such update-induced operators. correspondingly, the stackelberg gradient is obtained by differentiating through these updates. this procedure is referred to as unrolling @xcite , and the only computational overhead caused by it is computing hessian vector products. as a result, when applying the finite difference method, computing the stackelberg gradient requires two backpropagation and an extra o(d) complexity operation, where d is the embedding dimension. therefore, the unrolling algorithm computes the stackelberg gradient without causing much computational overhead. we conduct experiments on neural machine translation (nmt) and natural language understanding (nlu) tasks. for the nmt tasks, we experiment on four low-resource and one richresource datasets. salt improves upon existing adversarial regularization algorithms by notable margins, especially on low-resource datasets, where it achieves up to 2 bleu score improvements. to test performance on nlu tasks, we evaluate salt on the glue @xcite benchmark. salt outperforms state-of-the-art models, such as bert @xcite , freeat @xcite , freelb @xcite , and smart @xcite . we build salt on the bert-base architecture, and we achieve an average score of 84.5 on the glue development set, which is at least 0.7 higher than existing methods. moreover, even though we adapt salt to bert-base, the performance is noticeably higher than the vanilla bert-large model (84.5 vs. 84.0). the unrolling procedure was first proposed for auto-differentiation @xcite , and later applied in various context, such as hyper-parameter optimization @xcite , meta-learning @xcite , and generative adversarial networks @xcite . to the best of our knowledge, we are the first to apply the unrolling technique to adversarial regularization to improve generalization performance. we summarize our contributions as the following: (1) we propose salt, which employs a stackelberg game formulation of adversarial regularization. (2) we use an unrolling algorithm to find the equilibrium of the stackelberg game. (3) extensive experiments on nmt and nlu tasks verify the efficacy of our method. notation. we use df (x)/dx to denote the gradient of f with respect to x. we use ∂f (x, y)/∂x to denote the partial derivative of f with respect to x. for a d-dimensional vector v, its ℓ 2 norm is defined as adversarial training was originally proposed for training adversarial robust classifiers in image classification @xcite @xcite . the idea is to synthesize strong adversarial samples, and the classifier is trained to be robust to them. theoretical understanding @xcite about adversarial training and various algorithms to generate the adversarial samples, such as learning-to-learn @xcite , are proposed. besides computer vision, adversarial training can also benefit reinforce-ment learning @xcite . different from the above fields, in nlp, the goal of adversarial training is to build models that generalize well on the unperturbed test data. note that robustness and generalization are different concepts. recent works @xcite showed that adversarial training can hurt generalization performance, i.e., accuracy on clean data. as such, adversarial training needs to be treated with great caution. therefore, in nlp, this technique requires refined tuning of, for example, the training algorithm and the perturbation strength. fine-tuning pre-trained language models @xcite @xcite @xcite is state-ofthe-art for natural language understanding tasks such as the glue @xcite benchmark. recently, there are works that use adversarial pre-training @xcite and adversarialregularized fine-tuning methods such as smart @xcite , freelb @xcite , and freeat @xcite to improve model generalization and robustness @xcite . ## method natural language inputs are discrete symbols (e.g., words), instead of continuous ones. therefore, a common approach to generate perturbations is to learn continuous embeddings of the inputs and operate on the embedding space @xcite @xcite @xcite . let f (x, θ) be our model, where x is the input embedding, and θ is the model parameter. further let y be the ground-truth output corresponding to x. for example, in nmt, f is a sequence-to-sequence model, x is the embedding of the source sentence, and y is the target sentence. in classification tasks, f is a classifier, x is the input sentence/document embedding, and y is the label. in both of these cases, the model is trained by minimizing the empirical risk over the training data, i.e., here {(x i , y i )} n @xmath0, and ℓ is a taskspecific loss function, e.g., cross-entropy loss. ## experiments in all the experiments, we use pytorch 2 @xcite as the backend. all the experiments are conducted on nvidia v100 32gb gpus. we use the higher package 3 @xcite to implement the proposed algorithm. ## conclusion we propose salt, an adversarial regularization method that employs a stackelberg game formulation. such a formulation induces a competition between a leader (the model) and a follower (the adversary). in salt, the leader is in an advantageous position by recognizing the follower's strategy, and this strategic information is captured by the stackelberg gradient. we compute the stackelberg gradient, and hence find the equilibrium of the stackelberg game, using an unrolled optimization approach. empirical results nmt and nlu tasks suggest the superiority of salt to existing adversarial regularization methods.