I MPROVING THE G ENERALIZATION OF A DVERSARIAL T RAINING WITH D OMAIN A DAPTATION
A BSTRACT
By injecting adversarial examples into training data, adversarial training is promising for improving the robustness of deep learning models.
However, most existing adversarial training approaches are based on a speciﬁc type of adversarial attack.
It may not provide sufﬁciently representative samples from the adversarial domain, leading to a weak generalization ability on adversarial examples from other attacks.
Moreover, during the adversarial training, adversarial perturbations on inputs are usually crafted by fast single-step adversaries so as to scale to large datasets.
This work is mainly focused on the adversarial training yet efﬁcient
FGSM adversary.
In this scenario, it is difﬁcult to train a model with great generalization due to the lack of representative adversarial samples, aka the samples are unable to accurately reﬂect the adversarial domain.
To alleviate this problem, we propose a novel Adversarial Training with Domain Adaptation (ATDA) method.
Our intuition is to regard the adversarial training on FGSM adversary as a domain adaption task with limited number of target domain samples.
The main idea is to learn a representation that is semantically meaningful and domain invariant on the clean domain as well as the adversarial domain.
Empirical evaluations on
Fashion-MNIST, SVHN, CIFAR-10 and CIFAR-100 demonstrate that ATDA
can greatly improve the generalization of adversarial training and the smoothness of the learned models, and outperforms state-of-the-art methods on standard benchmark datasets.
To show the transfer ability of our method, we also extend ATDA to the adversarial training on iterative attacks such as PGD-Adversial Training (PAT) and the defense performance is improved considerably.
I NTRODUCTION
Deep learning techniques have shown impressive performance on image classiﬁcation and many other computer vision tasks.
However, recent works have revealed that deep learning models are often vulnerable to adversarial examples (Szegedy et al., 2014; Goodfellow et al.;
Papernot et al., 2016), which are maliciously designed to deceive the target model by generating carefully crafted adversarial perturbations on original clean inputs.
Moreover, adversarial examples can transfer across models to mislead other models with a high probability (Papernot et al., 2017; Liu et al., 2017).
How to effectively defense against adversarial attacks is crucial for security-critical computer vision systems, such as autonomous driving.
As a promising approach, adversarial training defends from adversarial perturbations by training a target classiﬁer with adversarial examples.
Researchers have found (Goodfellow et al.;
Kurakin et al., 2016b; Madry et al., 2018) that adversarial training could increase the robustness of neural networks.
However, adversarial training often obtains adversarial examples by taking a speciﬁc attack technique (e.g., FGSM) into consideration, so the defense targeted such attack and the trained model exhibits weak generalization ability on adversarial examples from other adversaries (Kurakin et al., 2016b).
Tramr et al.
(2018) showed that the robustness of adversarial training can be easily circumvented by the attack that combines with random perturbation from other models.
Accordingly, for most existing adversarial training methods, there is a risk of overﬁtting to adversarial examples crafted on the original model with the speciﬁc attack.
In this paper, we propose a novel adversarial training method that is able to improve the generalization of adversarial training.
From the perspective of domain adaptation (DA) (Torralba & Efros, 2011), there is a big domain gap between the distribution of clean examples and the distribution of adversarial examples in the high-level representation space, even though adversarial perturbations are imperceptible to humans.
Liao et al.
(2018) showed that adversarial perturbations are progressively ampliﬁed along the layer hierarchy of neural networks, which maximizes the distance between the original and adversarial subspace representations.
In addition, adversarial training simply injects adversarial examples from a speciﬁc attack into the training set, but there is still a large sample space for adversarial examples.
Accordingly, training with the classiﬁcation loss on such a training set will probably lead to overﬁtting on the adversarial examples from the speciﬁc attack.
Even though Wong & Kolter (2018) showed that adversarial training with iterative noisy attacks has stronger robustness than the adversarial training with single-step attacks, iterative attacks have a large computational cost and there is no theoretical analysis to justify that the adversarial examples sampled in such way could be sufﬁciently representative for the adversarial domain.
Our contributions are focused on how to improve the generalization of adversarial training on the simple yet scalable attacks, such as FGSM (Goodfellow et al.).
The key idea of our approach is to formulate the learning procedure as a domain adaptation problem with limited number of target domain samples, where target domain denotes adversarial domain.
Speciﬁcally, we introduce unsupervised as well as supervised domain adaptation into adversarial training to minimize the gap and increase the similarity between the distributions of clean examples and adversarial examples.
In this way, the learned models generalize well on adversarial examples from different (cid:96) ∞ bounded attacks.
We evaluate our ATDA method on standard benchmark datasets.
Empirical results show that despite a small decay of accuracy on clean data, ATDA signiﬁcantly improves the generalization ability of adversarial training and has the transfer ability to extend to adversarial training on PGD (Madry et al., 2018).
B ACKGROUND AND R ELATED W ORK
In this section, we introduce some notations and provide a brief overview of the current advanced attack methods, as well as the defense methods based on adversarial training.
2. N OTATION
Denote the clean data domain and the adversarial data domain by D and A respectively, we consider a classiﬁer based on a neural network f ( x ) :
R d → R k .
f ( x ) outputs the probability distribution for an input x ∈ [0 , 1] d , and k denotes the number of classes in the classiﬁcation task.
Let ϕ be the mapping at the logits layer (the last neural layer before the ﬁnal softmax function), so that f ( x ) = softmax ( ϕ ( x )) .
Let (cid:15) be the magnitude of the perturbation.
Let x adv be the adversarial image computed by perturbing the original image x .
The cost function of image classiﬁcation is denoted as J ( x, y ) .
We deﬁne the logits as the logits layer representation, and deﬁne the logit space as the semantic space of the logits layer representation.
We divide attacks into two types: white-box attacks have the complete knowledge of the target model and can fully access the model; black-box attacks have limited knowledge of the target classiﬁer (e.g.,its architecture) but can not access the model weights.
2. A TTACK M ETHODS
We consider four attack methods to generate adversarial examples.
For all attacks, the components of adversarial examples are clipped in [0 , 1] .
Fast Gradient Sign Method (FGSM).
Goodfellow et al.
introduced FGSM to generate adversarial examples by applying perturbations in the direction of the gradient.
[Equation]
As compared with other attack methods, FGSM is a simple, yet fast and efﬁcient adversary.
Accordingly, FGSM is particularly amenable to adversarial training.
Projected Gradient Descent (PGD).
The Projected Gradient Descent (PGD) adversary was introduced by Madry et al.
(2018) without random start, which is a stronger iterative variant of FGSM.
This method applies FGSM iteratively for k times with a budget α instead of a single step.
[Equation]
Here clip ( · , a, b ) function forces its input to reside in the range of [ a, b ] .
PGD usually yields a higher success rate than FGSM does in the white-box setting but shows weaker capability in the black-box setting.
RAND+FGSM
[Equation]
Tramr et al.
(2018) proposed R+FGSM against adversarially trained models by applying a small random perturbation of step size α before applying FGSM.
[Equation]
Momentum Iterative Method (MIM).
MIM (Dong et al., 2018) is a modiﬁcation of the iterative
FGSM and it won the ﬁrst place of NIPS 2017 Adversarial Attacks Competition.
Its basic idea is to utilize the gradients of the previous t steps with a decay factor µ to update the gradient at step t +before applying FGSM with a budget α .
[Equation]
2. P ROGRESS ON ADVERSARIAL TRAINING
An intuitive technique to defend a deep model against adversarial examples is adversarial training, which injects adversarial examples into the training data during the training process.
First, Goodfellow et al. proposed to increase the robustness by feeding the model with both original and adversarial examples generated by FGSM and by learning with the modiﬁed objective function. ˆ
[Equation]
Kurakin et al.
(2016b) scaled the adversarial training to ImageNet (Russakovsky et al., 2015) and showed better results by replacing half the clean example at each batch with the corresponding adversarial examples.
Meanwhile, Kurakin et al.
(2016b) discovered the label leaking effect and suggested not to use the FGSM deﬁned with respect to the true label y true .
However, their approach has weak robustness to the RAND+FGSM adversary.
Tramr et al.
(2018) proposed an ensemble adversarial training to improve robustness on black-box attacks by injecting adversarial examples transferred from a number of ﬁxed pre-trained models into the training data.
For adversarial training, another approach is to train only with adversarial examples.
Nøkland (2015) proposed a specialization of the method (Goodfellow et al.)
that learned only with the objective function of adversarial examples.
Madry et al.
(2018) demonstrated successful defenses based on adversarial training with the noisy PGD, which randomly initialize an adversarial example within the allowed norm ball before running iterative attack.
However, this technique is difﬁcult to scale to large-scale neural networks (Kurakin et al., 2016a) as the iterative attack increases the training time by a factor that is roughly equal to the number of iterative steps.
Wong & Kolter (2018) developed a robust training method by linear programming that minimized the loss for the worst case within the perturbation ball around each clean data point.
However, their approach achieved high test error on clean data and it is still challenging to scale to deep or wide neural networks.
As described above, though adversarial training is promising, it is difﬁcult to select a representative adversary to train on and most existing methods are weak in generalization for various adversaries, as the region of the adversarial examples for each clean data is large and contiguous (Tram`er et al., 2017; Tabacof & Valle, 2016).
Furthermore, generating a representative set of adversarial examples for large-scale datasets is computationally expensive.
A DVERSARIAL T RAINING WITH D OMAIN A DAPTATION
In this work, instead of focusing on a better sampling strategy to obtain representative adversarial data from the adversarial domain, we are especially concerned with the problem of how to train with clean data and adversarial examples from the efﬁcient FGSM, so that the adversarially trained model is strong in generalization for different adversaries and has a low computational cost during the training.
We propose an Adversarial Training with Domain Adaptation (ATDA) method to defense adversarial attacks and expect the learned models generalize well for various adversarial examples.
Our motivation is to treat the adversarial training on FGSM as a domain adaptation task with limited number of target domain samples, where the target domain denotes adversarial domain.
We combine standard adversarial training with the domain adaptor, which minimizes the domain gap between clean examples and adversarial examples.
In this way, our adversarially trained model is effective on adversarial examples crafted by FGSM but also shows great generalization on other adversaries.
3. D OMAIN A DAPTATION ON L OGIT S PACE 3.1. U NSUPERVISED DOMAIN ADAPTATION
Suppose we are given some clean training examples { x i } ( x i ∈ R d ) with labels { y i } from the clean data domain D , and adversarial examples { x advi } ( x advi ∈ R d ) from adversarial data domain A .
The adversarial examples are obtained by sampling ( x i , y true ) from D , computing small perturbations on x i to generate adversarial perturbations, and outputting ( x advi , y true ) .
It’s known that there is a huge shift in the distributions of clean data and adversarial data in the high-level representation space.
Assume that in the logit space , data from either the clean domain or the adversarial domain follow a multivariate normal distribution, i.e., D ∼ N ( µ D , Σ D ) , A ∼
N ( µ A , Σ A ) .
Our goal is to learn the logits representation that minimizes the shift by aligning the covariance matrices and the mean vectors of the clean distribution and the adversarial distribution.
To implement the CORrelation ALignment (CORAL), we deﬁne a covariance distance between the clean data and the adversarial data as follows.
[Equation]
(6) where C ϕ ( D ) and C ϕ
( A ) are the covariance matrices of the clean data and the adversarial data in the logit space respectively, and (cid:107) · (cid:107) (cid:96) 1 denotes the L 1 norm of a matrix.
Note that L CORAL ( D , A ) is slightly different from the CORAL loss proposed by Sun & Saenko (2016).
Similarly, we use the standard distribution distance metric, Maximum Mean Discrepancy (MMD) (Borgwardt et al., 2006), to minimize the distance of the mean vectors of the clean data and the adversarial data.
[Equation] [Equation]
The loss function for Unsupervised Domain Adaptation (UDA) can be calculated as follows.
[Equation]
3.1. S UPERVISED DOMAIN ADAPTATION
Even though the unsupervised domain adaptation achieves perfect confusion alignment, there is no guarantee that samples of the same label from clean domain and adversarial domain would map nearby in the logit space.
To effectively utilize the labeled data in the adversarial domain, we introduce a supervised domain adaptation (SDA) by proposing a new loss function, denoted as margin loss, to minimize the intra-class variations and maximize the inter-class variations on samples of different domains.
The SDA loss is shown in Eq.
[Equation]
Here softplus denotes a function ln (1 + exp ( · )) ;
c y
true ∈
R k denotes the center of y true class in the logit space; C
= { c j | j = 1 , 2 , ..., k } is a set consisting of the logits center for each class, which will be updated as the logits changed.
Similar to the center loss (Wen et al., 2016), we update center c j for each class j :
[Equation]
where 1 condition = 1 if the condition is true, otherwise 1 condition = 0 ; α denotes the learning rate of the centers.
During the training process, the logits center for each class can integrate the logits representation from both the clean domain and the adversarial domain.
3. A DVERSARIAL T RAINING
For adversarial training, iterative attacks are fairly expensive to compute and single-step attacks are fast to compute.
Accordingly, we use a variant of FGSM attack (Kurakin et al., 2016b) that avoids the label leaking effect to generate a new adversarial example x advi for each clean example x i .
[Equation]
) where y target denotes the predicted class arg max { ϕ ( x i ) } of the model.
However, in this case, the sampled adversarial examples are aggressive but not sufﬁciently representative due to the fact that the sampled adversarial examples always lie at the boundary of the (cid:96) ∞ ball of radius (cid:15) (see Figure 1) and the adversarial examples within the boundary are ignored.
For adversarial training, if we train a deep neural network only on the clean data and the adversarial data from the FGSM attack, the adversarially trained model will overﬁt on these two kinds of data and exhibits weak generalization ability on the adversarial examples sampled from other attacks.
From a different perspective, such problem can be viewed as a domain adaptation problem with limited number of labeled target domain samples, as only some special data point can be sampled in the adversarial domain by FGSM adversary.
Consequently, it is natural to combine the adversarial training with domain adaptation to improve the generalization ability on adversarial data.
We generate new adversarial examples by the variant of FGSM attack shown in Eq.
(11), then we use the following loss function to meet the criteria of domain adaptation while training a strong classiﬁer.
[Equation]
x adv
[Equation]
Here λ is the hyper-parameter to balance the regularization term; m is the number of input clean examples; D indicates the input clean examples { x i } , and A the corresponding adversarial examples { x advi } ; L C denotes the classiﬁcation loss.
The training process is summarized in Algorithm 1.
[Equation] [Equation]
E XPERIMENTS
In this section, we evaluate our ATDA method on various benchmark datasets to demonstrate the robustness and contrast its performance against other competing methods under different white-box and black-box attacks with bounded (cid:96) ∞ norm.
Code for these experiments is available at https: //github.com/
JHL-HUST/ATDA .
4. E XPERIMENTAL S ETUP
Datasets.
We consider four popular datasets, namely Fashion-MNIST (Xiao et al., 2017),
SVHN (Netzer et al., 2011), CIFAR-10 and CIFAR-100 (Krizhevsky & Hinton, 2009).
For all experiments, we normalize the pixel values to [0 , 1] by dividing 255.
Baselines.
To evaluate the generalization power on adversarial examples in both the white-box and black-box settings, we report the clean test accuracy, the defense accuracy on FGSM, PGD,
R+FGSM and MIM in the non-targeted way.
The common settings for these attacks are shown in
Table 5 of the Appendix.
We compare our ATDA method with normal training as well as several state-of-the-art adversarial training methods:
• Normal Training (NT).
Training with cross-entropy loss on the clean training data.
• Standard Adversarial Training (SAT) (Goodfellow et al.).
Training with the cross-entropy on the clean training data and the adversarial examples from the FGSM variant with perturbation (cid:15) to avoid label leaking.
• Ensemble Adversarial Training (EAT) (Tramr et al., 2018).
Training with cross-entropy on the clean training data and the adversarial examples crafted from the currently trained model and the static pre-trained models by the FGSM variant with the perturbation (cid:15) to avoid label leaking.
• Provably Robust Training (PRT) (Wong & Kolter, 2018).
Training with cross-entropy loss on the worst case in the (cid:96) ∞ ball of radius (cid:15) around each clean training data point.
It could be seen as training with a complicated method of sampling in the (cid:96) ∞ ball
of radius (cid:15) .
Evaluation Setup.
For each benchmark dataset, we train a normal model and various adversarial models with perturbation (cid:15) on a main model with ConvNet architecture, and evaluate them on various attacks bounded by (cid:15) .
Moreover, for Ensemble Adversarial Training (EAT), we use two different models as the static pre-trained models.
For black-box attacks, we test trained models on the adversarial examples transferred from a model held out during the training.
All experiments are implemented on a single Titan X GPU.
For all experiments, we set the hyper-parameter λ in Eq.
(12) to 1 / 3 and the hyper-parameter α in Eq.
(10) to 0 .
For more details about neural network architectures and training hyper-parameters, see Appendix A. We tune the networks to make sure they work, not to post concentrates on optimizing these settings.
4. C OMPARISON OF D EFENSE P ERFORMANCE ON A CCURACY
We evaluate the defense performance of our ATDA method from the perspective of classiﬁcation accuracy on various datasets, and compare with the baselines
Evaluation on Fashion-MNIST.
The accuracy results on Fashion-MNIST are reported in Table 1a.
NT yields the best performance on the clean data, but generalizes poorly on adversarial examples.
SAT and EAT overﬁt on the clean data and the adversarial data from FGSM.
PRT achieves lower error against various adversaries, but higher error on the clean data.
ATDA achieves stronger robustness against different (cid:96) ∞ bounded adversaries as compared to SAT (adversarial training on FGSM).
Evaluation on SVHN.
The classiﬁcation accuracy on SVHN are summarized in Table 1b.
PRT seems to degrade the performance on the clean testing data and exhibits weak robustness on various attacks.
As compared to SAT, ATDA achieves stronger generalization ability on adversarial examples from various attacks and higher accuracy on the white-box adversaries, at the same time it only loses a negligible performance on clean data.
Evaluation on CIFAR-10.
Compared with Fashion-MNIST and SVHN, CIFAR-10 is a more difﬁcult dataset for classiﬁcation.
As PRT is challenging and expensive to scale to large neural networks due to its complexity, the results of PRT are not reported.
The accuracy results on CIFAR-10 are summarized in Table 1c.
ATDA outperforms all the competing methods on most adversaries, despite a slightly lower performance on clean data.
Evaluation on CIFAR-100.
The CIFAR-100 dataset contains 100 image classes, with 600 images per class.
Our goal here is not to achieve state-of-the-art performance on CIFAR-100, but to compare the generalization ability of different training methods on a comparatively large dataset.
The results on CIFAR-100 are summarized in Table 1d.
Compared to SAT, ATDA achieves better generalization on various adversarial examples and it does not degrade the performance on clean data.
In conclusion, the accuracy results provide empirical evidence that ATDA has great generalization ability on different adversaries as compared to SAT and outperforms other competing methods.
4. F URTHER A NALYSIS ON THE D EFENSE P ERFORMANCE
To further investigate the defence performance of the proposed method, we compute two other metrics: the local loss sensitivity to perturbations and the shift of adversarial data distribution with respect to the clean data distribution.
Local Loss Sensitivity.
One method to quantify smoothness and generalization to perturbations for models is the local loss sensitivity (Arpit et al., 2017).
It is calculated in the clean testing data as follows.
The lower the value is, the smoother the loss function is.
[Equation]
The results of the local loss sensitivity for the aforementioned learned models are summarized in
Table 2.
The results suggest that adversarial training methods do increase the smoothness of the model as compared with the normal training and ATDA performs the best.
Distribution Discrepancy.
To quantify the dissimilarity of the distributions between the clean data and the adversarial data, we compare our learned logits embeddings with the logits embeddings of the competing methods on Fashion-MNIST.
We use t-SNE (Maaten & Hinton, 2008) for the comparison on the training data, testing data and adversarial testing data from the white-box FGSM or PGD.
The comparisons are illustrated in Figure 2 and we report the detailed MMD distances across domains in Table 3.
Compared with NT, SAT and EAT actually increase the MMD distance across domains of the clean data and the adversarial data.
In contrast, PRT and ATDA can learn domain invariance between the clean domain and the adversarial domain.
Furthermore, our learned logits representation achieves the best performance on domain invariance.
[Table] [Table]
[Table] [Table]
[Table] [Table]
4. A BLATION S TUDIES ON ATDA
To individually dissect the effectiveness of different components in ATDA (Standard Adversarial
Training (SAT), Unsupervised Domain Adaptation (UDA)
[Table]
and Supervised Domain Adaptation (SDA))
[Table]
we conduct a series of ablation experiments in Figure
[Table]
For each model
[Table]
we report the
[Table]
[Figure]
average accuracy rates over all white-box attacks and all black-box attacks, respectively.
The results illustrate that, by aligning the covariance matrix and mean vector of the clean and adversarial examples, UDA plays a key role in improving the generalization of SAT on various attacks.
In general, the aware of margin loss on SDA can also improve the defense quality on standard adversarial training, but the effectiveness is not very stable over all datasets.
By combining UDA and SDA together with SAT, our ﬁnal algorithm ATDA can exhibits stable improvements on the standard adversarial training.
In general, the performance of ATDA is slightly better than SAT+UDA.
[Figure]
E XTENSION TO PGD-A DVERSARIAL T RAINING
ATDA can simply be extended to adversarial training on other adversaries.
We now consider to extend the ATDA method to PGD-Adversarial Training (PAT) (Madry et al., 2018): adversarial training on the noisy PGD with perturbation (cid:15) .
By combining adversarial training on the noisy PGD with domain adaptation, we implement an extension of ATDA for PAT, called PATDA.
For the noisy
PGD, we set the iterated step k as 10 and the budget α as (cid:15)/ 4 according to Madry et al.
As shown in Table 4, we evaluate the defense performance of PAT and PATDA on various datasets.
On Fashion-MNIST, we observe that PATDA fails to increase robustness to most adversaries as compared to PAT.
On SVHN, PAT and PATDA fail to converge properly.
The results are not surprising, as training with the hard and sufﬁcient adversarial examples (from the noisy PGD) requires the neural networks with more parameters.
On CIFAR-10 and CIFAR-100, PATDA achieves stronger robustness to various attacks than PAT.
In general, PATDA exhibits stronger robustness to various adversaries as compared to PAT.
The results indicate that domain adaptation can be applied ﬂexibly to adversarial training on other adversaries to improve the defense performance.
[Table]
C ONCLUSION
In this study
[Table]
we regard the adversarial training as a domain adaptation task with limited number of target labeled data.
By combining adversarial training on FGSM adversary with unsupervised and supervised domain adaptation, the generalization ability on adversarial examples from various attacks and the smoothness on the learned models can be highly improved for robust defense.
In addition, ATDA can easily be extended to adversarial training on iterative attacks (e.g., PGD) to improve the defense performance.
The experimental results on several benchmark datasets suggest that the proposed ATDA and its extension PATDA achieve signiﬁcantly better generalization results as compared with current competing adversarial training methods.
A CKNOWLEDGMENTS
This work is supported by National Natural Science Foundation (61772219).
A E XPERIMENTAL DETAILS
In the appendix
[Table]
we show all details of the common settings, neural network architectures and training hyper-parameters for the experiments.
A. H YPER - PARAMETERS FOR ADVERSARIES .
For each dataset
[Table]
the details about the hyper-parameters of various adversaries are shown in Table 5, where (cid:15) denotes the magnitude of adversarial perturbations
[Table]
[Table]
A. NEURAL NETWORK ARCHITECTURES AND TRAINING HYPER - PARAMETERS
Fashion-MNIST.
In the training phase, we use Adam optimizer with a learning rate of 0.001 and set the batch size to 64.
For Fashion-MNIST, the neural network architectures for the main model, the static pre-trained models and the model held out during training are depicted in Table 6.
For all adversarial training methods, the magnitude of perturbations is 0 .
1 in (cid:96) ∞ norm.
SVHN.
In the training phase, we use Adam optimizer with a learning rate of 0.001 and set the batch size to 32 and use the same architectures as in Fashion-MNIST.
For all adversarial training methods, the magnitude of perturbations is 0 .
02 in (cid:96) ∞ norm.
[Table]
CIFAR-10.
In the training phase, we use the same training settings as in SVHN.
we use Adam optimizer with a learning rate of 0 .
001 and set the batch size to 32.
In order to enhance the expressive power of deep neural networks, we use Exponential Linear Unit (ELU) (Clevert et al., 2015) as the activation function and introduce Group Normalization (Wu & He, 2018) into the architectures.
The neural network architectures for CIFAR-10 are shown in Table 7.
For all adversarial training methods, the magnitude of perturbations is 4 / 255 in (cid:96) ∞ norm.
CIFAR-100.
We use the same training settings as in CIFAR-10.
For CIFAR-100, the neural network architectures for the main model, the static pre-trained models and the model held out during training are shown in Table 8.
For all adversarial training methods, the magnitude of perturbations is 4 / 255 in (cid:96) ∞ norm.
[Table] [Table]