On the Strong Correlation Between Model Generalization and Invariance
Anonymous Author(s)
AffiliationAddress email
Abstract
Generalization and invariance are two essential properties of any machine learningmodel.
Generalization captures a model’s ability to classify unseen data whileinvariance measures consistency of model predictions on transformations of thedata.
Existing research suggests a positive relationship: a model generalizing wellshould be invariant to certain visual factors.
Building on this qualitative implication we make two contributions.
First, we introduce effective invariance (EI), asimple and reasonable measure of model invariance which does not rely on imagelabels.
Given predictions on a test image and its transformed version, EI measureshow well the predictions agree and with what level of confidence.
Second, usinginvariance scores computed by EI, we perform large-scale quantitative correlationstudies between generalization and invariance, focusing on rotation and grayscaletransformations.
From a model-centric view, we observe generalization and invariance of different models exhibit a strong linear relationship , on both in-distributionand out-of-distribution datasets.
From a dataset-centric view, we find a certainmodel’s accuracy and invariance linearly correlated on different test sets.
Apartfrom these major findings, other minor but interesting insights are also discussed.
Introduction
Generalization and invariance are two important model properties in machine learning.
The formercharacterizes how well a model performs when encountering in-distribution or out-of-distribution(OOD) test data [1–5].
The latter assesses how consistent model predictions are on transformedtest data [6–10].
Therefore, understanding how the two properties are related would benefit modeldecision analysis under dynamic environments.
However, there lacks quantitative and systematic analysis of the relationship between generalizationand invariance.
In fact, most existing works are qualitative. For example, adding rotation invarianceto the model improves its in-distribution (ID) classification accuracy [11–13]; a model that isgood at perturbation invariance will likely have a stronger generalization ability [9].
Moreover,existing research is limited to a few in-distribution datasets and classifier architectures.
As such, therelationship of interest remains unknown on many other scenarios, such as out-of-distribution andlarge-scale test data, and other types of models.
Before performing the quantitative study, it is necessary to first quantify generalization and invariance.
For the former, the deep models we consider are well trained, so we simply use the accuracy onthe test set, as in many previous works [3, 2, 14].
In comparison, quantifying invariance is not asstraightforward.
Some works use model accuracy drop when the test set undergoes transformations to indicate invariance ability [15, 7, 13].
While this strategy is useful for a single model, its effectivenessis limited when comparing invariance of multiple models.
Others resort to consistency, i.e. , modelsshould have the same decision [9, 6], but this method neglects prediction confidence , which we findcritical for describing invariance (see discussions in Section 3).
We make two contributions to the community.
First , we propose a new method to measure modelinvariance, named effective invariance (EI), which considers both the consistency and confidenceof predictions.
Given a test image and its transformed counterpart, if the model predicts the sameclass and with high confidence, the EI value or invariance strength is high.
Otherwise, if the modelmakes different class predictions or the confidence is low, the EI score will be low.
We show thisnew measure solves invariance valuation in canonical cases where the commonly used metrics ( e.g. ,
Jensen-Shannon divergence) may fail.
Second , we conduct a broad correlation study to quantitativelyunderstand the relationship between model generalization and invariance.
Specifically, we use 8 testsets with various distribution types, such as the in-distribution ImageNet validation set [16], and outof-distribution ImageNet-Rendition with style shift [4].
We evaluate 150 ImageNet models rangingfrom traditional convolution neural networks ( VGGs [17]) to the very recent vision transformers ( e.g. ,
BEiT [18]).
Below we list two key observations and example insights.
• For various models , there is a strong correlation between their accuracy and invariance on bothin-distribution and out-of-distribution datasets (Sections 5.1 and 5.2).
This finding can be usefulfor unsupervised model selection, because EI does not require test ground truths.•
On various out-of-distribution datasets , a model’s accuracy and EI scores are also strongly correlated (Section 5.7).
This observation can be used to predict model accuracy on out-of-distributiondatasets without access to ground truths.
• Compared with data augmentation , training with more data seems more effective to improveinvariance and generalization (Section 5.6).
Related Work
Predicting generalization gap: a model-centric view.
This task aims to predict the generalizationgap of machine learning models on in-distribution data, i.e. , the difference between training andtest accuracy.
Most existing works focus on developing complexity measurement of trained networkparameters and training data [19–27], such as persistent topology [22] and the product of norms ofthe weights across layers [24].
These methods, assuming the training and test distributions are thesame, do not consider the characteristics of the test distribution.
Thus, they have limited effectivenessto predict generalization under out-of-distribution test sets.
In addressing this limitation, we showthat invariance, under our definition, serves as a strong indicator of model generalization ability oraccuracy on both in-distribution and out-of-distribution test data.
Closely related to our work, two recent methods [28, 15] predict ID generalization gap based on howa network performs on perturbed data points.
Specifically, Kashyap et al.
[28] use confidence drop torepresent invariance, which is less effective for invariance measurement under OOD data.
Schiff et al.[15] uses accuracy drop to measure invariance, which needs test labels, while EI does not require testlabels and is more reasonable than accuracy under OOD data.
Besides, drawing a response curve in[15] is computationally heavy, while our method is efficient.
Further, both studies are limited in theirscope: they mainly study ID generalization, has two types of networks and lack large-scale test sets,while our work is much more comprehensive.
Predicting generalization gap: a dataset-centric view.
The overall goal of this task is to predict theperformance of a given model on various unlabeled test sets [29–31].
Many methods take into accountthe statistics of the test set for accuracy prediction [29, 30, 32, 31, 33], such as distribution shift[29], average Softmax score on each test set [30].
We contribute a new solution: using the model’sinvariance on the OOD dataset to predict its accuracy.
This is supported by our new observation ofstrong linear correlation between a certain model’s accuracy and invariance on various test sets.
c c c c  0.
c c c c
JS=0.019EI=0.
𝑦" = 𝑦" ! 0.
c c c c  0.
c
c c c f ( 𝒙 ) f ( 𝒙 ! )
JS=0.076EI=0.
𝑦" = 𝑦" !
(a) (b) (d)
JS=0.029EI= 𝑦" ≠ 𝑦" !
JS=0.
EI= 𝑦" ≠ 𝑦" !
(c)
Improving robustness with data augmentation.
Data augmentation transforms training data toincrease its diversity which helps learn more robust models [34–40].
For example, Mixup [41, 42]and AutoAugment [36] are shown to improve model performance under distribution changes [43, 39].
Instead of using common transformations, adversarial training [44–46] augments training images withan adversarially learned noise distribution.
While these works aim to improve corruption robustnesswith data augmentation, we instead use the latter to analyze model invariance (and generalization).
Proposed Effective Invariance (EI)
Notations .
Considering an K -way classification task, we define input space X ∈ R d and label space
Y = { 1 , ..., K } .
Given a sample ( x , y ) drawn from an unknown distribution π on X × Y , a neuralnetwork classifier f : R d → ∆ K produces a probability distribution for x on K classes, where∆ K denotes the K − 1 dimensional unit simplex.
Specifically, f i ( x ) denotes the i -th element ofthe Softmax output vector produced by f .
Then, ˆ y =:
arg max i f i ( x ) is the predicted class, andˆ p
=: max i f i ( x ) is the associated confidence score.
Image transformation is defined as T : R d → R d .
Then, the transformed image is x ′ =
T ( x ) , and its predicted class is ˆ y t
=: arg max i f i ( x ′ )
withconfidence score ˆ p t =: max
i f i ( x ′ ) .
Drawback of existing invariance measures .
A commonly seen strategy is to directly use the distancebetween the Softmax vectors of two predictions as invariance measure: a lower distance means higherinvariance, and vice versa.
Examples of the similarity metrics are Jensen-Shannon divergence (JS)[47, 34] and ℓ 2 distance [48, 49] and Kullback–Leibler divergence [50, 51].
However, they onlyleverage the global similarity between two Softmax vectors without an explicitly consideration ofprediction class consistency and confidence.
We illustrate this drawback taking JS divergence as anexample in Fig.
In cases (a) and (b) where the predicted classes are both consistent, JS decidesclassifier f in (b) has higher, which ignores the low confidence in (b).
In cases (d) where the predictedclasses are different, JS still gives high invariance (small JS score), indicating a clear error.
Definition of EI.
In its definition, given an image and its transformed sample, a model with highinvariance should give the same predicted class, and vice versa [6, 9].
In our definition of EI, wefurther use the prediction confidence.
Our motivation is as follows.
When a model predicts the sameclass for the two images, if either of the two predictions is of low confidence, we should not considerit as highly invariant but give a penalty.
A model should have a high invariance if and only if it ishighly confident in predicting the same class.
Based on these considerations, EI is defined as:
[Equation]
To better understand the soundness of EI, we depict four representative cases in Fig.
In cases (a)and (b), classifier f gives the same predicted class ( ˆ y
= ˆ y t ) on original and transformed images.
Under EI, Classifier f has a higher invariance ability in (a) because it has high confidence scores.
Incase (c), the predicted class are different ( ˆ y ̸
= ˆ y t ), and in (d), the predictions are low in confidence(and give different classes), so we define invariance in both cases to be 0.
Note that while we use thegeometric mean of ˆ p t and ˆ p , our preliminary experiment shows the arithmetic gives similar effect.
Computation of EI in practice.
Given a test image, we generate a transformed image using acertain transformation.
Then, we compute the EI score based their Softmax vectors (Eq. 3).
Weobtain the model invariance by averaging the EI scores over all the test images.
In this work, wemainly investigate the rotation transformation and grayscale transformation.
For the former, to avoidinterpolation that would introduce artifact, we only use three transformation angles ( 90 ◦ , 180 ◦ , 270 ◦ ).
For each rotation angle, we compute an overall invariance score by comparing with the predictionsof original data.
By averaging the three EI scores, we obtain rotation invariance on the test set.
For the grayscale transformation, we remove color information and keep only luminous intensityinformation.
Then, we compare with the predictions of grayscale and original data and compute theoverall grayscale invariance on each test set.
Experimental Setup
4.1 Models to Be Evaluated
We considers both very recent and classic image classification models with different architectures,including Convolutional Neural Networks ( e.g. , standard VGGs [17], ResNets [52], and modern
ConvNeXt [53]), Vision Transformers ( e.g. , ViTs [54], Swin [55], and BEiT [18]), and all-MLParchitectures [56, 57] ( i.e , MLP-Mixer [57]).
In addition to different architectures, we also cover models with various training and regularizationstrategies ( e.g. , learning rate schedule [58], label smooth [59] and data augmentation [34, 35, 37, 36]),scaling strategies in model dimension (width, depth and resolution) [60–62], and learning manners(supervised learning, semi-supervised learning [63] and knowledge distillation [64, 65]).
In total, wehave 150 models provided by TIMM [66].
They are either trained or fine-tuned on the ImageNet-1ktraining set [16].
The selected models can be roughly divided into the following three categories:
Standard neural networks.
This category includes 100 models only trained on ImageNet trainingset.
These networks cover various architectures ranging from VGGs [17] to EfficientNet [61].
Semi-supervised learning.
We include 15 models trained in a semi-supervised learning manner.
They leverage a large collection of unlabelled images of YFCC100M [67] or Instagram 900M
[68]to improve the performance.
We use models trained based on a teacher-student paradigm ( e.g. ,
SWSL-ResNet [63] and SSL-ResNet [63]).
Models trained with self-training methods on unlabeled
JFT-300M [69] ( e.g , EfficientNet-L2-NS [70]) are also included.
Pretraining on more data.
We use another 35 models that are pre-trained on significantly largerdatasets than the standard ImageNet training set.
Specifically, we consider three pre-training methods:
a) weakly supervised pretraining on IG-3.6B ( i e. , RegNetY [71] and ResNeXt101-WSL [68]);
b) supervised pre-training on ImageNet-21K [16] ( e.g. , BiT [72] and Swin [55]); (c) supervised
pretraining on JFT-300M [69] ( e.g. , ViT L/16 [54]).
4.2 Test Sets
We use both in-distribution (ID) and out-of-distribution (OOD) datasets for the correlation study.
Specifically, the ImageNet validation set (ImageNet-Val) is used as ID test set.
For OOD test sets,we use of seven datasets, each with a different distribution from standard ImageNet.
Their distributionshift can be divided into the following five types. [Figure] a [Figure]
y
[Figure]
T o p - 1 a cc [Figure] r a c y ( % ) Dataset reproduction shift.
ImageNet-V2 [14] is a recollected version of ImageNet-Val.
It containsthree versions resulting from different data sampling strategies : Matched-Frequency (A), Threshold- 0.7 (B), Top-Images (C).
version has 10 , 000 images from 1 , 000 classes.
Natural adversarial shift.
ImageNet-Adv(ersarial) [73] is adversarially selected to be misclassifiedby ResNet-50.
Its natural adversarial examples are unmodified real-world images and have beenshown to be hard for other models [73, 3].
It has 7 , 500 samples from 200 ImageNet classes.
Sketch shift.
ImageNet-S(ketch) [74] consists of sketch-like images and matches ImageNet-Val incategories and scale.
It contains 50 , 000 images and shares the same 1 , 000 classes as ImageNet.
Blur shift.
We use ImageNet-Blur with the highest severity provided by [75].
This dataset issynthesized by blurring ImageNet-Val images with a Gaussian function.
Style shift.
ImageNet-R(endition) [4] contains various abstract visual renditions ( e.g. , art, paintings,and video game) of ImageNet classes.
ImageNet-R has 30 , 000 images of 200 ImageNet classes.
4.3 Correlation Measures
We use Pearson Correlation coefficient ( r ) [76] and Spearman’s Rank Correlation coefficient ( ρ )[77] to measure the linearity and monotonicity between invariance and generalization, respectively.
Both coefficients range from [ − 1 , 1] .
A value closer to − 1 or 1 indicates strong negative or positivecorrelation, respectively, and 0 implies no correlation [76].
To precisely show the correlation, we uselogit axis scaling that maps the accuracy range from [0 , 1] to [ −∞ , + ∞ ] , following [3, 2].
Unlessnoted otherwise, the correlation coefficients are calculated using invariance score and non-linearlyscaled accuracy numbers.  Experimental Observations
Most of our experiment is from a model-centric perspective, where we investigate how differentmodels’ accuracy and invariance correlate, and series of observations are made (Section 5.1 - Section5.6).
We also adopt a dataset-centric view, where we study how a given model’s generalization andinvariance correlate on different OOD test sets (Section 5.7).
[Figure]
pa cc u r a c y (
[Figure]
[Figure]
c l a ss i f
i c a t i o n a cc u r a c y (
[Figure]
[Figure]
Similar to Fig.
2, each dot denotes a model, where different colors denote different training strategies (see Section 4.1).
The subfigures correspond to three ImageNet test sets, respectively.
We observe the correlation is also strong: Pearson’s Correlation r > 0 .
840 , and Spearman’s Correlation ρ > 0 .
5.1 Strong Correlation Between Model’s Rotation Invariance and Accuracy
In Fig.
2, we show the correlation results of rotation invariance and generalization.
We have twoobservations.
First , we find that for different models, their rotation EI scores have a linear relationshipwith their classification accuracy.
The correlation holds for both ID test and OOD test sets, variousarchitectures and training strategies.
Specifically, both correlation metrics λ and ρ are higher than0.840.
This indicates models with higher accuracy numbers are most likely to have stronger rotationinvariance (measured by EI), and vice versa.
To our knowledge, it is a very early observation of thequantitative relationship between generalization and invariance (to a certain factor).
Second , training with more data benefits rotation invariance and generalization.
Large datasetscontain images with various geometric variations.
When (pre)trained with large datasets, models(blue and orange dots in Fig. 2) adapt to the rotation variations and gains stronger invariance, which,according to our study, likely means a higher generalization accuracy on ID and OOD test sets.
5.2 Strong Correlation Between Model’s Grayscale Invariance and Accuracy
We now focus on grayscale invariance and report the correlation results in Fig.
We have thefollowing conclusions.
First , among the 150 models, there is strong linear correlation betweenaccuracy and grayscale invariance measured by EI.
Specifically, both correlation coefficients r and ρare higher than 0 .
820 on all test sets.
Second , we find that pretrained or semi-supervised models tendto have higher grayscale invariance and accuracy than standard models, which again indicates theusefulness of large training sets.
Third and interestingly, the correlation is stronger on ImageNet-Rthan ImageNet-Val and ImageNet-V2-A ( 0 . 940 vs. 0 .
900 vs. 0 . 868 ).
In fact, ImageNet-R is featuredby style shift during its collection [4], so for this test set being invariant to color changes is animportant property for a stronger generalization ability.
5.3 Correlation Exists on Very Hard Test Sets
We now study the correlation under very hard test sets and use ImageNet-A and ImageNet-GaussBlurfor testing, on which accuracy of 83 models is lower than 20% .
We study rotation invariance and [Figure] [Figure] grayscale invariance.
In Fig.
4, we find strong linear correlation in the four cases ( r ≤ 0 . 88 ).
The rank correlation is less consistent but still indicates clear trends.
Moreover, most standard models havelow accuracy, while models (pre)trained with more data tend to have high accuracy and invariance.
We also notice standard models are scattered differently in the low-accuracy regime of ImageNet-Aand ImageNet-Blur, possibly due to their dataset bias introduced during dataset construction [79–81].
5.4 Correlation Holds Under the CIFAR-10 Setup
We conduct correlation study on CIFAR-10 setup.
We collect 90 CIFAR models ranging from LeNetto EfficientNet.
We use the ID CIFAR-10 test set and two OOD test sets.
1) CIFAR-10.1 [82] isa reproduction of the CIFAR-10 test set but with distribution shift arising from changes in datacollection.
It contains 2 , 000 test images sampled from TinyImages [83].
2) CINIC-10 test set [84] isan extended alternative for CIFAR-10.
It has has 90 , 000 images sampled from ImageNet.
We show the relationship between model generalization and invariance in Fig.
On all the threetest sets, we observe the strong correlation between model accuracy numbers and rotation invariancescores, where both r and ρ are greater than 0 . 90 .
In the supplementary materials, we show correlationmaintains with EI measured grayscale invariance.
5.5 EI Gives Stronger Correlation Than JS
In the representative cases (Fig. 1), EI shows superiority to JS in measuring model invariance byexplicitly considering Softmax consistency and confidence.
Now we compare the invariance scoresmeasured by EI and JS w.r.t their correlation strength with accuracy in Fig.
6 and Table 1.
We findthat JS provides good correlation on the ID ImageNet-Val test set, but much weaker correlation onthe more difficult OOD tests than EI.
As discussed in Section 3, the advantage of EI is that it not Test set JS EI
ImageNet-Val − 0 .+0 .
ImageNet-R − 0 .+0 . ImageNet-S +0 .+0 . ImageNet-A − 0 .+0 . [Figure]
only follows the definition of invariance, but also integrates confidence to strength it while
compares the overall Softmax vector.
In fact, the drawback of JS is primarily reflected in hard testsets, where the Softmax vector usually has a flat shape.
This explains why JS does not give strongcorrelation on the OOD test sets (see Section 5.3 for EI’s performance on very hard test sets).
Werefer readers to the supplementary material for comparisons with other measures.
5.6 Comparing Two Training Manners of Their Generalization and Invariance
Existing works report that using more diverse training data artificially ( i.e. , data augmentation) ornaturally ( i.e. , more real-world data) improves model invariance [4, 34–39, 85].
In this study wecompare the two strategies of their generalization and invariance abilities.
We use 14 ResNet-50models trained with strong data augmentation such as PixMix [40] and AutoAugment [36].
Forcomparison, we employ another 3 ResNet-50 models that learn invariance using more trainingsamples: SWSL-ResNet-50, SSL-ResNet-50, and ResNet-50-BiTM.
In Fig.
7, we observe thatmodels with heavy and strong augmentation exhibit linear trends on ImageNet-Val and ImageNet-R,but deviate on ImageNet-A.
In comparison, models (pre)trained with more data follow the linear trendon three test sets.
The latter seem more effective in improving invariance and accuracy comparedwith data augmentation, especially on ImageNet-A.
To thoroughly understand and extend this initialobservation, we will conduct more comprehensive experiment on this specific point in future.
5.7 Correlation of A Model’s Invariance and Generalization on Various OOD Test Sets
From a dataset-centric perspective, we study the relationship between generalization and invarianceof a given model on different OOD test sets.
Specifically, given a model, we calculate its accuracyon every test test of ImageNet-C [75] and compute its rotation and grayscale invariance scores. We [Figure] r a c
[Figure]
o p -
[Figure] a cc u r a c y [Figure] evaluate ResNet-152, ViT-Base-16, and DenseNet-121 classifiers for the correlation study.
As shownin Fig. 8, there is a very strong correlation between classifier accuracy and invariance on variousdatasets ( r > 0 .
93 and ρ > 0 . 95 ).
The results indicate that a classifier tends to have a high accuracyon the test set where it has a high EI score.
The above analysis indicates that it is feasible to use EI toaccess out-of-distribution error of a model.
Conclusion
This work considers two critical properties of a machine learning model: invariance and generalization.
To study their relationship, we first introduce effective invariance (EI) to more reasonably measureinvariance and then provide in-depth and comprehensive correlation experiment.
From a model- centric perspective, we report accuracy and EI of various models has a strong linear relationship on both ID and OOD datasets , which is validated on many scenarios such as large-scale test sets
CIFAR -10 and very hard test sets.
From a dataset-centric perspective, we show the accuracy and EI of a model have a strong linear relationship on various OOD datasets
Limitations and potential directions.
First , some networks with specially designed modules canbe highly invariant to some transformations [7–10, 12, 86], such as rotation invariance networks[11, 87, 13].
The rotation invariance scores of these models may present very different correlationfrom our observations.
That said, their color invariance scores may still exhibit similar correlationwith this work.
In future works, it would be interesting to understand how these models deviate fromothers.
Second , our ImageNet test sets do not include the geographic shift where images are capturedfrom various locations [4].
Recent works show this shift is also a key factor influencing modelaccuracy [88].
We leave this question to future study.
Lastly , this work focuses on classification tasks,where models are supervised by image-level annotations.
In other computer vision tasks, models maybe trained with different levels of supervision, such as instance-level bounding box annotations [89],pixel-level labels [90] and temporal context supervision [91].
Different types of supervision may leadto invariance ability to different factors, which may exhibit different correlation profiles.
Potential negative social impact.
We study fundamental model properties with public classificationdatasets, which might be misused in certain applications with ethical concerns. Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes] See Section
(b) Did you describe the limitations of your work?
[Yes] Please see Section
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] Please seeSection 6.(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
a)
Did you include the code
, data, and instructions needed
to reproduce the main
exper
-imental results (either in the supplemental material or as a URL)?
[Yes] We clearlyillustrated the publicly available datasets and models in Section 4.
We also providedetails about the calculation of EI in Section 3.
(b)
Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)
N/A
] This work does not require any training process.
Both datasetsand models are fixed following the standard protocol.
(c)
Did you report error bars (
e
.g
., with respect to
the random seed after running experiments multiple times)?
[N/A]
In our work, the trained models
are
provided by PyTorch
Image Models (timm), the correlation study does not require retraining the model, andthe datasets are also commonly used standard benchmarks.
Therefore, our results donot have any randomness.
d)
Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)
Yes
] We illustrate the computationalresources in Supplementary material.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] In Section 4, we cite
PyTorch Image Models (timm) where trained models are provided.
The datasets
we
used are also clearly cited and introduced.
(b) Did you mention the license of the assets?
[Yes] We use publicly released datasets and
timm models where the licenses are provided.
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
This work does not involve any new assets.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]