Finding Differences Between Transformers and ConvNets Using Counterfactual Simulation Testing
[Figure]
email
Abstract
Contemporary deep neural networks tend to be evaluated on static test sets.
Oneshortcoming of this is the fact that these deep neural networks cannot be easilyevaluated for robustness issues with respect to specific scene variations.
Forexample, it is hard to study the robustness of these networks to variations of objectscale, object pose, scene lighting and 3D occlusions.
The main reason is thatcollecting real datasets with fine-grained naturalistic variations can be extremelytime-consuming and expensive.
In this work, we present Counterfactual Simulation
Testing , a counterfactual framework that allows us to study the robustness ofneural networks with respect to some of these naturalistic variations by buildingrealistic synthetic scenes that allow us to ask counterfactual questions to the models,ultimately providing answers to questions such as “Would your classification stillbe correct if the object were viewed from the top?”
or “Would your classificationstill be correct if the object would be partially occluded by another object?” .
Ourmethod allows for a fair comparison of the robustness of recently released, state-ofthe-art Convolutional Neural Networks and Vision Transformers, with respect tothese naturalistic variations.
Among other interesting differences between theseclasses of architectures, we find evidence for differences in performance withrespect to object viewpoint, object scale and occlusions. Submitted to 36th
Introduction
Testing computer vision models is a challenging endeavour.
In order to claim the superiority of amodel, the computer vision community usually studies validation accuracy on the ImageNet [12]dataset.
Comparing models uniquely using Top-1 and Top-5 accuracy on this dataset can resultin an incomplete evaluation of the advantages and disadvantages of each model.
Recently, Vision
Transformers (ViTs) [15] have been proposed as an alternative deep neural network model to rival
Convolutional Neural Networks (ConvNets) [36] for computer vision tasks.
ViTs have achieved impressive results, rivaling ConvNets and sometimes outperforming them in
ImageNet accuracy.
As it stands, if we restrict training data to ImageNet-22k, BEiT-L [5], a ViTvariant, stands at the top of the ImageNet-1k leaderboard, with ConvNext-XL [38], a ConvNet, as aclose second.
Although the competition between these two classes of architectures has not yet beendecided, there have been high-profile studies of potential advantages of ViTs compared to ConvNets.
ViTs are believed to be more robust to certain adversarial attacks compared to ConvNets [51, 2, 8].
Although, recent work has also shown that this is not necessarily the case when ConvNets adopt thetraining recipes of ViTs [4].
Also, ViTs are believed to be more robust to domain generalization than
CNNs [42, 4] and against occlusion, as described in Naseer et al.
This last work shows that
ViTs are more resistant than ConvNets to different types of patch removal from images, which is away to simulate occlusions.
One limitation of this study is that patch removal is a convenient but notfully realistic proxy for occlusion.
It is natural to use such a proxy since datasets to study the effectsof occlusion on object recognition and detection are scarce.
A more general limitation to works that compare properties of ViTs and ConvNets is that, even thoughthey try to compare models of similar sizes and ImageNet accuracies, they do not account for the factthat the compared ConvNets use slightly out-of-date design and training recipes .
In their impressivework, Liu et al.
[38] propose a new class of ConvNets with modernized architecture design that seeksto closely resemble the design of ViTs, yet only uses convolutional layers.
This work allows for acloser inspection of whether Transformers are superior to ConvNets due to the difference in inductivebiases between transformer and convolutional layers.
We believe that studying the differences that arise in learned representations between Transformersand ConvNets to natural variations such as lighting, occlusions, object scale, object pose andothers is important.
A priori, convolutional and transformer layers have different inductive biasesthat should manifest themselves in different performance characteristics.
For example, there havebeen conjectures that ViTs should outperform ConvNets with respect to partial occlusion since thetransformer layers allow for early capture of long-range dependencies.
Until now, there have been two main obstacles that prevent the careful study of these differences: (1)
Transformer and ConvNet architectures were not comparable in terms of overall design techniquesand training recipe details, entangling these differences with transformer vs. convolutional layerdifferences (2) there is a scarcity of datasets that include fine-grained naturalistic variations of objectscale, object pose, scene lighting and 3D occlusions, among others.
In our work we propose an initial attempt to bridge these two obstacles, and strive to tackle thefundamental question:
Between Transformers and ConvNets; which of these models is more robust to naturalistic scenevariations such as object pose, object scale, camera viewpoint, lighting and occlusions?
In order to overcome (1) we compare the ConvNext [38] convolutional architecture to the Swin [37]
Transformer architecture.
Naturally, our contributions lie in our answers to obstacle (2).
For this, wepropose a method to test a computer vision architecture in a counterfactual manner using simulatedimages.
We call this method Counterfactual Simulation Testing.
Specifically, our method allowsus to ask counterfactual questions to the models, ultimately providing answers to questions such as“Would your classification still be correct if the object were viewed from the top?”
or “Would yourclassification still be correct if the object were partially occluded by another object?” .
This allows usto abstract from the base rate of domain gap between synthetic and real images for each architectureand to compare them fairly.
Using Counterfactual Simulation Testing we find evidence of performance differences betweencomparable ConvNets and Transformers with respect to object viewpoint, camera viewpoint, objectscale and occlusions.
We observe that consistently across different network sizes ConvNext is on average more robust than Swin with respect to object pose and camera rotations .
We also observethat ConvNext architecture usually outperform Swin architectures in terms of recognizing small scaleobjects.
Finally, we find that Swin and ConvNext architectures are roughly equivalent in terms ofrobustness with respect to occlusion, with Swin pulling ahead of ConvNext for severe occlusion.
Inorder to find these differences we generate five different realistic synthetic test sets of objects usingthe MIT ThreeDWorld
(TDW)
[18] platform with these specific naturalistic scene variations.
Thefull dataset, named Naturalistic Variation Object Dataset (NVD), contains 272k images of 92 objectmodels with 27 HDRI skybox lighting environments in an indoor scene.
In summary, our contributions include:
Counterfactual Simulation Testing , a method to test computer vision models for robustness withrespect to naturalistic scene variations in a counterfactual manner using simulated images by varyingscene parameters one-at-a-time and evaluating the stability of predictions.
Naturalistic Variation Object Dataset (NVD) , a dataset containing 272k images of 92 object modelswith 27 HDRI skybox lighting environments in a kitchen scene with 5 subsets of naturalistic scenevariations: object pose, object scale, 360° panoramic camera rotation, top-to-frontal object viewand occlusion with different objects.
We hope that this dataset will allow for evaluation of moderncomputer vision models with respect to generalization to the synthetic domain and robustness to thenaturalistic variations contained therein.
A sample of NVD is shown in Fig.
Counterfactual Simulation Testing
Testing robustness to natural variations in data.
One major obstacle in evaluating models is thatoften, an aggregate metric, such as top-1 accuracy, will be used for comparison purposes.
Thesemetrics can give a certain sense of the power of the model, but can hide intricacies in performancewith respect to natural data variations.
For example, it is not possible to know to what degree a modelis robust to occlusion given top-1 an top-5 ImageNet accuracy metrics.
We tackle this problem by generating large amounts of realistic synthetic data, due to the dearth ofreal datasets exploring these variations.
An object recognition network can be written as y = f
( x ) ,where y is the predicted label from the image x by the network f .
The network f can be eithera convolutional neural network of a vision transformer.
Our scene generator can be expressed asfollows: x = g ( θ i , ψ i , κ i 0 ) , where g is the simulator, θ i is the variable scene parameter of interest(e.g.
object pose, scale, etc.), ψ i are the constant parameters controlling the main object modeltype, occluder object model type and lighting environment and κ i 0 are the constant scene parameters(kitchen objects in scene, camera focal length, field of view, etc.).
The variable i denotes the differenttrials where the selection of variable scene parameter θ i changes (thus κ i 0 also changes).
In terms of scene content, in our work, only the main object (and occluder objects) vary in thescene.
The rest of the scene is a pre-designed indoor scene.
For same θ i we generate different sceneswith different object models (determined by ψ i ) and different lighting environments (also in ψ i ) fordiversity purposes.
For different scene variations encoded in θ
i
we select object occlusions, objectscale, object rotation around its vertical axis, camera elevation and camera panoramic rotation aroundthe main object (i.e. i = 5 ).
We describe the details of the dataset in Section 3.
Once we have generated such a dataset in this manner, we would like to test our network for robustnesswith respect to all selected variations θ i .
One way is to test the network on all generated images andproduce an expectation metric (averaged over ψ i ) with respect to the variable scene parameter θ i :
[Equation]
where x θ i =
g ( θ i , ψ i , κ i 0 ) , ˆ y is the true label, and m can be any metric.
Naturally, we could usetop-1 or top-5 accuracy as this metric
and we could plot M with respect to θ i .
This could allow us tocompare different networks to find which network is more robust.
Unfortunately, this solution givesrise to another important problem: domain generalization.
By comparing two networks f and h , weseek to understand their differences when applied to real data.
Yet both models might have differentgeneralization performance to the synthetic domain, having been trained on real data.
This render acomparison of M f and M h unfeasible.
How to address the Real to Synthetic domain gap.
Comparing pre-trained models out-of-the-boxon a synthetic domain, even if highly realistic like ours, will elicit differing performances given the capabilities of models to generalize to this out-of-distribution domain.
In order to avoid the brunt ofthis issue, we propose a way of asking counterfactual questions such as “Is your answer still correctwhen I rotate the object by 60 degrees?” or “Is your answer still correct when I shrink the object tohalf of its size?” to a model.
As opposed to naively computing expected metrics, this abstracts fromall prediction failures due to the model’s inability to classify an object in the simulated domain underideal circumstances due to the domain gap.
This allows us to ask the deeper question “On average, is
ConvNext-Small better at recognizing small objects compared to Swin-Small?” .
Let ˜ θ
i be the reference condition with respect to which we will ask all counterfactual questions.
Inmost cases, this condition should be selected to be the average ideal condition for object recognition.
For example, with respect to pose, this should be the canonical pose that elicits highest modelaccuracy.
We compute the proportion of correct conserved predictions (PCCP) metric with respect tothis reference condition as follows:
[Equation]
where S i is the set of all θ i in which f
( x ˜ θ i ) =
y , and n ( S i ) is the cardinality of S i .
We can thenstudy this metric under variable θ i and compare different models using point estimates, integralsover intervals (or spaces) of θ i , as well as plots of the metric.
Under our counterfactual framework,computing C f ( θ i ) is effectively asking the question “What proportion of your answers would stillbe correct if ˜ θ i
→ θ i happened?” .
In this way, PCCP metrics of different networks are comparablequantities of the relative robustness of the network’s predictions with respect to the reference condition.
In our problem we select reference conditions that present the easiest conditions for object recognitionin order to have a large amount of initial correct predictions.
In summary, this approach, along with our selection of object models that are easy to classify, andhigh performance of our tested networks on these objects in the synthetic domain under canonicalpose and ideal lighting, allows us to abstract from the synthetic domain gap.
We can also considerdifferent metrics such as stability of predictions , or even proportion of incorrect predictions thatbecome correct .
We discuss these in the supplementary material, we limit ourselves to PCCP in ourmain work since it suffices to study differences between ConvNets and Transformers.
Naturalistic Variation Object Dataset (NVD)
NVD is composed of five different subsets which seek to study naturalistic variations of scenesfor object classification.
Specifically, object pose, object scale, 360° panoramic camera rotation,top-to-frontal object view and occlusion with different objects.
NVD is built using our customizablescene generator built on top of the MIT ThreeDWorld (TDW)
[18] platform.
Main objects.
We first study the intersection of the ImageNet-1k label space with the availableobject models in the TDW model library.
After selecting all context appropriate object types weend up with 18 model classes.
From these classes, we filter the ’iPod’ and ’paintbrush’ classesdue to very low generalization accuracy across all architectures.
The final classes are: ’banana’ ,’baseball’ , ’cowboy hat, ten-gallon hat’ , ’cup’ , ’dumbbell’ , ’frying pan, frypan, skillet’ , ’hammer’ ,’ice cream, icecream’ , ’laptop, laptop computer’ , ’microwave, microwave oven’ , ’mouse, computermouse’ , ’orange’ , ’pillow’ , ’plate’ , ’screwdriver’ , ’spatula’ and ’vase’ .
We found 99 object models inthe TDW model library that are contained in this set of classes.
We filter out 7 object models due tolow quality or inconsistent size with other objects.
In total we use 92 models to generate NVD.
The
TDW model library provides high-quality objects that are useful for studying object recognition dueto the level of realism that is hard to find in open-source projects.
A subset of the models can be seenin Fig.
Due to the relatively limited amount of realistic indoor object models in this library, someclasses have more diversity in terms of different object models than others.
We provide details on theamount of objects per class in the supplementary material.
HDRI skybox lighting environments.
In order to increase the amount of variability we lightthe scene using 27 different HDRI skybox lighting environments that span the range of dark tooversaturated and contain unique features such as overhead artificial lighting, shadows, coloredsunlight/moonlight, among others.
We use the InteriorSceneLighting addon for TDW for this, and the 27 pre-defined available HDRI skyboxes from TDW.
Some examples of different lighting can beobserved in Fig.
1 and examples of all of them are included in the supplementary material.
Naturalistic variations.
We generate 5 subsets of NVD with variations in object pose, panoramiccamera rotation, object scale, top-to-frontal object view and occlusion with different objects.
Forobject pose, we perform a full rotation of the main object around its vertical axis with step size of15°.
For 360° panoramic camera rotation we perform a full camera rotation around the main objectwith step size of 30°, for object scale we scale the main object by s = u ( o ) ∗ 0 .
25 , where u ( o ) isthe unit scale of object o computed using the TDW get_unit_scale function.
We multiply by 0 .
25in order for the objects to be scaled in line with the context objects in the kitchen scene.
Then wevary this scale multiplier by multipying it by a scale factor in the [0 . 2 , 1] interval, with step size0 .
For top-to-frontal object view, we vary the camera elevation in the [0 ◦ , 90 ◦ ] range with stepsize of 5.
For occlusion, we position an occluder object between the camera and the main object,and vary its x-axis position, such that it starts at the left of the frame and ends at the right, passingdirectly between the camera center and the main object.
Thus, the main object is occluded by theoccluder in a variable manner.
We use three different occluder objects that have very distinct visualcharacteristics and occlude the object in different manners: a stone bookend, a red die and a green
L-shaped block.
These objects do not have a corresponding label in the ImageNet-1k dataset, andthus do not act as first-order distractors with respect to the main object, although they may havesecond-order associations with classes in ImageNet and thus may still act as distractors.
Experiments
For all experiments, we compare ConvNext architectures with their “twin” Swin architectures.
ConvNext and Swin network both have five overlapping architectures that have extremely similarnumber of parameters and ImageNet validation accuracy.
For a more detailed inspection of thesestatistics, see Table 1.
We use the official open-sourced code for both models, as well as the publicmodel weights.
All PCCP metrics in this section are computed using top-5 predictions for greaterstability of results.
We use two GeForce RTX 2080 GPU to perform all experiments.
ConvNext networks generalize better to the NVD synthetic domain than Swin Transformers.
We compare the mean performance of twin ConvNext and Swin architectures in Table 1.
Firstly, weobserve that all twin architectures have almost identical model sizes and FLOPs.
They also havevery comparable ImageNet accuracies, with some architecture pairs only differing by tenths of apercentage point.
We observe that all ConvNext models obtain higher mean accuracy on NVD than
[Table]
their twin Swin models.
In particular, even when the corresponding ConvNext network performs [Table] worse than the twin Swin network on ImageNet (e.g. ConvNext-S/Swin-S) the ConvNext networksperforms better on NVD.
This is a surprising finding that goes against the belief that Transformers are more robust to OODgeneralization with findings of greater OOD generalization detailed in Bai et al.
[4] and Paul etal.
The improvement in OOD generalization in ConvNext networks is most likely a consequenceof the modernized design of the architecture, since Bai et al.
[4] align training recipes in their work.
4.1 Counterfactual Simulation Testing
Swin Transformers are more robust to occlusions than ConvNext networks.
Several workshave claimed superiority of Transformers over ConvNets in the case of occlusions.
Naseer et al.
[41]compare ResNets to DeiTs by dropping random test image patches.
They show that DeiTs are muchmore robust than ResNets to this transformation.
Although DeiTs are superior to ResNets in thisscenario, we can question whether (1) this is a general phenomenon brought about by the differencesbetween conv layers and transformer layers and (2) this type of patch dropping actually approximatesnaturalistic occlusions due to other objects in a scene.
For (1), we provide evidence that this difference between ResNets and DeiTs is not due to the natureof transformer vs. conv layers.
In order to do so, we re-run the random patch drop experiments in [41]on all pairs of ConvNext and Swin networks.
We present results for different levels of informationloss in Fig.
For all networks we observe a drop in performance as information loss increases.
We show that ConvNext networks do not suffer the same critical failure mode that DenseNet121,
ResNet50, SqueezeNet and VGG19 exhibit in Naseer et al.
The top accuracy of these fourclassic ConvNets collapses to half after about 10% of patches are occluded and to 0 after about40% information loss.
ConvNext shows that it is much more resistant, conserving a large part of itsaccuracy even after 50% information loss.
Next, we observe that ConvNext networks are slightly lessrobust than Swin Transformers for low amounts of information loss (under 50%).
Thereafter, Swinbecomes comparatively much more robust as occlusion becomes severe.
For (2), we conduct counterfactual simulation testing of ConvNext and Swin networks with variableocclusion.
We generate images of the main object on the center table of the scene, and add an occluderobject between the camera and the main object.
The details of this subset are explained in Section 3.
In this counterfactual experiment we compare all occluded scenes with the non-occluded initial scenewhere the occluder is not in the scene.
We observe in Fig.
4 that both Swin and ConvNext tend tohave similar failure responses to the variable occlusions and that performance drops precipitously,achieving a minimum at zero in the x-axis.
It is important to note that for maximal occlusions(i.e.
occluder object at zero in x-axis)
all Swin networks exhibit stronger robustness than ConvNextnetworks.
For many of the lesser occlusions, Swin networks are still slightly ahead of ConvNextnetworks.
We conclude that Swin Transformers are slightly more robust to occlusions than ConvNextnetworks and this echoes the findings of our experiment on real ImageNet images with patch drop.
ConvNext networks are more robust to object scale than Swin Transformers.
In order tocompare the robustness of ConvNext and Swin networks to scale, we first generate images of allmain object models with unit object scale.
Although objects have different volume in this state (e.g.microwaves are larger than spatulas), they take a fair amount of space in the frame and don’t presentmuch of a challenge to large architectures.
Specifically, top-5 accuracies for Swin-L and ConvNext-L validation set. Swin Transformers are slightly more robust to this type of artificial occlusion than ConvNext networks when the information loss is small, although they become comparatively much stronger as the information loss is increased. are 90.5% and 93.8% respectively.
We then perform counterfactual simulation testing by plottingthe stability of correct predictions when the object scale multiplier is reduced from 1.
We show theseresults in Fig.
We observe that for Small, Base-22k and Large-22k networks, ConvNext vastlyoutperforms Swin for smaller objects.
For Base-22k and Large, the difference becomes large afterthe scale multiplier is under 0.6.
For Tiny and Base networks, ConvNext and Swin perform in verysimilar manner, with Swin slightly outperforming ConvNext on very small objects by a small margin.
ConvNext networks are more robust to rare top and frontal object views than Swin networks.
Here we compare the robustness of ConvNext and Swin networks to different viewpoints of themain object.
Specifically, we start with a camera viewpoint that captures the main object from afully-frontal view to a fully-top view.
Object recognition is known to fail when the object is viewedfrom rare poses such as from the top [1].
In this experiment we compare all views to the canonicalview with a camera pointing at the object with a 45° elevation.
We show in Figure 6, that on ImageNet-1k and ImageNet-22k-trained ConvNext and Swin networks,this is still the case.
Both network classes exhibit drastic drops in proportion of conserved correctpredictions when the pose is modified from a canonical pose with a camera at 45° of elevation toeither a fully-frontal pose at 0° or a fully-top view at 90°.
We observe that both increasing networksize and pre-training on ImageNet-22k increase robustness on both architectures, with the best modelbeing ConvNext-L-22k which achieves a conservation of correct answers in top view of around 73%.
We observe as well that all ConvNext networks obtain a higher proportion of conserved correctpredictions for views from the top (e.g. with high elevation) than comparable Swin networks.
Weconclude that ConvNext networks are more robust to rare top views of objects than Swin Transformersgiven comparable architecture sizes and identical training data and recipes.
Finally, we find thatmost ConvNext networks have higher proportion of conserved correct predictions across differentelevation angles, with some notable exceptions such as a plummeting robustness of ConvNext-S inlow elevation, and slightly lower robustness of ConvNext-B and ConvNext-B-22k compared to theirtwin Swin networks in certain elevation intervals.
ConvNext networks are, on average, more robust than Swin networks to different lateral objectviewpoints.
In order to test the robustness of both classes of networks to different lateral views ofobjects, we conduct two counterfactual studies.
The first scene variation consists in a panoramic 360°camera rotation around the object.
In this scenario the lighting on the object can change drastically.
Many lighting skyboxes in NVD contain a light source coming through a window behind the object,and when the camera is rotated we can either have a high contrast view of the object with the mainlight source behind the camera, or a shaded view of the object against the light.
The second variationconsists of a fixed camera setting with the main object performing a full 360° rotation around itsvertical axis.
This allows us to keep lighting fixed while we evaluate the networks on different poses for the same object.
Using both experiments we can disentangle decreases in performance due toobject pose or the interplay between object pose and lighting.
We plot the proportion of conserved correct predictions for the panoramic camera rotation in Fig.
First, we observe that both networks struggle to a higher extent with images taken on opposite to theposition, with the light source behind the camera.
This is due to two factors: the rear-view object are,in general, rare cases (e.g. the back of a microwave) and some lighting environments provide verystrong light resulting in overexposed scenes with less object detail.
Interestingly, we observe thatfor all networks ConvNext is more robust than Swin, with the slight exception of the 150° to 180°range for Base-22k networks.
We also see in this experiment that curves are much less smooth andirregular.
This is partly due to the nature of object rotations: some objects are easy to recognize inmultiple different views, while others present very challenging views within small rotation intervals.
This experiment entangles lighting and object pose.
In order to partially disentangle them, we performa study where the camera is fixed and the main object is rotated along its vertical axis.
We canobserve in Fig.
8, that, on average, ConvNext is more robust to these rotations than Swin.
Even so, thedifference between both networks is lower than in the previous experiment, suggesting that ConvNextis more robust to different lighting and that this contributes to performance differences in Fig.
Related Work and Discussion
Simulation In recent years, the community has explored using synthetic data for training deepnetworks [17, 48, 47, 16, 44, 6].
Some work goes a step further and learns the distribution of dataneeded to improve model performance [49, 39, 19, 7, 30, 3].
There is an easy way of measuringsuccess of these models by evaluating them on real data.
There are also attempts to benchmark andtest models using synthetic data [45, 40, 28, 6, 31, 50].
The synthetic-real domain gap complicatesthis endeavour, making it hard to generalize insights to the real-world.
In contrast, we propose acounterfactual study of model failure using a photorealistic simulator.
This allows us to abstract fromthe model domain gap base rate.
A different direction of work that could allow for this is the domainadaptation literature which adapts pixels or features to bridge the gap [20, 9, 56, 55, 26, 43].
ConvNet and Transformer comparative studies ConvNets have dominated the field of computervision for the last decade [36, 34, 52, 23, 27, 53, 38].
Vision Transformers have recently provento be extremely competitive, sometimes outperforming ConvNets in computer vision tasks [15, 54,57, 58, 37, 5, 14].
There have recently been studies into the differences between ConvNets and
Transformers.
Among other studies [41, 59], Raghu et al.
[46] study the internal representation of
ViTs, finding early aggregation of global information and strong propagation of features from lowerto higher layers.
Bai et al.
[4] show that CNNs can be as adversarially robust as ViTs, but lag behindin OOD generalization.
In contrast to this work, we compare recent CNNs and ViTs with respect tonaturalistic scene variations such as occlusion and object pose in a fine-grained manner, with somesurprising findings.
To the best of our knowledge, ours is the first work that seeks to compare these classes of architectures in this way and that proposes a way to equalize the (synthetic domain) playingfield using counterfactual analysis.
Robustness and counterfactual fairness There is research on OOD robustness of networksincluding synthetic images [44], stylized images [21], corrupted images [24] and natural adversarialimages [25].
Our effort complements this direction with a study of robustness with respect to naturalscene variations.
There exist investigations into network fragility regarding rare object poses [1] andocclusion [60, 33] - and work that tries to address the weaknesses [32].
Our work represents a seriousattempt at providing a flexible and general method with which to study these types of variations, alongwith many others that are less common in the literature such as realistic lighting environment changes.
Our counterfactual framework allows us to compare the robustness of different architectures thatmight not have the same base-level performance in a fair setting.
There exists work on counterfactualfairness [35, 10, 29], a field that studies ML decision fairness from a counterfactual viewpoint.
The closest literature to our work is the sparse literature of counterfactual interpretability [22] andsensitivity [11, 13, 50].
To the best of our knowledge, we are the first to propose a method to studyobject recognition networks for robustness with respect to natural scene variations by combining twokey ingredients: a counterfactual approach and realistic synthetic data with fine-grained variations.
Limitations Due to limited real data, we test networks on simulated data.
It can be hard togeneralize insight into the real world.
We tackle this limitation by (1) generating a photorealisticdataset using state-of-the-art research software, with networks achieving very high accuracy on objectrecognition under ideal scene parameters (2) proposing counterfactual studies in order to furtherabstract from domain gap.
General statements about ConvNets and Transformers are hard to make,given that architectures evolve.
In contrast to previous explorations, we propose to study the mostcomparable architectures to date; ConvNext and Swin, due to their similar design choices and alignedtraining recipes.
To further increase robustness, we study 4 different sizes of these networks with 2different training datasets (ImageNet-1k and ImageNet-22k).
Conclusion
We conduct a study of Transformers and ConvNets by proposing NVD; a novel realistic syntheticdataset of naturalistic scene variations.
By using our proposed Counterfactual Simulation Testingapproach, we are able to fairly compare state-of-the-art ConvNets and Transformers and find evidenceof differences in their ability to handle variations in object pose, scale, camera pose and occlusion.
We release both NVD and our flexible and customizable scene generator.
Broader Impact Our work can be used to address model bias, e.g. face analysis networks with arespective simulator.
Interactions exist with applications that can negative for society, for example,knowing weaknesses of networks can lead to simple ways of attacking them using natural inputs.
Asalways, improving computer vision models goes hand in hand with enabling applications such asmalicious surveillance.
We include a more detailed discussion in the supplementary material.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] Extended discussion in supp.
material.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e
g., data splits, hyperparameters,
how they
were chosen)?
[Yes]
(c) Did you report error bars (e
g., with respect to the random seed after running experi- ments multiple times)?
[N/A]
(d) Did you include the total amount of compute and the type of resources used (e
g., typeof GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes] Supp.
Material
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
No people involved in study.
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A] No personally identifiable information.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]