Resolution Adaptive Networks for Eﬃcient Inference
Anonymous Author(s)
AﬃliationAddress email
Reproducibility Summary
In this paper the authors (Yang et al
) propose a Resolution Adaptive CNN network (RANet) to eﬃcientlyclassify images at diﬀerent resolutions depending based on the classiﬁcation diﬃculty (prediction conﬁdence).
The main idea is that some images are easy to classify, only requiring low resolution and simple features (lowcomputational cost) while other images require higher resolution and larger networks (large computationalcost).
RANet attempts to classify a low-resolution representation of the image and if the classiﬁcationconﬁdence is below some threshold, will attempt will try to classify the image with higher and higher resolutionwith subsequent subnetworks.
9 The authors make two main assertions that we (The Einsteins) hope to verify in this reproducibility study.
Firstly does the RANet architecture oﬀer signiﬁcant reductions in computational cost while maintaining accuracy to comparable models.
Secondly, in an arbitrary dataset there’s such a distribution of ‘easy’ and
‘hard’ classiﬁcations tasks that would warrant the use of and adaptive networks.
We tested the ﬁrst assumptionby creating our implementation of RANet in Tensorﬂow Keras and found it to have comparable accuracy with
VGG16 and Resnet50, while having a reduction in computational time by up to 64%.
We tested the secondassumption by creating our own “spalling” classiﬁer using the authors’ provided code and kept track of howoften each subnetwork returned the prediction.
We found that the assertion that some images are ‘easier’ ormore ‘diﬃcult’ to classify is valid, with 74% of spalling images being considered ‘hard’ to classify and the restof the spalling images being ‘easy’ to classify.
20 While
, we have not reproduced the authors’ results directly, we show that the proposed method shows promiseand may have application for real-time and/or AI at the edge applications
Introduction
[Figure]
Figure 1, from Yang et al.
Provides a good overview of the proposed RANet architecture.
To start the inputimage is resized into 3 scales, where each scale is half of the previous.
In a forward pass, the smallest image(scale 1) is passed into the ﬁrst subnetwork.
If the classiﬁcation conﬁdence of the ﬁrst subnet is below somethreshold, the second subnetwork at scale 2 will be used for classiﬁcation.
The output of the Conv Block atthe scale minus one is use ‘fused’ with the feature maps of the Conv Block of the current scale.
This process is repeated until a prediction of high conﬁdence is
The typical Conv Block in RANet consists of a convolution layer (CONV2D), batch-normalization (BN) andrectiﬁed linear (ReLU) activation.
To combine the output of the Conv Block at scale and scale minus one, thefeature maps of both are concatenated, but the shapes of these outputs must match.
The authors propose2 diﬀerent methods: up sample feature maps of scale minus one or down sample the feature maps at scalebefore concatenation.
RANet is trained from end to end using the categorical cross-entropy metric.
There are many intermediaryclassiﬁers in RANet, this is used in training where the ﬁnal loss is a weighted sum of these intermediarylosses.
RANet is trained using stochastic gradient decent (SGD), for 300 epochs, with varying learning rate,momentum of 0.9 and weights decomposition.
The dataset is augmented by pixel standardization and randomhorizontal ﬂips.
Scope of Reproducibility
The authors assert that some images are easier to classify than others, such that an adaptive network duringinference can decrease computational cost by using less computation for easily classiﬁed samples.
The authorsclaim that their RANet maintains the same classiﬁcation accuracy while reducing computation cost of 65%,56% and 44% compared to GoogLeNet, ResNet and DenseNet respectively.
We assert that for some hardwarethe computational costs are linearly related to computational time.
We will implement our own RANet with
Tensorﬂow Keras and compare the computation time with popular CNN models.
The authors will trainour own networks from scratch and will not train by using Imagenet dataset.
For this reason, we will alsorefrain from training large models due to time and computational costs.
Due to the diﬃculty of working withlarge datasets, we will also restrict our study to the CIFAR10 dataset.
In this study we will compare our
RANet to VGG16 and Resnet50, where we expect a roughly 2x decrease in computational time for no loss inclassiﬁcation accuracy.
Additionally, to test the assertion that in a dataset there are images that are easier to classify than others,we will modify one of our datasets to detect concrete spalling to train a binary classiﬁer using the authors’code.
We will implement four subnetworks to determine what percentage of ﬁnal predictions are producedby each subnetwork.
To validate this, the authors hope to observe that a signiﬁcant portion of images arebeing predicted using the earlier, less computationally intensive subnetworks, proving that spalling imagesare ‘easy’ to classify and that RANet can be used to improve the eﬃciency of existing spalling classiﬁers.
Methodology
For the reproducibility project we implemented the proposed RANet architecture with Tensorﬂow Keras.
The original authors had implemented their code in Pytorch, from which we based our layer initializations,but we based our architecture entirely on what was described in the authors’ paper.
The result is thatour implementation and the authors are superﬁcially diﬀerent but theoretically functions the same.
Our
RANet model was trained from scratch using CIFAR10 dataset with 80/20 train-test split.
We used
ImageDataGenerator for image augmentation, learning rate schedule and SGD.
Our model achieved a peakvalidation accuracy of 87% during training, higher accuracies were possible but were not attempted due totime constraints.
During training we recorded loss from each subnetwork, and they are combined linearly tocalculate the total losses, they are provided in ﬁgure 2 and table 1.
Accuracy
[Figure]
From the ﬁgures above we observe that the small sub-model achieved an accuracy of 52%, the mediumsub-model achieved an accuracy of 78% and ﬁnally the large sub-model achieved an accuracy of 87%.
Theseresults match our intuition that a reduction in image resolution will result in reduced accuracy.
As stated inthe scope of reproducibility, we will not be conducting budgeted inference, rather we are interested in averagecomputation time per image.
We trained VGG16 and Resnet50 on CIFAR10 to baseline against our RANetmodel.
After a model was trained each was run on a similar experiment pipeline.
We recorded the averagetime for a given model to predict 100 randomly selected validation images, this was done three times permodel.
Since the speed of RANet depends on the accuracy threshold, the accuracy of the prediction is alsorecorded, while we assumed VGG16 and Resnet50 will average to the validation accuracy.
For the second experiment, the dataset was provided by the CVISS (Computer Vision and Smart Structures)
Lab in the department of Civil and Environmental Engineering at the University of Waterloo.
The datasetconsists of approximately 1000 earthquake images where spalled concrete damage was labeled with boundingboxes.
Using a 80/20 split, we divided the images into training and validation datasets. We then extractedmultiple square patches from each image where spalling damage may or may not be present.
In total, weextracted approximately 20000 patches for training and 5000 patches for validation of the model.
In ﬁgure 3, the ‘red’ annotations are the spalling labels, ‘green’ are the extracted spalling patches and‘yellow’ are non-spalling patches.
These image patches are resized to 224 x 224 (ImageNet resolution) and areused for training of the binary classiﬁer.
We implemented the author’s code and used four subnetworks forclassiﬁcation.
We tested our hypothesis on 812 images, and
Results
We found that RANet decreased computational cost as claimed in the original paper.
With no decrease inaccuracy RANet was 2x and 2.8x faster than VGG16 and Resnet50.
[Table]
The metrics that we used diﬀer from those the authors used in the paper, however our results do support theauthor’s conclusions about the potential such an architecture has reduce the computational cost of inference.
For example, we observed a 64% decrease in computational time compared to Resnet50, which is not farfrom the 56% decrease in computation cost for Resnet reported by the original authors.
The proposed adaptive architecture yielded promising results, we also wanted to test how applicable thiswould be to a real-world problem in the domain of civil engineering such as spalling classiﬁcation.
Weadapted the authors’ code and used four subnetworks labelled ’ conv_net1 ’, ’ conv_net2 ’, ’ conv_net3 ’, and’ conv_net4 ’, which scale up based on complexity and size of each subnetwork.
Of the 812 images that wereclassiﬁed using RANet, 604 utilized the largest network ’ conv_net4 ’, 157 images utilized ’ conv_net3 ’, 41images utilized ’ conv_net2 ’, and only 10 images utilized the smallest ’ conv_net1 ’ network.
We found that there was indeed a distribution of ’easy’ and ’diﬃcult’ to classify images.
However, 74% ofspalling images can be considered ‘hard’ to classify and required ’ conv_net4 ’, and the rest of the spallingimages are ‘easy’ to classify.
This proves that spalling images require a high computational cost to classify andmight not necessarily beneﬁt from implementing RANet because we would still be relying on large networksthat are just as computationally intensive as traditional CNN classiﬁcation techniques.
What was easy
[Figure]
The theory behind the proposed adaptive architecture the authors describe is straight forward and intuitiveto understand.
The concept can be easily adapted for various and diverse applications.
The authors’ choiceto use standard datasets also made reproducibility easier.
What was diﬃcult
A diﬃcult task in this project is to code RANet in Tensorﬂow Keras, due to the diﬀering structure of the
Keras model class.
We found very few resources on how to make an adaptive model in Tensorﬂow and Kerasdocumentation and tutorials.
Additionally, it was diﬃcult to exactly replicate the author’s RANet as theirlayer’s parameters were not provided.
This also resulted in diﬃculties during training as we tried to achievehigher validation accuracies.
The author’s code is heavily borrowed from previous work and there were very few comments nor direction. The author’s code was easy to use and it was easy to implement our custom spalling dataset. However, wefound it diﬃcult to modify the author’s code to produce Figure 4, the number of image samples classiﬁedusing each subnetwork of RANet.
Finally, due to the university closure and that our team members operated in diﬀerent time zones, it causeddelays and diﬃculties in communication.
With much hard work we were able to produce this deliverable,while may not be to the standard of this journal, the journey here has been very rewarding.
We should liketo thank Dr. Chul Min Yeum for access to the dataset and Juan Park for hosting our test machines as weleached many kilowatt hours from his meter.
Communication with original authors
We had no contact with the original paper’s authors.
Open-Source Repository
We published our TF2 implementation in a GitHub repository: https://github.com/MACILLAS/syde671.git