The Unreasonable Effectiveness of Fully-Connected Layers for Low-Data Regimes
Anonymous Author(s)
AffiliationAddress email
Abstract
Convolutional neural networks were the standard for solving many computer visiontasks until recently, when Transformers of MLP-based architectures have started toshow competitive performance.
These architectures typically have a vast numberof weights and need to be trained on massive datasets; hence, they are not suitablefor their use in low-data regimes.
In this work, we propose a simple yet effectiveframework to improve generalization from small amounts of data.
We augmentmodern CNNs with fully-connected (FC) layers and show the massive impact thisarchitectural change has in low-data regimes.
We further present an online jointknowledge-distillation method to utilize the extra FC layers at train time but avoidthem during test time.
This allows us to improve the generalization of a CNN-basedmodel without any increase in the number of weights at test time.
We performclassification experiments for a large range of network backbones and severalstandard datasets on supervised learning and active learning.
Our experimentssignificantly outperform the networks without fully-connected layers, reaching arelative improvement of up to 16% validation accuracy in the supervised settingwithout adding any extra parameters during inference.
Introduction
Convolutional neural networks (CNNs) [1, 2] have been the dominant architecture in the field ofcomputer vision.
Traditionally, CNNs consisted of convolutional (often called cross-correlation) andpooling layers, followed by several fully-connected layers [3–5].
The need for fully-connected layerswas challenged in an influential paper [6] and recent modern CNN architectures [7–11] discardedthem without a noticeable loss of performance and a drastic decrease in the number of trainableparameters.
Recently, the "reign" of all-convolutional networks has been challenged in several papers [12–16],where CNNs were either replaced (or augmented) by vision transformers or replaced by multi-layerperceptrons (MLPs).
These methods remove the inductive biases of CNNs, leaving more learningfreedom to the network.
While showing competitive performance and often outperforming CNNs,these methods come with some major disadvantages.
Because of their typically large number ofweights and the removed inductive biases, they need to be trained on massive datasets to reachtop performance.
As a consequence, this leads to long training times and the need for massivecomputational resources.
For example, MLPMixer [14] requires a thousand TPU days to be trainedon the ImageNet dataset [17].
[Figure]
Refiner
F C + S o f t m a x
Predictions 64 F C
L a y e r
N o r m
F C
R e L U
F C
L a y e r
N o r m
[Figure]
In this paper, instead of entirely replacing convolutional layers, we go back to the basics, and augmentmodern CNNs with fully-connected neural networks, combining the best of both worlds.
Contraryto new alternative architectures [14, 15] that usually require huge training sets, we focus our studyon the opposite scenario: the low-data regime, where the number of labeled samples is very-lowto moderately low.
Remarkably, adding fully-connected layers yields a significant improvementin several standard vision datasets.
In addition, our experiments show that this is agnostic to theunderlying network architecture and find that fully-connected layers are required to achieve the bestresults.
Furthermore, we extend our study with two other settings that typically deal with a low-dataregime: active and semi-supervised learning.
We find that the same pattern holds in both cases.
An obvious explanation for the performance increase would be that adding fully-connected layerslargely increases the number of learnable parameters, which explains the increase in performance.
To disprove this theory, we use knowledge distillation based on a gradient gating mechanism thatreduces the number of used weights during inference to be equal to the number of weights of theoriginal networks, e.g., ResNet18.
We show in our experiments that the network with the train timeadded fully connected layers still significantly outperforms the original architecture, even if bothhave an equal number of weights.
In summary, our contributions are the following:
• We show that adding fully-connected layers is beneficial for the generalization of convolutional networks in the tasks working in the low-data regime.•
We present a novel online joint knowledge distillation method (OJKD), which allows us toutilize additional final fully-connected layers during training but drop them during inferencewithout a noticeable loss in performance.
Doing so, we keep the same number of weights
during test time.
• We show state-of-the-art results in supervised learning and active learning, outperformingall convolutional networks by up to 16% in the low data regime.
Methodology
We propose a simple yet effective framework for improving the generalization from a small amountof data.
In our work, we bring back fully-connected layers at the end of CNN-based architectures.
Weshow that by adding as little as 42 k ( 0 .
37% ) extra parameters during training, we can significantlyimprove the generalization in the low-data regime.
Our network architecture consists of two mainparts: a convolutional backbone network and our proposed Feature Refiner (FR) based on multi-layerperceptrons.
Our method is task and model-agnostic and can be applied to many convolutionalnetworks.
In our method, we extract features with the convolutional backbone network.
Then, we apply our FRfollowed by a task-specific head.
More precisely, we first reduce the feature dimension to 64 with asingle linear layer to reduce the number of extra parameters.
Then we apply a symmetric two-layer
Backbone
FeatureRefiner
F C + S o f t m a x
Gradient Gate
F C + S o f t m a x
Loss
[Figure]
CE
L FR
L Original CE multi-layer perceptron wrapped around by normalization layers.
We present the precise architectureof our Feature Refiner in Figure 1.
2.1 Online Joint Knowledge Distillation
One could argue that using more parameters can improve the performance just because of theincreased expressivity of the network.
To disprove this argument, we develop an online jointknowledge distillation (OJKD) method.
Our OJKD enables us to use the exact same architecture asour baseline networks during inference and utilizes our FR solely during training.
We base our training pipeline on the baseline network’s architecture.
We split the baseline networkinto two parts, the convolutional backbone for feature extraction and the final fully-connectedclassification head.
We append an additional head with our Feature Refiner.
We devise the finalloss as the sum of the two head’s losses, making sure both heads are trained in parallel (online) andenforcing that they share the same network backbone (joint).
During inference, we drop the additionalhead and use only the original one, resulting in the exact same test time architecture as our baseline.
In other words, our FR head is the teacher network that shares the backbone with the student originalhead, and we distill the knowledge of the teacher head into the student head.
However, the key ingredient of our OJKD is the gradient-gating mechanism; we call it Gradient
Gate (GG).
This gating mechanism blocks the gradient of the original head during training, makingthe backbone only depend on our FR head.
We implement this functionality with a single layer.
During the forward pass, GG works as identity and just forwards the input without any modification.
However, during the backward pass, it sets the gradient to zero.
This way, the original head’s gradientsare backpropagated only until the GG.
While the original head gets optimized, it does not influencethe training of the backbone but only adapts to it.
Consequently, the backbone is trained only withthe gradients of our FR head.
Furthermore, the original head can still fit to the backbone and reach asimilar performance as the FR head.
We find that we can still improve upon the baseline without ourgating mechanism but reach lower accuracy than when we use it.
We show the pipeline of our OJKDin Figure 2.
Experiments
In this section, we demonstrate the substantial effectiveness of our simple approach improving theperformance of neural networks in low-data regimes.
1k 2k 3k 4k 5k 6k 7k 8k 9k 10k Datasetsize[samples] V a li d a ti on acc u r ac
y ResNet FR-ResNet (a) CIFAR 5k 6k 7k 8k 9k 10k 11k 12k 13k 14k Datasetsize[samples]
V a li d a ti on acc u r ac y
{+ResNet FR-ResNet (b) CIFAR 1k 2k 3k 4k 5k 6k Datasetsize[samples] 82 .85 .
 V a li d a ti on acc u r ac+}
y ResNet
FR-ResNet (c) Caltech 5k 6k 7k 8k 9k 10k 11k 12k 13k 14k Datasetsize[samples]
V a li d a ti on acc u r ac y
ResNet
FR-ResNet (
[Figure]
) Caltech
Datasets and the number of labels.
For all experiments, we report accuracy as the primary metricand use four public datasets: CIFAR10 [18], CIFAR100 [18], Caltech101 [19], and Caltech256 [19].
We use the predefined train/test split for the CIFAR datasets, while we split the Caltech datasets into70% training and 30% testing, maintaining the class distribution.
In the simpler datasets, CIFAR10and Caltech101, we start with an initial labeled pool of 1000 images, while in the more complicated,
CIFAR100 and Caltech256, we start with 5000 labeled images.
In both cases, we incrementally add1000 samples to the labeled pool in each cycle and evaluate the performance with the larger andlarger training datasets.
We use the active learning terminology for a’ cycle’, where a cycle is definedas a complete training loop.
CNN backbones.
For most experiments, we use ResNet18 [8] as our backbone.
We compare theresults of our method with those of pure ResNet18 on both the supervised and active learning setups.
Note that the supervised case (Figure 3) is equivalent to a random labeling strategy in an activelearning setup.
We compare our method with various active learning strategies.
We also compare toa non-convolutional network, the MLPMixer [14].
Finally, to show the generability of our method,we also experiment with other backbone networks: ResNet34, EfficientNet, and DenseNet.
We runeach experiment 5 times and report the mean and standard deviation.
We train each network in asingle GPU.
We summarize the results in plots and refer to our supplementary material for the exactnumbers and the complete implementation details.
Training details.
For the CIFAR experiments, we follow the training procedure of [20].
More precisely, we train our networks for 200 epochs using SGD optimizer with learning rate 0 .
1 , momentum0 . 9 , weight decay 5 e − 4 , and divide the learning rate by 10 after 80% epochs.
We used cross-entropyloss as supervision.
For the more complex Caltech datasets, we start with an Imagenet-pre-trainedbackbone and reduce the dimensionality in our FR only to 256 .
We use the same training setup for afull fine-tuning, except that we reduce the initial learning rate to 1 e − 3 and train for only 100 epochs. 1k 2k 3k 4k 5k 6k 7k 8k 9k 10k Datasetsize[samples] V a li d a ti on acc u r ac y
ResNet FR-ResNet MLPMixer ViT-B (a) CIFAR10 test accuracy.
5k 6k 7k 8k 9k 10k 11k 12k 13k 14k Datasetsize[samples]
V a li d a ti on acc u r ac
y
ResNet
FR-ResNet
MLPMixer
ViT-B (b) CIFAR100 test accuracy.
3.1 Supervised Learning
Comparisons with ResNet18 [8].
We compare the results of our method with those of ResNet18.
Asshown in Figure 3a, on the first training cycle ( 1000 labels), our method outperforms ResNet18 by7 .
6 percentage points ( pp ).
On the second cycle, we outperform ResNet18 by more than 10 pp .
Wekeep outperforming ResNet18 until the seventh cycle, where our improvement is half a percentagepoint.
For the remaining iterations, both methods reach the same accuracy.
On the CIFAR100 dataset (see Figure 3a), we start outperforming ResNet18 by 5 .
7 pp , and in thesecond cycle, we are better by almost 7 pp .
We continue outperforming ResNet18 in all ten cycles,in the last one being better by half a percentage point.
We see similar behavior in Caltech101 (see
Figure 3c) and Caltech256 (see Figure 3d).
A common tendency for all datasets is that with an increasing number of labeled samples, the gapbetween our method and the baseline shrinks.
Therefore, dropping the fully-connected layers in caseof a large labeled dataset does not cause any disadvantage, as was found in [6].
However, that workdid not analyze this question in the low-data regime, where using FC layers after CNN architecturesis clearly beneficial.
Comparisons with MLPMixer [14] and ViT [12].
We also compare the results of our methodwith that of two non-convolutional networks, the MLPMixer and ViT. Similar to us, the power of
MLPMixer is on the strengths of the fully-connected layers.
On the other hand, ViT is a transformerbased architecture.
Each Transformer block contains an attention block and a block consistingof fully-connected layers.
Unlike these methods, our method uses both convolutional and fullyconnected layers to leverage the advantages of both the high-level convolutional features and theglobal interrelation from the fully-connected layers.
In Figure 4a, we compare to MLPMixer and ViT on the CIFAR10 dataset.
On the first cycle, we outperform MLPMixer by 15 pp and ViT by circa6 pp .
Note that MLPMixer has worse results (by 7 . 5 pp ) than the original ResNet18, showing the needfor convolutional layers in the low-data regime.
We keep outperforming both methods in all othercycles, including the last one where we do better than them by circa 20 pp .
As we can see, MLPMixerand ViT do not perform well even in the latest training cycles and require to be trained on massivedatasets.
We show a similar comparison for CIFAR100 in Figure 4b.
3.2 Active Learning
We put our method to the test under a typical low-data setting, active learning, where instead ofrandomly choosing samples to label, we choose them based on an acquisition score.
The results in
Section 3.1 can be considered a special case of active learning with a random acquisition score.
Weuse maximum entropy acquisition score with our network and compare it with plain ResNet18 withmaximum entropy and two state-of-the-art active learning methods, LLAL [20] and core-set [21]. 1k 2k 3k 4k 5k 6k 7k 8k 9k 10k Datasetsize[samples] V a li d a ti on acc u r ac y
ResNet
[Figure] Entropy FR [Figure]
ResNet
[Figure]
Entropy
LLAL
ResNet
[Figure]
core
[Figure]
[Figure]
5k 6k 7k 8k 9k 10k
[Figure]
[Figure]
V
[Figure]
li d a ti on acc u
[Figure]
ac y
[Figure]
Entropy
[Figure]
Entropy
LLAL
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
d
[Figure]
[Figure]
Entropy
FR
[Figure]
Entropy
[Figure] 6k 7k 8k 9k [Figure] [Figure]
[Figure]
Entropy [Figure] In Figure 5, we summarize the results on all four datasets, showing that our method outperformsall others. On CIFAR10, we outperform all methods in early cycles by a large margin, up to 7 . 3 ppcompared to the core-set approach in the second cycle.
As more labels become available duringtraining, all the methods tend to converge to the same result.
A similar trend can be observed on the[-CIFAR
-]
CIFAR100 dataset.
Here, our approach achieves 5 .
2 pp better accuracy in the second cycle.
We then compare our methods on the Caltech datasets.
On the Caltech101 dataset, we have a slightadvantage ( < 0 .
5 pp )
compared with the vanilla ResNet18.
We have a larger advantage on the Caltech256 dataset, where we tend to outperform the second-best method by 1 − 1 .
5 pp .
Furthermore,note that core-set and LLAL approaches perform much worse on these more complex datasets.
Thefinal gap between the maximum entropy acquisition and LLAL is 4 .
5 pp on the Caltech256 and 1 .
4 ppon the Caltech101 dataset.
3.3 Semi-Supervised Learning
We now do an experiment where we present preliminary results in semi-supervised learning.
Wechoose to couple MixMatch [22] with our method and compare it with the original version of
MixMatch.
We use ResNet18 for the experiment and train the methods using only 250 labels.
MixMatch reaches 91 .
07 accuracy, while our method reaches 91 .
56 accuracy, improving by half apercentage point, and showing that our method can be used to improve semi-supervised learning.
3.4 Backbone Agnosticism
We check if our method can be used with other backbones or is limited to being used only with ResNet18.
We show the results for ResNet34, DenseNet121, and EfficientNetB3 on CIFAR10 and CIFAR100 in Figure 6.
The goal of the experiment is to show that our method is backbone agnostic [Figure] , DenseNet121, and EfficientNet-B3 backbones. In all cases, our method significantly improves over the original network.
and generalizes both to different versions of ResNet as well as to other types of convolutional neuralnetworks.
As we can see, our method significantly outperforms the baselines on both datasets and forall three types of backbones.
3.5 Ablation
Feature Refiner We conduct a detailed ablation of the architecture of our Feature Refiner, showingthe effect of the specific elements.
We start with the backbone ResNet18 and build our Feature
Refiner up step-by-step.
Figure 7a shows the results on the CIFAR10 dataset.
First, we apply only asingle linear layer without any activation function before the output layer (512x512 w/o Activation).
Interestingly, we can already see a great improvement of 4 .
9 pp in the first cycle, reaching 5 .
pp inthe second cycle.
Second, we change the previously applied linear layer to reduce the dimension,as we did in our Feature Refiner (512x64 w/o Activation).
This step further improves the results upto 2 .
3 pp in the third cycle.
Third, we add our fully-connected layers with the ReLU activation (FRw/o LayerNorm), which yields an improvement of 3 . 1 pp in the second cycle.
Finally, we use ourcomplete architecture by applying the normalization layers, reaching the best results in the earlierstages, giving an additional performance boost of 1 . 2 pp in the second stage.
Online joint knowledge distillation We now evaluate our proposed OJKD.
In these experiments,we highlight that during inference, we can drop the Feature Refiner head without any noticeableperformance loss.
We use the same training strategy as described in Section 2.1.
We train togetherwith the original and Feature Refiner heads.
However, during inference, we evaluate both heads.
In
Figure 7b we show both heads’ performance on the CIFAR10 dataset.
As we can see, both headshave barely distinguishable performance, showing that the knowledge of our teacher FR head can beproperly distilled into the student original head.
Furthermore, we ablate the effect of our proposed
Gradient Gate.
Without our gating mechanism, we can still improve upon the baseline up to 4 .
5 pp ,but using it can give us a further 5 .
6 pp improvement in the second stage.
The effect of the number of layers.
We now study the number of layers with nonlinearities.
To doso, we add nonlinear layers in our Feature Refinement.
We show the results on the CIFAR10 datasetin Figure 7c.
As we can see, adding more nonlinear layers comes with a decrease in performance.
[Figure]
Number of layers ablation 7c: We study the effect of the number of nonlinear fully-connected layers.
More layers do not lead to better performance.
In fact, the more layers we add, the larger is the decrease in performance compared to our originalmodel.
This makes sense considering that by increasing the number of layers, we increase the numberof learnable parameters, and thus we might cause overfitting.
Related Work
[Figure]
All-convolutional networks.
Convolutional neural networks [1, 2] have made a breakthrough inmany computer vision tasks since the pioneering work of [4].
Other works [23, 5] improved overthose architectures, typically by increasing the number of convolutional layers but by keeping thesame architecture: many convolutional and pooling layers followed by a few fully-connected layers.
However, this design choice was questioned in the famous Striving for simplicity [6] paper, where itwas argued that there is no need for fully connected layers.
Fully convolutionized architectures suchas Google-Inception [7] or ResNets [8] followed, significantly outperforming the previous state-ofthe-art.
Some more recent works typically tended to improve the architecture of ResNet architectureseither by enlarging the number of convolutional filters [8] or densely connecting convolutional layers[10].
Probably the best solution was found in the EfficientNet [24], which simultaneously increasesthe image resolution, number of convolutional channels, and number of convolutional layers.
Aconstant of these approaches is that they completely get rid of the fully connected layers.
Contrary tothem, we find out that adding fully-connected layers comes with a massive benefit in performancewhen the number of labeled points is limited.
By doing so, we are able to improve the performanceof several backbones in different learning tasks.
Non-convolutional networks.
A parallel line of research has shown that convolutional neuralnetworks can be replaced either by vision transformers [12, 13] or multi-layer perceptrons [14–16].
The main idea behind these works is to leave the network as much freedom as possible duringthe learning procedure instead of injecting inductive bias into the network.
Indeed, in multi-layerperceptrons, every unit is connected to all the units in both the previous and the next layer, allowingit to use all the information of the proceeding layer as well as pass it into the next one.
Works like
MLPMixer [14] and others have shown promising results in several vision tasks.
Still, they come witha series of limitations, such as the need for pre-training on very large datasets and requiring a massiveamount of computational resources.
Our work is similar to MLPMixer [14] in the sense that we also leverage the power of multi-layer perceptrons to solve different vision tasks.
However, unlike them,we do not get rid of the convolutional layers, but we advocate for hybrid networks.
Our networks donot need pre-training and massive computational resources, while it manages to significantly improvestate-of-the-art results when trained with a limited number of labeled points.
Knowledge distillation.
With the advancements in computational power, larger neural networks arepossible to train.
However, effective neural networks are required during inference in many practicalapplications, especially mobile or embedded applications.
This requirement motivates the field of
Knowledge Distillation (KD).
The goal of KD is to reduce the computational demand of a trainednetwork’s inference by maintaining its performance.
This could be in the form of offline distillation,where one network is used as a teacher to distill its knowledge into another smaller network, thestudent [25].
However, online distillation has gained more attention lately, thanks to its simplerpipeline [26].
In such a setup, the student and teacher models are trained together end-to-end.
Formore details on this field, we refer to the work of Gou et al.
[26] The published online distillationmethodologies are mainly specifically applied for ensemble distillation [27].
On the contrary, wepropose an effective yet general online distillation method.
We attach a separate head during thetraining (the Feature Refiner) while using a gating mechanism that blocks the gradients coming fromthe original softmax layer of the network.
On inference, we can simply drop the Feature Refiner headwithout noticeable performance drop, thus, using no extra parameters.
Conclusion
In this paper, we question the long-believed idea that convolutional neural networks do not needfully-connected layers.
Perhaps surprisingly, we show that using hybrid networks that use bothconvolutional high features and interrelations coming from fully-connected layers improves thegeneralization performance in the low-data regime, without any drawback in the high-data regime.
Toshow a fair comparison, we also introduce a gating mechanism that allows distilling the knowledgefrom our extra head with the added FC layers to the baseline network.
We show that our approachis model agnostic by evaluating different backbone networks.
We show in our experiments that ourmethod yields a significant improvement over the state-of-the-art in supervised and active learning, ina wide range of standard classification datasets without an increase in the number of used parameters.
Broader Impact / Limitations
A shortcoming of our method is that we cannot improve the networks that already contain fullyconnected layers, such as the VGG Network [5].
In the supplementary material, we do an experimentshowing that our Feature Refiner cannot further improve VGG Network’s accuracy.
However, wewere able to decrease the number of learnable parameters by 67% .
Another deficiency of our paper is the lack of a theoretical explanation.
Although the empiricalresults underline our methodology, we could not find any mathematical proof for it.
Our speculativeexplanation is that if the network has only a single final layer, then the backbone’s feature space mustbe linearly separable.
Our intuition is that in the case of a low amount of data, the convolutionallayers do not get enough supervision to find the right local reasoning, which limits its flexibility.
Having additional final fully-connected layers gives global supervision and increases the flexibility ofthe network to reach better optima.
We note that the research in deep learning has been mostly headby empirical results.
We believe that theoretical explanations are desirable and ultimately needed.
At the same time papers without a clear (or a downright wrong) theoretical explanation have had amassive impact on deep learning, as is the case of the batch-normalization [28] paper.
We hope thatour work can open a new interesting line of research, and inspire other researchers to question the’common knowledge’ in deep learning.
Furthermore, it would be interesting to see if our results canbe generalized in domains where the number of labeled data is very scarce, such as medical imaging.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[Yes]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]