TDMPNet: Prototype Network with Recurrent Top-Down Modulation for Robust Object Classiﬁcation under Partial Occlusion
Abstract.
Despite deep convolutional neural networks’ great success in object classiﬁcation, recent work has shown that they suﬀer from a severe generalization performance drop under occlusion conditions that do not appear in the training data.
Due to the large variability of occluders in terms of shape and appearance, training data can hardly cover all possible occlusion conditions.
However, in practice we expect models to reliably generalize to various novel occlusion conditions, rather than being limited to the training conditions.
In this work, we integrate inductive priors including prototypes, partial matching and top-down modulation into deep neural networks to realize robust object classiﬁcation under novel occlusion conditions, with limited occlusion in training data.
We ﬁrst introduce prototype learning as its regularization encourages compact data clusters for better generalization ability.
Then, a visibility map at the intermediate layer based on feature dictionary and activation scale is estimated for partial matching, whose prior sifts irrelevant information out when comparing features with prototypes.
Further, inspired by the important role of feedback connection in neuroscience for object recognition under occlusion, a structural prior, i.e. top-down modulation, is introduced into convolution layers, purposefully reducing the contamination by occlusion during feature extraction.
Experiment results on partially occluded MNIST, vehicles from the PASCAL3D+ dataset, and vehicles from the cropped COCO dataset demonstrate the improvement under both simulated and real-world novel occlusion conditions, as well as under the transfer of datasets.
Introduction
In recent years, deep convolutional neural networks (DCNNs) have achieved great success in computer vision tasks, like image classiﬁcation [22,19,6] and object detection [18,17].
However, widely used deep learning models are not robust under occlusion conditions, especially when ldocclusion does not appear in the training data [3,25,31,8].
While occlusion conditions in accordance with the training data may be solved by e.g. multi-label classiﬁcation, it is impossible to collect data covering all possible occlusion conditions, and novel occlusion conditions are much tougher to tackle.
Over-ﬁtting on the limited training conditions results in failure of generalization to novel occlusion conditions, which can cause fatal consequences in real applications as shown in accidents of driver-assistant systems [2].
In the real world, unexpected occlusion such as a ﬂying tissue in front of objects, which would look like a white box patch on the captured image, always exists.
Deep networks can be misguided when they have not seen such a scene in the training data.
Humans, on the other hand, are still able to recognize objects under extreme occlusions by unexpected occluders [32].
Therefore, a reliable computer vision model must be robust to novel occlusion other than training conditions.
In the following, occlusion refers to novel occlusion conditions that do not appear in the training data .
A distribution inconsistency between training and testing data in terms of occlusion causes failures of traditional DCNNs at image classiﬁcation.
Because occlusion patterns are highly variable in terms of appearance and shape, including all possible patterns in the training data is impossible, while biased occlusion patterns do not improve the generalization performance in unbiased conditions [8].
Hence, the inconsistency cannot be avoided and data eﬃciency regarding occlusion should be considered.
Therefore, our work focuses on training a model on limited occlusions while being able to generalize to novel occlusion conditions without assumptions on the occlusion patterns.
There are two main challenges.
The ﬁrst is the over-ﬁtting on the training data, which reduces the generalization ability under novel occlusion conditions.
The second is that occluders will contaminate surrounding features during feature extraction.
We introduce partial prototype matching to deal with the ﬁrst problem, and a top-down feedback modulation to tackle the second problem.
In cognitive science, prototype-matching is a popular theory for object recognition.
From mathematical perspective, prototypes can be viewed as cluster centers of points from the same class in an embedding space, and distance performs as the matching function.
Prototype learning after feature extraction is able to deal with over-ﬁtting [20], as it imposes regularization with a nearest neighbor inductive bias to encourage compact data clusters.
Furthermore, diﬀerent prototypes in one class are able to account for large changes in spatial patterns, such as diﬀerent viewpoints for 3D objects [8].
Prototypes have been introduced and integrated into deep network structure in few-shot learning task [20] and for rejection and class-incremental learning [27].
However, their distances are simply euclidean distances, which cannot be used directly in occlusion conditions due to the distortion of features in the occluded regions.
To tackle the problem of prototype matching under occlusion conditions, we introduce partial matching with a visibility map to focus on target object parts as illustrated in Figure 1.
Wang et al.
[24] ﬁrst discovered that semantic part representations for objects can be found from the internal states of trained
DCNNs, based on which Wang et al.
[25] and Zhang et al.
[31] developed semantic part detection methods.
Besides, larger activation scales of internal states are also correlated with objects [30].
Inspired by these works, we employ a ﬁlter with a visibility map on possible target object parts based on internal DCNN states and a dictionary to sift out irrelevant information.
Experiments show the eﬀective functioning of partial matching according to the ﬁlter.
In addition, we propose a top-down feedback modulation with the estimated visibility map (the feedback connection shown in Figure 1) as a structural prior because occluders also contaminate surrounding features during the feature extraction stage.
The feedback modulation helps the bottom layers to ﬁlter occlusion-induced distortions in the feature activations with high-level information, so that areas around the occluders become less distorted.
Our experiments in Section 4 demonstrate the eﬀective contamination reduction.
Our top-down modulation is related with some neuroscience conjectures.
There are several neuroscience evidence show that recurrent and feedback connections play an important role in object recognition when stimuli are partially occluded [5,15,21,16].
The main conjectures include that the recurrence ﬁlls missing data and that it sharpens certain representations by attention reﬁnement [14].
Here we assume that top-down connection could be a neural modulation to ﬁlter occlusion-caused anomalous activations.
There are also other techniques that implicitly encourage robustness under occlusion, like cutout regularization [1].
Our model does not conﬂict with them and can be further combined with these techniques to improve the robustness under novel occlusion conditions.
In summary, this paper makes the following contributions:
– We introduce partial prototype matching with a visibility map based on a feature dictionary into deep neural networks for robust object classiﬁcation under novel occlusion, with limited occlusion in training data.
The prototypes and the visibility map are integrated into a neural network and can be trained end-to-end.
– We further propose a top-down feedback modulation in convolution layers.
It imitates the neurological modulation from higher cortex to lower cortex and serves as a structural inductive prior.
Experiments show that the feedback eﬀectively reduce the contamination of occlusion during feature extraction.
– Extensive experiments on PASCAL3D+, MNIST, and COCO demonstrate that the proposed model signiﬁcantly improves the robustness of DCNNs under both simulated and real novel occlusion conditions, as well as under the transfer of datasets.
Furthermore, our model can be combined with regularization methods for occlusion-robustness to improve the performance.
Related Work
Object classiﬁcation under partial occlusion .
Fawzi and Frossard [3] have shown that DCNNs are not robust to partial occlusion when inputs are masked out by patches.
Devries and Taylor [1] and Yun et al.
[29] proposed regularization methods, e.g. cutout, by masking out patches from the images during training, which can improve robustness under occlusion to some extent.
Kortylewski et al.
[8] proposed dictionary-based Compositional Model.
Their model is composed of a traditional DCNN and a compositional model based on the features extracted by DCNN.
At runtime, the input is ﬁrst classiﬁed by the DCNN, and will turn to compositional model only when the prediction uncertainty exceeds a threshold, because compositional models are less discriminative than DCNNs.
Their model is not end-to-end, does not consider contamination of occlusion during feature extraction and requires a model of occluders.
Kortylewski et al.
[7] further extended this model to be end-to-end. Diﬀerently, our proposed model follows the deep network architecture, reduces inﬂuence of occlusions both during and after feature extraction, and is generalizable to novel occlusion conditions.
Prototype learning in deep networks .
Prototype learning is a classical method in pattern recognition.
After the rise of deep neural networks, Yang et al.
[27] replace the traditional hand-designed features with features extracted by convolutional neural networks in prototype learning and integrate it into deep networks for both high accuracy and robust pattern classiﬁcation.
Prototypes are also introduced in few-shot and zero-shot learning as part of metric learning [23,20].
Nevertheless, all these works use basic measures like euclidean or cosine distance in prototype matching, which is not suitable for occlusion conditions.
We introduce a ﬁlter focusing on target object parts to extend prototype matching to occlusion conditions.
Object part representation inside DCNNs .
Wang et al.
[24] found that by clustering feature vectors at diﬀerent positions from the intermediate layer of a pre-trained deep neural network, e.g. pool-4 layer in VGG, the patterns of some cluster centers are able to reﬂect speciﬁc object parts.
Wang et al.
[25] and
Zhang et al.
[31] use it for semantic part detection, and Kortylewski et al.
[8] use it to obtain part components in the compositional model.
Related works also include [11], which added a regularizer to encourage the feature representations of DCNNs to cluster during learning, trying to obtain part representations.
From another perspective, Zhang et al.
[30] tried to encourage each ﬁlter to be a part detector by restricting the activations of each ﬁlter to be independent, and they estimated the part position by the activation scale.
These works demonstrate that object part representation is available inside DCNNs, and activation scale contains information.
Based on these ideas, we obtain a ﬁlter with a visibility map for partial prototype matching under occlusion by ﬁnding possible target object parts with their representations and activation scales, and sifting out other irrelevant information.
Feedback connections in deep networks .
Despite top-down feedback connection is an ubiquitous structure in biological vision systems, it is not used in typical feed-forward DCNNs.
Nayebi et al.
[14] has summarized the function conjectures of recurrence in the visual systems and explored possible recurrence structures in CNNs to improve classiﬁcation performance through architecture search.
As for classiﬁcation task under occlusion, Spoerer et al.
[21] explored top-down and lateral connections for digit recognition under occlusion, but their connections are simply convolutional layers without explicit functioning.
As for top-down feedback information, Fu et al.
[4] learned to focus on smaller areas in the image and Li et al.
[10] designed a feedback layer and an emphasis layer.
But all of their feedback layers are composed of fully connected layers, which is not interpretable.
Some DCNN architectures also borrow the top-down feedback idea, like CliqueNet [28].
Diﬀerent from these works, our top-down feedback modulation is composed of explainable visibility map focusing on target object parts and is purposefully for reduction in contamination of occlusion.
Method
Our model is composed of three main parts.
The ﬁrst is prototype learning after feature extraction.
Following it is partial matching based on a ﬁlter focusing on target object parts to extend prototype matching under occlusion.
Finally, top-down modulation is introduced to reduce the contamination of occlusion.
3. Prototype learning
We conduct prototype learning after feature extraction by DCNNs.
Let x ∈
R H
× W 0 × 3 denote the input image, our feature extractor is f θ : R H
× W 0 × 3 →
R H × W × C , which is composed of convolution layer blocks in typical DCNNs.
In contrast to related works [20,27], our feature is a tensor f θ
( x ) ∈
R H
× W × C rather than a vector, in order to maintain spatial information for partial matching in the next section.
Suppose there are N classes for classiﬁcation, we set M prototypes for each class to account for diﬀerences in spatial activation patterns.
Therefore prototypes are a set of tensors p i,j ∈ R H × W × C , where i ∈ { 1 , 2 , ..., N } denotes the class of the prototype, and j ∈ { 1 , 2 , ..., M } represents the index in its class.
For feedforward prediction, the image is classiﬁed to the class of its nearest prototype according to a distance function d : R H × W × C ×
[Equation]
H × W × C
[Equation]
[Equation]
[Equation]
The distance function d can simply be euclidean distance, but experiments in Section 4.2 show that it improves networks slightly due to the contamination of occlusion.
A distance for partial matching will be introduced next section.
For backward update of parameters, we use cross entropy loss based on the distances.
To be speciﬁc, distances between the feature f θ ( x ) and prototypes p i,j produce a probability distribution over classes:
[Equation]
where d k = min
j d ( f θ ( x ) , p k,j ), and γ is a parameter that control the hardness of probability assignment.
We set γ to be learned by network automatically.
Then based on the probability, cross entropy loss is deﬁned:
[Equation]
Further, a prototype loss is added as the regularization of prototype learning:
[Equation]
Diﬀerent from [27], we only consider the nearest prototype when computing distances and probabilities, because our M prototypes in the same class are designed to represent diﬀerent states of objects, such as diﬀerent viewpoints, which may vary a lot in spatial distribution.
We initialize the prototypes by clustering the features of a sub dataset using k-means algorithm [13].
It prevents the degeneration of multiple prototypes to a single prototype.
3. Partial matching under occlusion
The core problem for extending prototype learning directly to occlusion conditions is the matching function.
Since occlusion will contaminate the object feature representation, simple distance between the feature and prototypes won’t be valid enough to do classiﬁcation.
Experiments in Section 4.2 show that pure prototype matching improves deep neural networks slightly.
Focusing on valid parts in features is required.
We employ a ﬁlter with a visibility map based on feature dictionary and activation scale to focus on valid unoccluded parts in features, which enables partial matching.
We learn a feature dictionary in the intermediate layer by clustering
[Figure]
feature vectors over the whole dataset on the feature map, which can represent speciﬁc activation patterns of parts in the images.
Speciﬁcally, feature dictionary is obtained by clustering all normalized vector v k,i,j ∈ R 1 × 1 × C l at position ( i, j ) of the feature map f lθ ( x
k ) ∈
R H l × W l × C l at the intermediate layer l over the dataset { x k } .
Related works [24,8] show that cluster centers are mostly activated by similar parts in the images, most of which are object parts.
More detailed visualization refer to related works [24,25,8].
Based on the feature dictionary { D k } , we compare the similarity between the vectors f lθ ( x ) i,j of the feature at layer l and each component D k : S ( f lθ ( x ) i,j , D k ) =
[Equation]
The higher the maximum similarity over { D k } is, the more likely is the area a target object part.
Therefore, we can sift occlusion out by its low similarity.
However, there are also a few background activation patterns irrelevant to classiﬁcation in the dictionary.
We use the relative scale of activations to ﬁlter them out.
As shown in the Figure 2, the scales of activations in a trained network for most irrelevant background are much lower than objects.
It is probably because deep networks could learn to focus on image parts that contribute to discrimination most.
Considering activation scales is helpful to ﬁlter irrelevant background and maintain most informative signals.
Combining the similarity with the feature dictionary and the activation scale enables us to estimate a visibility map that focuses on unoccluded target object parts.
The formulation for focusing attention at postion ( i, j ) in layer l is:
[Equation]
Since the scale of the activation after the ReLU function could be large, we normalize a li,j .
We use the following linear function with clipping since it preserves proper relative relationship among activation scales:
[Equation]
where a l and a u are lower and upper thresholds that can be dynamically determined according to { a li,j } .
Subsequently, the visibility map is down-sampled to the same spatial scale of f θ ( x ) i,j for partial matching.
We let { A i,j } denote it.
Based on { A i,j } , partial matching between the feature and prototypes is enabled.
Let f θ ( x ) (cid:12) A denote the application of the ﬁlter by scaling vectors f θ ( x ) i,j with A i,j .
A distance for partial matching under occlusion used for Eq.(1), (2) and (4) is deﬁned as:
[Equation]
In this way, we only compare unoccluded target object parts based on the estimated visibility map.
Due to the high-dimension of f θ ( x ) and p i,j , we normalize them on a unit sphere at ﬁrst and compute the euclidean distance after applying the ﬁlter, in order to obtain a valid distance.
We learn the feature dictionary { D k } through clustering.
So similar to prototype learning, we initialize it with clustering result on the pre-trained neural network, and add the clustering loss in the whole loss function during training:
[Equation]
Note that we simply add a normalization layer in the network to normalize d k and ignore the notation in the formula.
3. Top-Down feedback modulation
Our proposed partial matching only sifts out irrelevant feature vectors when comparing features and prototypes.
However, occluders may also contaminate its nearby feature vectors.
We propose to ﬁlter the occlusion-caused anomalous activitions in the lower layers to reduce such contamination and thus obtain cleaner features around the occluder.
Based on the estimated visibility map at a higher layer, a top-down feedback connection is introduced to reduce the contamination of occlusion in lower layers.
Formally, let { A bi,j } denote the up-sampling ﬁlter result of { A li,j } to the same spatial size as the bottom layer b, such as pool-1 layer, and f bθ as the function from input to layer b.
A new activation pattern at layer b can be obtained by applying the ﬁlter to the old activation:
[Equation]
The new activation is again feed-forwarded, as a recurrent procedure.
The recurrence can be carried out for multiple times, gradually reﬁning features to reduce the contamination of occlusion.
The upper threshold in Eq.(6) prevents degeneration of the ﬁlter attention to only one point, and the lower threshold in
Eq.(6) prevents mistaken ﬁltration due to the possible contamination of occlusion from the bottom layer to top layers.
In summary, the overall architecture with our three components is shown in
Figure 1.
Our overall loss function for training is:
[Equation]
[Figure]
Experiments 4. Dataset and settings
We evaluate our model for object classiﬁcation on partially occluded MNIST digits [9], vehicles from the PASCAL3D+ dataset [26], and vehicles from the
COCO dataset [12].
For PASCAL3D+ and MNIST, we simulate novel occlusion, while for COCO, we split non-occlusion images and occlusion images, and test both direct generalization from PASCAL3D+ to novel occlusion in COCO and performance under training on non-occlusion COCO images.
First, to test the generalization ability to novel occlusion conditions in MNIST and PASCAL3D+, we train our model on original images and test under simulation of partial occlusion by masking out patches in the images and ﬁlling them with white boxes, random noise or textures following [8], to imitate unexpected occlusion in front of objects like ﬂying tissues.
In addition, we also use the images provided in the VehicleSemanticPart dataset [25] for the PASCAL3D+ vehicles, where occlusion was simulated by superimposing segmented objects over the target object.
Diﬀerent occlusion levels are also deﬁned corresponding to the percentage of occlusion over objects based on the object segmentation masks provided in the PASCAL3D+ and threshold segmentation of the MNIST digits.
Examples refer to Figure 3.
We use the standard splits for the train and test data.
For the PASCAL3D+ dataset, we follow the setup in [24] and [8], that is the task is to discriminate between 12 objects during training, while at test time the 6 vehicle categories are tested.
Then, we crop the COCO dataset with the bounding box ground truth and divide the occluded and unoccluded images manually, whose categories accord with the setup of PASCAL3D+ above.
We ﬁrst directly employ the model trained on PASCAL3D+ on the occluded images from COCO (occluders do not appear
[Table]
in PASCAL3D+), to evaluate the generalization ability under real novel occlusion and transferred datasets.
Then, we train our model on non-occlusion images from COCO and test on occluded images.
We enrich the training data with PASCAL3D+
due to the insuﬃciency of images.
The occlusion examples are shown in Figure 3.
We utilize convolution layers in a VGG-16 pre-trained on the ImageNet dataset as the feature extraction part.
Prototype learning is conducted on the pool-5 layer.
The visibility map is estimated from the pool-4 layer, and the top-down modulation is imposed on pool-1 layer.
We set feature dictionary components to be 512 for all datasets and use von Mises-Fisher clustering result [8] as the initialization.
Other training details refer to the Supplementary Material.
We compare our model with VGG-16 ﬁnetuned on the datasets, dictionary-based
Compositional Model [8], and human baseline.
We also compare cutout regularization [1] in VGG and our model, which is similar to adding occlusion in the training data as it masks out patches.
The hole number and the length of cutout is set to be 1 and 48.
Other training settings follow the previous settings.
The recurrence number of TDMPNet is three if unspeciﬁed.
4. Results on simulated novel occlusion
Results for classiﬁcation at diﬀerent occlusion levels are shown in Table 1.
They show that DCNNs do not generalize well under synthetic novel occlusion.
TDMPNet signiﬁcantly outperforms VGG in every occlusion conditions and remains about the same accuracy when there’s no occlusion.
Further augmented by cutout regularization, our model achieves signiﬁcantly best results.
Pure prototype learning improves DCNNs slightly.
As shown in the results, direct prototype learning with simple distance function has little improvement.
Though it outperforms VGG in some conditions, the improvements are low compared with follow-up results.
Partial matching plays a crucial role.
As illustrated by the results, partial matching signiﬁcantly improves the performance.
For the mean accuracy over all conditions, it improves 10.3 percent on PASCAL3D+
and 14.5 percent on MNIST compared with VGG.
In the low occlusion level on PASCAL3D+, partial matching achieves the best results even without top-down modulation.
Top-Down modulation works well for severe occlusions.
Top-down recurrence could eﬀectively improve the performance in relatively hard tasks that even human performance drops.
As recurrence times goes up, the features are more pure and therefore performance increases.
A more detailed analysis is in the following section.
With top-down modulation, the ﬁnial mean accuracy outperforms VGG 11 percent on PASCAL3D+ and 17.2 percent on MNIST, reﬂecting its robustness under partial occlusion.
Combination with other techniques can further improve the performance greatly.
As shown in Table 1, cutout regularization can signiﬁcantly boost both VGG and TDMPNet.
It shows that TDMPNet does not conﬂict with other occlusion-robust techniques, and their combination can lead to better results.
TDMPNet with cutout regularization achieves the best result for robustness under synthetic novel occlusion, with a boost of 7.7 percent on PASCAL3D+ and 11.6 percent on MNIST compared with VGG with cutout regularization.
Comparison between TDMPNet and CompDictModel.
Dictionarybased Compositional Model [8] is a model that uses both VGG and a compositional model for classiﬁcation under partial occlusion.
Details refer to Related
Work and the original paper.
Results show that TDMPNet outperforms CompDictModel in most conditions except Level-3 ’o’ condition in PASCAL3D+.
A possible reason is that CompDictModel requires a complex model of occlusion.
Diﬀerently, our model aims at generalization to novel occlusion conditions and makes no assumptions on occlusion.
Another reason is that CompDictModel learn compositional models from the pool-4 layer, which may beneﬁt certain conditions.
Detailed analysis refer to the Supplementary Material.
In addition, our model is end-to-end, with fewer parameters and is simpler in computation compared with CompDictModel.
[Table]
4. Results on real-world novel occlusion
Table 2 are the results of transfer generalization from PASCAL3D+ to novel occlusion in cropped COCO dataset, and the results of training on non-occlusion images from COCO with supplements.
It shows actual improvement of TDMPNet under real-world novel occlusion, even under the transfer of datasets.
As shown in Table 2, under the transfer generalization, TDMPNet outperforms
VGG by 0.26 percent and TDMPNet with cutout demonstrate a more considerable improvement with 1.22 percent accuracy boost, while VGG with cutout do not improve the performance.
When trained on unoccluded images from COCO,
TDMPNet still demonstrate its superiority over VGG, with a boost of 2.18 percent both without cutout and 1.74 percent both with cutout.
Note that VGG with cutout regularization does not generalize its improvement to transferred datasets, while TDMPNet maintains the superiority.
It demonstrates the better generalization ability of TDMPNet under novel real-world occlusion conditions and transferred datasets.
4. Comparison of prototype number
In the previous experiments, we set prototype number as 4 to compare with
CompDictModel.
We further compare diﬀerent prototype numbers and visualize images that are assigned to the same prototype.
Multiple prototypes improve the performance .
As shown in Table 3,prototypes outperform 1 prototype, while 8 prototypes are about the same asprototypes.
It implies that modeling diﬀerent spatial patterns enables prototypes to be more inclusive, and 4 prototypes are enough to account for the spatial variance in the PASCAL3D+ dataset.
Multiple prototypes maintain spatial structures .
As shown in the Supplementary Material, the four prototypes in our model mainly correspond to diﬀerent viewpoints with certain spatial structure.
When there is only one prototype for each class, network simply learns a metric to push all possible activation patterns at a position close with each other, to ensure that the prototype is the closest to all entities.
This may lose spatial structure of objects.
Multiple prototypes are able to tackle such problem eﬀectively.
[-ECCV-
-]
[Table]
[Figure]
4. Analysis of the ﬁlter functioning
[Figure]
The classiﬁcation results demonstrate the importance of the ﬁlter with visibility maps for partial matching.
We illustrate how the two components in the ﬁlter contribute to focusing on informative parts through visualization of visibility maps.
As shown in the Figure 4, the activation scale (4(b)) increases the ﬁltering to the background in 4(a).
Based on it the feature dictionary further increases the ﬁltering on the occluding parrots (4(e)) and enhance attention on several positions (4(d)), resulting in visibility map 4(c).
After a top-down recurrence, the ﬁlter further sifts irrelevant information out and mainly focuses on target object parts (4(f)).
4. Analysis of the top-down modulation eﬀect
We further validate the function of recurrent top-down modulation.
It is designed to reduce contamination of occlusion to its surroundings.
Therefore, we compare [Table] [Figure] the diﬀerences between the pool-4 feature of the clean images and the occluded images before and after top-down recurrence.
Speciﬁcally, let f 0 c , f rc , f
o , f
ro denote the pool-4 feature of the clean image before and after recurrence and the occluded image before and after recurrence respectively, and let m o denote the mask of occlusion area obtained by average down-sampling of the occlusion ground truth.
We compute R c = 1 − sum ( | f ro (cid:12) m o − f rc (cid:12) m o | ) sum ( | f 1 o (cid:12)
m o − f 1 c (cid:12)
m o | ) as the contamination reduction percentage.
Results in Table 4 clearly show that top-down recurrence is capable of reducing contamination in the bottom layer based on the information from the top layer, and nearly the more the recurrence, the more the reduction.
Further, the visualization of diﬀerence reduction is in Figure 5, showing the reduction of occlusion-caused diﬀerence in features surrounding the occluders.
Conclusion
In this work, we integrate inductive priors including prototypes, partial matching, and top-down modulation into deep neural networks for robust object classiﬁcation under novel occlusion conditions, with limited occlusion in training data.
The ﬁlter in partial matching extends prototype matching to occlusion conditions, and the top-down modulation deals with the contamination of occlusion during feature extraction.
Our model signiﬁcantly improves current deep networks, and its combination with other regularization methods leads to better results.
Experiments demonstrate the superiority under both simulated and real novel occlusion conditions and under the transfer of datasets.
Acknowledgements This work was partly supported by ONR N00014-18-12119.