M ID -V ISION F EEDBACK FOR C ONVOLUTIONAL N EURAL N ETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
Feedback plays a prominent role in biological vision, where perception is modulated based on agents’ continuous interactions with the world, and evolving expectations and world model.
We introduce a novel mechanism which modulates perception in Convolutional Neural Networks (CNNs) based on high level categorical expectations: Mid-Vision Feedback (MVF).
MVF associates high level contexts with linear transformations.
When a context is ”expected” its associated linear transformation is applied over feature vectors in a mid level of a CNN.
The result is that mid-level network representations are biased towards conformance with high level expectations, improving overall accuracy and contextual consistency.
Additionally, during training mid-level feature vectors are biased through introduction of a loss term which increases the distance between feature vectors associated with different contexts.
MVF is agnostic as to the source of contextual expectations, and can serve as a mechanism for top down integration of symbolic systems with deep vision architectures.
We show the superior performance of
MVF to post-hoc filtering for incorporation of contextual knowledge, and show superior performance of configurations using predicted context (when no context is known a priori) over configurations with no context awareness.
I NTRODUCTION
In most contemporary computer vision architectures information flows in a single direction: from low-level of pixels up to high level abstract concepts (e.g., object categories) - such architectures are termed feed-forward architectures.
In general, each successive layer of the network contains more abstract representations than the previous, and the representational hierarchy mirrors the architectural hierarchy.
It is also possible to introduce top-down connections into the network architecture, introducing high level information into processes involving lower levels of abstraction in a process of feedback .
Feedback plays a primary role in biological vision; in fact, the majority of neural connections in the visual cortex are top-down, rather than bottom-up, connections (Markov et al., 2014).
These topdown connections are thought to convey information of higher level expectation , and neurons of the visual cortex use both higher level expectation as well as lower level visual information in producing their representations.
Expectations in biological systems arise from continuous engagement with the environment.
In Computer Vision, this is reflected in the paradigm of Active Vision (Bajcsy, 1988), where perception is framed as an active problem involving evolving world models.
The task of producing mid-level visual representations from low level input is under-constrainedmany plausible mid-level interpretations may be consistent with input.
To give an intuition for how understanding of context can impact perception of mid-level features consider Figure 1 - characteristics of shrews and kiwi differ, but may be similar enough to be confused without context.
Top-down feedback - from high level context to mid-level visual features - provides a “map” for mid-level processes, constraining it towards high level consistency.
Introduction of contextual knowledge through feedback is superior to post-hoc application of contextual knowledge, e.g. through discarding interpretations (classifications, here) which are not context consistent.
We demonstrate this point empirically.
Interpretations selected after post-hoc filtering for context consistency will still be built upon underconstrained mid-level features.
Furthermore, in contrast to post-hoc filtering, feedback naturally allows for detection of out-of-context objects, as feedback functions through biasing of visual representations rather than filtering. It is valuable for methods to allow for out-of-context detections, even when biasing against them, as out-of-context objects on occasion appear (e g , a tree in an
CNNs have a natural tendency towards decoupled representations - representations with a tendency for feature vector angle correspond to characteristic type (e.g., ”fuzzy”), and for feature vector magnitude corresponds to characteristic variation or degree (Liu et al., 2018) (”very fuzzy” / ”not fuzzy”)
(See Figure 2 in the Appendix for an illustration).
This opens up a couple of possibilities in terms of directly manipulating feature representations: 1) We can differentiate between axes with different associations to high level contexts, 2) we can control magnitudes of characteristics through amplifying and dampening axes associated with those characteristics.
That is, w.r.t.
point #1, as CNNs produce representations which are, to a degree, separated by angle, certain axes will be more associated with some higher level contexts over others.
Also, w.r.t.
point #2, amplifying characteristics associated with a higher level context increases the likelihood of interpreting input as conforming to that context; dampening characteristics associated with that context reduces the likelihood of interpreting input as conforming to that context.
We present a principled method to feedback - Mid-Vision Feedback (MVF), illustrated in
Figure 3 - allowing the biasing of mid-level feature representations in networks such as CNNs towards conformance with high level categorical expectations.
This approach is comprised of two components: 1) linear transforms ( affine transformations ), and 2) orthogonalization bias .
Affine transformations enable direct control over the feature vectors at the injection level - the level into which feedback is being inserted.
If these vectors have been trained with a bias towards orthog- onality w.r.t.
contexts, then this allows for affine transformations to manipulate features associated with context presence or absence with less impact on other features.
The orthogonalization bias is introduced to increase the independence between contexts, so they can be manipulated with less interference. This bias is introduced at the level of the architecture into which feedback is to be introduced (the injection level).
This does not negatively impact the representational power or performance of the base network.
This orthogonality bias is introduced across contexts - e.g., mid-level feature vectors associated with animate contexts can be biased towards orthogonality w.r.t.
mid-level features associated with inanimate contexts.
Due to the resulting greater angular separation between features associated with different contexts, this biasing allows greater control over facets of mid-level representation which are meaningful to higher level contexts.
MVF then functions as follows.
During runtime a high level contextual expectation is associated with input.
This expectation is used in biasing mid-level visual features through use of an affine transformation associated with the context of that expectation.
This selects for characteristics associated with this context.
These affine transformations are better enabled as a consequence of the disentanglement of such characteristics at the injection level, effected through introduction of an orthogonalization bias during training.
Feedback then enables a synergy between high level categorical interpretations and mid-level visual feature representations, bridging the signal-symbol gap in both directions.
This approach to incorporation of context expectations is controlled.
This differs from an approach of connecting the upper level fully connected layers of a network directly to lower level convolutional levels in a scheme which includes neither categorical representations nor biasing w.r.t.
said categorical representations.
in accordance with high level context expectations ( E ).
A contrastive loss ( L O ) is inserted to put pressure on the network to help separate feature representations at the injection levels according to context, better enabling affine manipulation.
During training, samples from different contexts are fed through the network.
The context expectations determine the behavior of L O and the selection of appropriate affine transformations.
MVF both employs feedback from categorical knowledge and is agnostic w.r.t.
the source of that categorical knowledge - i.e., it is not a requirement that context expectations be produced from the same network.
As a consequence of this, our approach enables interfacing feedback with larger symbolic systems - e.g., models of scenes employing graphical models over scene elements and categories.
This top-down synergy across the signal-symbol gap opens up a wide range of applications.
The rest of this paper is structured as follows:
In Section 2 we cover related work; in Section 3 we detail methods; in Section 4 we cover experiments; and, in Section 5 we conclude.
R ELATED W ORK 2. B IOLOGY , F EEDBACK , AND P ARALLELS TO C OMPUTER V ISION
Previous works ( (Markov et al., 2014), (Gilbert & Sigman, 2007), Kreiman & Serre (2020), (Gilbert & Li, 2013), and (Paneri & Gregoriou, 2017)) have explored the importance of feedback connections in biological sensory perception.
Further work ((Liao & Poggio, 2016) and (van Bergen &
Kriegeskorte, 2020)) draw connections between feedback in computer vision architectures and the primate visual cortex.
(Tang et al., 2018) show that feedforward CNNs are not robust to occlusion, unlike in human perception, but that adding recurrence improves occlusion robustness.
(Lotter et al., 2016) introduce PredNet, a network based on predictive coding, and demonstrate benefits on the task of self-supervised frame prediction.
There is good reason to believe that modeling characteristics of biological vision in computer vision architectures will benefit computer vision (e.g., (Medathati et al., 2016)).
For example, (Linsley et al., 2020b) demonstrate a network with top-down connections which aligns with human perception of visual illusions, where feedback aids in prioritizing object boundary contours over simple edge contours.
(Linsley et al., 2020a) shows how recurrent hierarchical feedback model can improve segmentation.
(Konkle & Alvarez, 2020) introduce instance-prototype contrastive learning, and show that self-supervised models can learn representations which are more brain-like than supervised models.
(Li et al., 2021) introduce Contrastive Clustering, showing a benefit to instanceas well as cluster-level contrastive loss in clustering.
(Long et al., 2018) demonstrate large scale organization of the cortex based on mid-level visual features (below the level of object recognition), including those associated with animacy vs. inanimacy .
(Jagadeesh & Gardner, 2022) argue that representations in category selective regions of the visual cortex encode a basis representation for texture, rather than objecthood representations.
(Harrington & Deza, 2021) demonstrate that constraining networks to be robust to adversarial input produces network representations more in-line with human visual perception, and argue for the use of texture summary statistic representations.
2. F EEDBACK IN E XISTING C OMPUTER V ISION M ETHODS
The conventional approach to feedback in computer vision is the use of recurrent connections.
(Caswell et al., 2016) introduce recurrent connections into shallow CNN architecture for image classification.
(Pinheiro & Collobert, 2014) employ recurrency over convolutions for the purpose of enabling lateral information flow in the task of segmentation.
(Zamir et al., 2016) instantiate feedback through an RNN architecture which iteratively refines prediction categories from coarse to specific.
Alternatives to conventional recurrent connections for feedback include (Hu & Ramanan, 2016), which explore convolutions with hierarchical rectified Gaussians to enable top-down as well as bottom-up information flow, and apply them to the task of keypoint localization under occlusion.
Additionally, (Yao et al., 2012) apply a graphical model over scene representations, allowing higher and lower level decisions to influence each other.
M ETHODS
With MVF we seek a feedback mechanism which allows us to directly bias lower level feature representations based on categorical higher level context expectations.
This involves top-down interaction across two levels of abstraction: 1) high level contexts c i ∈ C , 2)
mid-level features f i ∈ F .
The structure of MVF is illustrated in Figure 3, and the loss and training formulation given in Section 3.1.
Context expectations sit at a level of abstraction above the classes of network output, and are used in selecting affine transformations placed above the output of injection levels.
When applied to injection level output, the affine transformations bias injection level feature representations towards conformance with the associated context expectation, as illustrated in Figure 4.
Injection level output is made more amenable to manipulation according to context through introduction of a contrastive loss L O , introducing a bias towards orthogonalization across context.
After feature vector modulation through application of an affine transformation, the characteristics of the context associated with that affine transformation are more prominent. For example, with reference to Figure 1, if it is known that the context is one involving animals, rather than food, then characteristics may be biased towards fur interpretations, rather than kiwi fuzz or pastry surfaces.
Affine transformations are applied over the features of the injection level for the purpose of amplifying or dampening certain characteristics.
This process aligns mid-level representations towards conformance with higher level context expectation.
The affine transformations are made more effective through the disentanglement of characteristics at the injection level produced by the orthogonalizing bias.
During test time the CNN runs as a single stream (without the connection across the streams of multiple samples which the contrastive loss introduces), and a context expectation selects the affine transformation to apply over the feature vectors of the injection level.
This expectation can come from any source - in Section 4 we compare performance across network produced context expectations and ground truth context expectations.
3. L OSS AND T RAINING
Training is broken into two stages, as detailed in Table 1.
In the first, the base network is trained on its own, and features are biased towards orthogonality at the injection levels.
In the second stage, the learning rate for the network parameters is reduced, and affine transformations are added to the injection levels according to the context categories of input samples, initialized to identity matrices with added random noise, and given their own optimizers and learning rate.
In the first stage, gradients backpropogate through the base network only, bypassing the affine transformations; in the second stage gradients pass through both the base network and the affine transformations.
In each stage we employ batches containing equal proportions of samples belonging to each context.
Training is broken into two stages for a few reasons: 1) this division allows the possibility of using pretrained networks and training affine transformations in injection levels with no modification to the base network ( λ = 0 and LR N 2 = 0 ), 2) allows fine-tuning of pretrained networks ( λ > 0 and/or LR N 2 > 0 ), 3) we find that starting training of affine transformations after the feature representations have had a chance to converge helps the affine transformations train - intuitively, the affine transformations have to adapt to less of a moving target.
We allow the network parameters to continue to train after affine transformations have been introduced - at a reduced learning rate - as we find that this benefits performance.
To illustrate this process, consider a batch containing a horse and a desk image, horse belongs to animate while desk belongs to inanimate context.
Both the images are fed in the same batch to the network during training.
When training is in Stage 1, each image passes through the base network, sans affine transformations.
The feature representations of the injection levels for horse and desk are then connected to each other via L O , and biased towards orthogonality with respect to each other.
When training progresses to Stage 2, the L O contrastive connection is removed, and affine transformations are inserted according to context expectation.
Both network and affine transformation parameters are then updated according to gradients that pass through both affine transformations and network parameters.
We find that the angles between mid-level features associated with higher level contexts can be increased to a much greater degree than they would be without this orthogonalizing loss, as measured by cosine similarity, without appreciable negative impacts on performance.
See Figure 7 in the Appendix for an illustration of the extent to which this cosine loss is reduced when introducing this orthogonalizing loss.
[Table]
[Equation]
[Table]
[Equation]
[Equation]
Where
[Equation]
) , IL
[Equation]
Where
[Equation]
Where | S F,Y | is a method hyper-parameter, Y C ( f ) is the context of the sample from which feature vector f was produced, IL ( f ) is the injection level from which f was taken, and U ( A ) is the uniform probability distribution over elements of A .
With L O we wish to separate the angles of features associated with different contexts, in order to better enable manipulation through affine transformations.
This can be seen as an exacerbation of CNN’s natural tendency towards decoupled representations - where feature type has a tendency to group according to feature vector angle - through a structuring of characteristics’ feature vector angles according to higher level context.
We do this through a cosine loss ( A · B ∥
A ∥∥ B ∥ ) applied to the features of the injection level.
As we wish to control the network representation in terms of context expectation, we apply this loss across context.
Figure 5 illustrates the behavior produced by the orthogonalizing bias.
E XPERIMENTS
Here we cover experiments demonstrating the utility of our feedback method.
We perform evaluations over CIFAR100 (Krizhevsky et al., 2009), ImageNet (Deng et al., 2009), and the Caltech
UCSD Birds Data set (Birds) (Wah et al., 2011), all with multiple context splits - these datasets are described in Section B of the Appendix and derived based on the CIFAR-100 superclasses, and the
[Figure]
CIFAR
[Table]
Comparison between base VGG and feedback based on Predicted Feedback (PF) context, where the context fed down the network is first predicted by the same network (accuracies for context prediction are given in Table 4 in the Appendix). The simple CNN was not applied over Birds data as its expected input resolution is small. attribute labels provided in the CUB dataset. We base splits on this information in order to evaluate over standard divisions in the data.
We evaluate our method using both ground truth context expectations ( GT Feedback ), as well as context expectations derived from a network of the same structure as the base network ( Pred Feedback ).
All experiments are conducted using a 6-layer CNN base architecture, a VGG-16 network (Simonyan & Zisserman, 2014), and a Transformer model (Mehta & Rastegari, 2021), with variants including added affine transformations for feedback runs.
The hyperparameters to the modifications made to each of the base architectures for feedback incorporation are described briefly in Section 4.1, and described in detail in the Appendix.
We show confusion matrices for a 10-class context split over CIFAR100 for both ground truth context expectations and network predicted context expectations, in Figures 6a and 6b, respectively.
4. M ODELS
Here we describe the architectures in which we incorporate feedback.
Each model consumes 1 GPU during train and test time.
For all feedback experiments we choose a λ (intermediate loss scaling) of 1 .
(otherwise set to 0 . 0 ).
[Table]
CNN was not applied over Birds data as its expected input resolution is small. (Entries with ”-” to be updated when complete).
Shallow CNN: This model comprises a 6-layer CNN architecture, shown in Appendix, consisting of 3 by 3 shaped kernels, max-pooling applied over every other layer, and dropout ( p = 0 .
375 , p = 0 . 1 ) applied over the penultimate fully connected layer and after each convolution operation, respectively.
The affine transformation is applied after the second to last convolution operation, though we observe high performance inserting the affine transformations anywhere throughout the second half of the architecture.
We train the first stage for roughly 5 million iterations for all splits.
A learning rate of 0 .
001 is chosen for the training of the base network during the first stage, and a learning rate of 5 ∗ 10 − 5 is chosen for the learning rate of the base network during the second stage, whereas the affine transformation learning rate is set to 1 ∗ 10 − 3 .
VGG:
Here we adopt a VGG-16 network with pre-trained weights over ImageNet.
The VGG network consists of 16 layers consisting of convolution and max-pooling operations.
The affine transformation is applied after the eleventh convolution operation, though we observe high performance inserting the affine transformations anywhere throughout the last six layers.
We train the first stage for roughly 1 . 5 million iterations for all splits, until smooth convergence.
A learning rate of 5 ∗ 10 −is chosen for the training of the base network during the first stage, and a learning rate of 2 .
5 ∗ 10 −is chosen for the learning rate of the base network during the second stage, whereas the affine transformation learning rate is set to 2 .
5 ∗ 10 − 3 .
Visual Transformer:
Here we adopt a variant of the Visual Transformer models (Mehta & Rastegari, 2021), a general-purpose vision transformer that outperforms many related visual transformer architectures while being lightweight and easy to train.
The affine transformation is applied immediately after the third to last attention block.
We train the first stage for roughly 2 . 0 million iterations for all splits, until smooth convergence.
A learning rate of 1 ∗ 10 − 3 is chosen for the training of the base network during the first stage, and a learning rate of 1 . 0 ∗ 10 − 6 is chosen for the learning rate of the base network during the second stage, whereas the affine transformation learning rate is set to 1 ∗ 10 − 3 .
4. O RACLE E VALUATION
The ground truth context model ( GT Feedback ) assumes access to the ground truth context belonging to each input sample during training and test time, using contextual knowledge to index into the affine transformations for application over mid-level features as described in Section 3.
Here we evaluate the extent to which our complete framework outperforms 1) a base architecture mirroring that of our framework, and 2) the same base architecture where a hard masking operation using the ground truth contextual knowledge is applied over its class-level outputs.
The GT Masking baseline corresponds to the same base architecture where a hard mask of c g ∈ { 0 , 1 } k is applied over the output of the network, where k corresponds to the number of classes and c g takes on a value of 1 for within-context classes and 0 for out-of-context classes. Results for ground truth context evaluation are shown in Figure 3.
[Figure]
4. P REDICTED C ONTEXT E VALUATION
2 Show performance of base architectures with and without feedback, when context is predicted rather than provided.
Context prediction is performed through the addition of a second logits classification head to the base networks.
This head is trained in conjunction with the object classification head using ground truth context labels.
Performance for context prediction is shown in Table 4 in the Appendix.
4. D ISCUSSION
We evaluate the utility of feedback under two scenarios: 1) where context is known (Table 3), 2) where context is not known (Table 2). This provides insight both on the performance of the feedback mechanism in ideal cases (providing an upper bound on the utility of feedback), as well as realistic cases with imperfect information derived from the same network. We observe consistently positive results in each case. We also compare feedback to a strong alternative of masking out (excluding) predictions not associated with the ground truth context.
Consistent with Figure 6 superior performance of feedback over masking demonstrates that it improves modeling beyond simply removing context inconsistent predictions.
C ONCLUSION
We have presented an argument for the utility of feedback in vision.
Feedback is 1) prominent in biological vision, with the majority of neural connections in the cortex consisting of feedback connections, 2) allows better constraining of under-constrained processes of abstraction, 3) allows for the online adaptation of vision systems towards alignment with high level understanding of the world.
We leverage the fact that CNNs have a tendency towards decoupled representations, exacerbating the separation of mid-level features associated with different higher level contexts.
This allows better direct manipulation of the level at which feedback is introduced, minimizing collateral effects on characteristics not being selected for.
In contrast to post-hoc filtering of interpretations for consistency with context expectations, MVF allows for cross-context detections and produces higher accuracies.
MVF involves a top-down bridging of the signal-symbol gap, making it applicable to a range of applications.
In the future this work will be extended to localization, e.g. object detection or semantic segmentation, as well as used in embodied contexts with an active agent.
A A PPENDIX A BSTRACT
We here present Appendix.
Section B describes datasets; Section D presents the base CNN architecture used in experiments; Section G details tuned parameters; and, Section I presents data splits.
B D ATASETS B. CIFAR
We adopt CIFAR100 for the 1) high-levels of visual ambiguity due to low resolution, 2) existence of several distinct ”superclasses” consisting of a roughly equal number of classes and 3) the crosscontext confusion across classes highly similar in appearance (e g., sharks and dolphins).
We adopt the official training and test split for the CIFAR100 dataset.
Each class contains exactly 500 training images and 100 testing images, with each superclass consisting of 5000 training images and [Table] testing images.
We use the CIFAR100 superclasses in constructing context splits.
Split 1: Vehicles 1 vs. Vehicles 2, Split 2: Household Devices vs. Furniture, Split 3: Aquatic Mammals vs. Fish.
For full class breakdown see Appendix Section I.
B. I MAGE N ET
We adopt ImageNet for several of the aforementioned reasons above, as well as its generality in that it spans 1000 unique classes.
Each class contains variable number of images - we designate 80% of the images in each class for training, but only 2% for testing due to the computational cost incurred by the high number of images and the need for frequent testing.
We employ the following context split over ImageNet, designed to be similar to CIFAR100 splits, the full class breakdown of which is given in Appendix: Split 1: Household Devices vs. Furniture,
Split 2: Aquatic Mammals vs. Fish, Split 3: Vehicles 1 vs. Vehicles 2.
C B ASE CNN
Figure 8 illustrates the base CNN model (apart from VGG and ViT), for which performance is reported in Tables in the main paper. D C ONTEXT P REDICTION
See Table 4 for accuracy on context prediction.
E D ECOUPLED R EPRESENTATIONS
CNNs have a natural tendency towards decoupled representations.
These are representations where characteristics have a tendency to be represented in such a way that feature vector angle corresponds to characteristic type, while feature vector magnitude corresponds to characteristic variation or degree (Liu et al., 2018). F O RTHOGONALIZING L OSS
An illustration of this is given in Figure 2.
G P ARAMETERS
We here list parameters’ tuned values not introduced in the main paper:
Image size: 32 × 32 for CIFAR100, 224 × 224 for ImageNet.
Model input image size: 32 × 32 for 6-layer CNN, 224 × 224 for VGG16.
Images resized using bilinear interpolation.
Size of feature set selected for orthogonalization: 25. 4.
Batch size: 256 (CIFAR100 splits), 64 (ImageNet splits).
Data augmentations: Random rotations (15 degrees), random resized crops, Random horizontal flips.
Feedback Base Model: ADAM’s optimizer, weight decay of 7 . 5 × 10 − 4 for both stages and both models.
Affine Transformation Optimizer: ADAM’s optimizer, affine transformation learning rate of 0 .
001 for second stage training of both models.
Context Model: ResNet18 model with pretrained weights over ImageNet and learning rate of 0.001 using SGD optimizer. H A BLATION
We present ablation in Table
I D ATA S PLITS
We derive context splits based on the superclass structure provided with CIFAR-100 (over both CIFAR-100 and ImageNet), and the attribute ontology provided with the CUB dataset. We base splits on this information in order to evaluate over standard divisions in the data.
[Table] We adopt the Caltech-UCSD-Birds dataset for several of the aformentioned reasons above, in particular for the high cross-context confusion across different species of birds highly similar in appearance.
It consists of 11,788 images with 200 classes corresponding to bird species.
Like the Imagenet dataset, we designate 80% of the dataset for training and 20% for testing.
We employ the following 3 splits over the CUB dataset, grouping images into contexts based on the listed attributes provided with the CUB dataset: Migration behavior (1, 2, 3) 2.
Trophic level (Carnivore, Herbivore, Omnivore) 3. Primary lifestyle (Aerial, Aquatic, Generalist, Insessorial, Terrestrial) I. SPLITCIFAR
[Table]
Full CIFAR100 Split: animate = beaver, dolphin, otter, seal, whale, aquarium fish, flatfish, ray, shark, trout, bear, leopard, lion, tiger, wolf, camel, cattle, chimpanzee, elephant, kangaroo, fox, porcupine, possum, raccoon, skunk, baby, boy, girl, man, woman, crocodile, dinosaur, lizard, snake, turtle, hamster, mouse, rabbit, shrew, squirrel, bee, beetle, butterfly, caterpillar, cockroach, crab, lobster, snail, spider, worm inanimate = orchid, poppy, rose, sunflower, tulip, bottle, bowl, can, cup, plate, apple, mushroom, orange, pear, sweet pepper, clock, keyboard, lamp, telephone, television, bed, chair, couch, table, wardrobe, bridge, castle, house, road, skyscraper, cloud, forest, mountain, plain, sea, maple tree, oak tree, palm tree, pine tree, willow tree, bicycle, bus, motorcycle, pickup truck, train, lawn mower, rocket, streetcar, tank, tractor
[Table]
barracouta
[Table]
bicycle-built-for-two
[Table] school bus [Table] moped
[Table]
tricycle
[Table] bullet train [Table]
passenger car
[Table]
pickup
[Table]