pwc-archive/papers-with-abstracts huggingface dataset:
Transforming neural network visual representations to predict human judgments of similarity
Authors: Ioanna Maria Attarian, Brett D Roads, Michael Curtis Mozer
Date: 2020-10-09 00:00:00
Proceeding: NeurIPS Workshop SVRHM 2020 12

Deep-learning vision models have shown intriguing similarities and differences with respect to human vision. We investigate how to bring machine visual representations into better alignment with human representations.  Human representations are often inferred from behavioral evidence such as the selection of an image most similar to a query image. We find that with appropriate linear transformations of deep embeddings, we can improve prediction of human binary choice on  a data set of bird images from 67.8% at baseline to 90.3%. We hypothesized that deep embeddings have redundant, high (4096) dimensional representations; however, reducing the rank of these representations results in a loss of explanatory power. We hypothesized that the dilation transformation of representations explored in past research is too restrictive, and indeed we found that model explanatory power can be significantly improved with a more expressive linear transform. Most surprising and exciting,  we found that, consistent with classic psychological literature, human similarity  judgments are asymmetric: the similarity of X to Y is not necessarily equal to the similarity of Y to X, and allowing models to express  this asymmetry improves explanatory power.
Anatomically Constrained ResNets Exhibit Opponent Receptive Fields; So What?
Authors: Ethan Harris, Daniela Mihai, Jonathon Hare
Date: 2020-10-09 00:00:00
Proceeding: NeurIPS Workshop SVRHM 2020 12

Primate visual systems are well known to exhibit varying degrees of bottlenecks in the early visual pathway. Recent works have shown that the presence of a bottleneck between 'retinal' and 'ventral' parts of artificial models of visual systems, simulating the optic nerve, can cause the emergence of cellular properties that have been observed in primates: namely centre-surround organisation and opponency. To date, however, state-of-the-art convolutional network architectures for classification problems have not incorporated such an early bottleneck. In this paper, we ask what happens if such a bottleneck is added to a ResNet-50 model trained to classify the ImageNet data set. Our experiments show that some of the emergent properties observed in simpler models still appear in these considerably deeper and more complex models, however, there are some notable differences particularly with regard to spectral opponency. The introduction of the bottleneck is experimentally shown to introduce a small but consistent shape bias into the network. Tight bottlenecks are also shown to only have a very slight affect on the top-1 accuracy of the models when trained and tested on ImageNet.
Task-Driven Learning of Contour Integration Responses in a V1 Model
Authors: Salman Khan, Alexander Wong, Bryan P. Tripp
Date: 2020-10-09 00:00:00
Proceeding: NeurIPS Workshop SVRHM 2020 12

Under difficult viewing conditions, the brain's visual system uses a variety of modulatory techniques to augment its core feed-forward signals. Incorporating these into artificial neural networks can potentially improve their robustness. However, before such mechanisms can be recommended, they need to be fully understood. Here, we present a biologically plausible model of one such mechanism, contour integration, embedded in a task-driven artificial neural network. The model is neuroanatomically realistic and all its connections can be mapped onto existing connections in the V1 cortex. We find that the model learns to integrate contours from high-level tasks including those involving natural images. Trained models exhibited several observed neurophysiological and behavioral properties. In contrast, a parameter matched feed-forward control achieved comparable task-level performance but was largely inconsistent with neurophysiological data.
Disentangled Face Representations in Deep Generative Models and the Human Brain
Authors: Paul Soulos, Leyla Isik
Date: 2020-10-09 00:00:00
Proceeding: NeurIPS Workshop SVRHM 2020 12
Tasks: Representation Learning

How does the human brain recognize faces and represent their many features? Despite decades of research, we still lack a thorough understanding of the computations carried out in face-selective regions of the human brain.  Deep networks provide good match to neural data, but lack interpretability. Here we use a new class of deep generative models, disentangled representation learning models, which learn a latent space where each dimension “disentangles” a different interpretable dimension of faces, such as rotation, lighting, or hairstyle.  We show that these disentangled networks are a good encoding model for human fMRI data. We further find that the latent dimensions in these models map onto non-overlapping regions in fMRI data, allowing us to "disentangle" different features such as 3D rotation, skin tone, and facial expression in the human brain.  These methods provide an exciting alternative to standard “black box” deep learning methods, and have the potential to change the way we understand representations of visual processing int he human brain.
How does task structure shape representations in deep neural networks?
Authors: Kushin Mukherjee, Timothy T. Rogers
Date: 2020-10-09 00:00:00
Proceeding: NeurIPS Workshop SVRHM 2020 12
Tasks: Decoder, Object Recognition

While modern deep convolutional neural networks can be trained to perform at human levels of object recognition and learn visual features in the process, humans use vision for a host of tasks beyond object recognition including — drawing, acting, and making propositional statements. To investigate the role of task structure on the learned representations in deep networks we trained separate models to perform two tasks that are simple for humans — imagery and sketching. Both models encoded a bitmap image with the same encoder architecture but used either a deconvolutional decoder for the imagery task or an LSTM sequence decoder for the sketching task. We find that while both models learn to perform their respective tasks well, the sketcher model learns representations that can be better decoded to provide visual information about an input including — shape, location, and semantic category highlighting the importance of output task modality in learning robust visual representations.
Modeling human visual search: A combined Bayesian searcher and saliency map approach for eye movement guidance in natural scenes
Authors: Melanie Sclar, Gaston Bujia, Sebastian Vita, Guillermo Solovey, Juan Esteban Kamienkowski
Date: 2020-10-09 00:00:00
Proceeding: NeurIPS Workshop SVRHM 2020 12

Finding objects is essential for almost any daily-life visual task. Saliency models have been useful to predict fixation locations in natural images, but they provide no information about the time-sequence of fixations. Nowadays, one of the biggest challenges in the field is to go beyond saliency maps to predict a sequence of fixations related to a visual task, such as searching for a given target. Bayesian observer models have been proposed for this task, as they represent visual search as an active sampling process. Nevertheless, they were mostly evaluated on artificial images, and how they adapt to natural images remains largely unexplored.

Here, we propose a unified Bayesian model for visual search guided by saliency maps as prior information. We validated our model with a visual search experiment in natural scenes recording eye movements. We show that, although state-of-the-art saliency maps are good to model bottom-up first impressions in a visual search task, their performance degrades to chance after the first fixations, when top-down task information is critical. Thus, we propose to use them as priors of Bayesian searchers. This approach leads to a behavior very similar to humans for the whole scanpath, both in the percentage of target found as a function of the fixation rank and the scanpath similarity, reproducing the entire sequence of eye movements.
Curvature as an Organizing Principle of Mid-level Visual Representation: A Semantic-preference Mapping Approach
Authors: Shi Pui Donald Li, Michael Bonner
Date: 2020-10-09 00:00:00
Proceeding: NeurIPS Workshop SVRHM 2020 12
Tasks: Object, Preference Mapping

A central challenge in visual neuroscience is understanding the mid-level representations of the ventral stream. We used a novel, data-driven approach (semantic-preference mapping) combined with an image-statistics approach (curvature index) to characterize the mid-level features of category-selective visual regions. First, we fit voxelwise encoding models using a deep convolutional neural network (DCNN) to predict scene-evoked fMRI responses in object-selective and scene-selective regions. We then performed semantic-preference mapping to examine how the responses of these encoding models changed when specific object classes were removed from natural images. This analysis motivated the hypothesis that object-selective cortex model is best predicted by mid-level features with curved contours, while scene-selective cortex model is best predicted by mid-level features with rectilinear contours. We further developed an image-computable model that outputs a summary statistic for the prevalence of curved contours in local image patches, and we used this model to demonstrate the importance of curvature-preferences for linking DCNN representations with the representations of category-selective cortex models. Overall, our findings suggest that curvature is a key property of the mid-level representations that are shared between DCNNs and category-selective cortex models of the ventral visual stream.
Natural Images are More Informative for Interpreting CNN Activations than State-of-the-Art Synthetic Feature Visualizations
Authors: Judy Borowski, Roland Simon Zimmermann, Judith Schepers, Robert Geirhos, Thomas S. A. Wallis, Matthias Bethge, Wieland Brendel
Date: 2020-10-09 00:00:00
Proceeding: NeurIPS Workshop SVRHM 2020 12
Tasks: Informativeness

Feature visualizations such as synthetic maximally activating images are a widely used explanation method to better understand the information processing of convo- lutional neural networks (CNNs). At the same time, there are concerns that these visualizations might not accurately represent CNNs’ inner workings. Here, we measure how much extremely activating images help humans in predicting CNN activations. Using a well-controlled psychophysical paradigm, we compare the informativeness of synthetic images by Olah et al. [45] with a simple baseline visualization, namely natural images that also strongly activate a specific feature map. Given either synthetic or natural reference images, human participants choose which of two query images leads to strong positive activation. The experiment is designed to maximize participants’ performance, and is the first to probe interme- diate instead of final layer representations. We find that synthetic images indeed provide helpful information about feature map activations (82 ± 4% accuracy; chance would be 50%). However, natural images—originally intended to be a baseline—outperform these synthetic images by a wide margin (92 ± 2% accuracy). The superiority of natural images holds across the investigated network and various conditions. Therefore, we argue that visualization methods should improve over this simple baseline.
CNNs efficiently learn long-range dependencies
Authors: Timo Lüddecke, Alexander S Ecker
Date: 2020-10-09 00:00:00
Proceeding: NeurIPS Workshop SVRHM 2020 12

The role of feedback (or recurrent) connections is a fundamental question in neuroscience and machine learning. Recently, two benchmarks [1,2], which require following paths in images, have been proposed as examples where recurrence was considered helpful for efficiently solving them.
In this work, we demonstrate that these tasks can be solved equally well or even better using a single efficient convolutional feed-forward neural network architecture.

We analyze ResNet training regarding model complexity and sample efficiency and show that a narrow, parameter-efficient ResNet performs on par with the recurrent and computationally more complex hCNN and td+hCNN models from previous work on both benchmarks.
Code: https://eckerlab.org/code/cnn-efficient-path-tracing
Brain-inspired predictive coding dynamics improve the robustness of deep neural networks
Authors: Bhavin Choksi, Milad Mozafari, Callum Biggs O'May, B. ADOR, Andrea Alamia, Rufin VanRullen
Date: 2020-10-09 00:00:00
Proceeding: NeurIPS Workshop SVRHM 2020 12
Tasks: image-classification, Image Classification

Deep neural networks excel at image classification, but their performance is far less robust to input perturbations than human perception. In this work we address this shortcoming by incorporating brain-inspired recurrent dynamics in deep convolutional networks. We augment a pretrained feedforward classification model (VGG16 trained on ImageNet) with a “predictive coding” strategy: a framework popular in neuroscience for characterizing cortical function. At each layer of the hierarchical model, generative feedback “predicts” (i.e., reconstructs) the pattern of activity in the previous layer. The reconstruction errors are used to iteratively update the network’s representations across timesteps, and to optimize the network's feedback weights over the natural image dataset--a form of unsupervised training. We demonstrate that this results in a network with improved robustness compared to the corresponding feedforward baseline, not only against various types of noise but also against a suite of adversarial attacks. We propose that most feedforward models could be equipped with these brain-inspired feedback dynamics, thus improving their robustness to input perturbations.
Rethinking supervised learning: insights from biological learning and from calling it by its name
Authors: Alex Hernández-García
Date: 2020-10-09 00:00:00
Proceeding: NeurIPS 2021 12
Tasks: Self-Supervised Learning

The renaissance of artificial neural networks was catalysed by the success of classification models, tagged by the community with the broader term supervised learning. The extraordinary results gave rise to a hype loaded with ambitious promises and overstatements. Soon the community realised that the success owed much to the availability of thousands of labelled examples. And supervised learning went, for many, from glory to shame. Some criticised deep learning as a whole and others proclaimed that the way forward had to be "alternatives" to supervised learning: predictive, unsupervised, semi-supervised and, more recently, self-supervised learning. However, these seem all brand names, rather than actual categories of a theoretically grounded taxonomy. Moreover, the call to banish supervised learning was motivated by the questionable claim that humans learn with little or no supervision. Here, we review insights about learning and supervision in nature, revisit the notion that learning is not possible without supervision and argue that we will make better progress if we just call it by its name.
Sensory complexity and global gain in a DCNN codetermine optimal arousal state
Authors: Lynn Katrina Annika Sörensen, Heleen A. Slagter, H. Steven Scholte, Sander Bohte
Date: 2020-10-09 00:00:00
Proceeding: NeurIPS Workshop SVRHM 2020 12

 Arousal deeply impacts behaviour and sensory processing. Whereas for an easy task, perceptual performance linearly increases with higher arousal levels, for a more challenging task, an inverted-U-shaped relationship has been described. These findings are commonly referred to as the Yerkes-Dodson law (1908). Yet, it remains unclear why perceptual performance decays with high levels of arousal in difficult but not in simple tasks. Based on recent studies linking a global gain change in sensory processing to changes in arousal state, we augmented a deep convolutional neural network with a global gain mechanism to mimic the effects of cortical arousal. With this approach, we show that the Yerkes-Dodson law can be accounted for by this global gain mechanism, acting on a hierarchical sensory system that processes stimuli of varying sensory complexity. By leveraging the full observability of our model, we reconcile conflicting findings from previous studies on sensory processing, by showing that both linear as well inverted-U-shaped gain profiles emerge in the interaction of hierarchical sensory processing and global arousal changes.
Understanding CNNs as a model of the inferior temporal cortex: using mediation analysis to unpack the contribution of perceptual and semantic features in random and trained networks
Authors: Anna Truzzi, Rhodri Cusack
Date: 2020-10-09 00:00:00
Proceeding: NeurIPS Workshop SVRHM 2020 12

Convolutional neural networks (CNNs) trained for visual recognition can predict activity in primate inferior temporal cortex (IT). It was generally accepted that this is because training leads the CNNs to develop tuning to visual features similar to those in the brain. However, recent evidence that untrained random-weight CNNs explain IT variance to a similar magnitude appears inconsistent with this view. 
Since IT contains rich representations of both perceptual and semantic features, here we propose a resolution to this conflict, that random and trained networks capture different aspects of IT activity. Specifically, we hypothesised that random networks capture perceptual aspects of IT, while trained networks capture semantic aspects but not perceptual ones.
We evaluated a trained standard AlexNet and an untrained random network shown to correlate better with the brain, DeepCluster. The ability of the CNNs to predict IT activity patterns and the role played by perceptual and semantic features was evaluated using regression models, multidimensional scaling, and mediation analysis.
The results support the hypothesis and highlight that, whether CNNs are used as models of the brain, or the brain is used to inspire advances in neural networks, it is not enough to know how similar a given model is to the brain: we also need to know why.
The representation of object drawings and sketches in deep convolutional neural networks
Authors: Johannes Singer, Katja Seeliger, Martin N Hebart
Date: 2020-10-09 00:00:00
Proceeding: NeurIPS Workshop SVRHM 2020 12
Tasks: Cultural Vocal Bursts Intensity Prediction, Object

Drawings are universal in human culture and serve as tools to efficiently convey meaning with little visual information. Humans are adept at recognizing even highly abstracted drawings of objects, and their visual system has been shown to respond similarly to different object depictions. Yet, the processing of object drawings in deep convolutional neural networks (CNNs) has yielded conflicting results. While CNNs have been shown to perform poorly on drawings, there is evidence that representations in CNNs are similar for object photographs and drawings. Here, we resolve these disparate findings by probing the generalization ability of a CNN trained on natural object images for a set of photos, drawings and sketches of the same objects, with each depiction representing a different level of abstraction. We demonstrate that despite poor classification performance on drawings and sketches, the network exhibits a similar representational structure across levels of abstraction in intermediate layers which, however, disappears in later layers. Further, we show that a texture bias found in CNNs contributes both to the poor classification performance for drawings and the dissimilar representational structure, specifically in the later layers of the network. By finetuning only those layers on a database of object drawings, we show that features in early and intermediate layers learned on natural object photographs are indeed sufficient for downstream recognition of drawings. Our findings reconcile previous investigations on the generalization ability of CNNs for drawings and reveal both opportunities and limitations of CNNs as models for the representation and recognition of drawings and sketches.
Observer-dependent Collective Behavior For Biologically-inspired Processing Models
Authors: Bradly Alicea, Jesse Parent, Ujjwal Singh
Date: 2020-10-09 00:00:00

The role of observers in computational learning models is rarely discussed but offers a unique perspective on the practice of analysis and holistic interpretation of data. In this paper,  we present an observer-dependent perspective of data acquisition and preprocessing. Observers consist of observers that exhibit environmental relativism, biorealism, and collective behavior. Environmental realism is achieved by introducing a variety of viewpoints to a population of observers, while biorealism provides observers that have an innate component that resembles a genotype. Collectively, observers can enable emergent features in a given dataset, as well as contextual understanding. The individual and collective behavior of observers can serve as a preprocessing layer or pre-trained model for a variety of machines and deep learning models. In considering the collective behavior of our computational observers, we make a number of predictions about their behavior that may facilitate specific applications.
Visual representations derived from multiplicative interactions
Authors: Eric Elmoznino, Michael Bonner
Date: 2020-10-09 00:00:00
Proceeding: NeurIPS Workshop SVRHM 2020 12

Biological sensory systems appear to rely on canonical nonlinear computations that can be readily adapted to a broad range of representational objectives. Here we test the hypothesis that one such computation—multiplicative interaction—is a pervasive nonlinearity that underlies the representational transformations in human vision. We computed local multiplicative interactions of features in several classes of convolutional models and used the resulting representations to predict object-evoked responses in voxelwise models of human fMRI data. We found that multiplicative interactions predicted widespread representations throughout the ventral stream and were competitive with state-of-the-art supervised deep nets. Surprisingly, the performance of multiplicative interactions did not require supervision and could be achieved even with random or hand-engineered convolutional filters. These findings suggest that multiplicative interaction may be a canonical computation for feature transformations in human vision.
Brain-Like Object Recognition Neural Networks are more robustness to common corruptions
Authors: Anonymous
Date: 2020-10-09 00:00:00
Tasks: Object Recognition

Previous work (Schrimpf et al, 2018, Schrimpf et al, 2020) has shown that there exists a correlation between the performance of neural networks in object recognition tasks and its ability to match behavioral and neural recordings. We expanded on this work to ask the question: Does the behavioral and neural recordings are also correlated to the robustness of neural networks to common corruptions (e.g ImageNet-C). We selected several models from the leaderboard in Brain-Score, a platform that hosts neural and behavioral benchmarks for brain-model similarity, and tested their robustness to the corruption from ImageNet-C. We showed that higher brain-score is correlated with lower mean corruption error across models. Particularly, we show a correlation between the V4 and Behavioral datasets and the model's robustness to ImageNet-C. These finds suggest that explicitly modeling/matching data from V4 might be a good strategy for developing robust models to common corruptions.