KDSalBox: A toolbox of efﬁcient knowledge-distilled saliency models
Abstract
Dozens of saliency models have been designed over the last few decades, targeted at diverse applications ranging from image compression and retargeting to robot navigation, surveillance, and distractor detection.
Barriers to their use include the different and often incompatible software environments that they rely on, as well as the computational inefﬁciency of older implementations.
For application-purposes models are then frequently chosen based on convenience and efﬁciency, at the expense of optimizing for task performance.
To facilitate the evaluation and selection of saliency models for different applications, we present KDSalBox - a toolbox of 10 knowledge-distilled saliency models.
Using the original model implementations available in their native environments, we produce saliency training data for efﬁcient MobileNet-based architectures, that are identical in their architecture but differ in how they learn to distribute saliency over an image.
The resulting toolbox allows these 10 models to be efﬁciently run, compared, and be practically applied.
Introduction
At the speed at which technology and software libraries change, image processing applications like segmentation, stylization etc., become quickly outdated.
New implementations replace older ones, in some cases re-inventing prior solutions and in others supplanting older algorithms with newer, data-driven ones.
In the case of saliency modeling, the top performing models on saliency benchmarks (Kümmerer et al.
; Yu et al.
(2017)) are deep neural networks, all trained on the same dataset of visual attention data - the only one of its size (Jiang et al.
As a result, the top models all fall prey to similar biases and have similar limitations (Bruce et al.
; Bylinskii et al.
At the same time, older models including the one by Itti et al.
(1998b), based on low-level image features rather than being directly trained on eye movement data fare better for certain applications (Bruce et al.
For these reasons, they continue to serve as comparisons in saliency papers today.
On the other hand, the increasing availability of constrained computing environments (e.g., mobile phones) have led to increased focus on the design of efﬁcient models.
A number of compact architectures have been proposed recently (Sandler et al.
; Tan & Le (2019)).
These models are carefully designed with components such as inverted bottleneck units, depthwise separability and pointwise convolutions with batch normalization.
Key techniques that have been used to train these networks include model compression and knowledge distillation (Caruana et al.
; Hinton et al.
(2015)), where a student (the efﬁcient network) uses the teacher
’s (the complex network) supervision during the training process.
Knowledge distillation of neural networks commonly uses the intermediate activations as hints (Romero et al.
for better transfer learning.
However, the nature of the student-teacher paradigm allows to use the teacher in a black-box manner where the cross-entropy of only the output of the teacher and the student is backpropagated as a signal to update the weights (Hinton et al.
Recent work in saliency has shown that efﬁcient models for saliency prediction can perform on par with the more complex models while being several orders of magnitude faster (Hu & McGuinness (2020); Zhang et al.
The sheer quantity and diversity of the available computational saliency models makes it difﬁcult to evaluate what ideas or mechanisms are generalizable, scalable or better suited for a given application.
To this end, the MIT/Tuebingen Saliency Benchmark (Kümmerer et al.
(2019)) was introduced and has shown to be beneﬁcial for driving progress in saliency research.
The MIT/Tuebingen Saliency Benchmark provides for each model only a coarse summary of performance and a model ranking based on a set of standardized metrics.
However, beyond comparing summary numbers, re-implementing or re-running these models to evaluate them on new datasets and applications often proves difﬁcult due to different (and occasionally deprecated or outdated) technical requirements, environments, and dependencies.
A few saliency frameworks (Kuemmerer et al. (2015), Wloka et al.
(2018)) have recently been proposed to reduce this effort and create a standardized interface to many saliency models at once.
These frameworks not only help researchers explore these models but also facilitate reproducibility.
While these frameworks can facilitate saliency research, continuous maintenance of these frameworks and dependencies is required as the number of models continues to grow.
Wloka et al.
(2018) tackles this issue by providing containerized images for each model.
However, practical technical barriers remain, such as that all Matlab-based models require a licence and are typically not efﬁcient to run.
Given that saliency is often used as a preprocessing step for downstream image processing tasks, the traditional (non-deep) models are often rendered impractical, even though the predictions they make may be better suited for some applications.
Here we present a technical report to supplement the release of our KDSalBox toolbox of saliency models 2 .
Our toolbox is available to the research community to facilitate the evaluation and use of saliency models in different applications, by removing technical barriers. Our toolbox provides a one-stop shop for different saliency models, equivalent in dependencies and compute time.
The KDSalBox
Our toolbox consists of ten knowledge-distilled saliency models that try to mimic as closely as possible their original implementations.
This approach provides a uniﬁed framework for saliency research that is easy to maintain, facilitates the exploration, evaluation and selection of saliency models, and levels the playing ﬁeld for different types of models (e.g., including Matlab-based libraries with bulky wrappers).
All the knowledged-distilled models are identical in their architecture but only differ in how they learn to distribute saliency over an image.
This makes the framework easy to use, and we show later that despite its simplicity, the knowledge-distilled models learn to behave very closely to the original counterparts.
Interface of KDSalBox.
Our toolbox offers a similar interface as Wloka et al.
This includes pre- and postprocessing steps such as center prior, blurring, scaling as well as standardized evaluation (Bylinskii et al.
In addition, due to the simplicity and uniﬁed design of the framework, it also provides the infrastructure to distill the knowledge of new models with minimal effort.
Finally, we also implemented a graphical user interface to plot the saliency maps (see Figure 1) with the goal to facilitate the evaluation and selection of saliency models for different applications.
Ten saliency models.
The models that the toolbox currently includes are the following: AIM (Bruce & Tsotsos (2009)), IKN (Itti et al.
(1998a))
, GBVS (Harel et al.
, BMS (Zhang & Sclaroff (2013)), IMSIG (Hou et al. (2012)), RARE2012 (Riche et al.
, SUN (Zhang et al.
, UniSal (Droste et al.
, SAM (Cornia et al.
, DGII (Kümmerer et al.
The ﬁrst seven models are originally implemented in Matlab and mostly focus on lower level image features.
The remaining three are deep neural networks that capture higher level object-based features (such as understanding of objects, faces, etc.) and perform the best on current saliency benchmarks, where they are tested on predicting eye movements on natural images (Kümmerer et al.
2.1 Knowledge distillation
To perform knowledge distillation, we generated saliency maps from each original model on the
SALICON dataset (Jiang et al.
The generated saliency maps were then used for supervision during the training procedure.
Note that, we used only the generated saliency images, and not the ground truth (eye movements) of the SALICON dataset, as input during training, since our goal was to mimic as closely as possible the original model behaviors, irrespective of similarity to eye movements. The goal is not to train a new model to perform better at the task of eye movement prediction since the target saliency applications may be different.
Model architecture.
Our model design follows the encoder-decoder pattern.
We used the MobileNetV2 (Sandler et al.
(2018)) for the encoder and a simple convolutional neural network (composed of seven convolutional layers with batch normalization) as a decoder.
The MobileNet-based encoder serves as a lightweight backbone for image feature computation, while the decoder generates the ﬁnal saliency map.
Through some pilot testing, we converged on a decoder that was simple and compact while performing well in knowledge distilling the ten different saliency models.
It contains 6 convolutional layers and 5 intermediate bilinear upsampling layers, motivated by the fact that the original MobileNetv2 downsamples the original image (h x w) to a size (h//32 x w//32).
Training procedure.
For our training and evaluation we used the generated saliency images of each model on the SALICON training and evaluation datasets, respectively.
First the encoder was initialized with the pretrained weights on ImageNet.
Then for the ﬁrst three epochs we froze the encoder, and trained the decoder with our generated images.
After three epochs the encoder was unfrozen and the training proceeded normally.
We used a learning rate of 0.1.
During training we used a binary cross entropy loss and the Adam optimizer.
We also experimented with other custom losses (such as a weighted combination of KL-divergence, Correlation Coefﬁcient, etc.)
but we didn’t observe a noticeable improvement.
All our models were trained on a TITAN Xp (12G) GPU.
Evaluation
Metrics.
Following Bylinskii et al.
(2018), we report the distribution-based metrics CC and KL for evaluating saliency models.
We use these metrics and not others like NSS and AUC, because the goal is to compare pairs of distributions to each other (student and teacher saliency maps), rather than evaluating against ground truth ﬁxation locations.
CC measures Pearson’s Correlation Coefﬁcient between two saliency maps as a ratio of their covariance to the product of their variances.
KL is the Kullback-Leibler divergence between two saliency maps, evaluating the loss of information when one of the maps is used to approximate the other.
From their deﬁnitions, CC is a symmetric metric, while KL is not.
Two maps are similar as CC approaches 1 (higher correlation) and KL approaches(lower divergence).
How well have the students learned?
We evaluate the ability of each of the knowledge distilled student models to approximate their teachers - the original saliency models run in their native code bases.
We report the similarity between the saliency maps produced by the teacher and student models on the SALICON validation data of 5000 images, in Table 1.
The trained models perform very similarly compared to their original counterparts.
Interestingly, the performance of the Matlab-based models was easier to reproduce compared to the deep learning models.
We hypothesize that this is due to the limited dataset size during training making it difﬁcult to capture high-level features, as well as the simple model architecture we used. [Table] [Table] [Table]
MoviePosters
[Table] [Figure] Image (CAT [Figure]
Can the students generalize?
Our student models were trained on saliency maps computed on the
SALICON training images with the intention of capturing saliency behavior on a general sample of natural images.
Therefore by design, our student models are intended to mimic the behavior of the original models on natural images, without guarantees about other image types.
To investigate generalizability, in Table 2, we report the similarity of our student models to the original models on all 20 classes from the CAT2000 dataset by Borji & Itti (2015), which include art and pattern images, as well as on the 5 classes of the Imp1K dataset of graphic design classes Fosco et al.
The knowledge-distilled AIM model generalized the best to other image types, performing least similar to the original AIM model on low resolution images in the CAT2000 dataset and infographics from
Imp1K. The student models IKN, GBVS, BMS, IMSIG, and RARE2012 generalized moderately well, where the most difﬁcult classes proved to be pattern, satelite, sketch, low resolution, and line drawing from the CAT2000 dataset, and infographics and mobile UIs from the Imp1K dataset. The three deep learning saliency models, UniSal, SAM, and DGII produced student models that generalized better on some classes than others.
In Figure 2 we show how each model performs for two sample images, a natural image from the SALICON validation set, and a pattern image from the CAT dataset.
For the sample pattern image shown, all knowledge-distilled models output saliency along the edges of the triangles and the circle, whereas their original counterparts (especially the deep neural networks) predict differently, utilizing higher level features for the saliency maps.
On the other hand, the original counterpart of the low level models (such as AIM, SUN, etc.) fail to capture these features as well, which explains why the knowledge-distilled models generalize well in this domain (i.e., reproducing their failures).
In general, we caution against using the knowledge distilled models on images that deviate too much from the natural images they were trained on.
Table 2 can be used to get an approximate sense of how representative a given model from the KDSalBox might be of the original saliency model on applications targeting speciﬁc image types.
Data preprocessing during training.
Models differ in the ranges of their saliency maps (e.g., compare AIM and SAM in Figure 2).
It is difﬁcult to visually compare the suitability of models for an application by ignoring these differences in saliency values, independently of the actual locations the models predict.
For this reason we provide histogram matching as a possible processing step to facilitate apples-to-apples
model comparisons.
For these reasons, we also trained histogram matched model versions, but did not ﬁnd that they improved model performance (more details in Appendix A).
Discussion
Contributions.
The toolbox presented in this work provides an approachable framework to conduct research on saliency with minimal effort.
Our approach of offering a standardized but also very simple interface and environment for all saliency models can ﬁnd use in the research community and in industry.
Users of the toolbox can currently generate saliency images from ten saliency models with great efﬁciency, to evaluate them for application purposes.
To allow the toolbox to continue to evolve, we also provide code to knowledge distill other models using the student architecture and training procedure we followed.
Limitations and Future Work.
In our work, we have used a uniﬁed architecture for all models for simplicity and easier maintainability.
Depending on the application and setting (e.g. on-device deployment) a further reduction in the size or inference time might be needed in practice.
This can be accomplished by other techniques such as optimizing for the memory hierarchy, for special instruction sets, change of datatypes, etc.
Depending on the model, one can also design separate decoders that are more suitable for each model.
This, however, comes at the expense of simplicity and generalizability.
Another current limitation is that our models are trained only on the natural images from the SALICON dataset and as our preliminary generalization experiments show, some models (particularly students of deep neural net based teacher models) struggle more with generalizing to new image types than others.
Extensions of this work call for training on other domains and with larger datasets.
Finally, since all models share the same architecture and reside in the same framework, a natural extension of this work would be to combine these models like building blocks to increase performance on saliency benchmarks and on particular tasks where different models can offer complementary features (combined expertise).
A Histogram Analysis
As in Table
[Table]
we report our results in Table 3 for the case where the students are trained after all saliency maps are matched to the histogram of the ground truth of SALICON dataset.
This makes it possible to compare the models with each other and facilitate the evaluation and selection of saliency models for different applications.
As we can see the trained student models perform similarly as in the case without histogram matching in the SALICON dataset, but illustrate better where the saliency is distributed across the images.
This is however not the case for the Pattern dataset, since the distribution in these images is quite different and reduces the ability of the student models to mimic their original counterparts.
We also show in Figure 3 a sample image for both datasets for the histogram-matched case.
Similar to our analysis in Section 3, the same conclusions apply here as well — i.e. the knowledge-distillation of the deep learning models generalize worse then the lower-level models.
A potential way to improve this gap is to use a separate knowledge-distillation procedure for the deep learning models and use the intermediate layers to guide the training process (Romero et al.
[Figure]