[Re] Interactive Two-Stream Decoder for Accurate and Fast Saliency Detection
Anonymous Author(s)
AfﬁliationAddress email
Summary This article describes our attempts to reproduce the work presented in the paper - Interactive Two-Stream Decoderfor Accurate and Fast Saliency Detection which published in CVPR2020, as part of the ML Reproducibility Challenge2020.
Our work consists of three parts: (1) evaluation using the provided pre-trained models from the original paper;(2) evaluation using our retrained models; (3) parameters analysis Scope of
Reproducibility
There are two main objectives of our report: (1) evaluation using the provided pre-trained models from the originalpaper; (2) evaluation using our retrained models Methodology
We used publicly available source code provided by the authors. Minor changes were made to the source code in orderto load the model weight properly. The reproducibility experiments followed the protocal as described in the originalpaper. We performed the training on a machine with two GTX2080TI GPUs.
The training and validation took about 5and 7.5 hours for ResNet-based and VGG-based models, respectively.
Results
The reproducibily using authors’ pretrained model was of no success and we recorded the discrepancy of as high as11.4%.
While for the retrained models, the difference was observed to be as high as 8.6%.
Therefore, we carried outstatistical test to further conﬁrm the reproducibility of the investigated work.
We failed to reject the null hypothesis,which implies that there is no signﬁicant difference between the reported results and our results.
What was easy
The pretrained models have been shared by authors.
One can try out the models with little effort as minor changes tothe name of the parameter keys of the provided PyTorch models are expected.
What was difﬁcult
We observed some differences between the reported results.
We spent longer time to look into the issue and also haddiscussion with the authors.
We repeated the experiments several times for each model to ensure the correctness ofour results.
Despite all the challenges, we ﬁnally come to the conclusion that the investigated work is reproducible, as evidenced by the statistical test technical details about the
Introduction
Saliency detection is a mechanism to detecting visually attentive objects or regions in an image.
To better discover thevisually distinctive objects or regions, the saliency detection methods are expected to understand both the global andlocal features Qin et al.
The global features are essential to provide high-level semantic information for identifying salient objects while the local features are exploited for features reﬁnement.
The investigated paper Interactive Two-Stream Decoder for Accurate and Fast Saliency Detection (ITSD) Zhou et al. [2020b] explores the correlationbetween saliency (global) and contour (local) information by introducing interactive connections between the saliencyand contour streams in the decoder.
An adaptive contour loss is also introduced in the saliency stream to weight moreon near boundary pixels of the salient objects, thus improves the network capability to handling the salient objectboundary.
The method was evaluated on six publicly available saliency datasets and encouraging results were reported.
The investigated work is considered lightweight and able to achieve real-time inference speed at about 88 FPS based onour evaluation.
Furthermore, the reported results on six benchmark datasets are also quite encouraging, which achivecomparable performance as compared to several SOTA methods. Our reproduction of work consists of the following: • Reproduce the reported results using the pretrained models as made available by the authors
• Reproduce the reported results by training the models following the exact protocol described in the original paper
• Evaluation of various parameters as reported in the original paper  Methodology
The authors have made the source code associated with the paper publicly available on GitHub as well as links todownload their pretrained VGG-based and ResNet-based models Zhou et al. [2020a]. In order to properly load themodel weights, a minor modiﬁcation is expected to make to the name of the parameter keys in the given PyTorchmodels as the parameter names are inconsistent with the model deﬁnition.
To retrain the ITSD models, we followedthe training protocol as described in the original paper. The VGG and ResNet, which pretrained on ImageNet dataset,were employed as the backbone of the model. Data augmentation such as random ﬂipping, random cropping andscaling was applied on the DUTS-TR dataset with the aim to increase variability in the dataset. The models weretrained on two GTX2080TI GPUs using SGD at the learning rate of 0.01 for the ﬁrst 20k iterations and decayed by afactor of 0.1 for the remaining 5k iterations.
The ITSD employs an encoder-decoder network architecture.
The backbone of the encoder can be either of apretrained VGG or ResNet.
In order to keep the model lightweight, two measures are adopted: (1) the fully-connected(FC) layers in the standard VGG network is truncated with the aim to reduce the model size while there is no suchneed for the ResNet-based encoder since it has no FC layer; (2) channel pooling is applied to each of the encodedfeature maps.
To better recover the saliency map, the decoder of the network employs a two-stream structure, whichcomprises of ﬁve cascaded Feature Correlation Module (FCF) modules.
The FCF provides interactive connection(fusion stream) which allows the respective saliency and contour information to be transmitted across the stream, andtherefore encourages the network to learn the correlation.
To generate the ﬁnal saliency map, all of the intermediatefeatures from the saliency stream are ﬁrst up-sampled to the input image size followed by concatenation operation.
An adaptive contour loss (ACT) which leverages on the contour information is proposed to provide adaptive weightto the saliency stream, which allows the model to pay more attention to those pixels near to the salient object boundary.
3.2 Datasets
Six commonly used publicly available benchmark datasets, namely, DUTS, SOD, PASCAL-S, ECSSD, HKU-IS and DUTS-OMRON were employed in our reproduction work. The number of images for each dataset is summarized in
Table 1.
[Table]
As our primary object was to evaluate the reproducibility of the investigated work, we employed the default parametersreported in the {+original paper, i.e. , input image size = 
 × 288 , λ = 1 , m = 4 , and batch size = 8 , to train the ITSDmodels.+}
In additional, we also carried out further analysis on various parameters as reported in the paper.
3.4 Experimental setup
All the experiments conducted in this work followed the exact protocol as described in the original paper, which wehoped to mitigate the possible differences in some software libraries/packages, in order to reproduce the results assimilarly to the reported values. Furthermore, we used the source code as made publicly available by the authors toperform both the training and the evaluation, which ensure there exists no bias in the implementation.
All the trainingand evaluation were performed on the same machine with two Nvdia GTX2080TI GPUs, except for the inferencespeed evaluation which was done using single GPU.
Our source code is also made publicly available at GitHub. 3.5 Computational requirements The ResNet-based and VGG-based ITSD models had different requirements on the computational expectations. According to our evaluations, the times required to perform the training and validation were around 5 and 7.5 hours,respectively for ResNet-based and VGG-based models.
The overall required time to perform training and validationwas observed to be shorter for the ResNet-based model, which may be attributed to its efﬁcient bottleneck architecture.
Results
In this section, we provide our evaluation results obtained from: (1) the pretrained models as shared by the authors; (2)the retrained ITSD models according to the described protocol and (3) evaluation of various parameters.
We presentedthe evaluation results using two commonly used performance metrics, i.e., F β and MAE .
As can be observed from Table 2 and 3, the evaluation results in terms of F β were close to the reported values, withthe neglectable differences of up to 1%.
However, the degree of differences in terms of MAE was quite noticeable,with the difference as high as 11.4% was observed.
4.2 Performance evaluation using retrained models
Table 4 and 5 summarize our evaluation results using the retrained ITSD models.
It can be observed that the degreeof differences can be as high as 2.4% in F β and 8.6% in MAE.
In order to quantitatively justifying the reproducibilityof the results, we carried out statistical analysis using paired t -test at signiﬁcance level of 0.05 to verify our null
[Table] [Table]
hypothesis if there was no signiﬁcant difference between our evaluation results and the reported results in the article.
The paired t-test statistics value can be calculated as follows:
[Equation]
(1) where ¯ d is the mean differences [Table] s is the standard deviation and n d is the number of observations, i
[Table] e. , number of theemployed benchmark datasets. The p -value obtained from the calculated t -score is given in the Table [Table]
[Table]
[Table] [Table] 4.3 Additional results not present in the original paper In this section
[Table]
we provide further analysis on various parameters that had not been elaborated in the original paperto provide more insights which may be deemed beneﬁt for those working in the ﬁeld.
All the experiments conducted
[Figure] [Figure]
in this section employed the ResNet-based ITSD model, with all the training parameters kept unchanged (see Section3.3) unless speciﬁed otherwise.
4.3.1 Image sizes and inference speed
We have seen different input image sizes were employed for the saliency detection, such as Liu et al. [2019], Zhao et al.[2019], Qin et al.
Ref.
Zhou et al. [2020b] arbitrary chosen an input image size of 288 × 288 which motivatesus to determine the impact of varying input image sizes on the performance of the ITSD model.
We performed theevaluation using the models which trained with input image sizes of 160 × 160 , 192 × 192 , 224 × 224 , 256 × 256 , 288 ×288 , 320 × 320 .
Generally, the performance was better by using larger input image size, as can be observed from theevaluation results as shown in Fig.
1(a) and (b).
In addition, we also provide the evaluation of inference speed withdifferent input image sizes on a single GPU, as can be observed from Fig. 1(c).
The default input image size 288 × 288reported the inference speed ran at around 88 FPS while the inference speed for the input image size of 160 × 160achieved 89 FPS, at the expense of the saliency performance.
The input image size of 288 × 288 emerges to be a betterchoice which balances the saliency performance and the inference speed.
4.3.2 Hard examples factor m
The hard examples factor m in the saliency loss function serves to emphasize the boundary pixels so the model canbetter handling the boundary of the salient regions.
However, the effect of such parameter had not been discussed in theoriginal paper.
Fig.
2 shows our evaluation results of using various models respectively trained with m = { 1 , 2 , 3 , 4 , 5 } .
On average, some marginal gains were observed in both metrics with the hard examples factor m = 2 . [Figure] [Figure]
4.3.3 Dilation and erosion
[Figure] The contour map is obtained by computing the difference between the saliency maps produced by two morphologicaloperations, i.e. , dilation and erosion.
The resultant from such operation is a contour map with the border widthdetermined by the morphological kernel size, which in turn affecting how the model learn about the hard examplepixels near the salient object boundary.
In the original authors implementation, morphological kernel size of 5 × 5 wasused.
In order to study the effect of different morphological kernel sizes, we considered 1 × 1 , 3 × 3 , 5 × 5 , 7 × 7 , 9 × 9in our experiments.
Fig 3 shows samples of contour maps generated using different morphological kernel sizes.
Notethat no contour information is produced by using the morphological kernel size of 1 × 1 and therefore the Eq. ??
willbe solely depend on the P c to weight the saliency loss function.
As can be observed from Fig.
4, the morphologicalkernel size of 3 × 3 can generally provide better performance than the default kernel size, but such improvement areonly marginal. [Figure]
[Figure]
The λ in the total loss function serves to balance between the saliency and contour losses, which was set to λ
= 1in the original paper.
In other words, one can determine how much emphasis to be placed on the contour loss byadjusting the λ .
In our experiments, we performed evaluation for a range of values, i.e., λ = [0 . 0 , 1 . 6] , with a stepsize of 0.2.
Based on the average results from our evaluation, the λ = 0 .
6 had shown to achieve better performance,which suggested that the model is biased towards the saliency information as such information is more crucial, whilecomplemented by the contour information for further performance gain.
The λ = 0 totally suppresses the contour loss,which can be considered as a special case of the total loss function, where only saliency information is considered.
4.3.5 Optimized vs. default parameters
Sections 4.3.1 - 4.3.4 show our evaluations on the various parameters of the ITSD model, and each individual parameterhas been shown to induce certain degree of improvement on the average performance metrics.
As such, we carriedout experiment by using the optimized parameters, as summarized in Table 7.
Fig.
6 presents the average results ofthe comparison between the models trained with optimized and the default parameters, which only marginal gain wasobserved.
Our primary intention to carry out the experiments in Sections 4.3.1 - 4.3.4 was to study the implication
[Figure]
[Figure] λ 0. of each parameter towards the saliency detection performance.
Therefore, each of the experiment conducted onlyevaluated the parameter of concerned while other parameters were kept unchanged in order to better facilitate theanalysis.
One should consider a proper parameter optimization approach such as grid search to exhaustively searchesthrough the parameter spaces to better estimate the best parameters combination.
4.4 Constrained lambda
Although the free parameter, i.e., λ in the total loss function was analyzed in Section 4.3.4, it is still uncertain which ofthe losses contribute more towards the optimization of the network.
In order to further analyze the importance of eachloss term, we imposed constraint to the [-Eq. 
-]toal loss function by considering a weighted sum rule, as given as follows:
[Equation]
By constraining the lambda would enable us to analyze the correlation between the saliency and contour losses, ascan be observed from average performance metrics in Fig.
The best saliency detection performance was achievedwhen the λ = 0 . 5 , which suggested that both the saliency and contour losses contributed equally and complementedeach other.
While the worst performance was reported when the λ = 1 . 0 , which implied that the contour loss aloneappeared to be suboptimal for optimizing the model.
Discussion and conclusion
We had some issues to get the published code working initially but those issues were solved after some discussionwith the authors.
One of the major issues was the inconsistent use of the name of the parameter keys in the given PyTorch models, which resulted in failing to load the model weight.
The issue can be solved by updating the name of [Figure] the parameter keys to be compatible with the deﬁned models.
Despite of all the challenges we faced during the work reproduction, we eventually managed to reproduce the workof the investigated paper, i.e., ITSD.
Our reproduction attempts were mainly comprised of two parts: (1) reproductionusing the pretrained models as shared by the authors; (2) reproduction by training the models.
Our attempt using thepretrained models was of no success to reproduce the exact results as reported in the original paper, as can be observedin Table 2 and 3.
After seeking clariﬁcation from the authors, one of the possible causes of such discrepancy may bedue to the environment settings.
Another possible reason may be due to the best epoch results were reported in thepaper, while the shared models represented the ﬁnal training artifacts.
Our assumption was evidenced by carrying outthe experiments to capture the best epoch results, which we successfully obtained either similar or better performanceas compared to the reported results.
Our second reproduction attempt was to perform the retraining of the ITSDmodels.
Despite differences in the experimental results, we carried out statistical test, i.e., paired t -test, to furtherconﬁrmed the reproducibility of the presented work in the studied paper.
Further to our reproduction attempts, we hadalso carried out experiments to perform evaluation on the various parameters which had not been elaborated in theoriginal paper.
We hope that our reproduction work can be beneﬁcial to those working in the same direction, as wellas serving as a recognition to the contributions of the original authors.