From: ICLR.cc/2023/Conference/Paper4729/Authors
title: Submission Withdrawn by the Authors

From: ICLR.cc/2023/Conference/Program_Chairs
title: Paper Decision
decision: Reject
metareview:_summary,_strengths_and_weaknesses: This paper aims to estimate emotion (based on valence, arousal) from images, using a Cognitive Contextual Summarization (CCS) model based on One-For-All (OFA) backbone with BERT embeddings, and shows better results than SOTA. The problem is challenging especially when no humans are present in the image.
However the premise of the motivation, the design of the model, and the analysis of the results are not articulated well to convince the reviewers. 
justification_for_why_not_higher_score: The paper attempts a challenging problem but the current content and results are not ready for publication
justification_for_why_not_lower_score: NA
summary_of_AC-reviewer_meeting: Not needed as all reviewers proposed rejection, and no change after discussion

From: ICLR.cc/2023/Conference/Paper4729/Reviewer_W6QT
title: Weak Motivation, Weak Experiments 
comment: *Weak Motivation*

1. If the premise of the article is that of assessing evoked emotions in an observer through a visual stimuli (images), then the title is mis-representing this. Therefore, the title should be in the lines of  ‘Image-based Evoked Emotion…‘.
    
2. 3/4 reviewers are confused about the premise of the motivation, as is apparent from the reviews. The paper has major flaw in this regard.  
    

*Weak Experimental Evidence*

1. The Fig. 1 in supplementary material clearly shows that the model is able to model Valence earlier (around 20 epochs) than Arousal (around 60 epochs). It is unclear where the model should be stopped to train.
    
2. Lack of rigor in evaluating the method across different datasets. The authors argue that their premise is different from that of EMOTIC dataset. However, if the premise is reversed (estimating emotions of persons in an image), it is possible to evaluate the effectiveness of their proposed approach on a similar problem domain across different datasets. This could highlight the usefulness of their method and also probably a better empirical support to their argument of using ICN.  
    
3. IAPS is a dataset which captures an observers reaction (categorized into 3 states - positive, neutral and negative). The authors could show the usefulness of their method on this dataset by reformulating positive-neutral-negative as valence dimesion.
    
4. The argument for using ICN is also weak. It is unclear due to lack of empirical proof as to how ICN is able to better capture the semantics of an image including the relationship between different objects.

Due to above reasons it is not possible to consider the paper in the current form.

From: ICLR.cc/2023/Conference/Paper4729/Reviewer_W6QT
confidence: 5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
summary_of_the_paper: The paper focuses on the problem of recognizing emotions from an image, based on the visual stimuli. Current methods rely on the presence of a person in an images, however, images often do not contain people which makes the task more challenging.

Therefore, the paper suggests using a cognitive contextual summarization (CCS) network with One-For-All (OFA) backbone with BERT embeddings to train an end-end model for emotion dimension (valence, arousal) prediction.  The method outperform the state-of-the art (sota).
strength_and_weaknesses: **Strength**

1. The model outperforms sota.
    

**Weakness**

1. Unclear context of the work: Is it (a) estimating emotions that evokes in an observer via visual stimuli (images in this work), or (b) perceived emotion of the image itself (whether or not a person is present in the image)?
    
    The first 1-2 lines of abstract and introduction seem to hint (a). Whereas the studies mentioned in the first paragraph and the last line of that paragraph seem to hint at (b)
    
2. Premise of using all objects in the image for estimation: The paper mentions that previous methods rely on recognition of all different objects and elements of the image to make a prediction about image emotion. To address this, the paper uses image captioning network (ICN) to extract captions via natural language as a way of summarizing visual information. On what basis is the ICN superior than object detectors in terms of capturing all elements and objects in the image?    
    
3. Too many FC layers?: it is unclear why the CCS model requires 6 FC layers. Also, there is no ablation study to support this choice.
clarity,_quality,_novelty_and_reproducibility: 1. Fig 1 and 2 are quite similar. It is possible to remove one. Rows can have a constant Valence value, and different Arousal value, or vice-versa  
    
2. Novelty is highly limited. (a) Architectural Novelty: The proposed CCS network is a fusion network of OFA Encoder and BERT embeddings -- both of which are previous works. Also, late fusion is a well studied problem; (b) Method Novelty: It is a regression task which is accomplished with a standard way to train a deep network.
    
3. Figures 4-7 are not useful. For a better comparison between BE and CCS model, it would behove to use a box plot in a side-by-side setting to compare for valence and arousal for BE and CCS together. Scatter plots are useful to see the distribution, but here a comparison is required so a box plot would be better.
    
4. Fig.8 is not standardized. The ranges (x and y axis) shown are quite different for each plot making the comparison difficult and unfair. For example in Fig.8, (c) and (d) look like they are cases of model under-fitting vs. over-fitting. Also, (c) shows a max validation error of 1.7, whereas (d) shows errors that start from 6 and decrease to somewhere 1.5. So, the paper’s assertion that CCS is better than BE is unjustified. Similar arguments can be made for (a) vs. (b)
summary_of_the_review: The paper uses pretrained models (OFA, BERT) in a fusion architecture (CCS) for a regression task of estimating emotions via valence and arousal.  

There is a serious lack of motivation for the problem and the context of the problem being targeted is not clear (weakness #1,2).

There is no new knowledge gained in terms of either architecture, training method or the motivation of the work  (weakness #3).

Additionally, the figures need lot of work to be useful. It is, therefore, difficult to justify if the paper will have any kind of impact to further the understanding of image emotion recognition methods.
correctness: 2: Several of the paper’s claims are incorrect or not well-supported.
technical_novelty_and_significance: 1: The contributions are neither significant nor novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: Yes, Other reasons (please specify below)
details_of_ethics_concerns: The last line of the conclusion about Autistic and Dyslexic children seems out-of-the-blue. No where in the paper is there a mention about where or how image emotion estimation methods could be used. Suddenly, just at conclusion the paper mentions that it could be *useful* without any citation of a previous studies based on this claim.   
recommendation: 1: strong reject

From: ICLR.cc/2023/Conference/Paper4729/Reviewer_7Zax
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper presents an approach for image emotion estimation.  The authors use a dataset (BE) of approximately 10K images labeled by annotators for valence and arousal.  The authors combine a OFA encoder with BERT embedding to capture features they claim provides a contextual summarization of the scene.  The solution is an application of relatively well established neural models to an interesting domain.  The authors recognize the important of combing context with the assessment of emotions. 


strength_and_weaknesses: Personally, I don’t think that the justification of why their model is capturing a cognitive contextual summarization is very well justified.  I recognize that an embedding trained to capture caption data does somehow encode some more semantic information than a model just trained on pixels, but to describe that as cognitive and contextual I think is a stretch.  I think there needs to be a much clearer explanation there and or illustration of what it is in the image caption pre-training is being captured.

The related work referenced in the paper is a little weak.  There is quite a lot of work more recently building models that consider context. And it raises the question about why the EMOTIC dataset was not used?  Could authors justify why they did not use these data for training or testing?  I could not tell if the Building Emotional Machines dataset is available to other researchers, but if not, and because the code does not seem to be available it makes it quite hard for this work to be reproduced.  

In Section 4 the authors write “it is clearly seen that the BE model moderately learns valence representations across images. In other words, it is able to predict the valence of the image fairly or in other words the goodness of the linear fit (learning ability) increases across epochs while training and the model performs comparatively less during validation.” However, to me looking at Figure 4 (and 5 for arousal) the picture is far from clear.  Yes, the model fits to the training set, as one would expect; however performance on the validation set does not clearly change from Epoch 1 to 20.  I would agree that the arousal model is even worse than the valence model but neither can be described as good.  Looking at plots 6 and 7 it does seem that with the CCS model performance is a little bit better, but there are still very large errors.  I am not sure the Valence and Arousal MAEs are very helpful for the reader - I think a correlation metric might be more indicative, in which. Case I would estimate the correlations to be in the range 0.3-0.4 which is quite weak.  I would like to see a much deeper discussion of the potential issues associated with such weak results.  It is also slightly surprising to me that the baseline is so weak.  Again, this is not the first work in this domain and previous papers (see the work of Lapedriza et al.) have been able to build models that perform better (albeit on different dataset).  This highlights the problems with only presenting results on one dataset.
 
The results section of the paper is a little limited.  Given that the premise of the model is that it capture contextual information from the caption embeddings, it would be good fro a reader to gain some insight about what exactly that is being learnt via those captions. Quite a few examples are given in Figure 2 but unfortunately they don’t shed much  light on what exactly the model is capturing.  

clarity,_quality,_novelty_and_reproducibility: Overall the paper is clearly written. The novelty is not incredibly high from a learning perspective and I have some concerns about the reproducibility. Specifically, it is unclear to me whether the code or data are available and if not, I don't think it would be possible for someone to replicate this work from the paper alone.  For these reason, combined with the summary above I don't think this paper quite meets the bar for ICLR.  
summary_of_the_review: 
Overall, the paper tackles an interesting topic. However, I think there a few major weaknesses with the way the solution is framed and how the results are presented.  Integrating contextual information into models that interpret emotion is interesting, the ground truth is extremely subjective.  The author only perform experiments on one dataset and the results are not very convincing with regard to how well they capture the valence and around of there images.  In fact the lllistrative figure (Figure 2) has some examples that seem very counter intuitive. Why is an image of a big cat or a baby low arousal or a puppy low valence? 
correctness: 2: Several of the paper’s claims are incorrect or not well-supported.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough

From: ICLR.cc/2023/Conference/Paper4729/Reviewer_XWhe
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper proposes a method for recognizing emotions by identify interactions between elements in an image through captions. The authors build a model based on a pre-trained One-For-All (OFA) backbone and BERT, and validate its performance through experiments on the Building Emotion Machines dataset.
strength_and_weaknesses: Pros
- Suggest an emotion recognition solution that does not depend on the person in the image.
- Complex emotions can be analyzed on a continuous dimension space .

Cons
- The analysis of experimental results is not sufficient enough to support the novelty.

clarity,_quality,_novelty_and_reproducibility: The problem tackled in this paper is clear, but the proposed method is not sufficient in originality and is of low quality. It seems just combining two pre-trained models (OFA, BERT) with a simple network. Also, it is hard to say that the proposed method clearly figures out the problem in terms of experimental results. 
summary_of_the_review: In this paper, the explanation of the fusion process (Sec. 3.3) is insufficient. In addition, it is difficult to clearly understand the overall process, such as the flow of data, because there is no nomenclature for the entire paper. The experimental results in Sec. 4 simply show a quantitative performance difference from the other method. This is insufficient to prove whether the proposed model is capturing context or interaction well from the image.
correctness: 2: Several of the paper’s claims are incorrect or not well-supported.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: Not applicable
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough

From: ICLR.cc/2023/Conference/Paper4729/Reviewer_MNYU
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper works on a practical problem of image emotion estimation. In particular, it focuses on prediction of the elicited valence and arousal of the images. To achieve this, the authors proposed a Cognitive Contextual Summarization (CCS) model based on an One-For-All (OFA) backbone trained on multiple tasks, including image captioning. The ability of the backbone to recognize elements in the image and generate captions helps capture interactions through captions, which are later decoded using BERT for contextual understanding. The authors train and test their framework on BE dataset and report a new benchmark for the image emotion estimation task.
strength_and_weaknesses: Strength: 

The topic of this paper is practical with many potential applications. The prediction of valence and arousal is interesting and helpful.

Authors have performed intensive experiments on BE dataset, and reported detailed quantitative results.

Weakness:

The rationale of the proposed model design is not clear enough. It is not very convincing to me why the authors based the model on the OFA backbone trained on multiple tasks. Have the authors evaluated the trade-off, say, between incorporating image captioning task and computing time and resources?

It is not true that “existing frameworks fail to capture the context or the interaction between multiple elements in the image“. Indeed, most of the current emotion prediction models are based on deep learning and some of these models already have certain ability of learning the context in the image. 

Experiments are performed on one single dataset. It will be more convincing if the authors can train and test their model on other emotion datasets. It will also be better for authors to show some qualitative results (similar to Figs. 1 and 2).

The paper needs proof reading. The English writing and presentation can be further improved. For example, Fig. 1 and Fig. 2 can be combined. 


clarity,_quality,_novelty_and_reproducibility: The paper is of medium clarity, quality and originality. 
summary_of_the_review: As I have stated in the weakness above, the paper can be further improved on its motivation, experiment and presentation. It is a good work but I feel it’s not ready yet for publication in this ICLR conference. 
correctness: 2: Several of the paper’s claims are incorrect or not well-supported.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: Not applicable
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough