From: NIPS.cc/2018/Workshop/IRASL
title: Acceptance Decision
decision: Accept

From: NIPS.cc/2018/Workshop/IRASL/Paper15/Authors
title: thank you for your feedback!
comment: Thank you for your positive comments and suggestions for improvement. We address your comments and questions below.

[Feature extraction and alignment] We performed the same feature extraction and alignment as the previous baselines for fair comparison. This ensures that the improvement in performance results from our proposed cyclic modality translation model rather than preprocessing decisions. For future work, we will further investigate these feature extraction and alignment decisions in more detail.

[Remarks 1,2] Thank you for pointing these out. We will revise our paper accordingly.

[Loss function] Yes, we use a cyclic loss function with forward S -> \hat{T} and backward \hat{T} -> \hat{S} translations. Our intuition is that using cyclic translations: (1) encourages the model to enforce symmetry between the joint representations from source and target modalities, and (2) ensures that the joint representation retains maximal information from all modalities. Our intuition is backed up by our ablation studies in Section 5, Table 3.

[Remark 3, Equation 17] E_{S \leftrightarrows T} represents the final joint representation between S and T. E_{S \rightarrows T} refers to the intermediate representation when translating from S to T, and E_{T \rightarrows S} refers to the intermediate representation when back translating from T to S.

Equation 17 (and first equation in line 156) E_{S \leftrightarrows T} = E_{S \rightarrows T} refers to setting the final joint representation as the intermediate representation from forward translation.

An alternative definition could be E_{S \leftrightarrows T} = [ E_{S \rightarrows T}, E_{T \rightarrows S} ] i.e. the concatenation of both intermediate representations from forward and backward translations, but we found that simply using one of the translated representations was sufficient for good performance. This makes sense since we expect that using cyclic translations encourages the model to enforce symmetry between the joint representations from source and target modalities. As a result, E_{S \rightarrows T} \approx E_{T \rightarrows S} so using the forward representation is sufficient.

From: NIPS.cc/2018/Workshop/IRASL/Paper15/AnonReviewer2
title: Interesting application of cyclic translation to multi-modal sentiment analysis
review: This problem tackles the problem of multi-modal sentiment analysis where modalities are audio, text and videos. They propose a novel Multimodal Cyclic Translation Network (MCTN), a variant of Seq2Seq models which uses cyclic translation loss to train the representation and log-likelihood loss for label prediction. This helps in the prediction stage as it only needs one modality to predict the sentiment. The author also proposes the hierarchal version of MCTN which can incorporate more than two modalities.

Pros:
1. Very well motivated and a well written paper. 
2. Experimental setup and abalation studies appears to be done industriously.
1. Strong experimental results. 

Cons:
1. The method need the extraction of features for different modalities extrinsically and needs alignment of the sequences of features. Each of these steps might incorporate their own errors. I will like to see some performance metrics on each of these steps (for example, in auto-encoder loss for feature extraction).


A few remarks:
1. Grammatical error in line 5. 
2. In line 36, you might want to cite a source that claims that joint representation are sensitive to noise.
3.  In line 156, is there a typo at the beginning equation?. I see bi-directional s and t on both sides of the equation. 
4. Did you use uni-directional loss function in reverse direction (X_T -> X_S) as well in your loss function? If not, why not?
5. Verify if equation 17 is correct. It seems to be inconsistent with line 199. Maybe, I am misunderstanding something.






rating: 4: Top 50% of accepted papers, clear accept
confidence: 2: The reviewer is fairly confident that the evaluation is correct

From: NIPS.cc/2018/Workshop/IRASL/Paper15/Authors
title: thank you for your feedback!
comment: Thank you for your positive comments and suggestions for improvement. We address your comments and questions below.

[Sharing parameters] The model that does not share seq2seq parameters is presented in Figure 4(d), and its result is in Table 3. We also ran the same baseline with the decoded target sequence (instead of the true target sequence) and obtained results of Acc=73.18%, F1=73.24%, MAE=1.011, Corr=0.610 which are all below our results. We will add these to the paper.

[Sarcastic statements] We would like to emphasize that our model does not use only text - the model has been trained to 'hallucinate' the target visual and acoustic modalities during training. This would move the joint representations closer to the likely visual and acoustic representations. Detecting sarcasm in spoken language remains a difficult problem for artificial intelligence: we believe that our work can be extended to work with state-of-the-art multimodal models for sarcasm detection, thereby creating a hybrid model that remains robust to noisy modalities and also extracts useful information from the nonverbal behaviors to detect sarcasm and deception when required.

[Neural Machine Translation] You are right, we will include this reference in our paper.

[Negative log-likelihood] We used mean squared error since our features are real-valued. If one was working with binary or categorical features, the negative log-likelihood should be used as the reconstruction metric. The encoder and decoder can also be flexibly chosen depending on features used.

[Cycle consistency for T_2] The parallel approach would map the source modality X^S to an intermediate representation and then map that intermediate representation to two target modalities  X^{T_1} and X^{T_2} using two parallel decoders separately. We experimented with this baseline with one encoder and two decoders. On the CMU-MOSI dataset, we obtained results of Acc=73.76%, F1=73.83%, MAE=1.022, Corr=0.611, below our results. We are happy to add it in follow-up versions of the paper.

From: NIPS.cc/2018/Workshop/IRASL/Paper15/AnonReviewer3
title: Interesting paper, good results
review: This paper presents the Multimodal Cyclic Translation Network (MCTN) and
evaluates it for multimodal sentiment analysis. With two modalities (source/target),
the source input is mapped to the target with a sequence-to-sequence model.
The decoded output is then back-translated with the same model to ensure
cycle consistency. The authors argue that this process will lead to intermediate
representations that capture as much information as possible from both modalities.
The intermediate representation of the forward model is used to predict sentiment.
A hierarchical variant of MCTN, which handles more than two modalities, is also
introduced.

Strengths:

Results are fairly impressive, especially given that only text is used for
inference. Many other recent baselines are used for comparison.

There is a fairly comprehensive analysis section, answering many questions
about the contributions of different design decisions.

The paper is well-written.

Weaknesses:

The answer to Q4 (sharing parameters) is not really convincing. It would have been
preferable to use the decoded output instead of the true target. Moreover, the
choice of intermediate representation differs between Figure 5(c) and 5(d).

While using only text for evaluation has some advantages, such as robustness,
there may be cases where it is insufficient, especially for sarcastic statements
(e.g. saying "I love this!" while rolling your eyes).

Other remarks and questions:

"Neural Machine Translation with Reconstruction (Tu et al.)" should likely be
cited.

For the translation and cycle consistency loss, why mean squared error? Could
negative log-likelihood be used?

The hierarchical approach doesn't ensure cycle consistency for T_2. It might be
possible to do so with a non-hierarchical, but parallel approach. The decoder
would need to be given the output modality (e.g. with a special token). X^S could
be encoded into an intermediate representation that would be used to predict
both X^{T_1} and X^{T_2} separately.
rating: 4: Top 50% of accepted papers, clear accept
confidence: 2: The reviewer is fairly confident that the evaluation is correct

From: NIPS.cc/2018/Workshop/IRASL/Paper15/Authors
title: thank you for your feedback!
comment: Thank you for your positive comments and suggestions for improvement. We address your comments and questions below.

[comparison with neural machine translation] Our inspiration for this paper was indeed from the huge successes of sequence to sequence models for neural machine translation. To the best of our knowledge, our paper is the first to propose cyclic translations as a method to learn robust joint multimodal representations.

[Pang and Lee, 2002] You are absolutely right, we have added this reference to the paper.

[Segments] To be consistent with previous work, we followed the segments as defined in the original dataset (Zadeh et al., 2016): each segment is a sentence uttered by the speaker. The visual and acoustic features are chunked according to these sentence boundaries.

[Length L] You are right that the input sequences do not need to be padded/truncated to the same length L. We made this assumption to be consistent with previous models for these tasks and datasets. This ensures that the improvement in performance results from our proposed cyclic modality translation, rather than preprocessing decisions.

[Forced Alignment] This preprocessing step was performed to be consistent with previous works. Chen et al., (2017) motivated the use of fine-grained word-level alignment (i.e. aligning visual and acoustic features by computing their average over each word utterance interval) to learn synchronous and asynchronous multimodal interactions. Previous works that average temporal information or ignore multimodal alignment tend to have difficulties capturing temporal multimodal interactions.

From: NIPS.cc/2018/Workshop/IRASL/Paper15/AnonReviewer1
title: Multi-modal cyclic translations for sentiment analysis
review: An excellent paper, well presented and a good read.  That doesn't happen too often.  I don't really work in the multi-modal field so I can't say how original the work is -- hopefully you'll get other reviews for that.  This seems somehow related to the many-to-many translation paradigm except that the states are used for making a prediction (in this case sentiment) -- look at 
https://www.aclweb.org/anthology/Q/Q17/Q17-1024.pdf

A few comments and questions:

Line 72
You reference later work by these authors -- why not the 2002 work?

 Pang, Bo; Lee, Lillian; Vaithyanathan, Shivakumar (2002). "Thumbs up? Sentiment Classification using Machine Learning Techniques". Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP). pp. 79–86.

Line 98
Its unclear what you mean by segments in a short video -- also are segments silence breaks in the audio? Is the text from speech recognition and if so are the sentences segments?

Q: Why must all the input sequences by constrained to length L?  Encoder/Decoder architecture can handle length differences so not sure why this constraint.

"remains fully robust when **missing** other modalities during testing."

Lines 216-217
Forced alignment is performed using P2FA [63] to obtain word utterance times and we align the
visual and acoustic features by computing their average over each word utterance interval.

Not sure at all why this needs to be done -- can you motivate this?  What happens if you don't?

Nice results.



rating: 5: Top 15% of accepted papers, strong accept
confidence: 2: The reviewer is fairly confident that the evaluation is correct