From: NIPS.cc/2018/Workshop/Spatiotemporal/Paper32/Authors
title: thank you for your feedback!
comment: Thank you for your positive comments and suggestions for improvement. We address your comments and questions below.

[discretization] The multimodal time series datasets that we use undergo a preprocessing step where the visual and acoustic features are aligned to the level of words. We decided to use the same preprocessing step as the previously published papers [68] in order to ensure a fair comparison with the baseline models. This turns out to result in video segments that have a reasonable length (20 time steps after truncating and padding) for recurrent models to perform well. It would be interesting to explore extensions of our model to the more fine-grained temporal modeling of visual and acoustic features that are extracted at a higher frequency as compared to the word features. We hypothesize that it would help to relax the discretization assumption since the transcription of words is not perfect and the alignment of audio to language is not perfect either. As a result, informative visual features that occur when there are no words being spoken would be aligned to a speech pause token (SP). Future work could explore extensions of our model to settings where the word boundary discretization assumption is not valid. In this case, the model would have to dynamically decide when to highlight and fusion informative multimodal features towards the prediction task.

    [68] Amir Zadeh, Paul Pu Liang, Soujanya Poria, Prateek Vij, Erik Cambria, and Louis-Philippe Morency. Multi-attention recurrent network for human communication comprehension. AAAI, 2018.

From: ~Ransalu_Senanayake1
comment: Dear authors,

You should now be able to see the reviews. We encourage you to engage in the discussion by commenting on the reviews. Please attempt to clarify the issues highlighted by the reviewers.  

Regards,
Ransalu, Neal, Fabio, and Girish
title: Rebuttal

From: NIPS.cc/2018/Workshop/Spatiotemporal/Paper32/AnonReviewer1
evaluation: The paper introduces a neural architecture for modeling interactions between three modalities of spoken language: language content, visual content and acoustics. The model is based on an RNN where at each step a latent representation for each modality is updated and is dependent on a joint cross-modality latent representation. At each step, the so called fusion process takes place, where the latent representations are iteratively refined through the highlight stage which attends over a subset of the latent representation, and the fuse stage which integrates the new and old representation together. At the end of the step, a summary action takes place, which generates a joint representation for all modalities.
Authors evaluate their model on three tasks related to sentiment analysis, emotion recognition and speaker traits recognition, achieving state of the art results.

The paper is very interesting. It presents a convincing model for an important problem, and experimentally demonstrates its usefulness. The ablation studies are insightful.

A question to the authors: what is an impact of the discretization on the model? We have discrete steps in the RNN (Fig 2). What if we made the buckets longer than a single word? What if we make them smaller/larger? How are we dealing with visual happening when a word does not occur?  In general, would it help to relax a discretization assumption?

From: NIPS.cc/2018/Workshop/Spatiotemporal/Paper32/AnonReviewer1
relevance: 5: very high
impact: 4: high
novelty: 4: high

From: NIPS.cc/2018/Workshop/Spatiotemporal/Paper32/Authors
title: thank you for your feedback!
comment: Thank you for your positive comments and suggestions for improvement. We address your comments and questions below.

[recursive] We agree that multi-stage fusion is a more intuitive name since we technically do not recurse on smaller instances of the problem. We will replace all instances of "recursive" with "multistage" in the paper.

[summarize] We decided to include a summarize module so that it is clear what the final multimodal representation learned from the model is. Otherwise, the user would obtain a sequence of representations from the multiple stages of fuse and highlight. Summarize can be defined flexibly based on the task requirements. For example, one could define summarize as simply taking the last fuse LSTM output, or it could be defined as a weighted combination of all fuse LSTM outputs.

[final FUSE output] In its most general form, summarize should be used on all fusion outputs to encompass all learned intra-model and cross-modal interactions. However, in practice, we used an LSTM network that performed the fusion operations across multiple stages. As a result of using a recurrent model, all fusion outputs are stored in the final fusion LSTM memory and the summarize network can be used on this last stage only. In our implementation, the summarize network and the final prediction network only consisted of 1/2 hidden layers, depending on the dataset. As a result, we hypothesize that performance will not be significantly affected if the last fusion output is directly used for prediction. We are working on this ablation experiment and will update the paper accordingly.

[LSTHM] At time t, LSTHM outputs h_t^l, h_t^v, and h_t^a are used as input to the multistage fusion process, which returns as output a cross-modal representation z_t. z_t is integrated into the LSTHMs at the next time step using the update equations defined in [68]. At time t+1, the process is repeated until final time step T. We were not able to include these due to the space constraints but we will include more details in the appendix of the paper.

      [68] Amir Zadeh, Paul Pu Liang, Soujanya Poria, Prateek Vij, Erik Cambria, and Louis-Philippe Morency. Multi-attention recurrent network for human communication comprehension. AAAI, 2018.

[hyperparameters] Depending on the dataset, the number of LSTM units was 64/128/256 for language and 8/16/32 each for visual and acoustic LSTHMs. The fuse and highlight LSTMs also contain 64/128/256 units. The datasets contain multimodal time series data of 20 steps after multimodal alignment.

[minor comments] Ac refers to multiclass accuracy across c classes, we will clarify this in the paper, along with the remaining fixes with regards to phrasing. Thank you for pointing these out!

[K=3] We believe that having too many stages leads to overfitting on the dataset. Depending on the dataset, the number of stages may vary. We are exploring extensions of this model to scenarios where the number of modalities is large, not known prior to the model design, or dynamically changing.

From: NIPS.cc/2018/Workshop/Spatiotemporal/Paper32/AnonReviewer2
relevance: 5: very high
impact: 4: high
novelty: 3: borderline

From: NIPS.cc/2018/Workshop/Spatiotemporal/Paper32/AnonReviewer2
evaluation: This paper discusses a novel approach to perform multi modal language modelling called Recurrent Multistage Fusion Network (RMFN). The novelty of model lies at the way the problem is approached. Instead of trying to jointly model the multi modal data, a divide-and-conquer approach is taken where the modelling is done in several stages, where at each stage, most important modalities are highlighted (using an attention module) and then fused with the previous fused results. The paper reports results on three different datasets CMU-MOSI (sentiment analysis), IEMO-CAP (emotion recognition), and POM (speaker trait recognition), and show that RMFN outperforms all the other counterparts.

Strengths
-------------
+ The approach is well motivated by taking examples from the field of cognitive neuroscience as well as the limitations of the previous work
+ Sound experimental results to both demonstrate performance of the model, and explain the model

Major comments
-----------------------
The paper mentions the word "recursive stages" several times throughout the paper, but I do not understand why the stages are recursive? Please make it more clear, or refer to them as "multi-stage".

I am not convinced about the purpose of the SUMMARIZE module in the RMFN. For example, how would the model perform if the output of last stage's FUSE module is provided, i.e. s_t^K? From the figure, it looks to me like the SUMMARIZE is just a non-linear transformation on the "last" FUSE output. 

The paper says that, "all" the FUSE representations are connected to the SUMMARIZE. However I do not see this being reflected in Figure 2. The SUMMARIZE is only connected to the final FUSE output.

Is it possible to show the LSTM hybrid model (LSTMH) and how it is connected to the rest of RMFN in Figure 2?

At least important hyperparameters such as number of hidden units in LSTM modules, and the number of unrolled time steps, should be mentioned.

Minor comments
------------------------
Two sentences between line 6-10 can be combined to a single sentence.

Line 21: visual (facial expressions, gestures) and ->  visual (facial expressions, gestures), and

Ac represents the accuracy of the c th class or the accuracy upto the c th class?

Correlation is denoted by r only once in text, in the tables it is referred to as "Corr".

Further thoughts
------------------------
I understand about the space limitations, however encourage the authors to investigate following questions in more detail.

Why does K=3 show the best results? Is it because there are three different modalities? What happens if you increase/decrease the number of modalities?

What would happen if the SUMMARIZE layer is not present, and the last FUSE layer's output is used as z_t?