From: NIPS.cc/2018/Workshop/IRASL
title: Acceptance Decision
decision: Accept

From: NIPS.cc/2018/Workshop/IRASL/Paper11/Authors
title: Thank you + replies
comment: Thank you for taking time to read this paper and write comments!

We will update the paper according to your inputs (do more experiments to compare with SOTA + emphasize the interpretability aspect).

From: NIPS.cc/2018/Workshop/IRASL/Paper11/AnonReviewer1
title: The authors propose an extension to the Consensus networks for semi-supervised learning.
review: The authors propose an alternative method to perform semi-supervised learning when the input contains different modalities.
The overall idea consists of learning abstract compressed representation of the different modalities which allow to represent the common semantics between them. This is done in order to perform the main task at hand which is classification given the small amount of labeled data. The abstract compressed representation are learned in an unsupervised way, whereas, the classification task is learned using the labeled data. The results show that the proposed extension outperforms the original model.

The paper is very well written and the mathematical details are very well explained. The authors propose a complex loss where they combine:
1- an adversarial part: in order to learn the abstract compressed representation related to the different modalities. The adversarial loss aims to make the representation of the different modalities indistinguishable by the adversarial discriminator in order to obtain a semantic which is as independent as possible from the nature of the modality that representation is coming from;
2- a classification part: this part is related to the main task at hand we want the network to perform which is the classification.

The authors also propose to use the normalized Jensen-Shannon Divergence as a metric of similarity between for the learned abstract representation coming from different modalities.

The authors might need to compare their method to state-of-the-art semi-supervised learning methods if they want to extend their paper to a conference paper. The paper is also not really relevant to the IRASL workshop. Still the idea, from a machine learning perspective is very appealing and, in my humble opinion, the authors should pursue in this direction.
rating: 4: Top 50% of accepted papers, clear accept
confidence: 1: The reviewer's evaluation is an educated guess

From: NIPS.cc/2018/Workshop/IRASL/Paper11/Authors
title: Thank you + replies
comment: Thank you for taking time to read (multiple) papers and write comments!

The inspiration of dividing features into multiple modalities could trace back to an ancient fable of multiple blind people observing an elephant. Each one might observe (touch) very different components (e.g., large pillar-like legs, long dexterous nose, sharp and hard teeth) but when they communicate and reach a consensus they will agree it is an elephant. 

For modality division, the methods up till now are mostly according to their natural category. In the previous paper (https://arxiv.org/abs/1808.06570) there were some experiments showing their natural category is better than random division, but the natural category really can vary according to different people. Our intuition is that the natural categories contain information encoded in very similar manner.

We will update the paper according to your inputs.

From: NIPS.cc/2018/Workshop/IRASL/Paper11/AnonReviewer3
title: The paper extends an earlier study on Consensus Networks to show how these models can be trained in a semi-supervised manner. The paper also present several experiments to study the behavior of the network components. The paper seems to be technically correct, the references are adequate. The experimental evaluations use quite specific datasets, which is not perfectly convincing to me.
review: The paper extends an earlier study on Consensus Networks to show how these models can be trained in a semi-supervised manner. (Well, the "earlier paper" was submitted to arxiv in August, so perhaps is should be called a parallel paper...)
The main concept is that the "consensus" component of the model requires only a small subset of the data points to be labelled. I have to admit that I was not familiar with the concept of conensus learning or co-training earlier. In spite of this, I understood the basic concept, so the presentation of the paper is clear and the formulas look to be correct.
The paper contains several experiments the demonstrate the role and behaviour of the various components in the model.
The experiments are carried out on two quite special datasets (dementia detection and bank marketing). These experiments really help understand the importance of the components. I had more difficulties getting an intuition of why dividing the features into "modalities" is good  (in fact, the related experiments are in the earlier paper).  I really wonder how crucial the division of the features into modalities is? Would it work if we took a random task, and divided the features into modalities in a totally random manner? Put another way, would the method generalize to any task? To me, the paper would be more convincing if it contained more experiments on more datasets.

Technical remarks:
1. in additional --> in addition
2. Internatilnal --> international
2. Please take care to force the abbreviations in the references to capital (gan, t-sne, etc.)
rating: 4: Top 50% of accepted papers, clear accept
confidence: 1: The reviewer's evaluation is an educated guess

From: NIPS.cc/2018/Workshop/IRASL/Paper11/Authors
comment: Thank you for taking time to read this paper and give inputs!

In addition to proposing a new model, this paper puts a lot of work into interpretability of this multi-modal classification model.
- Ablation studies are carried out towards analyzing the two mechanisms, consensus and classification.
- T-SNE plots illustrate how the modal representations evolve over the course of training.

The multi-modal classification setting is suitable for speeches and languages. The whole multi-modality agreement idea is inspired by how we hear and understand speeches -- what we hear and what the texts actually mean should have the same information. In future speech-based classification tasks, modalities could be (1) acoustic features, (2) semantic features, and (3) more abstract syntactic features, just as what we did to the DementiaBank dataset. 

We will incorporate your comments into the paper and emphasize the aspect of interpretability in speech & languages.
title: Thank you + replies

From: NIPS.cc/2018/Workshop/IRASL/Paper11/AnonReviewer2
title: An extension of Consensus Networks to Transductive Consensus Networks 
review: This paper proposes to merge the semi-supervised ability (Transductive) with the multi-modality representation (Consensus) in a novel Transductive Consensus Network (TCN), that deals with mixed (labelled and unlabelled samples) and multimodal data sets. The work also proposes to investigate the impact of both parts (Consensus and Transductive) by benchmarking and visualizing (T-SNE) 3 different TCN architectures. 

Quality : 
TCN is a novel architecture that can help to solve a very realistic context, where samples are multimodal and mixed (labelled and unlabelled). Also, the proposed model is well introduced.

Clarity :
The paper is easy to read and self-contained. Nonetheless few typos remain such as (l. 60 denoise autoencoders ; Figure 4 the the …)

Originality :
TCN is a combination of well-known techniques and is novel in that way. 

Significance :
The proposed TCN always obtains better results than the fully supervised CN. It is therefore, better. TCN could be easily reused by the community for further explorations. The only problem i see with this work concerns the adequation with the IRASL workshop. Indeed, the work isn’t presented as a solution to « robustness and interpretability in audio speech and language » but more as a generic solution. 

Pros : 
+ Novelty of the TCN 
+ Helps to answer a realistic context (Multimodal + mixed datas)
+ The paper is very easy to read. 
 
Cons :
- Typos
- Relevance with the IRASL workshop



rating: 3: Marginally above acceptance threshold
confidence: 2: The reviewer is fairly confident that the evaluation is correct