Paper:
Incremental Boosting Convolutional Neural Network for Facial Action Unit Recognition
Recognizing facial action units (AUs) from spontaneous facial expressions is still a challenging problem. Most recently, CNNs have shown promise on facial AU recognition. However, the learned CNNs are often overfitted and do not generalize well to unseen subjects due to limited AU-coded training images. We proposed a novel Incremental Boosting CNN (IB-CNN) to integrate boosting into the CNN via an incremental boosting layer that selects discriminative neurons from the lower layer and is incrementally updated on successive mini-batches. In addition, a novel loss function that accounts for errors from both the incremental boosted classifier and individual weak classifiers was proposed to fine-tune the IB-CNN. Experimental results on four benchmark AU databases have demonstrated that the IB-CNN yields significant improvement over the traditional CNN and the boosting CNN without incremental learning, as well as outperforming the state-of-the-art CNN-based methods in AU recognition. The improvement is more impressive for the AUs that have the lowest frequencies in the databases.

Review:
The paper presents and validates a new method to improve classification of facial action units (AUs) from single frames. The challenges and applications of this problem are discussed. The presented method is based on a traditional convolutional neural network (CNN) with a boosting algorithm on top of fully connected features (B-CNN). The authors suggest an incremental B-CNN (IB-CNN), in which the weights of boosting classifiers are trained in a soft way, better accumulating information from all batches. The theoretical formulations and description of the method are presented. In the experimental part, the classification results for two quite popular datasets are presented, compared with baseline methods (CNN and B-CNN) and with previous works, improvements are noticeable. Additional experiments analyzing properties of the developed method are also presented.
Some answers in the rebuttal appeared useful for me, so I can keep the same scores.  “Technical quality” The overall quality of experiments is good. - Ideally, it would be also great to present results on the CK+ database to compare this submission with [21] (which is very related) and to better compare it to [28]. - Line 249: “Data analysis of the parameter η:” The interpretation and analysis of the result in this experiment is missing. Maybe it is not necessary to present results of this experiment? - It would be interesting to see how many active neurons (i.e., with α_j > 0) were in each case in the experiment corresponding to Fig. 5. Is the proposed IB-CNN more robust to the number of input neurons because the number of active neurons is always the same after training? - For some AUs the results are worse than using a CNN or a B-CNN on both datasets? Do the authors have any explanation? - Also, how the authors chose β (beta) in (2) and (8)?  “Novelty/originality” The core idea of incrementally updating weights of classifiers (Eq. (5)) seems to be novel. However, little is said about the work on “Facial Expression Recognition via a Boosted Deep Belief Network” (ref [21] in the submission), which seems to more related than other works discussed. For instance, definition of h(x_ij , λ_j) in (1) of the submission is analogous to (7) in [21], whereas (2) is analogous to (8) in [21], and (4) is very similar to (9) in [21]. So, [21] should be better credited.  “Potential impact or usefulness” The paper presents results for facial action classification. Experimentally, the proposed method is shown to be useful for this task only. But the method is quite general and can be potentially useful in any task where the number of (positive) training samples is small, i.e. can be applied to other (image, video) datasets. As the authors say in lines 288-289: “The IB-CNN is a general machine learning method and can be adopted by other learning tasks, especially those with limited and insufficient training data”. Ideally, this conclusion should be supported in the paper by testing their method on other databases, not necessary facial expressions, to extend their impact. For instance, most of video datasets (KTH, HMDB51) with human actions have a relatively small number of samples per class. To sum up, the paper could have a greater impact (i.e., oral level) if it were more general (from the experimental perspective), something like “Incremental Boosting Convolutional Neural Network for Limited Training Data”. However, admittedly the methods particularly suited for facial action recognition can be indeed very useful in practice.  “Clarity and presentation” Overall, the paper is well written, except for the following issues which can be fixed in a short time. - There must be more space between paragraphs, headings, tables and figures. So, the authors need to remove something less significant (e.g., "Data analysis of the parameter η") from the final version of the paper. Headings to sections, subsections and paragraphs should have consistent case. Please, consult the NIPS style file. - For Tables 1, 2 three digits after decimal look not good, i.e., consider “30.4±6.5” instead of “0.304±0.065”. Then also replace “F1” with “F1x100”. This will free some space, which can be used to add columns describing what are AU2, 12, etc., and, more interestingly, their frequencies in the datasets. For instance, the authors say in lines 287-288: “Furthermore, the IB-CNN is more effective in recognizing infrequent AUs with insufficient training data” and in lines  279-280: “To deal with insufficient positive samples in a mini-batch.” In Tables 1, 2 it would be great to provide some numbers relating to the frequencies of occurrence to support these statements. - The paragraph starting at line 83 is a little bit confusing. The authors say that “A few approaches developed new objective functions to improve recognition performance...”. Then, in the same paragraph they say that “Hinton et al. [23] utilized a dropout ...”, which is not really related to the objective functions nor to making decision (dropout is not used during inference, so it is still the inner product). Consider polishing this paragraph. It’s probably better to say “A few works developed improved training procedures...”.  - Fig. 1 has description for (a) and (b), but there is no (a) and (b) on the figure.  - The authors are suggested to follow punctuation rules in expressions, which are part of sentences (in fact, all expressions (1)-(10) in the paper miss commas and dots). - Lines 142: “However, the information captured previously, e.g., the weights and thresholds of the active neurons, is discarded for a new batch.” And related line 170: “Compared to the B-CNN, the IB-CNN exploits the information from all mini-batches.” Although, Fig. 2 compares a B-CNN and an IB-CNN, in text these sentences may sound confusing. Could the authors specify more concretely what is discarded and what is not, i.e. which weights and thresholds? Otherwise, it sounds like all information from the previous batch is discarded and it is not clear then how the network can be trained.  - Can the authors solve the issue with references to make them hyperlinked?