Paper:
Fine-tuning with pre-trained models has achieved exceptional results for many language tasks. In this study, we focused on one such self-attention network model, namely BERT, which has performed well in terms of stacking layers across diverse language-understanding benchmarks. However, in many downstream tasks, information between layers is ignored by BERT for fine-tuning. In addition, although self-attention networks are well-known for their ability to capture global dependencies, room for improvement remains in terms of emphasizing the importance of local contexts. In light of these advantages and disadvantages, this paper proposes SesameBERT, a generalized fine-tuning method that (1) enables the extraction of global information among all layers through Squeeze and Excitation and (2) enriches local information by capturing neighboring contexts via Gaussian blurring. Furthermore, we demonstrated the effectiveness of our approach in the HANS dataset, which is used to determine whether models have adopted shallow heuristics instead of learning underlying generalizations. The experiments revealed that SesameBERT outperformed BERT with respect to GLUE benchmark and the HANS evaluation set.

Review:
Summary:
The paper proposes adding two mechanisms to the BERT architecture for NLU. The first is based on integrating information from all layers of the encoder via a method called Squeeze and Excitation. The second uses Gaussian blurring to encourage information sharing among neighboring words. The proposed method improves modestly on BERT on the GLUE suite of problems. It also substantially improves on BERT with respect to a class of examples that are designed to confound models that learn superficial heuristics based on word occurrence.
I learn toward rejecting this paper. The method shows some performance gains over BERT on some GLUE tasks, but these are fairly small for the most part, and BERT outperforms the proposed method by a similar amount on a similar number of tasks. The strongest result is the HANS "lexical_overlap" case, where the proposed method has a clear advantage. I have no experience with these kinds of NLU models, so I can't say with confidence whether the architectural additions proposed are well-motivated, but to me it feels like there is not a strong justification for adding these particular features to the BERT architecture, and the results do not clearly demonstrate their utility except in the "lexical_overlap" case.
Details / Questions:
* It seems to me that the GLUE results might be within the margin of error. Is it feasible to replicate training with different random seeds to see what the variance in the performance numbers might be? I suspect that a statistical analysis [1] might conclude that BERT and the proposed method are indistinguishable on the GLUE suite.
* Were the proposed architectural additions conceived with the HANS "counterexamples" in mind (i.e. is there a specific reason to think that these types of methods would avoid the "superficial" reasoning that these examples are supposed to reveal)? Were other methods of adding context considered?
* I suggest using the same x-axis scale on the two charts in Figure 3 to avoid confusion about the magnitudes of the differences.
References:
[1] Demšar, J. (2006). Statistical comparisons of classifiers over multiple data sets. Journal of Machine Learning Research, 7(Jan), 1-30.