Hitachi at S em E val-2020 Task 7: Stacking at Scale with Heterogeneous Language Models for Humor Recognition (2020)

Abstract: This paper describes the winning system for SemEval-2020 task 7: Assessing Humor in Edited News Headlines. Our strategy is Stacking at Scale (SaS) with heterogeneous pre-trained language models (PLMs) such as BERT and GPT-2. SaS first performs fine-tuning on numbers of PLMs with various hyperparameters and then applies a powerful stacking ensemble on top of the fine-tuned PLMs. Our experimental results show that SaS outperforms a naive average ensemble, leveraging weaker PLMs as well as high-performing PLMs. Interestingly, the results show that SaS captured non-funny semantics. Consequently, the system was ranked 1st in all subtasks by significant margins compared with other systems.

## introduction the recognition of humor in text has been receiving much attention @xcite . accordingly, semeval-2020 task 7, assessing humor in edited news headlines @xcite , which aims at automatically recognizing humor in hand-edited news headlines, was held with two subtasks: subtask 1, which aims at predicting a funny score for an edited news headline, and subtask 2, which aims at predicting the funnier headline of two given edited headlines. in this paper, we pursue humor recognition with a large-scale stacking ensemble (hereafter stacking at scale or sas), by leveraging pre-trained language models (plms). sas is based on an ensemble method where a meta-estimator is trained to predict labels from the outputs of base models, finding the best combinations of the base models @xcite . hence, there are two steps in sas: (i) fine-tuning numbers of heterogeneous plms, including bert @xcite , gpt-2 @xcite , roberta @xcite , transformer-xl @xcite , xlnet @xcite , and xlm @xcite , with various hyperparameters, obtaining rich and diverse models, and (ii) training a meta-estimator on top of these plms. our experiments, fusing up to 1750 plms in total, indicate that sas successfully leverages weaker plms as well as high-performing plms. consequently, our system is ranked 1st on both subtasks with significant margins to others. interestingly, analyses show that sas learned (relatively) non-funny semantics while still struggling to understand the funniest semantics. to the best of our knowledge, this is the first experiment that involves thousands of diverse of plms, revealing the current strengths and limitations of plms in automatic humor recognition. we also provide useful insights obtained from rich analyses. ## background work related to humor recognition has been done in recent years @xcite @xcite . @xcite introduced a large-scale annotated corpus of sarcasm and provided baseline systems for sarcasm detection. @xcite widely investigated features for automatically detecting irony and humor. semeval-2020 task 7 @xcite aims at automatically detecting humor in hand-edited news headlines and was introduced by @xcite . we worked to solve the problem by utilizing a number of plms with stacking. ## task formalization as we described in the above, subtask 1 aims at predicting a "funny score", a real-value in the range of [0, 3] (0 = "not", 1 = "slightly", 2 = "moderately", 3 = "funny") for an edited headline. we formalized the task as a sentence-pair regression. subtask 2 aims at predicting the funnier headline of two edited headlines originating from the same headline. we take an approach to utilizing the model of subtask 1, that is, estimating the scores of the edited headlines and choosing the one having the higher score. ## fine-tuning pre-trained language model (plm) on sentence-pair regression plm specific pooling tokenization is conducted by a plm-specific tokenizer. we surround the edited tokens with two special marking tokens, "<" and ">." we insert special tokens (e.g., @xcite and @xcite ) if necessary as required for each plm. the implementations are described in detail in section 6.1. ## stacking at scale we further propose large-scaled ensemble, called stacking at scale (sas), based on a two-layer stacking ensemble @xcite , where the first-layer models (i.e., base models) are fine-tuned plms with different hyperparameter sets, and the second-layer model (i.e., meta-estimator) is another regression model. this may select the best combinations of the base models to produce more robust predictions. figure 2 shows a schematic view and the algorithm steps of sas. the key attributes are (i) using heterogeneous plms for base models, (ii) generating diverse hyperparameter sets for the base models, model selection fold2 fold3 humicroedit fold1 humicroedit funlines fold2 fold3 funlines fold3 funlines fold1 fold2 fold1 𝑆 1,1 𝐵𝐸𝑅𝑇 𝑆 1,2 𝐵𝐸𝑅𝑇 𝑆 1,3 𝐵𝐸𝑅𝑇 𝑀 1,1 𝐵𝐸𝑅𝑇 𝑀 1,2 𝐵𝐸𝑅𝑇 𝑀 1,3 𝐵𝐸𝑅𝑇 val-fold predictions fold2 fold3 humicroedit fold1 humicroedit funlines fold2 fold3 funlines fold3 funlines fold1 fold2 fold1 grouping by data points meta estimator 𝑆 𝐵,1 𝑋𝐿𝑀 𝑆 𝐵,2 𝑋𝐿𝑀 𝑆 𝐵,3 𝑋𝐿𝑀 𝑀 𝐵,1 𝑋𝐿𝑀 𝑀 𝐵,2 𝑋𝐿𝑀 𝑀 𝐵,3 𝑋𝐿𝑀 cv averaging cv averaging step 1: divide humicroedit into 𝑘 folds for τ in plms: step 2: initialize hyparparamoptimizer for 𝑖 = 1 … b: step 3: get a hyperparameter set suggestion 𝐻 𝑖 from hyparparamoptimizer step 4: k-fold cross-validation: for 𝑓 = 1 … 𝑘: 𝑀 𝑖,𝑓 𝜏 ← fine-tuned plm with 𝐻 𝑖 on non 𝑓-th (i.e. training) folds plus funlines 𝑆 𝑖,𝑓 𝜏 ← predicted funny scores on 𝑓-th (i.e. validation) fold step 5: build cv-averaging models 𝑀 𝑖 𝜏 ← a model that takes an average of predictions of 𝑀 𝑖,1 𝜏 … 𝑀 𝑖,𝑘 𝜏 𝑆 𝑖 𝜏 ∈ ℝ 𝑑 ← a concatenation of 𝑆 𝑖,1 𝜏 … 𝑆 𝑖,𝑘 𝜏 , where 𝑑 is data size step 4: feedback the performance of 𝑀 𝑖 𝜏 to hyparparamoptimizer train base models train meta estimator step 1: 𝑺 ∈ ℝ 𝑑×|ℳ| ← a concatenation of 𝑆 1 … 𝑆 |𝒮| ∈ 𝒮 step 2: 𝔐 ← meta estimator trained on 𝑺 apply meta estimator input: test data consists of 𝐷 samples for 𝑖 = 1 … |ℳ|: step 1: 𝑆 𝑖 ′ ∈ ℝ 𝐷 ← predicted funny scores on test data with model 𝑀 𝑖 ∈ ℳ step 2: s′ ∈ ℝ 𝐷×|ℳ| ← concatenation of 𝑆 1 ′ … 𝑆 |ℳ| ′ step 3: apply the meta estimator 𝔐 on s′and get the final prediction 𝑑 fold1 fold1 1 fold1 fold1 2 fold1 fold1 3 |ℳ| 𝑆 1 𝐵𝐸𝑅𝑇 𝑆 𝐵 𝑋𝐿𝑀 select base models ℳ ← ∅, 𝒮 ← ∅ for τ in plms: ℳ ← ℳ ∪ { top-𝑛 performed models from 𝑀 1 𝜏 … 𝑀 𝐵 𝜏 } 𝒮 ← 𝒮 ∪ { corresponding predictions from 𝑆 1 𝜏 … 𝑆 𝐵 𝜏 } input: humicroedit data, funlines data, 𝑘 (cross-validation folds), 𝑛, b (trial budget), hyparparamoptimizer, plms (a set of plm types) stacking at scale 𝑆 1 𝑆 |ℳ| 2 1 3 1 2 3 hyparparam optimizer 𝐻 1 𝐻 𝐵 𝔐 𝑀 1 𝐵𝐸𝑅𝑇 𝑀 𝐵 𝑋𝐿𝑀 figure 2: simplified example of stacking at scale with 3-fold cross-validation and (iii) performing cross-validation (cv) during the whole process. cv is used for accumulating label leakage-free prediction data over the whole training dataset used for the meta-estimator training as well as measuring the accurate performances of models to select better base models. since sas requires enormous computations, discussions on complexity are given in appendix a. ## conclusion in this paper, we proposed a top performing model for the task of humor recognition. we fused thousands of pre-trained language models by stacking at scale. experimental results showed the incredible performance of the stacking at scale, and at the same time, also revealed the current limitation of pre-trained language models. for future work, we will explore injecting common sense or background knowledge into models to understand humor better.