pwc-archive/papers-with-abstracts huggingface dataset:
TeamJUST at SemEval-2020 Task 4: Commonsense Validation and Explanation Using Ensembling Techniques
Authors: Roweida Mohammed, Malak Abdullah
Date: 2020-12-01 00:00:00
Proceeding: SEMEVAL 2020
Tasks: Common Sense Reasoning, Sentence

Common sense for natural language processing methods has been attracting a wide research interest, recently. Estimating automatically whether a sentence makes sense or not is considered an essential question. Task 4 in the International Workshop SemEval 2020 has provided three subtasks (A, B, and C) that challenges the participants to build systems for distinguishing the common sense statements from those that do not make sense. This paper describes TeamJUST{'}s approach for participating in subtask A to differentiate between two sentences in English and classify them into two classes: common sense and uncommon sense statements. Our approach depends on ensembling four different state-of-the-art pre-trained models (BERT, ALBERT, Roberta, and XLNet). Our baseline model which we used only the pre-trained model of BERT has scored 89.1, while the TeamJUST model outperformed the baseline model with an accuracy score of 96.2. We have improved the results in the post-evaluation period to achieve our best result, which would rank the 4th in the competition if we had the chance to use our latest experiment.
TR at SemEval-2020 Task 4: Exploring the Limits of Language-model-based Common Sense Validation
Authors: Don Teo
Date: 2020-12-01 00:00:00
Proceeding: SEMEVAL 2020
Tasks: Common Sense Reasoning, Graph Embedding, Image Generation, Knowledge Graph Embedding, Language Modeling, Language Modelling, Text to Image Generation, Text-to-Image Generation, World Knowledge

In this paper, we present our submission for subtask A of the Common Sense Validation and Explanation (ComVE) shared task. We examine the ability of large-scale pre-trained language models to distinguish commonsense from non-commonsense statements. We also explore the utility of external resources that aim to supplement the world knowledge inherent in such language models, including commonsense knowledge graph embedding models, word concreteness ratings, and text-to-image generation models. We find that such resources provide insignificant gains to the performance of fine-tuned language models. We also provide a qualitative analysis of the limitations of the language model fine-tuned to this task.
UAICS at SemEval-2020 Task 4: Using a Bidirectional Transformer for Task a
Authors: Ciprian-Gabriel Cusmuliuc, Lucia-Georgiana Coca, Adrian Iftene
Date: 2020-12-01 00:00:00
Proceeding: SEMEVAL 2020

Commonsense Validation and Explanation has been a difficult task for machines since the dawn of computing. Although very trivial to humans it poses a high complexity for machines due to the necessity of inference over a pre-existing knowledge base. In order to try and solve this problem the SemEval 2020 Task 4 - {''}Commonsense Validation and Explanation (ComVE){''} aims to evaluate systems capable of multiple stages of ComVE. The challenge includes 3 tasks (A, B and C), each with it{'}s own requirements. Our team participated only in task A which required selecting the statement that made the least sense. We choose to use a bidirectional transformer in order to solve the challenge, this paper presents the details of our method, runs and result.
UI at SemEval-2020 Task 4: Commonsense Validation and Explanation by Exploiting Contradiction
Authors: Kerenza Doxolodeo, Rahmad Mahendra
Date: 2020-12-01 00:00:00
Proceeding: SEMEVAL 2020
Tasks: Sentence

This paper describes our submissions into the ComVe challenge, the SemEval 2020 Task 4. This evaluation task consists of three sub-tasks that test commonsense comprehension by identifying sentences that do not make sense and explain why they do not. In subtask A, we use Roberta to find which sentence does not make sense. In subtask B, besides using BERT, we also experiment with replacing the dataset with MNLI when selecting the best explanation from the provided options why the given sentence does not make sense. In subtask C, we utilize the MNLI model from subtask B to evaluate the explanation generated by Roberta and GPT-2 by exploiting the contradiction of the sentence and their explanation. Our system submission records a performance of 88.2{\%}, 80.5{\%}, and BLEU 5.5 for those three subtasks, respectively.
Warren at SemEval-2020 Task 4: ALBERT and Multi-Task Learning for Commonsense Validation
Authors: Yuhang Wu, Hao Wu
Date: 2020-12-01 00:00:00
Proceeding: SEMEVAL 2020
Tasks: Multi-Task Learning, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL), text-classification, Text Classification

This paper describes our system in subtask A of SemEval 2020 Shared Task 4. We propose a reinforcement learning model based on MTL(Multi-Task Learning) to enhance the prediction ability of commonsense validation. The experimental results demonstrate that our system outperforms the single-task text classification model. We combine MTL and ALBERT pretrain model to achieve an accuracy of 0.904 and our model is ranked 16th on the final leader board of the competition among the 45 teams.
YNU-oxz at SemEval-2020 Task 4: Commonsense Validation Using BERT with Bidirectional GRU
Authors: Xiaozhi Ou, Hongling Li
Date: 2020-12-01 00:00:00
Proceeding: SEMEVAL 2020
Tasks: Explanation Generation

This paper describes the system and results of our team participated in SemEval-2020 Task4: Commonsense Validation and Explanation (ComVE), which aim to distinguish meaningful natural language statements from unreasonable natural language statements. This task contains three subtasks: Subtask A{--}Validation, Subtask B{--}Explanation (Multi-Choice), and Subtask C{--} Explanation (Generation). In these three subtasks, we only participated in Subtask A, which aims to distinguish whether a given two natural language statements with similar wording are meaningful. To solve this problem, we proposed a method using a combination of BERT with the Bidirectional Gated Recurrent Unit (Bi-GRU). Our model achieved an accuracy of 0.836 in Subtask A (ranked 27/45).
BLCU-NLP at SemEval-2020 Task 5: Data Augmentation for Efficient Counterfactual Detecting
Authors: Chang Liu, Dong Yu
Date: 2020-12-01 00:00:00
Proceeding: SEMEVAL 2020
Tasks: Common Sense Reasoning, counterfactual, Data Augmentation

Counterfactuals describe events counter to facts and hence naturally involve common sense, knowledge, and reasoning. SemEval 2020 task 5 is focusing on this field. We participate in the subtask 1 and we use BERT as our system. Our Innovations are feature extraction and data augmentation. We extract and summarize features of counterfactual statements, augment counterfactual examples in training set with the help of these features, and two general methods of data augmentation is experimented in our work. We demonstrate the effectiveness of our approaches, which achieves 0.95 of subtask 1 in F1 while using only a subset of giving training set to fine-tune the BERT model, and our official submission achieves F1 0.802, which ranks us 16th in the competition.
BYteam at SemEval-2020 Task 5: Detecting Counterfactual Statements with BERT and Ensembles
Authors: Yang Bai, Xiaobing Zhou
Date: 2020-12-01 00:00:00
Proceeding: SEMEVAL 2020
Tasks: Classification, counterfactual

We participate in the classification tasks of SemEval-2020 Task: Subtask1: Detecting counterfactual statements of semeval-2020 task5(Detecting Counterfactuals). This paper examines different approaches and models towards detecting counterfactual statements classification. We choose the Bert model. However, the output of Bert is not a good summary of semantic information, so in order to obtain more abundant semantic information features, we modify the upper layer structure of Bert. Finally, our system achieves an accuracy of 88.90{\%} and F1 score of 86.30{\%} by hard voting, which ranks 6th on the final leader board of the in subtask 1 competition.
ETHAN at SemEval-2020 Task 5: Modelling Causal Reasoning in Language Using Neuro-symbolic Cloud Computing
Authors: Len Yabloko
Date: 2020-12-01 00:00:00
Proceeding: SEMEVAL 2020
Tasks: Cloud Computing, Language Modeling, Language Modelling

I present ETHAN: Experimental Testing of Hybrid AI Node implemented entirely on free cloud computing infrastructure. The ultimate goal of this research is to create modular reusable hybrid neuro-symbolic architecture for Artificial Intelligence. As a test case I model natural language comprehension of causal relations from open domain text corpus that combines semi-supervised language model (Huggingface Transformers) with constituency and dependency parsers (Allen Institute for Artificial Intelligence.)
Ferryman as SemEval-2020 Task 5: Optimized BERT for Detecting Counterfactuals
Authors: Weilong Chen, Yan Zhuang, Peng Wang, Feng Hong, Yan Wang, Yanru Zhang
Date: 2020-12-01 00:00:00
Proceeding: SEMEVAL 2020
Tasks: counterfactual, Counterfactual Detection, Counterfactual Reasoning, Sentence

The main purpose of this article is to state the effect of using different methods and models for counterfactual determination and detection of causal knowledge. Nowadays, counterfactual reasoning has been widely used in various fields. In the realm of natural language process(NLP), counterfactual reasoning has huge potential to improve the correctness of a sentence. In the shared Task 5 of detecting counterfactual in SemEval 2020, we pre-process the officially given dataset according to case conversion, extract stem and abbreviation replacement. We use last-5 bidirectional encoder representation from bidirectional encoder representation from transformer (BERT)and term frequency{--}inverse document frequency (TF-IDF) vectorizer for counterfactual detection. Meanwhile, multi-sample dropout and cross validation are used to improve versatility and prevent problems such as poor generosity caused by overfitting. Finally, our team Ferryman ranked the 8th place in the sub-task 1 of this competition.
Lee at SemEval-2020 Task 5: ALBERT Model Based on the Maximum Ensemble Strategy and Different Data Sampling Methods for Detecting Counterfactual Statements
Authors: Junyi Li, Yuhang Wu, Bin Wang, Haiyan Ding
Date: 2020-12-01 00:00:00
Proceeding: SEMEVAL 2020
Tasks: counterfactual

This article describes the system submitted to SemEval 2020 Task 5: Modelling Causal Reasoning in Language: Detecting Counterfactuals. In this task, we only participate in the subtask A which is detecting counterfactual statements. In order to solve this sub-task, first of all, because of the problem of data balance, we use the undersampling and oversampling methods to process the data set. Second, we used the ALBERT model and the maximum ensemble method based on the ALBERT model. Our methods achieved a F1 score of 0.85 in subtask A.
NLU-Co at SemEval-2020 Task 5: NLU/SVM Based Model Apply Tocharacterise and Extract Counterfactual Items on Raw Data
Authors: Elvis Mboning Tchiaze, Damien Nouvel
Date: 2020-12-01 00:00:00
Proceeding: SEMEVAL 2020
Tasks: counterfactual, Natural Language Understanding

In this article, we try to solve the problem of classification of counterfactual statements and extraction of antecedents/consequences in raw data, by mobilizing on one hand Support vector machine (SVMs) and on the other hand Natural Language Understanding (NLU) infrastructures available on the market for conversational agents. Our experiments allowed us to test different pipelines of two known platforms (Snips NLU and Rasa NLU). The results obtained show that a Rasa NLU pipeline, built with a well-preprocessed dataset and tuned algorithms, allows to model accurately the structure of a counterfactual event, in order to facilitate the identification and the extraction of its components.
Pheonix at SemEval-2020 Task 5: Masking the Labels Lubricates Models for Sequence Labeling
Authors: Pouria Babvey, Dario Borrelli, Yutong Zhao, Carlo Lipizzi
Date: 2020-12-01 00:00:00
Proceeding: SEMEVAL 2020

This paper presents the deep-learning model that is submitted to the SemEval-2020 Task 5 competition: {``}Detecting Counterfactuals{''}. We participated in both Subtask1 and Subtask2. The model proposed in this paper ranked 2nd in Subtask2 {``}Detecting antecedent and consequence{''}. Our model approaches the task as a sequence labeling. The architecture is built on top of BERT, and a multi-head attention layer with label masking is used to benefit from the mutual information between nearby labels. Also, for prediction, a multi-stage algorithm is used in which the model finalize some predictions with higher certainty in each step and use them in the following. Our results show that masking the labels not only is an efficient regularization method but also improves the accuracy of the model compared with other alternatives like CRF. Label masking can be used as a regularization method in sequence labeling. Also, it improves the performance of the model by learning the specific patterns in the target variable.
YNU-oxz at SemEval-2020 Task 5: Detecting Counterfactuals Based on Ordered Neurons LSTM and Hierarchical Attention Network
Authors: Xiaozhi Ou, Shengyan Liu, Hongling Li
Date: 2020-12-01 00:00:00
Proceeding: SEMEVAL 2020
Tasks: counterfactual, Dimensionality Reduction, Sentence

This paper describes the system and results of our team{'}s participation in SemEval-2020 Task5: Modelling Causal Reasoning in Language: Detecting Counterfactuals, which aims to simulate counterfactual semantics and reasoning in natural language. This task contains two subtasks: Subtask1{--}Detecting counterfactual statements and Subtask2{--}Detecting antecedent and consequence. We only participated in Subtask1, aiming to determine whether a given sentence is counterfactual. In order to solve this task, we proposed a system based on Ordered Neurons LSTM (ON-LSTM) with Hierarchical Attention Network (HAN) and used Pooling operation for dimensionality reduction. Finally, we used the K-fold approach as the ensemble method. Our model achieved an F1 score of 0.7040 in Subtask1 (Ranked 16/27).
BERTatDE at SemEval-2020 Task 6: Extracting Term-definition Pairs in Free Text Using Pre-trained Model
Authors: Huihui Zhang, Feiliang Ren
Date: 2020-12-01 00:00:00
Proceeding: SEMEVAL 2020
Tasks: Definition Extraction, Sentence, Sentence Classification

Definition extraction is an important task in Nature Language Processing, and it is used to identify the terms and definitions related to terms. The task contains sentence classification task (i.e., classify whether it contains definition) and sequence labeling task (i.e., find the boundary of terms and definitions). The paper describes our system BERTatDE1 in sentence classification task (subtask 1) and sequence labeling task (subtask 2) in the definition extraction (SemEval-2020 Task 6). We use BERT to solve the multi-domain problems including the uncertainty of term boundary that is, different areas have different ways to definite the domain related terms. We use BERT, BiLSTM and attention in subtask 1 and our best result achieved 79.71{\%} in F1 and the eighteenth place in subtask 1. For the subtask 2, we use BERT, BiLSTM and CRF to sequence labeling, and achieve 40.73{\%} in Macro-averaged F1.
DeftPunk at SemEval-2020 Task 6: Using RNN-ensemble for the Sentence Classification.
Authors: Jekaterina Kaparina, Anna Soboleva
Date: 2020-12-01 00:00:00
Proceeding: SEMEVAL 2020
Tasks: Sentence, Sentence Classification

This paper describes participation in DeftEval 2020 (part of SemEval sharing task competition), and is focused on the sentence classification. Our approach to the task was to create an ensemble of several RNNs combined with fasttext and ELMo embeddings. Results show that various types of models in an ensemble give a performance boost in comparison to standard models. Our model achieved F1-score of 78{\%} for a positive class on the DeftEval dataset.
RGCL at SemEval-2020 Task 6: Neural Approaches to DefinitionExtraction
Authors: Tharindu Ranasinghe, Alistair Plum, Constantin Orasan, Ruslan Mitkov
Date: 2020-12-01 00:00:00
Proceeding: SEMEVAL 2020
Tasks: Sentence

This paper presents the RGCL team submission to SemEval 2020 Task 6: DeftEval, subtasks 1 and 2. The system classifies definitions at the sentence and token levels. It utilises state-of-the-art neural network architectures, which have some task-specific adaptations, including an automatically extended training set. Overall, the approach achieves acceptable evaluation scores, while maintaining flexibility in architecture selection.
T\"uKaPo at SemEval-2020 Task 6: Def(n)tly Not BERT: Definition Extraction Using pre-BERT Methods in a post-BERT World
Authors: Madeeswaran Kannan, Haemanth Santhi Ponnusamy
Date: 2020-12-01 00:00:00
Proceeding: SEMEVAL 2020
Tasks: Definition Extraction, Sentence, Sentence Classification

We describe our system (T{\"u}KaPo) submitted for Task 6: DeftEval, at SemEval 2020. We developed a hybrid approach that combined existing CNN and RNN methods and investigated the impact of purely-syntactic and semantic features on the task of definition extraction. Our final model achieved a F1-score of 0.6851 in subtask 1, i.e, sentence classification.
UNIXLONG at SemEval-2020 Task 6: A Joint Model for Definition Extraction
Authors: Shuyi Xie, Jian Ma, Haiqin Yang, Jiang Lianxin, Mo Yang, Jianping Shen
Date: 2020-12-01 00:00:00
Proceeding: SEMEVAL 2020
Tasks: Definition Extraction, Relation, Relation Extraction, TAG

Definition Extraction is the task to automatically extract terms and their definitions from text. In recent years, it attracts wide interest from NLP researchers. This paper describes the unixlong team{'}s system for the SemEval 2020 task6: DeftEval: Extracting term-definition pairs in free text. The goal of this task is to extract definition, word level BIO tags and relations. This task is challenging due to the free style of the text, especially the definitions of the terms range across several sentences and lack explicit verb phrases. We propose a joint model to train the tasks of definition extraction and the word level BIO tagging simultaneously. We design a creative format input of BERT to capture the location information between entity and its definition. Then we adjust the result of BERT with some rules. Finally, we apply TAG{\_}ID, ROOT{\_}ID, BIO tag to predict the relation and achieve macro-averaged F1 score 1.0 which rank first on the official test set in the relation extraction subtask.
SemEval-2020 Task 8: Memotion Analysis- the Visuo-Lingual Metaphor!
Authors: Chhavi Sharma, Deepesh Bhageria, William Scott, Srinivas PYKL, Amitava Das, Tanmoy Chakraborty, Viswanath Pulabaigari, Bj{\"o}rn Gamb{\"a}ck
Date: 2020-12-01 00:00:00
Proceeding: SEMEVAL 2020
Tasks: Emotion Recognition

Information on social media comprises of various modalities such as textual, visual and audio. NLP and Computer Vision communities often leverage only one prominent modality in isolation to study social media. However, computational processing of Internet memes needs a hybrid approach. The growing ubiquity of Internet memes on social media platforms such as Facebook, Instagram, and Twitter further suggests that we can not ignore such multimodal content anymore. To the best of our knowledge, there is not much attention towards meme emotion analysis. The objective of this proposal is to bring the attention of the research community towards the automatic processing of Internet memes. The task Memotion analysis released approx 10K annotated memes- with human annotated labels namely sentiment(positive, negative, neutral), type of emotion(sarcastic,funny,offensive, motivation) and their corresponding intensity. The challenge consisted of three subtasks: sentiment (positive, negative, and neutral) analysis of memes,overall emotion (humor, sarcasm, offensive, and motivational) classification of memes, and classifying intensity of meme emotion. The best performances achieved were F1 (macro average) scores of 0.35, 0.51 and 0.32, respectively for each of the three subtasks.
Hitachi at SemEval-2020 Task 7: Stacking at Scale with Heterogeneous Language Models for Humor Recognition
Authors: Terufumi Morishita, Gaku Morio, Hiroaki Ozaki, Toshinori Miyoshi
Date: 2020-12-01 00:00:00
Proceeding: SEMEVAL 2020

This paper describes the winning system for SemEval-2020 task 7: Assessing Humor in Edited News Headlines. Our strategy is Stacking at Scale (SaS) with heterogeneous pre-trained language models (PLMs) such as BERT and GPT-2. SaS first performs fine-tuning on numbers of PLMs with various hyperparameters and then applies a powerful stacking ensemble on top of the fine-tuned PLMs. Our experimental results show that SaS outperforms a naive average ensemble, leveraging weaker PLMs as well as high-performing PLMs. Interestingly, the results show that SaS captured non-funny semantics. Consequently, the system was ranked 1st in all subtasks by significant margins compared with other systems.
SESAM at SemEval-2020 Task 8: Investigating the Relationship between Image and Text in Sentiment Analysis of Memes
Authors: Lisa Bonheme, Marek Grzes
Date: 2020-12-01 00:00:00
Proceeding: SEMEVAL 2020
Tasks: Sentiment Analysis

This paper presents our submission to task 8 (memotion analysis) of the SemEval 2020 competition. We explain the algorithms that were used to learn our models along with the process of tuning the algorithms and selecting the best model. Since meme analysis is a challenging task with two distinct modalities, we studied the impact of different multimodal representation strategies. The results of several approaches to dealing with multimodal data are therefore discussed in the paper. We found that alignment-based strategies did not perform well on memes. Our quantitative results also showed that images and text were uncorrelated. Fusion-based strategies did not show significant improvements and using one modality only (text or image) tends to lead to better results when applied with the predictive models that we used in our research.
Buhscitu at SemEval-2020 Task 7: Assessing Humour in Edited News Headlines Using Hand-Crafted Features and Online Knowledge Bases
Authors: Kristian N{\o}rgaard Jensen, Nicolaj Filrup Rasmussen, Thai Wang, Marco Placenti, Barbara Plank
Date: 2020-12-01 00:00:00
Proceeding: SEMEVAL 2020
Tasks: Language Modeling, Language Modelling, regression

This paper describes a system that aims at assessing humour intensity in edited news headlines as part of the 7th task of SemEval-2020 on {``}Humor, Emphasis and Sentiment{''}. Various factors need to be accounted for in order to assess the funniness of an edited headline. We propose an architecture that uses hand-crafted features, knowledge bases and a language model to understand humour, and combines them in a regression model. Our system outperforms two baselines. In general, automatic humour assessment remains a difficult task.
Hasyarasa at SemEval-2020 Task 7: Quantifying Humor as Departure from Expectedness
Authors: Ravi Theja Desetty, Ranit Chatterjee, Smita Ghaisas
Date: 2020-12-01 00:00:00
Proceeding: SEMEVAL 2020

This paper describes our system submission Hasyarasa for the SemEval-2020 Task-7: Assessing Humor in Edited News Headlines. This task has two subtasks. The goal of Subtask 1 is to predict the mean funniness of the edited headline given the original and the edited headline. In Subtask 2, given two edits on the original headline, the goal is to predict the funnier of the two. We observed that the departure from expected state/ actions of situations/ individuals is the cause of humor in the edited headlines. We propose two novel features: Contextual Semantic Distance and Contextual Neighborhood Distance to estimate this departure and thus capture the contextual absurdity and hence the humor in the edited headlines. We have used these features together with a Bi-LSTM Attention based model and have achieved 0.53310 RMSE for Subtask 1 and 60.19{\%} accuracy for Subtask 2.