pwc-archive/papers-with-abstracts huggingface dataset:
CitiusNLP at SemEval-2020 Task 3: Comparing Two Approaches for Word Vector Contextualization
Authors: Pablo Gamallo
Date: 2020-12-01 00:00:00
Proceeding: SEMEVAL 2020
Tasks: Word Embeddings, Word Similarity

This article describes some unsupervised strategies submitted to SemEval 2020 Task 3, a task which consists of considering the effect of context to compute word similarity. More precisely, given two words in context, the system must predict the degree of similarity of those words considering the context in which they occur, and the system score is compared against human prediction. We compare one approach based on pre-trained BERT models with other strategy relying on static word embeddings and syntactic dependencies. The BERT-based method clearly outperformed the dependency-based strategy.
Ferryman at SemEval-2020 Task 3: Bert with TFIDF-Weighting for Predicting the Effect of Context in Word Similarity
Authors: Weilong Chen, Xin Yuan, Sai Zhang, Jiehui Wu, Yanru Zhang, Yan Wang
Date: 2020-12-01 00:00:00
Proceeding: SEMEVAL 2020
Tasks: Position, Sentence, Word Similarity

Word similarity is widely used in machine learning applications like searching engine and recommendation. Measuring the changing meaning of the same word between two different sentences is not only a way to handle complex features in word usage (such as sentence syntax and semantics), but also an important method for different word polysemy modeling. In this paper, we present the methodology proposed by team Ferryman. Our system is based on the Bidirectional Encoder Representations from Transformers (BERT) model combined with term frequency-inverse document frequency (TF-IDF), applying the method on the provided datasets called CoSimLex, which covers four different languages including English, Croatian, Slovene, and Finnish. Our team Ferryman wins the the first position for English task and the second position for Finnish in the subtask 1.
Hitachi at SemEval-2020 Task 3: Exploring the Representation Spaces of Transformers for Human Sense Word Similarity
Authors: Terufumi Morishita, Gaku Morio, Hiroaki Ozaki, Toshinori Miyoshi
Date: 2020-12-01 00:00:00
Proceeding: SEMEVAL 2020
Tasks: Word Similarity

In this paper, we present our system for SemEval-2020 task 3, Predicting the (Graded) Effect of Context in Word Similarity. Due to the unsupervised nature of the task, we concentrated on inquiring about the similarity measures induced by different layers of different pre-trained Transformer-based language models, which can be good approximations of the human sense of word similarity. Interestingly, our experiments reveal a language-independent characteristic: the middle to upper layers of Transformer-based language models can induce good approximate similarity measures. Finally, our system was ranked 1st on the Slovenian part of Subtask1 and 2nd on the Croatian part of both Subtask1 and Subtask2.
JUSTMasters at SemEval-2020 Task 3: Multilingual Deep Learning Model to Predict the Effect of Context in Word Similarity
Authors: Nour Al-khdour, Mutaz Bni Younes, Malak Abdullah, Mohammad AL-Smadi
Date: 2020-12-01 00:00:00
Proceeding: SEMEVAL 2020
Tasks: Word Similarity

There is a growing research interest in studying word similarity. Without a doubt, two similar words in a context may considered different in another context. Therefore, this paper investigates the effect of the context in word similarity. The SemEval-2020 workshop has provided a shared task (Task 3: Predicting the (Graded) Effect of Context in Word Similarity). In this task, the organizers provided unlabeled datasets for four languages, English, Croatian, Finnish and Slovenian. Our team, JUSTMasters, has participated in this competition in the two subtasks: A and B. Our approach has used a weighted average ensembling method for different pretrained embeddings techniques for each of the four languages. Our proposed model outperformed the baseline models in both subtasks and acheived the best result for subtask 2 in English and Finnish, with score 0.725 and 0.68 respectively. We have been ranked the sixth for subtask 1, with scores for English, Croatian, Finnish, and Slovenian as follows: 0.738, 0.44, 0.546, 0.512.
Will\_Go at SemEval-2020 Task 3: An Accurate Model for Predicting the (Graded) Effect of Context in Word Similarity Based on BERT
Authors: Wei Bao, Hongshu Che, Jiandong Zhang
Date: 2020-12-01 00:00:00
Proceeding: SEMEVAL 2020
Tasks: Word Similarity

Natural Language Processing (NLP) has been widely used in the semantic analysis in recent years. Our paper mainly discusses a methodology to analyze the effect that context has on human perception of similar words, which is the third task of SemEval 2020. We apply several methods in calculating the distance between two embedding vector generated by Bidirectional Encoder Representation from Transformer (BERT). Our team will go won the 1st place in Finnish language track of subtask1, the second place in English track of subtask1.
HIT-SCIR at SemEval-2020 Task 5: Training Pre-trained Language Model with Pseudo-labeling Data for Counterfactuals Detection
Authors: Xiao Ding, Dingkui Hao, Yuewei Zhang, Kuo Liao, Zhongyang Li, Bing Qin, Ting Liu
Date: 2020-12-01 00:00:00
Proceeding: SEMEVAL 2020
Tasks: counterfactual, Language Modeling, Language Modelling, Task 2

We describe our system for Task 5 of SemEval 2020: Modelling Causal Reasoning in Language: Detecting Counterfactuals. Despite deep learning has achieved significant success in many fields, it still hardly drives today{'}s AI to strong AI, as it lacks of causation, which is a fundamental concept in human thinking and reasoning. In this task, we dedicate to detecting causation, especially counterfactuals from texts. We explore multiple pre-trained models to learn basic features and then fine-tune models with counterfactual data and pseudo-labeling data. Our team HIT-SCIR wins the first place (1st) in Sub-task 1 {---} Detecting Counterfactual Statements and is ranked 4th in Sub-task 2 {---} Detecting Antecedent and Consequence. In this paper we provide a detailed description of the approach, as well as the results obtained in this task.
Cardiff University at SemEval-2020 Task 6: Fine-tuning BERT for Domain-Specific Definition Classification
Authors: Shelan Jeawak, Luis Espinosa-Anke, Steven Schockaert
Date: 2020-12-01 00:00:00
Proceeding: SEMEVAL 2020
Tasks: Language Modeling, Language Modelling, Sentence

We describe the system submitted to SemEval-2020 Task 6, Subtask 1. The aim of this subtask is to predict whether a given sentence contains a definition or not. Unsurprisingly, we found that strong results can be achieved by fine-tuning a pre-trained BERT language model. In this paper, we analyze the performance of this strategy. Among others, we show that results can be improved by using a two-step fine-tuning process, in which the BERT model is first fine-tuned on the full training set, and then further specialized towards a target domain.
Masked Reasoner at SemEval-2020 Task 4: Fine-Tuning RoBERTa for Commonsense Reasoning
Authors: Daming Lu
Date: 2020-12-01 00:00:00
Proceeding: SEMEVAL 2020
Tasks: Language Modeling, Language Modelling

This paper describes the masked reasoner system that participated in SemEval-2020 Task 4: Commonsense Validation and Explanation. The system participated in the subtask B.We proposes a novel method to fine-tune RoBERTa by masking the most important word in the statement. We believe that the confidence of the system in recovering that word is positively correlated to the score the masked language model gives to the current statement-explanation pair. We evaluate the importance of each word using InferSent and do the masked fine-tuning on RoBERTa. Then we use the fine-tuned model to predict the most plausible explanation. Our system is fast in training and achieved 73.5{\%} accuracy.
SWAGex at SemEval-2020 Task 4: Commonsense Explanation as Next Event Prediction
Authors: Wiem Ben Rim, Naoaki Okazaki
Date: 2020-12-01 00:00:00
Proceeding: SEMEVAL 2020
Tasks: Language Modeling, Language Modelling

We describe the system submitted by the SWAGex team to the SemEval-2020 Commonsense Validation and Explanation Task. We use multiple methods on the pre-trained language model BERT (Devlin et al., 2018) for tasks that require the system to recognize sentences against commonsense and justify the reasoning behind this decision. Our best performing model is BERT trained on SWAG and fine-tuned for the task. We investigate the ability to transfer commonsense knowledge from SWAG to SemEval-2020 by training a model for the Explanation task with Next Event Prediction data
UoR at SemEval-2020 Task 4: Pre-trained Sentence Transformer Models for Commonsense Validation and Explanation
Authors: Thanet Markchom, Bhuvana Dhruva, Chandresh Pravin, HuiZhi Liang
Date: 2020-12-01 00:00:00
Proceeding: SEMEVAL 2020
Tasks: Common Sense Reasoning, Sentence

SemEval Task 4 Commonsense Validation and Explanation Challenge is to validate whether a system can differentiate natural language statements that make sense from those that do not make sense. Two subtasks, A and B, are focused in this work, i.e., detecting against-common-sense statements and selecting explanations of why they are false from the given options. Intuitively, commonsense validation requires additional knowledge beyond the given statements. Therefore, we propose a system utilising pre-trained sentence transformer models based on BERT, RoBERTa and DistillBERT architectures to embed the statements before classification. According to the results, these embeddings can improve the performance of the typical MLP and LSTM classifiers as downstream models of both subtasks compared to regular tokenised statements. These embedded statements are shown to comprise additional information from external resources which help validate common sense in natural language.
CLaC at SemEval-2020 Task 5: Muli-task Stacked Bi-LSTMs
Authors: MinGyou Sung, Parsa Bagherzadeh, Sabine Bergler
Date: 2020-12-01 00:00:00
Proceeding: SEMEVAL 2020
Tasks: Multi-Task Learning, POS, Word Embeddings

We consider detection of the span of antecedents and consequents in argumentative prose a structural, grammatical task. Our system comprises a set of stacked Bi-LSTMs trained on two complementary linguistic annotations. We explore the effectiveness of grammatical features (POS and clause type) through ablation. The reported experiments suggest that a multi-task learning approach using this external, grammatical knowledge is useful for detecting the extent of antecedents and consequents and performs nearly as well without the use of word embeddings.
ACNLP at SemEval-2020 Task 6: A Supervised Approach for Definition Extraction
Authors: Fabien Caspani, Pirashanth Ratnamogan, Mathis Linger, Mhamed Hajaiej
Date: 2020-12-01 00:00:00
Proceeding: SEMEVAL 2020
Tasks: Classification, Definition Extraction, Relation Classification, Sentence, Sentence Classification, Transfer Learning

We describe our contribution to two of the subtasks of SemEval 2020 Task 6, DeftEval: Extracting term-definition pairs in free text. The system for Subtask 1: Sentence Classification is based on a transformer architecture where we use transfer learning to fine-tune a pretrained model on the downstream task, and the one for Subtask 3: Relation Classification uses a Random Forest classifier with handcrafted dedicated features. Our systems respectively achieve 0.830 and 0.994 F1-scores on the official test set, and we believe that the insights derived from our study are potentially relevant to help advance the research on definition extraction.
Gorynych Transformer at SemEval-2020 Task 6: Multi-task Learning for Definition Extraction
Authors: Adis Davletov, Nikolay Arefyev, Alexander Shatilov, Denis Gordeev, Alexey Rey
Date: 2020-12-01 00:00:00
Proceeding: SEMEVAL 2020
Tasks: Classification, Definition Extraction, Multi-Task Learning, named-entity-recognition, Named Entity Recognition, Named Entity Recognition (NER), Relation Classification, Sentence, Sentence Classification

This paper describes our approach to {``}DeftEval: Extracting Definitions from Free Text in Textbooks{''} competition held as a part of Semeval 2020. The task was devoted to finding and labeling definitions in texts. DeftEval was split into three subtasks: sentence classification, sequence labeling and relation classification. Our solution ranked 5th in the first subtask and 23rd and 21st in the second and the third subtasks respectively. We applied simultaneous multi-task learning with Transformer-based models for subtasks 1 and 3 and a single BERT-based model for named entity recognition.
CN-HIT-IT.NLP at SemEval-2020 Task 4: Enhanced Language Representation with Multiple Knowledge Triples
Authors: Yice Zhang, Jiaxuan Lin, Yang Fan, Peng Jin, Yuanchao Liu, Bingquan Liu
Date: 2020-12-01 00:00:00
Proceeding: SEMEVAL 2020
Tasks: Knowledge Graphs

This paper describes our system that participated in the SemEval-2020 task 4: Commonsense Validation and Explanation. For this task, it is obvious that external knowledge, such as Knowledge graph, can help the model understand commonsense in natural language statements. But how to select the right triples for statements remains unsolved, so how to reduce the interference of irrelevant triples on model performance is a research focus. This paper adopt a modified K-BERT as the language encoder, to enhance language representation through triples from knowledge graphs. Experiments show that our method is better than models without external knowledge, and is slightly better than the original K-BERT. We got an accuracy score of 0.97 in subtaskA, ranking 1/45, and got an accuracy score of 0.948, ranking 2/35.
DEEPYANG at SemEval-2020 Task 4: Using the Hidden Layer State of BERT Model for Differentiating Common Sense
Authors: Yang Bai, Xiaobing Zhou
Date: 2020-12-01 00:00:00
Proceeding: SEMEVAL 2020
Tasks: Common Sense Reasoning, Natural Language Understanding

Introducing common sense to natural language understanding systems has received increasing research attention. To facilitate the researches on common sense reasoning, the SemEval-2020 Task 4 Commonsense Validation and Explanation(ComVE) is proposed. We participate in sub-task A and try various methods including traditional machine learning methods, deep learning methods, and also recent pre-trained language models. Finally, we concatenate the original output of BERT and the output vector of BERT hidden layer state to obtain more abundant semantic information features, and obtain competitive results. Our model achieves an accuracy of 0.8510 in the final test data and ranks 25th among all the teams.
HR@JUST Team at SemEval-2020 Task 4: The Impact of RoBERTa Transformer for Evaluation Common Sense Understanding
Authors: Heba Al-Jarrah, Rahaf Al-Hamouri, Mohammad AL-Smadi
Date: 2020-12-01 00:00:00
Proceeding: SEMEVAL 2020
Tasks: Common Sense Reasoning

This paper describes the results of our team HR@JUST participation at SemEval-2020 Task 4 - Commonsense Validation and Explanation (ComVE) for POST evaluation period. The provided task consists of three sub-tasks, we participate in task A. We considered a state-of-the-art approach for solving this task by performing RoBERTa model with no Next Sentences Prediction (NSP), dynamic masking, larger training data, and larger batch size. The achieved results show that we got the 11th rank on the final test set leaderboard with an accuracy of 91.3{\%}.
JBNU at SemEval-2020 Task 4: BERT and UniLM for Commonsense Validation and Explanation
Authors: Seung-Hoon Na, Jong-Hyeon Lee
Date: 2020-12-01 00:00:00
Proceeding: SEMEVAL 2020
Tasks: Sentence

This paper presents our contributions to the SemEval-2020 Task 4 Commonsense Validation and Explanation (ComVE) and includes the experimental results of the two Subtasks B and C of the SemEval-2020 Task 4. Our systems rely on pre-trained language models, i.e., BERT (including its variants) and UniLM, and rank 10th and 7th among 27 and 17 systems on Subtasks B and C, respectively. We analyze the commonsense ability of the existing pretrained language models by testing them on the SemEval-2020 Task 4 ComVE dataset, specifically for Subtasks B and C, the explanation subtasks with multi-choice and sentence generation, respectively.
JUSTers at SemEval-2020 Task 4: Evaluating Transformer Models against Commonsense Validation and Explanation
Authors: Ali Fadel, Mahmoud Al-Ayyoub, Erik Cambria
Date: 2020-12-01 00:00:00
Proceeding: SEMEVAL 2020

In this paper, we describe our team{'}s (JUSTers) effort in the Commonsense Validation and Explanation (ComVE) task, which is part of SemEval2020. We evaluate five pre-trained Transformer-based language models with various sizes against the three proposed subtasks. For the first two subtasks, the best accuracy levels achieved by our models are 92.90{\%} and 92.30{\%}, respectively, placing our team in the 12th and 9th places, respectively. As for the last subtask, our models reach 16.10 BLEU score and 1.94 human evaluation score placing our team in the 5th and 3rd places according to these two metrics, respectively. The latter is only 0.16 away from the 1st place human evaluation score.
KDE SenseForce at SemEval-2020 Task 4: Exploiting BERT for Commonsense Validation and Explanation
Authors: Khanddorj Mendbayar, Masaki Aono
Date: 2020-12-01 00:00:00
Proceeding: SEMEVAL 2020
Tasks: Natural Language Understanding

Using a natural language understanding system for commonsense comprehension is getting increasing attention from researchers. Current multi-purpose state-of-the-art models suffer on commonsense validation and explanation tasks. We have adopted one of the state-of-the-art models and proposing a method to boost the performance of the model in commonsense related tasks.
Lijunyi at SemEval-2020 Task 4: An ALBERT Model Based Maximum Ensemble with Different Training Sizes and Depths for Commonsense Validation and Explanation
Authors: Junyi Li, Bin Wang, Haiyan Ding
Date: 2020-12-01 00:00:00
Proceeding: SEMEVAL 2020
Tasks: Sentence

This article describes the system submitted to SemEval 2020 Task 4: Commonsense Validation and Explanation. We only participated in the subtask A, which is mainly to distinguish whether the sentence has meaning. To solve this task, we mainly used ALBERT model-based maximum ensemble with different training sizes and depths. To prove the validity of the model to the task, we also used some other neural network models for comparison. Our model achieved the accuracy score of 0.938(ranked 10/41) in subtask A.
Mxgra at SemEval-2020 Task 4: Common Sense Making with Next Token Prediction
Authors: Kris Collins, Max Grathwohl, Heba Ahmed
Date: 2020-12-01 00:00:00
Proceeding: SEMEVAL 2020
Tasks: Common Sense Reasoning, Language Modeling, Language Modelling, Sentence

In this paper, we explore solutions to a common sense making task in which a model must discern which of two sentences is against common sense. We used a pre-trained language model which we used to calculate complexity scores for input to discern which sentence contained an unlikely sequence of tokens. Other approaches we tested were word vector distances, which were used to find semantic outliers within a sentence, and siamese network. By using the pre-trained language model to calculate perplexity scores based on the sequence of tokens in input sentences, we achieved an accuracy of 75 percent.
NLP@JUST at SemEval-2020 Task 4: Ensemble Technique for BERT and Roberta to Evaluate Commonsense Validation
Authors: Emran Al-Bashabsheh, Ayah Abu Aqouleh, Mohammad AL-Smadi
Date: 2020-12-01 00:00:00
Proceeding: SEMEVAL 2020
Tasks: Common Sense Reasoning

This paper presents the work of the NLP@JUST team at SemEval-2020 Task 4 competition that related to commonsense validation and explanation (ComVE) task. The team participates in sub-taskA (Validation) which related to validation that checks if the text is against common sense or not. Several models have trained (\textit{i.e. Bert, XLNet, and Roberta}), however, the main models used are the RoBERTa-large and BERT Whole word masking. As well as, we utilized the results from both models to generate final prediction by using the average Ensemble technique, that used to improve the overall performance. The evaluation result shows that the implemented model achieved an accuracy of 93.9{\%} obtained and published at the post-evaluation result on the leaderboard.
SSN-NLP at SemEval-2020 Task 4: Text Classification and Generation on Common Sense Context Using Neural Networks
Authors: Rishivardhan K., Kayalvizhi S, Thenmozhi D., Raghav R., Kshitij Sharma
Date: 2020-12-01 00:00:00
Proceeding: SEMEVAL 2020
Tasks: Common Sense Reasoning, Machine Translation, Majority Voting Classifier, NMT, Sentence, text-classification, Text Classification, Translation

Common sense validation deals with testing whether a system can differentiate natural language statements that make sense from those that do not make sense. This paper describes the our approach to solve this challenge. For common sense validation with multi choice, we propose a stacking based approach to classify sentences that are more favourable in terms of common sense to the particular statement. We have used majority voting classifier methodology amongst three models such as Bidirectional Encoder Representations from Transformers (BERT), Micro Text Classification (Micro TC) and XLNet. For sentence generation, we used Neural Machine Translation (NMT) model to generate explanatory sentences.
Team Solomon at SemEval-2020 Task 4: Be Reasonable: Exploiting Large-scale Language Models for Commonsense Reasoning
Authors: Vertika Srivastava, Sudeep Kumar Sahoo, Yeon Hyang Kim, Rohit R.R, Mayank Raj, Ajay Jaiswal
Date: 2020-12-01 00:00:00
Proceeding: SEMEVAL 2020
Tasks: Natural Language Understanding, Reading Comprehension

In this paper, we present our submission for SemEval 2020 Task 4 - Commonsense Validation and Explanation (ComVE). The objective of this task was to develop a system that can differentiate statements that make sense from the ones that don{'}t. ComVE comprises of three subtasks to challenge and test a system{'}s capability in understanding commonsense knowledge from various dimensions. Commonsense reasoning is a challenging task in the domain of natural language understanding and systems augmented with it can improve performance in various other tasks such as reading comprehension, and inferencing. We have developed a system that leverages commonsense knowledge from pretrained language models trained on huge corpus such as RoBERTa, GPT2, etc. Our proposed system validates the reasonability of a given statement against the backdrop of commonsense knowledge acquired by these models and generates a logical reason to support its decision. Our system ranked 2nd in subtask C with a BLEU score of 19.3, which by far is the most challenging subtask as it required systems to generate the rationale behind the choice of an unreasonable statement. In subtask A and B, we achieved 96{\%} and 94{\%} accuracy respectively standing at 4th position in both the subtasks.