pwc-archive/papers-with-abstracts huggingface dataset:
Lotus at SemEval-2021 Task 2: Combination of BERT and Paraphrasing for English Word Sense Disambiguation
Authors: Niloofar Ranjbar, Hossein Zeinali
Date: 2021-08-01 00:00:00
Proceeding: SEMEVAL 2021
Tasks: Task 2, Word Sense Disambiguation

In this paper, we describe our proposed methods for the multilingual word-in-Context disambiguation task in SemEval-2021. In this task, systems should determine whether a word that occurs in two different sentences is used with the same meaning or not. We proposed several methods using a pre-trained BERT model. In two of them, we paraphrased sentences and add them as input to the BERT, and in one of them, we used WordNet to add some extra lexical information. We evaluated our proposed methods on test data in SemEval- 2021 task 2.
Cambridge at SemEval-2021 Task 2: Neural WiC-Model with Data Augmentation and Exploration of Representation
Authors: Zheng Yuan, David Strohmaier
Date: 2021-08-01 00:00:00
Proceeding: SEMEVAL 2021
Tasks: Data Augmentation, Language Modeling, Language Modelling, Task 2

This paper describes the system of the Cambridge team submitted to the SemEval-2021 shared task on Multilingual and Cross-lingual Word-in-Context Disambiguation. Building on top of a pre-trained masked language model, our system is first pre-trained on out-of-domain data, and then fine-tuned on in-domain data. We demonstrate the effectiveness of the proposed two-step training strategy and the benefits of data augmentation from both existing examples and new resources. We further investigate different representations and show that the addition of distance-based features is helpful in the word-in-context disambiguation task. Our system yields highly competitive results in the cross-lingual track without training on any cross-lingual data; and achieves state-of-the-art results in the multilingual track, ranking first in two languages (Arabic and Russian) and second in French out of 171 submitted systems.
UoB\_UK at SemEval 2021 Task 2: Zero-Shot and Few-Shot Learning for Multi-lingual and Cross-lingual Word Sense Disambiguation.
Authors: Wei Li, Harish Tayyar Madabushi, Mark Lee
Date: 2021-08-01 00:00:00
Proceeding: SEMEVAL 2021
Tasks: Few-Shot Learning, Task 2, Transfer Learning, Word Sense Disambiguation

This paper describes our submission to SemEval 2021 Task 2. We compare XLM-RoBERTa Base and Large in the few-shot and zero-shot settings and additionally test the effectiveness of using a k-nearest neighbors classifier in the few-shot setting instead of the more traditional multi-layered perceptron. Our experiments on both the multi-lingual and cross-lingual data show that XLM-RoBERTa Large, unlike the Base version, seems to be able to more effectively transfer learning in a few-shot setting and that the k-nearest neighbors classifier is indeed a more powerful classifier than a multi-layered perceptron when used in few-shot learning.
PAW at SemEval-2021 Task 2: Multilingual and Cross-lingual Word-in-Context Disambiguation : Exploring Cross Lingual Transfer, Augmentations and Adversarial Training
Authors: Harsh Goyal, Aadarsh Singh, Priyanshu Kumar
Date: 2021-08-01 00:00:00
Proceeding: SEMEVAL 2021
Tasks: Binary Classification, Cross-Lingual Transfer, Data Augmentation, Task 2

We experiment with XLM RoBERTa for Word in Context Disambiguation in the Multi Lingual and Cross Lingual setting so as to develop a single model having knowledge about both settings. We solve the problem as a binary classification problem and also experiment with data augmentation and adversarial training techniques. In addition, we also experiment with a 2-stage training technique. Our approaches prove to be beneficial for better performance and robustness.
GlossReader at SemEval-2021 Task 2: Reading Definitions Improves Contextualized Word Embeddings
Authors: Maxim Rachinskiy, Nikolay Arefyev
Date: 2021-08-01 00:00:00
Proceeding: SEMEVAL 2021
Tasks: Cross-Lingual Transfer, Language Modelling, Task 2, Word Embeddings, Word Sense Disambiguation, XLM-R, Zero-Shot Cross-Lingual Transfer

Consulting a dictionary or a glossary is a familiar way for many humans to figure out what does a word in a particular context mean. We hypothesize that a system that can select a proper definition for a particular word occurrence can also naturally solve tasks related to word senses. To verify this hypothesis we developed a solution for the Multilingual and Cross-lingual Word-in-Context (MCL-WiC) task, that does not use any of the shared task data or other WiC data for training. Instead, it is trained to embed word definitions from English WordNet and word occurrences in English texts into the same vector space following an approach previously proposed for Word Sense Disambiguation (WSD). To estimate the similarity in meaning of two word occurrences, we compared different metrics in this shared vector space and found that L1-distance between normalized contextualized word embeddings outperforms traditionally employed cosine similarity and several other metrics. To solve the task for languages other than English, we rely on zero-shot cross-lingual transfer capabilities of the multilingual XLM-R masked language model. Despite not using MCL-WiC training data, in the shared task our approach achieves an accuracy of 89.5{\%} on the English test set, which is only 4{\%} less than the best system. In the multilingual subtask zero-shot cross-lingual transfer shows competitive results, that are within 2{\%} from the best systems for Russian, French, and Arabic. In the cross-lingual subtask are within 2-4{\%} from the best systems.
UAlberta at SemEval-2021 Task 2: Determining Sense Synonymy via Translations
Authors: Bradley Hauer, Hongchang Bao, Arnob Mallik, Grzegorz Kondrak
Date: 2021-08-01 00:00:00
Proceeding: SEMEVAL 2021
Tasks: Explainable Models, Task 2, Translation

We describe the University of Alberta systems for the SemEval-2021 Word-in-Context (WiC) disambiguation task. We explore the use of translation information for deciding whether two different tokens of the same word correspond to the same sense of the word. Our focus is on developing principled theoretical approaches which are grounded in linguistic phenomena, leading to more explainable models. We show that translations from multiple languages can be leveraged to improve the accuracy on the WiC task.
LIORI at SemEval-2021 Task 2: Span Prediction and Binary Classification approaches to Word-in-Context Disambiguation
Authors: Adis Davletov, Nikolay Arefyev, Denis Gordeev, Alexey Rey
Date: 2021-08-01 00:00:00
Proceeding: SEMEVAL 2021
Tasks: Binary Classification, Data Augmentation, Question Answering, Task 2, XLM-R

This paper presents our approaches to SemEval-2021 Task 2: Multilingual and Cross-lingual Word-in-Context Disambiguation task. The first approach attempted to reformulate the task as a question answering problem, while the second one framed it as a binary classification problem. Our best system, which is an ensemble of XLM-R based binary classifiers trained with data augmentation, is among the 3 best-performing systems for Russian, French and Arabic in the multilingual subtask. In the post-evaluation period, we experimented with batch normalization, subword pooling and target word occurrence aggregation methods, resulting in further performance improvements.
FII\_CROSS at SemEval-2021 Task 2: Multilingual and Cross-lingual Word-in-Context Disambiguation
Authors: Ciprian Bodnar, Andrada Tapuc, Cosmin Pintilie, Daniela Gifu, Diana Trandabat
Date: 2021-08-01 00:00:00
Proceeding: SEMEVAL 2021
Tasks: Task 2

This paper presents a word-in-context disambiguation system. The task focuses on capturing the polysemous nature of words in a multilingual and cross-lingual setting, without considering a strict inventory of word meanings. The system applies Natural Language Processing algorithms on datasets from SemEval 2021 Task 2, being able to identify the meaning of words for the languages Arabic, Chinese, English, French and Russian, without making use of any additional mono- or multilingual resources.
UoR at SemEval-2021 Task 4: Using Pre-trained BERT Token Embeddings for Question Answering of Abstract Meaning
Authors: Thanet Markchom, HuiZhi Liang
Date: 2021-08-01 00:00:00
Proceeding: SEMEVAL 2021
Tasks: Question Answering, Reading Comprehension

Most question answering tasks focuses on predicting concrete answers, e.g., named entities. These tasks can be normally achieved by understanding the contexts without additional information required. In Reading Comprehension of Abstract Meaning (ReCAM) task, the abstract answers are introduced. To understand abstract meanings in the context, additional knowledge is essential. In this paper, we propose an approach that leverages the pre-trained BERT Token embeddings as a prior knowledge resource. According to the results, our approach using the pre-trained BERT outperformed the baselines. It shows that the pre-trained BERT token embeddings can be used as additional knowledge for understanding abstract meanings in question answering.
Noobs at Semeval-2021 Task 4: Masked Language Modeling for abstract answer prediction
Authors: Shikhar Shukla, Sarthak Sarthak, Karm Veer Arya
Date: 2021-08-01 00:00:00
Proceeding: SEMEVAL 2021
Tasks: Language Modeling, Language Modelling, Masked Language Modeling, Reading Comprehension

This paper presents the system developed by our team for Semeval 2021 Task 4: Reading Comprehension of Abstract Meaning. The aim of the task was to benchmark the NLP techniques in understanding the abstract concepts present in a passage, and then predict the missing word in a human written summary of the passage. We trained a Roberta-Large model trained with a masked language modeling objective. In cases where this model failed to predict one of the available options, another Roberta-Large model trained as a binary classifier was used to predict correct and incorrect options. We used passage summary generated by Pegasus model and question as inputs. Our best solution was an ensemble of these 2 systems. We achieved an accuracy of 86.22{\%} on subtask 1 and 87.10{\%} on subtask 2.
PINGAN Omini-Sinitic at SemEval-2021 Task 4:Reading Comprehension of Abstract Meaning
Authors: Ye Wang, Yanmeng Wang, Haijun Zhu, Bo Zeng, Zhenghong Hao, Shaojun Wang, Jing Xiao
Date: 2021-08-01 00:00:00
Proceeding: SEMEVAL 2021
Tasks: Denoising, Language Modeling, Language Modelling, Reading Comprehension

This paper describes the winning system for subtask 2 and the second-placed system for subtask 1 in SemEval 2021 Task 4: ReadingComprehension of Abstract Meaning. We propose to use pre-trianed Electra discriminator to choose the best abstract word from five candidates. An upper attention and auto denoising mechanism is introduced to process the long sequences. The experiment results demonstrate that this contribution greatly facilitatesthe contextual language modeling in reading comprehension task. The ablation study is also conducted to show the validity of our proposed methods.
YNU-HPCC at SemEval-2021 Task 5: Using a Transformer-based Model with Auxiliary Information for Toxic Span Detection
Authors: Ruijun Chen, Jin Wang, Xuejie Zhang
Date: 2021-08-01 00:00:00
Proceeding: SEMEVAL 2021
Tasks: Toxic Spans Detection

Toxic span detection requires the detection of spans that make a text toxic instead of simply classifying the text. In this paper, a transformer-based model with auxiliary information is proposed for SemEval-2021 Task 5. The proposed model was implemented based on the BERT-CRF architecture. It consists of three parts: a transformer-based model that can obtain the token representation, an auxiliary information module that combines features from different layers, and an output layer used for the classification. Various BERT-based models, such as BERT, ALBERT, RoBERTa, and XLNET, were used to learn contextual representations. The predictions of these models were assembled to improve the sequence labeling tasks by using a voting strategy. Experimental results showed that the introduced auxiliary information can improve the performance of toxic spans detection. The proposed model ranked 5th of 91 in the competition. The code of this study is available at https://github.com/Chenrj233/semeval2021{\_}task5
GHOST at SemEval-2021 Task 5: Is explanation all you need?
Authors: Kamil Pluci{\'n}ski, Hanna Klimczak
Date: 2021-08-01 00:00:00
Proceeding: SEMEVAL 2021
Tasks: All, Binary Classification, Classification, Explainable Artificial Intelligence (XAI), token-classification, Token Classification, Toxic Spans Detection

This paper discusses different approaches to the Toxic Spans Detection task. The problem posed by the task was to determine which words contribute mostly to recognising a document as toxic. As opposed to binary classification of entire texts, word-level assessment could be of great use during comment moderation, also allowing for a more in-depth comprehension of the model{'}s predictions. As the main goal was to ensure transparency and understanding, this paper focuses on the current state-of-the-art approaches based on the explainable AI concepts and compares them to a supervised learning solution with word-level labels. The work consists of two xAI approaches that automatically provide the explanation for models trained for binary classification of toxic documents: an LSTM model with attention as a model-specific approach and the Shapley values for interpreting BERT predictions as a model-agnostic method. The competing approach considers this problem as supervised token classification, where models like BERT and its modifications were tested. The paper aims to explore, compare and assess the quality of predictions for different methods on the task. The advantages of each approach and further research direction are also discussed.
GoldenWind at SemEval-2021 Task 5: Orthrus - An Ensemble Approach to Identify Toxicity
Authors: Marco Palomino, Dawid Grad, James Bedwell
Date: 2021-08-01 00:00:00
Proceeding: SEMEVAL 2021

Many new developments to detect and mitigate toxicity are currently being evaluated. We are particularly interested in the correlation between toxicity and the emotions expressed in online posts. While toxicity may be disguised by amending the wording of posts, emotions will not. Therefore, we describe here an ensemble method to identify toxicity and classify the emotions expressed on a corpus of annotated posts published by Task 5 of SemEval 2021{--}our analysis shows that the majority of such posts express anger, sadness and fear. Our method to identify toxicity combines a lexicon-based approach, which on its own achieves an F1 score of 61.07{\%}, with a supervised learning approach, which on its own achieves an F1 score of 60{\%}. When both methods are combined, the ensemble achieves an F1 score of 66.37{\%}.
LISAC FSDM USMBA at SemEval-2021 Task 5: Tackling Toxic Spans Detection Challenge with Supervised SpanBERT-based Model and Unsupervised LIME-based Model
Authors: Abdessamad Benlahbib, Ahmed Alami, Hamza Alami
Date: 2021-08-01 00:00:00
Proceeding: SEMEVAL 2021
Tasks: Toxic Spans Detection

Toxic spans detection is an emerging challenge that aims to find toxic spans within a toxic text. In this paper, we describe our solutions to tackle toxic spans detection. The first solution, which follows a supervised approach, is based on SpanBERT model. This latter is intended to better embed and predict spans of text. The second solution, which adopts an unsupervised approach, combines linear support vector machine with the Local Interpretable Model-Agnostic Explanations (LIME). This last is used to interpret predictions of learning-based models. Our supervised model outperformed the unsupervised model and achieved the f-score of 67,84{\%} (ranked 22/85) in Task 5 at SemEval-2021: Toxic Spans Detection.
HITMI\&T at SemEval-2021 Task 5: Integrating Transformer and CRF for Toxic Spans Detection
Authors: Chenyi Wang, Tianshu Liu, Tiejun Zhao
Date: 2021-08-01 00:00:00
Proceeding: SEMEVAL 2021
Tasks: Toxic Spans Detection

This paper introduces our system at SemEval-2021 Task 5: Toxic Spans Detection. The task aims to accurately locate toxic spans within a text. Using BIO tagging scheme, we model the task as a token-level sequence labeling task. Our system uses a single model built on the model of multi-layer bidirectional transformer encoder. And we introduce conditional random field (CRF) to make the model learn the constraints between tags. We use ERNIE as pre-trained model, which is more suitable for the task accroding to our experiments. In addition, we use adversarial training with the fast gradient method (FGM) to improve the robustness of the system. Our system obtains 69.85{\%} F1 score, ranking 3rd for the official evaluation.
AStarTwice at SemEval-2021 Task 5: Toxic Span Detection Using RoBERTa-CRF, Domain Specific Pre-Training and Self-Training
Authors: Thakur Ashutosh Suman, Abhinav Jain
Date: 2021-08-01 00:00:00
Proceeding: SEMEVAL 2021
Tasks: Language Modeling, Language Modelling, Toxic Spans Detection

This paper describes our contribution to SemEval-2021 Task 5: Toxic Spans Detection. Our solution is built upon RoBERTa language model and Conditional Random Fields (CRF). We pre-trained RoBERTa on Civil Comments dataset, enabling it to create better contextual representation for this task. We also employed the semi-supervised learning technique of self-training, which allowed us to extend our training dataset. In addition to these, we also identified some pre-processing steps that significantly improved our F1 score. Our proposed system achieved a rank of 41 with an F1 score of 66.16{\%}.
NLP\_UIOWA at Semeval-2021 Task 5: Transferring Toxic Sets to Tag Toxic Spans
Authors: Jonathan Rusert
Date: 2021-08-01 00:00:00
Proceeding: SEMEVAL 2021
Tasks: TAG

We leverage a BLSTM with attention to identify toxic spans in texts. We explore different dimensions which affect the model{'}s performance. The first dimension explored is the toxic set the model is trained on. Besides the provided dataset, we explore the transferability of 5 different toxic related sets, including offensive, toxic, abusive, and hate sets. We find that the solely offensive set shows the highest promise of transferability. The second dimension we explore is methodology, including leveraging attention, employing a greedy remove method, using a frequency ratio, and examining hybrid combinations of multiple methods. We conduct an error analysis to examine which types of toxic spans were missed and which were wrongly inferred as toxic along with the main reasons why they occurred. Finally, we extend our method via ensembles, which achieves our highest F1 score of 55.1.
S-NLP at SemEval-2021 Task 5: An Analysis of Dual Networks for Sequence Tagging
Authors: Viet Anh Nguyen, Tam Minh Nguyen, Huy Quang Dao, Quang Huu Pham
Date: 2021-08-01 00:00:00
Proceeding: SEMEVAL 2021
Tasks: Language Modeling, Language Modelling, Toxic Spans Detection

The SemEval 2021 task 5: Toxic Spans Detection is a task of identifying considered-toxic spans in text, which provides a valuable, automatic tool for moderating online contents. This paper represents the second-place method for the task, an ensemble of two approaches. While one approach relies on combining different embedding methods to extract diverse semantic and syntactic representations of words in context; the other utilizes extra data with a slightly customized Self-training, a semi-supervised learning technique, for sequence tagging problems. Both of our architectures take advantage of a strong language model, which was fine-tuned on a toxic classification task. Although experimental evidence indicates higher effectiveness of the first approach than the second one, combining them leads to our best results of 70.77 F1-score on the test dataset.
UAntwerp at SemEval-2021 Task 5: Spans are Spans, stacking a binary word level approach to toxic span detection
Authors: Ben Burtenshaw, Mike Kestemont
Date: 2021-08-01 00:00:00
Proceeding: SEMEVAL 2021

This paper describes the system developed by the Antwerp Centre for Digital humanities and literary Criticism [UAntwerp] for toxic span detection. We used a stacked generalisation ensemble of five component models, with two distinct interpretations of the task. Two models attempted to predict binary word toxicity based on ngram sequences, whilst 3 categorical span based models were trained to predict toxic token labels based on complete sequence tokens. The five models{'} predictions were ensembled within an LSTM model. As well as describing the system, we perform error analysis to explore model performance in relation to textual features. The system described in this paper scored 0.6755 and ranked 26th.
hub at SemEval-2021 Task 5: Toxic Span Detection Based on Word-Level Classification
Authors: Bo Huang, Yang Bai, Xiaobing Zhou
Date: 2021-08-01 00:00:00
Proceeding: SEMEVAL 2021
Tasks: Toxic Spans Detection

This article introduces the system description of the hub team, which explains the related work and experimental results of our team{'}s participation in SemEval 2021 Task 5: Toxic Spans Detection. The data for this shared task comes from some posts on the Internet. The task goal is to identify the toxic content contained in these text data. We need to find the span of the toxic text in the text data as accurately as possible. In the same post, the toxic text may be one paragraph or multiple paragraphs. Our team uses a classification scheme based on word-level to accomplish this task. The system we used to submit the results is ALBERT+BILSTM+CRF. The result evaluation index of the task submission is the F1 score, and the final score of the prediction result of the test set submitted by our team is 0.6640226029.
Sefamerve ARGE at SemEval-2021 Task 5: Toxic Spans Detection Using Segmentation Based 1-D Convolutional Neural Network Model
Authors: Selman Delil, Birol Kuyumcu, C{\"u}neyt Aksakall{\i}
Date: 2021-08-01 00:00:00
Proceeding: SEMEVAL 2021
Tasks: Segmentation, Semantic Segmentation, Text Segmentation, Toxic Spans Detection

This paper describes our contribution to SemEval-2021 Task 5: Toxic Spans Detection. Our approach considers toxic spans detection as a segmentation problem. The system, Waw-unet, consists of a 1-D convolutional neural network adopted from U-Net architecture commonly applied for semantic segmentation. We customize existing architecture by adding a special network block considering for text segmentation, as an essential component of the model. We compared the model with two transformers-based systems RoBERTa and XLM-RoBERTa to see its performance against pre-trained language models. We obtained 0.6251 f1 score with Waw-unet while 0.6390 and 0.6601 with the compared models respectively.
SkoltechNLP at SemEval-2021 Task 5: Leveraging Sentence-level Pre-training for Toxic Span Detection
Authors: David Dale, Igor Markov, Varvara Logacheva, Olga Kozlova, Nikita Semenov, Alexander Panchenko
Date: 2021-08-01 00:00:00
Proceeding: SEMEVAL 2021
Tasks: Sentence, Toxic Spans Detection

This work describes the participation of the Skoltech NLP group team (Sk) in the Toxic Spans Detection task at SemEval-2021. The goal of the task is to identify the most toxic fragments of a given sentence, which is a binary sequence tagging problem. We show that fine-tuning a RoBERTa model for this problem is a strong baseline. This baseline can be further improved by pre-training the RoBERTa model on a large dataset labeled for toxicity at the sentence level. While our solution scored among the top 20{\%} participating models, it is only 2 points below the best result. This suggests the viability of our approach.