pwc-archive/papers-with-abstracts huggingface dataset:
\#NonDicevoSulSerio at SemEval-2018 Task 3: Exploiting Emojis and Affective Content for Irony Detection in English Tweets
Authors: Endang Wahyu Pamungkas, Viviana Patti
Date: 2018-06-01 00:00:00
Proceeding: SEMEVAL 2018 6
Tasks: Binary Classification, General Classification, Sentiment Analysis

This paper describes the participation of the {\#}NonDicevoSulSerio team at SemEval2018-Task3, which focused on Irony Detection in English Tweets and was articulated in two tasks addressing the identification of irony at different levels of granularity. We participated in both tasks proposed: Task A is a classical binary classification task to determine whether a tweet is ironic or not, while Task B is a multiclass classification task devoted to distinguish different types of irony, where systems have to predict one out of four labels describing verbal irony by clash, other verbal irony, situational irony, and non-irony. We addressed both tasks by proposing a model built upon a well-engineered features set involving both syntactic and lexical features, and a wide range of affective-based features, covering different facets of sentiment and emotions. The use of new features for taking advantage of the affective information conveyed by emojis has been analyzed. On this line, we also tried to exploit the possible incongruity between sentiment expressed in the text and in the emojis included in a tweet. We used a Support Vector Machine classifier, and obtained promising results. We also carried on experiments in an unconstrained setting.
KNU CI System at SemEval-2018 Task4: Character Identification by Solving Sequence-Labeling Problem
Authors: Cheoneum Park, Heejun Song, Chang-Ki Lee
Date: 2018-06-01 00:00:00
Proceeding: SEMEVAL 2018 6
Tasks: Coreference Resolution, Decoder, Entity Linking, Position

Character identification is an entity-linking task that finds words referring to the same person among the nouns mentioned in a conversation and turns them into one entity. In this paper, we define a sequence-labeling problem to solve character identification, and propose an attention-based recurrent neural network (RNN) encoder{--}decoder model. The in-put document for character identification on multiparty dialogues consists of several conversations, which increase the length of the input sequence. The RNN encoder{--}decoder model suffers from poor performance when the length of the input sequence is long. To solve this problem, we propose applying position encoding and the self-matching network to the RNN encoder{--}decoder model. Our experimental results demonstrate that of the four models proposed, Model 2 showed an F1 score of 86.00{\%} and a label accuracy of 85.10{\%} at the scene-level.
NewsReader at SemEval-2018 Task 5: Counting events by reasoning over event-centric-knowledge-graphs
Authors: Piek Vossen
Date: 2018-06-01 00:00:00
Proceeding: SEMEVAL 2018 6
Tasks: Knowledge Graphs

In this paper, we describe the participation of the NewsReader system in the SemEval-2018 Task 5 on Counting Events and Participants in the Long Tail. NewsReader is a generic unsupervised text processing system that detects events with participants, time and place to generate Event Centric Knowledge Graphs (ECKGs). We minimally adapted these ECKGs to establish a baseline performance for the task. We first use the ECKGs to establish which documents report on the same incident and what event mentions are coreferential. Next, we aggregate ECKGs across coreferential mentions and use the aggregated knowledge to answer the questions of the task. Our participation tests the quality of NewsReader to create ECKGs, as well as the potential of ECKGs to establish event identity and reason over the result to answer the task queries.
FEUP at SemEval-2018 Task 5: An Experimental Study of a Question Answering System
Authors: Carla Abreu, Eug{\'e}nio Oliveira
Date: 2018-06-01 00:00:00
Proceeding: SEMEVAL 2018 6
Tasks: Articles, Named Entity Recognition (NER), Question Answering

We present the approach developed at the Faculty of Engineering of the University of Porto to participate in SemEval-2018 Task 5: Counting Events and Participants within Highly Ambiguous Data covering a very long tail.The work described here presents the experimental system developed to extract entities from news articles for the sake of Question Answering. We propose a supervised learning approach to enable the recognition of two different types of entities: Locations and Participants. We also discuss the use of distance-based algorithms (using Levenshtein distance and Q-grams) for the detection of documents{'} closeness based on the entities extracted. For the experiments, we also used a multi-agent system that improved the performance.
NAI-SEA at SemEval-2018 Task 5: An Event Search System
Authors: Yingchi Liu, Quanzhi Li, Luo Si
Date: 2018-06-01 00:00:00
Proceeding: SEMEVAL 2018 6
Tasks: Clustering, Retrieval

In this paper, we describe Alibaba{'}s participating system in the semEval-2018 Task5: Counting Events and Participants in the Long Tail. We designed and implemented a pipeline system that consists of components to extract question properties and document features, document event category classifications, document retrieval and document clustering. To retrieve the majority of the relevant documents, we carefully designed our system to extract key information from each question and document pair. After retrieval, we perform further document clustering to count the number of events. The task contains 3 subtasks, on which we achieved F1 score of 78.33, 50.52, 63.59 , respectively, for document level retrieval. Our system ranks first in all the three subtasks on document level retrieval, and it also ranks first in incident-level evaluation by RSME measure in subtask 3.
SemEval-2018 Task 7: Semantic Relation Extraction and Classification in Scientific Papers
Authors: Kata G{\'a}bor, Davide Buscaldi, Anne-Kathrin Schumann, Behrang Qasemizadeh, Ha{\"\i}fa Zargayouna, Thierry Charnois
Date: 2018-06-01 00:00:00
Proceeding: SEMEVAL 2018 6
Tasks: Classification, General Classification, Keyword Extraction, Relation, Relation Classification, Relation Extraction

This paper describes the first task on semantic relation extraction and classification in scientific paper abstracts at SemEval 2018. The challenge focuses on domain-specific semantic relations and includes three different subtasks. The subtasks were designed so as to compare and quantify the effect of different pre-processing steps on the relation classification results. We expect the task to be relevant for a broad range of researchers working on extracting specialized knowledge from domain corpora, for example but not limited to scientific or bio-medical information extraction. The task attracted a total of 32 participants, with 158 submissions across different scenarios.
SemEval-2018 Task 8: Semantic Extraction from CybersecUrity REports using Natural Language Processing (SecureNLP)
Authors: Ph, Peter i, Amila Silva, Wei Lu
Date: 2018-06-01 00:00:00
Proceeding: SEMEVAL 2018 6
Tasks: Malware Detection

This paper describes the SemEval 2018 shared task on semantic extraction from cybersecurity reports, which is introduced for the first time as a shared task on SemEval. This task comprises four SubTasks done incrementally to predict the characteristics of a specific malware using cybersecurity reports. To the best of our knowledge, we introduce the world{'}s largest publicly available dataset of annotated malware reports in this task. This task received in total 18 submissions from 9 participating teams.
DM\_NLP at SemEval-2018 Task 8: neural sequence labeling with linguistic features
Authors: Chunping Ma, Huafei Zheng, Pengjun Xie, Chen Li, Linlin Li, Luo Si
Date: 2018-06-01 00:00:00
Proceeding: SEMEVAL 2018 6
Tasks: Chinese Word Segmentation, Chunking, Clustering, Dependency Parsing, NER, POS, Sentence

This paper describes our submissions for SemEval-2018 Task 8: Semantic Extraction from CybersecUrity REports using NLP. The DM{\_}NLP participated in two subtasks: SubTask 1 classifies if a sentence is useful for inferring malware actions and capabilities, and SubTask 2 predicts token labels ({``}Action{''}, {``}Entity{''}, {``}Modifier{''} and {``}Others{''}) for a given malware-related sentence. Since we leverage results of Subtask 2 directly to infer the result of Subtask 1, the paper focus on the system solving Subtask 2. By taking Subtask 2 as a sequence labeling task, our system relies on a recurrent neural network named BiLSTM-CNN-CRF with rich linguistic features, such as POS tags, dependency parsing labels, chunking labels, NER labels, Brown clustering. Our system achieved the highest F1 score in both token level and phrase level.
SemEval-2018 Task 9: Hypernym Discovery
Authors: Jose Camacho-Collados, Claudio Delli Bovi, Luis Espinosa-Anke, Sergio Oramas, Tommaso Pasini, Enrico Santus, Vered Shwartz, Roberto Navigli, Horacio Saggion
Date: 2018-06-01 00:00:00
Proceeding: SEMEVAL 2018 6
Tasks: Binary Classification, Hypernym Discovery, Natural Language Inference, Question Answering

This paper describes the SemEval 2018 Shared Task on Hypernym Discovery. We put forward this task as a complementary benchmark for modeling hypernymy, a problem which has traditionally been cast as a binary classification task, taking a pair of candidate words as input. Instead, our reformulated task is defined as follows: given an input term, retrieve (or discover) its suitable hypernyms from a target corpus. We proposed five different subtasks covering three languages (English, Spanish, and Italian), and two specific domains of knowledge in English (Medical and Music). Participants were allowed to compete in any or all of the subtasks. Overall, a total of 11 teams participated, with a total of 39 different systems submitted through all subtasks. Data, results and further information about the task can be found at \url{https://competitions.codalab.org/competitions/17119}.
CRIM at SemEval-2018 Task 9: A Hybrid Approach to Hypernym Discovery
Authors: Gabriel Bernier-Colborne, Caroline Barri{\`e}re
Date: 2018-06-01 00:00:00
Proceeding: SEMEVAL 2018 6
Tasks: Hypernym Discovery, Relation Extraction

This report describes the system developed by the CRIM team for the hypernym discovery task at SemEval 2018. This system exploits a combination of supervised projection learning and unsupervised pattern-based hypernym discovery. It was ranked first on the 3 sub-tasks for which we submitted results.
SemEval-2018 Task 10: Capturing Discriminative Attributes
Authors: Alicia Krebs, Aless Lenci, ro, Denis Paperno
Date: 2018-06-01 00:00:00
Proceeding: SEMEVAL 2018 6
Tasks: Attribute, Semantic Similarity, Semantic Textual Similarity, Word Sense Disambiguation

This paper describes the SemEval 2018 Task 10 on Capturing Discriminative Attributes. Participants were asked to identify whether an attribute could help discriminate between two concepts. For example, a successful system should determine that {`}urine{'} is a discriminating feature in the word pair {`}kidney{'}, {`}bone{'}. The aim of the task is to better evaluate the capabilities of state of the art semantic models, beyond pure semantic similarity. The task attracted submissions from 21 teams, and the best system achieved a 0.75 F1 score.
SUNNYNLP at SemEval-2018 Task 10: A Support-Vector-Machine-Based Method for Detecting Semantic Difference using Taxonomy and Word Embedding Features
Authors: Sunny Lai, Kwong Sak Leung, Yee Leung
Date: 2018-06-01 00:00:00
Proceeding: SEMEVAL 2018 6
Tasks: Dialogue State Tracking, Question Answering, Relation Extraction, Semantic Textual Similarity

We present SUNNYNLP, our system for solving SemEval 2018 Task 10: {``}Capturing Discriminative Attributes{''}. Our Support-Vector-Machine(SVM)-based system combines features extracted from pre-trained embeddings and statistical information from Is-A taxonomy to detect semantic difference of concepts pairs. Our system is demonstrated to be effective in detecting semantic difference and is ranked 1st in the competition in terms of F1 measure. The open source of our code is coined SUNNYNLP.
SemEval-2018 Task 11: Machine Comprehension Using Commonsense Knowledge
Authors: Simon Ostermann, Michael Roth, Ashutosh Modi, Stefan Thater, Manfred Pinkal
Date: 2018-06-01 00:00:00
Proceeding: SEMEVAL 2018 6
Tasks: Reading Comprehension, Story Completion

This report summarizes the results of the SemEval 2018 task on machine comprehension using commonsense knowledge. For this machine comprehension task, we created a new corpus, MCScript. It contains a high number of questions that require commonsense knowledge for finding the correct answer. 11 teams from 4 different countries participated in this shared task, most of them used neural approaches. The best performing system achieves an accuracy of 83.95{\%}, outperforming the baselines by a large margin, but still far from the human upper bound, which was found to be at 98{\%}.
SemEval-2018 Task 12: The Argument Reasoning Comprehension Task
Authors: Ivan Habernal, Henning Wachsmuth, Iryna Gurevych, Benno Stein
Date: 2018-06-01 00:00:00
Proceeding: SEMEVAL 2018 6
Tasks: Common Sense Reasoning

A natural language argument is composed of a claim as well as reasons given as premises for the claim. The warrant explaining the reasoning is usually left implicit, as it is clear from the context and common sense. This makes a comprehension of arguments easy for humans but hard for machines. This paper summarizes the first shared task on argument reasoning comprehension. Given a premise and a claim along with some topic information, the goal was to automatically identify the correct warrant among two candidates that are plausible and lexically close, but in fact imply opposite claims. We describe the dataset with 1970 instances that we built for the task, and we outline the 21 computational approaches that participated, most of which used neural networks. The results reveal the complexity of the task, with many approaches hardly improving over the random accuracy of about 0.5. Still, the best observed accuracy (0.712) underlines the principle feasibility of identifying warrants. Our analysis indicates that an inclusion of external knowledge is key to reasoning comprehension.
GIST at SemEval-2018 Task 12: A network transferring inference knowledge to Argument Reasoning Comprehension task
Authors: HongSeok Choi, Hyunju Lee
Date: 2018-06-01 00:00:00
Proceeding: SEMEVAL 2018 6
Tasks: Common Sense Reasoning, Natural Language Inference, Transfer Learning

This paper describes our GIST team system that participated in SemEval-2018 Argument Reasoning Comprehension task (Task 12). Here, we address two challenging factors: unstated common senses and two lexically close warrants that lead to contradicting claims. A key idea for our system is full use of transfer learning from the Natural Language Inference (NLI) task to this task. We used Enhanced Sequential Inference Model (ESIM) to learn the NLI dataset. We describe how to use ESIM for transfer learning to choose correct warrant through a proposed system. We show comparable results through ablation experiments. Our system ranked 1st among 22 systems, outperforming all the systems more than 10{\%}.
LightRel at SemEval-2018 Task 7: Lightweight and Fast Relation Classification
Authors: Tyler Renslow, G{\"u}nter Neumann
Date: 2018-06-01 00:00:00
Proceeding: SEMEVAL 2018 6
Tasks: Feature Engineering, General Classification, Relation, Relation Classification, Relation Extraction, Word Embeddings

We present LightRel, a lightweight and fast relation classifier. Our goal is to develop a high baseline for different relation extraction tasks. By defining only very few data-internal, word-level features and external knowledge sources in the form of word clusters and word embeddings, we train a fast and simple linear classifier
The UWNLP system at SemEval-2018 Task 7: Neural Relation Extraction Model with Selectively Incorporated Concept Embeddings
Authors: Yi Luan, Mari Ostendorf, Hannaneh Hajishirzi
Date: 2018-06-01 00:00:00
Proceeding: SEMEVAL 2018 6
Tasks: Classification, General Classification, Relation, Relation Classification, Relation Extraction, Word Embeddings

This paper describes our submission for SemEval 2018 Task 7 shared task on semantic relation extraction and classification in scientific papers. Our model is based on the end-to-end relation extraction model of (Miwa and Bansal, 2016) with several enhancements such as character-level encoding attention mechanism on selecting pretrained concept candidate embeddings. Our official submission ranked the second in relation classification task (Subtask 1.1 and Subtask 2 Senerio 2), and the first in the relation extraction task (Subtask 2 Scenario 1).
UC3M-NII Team at SemEval-2018 Task 7: Semantic Relation Classification in Scientific Papers via Convolutional Neural Network
Authors: V{\'\i}ctor Su{\'a}rez-Paniagua, Isabel Segura-Bedmar, Akiko Aizawa
Date: 2018-06-01 00:00:00
Proceeding: SEMEVAL 2018 6
Tasks: Clustering, General Classification, Relation, Relation Classification, Relation Extraction, Sentence Classification, Sentiment Analysis

This paper reports our participation for SemEval-2018 Task 7 on extraction and classification of relationships between entities in scientific papers. Our approach is based on the use of a Convolutional Neural Network (CNN) trained on350 abstract with manually annotated entities and relations. Our hypothesis is that this deep learning model can be applied to extract and classify relations between entities for scientific papers at the same time. We use the Part-of-Speech and the distances to the target entities as part of the embedding for each word and we blind all the entities by marker names. In addition, we use sampling techniques to overcome the imbalance issues of this dataset. Our architecture obtained an F1-score of 35.4{\%} for the relation extraction task and 18.5{\%} for the relation classification task with a basic configuration of the one step CNN.
MIT-MEDG at SemEval-2018 Task 7: Semantic Relation Classification via Convolution Neural Network
Authors: Di Jin, Franck Dernoncourt, Elena Sergeeva, Matthew McDermott, Geeticka Chauhan
Date: 2018-06-01 00:00:00
Proceeding: SEMEVAL 2018 6
Tasks: Articles, Common Sense Reasoning, Data Augmentation, General Classification, Relation, Relation Classification, Relation Extraction, Sentence, Word Embeddings

SemEval 2018 Task 7 tasked participants to build a system to classify two entities within a sentence into one of the 6 possible relation types. We tested 3 classes of models: Linear classifiers, Long Short-Term Memory (LSTM) models, and Convolutional Neural Network (CNN) models. Ultimately, the CNN model class proved most performant, so we specialized to this model for our final submissions. We improved performance beyond a vanilla CNN by including a variant of negative sampling, using custom word embeddings learned over a corpus of ACL articles, training over corpora of both tasks 1.1 and 1.2, using reversed feature, using part of context words beyond the entity pairs and using ensemble methods to improve our final predictions. We also tested attention based pooling, up-sampling, and data augmentation, but none improved performance. Our model achieved rank 6 out of 28 (macro-averaged F1-score: 72.7) in subtask 1.1, and rank 4 out of 20 (macro F1: 80.6) in subtask 1.2.
IRCMS at SemEval-2018 Task 7 : Evaluating a basic CNN Method and Traditional Pipeline Method for Relation Classification
Authors: Zhongbo Yin, Zhunchen Luo, Wei Luo, Mao Bin, Changhai Tian, Yuming Ye, Shuai Wu
Date: 2018-06-01 00:00:00
Proceeding: SEMEVAL 2018 6
Tasks: General Classification, Named Entity Recognition (NER), Relation, Relation Classification, Relation Extraction, Semantic Textual Similarity

This paper presents our participation for sub-task1 (1.1 and 1.2) in SemEval 2018 task 7: Semantic Relation Extraction and Classification in Scientific Papers (G{\'a}bor et al., 2018). We experimented on this task with two methods: CNN method and traditional pipeline method. We use the context between two entities (included) as input information for both methods, which extremely reduce the noise effect. For the CNN method, we construct a simple convolution neural network to automatically learn features from raw texts without any manual processing. Moreover, we use the softmax function to classify the entity pair into a specific relation category. For the traditional pipeline method, we use the Hackabout method as a representation which is described in section3.5. The CNN method{'}s result is much better than traditional pipeline method (49.1{\%} vs. 42.3{\%} and 71.1{\%} vs. 54.6{\%} ).
Bf3R at SemEval-2018 Task 7: Evaluating Two Relation Extraction Tools for Finding Semantic Relations in Biomedical Abstracts
Authors: Mariana Neves, Daniel Butzke, Gilbert Sch{\"o}nfelder, Barbara Grune
Date: 2018-06-01 00:00:00
Proceeding: SEMEVAL 2018 6
Tasks: Information Retrieval, Relation Extraction

Automatic extraction of semantic relations from text can support finding relevant information from scientific publications. We describe our participation in Task 7 of SemEval-2018 for which we experimented with two relations extraction tools - jSRE and TEES - for the extraction and classification of six relation types. The results we obtained with TEES were significantly superior than those with jSRE (33.4{\%} vs. 30.09{\%} and 20.3{\%} vs. 16{\%}). Additionally, we utilized the model trained with TEES for extracting semantic relations from biomedical abstracts, for which we present a preliminary evaluation.
Texterra at SemEval-2018 Task 7: Exploiting Syntactic Information for Relation Extraction and Classification in Scientific Papers
Authors: Andrey Sysoev, Vladimir Mayorov
Date: 2018-06-01 00:00:00
Proceeding: SEMEVAL 2018 6
Tasks: Classification, General Classification, Relation Classification, Relation Extraction, Sentence

In this work we evaluate applicability of entity pair models and neural network architectures for relation extraction and classification in scientific papers at SemEval-2018. We carry out experiments with representing entity pairs through sentence tokens and through shortest path in dependency tree, comparing approaches based on convolutional and recurrent neural networks. With convolutional network applied to shortest path in dependency tree we managed to be ranked eighth in subtask 1.1 ({``}clean data{''}), ninth in 1.2 ({``}noisy data{''}). Similar model applied to separate parts of the shortest path was mounted to ninth (extraction track) and seventh (classification track) positions in subtask 2 ranking.
UniMa at SemEval-2018 Task 7: Semantic Relation Extraction and Classification from Scientific Publications
Authors: Thorsten Keiper, Zhonghao Lyu, Sara Pooladzadeh, Yuan Xu, Jingyi Zhang, Anne Lauscher, Simone Paolo Ponzetto
Date: 2018-06-01 00:00:00
Proceeding: SEMEVAL 2018 6
Tasks: General Classification, Relation, Relation Classification, Relation Extraction

Large repositories of scientific literature call for the development of robust methods to extract information from scholarly papers. This problem is addressed by the SemEval 2018 Task 7 on extracting and classifying relations found within scientific publications. In this paper, we present a feature-based and a deep learning-based approach to the task and discuss the results of the system runs that we submitted for evaluation.