pwc-archive/papers-with-abstracts huggingface dataset:
DUTH at SemEval-2019 Task 8: Part-Of-Speech Features for Question Classification
Authors: Anastasios Bairaktaris, Symeon Symeonidis, Avi Arampatzis
Date: 2019-06-01 00:00:00
Proceeding: SEMEVAL 2019 6
Tasks: Community Question Answering, Fact Checking, feature selection, General Classification, POS, Question Answering

This report describes the methods employed by the Democritus University of Thrace (DUTH) team for participating in SemEval-2019 Task 8: Fact Checking in Community Question Answering Forums. Our team dealt only with Subtask A: Question Classification. Our approach was based on shallow natural language processing (NLP) pre-processing techniques to reduce the noise in data, feature selection methods, and supervised machine learning algorithms such as NearestCentroid, Perceptron, and LinearSVC. To determine the essential features, we were aided by exploratory data analysis and visualizations. In order to improve classification accuracy, we developed a customized list of stopwords, retaining some opinion- and fact-denoting common function words which would have been removed by standard stoplisting. Furthermore, we examined the usefulness of part-of-speech (POS) categories for the task; by trying to remove nouns and adjectives, we found some evidence that verbs are a valuable POS category for the opinion question class.
Fermi at SemEval-2019 Task 8: An elementary but effective approach to Question Discernment in Community QA Forums
Authors: Bakhtiyar Syed, Vijayasaradhi Indurthi, Manish Shrivastava, Manish Gupta, Vasudeva Varma
Date: 2019-06-01 00:00:00
Proceeding: SEMEVAL 2019 6
Tasks: Community Question Answering, Question Answering, Sentence

Online Community Question Answering Forums (cQA) have gained massive popularity within recent years. The rise in users for such forums have led to the increase in the need for automated evaluation for question comprehension and fact evaluation of the answers provided by various participants in the forum. Our team, \textbf{Fermi}, participated in sub-task A of Task 8 at SemEval 2019 - which tackles the first problem in the pipeline of factual evaluation in cQA forums, i.e., deciding whether a posed question asks for a factual information, an opinion/advice or is just socializing. This information is highly useful in segregating factual questions from non-factual ones which highly helps in organizing the questions into useful categories and trims down the problem space for the next task in the pipeline for fact evaluation among the available answers. Our system uses the embeddings obtained from Universal Sentence Encoder combined with XGBoost for the classification sub-task A. We also evaluate other combinations of embeddings and off-the-shelf machine learning algorithms to demonstrate the efficacy of the various representations and their combinations. Our results across the evaluation test set gave an accuracy of 84{\%} and received the first position in the final standings judged by the organizers.
SolomonLab at SemEval-2019 Task 8: Question Factuality and Answer Veracity Prediction in Community Forums
Authors: Ankita Gupta, Sudeep Kumar Sahoo, Divya Prakash, Rohit R. R, Vertika Srivastava, Yeon Hyang Kim
Date: 2019-06-01 00:00:00
Proceeding: SEMEVAL 2019 6
Tasks: Community Question Answering, Data Augmentation, Fact Checking, Question Answering

We describe our system for SemEval-2019, Task 8 on {``}Fact-Checking in Community Question Answering Forums (cQA){''}. cQA forums are very prevalent nowadays, as they provide an effective means for communities to share knowledge. Unfortunately, this shared information is not always factual and fact-verified. In this task, we aim to identify factual questions posted on cQA and verify the veracity of answers to these questions. Our approach relies on data augmentation and aggregates cues from several dimensions such as semantics, linguistics, syntax, writing style and evidence obtained from trusted external sources. In subtask A, our submission is ranked 3rd, with an accuracy of 83.14{\%}. Our current best solution stands 1st on the leaderboard with 88{\%} accuracy. In subtask B, our present solution is ranked 2nd, with 58.33{\%} MAP score.
TueFact at SemEval 2019 Task 8: Fact checking in community question answering forums: context matters
Authors: R{\'e}ka Juh{\'a}sz, Franziska Barbara Linnenschmidt, Teslin Roys
Date: 2019-06-01 00:00:00
Proceeding: SEMEVAL 2019 6
Tasks: Community Question Answering, Fact Checking, Question Answering

The SemEval 2019 Task 8 on Fact-Checking in community question answering forums aimed to classify questions into categories and verify the correctness of answers given on the QatarLiving public forum. The task was divided into two subtasks: the first classifying the question, the second the answers. The TueFact system described in this paper used different approaches for the two subtasks. Subtask A makes use of word vectors based on a bag-of-word-ngram model using up to trigrams. Predictions are done using multi-class logistic regression. The official SemEval result lists an accuracy of 0.60. Subtask B uses vectorized character n-grams up to trigrams instead. Predictions are done using a LSTM model and achieved an accuracy of 0.53 on the final SemEval Task 8 evaluation set.
YNU-HPCC at SemEval-2019 Task 8: Using A LSTM-Attention Model for Fact-Checking in Community Forums
Authors: Peng Liu, Jin Wang, Xue-jie Zhang
Date: 2019-06-01 00:00:00
Proceeding: SEMEVAL 2019 6
Tasks: Fact Checking

We propose a system that uses a long short-term memory with attention mechanism (LSTM-Attention) model to complete the task. The LSTM-Attention model uses two LSTM to extract the features of the question and answer pair. Then, each of the features is sequentially composed using the attention mechanism, concatenating the two vectors into one. Finally, the concatenated vector is used as input for the MLP and the MLP{'}s output layer uses the softmax function to classify the provided answers into three categories. This model is capable of extracting the features of the question and answer pair well. The results show that the proposed system outperforms the baseline algorithm.
DBMS-KU at SemEval-2019 Task 9: Exploring Machine Learning Approaches in Classifying Text as Suggestion or Non-Suggestion
Authors: Tirana Fatyanosa, Al Hafiz Akbar Maulana Siagian, Masayoshi Aritsugi
Date: 2019-06-01 00:00:00
Proceeding: SEMEVAL 2019 6
Tasks: General Classification, regression, Suggestion mining

This paper describes the participation of DBMS-KU team in the SemEval 2019 Task 9, that is, suggestion mining from online reviews and forums. To deal with this task, we explore several machine learning approaches, i.e., Random Forest (RF), Logistic Regression (LR), Multinomial Naive Bayes (MNB), Linear Support Vector Classification (LSVC), Sublinear Support Vector Classification (SSVC), Convolutional Neural Network (CNN), and Variable Length Chromosome Genetic Algorithm-Naive Bayes (VLCGA-NB). Our system obtains reasonable results of F1-Score 0.47 and 0.37 on the evaluation data in Subtask A and Subtask B, respectively. In particular, our obtained results outperform the baseline in Subtask A. Interestingly, the results seem to show that our system could perform well in classifying Non-suggestion class.
DS at SemEval-2019 Task 9: From Suggestion Mining with neural networks to adversarial cross-domain classification
Authors: Tobias Cabanski
Date: 2019-06-01 00:00:00
Proceeding: SEMEVAL 2019 6
Tasks: domain classification, General Classification, Suggestion mining, Word Embeddings

Suggestion Mining is the task of classifying sentences into suggestions or non-suggestions. SemEval-2019 Task 9 sets the task to mine suggestions from online texts. For each of the two subtasks, the classification has to be applied on a different domain. Subtask A addresses the domain of posts in suggestion online forums and comes with a set of training examples, that is used for supervised training. A combination of LSTM and CNN networks is constructed to create a model which uses BERT word embeddings as input features. For subtask B, the domain of hotel reviews is regarded. In contrast to subtask A, no labeled data for supervised training is provided, so that additional unlabeled data is taken to apply a cross-domain classification. This is done by using adversarial training of the three model parts label classifier, domain classifier and the shared feature representation. For subtask A, the developed model archives a F1-score of 0.7273, which is in the top ten of the leader board. The F1-score for subtask B is 0.8187 and is ranked in the top five of the submissions for that task.
Hybrid RNN at SemEval-2019 Task 9: Blending Information Sources for Domain-Independent Suggestion Mining
Authors: Aysu Ezen-Can, Ethem F. Can
Date: 2019-06-01 00:00:00
Proceeding: SEMEVAL 2019 6
Tasks: Sentiment Analysis, Suggestion mining

Social media has an increasing amount of information that both customers and companies can benefit from. These social media posts can include Tweets or be in the form of vocalization of complements and complaints (e.g., reviews) of a product or service. Researchers have been actively mining this invaluable information source to automatically generate insights. Mining sentiments of customer reviews is an example that has gained momentum due to its potential to gather information that customers are not happy about. Instead of reading millions of reviews, companies prefer sentiment analysis to obtain feedback and to improve their products or services. In this work, we aim to identify information that companies can act on, or other customers can utilize for making their own experience better. This is different from identifying if reviews of a product or service is negative, positive, or neutral. To that end, we classify sentences of a given review as suggestion or not suggestion so that readers of the reviews do not have to go through thousands of reviews but instead can focus on actionable items and applicable suggestions. To identify suggestions within reviews, we employ a hybrid approach that utilizes a recurrent neural network (RNN) along with rule-based features to build a domain-independent suggestion mining model. In this way, a model trained on electronics reviews is used to extract suggestions from hotel reviews.
INRIA at SemEval-2019 Task 9: Suggestion Mining Using SVM with Handcrafted Features
Authors: Ilia Markov, Eric Villemonte de la Clergerie
Date: 2019-06-01 00:00:00
Proceeding: SEMEVAL 2019 6
Tasks: Suggestion mining

We present the INRIA approach to the suggestion mining task at SemEval 2019. The task consists of two subtasks: suggestion mining under single-domain (Subtask A) and cross-domain (Subtask B) settings. We used the Support Vector Machines algorithm trained on handcrafted features, function words, sentiment features, digits, and verbs for Subtask A, and handcrafted features for Subtask B. Our best run archived a F1-score of 51.18{\%} on Subtask A, and ranked in the top ten of the submissions for Subtask B with 73.30{\%} F1-score.
Lijunyi at SemEval-2019 Task 9: An attention-based LSTM and ensemble of different models for suggestion mining from online reviews and forums
Authors: Junyi Li
Date: 2019-06-01 00:00:00
Proceeding: SEMEVAL 2019 6
Tasks: Suggestion mining

In this paper, we describe a suggestion mining system that participated in SemEval 2019 Task 9, SubTask A - Suggestion Mining from Online Reviews and Forums. Given some suggestions from online reviews and forums that can be classified into suggestion and non-suggestion classes. In this task, we combine the attention mechanism with the LSTM model, which is the final system we submitted. The final submission achieves 14th place in Task 9, SubTask A with the accuracy of 0.6776. After the challenge, we train a series of neural network models such as convolutional neural network(CNN), TextCNN, long short-term memory(LSTM) and C-LSTM. Finally, we make an ensemble on the predictions of these models and get a better result.
NTUA-ISLab at SemEval-2019 Task 9: Mining Suggestions in the wild
Authors: Rol Potamias, os Alex, Alex Neofytou, ros, Georgios Siolas
Date: 2019-06-01 00:00:00
Proceeding: SEMEVAL 2019 6
Tasks: Position, Suggestion mining

As online customer forums and product comparison sites increase their societal influence, users are actively expressing their opinions and posting their recommendations on their fellow customers online. However, systems capable of recognizing suggestions still lack in stability. Suggestion Mining, a novel and challenging field of Natural Language Processing, is increasingly gaining attention, aiming to track user advice on online forums. In this paper, a carefully designed methodology to identify customer-to-company and customer-to-customer suggestions is presented. The methodology implements a rule-based classifier using heuristic, lexical and syntactic patterns. The approach ranked at 5th and 1st position, achieving an f1-score of 0.749 and 0.858 for SemEval-2019/Suggestion Mining sub-tasks A and B, respectively. In addition, we were able to improve performance results by combining the rule-based classifier with a recurrent convolutional neural network, that exhibits an f1-score of 0.79 for subtask A.
OleNet at SemEval-2019 Task 9: BERT based Multi-Perspective Models for Suggestion Mining
Authors: Jiaxiang Liu, Shuohuan Wang, Yu Sun
Date: 2019-06-01 00:00:00
Proceeding: SEMEVAL 2019 6
Tasks: Sentence, Suggestion mining

This paper describes our system partici- pated in Task 9 of SemEval-2019: the task is focused on suggestion mining and it aims to classify given sentences into sug- gestion and non-suggestion classes in do- main specific and cross domain training setting respectively. We propose a multi- perspective architecture for learning rep- resentations by using different classical models including Convolutional Neural Networks (CNN), Gated Recurrent Units (GRU), Feed Forward Attention (FFA), etc. To leverage the semantics distributed in large amount of unsupervised data, we also have adopted the pre-trained Bidi- rectional Encoder Representations from Transformers (BERT) model as an en- coder to produce sentence and word rep- resentations. The proposed architecture is applied for both sub-tasks, and achieved f1-score of 0.7812 for subtask A, and 0.8579 for subtask B. We won the first and second place for the two tasks respec- tively in the final competition.
SSN-SPARKS at SemEval-2019 Task 9: Mining Suggestions from Online Reviews using Deep Learning Techniques on Augmented Data
Authors: Rajalakshmi S, Angel Suseelan, S Milton Rajendram, Mirnalinee T T
Date: 2019-06-01 00:00:00
Proceeding: SEMEVAL 2019 6
Tasks: Data Augmentation, feature selection, General Classification, Opinion Mining, Suggestion mining

This paper describes the work on mining the suggestions from online reviews and forums. Opinion mining detects whether the comments are positive, negative or neutral, while suggestion mining explores the review content for the possible tips or advice. The system developed by SSN-SPARKS team in SemEval-2019 for task 9 (suggestion mining) uses a rule-based approach for feature selection, SMOTE technique for data augmentation and deep learning technique (Convolutional Neural Network) for classification. We have compared the results with Random Forest classifier (RF) and MultiLayer Perceptron (MLP) model. Results show that the CNN model performs better than other models for both the subtasks.
Suggestion Miner at SemEval-2019 Task 9: Suggestion Detection in Online Forum using Word Graph
Authors: Usman Ahmed, Humera Liaquat, Luqman Ahmed, Syed Jawad Hussain
Date: 2019-06-01 00:00:00
Proceeding: SEMEVAL 2019 6
Tasks: Suggestion mining

This paper describes the suggestion miner system that participates in SemEval 2019 Task 9 - SubTask A - Suggestion Mining from Online Reviews and Forums. The system participated in the subtasks A. This paper discusses the results of our system in the development, evaluation and post evaluation. Each class in the dataset is represented as directed unweighted graphs. Then, the comparison is carried out with each class graph which results in a vector. This vector is used as features by a machine learning algorithm. The model is evaluated on hold on strategy. The organizers randomly split (8500 instances) training set (provided to the participant in training their system) and testing set (833 instances). The test set is reserved to evaluate the performance of participants systems. During the evaluation, our system ranked 31 in the Coda Lab result of the subtask A (binary class problem). The binary class system achieves evaluation value 0.34, precision 0.87, recall 0.73 and F measure 0.78.
Team Taurus at SemEval-2019 Task 9: Expert-informed pattern recognition for suggestion mining
Authors: Nelleke Oostdijk, Hans van Halteren
Date: 2019-06-01 00:00:00
Proceeding: SEMEVAL 2019 6
Tasks: BIG-bench Machine Learning, Suggestion mining

This paper presents our submissions to SemEval-2019 Task9, Suggestion Mining. Our system is one in a series of systems in which we compare an approach using expert-defined rules with a comparable one using machine learning. We target tasks with a syntactic or semantic component that might be better described by a human understanding the task than by a machine learner only able to count features. For Semeval-2019 Task 9, the expert rules clearly outperformed our machine learning model when training and testing on equally balanced testsets.
WUT at SemEval-2019 Task 9: Domain-Adversarial Neural Networks for Domain Adaptation in Suggestion Mining
Authors: Mateusz Klimaszewski, Piotr Andruszkiewicz
Date: 2019-06-01 00:00:00
Proceeding: SEMEVAL 2019 6
Tasks: Domain Adaptation, General Classification, Suggestion mining, text-classification, Text Classification, Transfer Learning

We present a system for cross-domain suggestion mining, prepared for the SemEval-2019 Task 9: Suggestion Mining from Online Reviews and Forums (Subtask B). Our submitted solution for this text classification problem explores the idea of treating different suggestions{'} sources as one of the settings of Transfer Learning - Domain Adaptation. Our experiments show that without any labeled target domain examples during training time, we are capable of proposing a system, reaching up to 0.778 in terms of F1 score on test dataset, based on Target Preserving Domain Adversarial Neural Networks.
Yimmon at SemEval-2019 Task 9: Suggestion Mining with Hybrid Augmented Approaches
Authors: Yimeng Zhuang
Date: 2019-06-01 00:00:00
Proceeding: SEMEVAL 2019 6
Tasks: Machine Reading Comprehension, Reading Comprehension, Suggestion mining, Translation

Suggestion mining task aims to extract tips, advice, and recommendations from unstructured text. The task includes many challenges, such as class imbalance, figurative expressions, context dependency, and long and complex sentences. This paper gives a detailed system description of our submission in SemEval 2019 Task 9 Subtask A. We transfer Self-Attention Network (SAN), a successful model in machine reading comprehension field, into this task. Our model concentrates on modeling long-term dependency which is indispensable to parse long and complex sentences. Besides, we also adopt techniques, such as contextualized embedding, back-translation, and auxiliary loss, to augment the system. Our model achieves a performance of F1=76.3, and rank 4th among 34 participating systems. Further ablation study shows that the techniques used in our system are beneficial to the performance.
YNU\_DYX at SemEval-2019 Task 9: A Stacked BiLSTM for Suggestion Mining Classification
Authors: Yunxia Ding, Xiaobing Zhou, Xue-jie Zhang
Date: 2019-06-01 00:00:00
Proceeding: SEMEVAL 2019 6
Tasks: General Classification, Suggestion mining

In this paper we describe a deep-learning system that competed as SemEval 2019 Task 9-SubTask A: Suggestion Mining from Online Reviews and Forums. We use Word2Vec to learn the distributed representations from sentences. This system is composed of a Stacked Bidirectional Long-Short Memory Network (SBiLSTM) for enriching word representations before and after the sequence relationship with context. We perform an ensemble to improve the effectiveness of our model. Our official submission results achieve an F1-score 0.5659.
YNU-HPCC at SemEval-2019 Task 9: Using a BERT and CNN-BiLSTM-GRU Model for Suggestion Mining
Authors: Ping Yue, Jin Wang, Xue-jie Zhang
Date: 2019-06-01 00:00:00
Proceeding: SEMEVAL 2019 6
Tasks: Opinion Mining, Suggestion mining

Consumer opinions towards commercial entities are generally expressed through online reviews, blogs, and discussion forums. These opinions largely express positive and negative sentiments towards a given entity,but also tend to contain suggestions for improving the entity. In this task, we extract suggestions from given the unstructured text, compared to the traditional opinion mining systems. Such suggestion mining is more applicability and extends capabilities.
ZQM at SemEval-2019 Task9: A Single Layer CNN Based on Pre-trained Model for Suggestion Mining
Authors: Qimin Zhou, Zhengxin Zhang, Hao Wu, Linmao Wang
Date: 2019-06-01 00:00:00
Proceeding: SEMEVAL 2019 6
Tasks: Position, Suggestion mining

This paper describes our system that competed at SemEval 2019 Task 9 - SubTask A: {''}Sug- gestion Mining from Online Reviews and Forums{''}. Our system fuses the convolutional neural network and the latest BERT model to conduct suggestion mining. In our system, the input of convolutional neural network is the embedding vectors which are drawn from the pre-trained BERT model. And to enhance the effectiveness of the whole system, the pre-trained BERT model is fine-tuned by provided datasets before the procedure of embedding vectors extraction. Empirical results show the effectiveness of our model which obtained 9th position out of 34 teams with F1 score equals to 0.715.
ProblemSolver at SemEval-2019 Task 10: Sequence-to-Sequence Learning and Expression Trees
Authors: Xuefeng Luo, Alina Baranova, Jonas Biegert
Date: 2019-06-01 00:00:00
Proceeding: SEMEVAL 2019 6
Tasks: Math, Question Answering

This paper describes our participation in SemEval-2019 shared task {``}Math Question Answering{''}, where the aim is to create a program that could solve the Math SAT questions automatically as accurately as possible. We went with a dual-pronged approach, building a Sequence-to-Sequence Neural Network pre-trained with augmented data that could answer all categories of questions and a Tree system, which can only answer a certain type of questions. The systems did not perform well on the entire test data given in the task, but did decently on the questions they were actually capable of answering. The Sequence-to-Sequence Neural Network model managed to get slightly better than our baseline of guessing {``}A{''} for every question, while the Tree system additionally improved the results.
RGCL-WLV at SemEval-2019 Task 12: Toponym Detection
Authors: Alistair Plum, Tharindu Ranasinghe, Pablo Calleja, Constantin Or{\u{a}}san, Ruslan Mitkov
Date: 2019-06-01 00:00:00
Proceeding: SEMEVAL 2019 6
Tasks: BIG-bench Machine Learning, Toponym Resolution

This article describes the system submitted by the RGCL-WLV team to the SemEval 2019 Task 12: Toponym resolution in scientific papers. The system detects toponyms using a bootstrapped machine learning (ML) approach which classifies names identified using gazetteers extracted from the GeoNames geographical database. The paper evaluates the performance of several ML classifiers, as well as how the gazetteers influence the accuracy of the system. Several runs were submitted. The highest precision achieved for one of the submissions was 89{\%}, albeit it at a relatively low recall of 49{\%}.