pwc-archive/papers-with-abstracts huggingface dataset:
Learning to Solve Geometry Problems from Natural Language Demonstrations in Textbooks
Authors: Mrinmaya Sachan, Eric Xing
Date: 2017-08-01 00:00:00
Proceeding: SEMEVAL 2017 8
Tasks: Question Answering

Humans as well as animals are good at imitation. Inspired by this, the learning by demonstration view of machine learning learns to perform a task from detailed example demonstrations. In this paper, we introduce the task of question answering using natural language demonstrations where the question answering system is provided with detailed demonstrative solutions to questions in natural language. As a case study, we explore the task of learning to solve geometry problems using demonstrative solutions available in textbooks. We collect a new dataset of demonstrative geometry solutions from textbooks and explore approaches that learn to interpret these demonstrations as well as to use these interpretations to solve geometry problems. Our approaches show improvements over the best previously published system for solving geometry problems.
Ways of Asking and Replying in Duplicate Question Detection
Authors: Jo{\~a}o Ant{\'o}nio Rodrigues, Chakaveh Saedi, Vladislav Maraev, Jo{\~a}o Silva, Ant{\'o}nio Branco
Date: 2017-08-01 00:00:00
Proceeding: SEMEVAL 2017 8
Tasks: Machine Translation, Question Answering, Semantic Textual Similarity

This paper presents the results of systematic experimentation on the impact in duplicate question detection of different types of questions across both a number of established approaches and a novel, superior one used to address this language processing task. This study permits to gain a novel insight on the different levels of robustness of the diverse detection methods with respect to different conditions of their application, including the ones that approximate real usage scenarios.
SemEval-2017 Task 1: Semantic Textual Similarity Multilingual and Crosslingual Focused Evaluation
Authors: Daniel Cer, Mona Diab, Eneko Agirre, I{\~n}igo Lopez-Gazpio, Lucia Specia
Date: 2017-08-01 00:00:00
Proceeding: SEMEVAL 2017 8
Tasks: Machine Translation, Natural Language Inference, Question Answering, Semantic Textual Similarity, STS, STS Benchmark, Translation

Semantic Textual Similarity (STS) measures the meaning similarity of sentences. Applications include machine translation (MT), summarization, generation, question answering (QA), short answer grading, semantic search, dialog and conversational systems. The STS shared task is a venue for assessing the current state-of-the-art. The 2017 task focuses on multilingual and cross-lingual pairs with one sub-track exploring MT quality estimation (MTQE) data. The task obtained strong participation from 31 teams, with 17 participating in \textit{all language tracks}. We summarize performance and review a selection of well performing methods. Analysis highlights common errors, providing insight into the limitations of existing models. To support ongoing work on semantic representations, the \textit{STS Benchmark} is introduced as a new shared training and evaluation set carefully selected from the corpus of English STS shared task data (2012-2017).
SemEval-2017 Task 2: Multilingual and Cross-lingual Semantic Word Similarity
Authors: Jose Camacho-Collados, Mohammad Taher Pilehvar, Nigel Collier, Roberto Navigli
Date: 2017-08-01 00:00:00
Proceeding: SEMEVAL 2017 8
Tasks: Information Retrieval, Machine Translation, Question Answering, Representation Learning, Semantic Similarity, Semantic Textual Similarity, Task 2, Text Summarization, Word Embeddings, Word Sense Disambiguation, Word Similarity

This paper introduces a new task on Multilingual and Cross-lingual SemanticThis paper introduces a new task on Multilingual and Cross-lingual Semantic Word Similarity which measures the semantic similarity of word pairs within and across five languages: English, Farsi, German, Italian and Spanish. High quality datasets were manually curated for the five languages with high inter-annotator agreements (consistently in the 0.9 ballpark). These were used for semi-automatic construction of ten cross-lingual datasets. 17 teams participated in the task, submitting 24 systems in subtask 1 and 14 systems in subtask 2. Results show that systems that combine statistical knowledge from text corpora, in the form of word embeddings, and external knowledge from lexical resources are best performers in both subtasks. More information can be found on the task website: \url{http://alt.qcri.org/semeval2017/task2/}
SemEval-2017 Task 6: \#HashtagWars: Learning a Sense of Humor
Authors: Peter Potash, Alexey Romanov, Anna Rumshisky
Date: 2017-08-01 00:00:00
Proceeding: SEMEVAL 2017 8
Tasks: Humor Detection

This paper describes a new shared task for humor understanding that attempts to eschew the ubiquitous binary approach to humor detection and focus on comparative humor ranking instead. The task is based on a new dataset of funny tweets posted in response to shared hashtags, collected from the {`}Hashtag Wars{'} segment of the TV show @midnight. The results are evaluated in two subtasks that require the participants to generate either the correct pairwise comparisons of tweets (subtask A), or the correct ranking of the tweets (subtask B) in terms of how funny they are. 7 teams participated in subtask A, and 5 teams participated in subtask B. The best accuracy in subtask A was 0.675. The best (lowest) rank edit distance for subtask B was 0.872.
SemEval-2017 Task 7: Detection and Interpretation of English Puns
Authors: Tristan Miller, Christian Hempelmann, Iryna Gurevych
Date: 2017-08-01 00:00:00
Proceeding: SEMEVAL 2017 8
Tasks: Word Sense Disambiguation

A pun is a form of wordplay in which a word suggests two or more meanings by exploiting polysemy, homonymy, or phonological similarity to another word, for an intended humorous or rhetorical effect. Though a recurrent and expected feature in many discourse types, puns stymie traditional approaches to computational lexical semantics because they violate their one-sense-per-context assumption. This paper describes the first competitive evaluation for the automatic detection, location, and interpretation of puns. We describe the motivation for these tasks, the evaluation methods, and the manually annotated data set. Finally, we present an overview and discussion of the participating systems{'} methodologies, resources, and results.
BIT at SemEval-2017 Task 1: Using Semantic Information Space to Evaluate Semantic Textual Similarity
Authors: Hao Wu, He-Yan Huang, Ping Jian, Yuhang Guo, Chao Su
Date: 2017-08-01 00:00:00
Proceeding: SEMEVAL 2017 8
Tasks: Information Retrieval, Machine Translation, Question Answering, Semantic Textual Similarity, STS, Text Summarization

This paper presents three systems for semantic textual similarity (STS) evaluation at SemEval-2017 STS task. One is an unsupervised system and the other two are supervised systems which simply employ the unsupervised one. All our systems mainly depend on the (SIS), which is constructed based on the semantic hierarchical taxonomy in WordNet, to compute non-overlapping information content (IC) of sentences. Our team ranked 2nd among 31 participating teams by the primary score of Pearson correlation coefficient (PCC) mean of 7 tracks and achieved the best performance on Track 1 (AR-AR) dataset.
IIT-UHH at SemEval-2017 Task 3: Exploring Multiple Features for Community Question Answering and Implicit Dialogue Identification
Authors: N, Titas i, Chris Biemann, Seid Muhie Yimam, Deepak Gupta, Sarah Kohail, Asif Ekbal, Pushpak Bhattacharyya
Date: 2017-08-01 00:00:00
Proceeding: SEMEVAL 2017 8
Tasks: Answer Selection, Community Question Answering, Question Answering, Question Similarity

In this paper we present the system for Answer Selection and Ranking in Community Question Answering, which we build as part of our participation in SemEval-2017 Task 3. We develop a Support Vector Machine (SVM) based system that makes use of textual, domain-specific, word-embedding and topic-modeling features. In addition, we propose a novel method for dialogue chain identification in comment threads. Our primary submission won subtask C, outperforming other systems in all the primary evaluation metrics. We performed well in other English subtasks, ranking third in subtask A and eighth in subtask B. We also developed open source toolkits for all the three English subtasks by the name cQARank [\url{https://github.com/TitasNandi/cQARank}].
HumorHawk at SemEval-2017 Task 6: Mixing Meaning and Sound for Humor Recognition
Authors: David Donahue, Alexey Romanov, Anna Rumshisky
Date: 2017-08-01 00:00:00
Proceeding: SEMEVAL 2017 8
Tasks: Humor Detection, Word Embeddings

This paper describes the winning system for SemEval-2017 Task 6: {\#}HashtagWars: Learning a Sense of Humor. Humor detection has up until now been predominantly addressed using feature-based approaches. Our system utilizes recurrent deep learning methods with dense embeddings to predict humorous tweets from the @midnight show {\#}HashtagWars. In order to include both meaning and sound in the analysis, GloVe embeddings are combined with a novel phonetic representation to serve as input to an LSTM component. The output is combined with a character-based CNN model, and an XGBoost component in an ensemble model which achieves 0.675 accuracy on the evaluation data.
Idiom Savant at Semeval-2017 Task 7: Detection and Interpretation of English Puns
Authors: Samuel Doogan, Aniruddha Ghosh, Hanyang Chen, Tony Veale
Date: 2017-08-01 00:00:00
Proceeding: SEMEVAL 2017 8
Tasks: Task 2

This paper describes our system, entitled Idiom Savant, for the 7th Task of the Semeval 2017 workshop, {``}Detection and interpretation of English Puns{''}. Our system consists of two probabilistic models for each type of puns using Google n-gram and Word2Vec. Our system achieved f-score of calculating, 0.663, and 0.07 in homographic puns and 0.8439, 0.6631, and 0.0806 in heterographic puns in task 1, task 2, and task 3 respectively.
UdL at SemEval-2017 Task 1: Semantic Textual Similarity Estimation of English Sentence Pairs Using Regression Model over Pairwise Features
Authors: Hussein T. Al-Natsheh, Lucie Martinet, Fabrice Muhlenbach, Djamel Abdelkader Zighed
Date: 2017-08-01 00:00:00
Proceeding: SEMEVAL 2017 8
Tasks: Ensemble Learning, Image Captioning, Model Selection, POS, Semantic Similarity, Semantic Textual Similarity, Sentence, Sentence Embedding, Word Embeddings

This paper describes the model UdL we proposed to solve the semantic textual similarity task of SemEval 2017 workshop. The track we participated in was estimating the semantics relatedness of a given set of sentence pairs in English. The best run out of three submitted runs of our model achieved a Pearson correlation score of 0.8004 compared to a hidden human annotation of 250 pairs. We used random forest ensemble learning to map an expandable set of extracted pairwise features into a semantic similarity estimated value bounded between 0 and 5. Most of these features were calculated using word embedding vectors similarity to align Part of Speech (PoS) and Name Entities (NE) tagged tokens of each sentence pair. Among other pairwise features, we experimented a classical tf-idf weighted Bag of Words (BoW) vector model but with character-based range of n-grams instead of words. This sentence vector BoW-based feature gave a relatively high importance value percentage in the feature importances analysis of the ensemble learning.
DT\_Team at SemEval-2017 Task 1: Semantic Similarity Using Alignments, Sentence-Level Embeddings and Gaussian Mixture Model Output
Authors: Nabin Maharjan, Rajendra Banjade, Dipesh Gautam, Lasang J. Tamang, Vasile Rus
Date: 2017-08-01 00:00:00
Proceeding: SEMEVAL 2017 8
Tasks: Lemmatization, Semantic Similarity, Semantic Textual Similarity, Sentence, Sentence Embeddings, STS, STS Benchmark, Word Alignment

We describe our system (DT Team) submitted at SemEval-2017 Task 1, Semantic Textual Similarity (STS) challenge for English (Track 5). We developed three different models with various features including similarity scores calculated using word and chunk alignments, word/sentence embeddings, and Gaussian Mixture Model(GMM). The correlation between our system{'}s output and the human judgments were up to 0.8536, which is more than 10{\%} above baseline, and almost as good as the best performing system which was at 0.8547 correlation (the difference is just about 0.1{\%}). Also, our system produced leading results when evaluated with a separate STS benchmark dataset. The word alignment and sentence embeddings based features were found to be very effective.
FCICU at SemEval-2017 Task 1: Sense-Based Language Independent Semantic Textual Similarity Approach
Authors: Basma Hassan, Samir AbdelRahman, Reem Bahgat, Ibrahim Farag
Date: 2017-08-01 00:00:00
Proceeding: SEMEVAL 2017 8
Tasks: Machine Translation, Semantic Textual Similarity, Sentence

This paper describes FCICU team systems that participated in SemEval-2017 Semantic Textual Similarity task (Task1) for monolingual and cross-lingual sentence pairs. A sense-based language independent textual similarity approach is presented, in which a proposed alignment similarity method coupled with new usage of a semantic network (BabelNet) is used. Additionally, a previously proposed integration between sense-based and sur-face-based semantic textual similarity approach is applied together with our proposed approach. For all the tracks in Task1, Run1 is a string kernel with alignments metric and Run2 is a sense-based alignment similarity method. The first run is ranked 10th, and the second is ranked 12th in the primary track, with correlation 0.619 and 0.617 respectively
HCTI at SemEval-2017 Task 1: Use convolutional neural network to evaluate Semantic Textual Similarity
Authors: Yang Shao
Date: 2017-08-01 00:00:00
Proceeding: SEMEVAL 2017 8
Tasks: Answer Selection, Machine Translation, Question Answering, Semantic Similarity, Semantic Textual Similarity, Sentence, STS

This paper describes our convolutional neural network (CNN) system for Semantic Textual Similarity (STS) task. We calculated semantic similarity score between two sentences by comparing their semantic vectors. We generated semantic vector of every sentence by max pooling every dimension of their word vectors. There are mainly two trick points in our system. One is that we trained a CNN to transfer GloVe word vectors to a more proper form for STS task before pooling. Another is that we trained a fully-connected neural network (FCNN) to transfer difference of two semantic vectors to probability of every similarity score. We decided all hyper parameters empirically. In spite of the simplicity of our neural network system, we achieved a good accuracy and ranked 3rd in primary track of SemEval 2017.
LIM-LIG at SemEval-2017 Task1: Enhancing the Semantic Similarity for Arabic Sentences with Vectors Weighting
Authors: El Moatez Billah Nagoudi, J{\'e}r{\'e}my Ferrero, Didier Schwab
Date: 2017-08-01 00:00:00
Proceeding: SEMEVAL 2017 8
Tasks: Descriptive, Information Retrieval, Machine Translation, Paraphrase Identification, Part-Of-Speech Tagging, Semantic Similarity, Semantic Textual Similarity, Sentence, STS, Word Sense Disambiguation

This article describes our proposed system named LIM-LIG. This system is designed for SemEval 2017 Task1: Semantic Textual Similarity (Track1). LIM-LIG proposes an innovative enhancement to word embedding-based model devoted to measure the semantic similarity in Arabic sentences. The main idea is to exploit the word representations as vectors in a multidimensional space to capture the semantic and syntactic properties of words. IDF weighting and Part-of-Speech tagging are applied on the examined sentences to support the identification of words that are highly descriptive in each sentence. LIM-LIG system achieves a Pearson{'}s correlation of 0.74633, ranking 2nd among all participants in the Arabic monolingual pairs STS task organized within the SemEval 2017 evaluation campaign
OPI-JSA at SemEval-2017 Task 1: Application of Ensemble learning for computing semantic textual similarity
Authors: Martyna {\'S}piewak, Piotr Sobecki, Daniel Kara{\'s}
Date: 2017-08-01 00:00:00
Proceeding: SEMEVAL 2017 8
Tasks: Ensemble Learning, POS, Semantic Similarity, Semantic Textual Similarity, STS, Word Embeddings

Semantic Textual Similarity (STS) evaluation assesses the degree to which two parts of texts are similar, based on their semantic evaluation. In this paper, we describe three models submitted to STS SemEval 2017. Given two English parts of a text, each of proposed methods outputs the assessment of their semantic similarity. We propose an approach for computing monolingual semantic textual similarity based on an ensemble of three distinct methods. Our model consists of recursive neural network (RNN) text auto-encoders ensemble with supervised a model of vectorized sentences using reduced part of speech (PoS) weighted word embeddings as well as unsupervised a method based on word coverage (TakeLab). Additionally, we enrich our model with additional features that allow disambiguation of ensemble methods based on their efficiency. We have used Multi-Layer Perceptron as an ensemble classifier basing on estimations of trained Gradient Boosting Regressors. Results of our research proves that using such ensemble leads to a higher accuracy due to a fact that each member-algorithm tends to specialize in particular type of sentences. Simple model based on PoS weighted Word2Vec word embeddings seem to improve performance of more complex RNN based auto-encoders in the ensemble. In the monolingual English-English STS subtask our Ensemble based model achieved mean Pearson correlation of .785 compared with human annotators.
Lump at SemEval-2017 Task 1: Towards an Interlingua Semantic Similarity
Authors: Cristina Espa{\~n}a-Bonet, Alberto Barr{\'o}n-Cede{\~n}o
Date: 2017-08-01 00:00:00
Proceeding: SEMEVAL 2017 8
Tasks: Language Identification, Machine Translation, Semantic Similarity, Semantic Textual Similarity, Word Embeddings

This is the Lump team participation at SemEval 2017 Task 1 on Semantic Textual Similarity. Our supervised model relies on features which are multilingual or interlingual in nature. We include lexical similarities, cross-language explicit semantic analysis, internal representations of multilingual neural networks and interlingual word embeddings. Our representations allow to use large datasets in language pairs with many instances to better classify instances in smaller language pairs avoiding the necessity of translating into a single language. Hence we can deal with all the languages in the task: Arabic, English, Spanish, and Turkish.
QLUT at SemEval-2017 Task 1: Semantic Textual Similarity Based on Word Embeddings
Authors: Fanqing Meng, Wenpeng Lu, Yuteng Zhang, Jinyong Cheng, Yuehan Du, Shuwang Han
Date: 2017-08-01 00:00:00
Proceeding: SEMEVAL 2017 8
Tasks: Lemmatization, Semantic Textual Similarity, Word Embeddings

This paper reports the details of our submissions in the task 1 of SemEval 2017. This task aims at assessing the semantic textual similarity of two sentences or texts. We submit three unsupervised systems based on word embeddings. The differences between these runs are the various preprocessing on evaluation data. The best performance of these systems on the evaluation of Pearson correlation is 0.6887. Unsurprisingly, results of our runs demonstrate that data preprocessing, such as tokenization, lemmatization, extraction of content words and removing stop words, is helpful and plays a significant role in improving the performance of models.
ResSim at SemEval-2017 Task 1: Multilingual Word Representations for Semantic Textual Similarity
Authors: Johannes Bjerva, Robert {\"O}stling
Date: 2017-08-01 00:00:00
Proceeding: SEMEVAL 2017 8
Tasks: Machine Translation, Semantic Similarity, Semantic Textual Similarity

Shared Task 1 at SemEval-2017 deals with assessing the semantic similarity between sentences, either in the same or in different languages. In our system submission, we employ multilingual word representations, in which similar words in different languages are close to one another. Using such representations is advantageous, since the increasing amount of available parallel data allows for the application of such methods to many of the languages in the world. Hence, semantic similarity can be inferred even for languages for which no annotated data exists. Our system is trained and evaluated on all language pairs included in the shared task (English, Spanish, Arabic, and Turkish). Although development results are promising, our system does not yield high performance on the shared task test sets.
ITNLP-AiKF at SemEval-2017 Task 1: Rich Features Based SVR for Semantic Textual Similarity Computing
Authors: Wenjie Liu, Chengjie Sun, Lei Lin, Bingquan Liu
Date: 2017-08-01 00:00:00
Proceeding: SEMEVAL 2017 8
Tasks: Feature Engineering, regression, Semantic Textual Similarity, Sentence, STS

Semantic Textual Similarity (STS) devotes to measuring the degree of equivalence in the underlying semantic of the sentence pair. We proposed a new system, ITNLP-AiKF, which applies in the SemEval 2017 Task1 Semantic Textual Similarity track 5 English monolingual pairs. In our system, rich features are involved, including Ontology based, word embedding based, Corpus based, Alignment based and Literal based feature. We leveraged the features to predict sentence pair similarity by a Support Vector Regression (SVR) model. In the result, a Pearson Correlation of 0.8231 is achieved by our system, which is a competitive result in the contest of this track.
SEF@UHH at SemEval-2017 Task 1: Unsupervised Knowledge-Free Semantic Textual Similarity via Paragraph Vector
Authors: Mirela-Stefania Duma, Wolfgang Menzel
Date: 2017-08-01 00:00:00
Proceeding: SEMEVAL 2017 8
Tasks: Semantic Similarity, Semantic Textual Similarity

This paper describes our unsupervised knowledge-free approach to the SemEval-2017 Task 1 Competition. The proposed method makes use of Paragraph Vector for assessing the semantic similarity between pairs of sentences. We experimented with various dimensions of the vector and three state-of-the-art similarity metrics. Given a cross-lingual task, we trained models corresponding to its two languages and combined the models by averaging the similarity scores. The results of our submitted runs are above the median scores for five out of seven test sets by means of Pearson Correlation. Moreover, one of our system runs performed best on the Spanish-English-WMT test set ranking first out of 53 runs submitted in total by all participants.
STS-UHH at SemEval-2017 Task 1: Scoring Semantic Textual Similarity Using Supervised and Unsupervised Ensemble
Authors: Sarah Kohail, Amr Rekaby Salama, Chris Biemann
Date: 2017-08-01 00:00:00
Proceeding: SEMEVAL 2017 8
Tasks: Graph Similarity, Information Retrieval, Natural Language Inference, Paraphrase Identification, regression, Semantic Textual Similarity, STS, Word Alignment

This paper reports the STS-UHH participation in the SemEval 2017 shared Task 1 of Semantic Textual Similarity (STS). Overall, we submitted 3 runs covering monolingual and cross-lingual STS tracks. Our participation involves two approaches: unsupervised approach, which estimates a word alignment-based similarity score, and supervised approach, which combines dependency graph similarity and coverage features with lexical similarity measures using regression methods. We also present a way on ensembling both models. Out of 84 submitted runs, our team best multi-lingual run has been ranked 12th in overall performance with correlation of 0.61, 7th among 31 participating teams.