pwc-archive/papers-with-abstracts huggingface dataset:
Interactively-Propagative Attention Learning for Implicit Discourse Relation Recognition
Authors: Huibin Ruan, Yu Hong, Yang Xu, Zhen Huang, Guodong Zhou, Min Zhang
Date: 2020-12-01 00:00:00
Proceeding: COLING 2020 8
Tasks: Relation, Representation Learning

We tackle implicit discourse relation recognition. Both self-attention and interactive-attention mechanisms have been applied for attention-aware representation learning, which improves the current discourse analysis models. To take advantages of the two attention mechanisms simultaneously, we develop a propagative attention learning model using a cross-coupled two-channel network. We experiment on Penn Discourse Treebank. The test results demonstrate that our model yields substantial improvements over the baselines (BiLSTM and BERT).
Dual Attention Network for Cross-lingual Entity Alignment
Authors: Jian Sun, Yu Zhou, Chengqing Zong
Date: 2020-12-01 00:00:00
Proceeding: COLING 2020 8
Tasks: Entity Alignment, Graph Attention, Knowledge Graphs, Relation

Cross-lingual Entity alignment is an essential part of building a knowledge graph, which can help integrate knowledge among different language knowledge graphs. In the real KGs, there exists an imbalance among the information in the same hierarchy of corresponding entities, which results in the heterogeneity of neighborhood structure, making this task challenging. To tackle this problem, we propose a dual attention network for cross-lingual entity alignment (DAEA). Specifically, our dual attention consists of relation-aware graph attention and hierarchical attention. The relation-aware graph attention aims at selectively aggregating multi-hierarchy neighborhood information to alleviate the difference of heterogeneity among counterpart entities. The hierarchical attention adaptively aggregates the low-hierarchy and the high-hierarchy information, which is beneficial to balance the neighborhood information of counterpart entities and distinguish non-counterpart entities with similar structures. Finally, we treat cross-lingual entity alignment as a process of linking prediction. Experimental results on three real-world cross-lingual entity alignment datasets have shown the effectiveness of DAEA.
Task-Aware Representation of Sentences for Generic Text Classification
Authors: Kishaloy Halder, Alan Akbik, Josip Krapac, Roland Vollgraf
Date: 2020-12-01 00:00:00
Proceeding: COLING 2020 8
Tasks: Articles, Binary Classification, Classification, text-classification, Text Classification, Transfer Learning

State-of-the-art approaches for text classification leverage a transformer architecture with a linear layer on top that outputs a class distribution for a given prediction problem. While effective, this approach suffers from conceptual limitations that affect its utility in few-shot or zero-shot transfer learning scenarios. First, the number of classes to predict needs to be pre-defined. In a transfer learning setting, in which new classes are added to an already trained classifier, all information contained in a linear layer is therefore discarded, and a new layer is trained from scratch. Second, this approach only learns the semantics of classes implicitly from training examples, as opposed to leveraging the explicit semantic information provided by the natural language names of the classes. For instance, a classifier trained to predict the topics of news articles might have classes like {``}business{''} or {``}sports{''} that themselves carry semantic information. Extending a classifier to predict a new class named {``}politics{''} with only a handful of training examples would benefit from both leveraging the semantic information in the name of a new class and using the information contained in the already trained linear layer. This paper presents a novel formulation of text classification that addresses these limitations. It imbues the notion of the task at hand into the transformer model itself by factorizing arbitrary classification problems into a generic binary classification problem. We present experiments in few-shot and zero-shot transfer learning that show that our approach significantly outperforms previous approaches on small training data and can even learn to predict new classes with no training examples at all. The implementation of our model is publicly available at: https://github.com/flairNLP/flair.
RANCC: Rationalizing Neural Networks via Concept Clustering
Authors: Housam Khalifa Bashier, Mi-Young Kim, Randy Goebel
Date: 2020-12-01 00:00:00
Proceeding: COLING 2020 8
Tasks: Classification, Clustering, Feature Importance, text-classification, Text Classification

We propose a new self-explainable model for Natural Language Processing (NLP) text classification tasks. Our approach constructs explanations concurrently with the formulation of classification predictions. To do so, we extract a rationale from the text, then use it to predict a concept of interest as the final prediction. We provide three types of explanations: 1) rationale extraction, 2) a measure of feature importance, and 3) clustering of concepts. In addition, we show how our model can be compressed without applying complicated compression techniques. We experimentally demonstrate our explainability approach on a number of well-known text classification datasets.
Emotion Classification by Jointly Learning to Lexiconize and Classify
Authors: Deyu Zhou, Shuangzhi Wu, Qing Wang, Jun Xie, Zhaopeng Tu, Mu Li
Date: 2020-12-01 00:00:00
Proceeding: COLING 2020 8
Tasks: Classification, Emotion Classification, Sentence

Emotion lexicons have been shown effective for emotion classification (Baziotis et al., 2018). Previous studies handle emotion lexicon construction and emotion classification separately. In this paper, we propose an emotional network (EmNet) to jointly learn sentence emotions and construct emotion lexicons which are dynamically adapted to a given context. The dynamic emotion lexicons are useful for handling words with multiple emotions based on different context, which can effectively improve the classification accuracy. We validate the approach on two representative architectures {--} LSTM and BERT, demonstrating its superiority on identifying emotions in Tweets. Our model outperforms several approaches proposed in previous studies and achieves new state-of-the-art on the benchmark Twitter dataset.
Multi-level Alignment Pretraining for Multi-lingual Semantic Parsing
Authors: Bo Shao, Yeyun Gong, Weizhen Qi, Nan Duan, Xiaola Lin
Date: 2020-12-01 00:00:00
Proceeding: COLING 2020 8
Tasks: Semantic Parsing, Sentence

In this paper, we present a multi-level alignment pretraining method in a unified architecture formulti-lingual semantic parsing. In this architecture, we use an adversarial training method toalign the space of different languages and use sentence level and word level parallel corpus assupervision information to align the semantic of different languages. Finally, we jointly train themulti-level alignment and semantic parsing tasks. We conduct experiments on a publicly avail-able multi-lingual semantic parsing dataset ATIS and a newly constructed dataset. Experimentalresults show that our model outperforms state-of-the-art methods on both datasets.
Conception: Multilingually-Enhanced, Human-Readable Concept Vector Representations
Authors: Simone Conia, Roberto Navigli
Date: 2020-12-01 00:00:00
Proceeding: COLING 2020 8
Tasks: Word Sense Disambiguation, Word Similarity

To date, the most successful word, word sense, and concept modelling techniques have used large corpora and knowledge resources to produce dense vector representations that capture semantic similarities in a relatively low-dimensional space. Most current approaches, however, suffer from a monolingual bias, with their strength depending on the amount of data available across languages. In this paper we address this issue and propose Conception, a novel technique for building language-independent vector representations of concepts which places multilinguality at its core while retaining explicit relationships between concepts. Our approach results in high-coverage representations that outperform the state of the art in multilingual and cross-lingual Semantic Word Similarity and Word Sense Disambiguation, proving particularly robust on low-resource languages. Conception {--} its software and the complete set of representations {--} is available at https://github.com/SapienzaNLP/conception.
Sentence Matching with Syntax- and Semantics-Aware BERT
Authors: Tao Liu, Xin Wang, Chengguo Lv, Ranran Zhen, Guohong Fu
Date: 2020-12-01 00:00:00
Proceeding: COLING 2020 8
Tasks: Sentence

Sentence matching aims to identify the special relationship between two sentences, and plays a key role in many natural language processing tasks. However, previous studies mainly focused on exploiting either syntactic or semantic information for sentence matching, and no studies consider integrating both of them. In this study, we propose integrating syntax and semantics into BERT with sentence matching. In particular, we use an implicit syntax and semantics integration method that is less sensitive to the output structure information. Thus the implicit integration can alleviate the error propagation problem. The experimental results show that our approach has achieved state-of-the-art or competitive performance on several sentence matching datasets, demonstrating the benefits of implicitly integrating syntactic and semantic features in sentence matching.
Temporal Relations Annotation and Extrapolation Based on Semi-intervals and Boundig Relations
Authors: Alejandro Pimentel, Gemma Bel Enguix, Gerardo Sierra Mart{\'\i}nez, Azucena Montes
Date: 2020-12-01 00:00:00
Proceeding: COLING 2020 8
Tasks: Management

The computational treatment of temporal relations is based on the work of Allen, who establishes 13 different types, and Freksa, who designs a cognitive procedure to manage them. Freksa{'}s notation is not widely used because, although it has cognitive and expressive advantages, it is too complex from the computational perspective. This paper proposes a system for the annotation and management of temporal relations that combines the richness and expressiveness of Freksa{'}s approach with the simplicity of Allen{'}s notation. Our method is summarized in the application of bounding relations, thanks to which it is possible to obtain the temporary representation of complete neighborhoods capable of representing vague temporal relations such as those that can be frequently found in a text. Such advantages are obtained without the need to greatly increase the complexity of the labeling process since the markup language is almost the same as TimeML, to which only a second temporary {``}relType{''}{'} type label relationship is added. Our experiments show that the temporal relationships that present vagueness are in fact much more common than those in which a single relationship can be established precisely. For these reasons, our new labeling system achieves a more agreeable representation of temporal relations.
Homonym normalisation by word sense clustering: a case in Japanese
Authors: Yo Sato, Kevin Heffernan
Date: 2020-12-01 00:00:00
Proceeding: COLING 2020 8
Tasks: Clustering, Language Modeling, Language Modelling, Transliteration

This work presents a method of word sense clustering that differentiates homonyms and merge homophones, taking Japanese as an example, where orthographical variation causes problem for language processing. It uses contextualised embeddings (BERT) to cluster tokens into distinct sense groups, and we use these groups to normalise synonymous instances to a single representative form. We see the benefit of this normalisation in language model, as well as in transliteration.
Verbal Multiword Expression Identification: Do We Need a Sledgehammer to Crack a Nut?
Authors: Caroline Pasquer, Agata Savary, Carlos Ramisch, Jean-Yves Antoine
Date: 2020-12-01 00:00:00
Proceeding: COLING 2020 8

Automatic identification of multiword expressions (MWEs), like {`}to cut corners{'} (to do an incomplete job), is a pre-requisite for semantically-oriented downstream applications. This task is challenging because MWEs, especially verbal ones (VMWEs), exhibit surface variability. This paper deals with a subproblem of VMWE identification: the identification of occurrences of previously seen VMWEs. A simple language-independent system based on a combination of filters competes with the best systems from a recent shared task: it obtains the best averaged F-score over 11 languages (0.6653) and even the best score for both seen and unseen VMWEs due to the high proportion of seen VMWEs in texts. This highlights the fact that focusing on the identification of seen VMWEs could be a strategy to improve VMWE identification in general.
An Unsupervised Method for Learning Representations of Multi-word Expressions for Semantic Classification
Authors: Robert Vacareanu, Marco A. Valenzuela-Esc{\'a}rcega, Rebecca Sharp, Mihai Surdeanu
Date: 2020-12-01 00:00:00
Proceeding: COLING 2020 8

This paper explores an unsupervised approach to learning a compositional representation function for multi-word expressions (MWEs), and evaluates it on the Tratz dataset, which associates two-word expressions with the semantic relation between the compound constituents (e.g. the label employer is associated with the noun compound government agency) (Tratz, 2011). The composition function is based on recurrent neural networks, and is trained using the Skip-Gram objective to predict the words in the context of MWEs. Thus our approach can naturally leverage large unlabeled text sources. Further, our method can make use of provided MWEs when available, but can also function as a completely unsupervised algorithm, using MWE boundaries predicted by a single, domain-agnostic part-of-speech pattern. With pre-defined MWE boundaries, our method outperforms the previous state-of-the-art performance on the coarse-grained evaluation of the Tratz dataset (Tratz, 2011), with an F1 score of 50.4{\%}. The unsupervised version of our method approaches the performance of the supervised one, and even outperforms it in some configurations.
SLICE: Supersense-based Lightweight Interpretable Contextual Embeddings
Authors: Cindy Aloui, Carlos Ramisch, Alexis Nasr, Lucie Barque
Date: 2020-12-01 00:00:00
Proceeding: COLING 2020 8

Contextualised embeddings such as BERT have become de facto state-of-the-art references in many NLP applications, thanks to their impressive performances. However, their opaqueness makes it hard to interpret their behaviour. SLICE is a hybrid model that combines supersense labels with contextual embeddings. We introduce a weakly supervised method to learn interpretable embeddings from raw corpora and small lists of seed words. Our model is able to represent both a word and its context as embeddings into the same compact space, whose dimensions correspond to interpretable supersenses. We assess the model in a task of supersense tagging for French nouns. The little amount of supervision required makes it particularly well suited for low-resourced scenarios. Thanks to its interpretability, we perform linguistic analyses about the predicted supersenses in terms of input word and context representations.
An Empirical Study of the Downstream Reliability of Pre-Trained Word Embeddings
Authors: Anthony Rios, Brandon Lwowski
Date: 2020-12-01 00:00:00
Proceeding: COLING 2020 8
Tasks: Imputation, Word Embeddings

While pre-trained word embeddings have been shown to improve the performance of downstream tasks, many questions remain regarding their reliability: Do the same pre-trained word embeddings result in the best performance with slight changes to the training data? Do the same pre-trained embeddings perform well with multiple neural network architectures? Do imputation strategies for unknown words impact reliability? In this paper, we introduce two new metrics to understand the downstream reliability of word embeddings. We find that downstream reliability of word embeddings depends on multiple factors, including, the evaluation metric, the handling of out-of-vocabulary words, and whether the embeddings are fine-tuned.
Sentence Analogies: Linguistic Regularities in Sentence Embeddings
Authors: Xunjie Zhu, Gerard de Melo
Date: 2020-12-01 00:00:00
Proceeding: COLING 2020 8
Tasks: Sentence, Sentence Embedding, Sentence-Embedding, Sentence Embeddings

While important properties of word vector representations have been studied extensively, far less is known about the properties of sentence vector representations. Word vectors are often evaluated by assessing to what degree they exhibit regularities with regard to relationships of the sort considered in word analogies. In this paper, we investigate to what extent commonly used sentence vector representation spaces as well reflect certain kinds of regularities. We propose a number of schemes to induce evaluation data, based on lexical analogy data as well as semantic relationships between sentences. Our experiments consider a wide range of sentence embedding methods, including ones based on BERT-style contextual embeddings. We find that different models differ substantially in their ability to reflect such regularities.
Manifold Learning-based Word Representation Refinement Incorporating Global and Local Information
Authors: Wenyu Zhao, Dong Zhou, Lin Li, Jinjun Chen
Date: 2020-12-01 00:00:00
Proceeding: COLING 2020 8
Tasks: Semantic Similarity, Semantic Textual Similarity, Word Embeddings, Word Similarity

Recent studies show that word embedding models often underestimate similarities between similar words and overestimate similarities between distant words. This results in word similarity results obtained from embedding models inconsistent with human judgment. Manifold learning-based methods are widely utilized to refine word representations by re-embedding word vectors from the original embedding space to a new refined semantic space. These methods mainly focus on preserving local geometry information through performing weighted locally linear combination between words and their neighbors twice. However, these reconstruction weights are easily influenced by different selections of neighboring words and the whole combination process is time-consuming. In this paper, we propose two novel word representation refinement methods leveraging isometry feature mapping and local tangent space respectively. Unlike previous methods, our first method corrects pre-trained word embeddings by preserving global geometry information of all words instead of local geometry information between words and their neighbors. Our second method refines word representations by aligning original and re-fined embedding spaces based on local tangent space instead of performing weighted locally linear combination twice. Experimental results obtained from standard semantic relatedness and semantic similarity tasks show that our methods outperform various state-of-the-art baselines for word representation refinement.
Handling Anomalies of Synthetic Questions in Unsupervised Question Answering
Authors: Giwon Hong, Junmo Kang, Doyeon Lim, Sung-Hyon Myaeng
Date: 2020-12-01 00:00:00
Proceeding: COLING 2020 8
Tasks: Question Answering

Advances in Question Answering (QA) research require additional datasets for new domains, languages, and types of questions, as well as for performance increases. Human creation of a QA dataset like SQuAD, however, is expensive. As an alternative, an unsupervised QA approach has been proposed so that QA training data can be generated automatically. However, the performance of unsupervised QA is much lower than that of supervised QA models. We identify two anomalies in the automatically generated questions and propose how they can be mitigated. We show our approach helps improve unsupervised QA significantly across a number of QA tasks.
Designing Templates for Eliciting Commonsense Knowledge from Pretrained Sequence-to-Sequence Models
Authors: Jheng-Hong Yang, Sheng-Chieh Lin, Rodrigo Nogueira, Ming-Feng Tsai, Chuan-Ju Wang, Jimmy Lin
Date: 2020-12-01 00:00:00
Proceeding: COLING 2020 8
Tasks: Multiple-choice, Natural Language Understanding, Open-Ended Question Answering, Question Answering

While internalized {``}implicit knowledge{''} in pretrained transformers has led to fruitful progress in many natural language understanding tasks, how to most effectively elicit such knowledge remains an open question. Based on the text-to-text transfer transformer (T5) model, this work explores a template-based approach to extract implicit knowledge for commonsense reasoning on multiple-choice (MC) question answering tasks. Experiments on three representative MC datasets show the surprisingly good performance of our simple template, coupled with a logit normalization technique for disambiguation. Furthermore, we verify that our proposed template can be easily extended to other MC tasks with contexts such as supporting facts in open-book question answering settings. Starting from the MC task, this work initiates further research to find generic natural language templates that can effectively leverage stored knowledge in pretrained models.
Towards the First Machine Translation System for Sumerian Transliterations
Authors: Ravneet Punia, Niko Schenk, Christian Chiarcos, {\'E}milie Pag{\'e}-Perron
Date: 2020-12-01 00:00:00
Proceeding: COLING 2020 8
Tasks: Cultural Vocal Bursts Intensity Prediction, Machine Translation, Transfer Learning, Translation

The Sumerian cuneiform script was invented more than 5,000 years ago and represents one of the oldest in history. We present the first attempt to translate Sumerian texts into English automatically. We publicly release high-quality corpora for standardized training and evaluation and report results on experiments with supervised, phrase-based, and transfer learning techniques for machine translation. Quantitative and qualitative evaluations indicate the usefulness of the translations. Our proposed methodology provides a broader audience of researchers with novel access to the data, accelerates the costly and time-consuming manual translation process, and helps them better explore the relationships between Sumerian cuneiform and Mesopotamian culture.
Using Bilingual Patents for Translation Training
Authors: John Lee, Benjamin Tsou, Tianyuan Cai
Date: 2020-12-01 00:00:00
Proceeding: COLING 2020 8
Tasks: Machine Translation, Translation

While bilingual corpora have been instrumental for machine translation, their utility for training translators has been less explored. We investigate the use of bilingual corpora as pedagogical tools for translation in the technical domain. In a user study, novice translators revised Chinese translations of English patents through bilingual concordancing. Results show that concordancing with an in-domain bilingual corpus can yield greater improvement in translation quality of technical terms than a general-domain bilingual corpus.