pwc-archive/papers-with-abstracts huggingface dataset:
Bayes Test of Precision, Recall, and F1 Measure for Comparison of Two Natural Language Processing Models
Authors: Ruibo Wang, Jihong Li
Date: 2019-07-01 00:00:00
Proceeding: ACL 2019 7
Tasks: Chunking, Two-sample testing

Direct comparison on point estimation of the precision (P), recall (R), and F1 measure of two natural language processing (NLP) models on a common test corpus is unreasonable and results in less replicable conclusions due to a lack of a statistical test. However, the existing t-tests in cross-validation (CV) for model comparison are inappropriate because the distributions of P, R, F1 are skewed and an interval estimation of P, R, and F1 based on a t-test may exceed [0,1]. In this study, we propose to use a block-regularized 3{\mbox{$\times$}}2 CV (3{\mbox{$\times$}}2 BCV) in model comparison because it could regularize the difference in certain frequency distributions over linguistic units between training and validation sets and yield stable estimators of P, R, and F1. On the basis of the 3{\mbox{$\times$}}2 BCV, we calibrate the posterior distributions of P, R, and F1 and derive an accurate interval estimation of P, R, and F1. Furthermore, we formulate the comparison into a hypothesis testing problem and propose a novel Bayes test. The test could directly compute the probabilities of the hypotheses on the basis of the posterior distributions and provide more informative decisions than the existing significance t-tests. Three experiments with regard to NLP chunking tasks are conducted, and the results illustrate the validity of the Bayes test.
Keeping Notes: Conditional Natural Language Generation with a Scratchpad Encoder
Authors: Ryan Benmalek, Madian Khabsa, Suma Desu, Claire Cardie, Michele Banko
Date: 2019-07-01 00:00:00
Proceeding: ACL 2019 7
Tasks: Decoder, Machine Translation, Question Generation, Question-Generation, Text Generation, Text Summarization, Translation

We introduce the Scratchpad Mechanism, a novel addition to the sequence-to-sequence (seq2seq) neural network architecture and demonstrate its effectiveness in improving the overall fluency of seq2seq models for natural language generation tasks. By enabling the decoder at each time step to write to all of the encoder output layers, Scratchpad can employ the encoder as a {``}scratchpad{''} memory to keep track of what has been generated so far and thereby guide future generation. We evaluate Scratchpad in the context of three well-studied natural language generation tasks {---} Machine Translation, Question Generation, and Text Summarization {---} and obtain state-of-the-art or comparable performance on standard datasets for each task. Qualitative assessments in the form of human judgements (question generation), attention visualization (MT), and sample output (summarization) provide further evidence of the ability of Scratchpad to generate fluent and expressive output.
Employing the Correspondence of Relations and Connectives to Identify Implicit Discourse Relations via Label Embeddings
Authors: Linh The Nguyen, Linh Van Ngo, Khoat Than, Thien Huu Nguyen
Date: 2019-07-01 00:00:00
Proceeding: ACL 2019 7
Tasks: Multi-Task Learning, Transfer Learning

It has been shown that implicit connectives can be exploited to improve the performance of the models for implicit discourse relation recognition (IDRR). An important property of the implicit connectives is that they can be accurately mapped into the discourse relations conveying their functions. In this work, we explore this property in a multi-task learning framework for IDRR in which the relations and the connectives are simultaneously predicted, and the mapping is leveraged to transfer knowledge between the two prediction tasks via the embeddings of relations and connectives. We propose several techniques to enable such knowledge transfer that yield the state-of-the-art performance for IDRR on several settings of the benchmark dataset (i.e., the Penn Discourse Treebank dataset).
Do You Know That Florence Is Packed with Visitors? Evaluating State-of-the-art Models of Speaker Commitment
Authors: Nanjiang Jiang, Marie-Catherine de Marneffe
Date: 2019-07-01 00:00:00
Proceeding: ACL 2019 7
Tasks: Negation, Question Answering

When a speaker, Mary, asks {``}Do you know that Florence is packed with visitors?{''}, we take her to believe that Florence is packed with visitors, but not if she asks {``}Do you think that Florence is packed with visitors?{''}. Inferring speaker commitment (aka event factuality) is crucial for information extraction and question answering. Here, we explore the hypothesis that linguistic deficits drive the error patterns of existing speaker commitment models by analyzing the linguistic correlates of model error on a challenging naturalistic dataset. We evaluate two state-of-the-art speaker commitment models on the CommitmentBank, an English dataset of naturally occurring discourses. The CommitmentBank is annotated with speaker commitment towards the content of the complement ({``}Florence is packed with visitors{''} in our example) of clause-embedding verbs ({``}know{''}, {``}think{''}) under four entailment-canceling environments (negation, modal, question, conditional). A breakdown of items by linguistic features reveals asymmetrical error patterns: while the models achieve good performance on some classes (e.g., negation), they fail to generalize to the diverse linguistic constructions (e.g., conditionals) in natural language, highlighting directions for improvement.
Multi-Relational Script Learning for Discourse Relations
Authors: I-Ta Lee, Dan Goldwasser
Date: 2019-07-01 00:00:00
Proceeding: ACL 2019 7

Modeling script knowledge can be useful for a wide range of NLP tasks. Current statistical script learning approaches embed the events, such that their relationships are indicated by their similarity in the embedding. While intuitive, these approaches fall short of representing nuanced relations, needed for downstream tasks. In this paper, we suggest to view learning event embedding as a multi-relational problem, which allows us to capture different aspects of event pairs. We model a rich set of event relations, such as Cause and Contrast, derived from the Penn Discourse Tree Bank. We evaluate our model on three types of tasks, the popular Mutli-Choice Narrative Cloze and its variants, several multi-relational prediction tasks, and a related downstream task{---}implicit discourse sense classification.
Open-Domain Why-Question Answering with Adversarial Learning to Encode Answer Texts
Authors: Jong-Hoon Oh, Kazuma Kadowaki, Julien Kloetzer, Ryu Iida, Kentaro Torisawa
Date: 2019-07-01 00:00:00
Proceeding: ACL 2019 7
Tasks: Question Answering, Sentence

In this paper, we propose a method for why-question answering (why-QA) that uses an adversarial learning framework. Existing why-QA methods retrieve {``}answer passages{''} that usually consist of several sentences. These multi-sentence passages contain not only the reason sought by a why-question and its connection to the why-question, but also redundant and/or unrelated parts. We use our proposed {``}Adversarial networks for Generating compact-answer Representation{''} (AGR) to generate from a passage a vector representation of the non-redundant reason sought by a why-question and exploit the representation for judging whether the passage actually answers the why-question. Through a series of experiments using Japanese why-QA datasets, we show that these representations improve the performance of our why-QA neural model as well as that of a BERT-based why-QA model. We show that they also improve a state-of-the-art distantly supervised open-domain QA (DS-QA) method on publicly available English datasets, even though the target task is not a why-QA.
AdaNSP: Uncertainty-driven Adaptive Decoding in Neural Semantic Parsing
Authors: Xiang Zhang, Shizhu He, Kang Liu, Jun Zhao
Date: 2019-07-01 00:00:00
Proceeding: ACL 2019 7
Tasks: Decoder, Semantic Parsing, Sentence

Neural semantic parsers utilize the encoder-decoder framework to learn an end-to-end model for semantic parsing that transduces a natural language sentence to the formal semantic representation. To keep the model aware of the underlying grammar in target sequences, many constrained decoders were devised in a multi-stage paradigm, which decode to the sketches or abstract syntax trees first, and then decode to target semantic tokens. We instead to propose an adaptive decoding method to avoid such intermediate representations. The decoder is guided by model uncertainty and automatically uses deeper computations when necessary. Thus it can predict tokens adaptively. Our model outperforms the state-of-the-art neural models and does not need any expertise like predefined grammar or sketches in the meantime.
Eliciting Knowledge from Experts: Automatic Transcript Parsing for Cognitive Task Analysis
Authors: Junyi Du, He Jiang, Jiaming Shen, Xiang Ren
Date: 2019-07-01 00:00:00
Proceeding: ACL 2019 7
Tasks: Relation Extraction, Sentence

Cognitive task analysis (CTA) is a type of analysis in applied psychology aimed at eliciting and representing the knowledge and thought processes of domain experts. In CTA, often heavy human labor is involved to parse the interview transcript into structured knowledge (e.g., flowchart for different actions). To reduce human efforts and scale the process, automated CTA transcript parsing is desirable. However, this task has unique challenges as (1) it requires the understanding of long-range context information in conversational text; and (2) the amount of labeled data is limited and indirect{---}i.e., context-aware, noisy, and low-resource. In this paper, we propose a weakly-supervised information extraction framework for automated CTA transcript parsing. We partition the parsing process into a sequence labeling task and a text span-pair relation extraction task, with distant supervision from human-curated protocol files. To model long-range context information for extracting sentence relations, neighbor sentences are involved as a part of input. Different types of models for capturing context dependency are then applied. We manually annotate real-world CTA transcripts to facilitate the evaluation of the parsing tasks.
Beyond BLEU:Training Neural Machine Translation with Semantic Similarity
Authors: John Wieting, Taylor Berg-Kirkpatrick, Kevin Gimpel, Graham Neubig
Date: 2019-07-01 00:00:00
Proceeding: ACL 2019 7
Tasks: Diversity, Machine Translation, NMT, Semantic Similarity, Semantic Textual Similarity, Translation

While most neural machine translation (NMT)systems are still trained using maximum likelihood estimation, recent work has demonstrated that optimizing systems to directly improve evaluation metrics such as BLEU can significantly improve final translation accuracy. However, training with BLEU has some limitations: it doesn{'}t assign partial credit, it has a limited range of output values, and it can penalize semantically correct hypotheses if they differ lexically from the reference. In this paper, we introduce an alternative reward function for optimizing NMT systems that is based on recent work in semantic similarity. We evaluate on four disparate languages trans-lated to English, and find that training with our proposed metric results in better translations as evaluated by BLEU, semantic similarity, and human evaluation, and also that the optimization procedure converges faster. Analysis suggests that this is because the proposed metric is more conducive to optimization, assigning partial credit and providing more diversity in scores than BLEU
AutoML Strategy Based on Grammatical Evolution: A Case Study about Knowledge Discovery from Text
Authors: Suilan Estevez-Velarde, Yoan Guti{\'e}rrez, Andr{\'e}s Montoyo, Yudivi{\'a}n Almeida-Cruz
Date: 2019-07-01 00:00:00
Proceeding: ACL 2019 7
Tasks: AutoML, BIG-bench Machine Learning, feature selection

The process of extracting knowledge from natural language text poses a complex problem that requires both a combination of machine learning techniques and proper feature selection. Recent advances in Automatic Machine Learning (AutoML) provide effective tools to explore large sets of algorithms, hyper-parameters and features to find out the most suitable combination of them. This paper proposes a novel AutoML strategy based on probabilistic grammatical evolution, which is evaluated on the health domain by facing the knowledge discovery challenge in Spanish text documents. Our approach achieves state-of-the-art results and provides interesting insights into the best combination of parameters and algorithms to use when dealing with this challenge. Source code is provided for the research community.
Distilling Discrimination and Generalization Knowledge for Event Detection via Delta-Representation Learning
Authors: Yaojie Lu, Hongyu Lin, Xianpei Han, Le Sun
Date: 2019-07-01 00:00:00
Proceeding: ACL 2019 7
Tasks: Event Detection, Representation Learning

Event detection systems rely on discrimination knowledge to distinguish ambiguous trigger words and generalization knowledge to detect unseen/sparse trigger words. Current neural event detection approaches focus on trigger-centric representations, which work well on distilling discrimination knowledge, but poorly on learning generalization knowledge. To address this problem, this paper proposes a Delta-learning approach to distill discrimination and generalization knowledge by effectively decoupling, incrementally learning and adaptively fusing event representation. Experiments show that our method significantly outperforms previous approaches on unseen/sparse trigger words, and achieves state-of-the-art performance on both ACE2005 and KBP2017 datasets.
Chinese Relation Extraction with Multi-Grained Information and External Linguistic Knowledge
Authors: Ziran Li, Ning Ding, Zhiyuan Liu, Hai-Tao Zheng, Ying Shen
Date: 2019-07-01 00:00:00
Proceeding: ACL 2019 7
Tasks: Relation, Relation Extraction, Segmentation

Chinese relation extraction is conducted using neural networks with either character-based or word-based inputs, and most existing methods typically suffer from segmentation errors and ambiguity of polysemy. To address the issues, we propose a multi-grained lattice framework (MG lattice) for Chinese relation extraction to take advantage of multi-grained language information and external linguistic knowledge. In this framework, (1) we incorporate word-level information into character sequence inputs so that segmentation errors can be avoided. (2) We also model multiple senses of polysemous words with the help of external linguistic knowledge, so as to alleviate polysemy ambiguity. Experiments on three real-world datasets in distinct domains show consistent and significant superiority and robustness of our model, as compared with other baselines. We will release the source code of this paper in the future.
A2N: Attending to Neighbors for Knowledge Graph Inference
Authors: Trapit Bansal, Da-Cheng Juan, Sujith Ravi, Andrew McCallum
Date: 2019-07-01 00:00:00
Proceeding: ACL 2019 7
Tasks: Knowledge Graph Completion, Link Prediction

State-of-the-art models for knowledge graph completion aim at learning a fixed embedding representation of entities in a multi-relational graph which can generalize to infer unseen entity relationships at test time. This can be sub-optimal as it requires memorizing and generalizing to all possible entity relationships using these fixed representations. We thus propose a novel attention-based method to learn query-dependent representation of entities which adaptively combines the relevant graph neighborhood of an entity leading to more accurate KG completion. The proposed method is evaluated on two benchmark datasets for knowledge graph completion, and experimental results show that the proposed model performs competitively or better than existing state-of-the-art, including recent methods for explicit multi-hop reasoning. Qualitative probing offers insight into how the model can reason about facts involving multiple hops in the knowledge graph, through the use of neighborhood attention.
Language Modeling with Shared Grammar
Authors: Yuyu Zhang, Le Song
Date: 2019-07-01 00:00:00
Proceeding: ACL 2019 7
Tasks: Language Modeling, Language Modelling

Sequential recurrent neural networks have achieved superior performance on language modeling, but overlook the structure information in natural language. Recent works on structure-aware models have shown promising results on language modeling. However, how to incorporate structure knowledge on corpus without syntactic annotations remains an open problem. In this work, we propose neural variational language model (NVLM), which enables the sharing of grammar knowledge among different corpora. Experimental results demonstrate the effectiveness of our framework on two popular benchmark datasets. With the help of shared grammar, our language model converges significantly faster to a lower perplexity on new training corpus.
Complex Question Decomposition for Semantic Parsing
Authors: Haoyu Zhang, Jingjing Cai, Jianjun Xu, Ji Wang
Date: 2019-07-01 00:00:00
Proceeding: ACL 2019 7
Tasks: Semantic Parsing

In this work, we focus on complex question semantic parsing and propose a novel Hierarchical Semantic Parsing (HSP) method, which utilizes the decompositionality of complex questions for semantic parsing. Our model is designed within a three-stage parsing architecture based on the idea of decomposition-integration. In the first stage, we propose a question decomposer which decomposes a complex question into a sequence of sub-questions. In the second stage, we design an information extractor to derive the type and predicate information of these questions. In the last stage, we integrate the generated information from previous stages and generate a logical form for the complex question. We conduct experiments on COMPLEXWEBQUESTIONS which is a large scale complex question semantic parsing dataset, results show that our model achieves significant improvement compared to state-of-the-art methods.
DisSent: Learning Sentence Representations from Explicit Discourse Relations
Authors: Allen Nie, Erin Bennett, Noah Goodman
Date: 2019-07-01 00:00:00
Proceeding: ACL 2019 7
Tasks: Dependency Parsing, Natural Language Understanding, Relation, Relation Prediction, Sentence, Sentence Embeddings

Learning effective representations of sentences is one of the core missions of natural language understanding. Existing models either train on a vast amount of text, or require costly, manually curated sentence relation datasets. We show that with dependency parsing and rule-based rubrics, we can curate a high quality sentence relation task by leveraging explicit discourse relations. We show that our curated dataset provides an excellent signal for learning vector representations of sentence meaning, representing relations that can only be determined when the meanings of two sentences are combined. We demonstrate that the automatically curated corpus allows a bidirectional LSTM sentence encoder to yield high quality sentence embeddings and can serve as a supervised fine-tuning dataset for larger models such as BERT. Our fixed sentence embeddings achieve high performance on a variety of transfer tasks, including SentEval, and we achieve state-of-the-art results on Penn Discourse Treebank{'}s implicit relation prediction task.
EigenSent: Spectral sentence embeddings using higher-order Dynamic Mode Decomposition
Authors: Subhradeep Kayal, George Tsatsaronis
Date: 2019-07-01 00:00:00
Proceeding: ACL 2019 7
Tasks: Sentence, Sentence Embeddings, Word Embeddings

Distributed representation of words, or word embeddings, have motivated methods for calculating semantic representations of word sequences such as phrases, sentences and paragraphs. Most of the existing methods to do so either use algorithms to learn such representations, or improve on calculating weighted averages of the word vectors. In this work, we experiment with spectral methods of signal representation and summarization as mechanisms for constructing such word-sequence embeddings in an unsupervised fashion. In particular, we explore an algorithm rooted in fluid-dynamics, known as higher-order Dynamic Mode Decomposition, which is designed to capture the eigenfrequencies, and hence the fundamental transition dynamics, of periodic and quasi-periodic systems. It is empirically observed that this approach, which we call EigenSent, can summarize transitions in a sequence of words and generate an embedding that can represent well the sequence itself. To the best of the authors{'} knowledge, this is the first application of a spectral decomposition and signal summarization technique on text, to create sentence embeddings. We test the efficacy of this algorithm in creating sentence embeddings on three public datasets, where it performs appreciably well. Moreover it is also shown that, due to the positive combination of their complementary properties, concatenating the embeddings generated by EigenSent with simple word vector averaging achieves state-of-the-art results.
Reranking for Neural Semantic Parsing
Authors: Pengcheng Yin, Graham Neubig
Date: 2019-07-01 00:00:00
Proceeding: ACL 2019 7
Tasks: Code Generation, Reranking, Semantic Parsing

Semantic parsing considers the task of transducing natural language (NL) utterances into machine executable meaning representations (MRs). While neural network-based semantic parsers have achieved impressive improvements over previous methods, results are still far from perfect, and cursory manual inspection can easily identify obvious problems such as lack of adequacy or coherence of the generated MRs. This paper presents a simple approach to quickly iterate and improve the performance of an existing neural semantic parser by reranking an n-best list of predicted MRs, using features that are designed to fix observed problems with baseline models. We implement our reranker in a competitive neural semantic parser and test on four semantic parsing (GEO, ATIS) and Python code generation (Django, CoNaLa) tasks, improving the strong baseline parser by up to 5.7{\%} absolute in BLEU (CoNaLa) and 2.9{\%} in accuracy (Django), outperforming the best published neural parser results on all four datasets.
Towards Multimodal Sarcasm Detection (An \_Obviously\_ Perfect Paper)
Authors: Santiago Castro, Devamanyu Hazarika, Ver{\'o}nica P{\'e}rez-Rosas, Roger Zimmermann, Rada Mihalcea, Soujanya Poria
Date: 2019-07-01 00:00:00
Proceeding: ACL 2019 7
Tasks: Sarcasm Detection

Sarcasm is often expressed through several verbal and non-verbal cues, e.g., a change of tone, overemphasis in a word, a drawn-out syllable, or a straight looking face. Most of the recent work in sarcasm detection has been carried out on textual data. In this paper, we argue that incorporating multimodal cues can improve the automatic classification of sarcasm. As a first step towards enabling the development of multimodal approaches for sarcasm detection, we propose a new sarcasm dataset, Multimodal Sarcasm Detection Dataset (MUStARD), compiled from popular TV shows. MUStARD consists of audiovisual utterances annotated with sarcasm labels. Each utterance is accompanied by its context of historical utterances in the dialogue, which provides additional information on the scenario where the utterance occurs. Our initial results show that the use of multimodal information can reduce the relative error rate of sarcasm detection by up to 12.9{\%} in F-score when compared to the use of individual modalities. The full dataset is publicly available for use at https://github.com/soujanyaporia/MUStARD.