pwc-archive/papers-with-abstracts huggingface dataset:
Coarse-to-Fine Query Focused Multi-Document Summarization
Authors: Yumo Xu, Mirella Lapata
Date: 2020-11-01 00:00:00
Proceeding: EMNLP 2020 11
Tasks: Document Summarization, Multi-Document Summarization, Question Answering, Retrieval

We consider the problem of better modeling query-cluster interactions to facilitate query focused multi-document summarization. Due to the lack of training data, existing work relies heavily on retrieval-style methods for assembling query relevant summaries. We propose a coarse-to-fine modeling framework which employs progressively more accurate modules for estimating whether text segments are relevant, likely to contain an answer, and central. The modules can be independently developed and leverage training data if available. We present an instantiation of this framework with a trained evidence estimator which relies on distant supervision from question answering (where various resources exist) to identify segments which are likely to answer the query and should be included in the summary. Our framework is robust across domains and query types (i.e., long vs short) and outperforms strong comparison systems on benchmark datasets.
Public Sentiment Drift Analysis Based on Hierarchical Variational Auto-encoder
Authors: Wenyue Zhang, XiaoLi Li, Yang Li, Suge Wang, Deyu Li, Jian Liao, Jianxing Zheng
Date: 2020-11-01 00:00:00
Proceeding: EMNLP 2020 11

Detecting public sentiment drift is a challenging task due to sentiment change over time. Existing methods first build a classification model using historical data and subsequently detect drift if the model performs much worse on new data. In this paper, we focus on distribution learning by proposing a novel Hierarchical Variational Auto-Encoder (HVAE) model to learn better distribution representation, and design a new drift measure to directly evaluate distribution changes between historical data and new data.Our experimental results demonstrate that our proposed model achieves better results than three existing state-of-the-art methods.
Point to the Expression: Solving Algebraic Word Problems using the Expression-Pointer Transformer Model
Authors: Bugeun Kim, Kyung Seo Ki, Donggeon Lee, Gahgene Gweon
Date: 2020-11-01 00:00:00
Proceeding: EMNLP 2020 11
Tasks: Math Word Problem Solving

Solving algebraic word problems has recently emerged as an important natural language processing task. To solve algebraic word problems, recent studies suggested neural models that generate solution equations by using {`}Op (operator/operand){'} tokens as a unit of input/output. However, such a neural model suffered two issues: expression fragmentation and operand-context separation. To address each of these two issues, we propose a pure neural model, Expression-Pointer Transformer (EPT), which uses (1) {`}Expression{'} token and (2) operand-context pointers when generating solution equations. The performance of the EPT model is tested on three datasets: ALG514, DRAW-1K, and MAWPS. Compared to the state-of-the-art (SoTA) models, the EPT model achieved a comparable performance accuracy in each of the three datasets; 81.3{\%} on ALG514, 59.5{\%} on DRAW-1K, and 84.5{\%} on MAWPS. The contribution of this paper is two-fold; (1) We propose a pure neural model, EPT, which can address the expression fragmentation and the operand-context separation. (2) The fully automatic EPT model, which does not use hand-crafted features, yields comparable performance to existing models using hand-crafted features, and achieves better performance than existing pure neural models by at most 40{\%}.
Routing Enforced Generative Model for Recipe Generation
Authors: Zhiwei Yu, Hongyu Zang, Xiaojun Wan
Date: 2020-11-01 00:00:00
Proceeding: EMNLP 2020 11
Tasks: model, Recipe Generation

One of the most challenging part of recipe generation is to deal with the complex restrictions among the input ingredients. Previous researches simplify the problem by treating the inputs independently and generating recipes containing as much information as possible. In this work, we propose a routing method to dive into the content selection under the internal restrictions. The routing enforced generative model (RGM) can generate appropriate recipes according to the given ingredients and user preferences. Our model yields new state-of-the-art results on the recipe generation task with significant improvements on BLEU, F1 and human evaluation.
Selection and Generation: Learning towards Multi-Product Advertisement Post Generation
Authors: Zhangming Chan, Yuchi Zhang, Xiuying Chen, Shen Gao, Zhiqiang Zhang, Dongyan Zhao, Rui Yan
Date: 2020-11-01 00:00:00
Proceeding: EMNLP 2020 11

As the E-commerce thrives, high-quality online advertising copywriting has attracted more and more attention. Different from the advertising copywriting for a single product, an advertisement (AD) post includes an attractive topic that meets the customer needs and description copywriting about several products under its topic. A good AD post can highlight the characteristics of each product, thus helps customers make a good choice among candidate products. Hence, multi-product AD post generation is meaningful and important. We propose a novel end-to-end model named S-MG Net to generate the AD post. Targeted at such a challenging real-world problem, we split the AD post generation task into two subprocesses: (1) select a set of products via the SelectNet (Selection Network). (2) generate a post including selected products via the MGenNet (Multi-Generator Network). Concretely, SelectNet first captures the post topic and the relationship among the products to output the representative products. Then, MGenNet generates the description copywriting of each product. Experiments conducted on a large-scale real-world AD post dataset demonstrate that our proposed model achieves impressive performance in terms of both automatic metrics as well as human evaluations.
Domain Adaptation of Thai Word Segmentation Models using Stacked Ensemble
Authors: Peerat Limkonchotiwat, Wannaphong Phatthiyaphaibun, Raheem Sarwar, Ekapol Chuangsuwanich, Sarana Nutanong
Date: 2020-11-01 00:00:00
Proceeding: EMNLP 2020 11
Tasks: Domain Adaptation, Ensemble Learning, Thai Word Segmentation, Transfer Learning

Like many Natural Language Processing tasks, Thai word segmentation is domain-dependent. Researchers have been relying on transfer learning to adapt an existing model to a new domain. However, this approach is inapplicable to cases where we can interact with only input and output layers of the models, also known as {``}black boxes{''}. We propose a filter-and-refine solution based on the stacked-ensemble learning paradigm to address this black-box limitation. We conducted extensive experimental studies comparing our method against state-of-the-art models and transfer learning. Experimental results show that our proposed solution is an effective domain adaptation method and has a similar performance as the transfer learning method.
A Joint Multiple Criteria Model in Transfer Learning for Cross-domain Chinese Word Segmentation
Authors: Kaiyu Huang, Degen Huang, Zhuang Liu, Fengran Mo
Date: 2020-11-01 00:00:00
Proceeding: EMNLP 2020 11
Tasks: Chinese Word Segmentation, Segmentation, Transfer Learning

Word-level information is important in natural language processing (NLP), especially for the Chinese language due to its high linguistic complexity. Chinese word segmentation (CWS) is an essential task for Chinese downstream NLP tasks. Existing methods have already achieved a competitive performance for CWS on large-scale annotated corpora. However, the accuracy of the method will drop dramatically when it handles an unsegmented text with lots of out-of-vocabulary (OOV) words. In addition, there are many different segmentation criteria for addressing different requirements of downstream NLP tasks. Excessive amounts of models with saving different criteria will generate the explosive growth of the total parameters. To this end, we propose a joint multiple criteria model that shares all parameters to integrate different segmentation criteria into one model. Besides, we utilize a transfer learning method to improve the performance of OOV words. Our proposed method is evaluated by designing comprehensive experiments on multiple benchmark datasets (e.g., Bakeoff 2005, Bakeoff 2008 and SIGHAN 2010). Our method achieves the state-of-the-art performances on all datasets. Importantly, our method also shows a competitive practicability and generalization ability for the CWS task.
Alignment-free Cross-lingual Semantic Role Labeling
Authors: Rui Cai, Mirella Lapata
Date: 2020-11-01 00:00:00
Proceeding: EMNLP 2020 11
Tasks: Machine Translation, Multilingual Word Embeddings, Semantic Role Labeling, Translation, Word Embeddings

Cross-lingual semantic role labeling (SRL) aims at leveraging resources in a source language to minimize the effort required to construct annotations or models for a new target language. Recent approaches rely on word alignments, machine translation engines, or preprocessing tools such as parsers or taggers. We propose a cross-lingual SRL model which only requires annotations in a source language and access to raw text in the form of a parallel corpus. The backbone of our model is an LSTM-based semantic role labeler jointly trained with a semantic role compressor and multilingual word embeddings. The compressor collects useful information from the output of the semantic role labeler, filtering noisy and conflicting evidence. It lives in a multilingual embedding space and provides direct supervision for predicting semantic roles in the target language. Results on the Universal Proposition Bank and manually annotated datasets show that our method is highly effective, even against systems utilizing supervised features.
Improving Out-of-Scope Detection in Intent Classification by Using Embeddings of the Word Graph Space of the Classes
Authors: Paulo Cavalin, Victor Henrique Alves Ribeiro, Ana Appel, Claudio Pinhanez
Date: 2020-11-01 00:00:00
Proceeding: EMNLP 2020 11
Tasks: Classification, General Classification, Graph Embedding, intent-classification, Intent Classification

This paper explores how intent classification can be improved by representing the class labels not as a discrete set of symbols but as a space where the word graphs associated to each class are mapped using typical graph embedding techniques. The approach, inspired by a previous algorithm used for an inverse dictionary task, allows the classification algorithm to take in account inter-class similarities provided by the repeated occurrence of some words in the training examples of the different classes. The classification is carried out by mapping text embeddings to the word graph embeddings of the classes. Focusing solely on improving the representation of the class label set, we show in experiments conducted in both private and public intent classification datasets, that better detection of out-of-scope examples (OOS) is achieved and, as a consequence, that the overall accuracy of intent classification is also improved. In particular, using the recently-released \textit{Larson dataset}, an error of about 9.9{\%} has been achieved for OOS detection, beating the previous state-of-the-art result by more than 31 percentage points.
Surprisal Predicts Code-Switching in Chinese-English Bilingual Text
Authors: Jes{\'u}s Calvillo, Le Fang, Jeremy Cole, David Reitter
Date: 2020-11-01 00:00:00
Proceeding: EMNLP 2020 11
Tasks: Sentence

Why do bilinguals switch languages within a sentence? The present observational study asks whether word surprisal and word entropy predict code-switching in bilingual written conversation. We describe and model a new dataset of Chinese-English text with 1476 clean code-switched sentences, translated back into Chinese. The model includes known control variables together with word surprisal and word entropy. We found that word surprisal, but not entropy, is a significant predictor that explains code-switching above and beyond other well-known predictors. We also found sentence length to be a significant predictor, which has been related to sentence complexity. We propose high cognitive effort as a reason for code-switching, as it leaves fewer resources for inhibition of the alternative language. We also corroborate previous findings, but this time using a computational model of surprisal, a new language pair, and doing so for written language.
Improving Word Sense Disambiguation with Translations
Authors: Yixing Luan, Bradley Hauer, Lili Mou, Grzegorz Kondrak
Date: 2020-11-01 00:00:00
Proceeding: EMNLP 2020 11
Tasks: Machine Translation, Translation, Word Sense Disambiguation

It has been conjectured that multilingual information can help monolingual word sense disambiguation (WSD). However, existing WSD systems rarely consider multilingual information, and no effective method has been proposed for improving WSD by generating translations. In this paper, we present a novel approach that improves the performance of a base WSD system using machine translation. Since our approach is language independent, we perform WSD experiments on several languages. The results demonstrate that our methods can consistently improve the performance of WSD systems, and obtain state-ofthe-art results in both English and multilingual WSD. To facilitate the use of lexical translation information, we also propose BABALIGN, an precise bitext alignment algorithm which is guided by multilingual lexical correspondences from BabelNet.
Towards Better Context-aware Lexical Semantics:Adjusting Contextualized Representations through Static Anchors
Authors: Qianchu Liu, Diana McCarthy, Anna Korhonen
Date: 2020-11-01 00:00:00
Proceeding: EMNLP 2020 11

One of the most powerful features of contextualized models is their dynamic embeddings for words in context, leading to state-of-the-art representations for context-aware lexical semantics. In this paper, we present a post-processing technique that enhances these representations by learning a transformation through static anchors. Our method requires only another pre-trained model and no labeled data is needed. We show consistent improvement in a range of benchmark tasks that test contextual variations of meaning both across different usages of a word and across different words as they are used in context. We demonstrate that while the original contextual representations can be improved by another embedding space from both contextualized and static models, the static embeddings, which have lower computational requirements, provide the most gains.
Do ``Undocumented Workers'' == ``Illegal Aliens''? Differentiating Denotation and Connotation in Vector Spaces
Authors: Albert Webson, Zhizhong Chen, Carsten Eickhoff, Ellie Pavlick
Date: 2020-11-01 00:00:00
Proceeding: EMNLP 2020 11
Tasks: Diversity, Information Retrieval, Retrieval

In politics, neologisms are frequently invented for partisan objectives. For example, {``}undocumented workers{''} and {``}illegal aliens{''} refer to the same group of people (i.e., they have the same denotation), but they carry clearly different connotations. Examples like these have traditionally posed a challenge to reference-based semantic theories and led to increasing acceptance of alternative theories (e.g., Two-Factor Semantics) among philosophers and cognitive scientists. In NLP, however, popular pretrained models encode both denotation and connotation as one entangled representation. In this study, we propose an adversarial nerual netowrk that decomposes a pretrained representation as independent denotation and connotation representations. For intrinsic interpretability, we show that words with the same denotation but different connotations (e.g., {``}immigrants{''} vs. {``}aliens{''}, {``}estate tax{''} vs. {``}death tax{''}) move closer to each other in denotation space while moving further apart in connotation space. For extrinsic application, we train an information retrieval system with our disentangled representations and show that the denotation vectors improve the viewpoint diversity of document rankings.
CLIRMatrix: A massively large collection of bilingual and multilingual datasets for Cross-Lingual Information Retrieval
Authors: Shuo Sun, Kevin Duh
Date: 2020-11-01 00:00:00
Proceeding: EMNLP 2020 11
Tasks: Cross-Lingual Information Retrieval, Information Retrieval, Retrieval

We present CLIRMatrix, a massively large collection of bilingual and multilingual datasets for Cross-Lingual Information Retrieval extracted automatically from Wikipedia. CLIRMatrix comprises (1) BI-139, a bilingual dataset of queries in one language matched with relevant documents in another language for 139x138=19,182 language pairs, and (2) MULTI-8, a multilingual dataset of queries and documents jointly aligned in 8 different languages. In total, we mined 49 million unique queries and 34 billion (query, document, label) triplets, making it the largest and most comprehensive CLIR dataset to date. This collection is intended to support research in end-to-end neural information retrieval and is publicly available at [url]. We provide baseline neural model results on BI-139, and evaluate MULTI-8 in both single-language retrieval and mix-language retrieval settings.
Ad-hoc Document Retrieval using Weak-Supervision with BERT and GPT2
Authors: Yosi Mass, Haggai Roitman
Date: 2020-11-01 00:00:00
Proceeding: EMNLP 2020 11
Tasks: Deep Learning, Information Retrieval, Retrieval

We describe a weakly-supervised method for training deep learning models for the task of ad-hoc document retrieval. Our method is based on generative and discriminative models that are trained using weak-supervision just from the documents in the corpus. We present an end-to-end retrieval system that starts with traditional information retrieval methods, followed by two deep learning re-rankers. We evaluate our method on three different datasets: a COVID-19 related scientific literature dataset and two news datasets. We show that our method outperforms state-of-the-art methods; this without the need for the expensive process of manually labeling data.
Learning Explainable Linguistic Expressions with Neural Inductive Logic Programming for Sentence Classification
Authors: Prithviraj Sen, Marina Danilevsky, Yunyao Li, Siddhartha Brahma, Matthias Boehm, Laura Chiticariu, Rajasekar Krishnamurthy
Date: 2020-11-01 00:00:00
Proceeding: EMNLP 2020 11
Tasks: General Classification, Inductive logic programming, Natural Language Understanding, Relational Reasoning, Sentence, Sentence Classification

Interpretability of predictive models is becoming increasingly important with growing adoption in the real-world. We present RuleNN, a neural network architecture for learning transparent models for sentence classification. The models are in the form of rules expressed in first-order logic, a dialect with well-defined, human-understandable semantics. More precisely, RuleNN learns linguistic expressions (LE) built on top of predicates extracted using shallow natural language understanding. Our experimental results show that RuleNN outperforms statistical relational learning and other neuro-symbolic methods, and performs comparably with black-box recurrent neural networks. Our user studies confirm that the learned LEs are explainable and capture domain semantics. Moreover, allowing domain experts to modify LEs and instill more domain knowledge leads to human-machine co-creation of models with better performance.
Identifying Elements Essential for BERT's Multilinguality
Authors: Philipp Dufter, Hinrich Sch{\"u}tze
Date: 2020-11-01 00:00:00
Proceeding: EMNLP 2020 11

It has been shown that multilingual BERT (mBERT) yields high quality multilingual representations and enables effective zero-shot transfer. This is surprising given that mBERT does not use any crosslingual signal during training. While recent literature has studied this phenomenon, the reasons for the multilinguality are still somewhat obscure. We aim to identify architectural properties of BERT and linguistic properties of languages that are necessary for BERT to become multilingual. To allow for fast experimentation we propose an efficient setup with small BERT models trained on a mix of synthetic and natural data. Overall, we identify four architectural and two linguistic elements that influence multilinguality. Based on our insights, we experiment with a multilingual pretraining setup that modifies the masking strategy using VecMap, i.e., unsupervised embedding alignment. Experiments on XNLI with three languages indicate that our findings transfer from our small setup to larger scale settings.
Pre-tokenization of Multi-word Expressions in Cross-lingual Word Embeddings
Authors: Naoki Otani, Satoru Ozaki, Xingyuan Zhao, Yucen Li, Micaelah St Johns, Lori Levin
Date: 2020-11-01 00:00:00
Proceeding: EMNLP 2020 11
Tasks: Cross-Lingual Word Embeddings, Translation, Word Embeddings, Word Translation

Cross-lingual word embedding (CWE) algorithms represent words in multiple languages in a unified vector space. Multi-Word Expressions (MWE) are common in every language. When training word embeddings, each component word of an MWE gets its own separate embedding, and thus, MWEs are not translated by CWEs. We propose a simple method for word translation of MWEs to and from English in ten languages: we first compile lists of MWEs in each language and then tokenize the MWEs as single tokens before training word embeddings. CWEs are trained on a word-translation task using the dictionaries that only contain single words. In order to evaluate MWE translation, we created bilingual word lists from multilingual WordNet that include single-token words and MWEs, and most importantly, include MWEs that correspond to single words in another language. We release these dictionaries to the research community. We show that the pre-tokenization of MWEs as single tokens performs better than averaging the embeddings of the individual tokens of the MWE. We can translate MWEs at a top-10 precision of 30-60{\%}. The tokenization of MWEs makes the occurrences of single words in a training corpus more sparse, but we show that it does not pose negative impacts on single-word translations.
Monolingual Adapters for Zero-Shot Neural Machine Translation
Authors: Jerin Philip, Alexandre Berard, Matthias Gall{\'e}, Laurent Besacier
Date: 2020-11-01 00:00:00
Proceeding: EMNLP 2020 11
Tasks: Machine Translation, Translation

We propose a novel adapter layer formalism for adapting multilingual models. They are more parameter-efficient than existing adapter layers while obtaining as good or better performance. The layers are specific to one language (as opposed to bilingual adapters) allowing to compose them and generalize to unseen language-pairs. In this zero-shot setting, they obtain a median improvement of +2.77 BLEU points over a strong 20-language multilingual Transformer baseline trained on TED talks.