pwc-archive/papers-with-abstracts huggingface dataset:
Deconstructing Supertagging into Multi-Task Sequence Prediction
Authors: Zhenqi Zhu, Anoop Sarkar
Date: 2019-11-01 00:00:00
Proceeding: CONLL 2019 11
Tasks: Multi-Task Learning, Prediction, TAG

Supertagging is a sequence prediction task where each word is assigned a piece of complex syntactic structure called a supertag. We provide a novel approach to multi-task learning for Tree Adjoining Grammar (TAG) supertagging by deconstructing these complex supertags in order to define a set of related but auxiliary sequence prediction tasks. Our multi-task prediction framework is trained over the exactly same training data used to train the original supertagger where each auxiliary task provides an alternative view on the original prediction task. Our experimental results show that our multi-task approach significantly improves TAG supertagging with a new state-of-the-art accuracy score of 91.39{\%} on the Penn Treebank supertagging dataset.
Multilingual Model Using Cross-Task Embedding Projection
Authors: Jin Sakuma, Naoki Yoshinaga
Date: 2019-11-01 00:00:00
Proceeding: CONLL 2019 11
Tasks: Cross-Lingual Word Embeddings, model, Sentiment Analysis, Topic Classification, Word Embeddings

We present a method for applying a neural network trained on one (resource-rich) language for a given task to other (resource-poor) languages. We accomplish this by inducing a mapping from pre-trained cross-lingual word embeddings to the embedding layer of the neural network trained on the resource-rich language. To perform element-wise cross-task embedding projection, we invent locally linear mapping which assumes and preserves the local topology across the semantic spaces before and after the projection. Experimental results on topic classification task and sentiment analysis task showed that the fully task-specific multilingual model obtained using our method outperformed the existing multilingual models with embedding layers fixed to pre-trained cross-lingual word embeddings.
Investigating Cross-Lingual Alignment Methods for Contextualized Embeddings with Token-Level Evaluation
Authors: Qianchu Liu, Diana McCarthy, Ivan Vuli{\'c}, Anna Korhonen
Date: 2019-11-01 00:00:00
Proceeding: CONLL 2019 11
Tasks: Retrieval, Sentence, Sentence Retrieval, Word Similarity

In this paper, we present a thorough investigation on methods that align pre-trained contextualized embeddings into shared cross-lingual context-aware embedding space, providing strong reference benchmarks for future context-aware crosslingual models. We propose a novel and challenging task, Bilingual Token-level Sense Retrieval (BTSR). It specifically evaluates the accurate alignment of words with the same meaning in cross-lingual non-parallel contexts, currently not evaluated by existing tasks such as Bilingual Contextual Word Similarity and Sentence Retrieval. We show how the proposed BTSR task highlights the merits of different alignment methods. In particular, we find that using context average type-level alignment is effective in transferring monolingual contextualized embeddings cross-lingually especially in non-parallel contexts, and at the same time improves the monolingual space. Furthermore, aligning independently trained models yields better performance than aligning multilingual embeddings with shared vocabulary.
Large-Scale, Diverse, Paraphrastic Bitexts via Sampling and Clustering
Authors: J. Edward Hu, Abhinav Singh, Nils Holzenberger, Matt Post, Benjamin Van Durme
Date: 2019-11-01 00:00:00
Proceeding: CONLL 2019 11
Tasks: Clustering, Diversity, Sentence, Translation

Producing diverse paraphrases of a sentence is a challenging task. Natural paraphrase corpora are scarce and limited, while existing large-scale resources are automatically generated via back-translation and rely on beam search, which tends to lack diversity. We describe ParaBank 2, a new resource that contains multiple diverse sentential paraphrases, produced from a bilingual corpus using negative constraints, inference sampling, and clustering.We show that ParaBank 2 significantly surpasses prior work in both lexical and syntactic diversity while being meaning-preserving, as measured by human judgments and standardized metrics. Further, we illustrate how such paraphrastic resources may be used to refine contextualized encoders, leading to improvements in downstream tasks.
Representing Movie Characters in Dialogues
Authors: Mahmoud Azab, Noriyuki Kojima, Jia Deng, Rada Mihalcea
Date: 2019-11-01 00:00:00
Proceeding: CONLL 2019 11
Tasks: Question Answering, Relation Classification, Visual Question Answering, Visual Question Answering (VQA)

We introduce a new embedding model to represent movie characters and their interactions in a dialogue by encoding in the same representation the language used by these characters as well as information about the other participants in the dialogue. We evaluate the performance of these new character embeddings on two tasks: (1) character relatedness, using a dataset we introduce consisting of a dense character interaction matrix for 4,378 unique character pairs over 22 hours of dialogue from eighteen movies; and (2) character relation classification, for fine- and coarse-grained relations, as well as sentiment relations. Our experiments show that our model significantly outperforms the traditional Word2Vec continuous bag-of-words and skip-gram models, demonstrating the effectiveness of the character embeddings we introduce. We further show how these embeddings can be used in conjunction with a visual question answering system to improve over previous results.
Cross-Lingual Word Embeddings and the Structure of the Human Bilingual Lexicon
Authors: Paola Merlo, Maria Andueza Rodriguez
Date: 2019-11-01 00:00:00
Proceeding: CONLL 2019 11
Tasks: Cross-Lingual Word Embeddings, Translation, Word Embeddings

Research on the bilingual lexicon has uncovered fascinating interactions between the lexicons of the native language and of the second language in bilingual speakers. In particular, it has been found that the lexicon of the underlying native language affects the organisation of the second language. In the spirit of interpreting current distributed representations, this paper investigates two models of cross-lingual word embeddings, comparing them to the shared-translation effect and the cross-lingual coactivation effects of false and true friends (cognates) found in humans. We find that the similarity structure of the cross-lingual word embeddings space yields the same effects as the human bilingual lexicon.
Learning Conceptual Spaces with Disentangled Facets
Authors: Rana Alshaikh, Zied Bouraoui, Steven Schockaert
Date: 2019-11-01 00:00:00
Proceeding: CONLL 2019 11
Tasks: Word Embeddings

Conceptual spaces are geometric representations of meaning that were proposed by G Ìˆardenfors (2000). They share many similarities with the vector space embeddings that are commonly used in natural language processing. However, rather than representing entities in a single vector space, conceptual spaces are usually decomposed into several facets, each of which is then modelled as a relatively low dimensional vector space. Unfortunately, the problem of learning such conceptual spaces has thus far only received limited attention. To address this gap, we analyze how, and to what extent, a given vector space embedding can be decomposed into meaningful facets in an unsupervised fashion. While this problem is highly challenging, we show that useful facets can be discovered by relying on word embeddings to group semantically related features.
Weird Inflects but OK: Making Sense of Morphological Generation Errors
Authors: Kyle Gorman, Arya D. McCarthy, Ryan Cotterell, Ekaterina Vylomova, Miikka Silfverberg, Magdalena Markowska
Date: 2019-11-01 00:00:00
Proceeding: CONLL 2019 11
Tasks: Form, Text Generation

We conduct a manual error analysis of the CoNLL-SIGMORPHON Shared Task on Morphological Reinflection. This task involves natural language generation: systems are given a word in citation form (e.g., hug) and asked to produce the corresponding inflected form (e.g., the simple past hugged). We propose an error taxonomy and use it to annotate errors made by the top two systems across twelve languages. Many of the observed errors are related to inflectional patterns sensitive to inherent linguistic properties such as animacy or affect; many others are failures to predict truly unpredictable inflectional behaviors. We also find nearly one quarter of the residual {``}errors{''} reflect errors in the gold data.
Improving Natural Language Understanding by Reverse Mapping Bytepair Encoding
Authors: Chaodong Tong, Huailiang Peng, Qiong Dai, Lei Jiang, Jianghua Huang
Date: 2019-11-01 00:00:00
Proceeding: CONLL 2019 11
Tasks: Natural Language Understanding, RTE, SST-2

We propose a method called reverse mapping bytepair encoding, which maps named-entity information and other word-level linguistic features back to subwords during the encoding procedure of bytepair encoding (BPE). We employ this method to the Generative Pre-trained Transformer (OpenAI GPT) by adding a weighted linear layer after the embedding layer. We also propose a new model architecture named as the multi-channel separate transformer to employ a training process without parameter-sharing. Evaluation on Stories Cloze, RTE, SciTail and SST-2 datasets demonstrates the effectiveness of our approach.
Fully Unsupervised Crosslingual Semantic Textual Similarity Metric Based on BERT for Identifying Parallel Data
Authors: Chi-kiu Lo, Michel Simard
Date: 2019-11-01 00:00:00
Proceeding: CONLL 2019 11
Tasks: Machine Translation, Natural Language Understanding, Semantic Textual Similarity, STS, Translation

We present a fully unsupervised crosslingual semantic textual similarity (STS) metric, based on contextual embeddings extracted from BERT {--} Bidirectional Encoder Representations from Transformers (Devlin et al., 2019). The goal of crosslingual STS is to measure to what degree two segments of text in different languages express the same meaning. Not only is it a key task in crosslingual natural language understanding (XLU), it is also particularly useful for identifying parallel resources for training and evaluating downstream multilingual natural language processing (NLP) applications, such as machine translation. Most previous crosslingual STS methods relied heavily on existing parallel resources, thus leading to a circular dependency problem. With the advent of massively multilingual context representation models such as BERT, which are trained on the concatenation of non-parallel data from each language, we show that the deadlock around parallel resources can be broken. We perform intrinsic evaluations on crosslingual STS data sets and extrinsic evaluations on parallel corpus filtering and human translation equivalence assessment tasks. Our results show that the unsupervised crosslingual STS metric using BERT without fine-tuning achieves performance on par with supervised or weakly supervised approaches.
Comparing Top-Down and Bottom-Up Neural Generative Dependency Models
Authors: Austin Matthews, Graham Neubig, Chris Dyer
Date: 2019-11-01 00:00:00
Proceeding: CONLL 2019 11
Tasks: Language Modeling, Language Modelling

Recurrent neural network grammars generate sentences using phrase-structure syntax and perform very well on both parsing and language modeling. To explore whether generative dependency models are similarly effective, we propose two new generative models of dependency syntax. Both models use recurrent neural nets to avoid making explicit independence assumptions, but they differ in the order used to construct the trees: one builds the tree bottom-up and the other top-down, which profoundly changes the estimation problem faced by the learner. We evaluate the two models on three typologically different languages: English, Arabic, and Japanese. While both generative models improve parsing performance over a discriminative baseline, they are significantly less effective than non-syntactic LSTM language models. Surprisingly, little difference between the construction orders is observed for either parsing or language modeling.
Representation Learning and Dynamic Programming for Arc-Hybrid Parsing
Authors: Joseph Le Roux, Antoine Rozenknop, Mathieu Lacroix
Date: 2019-11-01 00:00:00
Proceeding: CONLL 2019 11
Tasks: ARC, Representation Learning

We present a new method for transition-based parsing where a solution is a pair made of a dependency tree and a derivation graph describing the construction of the former. From this representation we are able to derive an efficient parsing algorithm and design a neural network that learns vertex representations and arc scores. Experimentally, although we only train via local classifiers, our approach improves over previous arc-hybrid systems and reach state-of-the-art parsing accuracy.
Policy Preference Detection in Parliamentary Debate Motions
Authors: Gavin Abercrombie, Federico Nanni, Riza Batista-Navarro, Simone Paolo Ponzetto
Date: 2019-11-01 00:00:00
Proceeding: CONLL 2019 11
Tasks: General Classification

Debate motions (proposals) tabled in the UK Parliament contain information about the stated policy preferences of the Members of Parliament who propose them, and are key to the analysis of all subsequent speeches given in response to them. We attempt to automatically label debate motions with codes from a pre-existing coding scheme developed by political scientists for the annotation and analysis of political parties{'} manifestos. We develop annotation guidelines for the task of applying these codes to debate motions at two levels of granularity and produce a dataset of manually labelled examples. We evaluate the annotation process and the reliability and utility of the labelling scheme, finding that inter-annotator agreement is comparable with that of other studies conducted on manifesto data. Moreover, we test a variety of ways of automatically labelling motions with the codes, ranging from similarity matching to neural classification methods, and evaluate them against the gold standard labels. From these experiments, we note that established supervised baselines are not always able to improve over simple lexical heuristics. At the same time, we detect a clear and evident benefit when employing BERT, a state-of-the-art deep language representation model, even in classification scenarios with over 30 different labels and limited amounts of training data.
Improving Neural Machine Translation by Achieving Knowledge Transfer with Sentence Alignment Learning
Authors: Xuewen Shi, He-Yan Huang, Wenguan Wang, Ping Jian, Yi-Kun Tang
Date: 2019-11-01 00:00:00
Proceeding: CONLL 2019 11
Tasks: Machine Translation, NMT, Sentence, Sentence Embedding, Sentence-Embedding, Transfer Learning, Translation

Neural Machine Translation (NMT) optimized by Maximum Likelihood Estimation (MLE) lacks the guarantee of translation adequacy. To alleviate this problem, we propose an NMT approach that heightens the adequacy in machine translation by transferring the semantic knowledge learned from bilingual sentence alignment. Specifically, we first design a discriminator that learns to estimate sentence aligning score over translation candidates, and then the learned semantic knowledge is transfered to the NMT model under an adversarial learning framework. We also propose a gated self-attention based encoder for sentence embedding. Furthermore, an N-pair training loss is introduced in our framework to aid the discriminator in better capturing lexical evidence in translation candidates. Experimental results show that our proposed method outperforms baseline NMT models on Chinese-to-English and English-to-German translation tasks. Further analysis also indicates the detailed semantic knowledge transfered from the discriminator to the NMT model.
Unsupervised Neural Machine Translation with Future Rewarding
Authors: Xiangpeng Wei, Yue Hu, Luxi Xing, Li Gao
Date: 2019-11-01 00:00:00
Proceeding: CONLL 2019 11
Tasks: Decoder, Machine Translation, NMT, Reinforcement Learning, Sentence, Translation, Variational Inference

In this paper, we alleviate the local optimality of back-translation by learning a policy (takes the form of an encoder-decoder and is defined by its parameters) with future rewarding under the reinforcement learning framework, which aims to optimize the global word predictions for unsupervised neural machine translation. To this end, we design a novel reward function to characterize high-quality translations from two aspects: n-gram matching and semantic adequacy. The n-gram matching is defined as an alternative for the discrete BLEU metric, and the semantic adequacy is used to measure the adequacy of conveying the meaning of the source sentence to the target. During training, our model strives for earning higher rewards by learning to produce grammatically more accurate and semantically more adequate translations. Besides, a variational inference network (VIN) is proposed to constrain the corresponding sentences in two languages have the same or similar latent semantic code. On the widely used WMT{'}14 English-French, WMT{'}16 English-German and NIST Chinese-to-English benchmarks, our models respectively obtain 27.59/27.15, 19.65/23.42 and 22.40 BLEU points without using any labeled data, demonstrating consistent improvements over previous unsupervised NMT models.
Automatically Extracting Challenge Sets for Non-Local Phenomena in Neural Machine Translation
Authors: Leshem Choshen, Omri Abend
Date: 2019-11-01 00:00:00
Proceeding: CONLL 2019 11
Tasks: Machine Translation, Translation

We show that the state-of-the-art Transformer MT model is not biased towards monotonic reordering (unlike previous recurrent neural network models), but that nevertheless, long-distance dependencies remain a challenge for the model. Since most dependencies are short-distance, common evaluation metrics will be little influenced by how well systems perform on them. We therefore propose an automatic approach for extracting challenge sets rich with long-distance dependencies, and argue that evaluation using this methodology provides a complementary perspective on system performance. To support our claim, we compile challenge sets for English-German and German-English, which are much larger than any previously released challenge set for MT. The extracted sets are large enough to allow reliable automatic evaluation, which makes the proposed approach a scalable and practical solution for evaluating MT performance on the long-tail of syntactic phenomena.
Improving Pre-Trained Multilingual Model with Vocabulary Expansion
Authors: Hai Wang, Dian Yu, Kai Sun, Jianshu Chen, Dong Yu
Date: 2019-11-01 00:00:00
Proceeding: CONLL 2019 11
Tasks: Language Modeling, Language Modelling, Machine Reading Comprehension, Machine Translation, model, named-entity-recognition, Named Entity Recognition, Named Entity Recognition (NER), Part-Of-Speech Tagging, Reading Comprehension, Sentence

Recently, pre-trained language models have achieved remarkable success in a broad range of natural language processing tasks. However, in multilingual setting, it is extremely resource-consuming to pre-train a deep language model over large-scale corpora for each language. Instead of exhaustively pre-training monolingual language models independently, an alternative solution is to pre-train a powerful multilingual deep language model over large-scale corpora in hundreds of languages. However, the vocabulary size for each language in such a model is relatively small, especially for low-resource languages. This limitation inevitably hinders the performance of these multilingual models on tasks such as sequence labeling, wherein in-depth token-level or sentence-level understanding is essential. In this paper, inspired by previous methods designed for monolingual settings, we investigate two approaches (i.e., joint mapping and mixture mapping) based on a pre-trained multilingual model BERT for addressing the out-of-vocabulary (OOV) problem on a variety of tasks, including part-of-speech tagging, named entity recognition, machine translation quality estimation, and machine reading comprehension. Experimental results show that using mixture mapping is more promising. To the best of our knowledge, this is the first work that attempts to address and discuss the OOV issue in multilingual settings.
On the Relation between Position Information and Sentence Length in Neural Machine Translation
Authors: Masato Neishi, Naoki Yoshinaga
Date: 2019-11-01 00:00:00
Proceeding: CONLL 2019 11
Tasks: Machine Translation, NMT, Position, Relation, Sentence, Translation

Long sentences have been one of the major challenges in neural machine translation (NMT). Although some approaches such as the attention mechanism have partially remedied the problem, we found that the current standard NMT model, Transformer, has difficulty in translating long sentences compared to the former standard, Recurrent Neural Network (RNN)-based model. One of the key differences of these NMT models is how the model handles position information which is essential to process sequential data. In this study, we focus on the position information type of NMT models, and hypothesize that relative position is better than absolute position. To examine the hypothesis, we propose RNN-Transformer which replaces positional encoding layer of Transformer by RNN, and then compare RNN-based model and four variants of Transformer. Experiments on ASPEC English-to-Japanese and WMT2014 English-to-German translation tasks demonstrate that relative position helps translating sentences longer than those in the training data. Further experiments on length-controlled training data reveal that absolute position actually causes overfitting to the sentence length.
Linguistic Analysis Improves Neural Metaphor Detection
Authors: Kevin Stowe, Sarah Moeller, Laura Michaelis, Martha Palmer
Date: 2019-11-01 00:00:00
Proceeding: CONLL 2019 11
Tasks: Deep Learning

In the field of metaphor detection, deep learning systems are the ubiquitous and achieve strong performance on many tasks. However, due to the complicated procedures for manually identifying metaphors, the datasets available are relatively small and fraught with complications. We show that using syntactic features and lexical resources can automatically provide additional high-quality training data for metaphoric language, and this data can cover gaps and inconsistencies in metaphor annotation, improving state-of-the-art word-level metaphor identification. This novel application of automatically improving training data improves classification across numerous tasks, and reconfirms the necessity of high-quality data for deep learning frameworks.
Mimic and Rephrase: Reflective Listening in Open-Ended Dialogue
Authors: Justin Dieter, Tian Wang, Arun Tejasvi Chaganty, Gabor Angeli, Angel X. Chang
Date: 2019-11-01 00:00:00
Proceeding: CONLL 2019 11

Reflective listening{--}demonstrating that you have heard your conversational partner{--}is key to effective communication. Expert human communicators often mimic and rephrase their conversational partner, e.g., when responding to sentimental stories or to questions they don{'}t know the answer to. We introduce a new task and an associated dataset wherein dialogue agents similarly mimic and rephrase a user{'}s request to communicate sympathy (I{'}m sorry to hear that) or lack of knowledge (I do not know that). We study what makes a rephrasal response good against a set of qualitative metrics. We then evaluate three models for generating responses: a syntax-aware rule-based system, a seq2seq LSTM neural models with attention (S2SA), and the same neural model augmented with a copy mechanism (S2SA+C). In a human evaluation, we find that S2SA+C and the rule-based system are comparable and approach human-generated response quality. In addition, experiences with a live deployment of S2SA+C in a customer support setting suggest that this generation task is a practical contribution to real world conversational agents.