pwc-archive/papers-with-abstracts huggingface dataset:
Federated Learning for Spoken Language Understanding
Authors: Zhiqi Huang, Fenglin Liu, Yuexian Zou
Date: 2020-12-01 00:00:00
Proceeding: COLING 2020 8
Tasks: Federated Learning, Intent Detection, slot-filling, Slot Filling, Spoken Language Understanding, Transfer Learning, Vertical Federated Learning

Recently, spoken language understanding (SLU) has attracted extensive research interests, and various SLU datasets have been proposed to promote the development. However, most of the existing methods focus on a single individual dataset, the efforts to improve the robustness of models and obtain better performance by combining the merits of various datasets are not well studied. In this paper, we argue that if these SLU datasets are considered together, different knowledge from different datasets could be learned jointly, and there are high chances to promote the performance of each dataset. At the same time, we further attempt to prevent data leakage when unifying multiple datasets which, arguably, is more useful in an industry setting. To this end, we propose a federated learning framework, which could unify various types of datasets as well as tasks to learn and fuse various types of knowledge, i.e., text representations, from different datasets and tasks, without the sharing of downstream task data. The fused text representations merge useful features from different SLU datasets and tasks and are thus much more powerful than the original text representations alone in individual tasks. At last, in order to provide multi-granularity text representations for our framework, we propose a novel Multi-view Encoder (MV-Encoder) as the backbone of our federated learning framework. Experiments on two SLU benchmark datasets, including two tasks (intention detection and slot filling) and federated learning settings (horizontal federated learning, vertical federated learning and federated transfer learning), demonstrate the effectiveness and universality of our approach. Specifically, we are able to get 1.53{\%} improvement on the intent detection metric accuracy. And we could also boost the performance of a strong baseline by up to 5.29{\%} on the slot filling metric F1. Furthermore, by leveraging BERT as an additional encoder, we establish new state-of-the-art results on SNIPS and ATIS datasets, where we get 99.33{\%} and 98.28{\%} in terms of accuracy on intent detection task as well as 97.20{\%} and 96.41{\%} in terms of F1 score on slot filling task, respectively.
Attentively Embracing Noise for Robust Latent Representation in BERT
Authors: Gwenaelle Cunha Sergio, Dennis Singh Moirangthem, Minho Lee
Date: 2020-12-01 00:00:00
Proceeding: COLING 2020 8
Tasks: Automatic Speech Recognition, Automatic Speech Recognition (ASR), Chatbot, Classification, intent-classification, Intent Classification, speech-recognition, Speech Recognition, Speech-to-Text, text-classification, Text Classification

Modern digital personal assistants interact with users through voice. Therefore, they heavily rely on automatic speech recognition (ASR) in order to convert speech to text and perform further tasks. We introduce EBERT, which stands for EmbraceBERT, with the goal of extracting more robust latent representations for the task of noisy ASR text classification. Conventionally, BERT is fine-tuned for downstream classification tasks using only the [CLS] starter token, with the remaining tokens being discarded. We propose using all encoded transformer tokens and further encode them using a novel attentive embracement layer and multi-head attention layer. This approach uses the otherwise discarded tokens as a source of additional information and the multi-head attention in conjunction with the attentive embracement layer to select important features from clean data during training. This allows for the extraction of a robust latent vector resulting in improved classification performance during testing when presented with noisy inputs. We show the impact of our model on both the Chatbot and Snips corpora for intent classification with ASR error. Results, in terms of F1-score and mean between 10 runs, show that our model significantly outperforms the baseline model.
A Comprehensive Evaluation of Incremental Speech Recognition and Diarization for Conversational AI
Authors: Angus Addlesee, Yanchao Yu, Arash Eshghi
Date: 2020-12-01 00:00:00
Proceeding: COLING 2020 8
Tasks: Automatic Speech Recognition, Automatic Speech Recognition (ASR), speaker-diarization, Speaker Diarization, speech-recognition, Speech Recognition

Automatic Speech Recognition (ASR) systems are increasingly powerful and more accurate, but also more numerous with several options existing currently as a service (e.g. Google, IBM, and Microsoft). Currently the most stringent standards for such systems are set within the context of their use in, and for, Conversational AI technology. These systems are expected to operate incrementally in real-time, be responsive, stable, and robust to the pervasive yet peculiar characteristics of conversational speech such as disfluencies and overlaps. In this paper we evaluate the most popular of such systems with metrics and experiments designed with these standards in mind. We also evaluate the speaker diarization (SD) capabilities of the same systems which will be particularly important for dialogue systems designed to handle multi-party interaction. We found that Microsoft has the leading incremental ASR system which preserves disfluent materials and IBM has the leading incremental SD system in addition to the ASR that is most robust to speech overlaps. Google strikes a balance between the two but none of these systems are yet suitable to reliably handle natural spontaneous conversations in real-time.
Decolonising Speech and Language Technology
Authors: Steven Bird
Date: 2020-12-01 00:00:00
Proceeding: COLING 2020 8

After generations of exploitation, Indigenous people often respond negatively to the idea that their languages are data ready for the taking. By treating Indigenous knowledge as a commodity, speech and language technologists risk disenfranchising local knowledge authorities, reenacting the causes of language endangerment. Scholars in related fields have responded to calls for decolonisation, and we in the speech and language technology community need to follow suit, and explore what this means for our practices that involve Indigenous languages and the communities who own them. This paper reviews colonising discourses in speech and language technology, and suggests new ways of working with Indigenous communities, and seeks to open a discussion of a postcolonial approach to computational methods for supporting language vitality.
Multitask Learning-Based Neural Bridging Reference Resolution
Authors: Juntao Yu, Massimo Poesio
Date: 2020-12-01 00:00:00
Proceeding: COLING 2020 8
Tasks: coreference-resolution, Coreference Resolution, Multi-Task Learning

We propose a multi task learning-based neural model for resolving bridging references tackling two key challenges. The first challenge is the lack of large corpora annotated with bridging references. To address this, we use multi-task learning to help bridging reference resolution with coreference resolution. We show that substantial improvements of up to 8 p.p. can be achieved on full bridging resolution with this architecture. The second challenge is the different definitions of bridging used in different corpora, meaning that hand-coded systems or systems using special features designed for one corpus do not work well with other corpora. Our neural model only uses a small number of corpus independent features, thus can be applied to different corpora. Evaluations with very different bridging corpora (ARRAU, ISNOTES, BASHI and SCICORP) suggest that our architecture works equally well on all corpora, and achieves the SoTA results on full bridging resolution for all corpora, outperforming the best reported results by up to 36.3 p.p..
Automatic Discovery of Heterogeneous Machine Learning Pipelines: An Application to Natural Language Processing
Authors: Suilan Estevez-Velarde, Yoan Guti{\'e}rrez, Andres Montoyo, Yudivi{\'a}n Almeida Cruz
Date: 2020-12-01 00:00:00
Proceeding: COLING 2020 8
Tasks: AutoML, BIG-bench Machine Learning

This paper presents AutoGOAL, a system for automatic machine learning (AutoML) that uses heterogeneous techniques. In contrast with existing AutoML approaches, our contribution can automatically build machine learning pipelines that combine techniques and algorithms from different frameworks, including shallow classifiers, natural language processing tools, and neural networks. We define the heterogeneous AutoML optimization problem as the search for the best sequence of algorithms that transforms specific input data into the desired output. This provides a novel theoretical and practical approach to AutoML. Our proposal is experimentally evaluated in diverse machine learning problems and compared with alternative approaches, showing that it is competitive with other AutoML alternatives in standard benchmarks. Furthermore, it can be applied to novel scenarios, such as several NLP tasks, where existing alternatives cannot be directly deployed. The system is freely available and includes in-built compatibility with a large number of popular machine learning frameworks, which makes our approach useful for solving practical problems with relative ease and effort.
Distill and Replay for Continual Language Learning
Authors: Jingyuan Sun, Shaonan Wang, Jiajun Zhang, Chengqing Zong
Date: 2020-12-01 00:00:00
Proceeding: COLING 2020 8
Tasks: Language Modelling, Natural Language Understanding

Accumulating knowledge to tackle new tasks without necessarily forgetting the old ones is a hallmark of human-like intelligence. But the current dominant paradigm of machine learning is still to train a model that works well on static datasets. When learning tasks in a stream where data distribution may fluctuate, fitting on new tasks often leads to forgetting on the previous ones. We propose a simple yet effective framework that continually learns natural language understanding tasks with one model. Our framework distills knowledge and replays experience from previous tasks when fitting on a new task, thus named DnR (distill and replay). The framework is based on language models and can be smoothly built with different language model architectures. Experimental results demonstrate that DnR outperfoms previous state-of-the-art models in continually learning tasks of the same type but from different domains, as well as tasks of different types. With the distillation method, we further show that it{'}s possible for DnR to incrementally compress the model size while still outperforming most of the baselines. We hope that DnR could promote the empirical application of continual language learning, and contribute to building human-level language intelligence minimally bothered by catastrophic forgetting.
Incorporating Noisy Length Constraints into Transformer with Length-aware Positional Encodings
Authors: Yui Oka, Katsuki Chousa, Katsuhito Sudoh, Satoshi Nakamura
Date: 2020-12-01 00:00:00
Proceeding: COLING 2020 8
Tasks: Machine Translation, Prediction, Sentence, Translation

Neural Machine Translation often suffers from an under-translation problem due to its limited modeling of output sequence lengths. In this work, we propose a novel approach to training a Transformer model using length constraints based on length-aware positional encoding (PE). Since length constraints with exact target sentence lengths degrade translation performance, we add random noise within a certain window size to the length constraints in the PE during the training. In the inference step, we predict the output lengths using input sequences and a BERT-based length prediction model. Experimental results in an ASPEC English-to-Japanese translation showed the proposed method produced translations with lengths close to the reference ones and outperformed a vanilla Transformer (especially in short sentences) by 3.22 points in BLEU. The average translation results using our length prediction model were also better than another baseline method using input lengths for the length constraints. The proposed noise injection improved robustness for length prediction errors, especially within the window size.
Rethinking Skip Connection with Layer Normalization
Authors: Fenglin Liu, Xuancheng Ren, Zhiyuan Zhang, Xu sun, Yuexian Zou
Date: 2020-12-01 00:00:00
Proceeding: COLING 2020 8
Tasks: image-classification, Image Classification, Machine Translation, Translation

Skip connection is a widely-used technique to improve the performance and the convergence of deep neural networks, which is believed to relieve the difficulty in optimization due to non-linearity by propagating a linear component through the neural network layers. However, from another point of view, it can also be seen as a modulating mechanism between the input and the output, with the input scaled by a pre-defined value one. In this work, we investigate how the scale factors in the effectiveness of the skip connection and reveal that a trivial adjustment of the scale will lead to spurious gradient exploding or vanishing in line with the deepness of the models, which could by addressed by normalization, in particular, layer normalization, which induces consistent improvements over the plain skip connection. Inspired by the findings, we further propose to adaptively adjust the scale of the input by recursively applying skip connection with layer normalization, which promotes the performance substantially and generalizes well across diverse tasks including both machine translation and image classification datasets.
Specializing Word Vectors by Spectral Decomposition on Heterogeneously Twisted Graphs
Authors: Yuanhang Ren, Ye Du
Date: 2020-12-01 00:00:00
Proceeding: COLING 2020 8
Tasks: Semantic Similarity, Semantic Textual Similarity

Traditional word vectors, such as word2vec and glove, have a well-known inclination to conflate the semantic similarity with other semantic relations. A retrofitting procedure may be needed to solve this issue. In this work, we propose a new retrofitting method called Heterogeneously Retrofitted Spectral Word Embedding. It heterogeneously twists the similarity matrix of word pairs with lexical constraints. A new set of word vectors is generated by a spectral decomposition of the similarity matrix, which has a linear algebraic analytic form. Our method has a competitive performance compared with the state-of-the-art retrofitting method such as AR (CITATION). In addition, since our embedding has a clear linear algebraic relationship with the similarity matrix, we carefully study the contribution of each component in our model. Last but not least, our method is very efficient to execute.
Deep Inside-outside Recursive Autoencoder with All-span Objective
Authors: Ruyue Hong, Jiong Cai, Kewei Tu
Date: 2020-12-01 00:00:00
Proceeding: COLING 2020 8
Tasks: All, Constituency Parsing, Sentence

Deep inside-outside recursive autoencoder (DIORA) is a neural-based model designed for unsupervised constituency parsing. During its forward computation, it provides phrase and contextual representations for all spans in the input sentence. By utilizing the contextual representation of each leaf-level span, the span of length 1, to reconstruct the word inside the span, the model is trained without labeled data. In this work, we extend the training objective of DIORA by making use of all spans instead of only leaf-level spans. We test our new training objective on datasets of two languages: English and Japanese, and empirically show that our method achieves improvement in parsing accuracy over the original DIORA.
Increasing Learning Efficiency of Self-Attention Networks through Direct Position Interactions, Learnable Temperature, and Convoluted Attention
Authors: Philipp Dufter, Martin Schmitt, Hinrich Sch{\"u}tze
Date: 2020-12-01 00:00:00
Proceeding: COLING 2020 8
Tasks: Language Modeling, Language Modelling, Part-Of-Speech Tagging, Position

Self-Attention Networks (SANs) are an integral part of successful neural architectures such as Transformer (Vaswani et al., 2017), and thus of pretrained language models such as BERT (Devlin et al., 2019) or GPT-3 (Brown et al., 2020). Training SANs on a task or pretraining them on language modeling requires large amounts of data and compute resources. We are searching for modifications to SANs that enable faster learning, i.e., higher accuracies after fewer update steps. We investigate three modifications to SANs: direct position interactions, learnable temperature, and convoluted attention. When evaluating them on part-of-speech tagging, we find that direct position interactions are an alternative to position embeddings, and convoluted attention has the potential to speed up the learning process.
Invertible Tree Embeddings using a Cryptographic Role Embedding Scheme
Authors: Coleman Haley, Paul Smolensky
Date: 2020-12-01 00:00:00
Proceeding: COLING 2020 8
Tasks: Position, Retrieval, Role Embedding

We present a novel method for embedding trees in a vector space based on Tensor-Product Representations (TPRs) which allows for inversion: the retrieval of the original tree structure and nodes from the vectorial embedding. Unlike previous attempts, this does not come at the cost of intractable representation size; we utilize a method for non-exact inversion, showing that it works well when there is sufficient randomness in the representation scheme for simple data and providing an upper bound on its error. To handle the huge number of possible tree positions without memoizing position representation vectors, we present a method (Cryptographic Role Embedding) using cryptographic hashing algorithms that allows for the representation of unboundedly many positions. Through experiments on parse tree data, we show a 30,000-dimensional Cryptographic Role Embedding of trees can provide invertibility with error {\textless} 1{\%} that previous methods would require 8.6 {\mbox{$\times$}} 1057 dimensions to represent.
Bridging Resolution: A Survey of the State of the Art
Authors: Hideo Kobayashi, Vincent Ng
Date: 2020-12-01 00:00:00
Proceeding: COLING 2020 8
Tasks: coreference-resolution, Coreference Resolution, Survey

Bridging reference resolution is an anaphora resolution task that is arguably more challenging and less studied than entity coreference resolution. Given that significant progress has been made on coreference resolution in recent years, we believe that bridging resolution will receive increasing attention in the NLP community. Nevertheless, progress on bridging resolution is currently hampered in part by the scarcity of large annotated corpora for model training as well as the lack of standardized evaluation protocols. This paper presents a survey of the current state of research on bridging reference resolution and discusses future research directions.
An analysis of language models for metaphor recognition
Authors: Arthur Neidlein, Philip Wiesenbach, Katja Markert
Date: 2020-12-01 00:00:00
Proceeding: COLING 2020 8
Tasks: Word Sense Disambiguation

We conduct a linguistic analysis of recent metaphor recognition systems, all of which are based on language models. We show that their performance, although reaching high F-scores, has considerable gaps from a linguistic perspective. First, they perform substantially worse on unconventional metaphors than on conventional ones. Second, they struggle with handling rarer word types. These two findings together suggest that a large part of the systems{'} success is due to optimising the disambiguation of conventionalised, metaphoric word senses for specific words instead of modelling general properties of metaphors. As a positive result, the systems show increasing capabilities to recognise metaphoric readings of unseen words if synonyms or morphological variations of these words have been seen before, leading to enhanced generalisation beyond word sense disambiguation.
What Meaning-Form Correlation Has to Compose With: A Study of MFC on Artificial and Natural Language
Authors: Timothee Mickus, Timoth{\'e}e Bernard, Denis Paperno
Date: 2020-12-01 00:00:00
Proceeding: COLING 2020 8
Tasks: Form

Compositionality is a widely discussed property of natural languages, although its exact definition has been elusive. We focus on the proposal that compositionality can be assessed by measuring meaning-form correlation. We analyze meaning-form correlation on three sets of languages: (i) artificial toy languages tailored to be compositional, (ii) a set of English dictionary definitions, and (iii) a set of English sentences drawn from literature. We find that linguistic phenomena such as synonymy and ungrounded stop-words weigh on MFC measurements, and that straightforward methods to mitigate their effects have widely varying results depending on the dataset they are applied to. Data and code are made publicly available.
Evaluating Pretrained Transformer-based Models on the Task of Fine-Grained Named Entity Recognition
Authors: Cedric Lothritz, Kevin Allix, Lisa Veiber, Tegawend{\'e} F. Bissyand{\'e}, Jacques Klein
Date: 2020-12-01 00:00:00
Proceeding: COLING 2020 8
Tasks: named-entity-recognition, Named Entity Recognition, Named Entity Recognition (NER), NER

Named Entity Recognition (NER) is a fundamental Natural Language Processing (NLP) task and has remained an active research field. In recent years, transformer models and more specifically the BERT model developed at Google revolutionised the field of NLP. While the performance of transformer-based approaches such as BERT has been studied for NER, there has not yet been a study for the fine-grained Named Entity Recognition (FG-NER) task. In this paper, we compare three transformer-based models (BERT, RoBERTa, and XLNet) to two non-transformer-based models (CRF and BiLSTM-CNN-CRF). Furthermore, we apply each model to a multitude of distinct domains. We find that transformer-based models incrementally outperform the studied non-transformer-based models in most domains with respect to the F1 score. Furthermore, we find that the choice of domains significantly influenced the performance regardless of the respective data size or the model chosen.
Unleashing the Power of Neural Discourse Parsers - A Context and Structure Aware Approach Using Large Scale Pretraining
Authors: Grigorii Guz, Patrick Huber, Giuseppe Carenini
Date: 2020-12-01 00:00:00
Proceeding: COLING 2020 8
Tasks: Discourse Parsing, Machine Translation, Opinion Mining, Translation

RST-based discourse parsing is an important NLP task with numerous downstream applications, such as summarization, machine translation and opinion mining. In this paper, we demonstrate a simple, yet highly accurate discourse parser, incorporating recent contextual language models. Our parser establishes the new state-of-the-art (SOTA) performance for predicting structure and nuclearity on two key RST datasets, RST-DT and Instr-DT. We further demonstrate that pretraining our parser on the recently available large-scale {``}silver-standard{''} discourse treebank MEGA-DT provides even larger performance benefits, suggesting a novel and promising research direction in the field of discourse analysis.