pwc-archive/papers-with-abstracts huggingface dataset:
Statistically significant detection of semantic shifts using contextual word embeddings
Authors: Yang Liu, Alan Medlar, Dorota Glowacka
Date: 2021-04-08 00:00:00
Proceeding: EMNLP (Eval4NLP) 2021 11
Tasks: Word Embeddings

Detecting lexical semantic change in smaller data sets, e.g. in historical linguistics and digital humanities, is challenging due to a lack of statistical power. This issue is exacerbated by non-contextual embedding models that produce one embedding per word and, therefore, mask the variability present in the data. In this article, we propose an approach to estimate semantic shift by combining contextual word embeddings with permutation-based statistical tests. We use the false discovery rate procedure to address the large number of hypothesis tests being conducted simultaneously. We demonstrate the performance of this approach in simulation where it achieves consistently high precision by suppressing false positives. We additionally analyze real-world data from SemEval-2020 Task 1 and the Liverpool FC subreddit corpus. We show that by taking sample variation into account, we can improve the robustness of individual semantic shift estimates without degrading overall performance.
Exploring Machine Speech Chain for Domain Adaptation and Few-Shot Speaker Adaptation
Authors: Fengpeng Yue, Yan Deng, Lei He, Tom Ko
Date: 2021-04-08 00:00:00
Tasks: Automatic Speech Recognition, Automatic Speech Recognition (ASR), Data Augmentation, Domain Adaptation, speech-recognition, Speech Recognition, text-to-speech, Text to Speech

Machine Speech Chain, which integrates both end-to-end (E2E) automatic speech recognition (ASR) and text-to-speech (TTS) into one circle for joint training, has been proven to be effective in data augmentation by leveraging large amounts of unpaired data. In this paper, we explore the TTS->ASR pipeline in speech chain to do domain adaptation for both neural TTS and E2E ASR models, with only text data from target domain. We conduct experiments by adapting from audiobook domain (LibriSpeech) to presentation domain (TED-LIUM), there is a relative word error rate (WER) reduction of 10% for the E2E ASR model on the TED-LIUM test set, and a relative WER reduction of 51.5% in synthetic speech generated by neural TTS in the presentation domain. Further, we apply few-shot speaker adaptation for the E2E ASR by using a few utterances from target speakers in an unsupervised way, results in additional gains.
Exploring the Relationship Between Algorithm Performance, Vocabulary, and Run-Time in Text Classification
Authors: Wilson Fearn, Orion Weller, Kevin Seppi
Date: 2021-04-08 00:00:00
Proceeding: NAACL 2021 4
Tasks: Classification, Document Classification, General Classification, Sentiment Analysis, text-classification, Text Classification

Text classification is a significant branch of natural language processing, and has many applications including document classification and sentiment analysis. Unsurprisingly, those who do text classification are concerned with the run-time of their algorithms, many of which depend on the size of the corpus' vocabulary due to their bag-of-words representation. Although many studies have examined the effect of preprocessing techniques on vocabulary size and accuracy, none have examined how these methods affect a model's run-time. To fill this gap, we provide a comprehensive study that examines how preprocessing techniques affect the vocabulary size, model performance, and model run-time, evaluating ten techniques over four models and two datasets. We show that some individual methods can reduce run-time with no loss of accuracy, while some combinations of methods can trade 2-5% of the accuracy for up to a 65% reduction of run-time. Furthermore, some combinations of preprocessing techniques can even provide a 15% reduction in run-time while simultaneously improving model accuracy.
Probing BERT in Hyperbolic Spaces
Authors: Boli Chen, Yao Fu, Guangwei Xu, Pengjun Xie, Chuanqi Tan, Mosha Chen, Liping Jing
Date: 2021-04-08 00:00:00
Proceeding: ICLR 2021 1
Tasks: Word Embeddings

Recently, a variety of probing tasks are proposed to discover linguistic properties learned in contextualized word embeddings. Many of these works implicitly assume these embeddings lay in certain metric spaces, typically the Euclidean space. This work considers a family of geometrically special spaces, the hyperbolic spaces, that exhibit better inductive biases for hierarchical structures and may better reveal linguistic hierarchies encoded in contextualized representations. We introduce a Poincare probe, a structural probe projecting these embeddings into a Poincare subspace with explicitly defined hierarchies. We focus on two probing objectives: (a) dependency trees where the hierarchy is defined as head-dependent structures; (b) lexical sentiments where the hierarchy is defined as the polarity of words (positivity and negativity). We argue that a key desideratum of a probe is its sensitivity to the existence of linguistic structures. We apply our probes on BERT, a typical contextualized embedding model. In a syntactic subspace, our probe better recovers tree structures than Euclidean probes, revealing the possibility that the geometry of BERT syntax may not necessarily be Euclidean. In a sentiment subspace, we reveal two possible meta-embeddings for positive and negative sentiments and show how lexically-controlled contextualization would change the geometric localization of embeddings. We demonstrate the findings with our Poincare probe via extensive experiments and visualization. Our results can be reproduced at https://github.com/FranxYao/PoincareProbe.
COVID-19 Named Entity Recognition for Vietnamese
Authors: Thinh Hung Truong, Mai Hoang Dao, Dat Quoc Nguyen
Date: 2021-04-08 00:00:00
Proceeding: NAACL 2021 4
Tasks: named-entity-recognition, Named Entity Recognition, Named Entity Recognition In Vietnamese, Named Entity Recognition (NER), NER, Vietnamese Word Segmentation, XLM-R

The current COVID-19 pandemic has lead to the creation of many corpora that facilitate NLP research and downstream applications to help fight the pandemic. However, most of these corpora are exclusively for English. As the pandemic is a global problem, it is worth creating COVID-19 related datasets for languages other than English. In this paper, we present the first manually-annotated COVID-19 domain-specific dataset for Vietnamese. Particularly, our dataset is annotated for the named entity recognition (NER) task with newly-defined entity types that can be used in other future epidemics. Our dataset also contains the largest number of entities compared to existing Vietnamese NER datasets. We empirically conduct experiments using strong baselines on our dataset, and find that: automatic Vietnamese word segmentation helps improve the NER results and the highest performances are obtained by fine-tuning pre-trained language models where the monolingual model PhoBERT for Vietnamese (Nguyen and Nguyen, 2020) produces higher results than the multilingual model XLM-R (Conneau et al., 2020). We publicly release our dataset at: https://github.com/VinAIResearch/PhoNER_COVID19
How Metaphors Impact Political Discourse: A Large-Scale Topic-Agnostic Study Using Neural Metaphor Detection
Authors: Vinodkumar Prabhakaran, Marek Rei, Ekaterina Shutova
Date: 2021-04-08 00:00:00
Tasks: Persuasiveness

Metaphors are widely used in political rhetoric as an effective framing device. While the efficacy of specific metaphors such as the war metaphor in political discourse has been documented before, those studies often rely on small number of hand-coded instances of metaphor use. Larger-scale topic-agnostic studies are required to establish the general persuasiveness of metaphors as a device, and to shed light on the broader patterns that guide their persuasiveness. In this paper, we present a large-scale data-driven study of metaphors used in political discourse. We conduct this study on a publicly available dataset of over 85K posts made by 412 US politicians in their Facebook public pages, up until Feb 2017. Our contributions are threefold: we show evidence that metaphor use correlates with ideological leanings in complex ways that depend on concurrent political events such as winning or losing elections; we show that posts with metaphors elicit more engagement from their audience overall even after controlling for various socio-political factors such as gender and political party affiliation; and finally, we demonstrate that metaphoricity is indeed the reason for increased engagement of posts, through a fine-grained linguistic analysis of metaphorical vs. literal usages of 513 words across 70K posts.
Machine Learning Based on Natural Language Processing to Detect Cardiac Failure in Clinical Narratives
Authors: Thanh-Dung Le, Rita Noumeir, Jerome Rambaud, Guillaume Sans, Philippe Jouvet
Date: 2021-04-08 00:00:00
Tasks: BIG-bench Machine Learning, Binary Classification, feature selection, Representation Learning, Word Embeddings

The purpose of the study presented herein is to develop a machine learning algorithm based on natural language processing that automatically detects whether a patient has a cardiac failure or a healthy condition by using physician notes in Research Data Warehouse at CHU Sainte Justine Hospital. First, a word representation learning technique was employed by using bag-of-word (BoW), term frequency inverse document frequency (TFIDF), and neural word embeddings (word2vec). Each representation technique aims to retain the words semantic and syntactic analysis in critical care data. It helps to enrich the mutual information for the word representation and leads to an advantage for further appropriate analysis steps. Second, a machine learning classifier was used to detect the patients condition for either cardiac failure or stable patient through the created word representation vector space from the previous step. This machine learning approach is based on a supervised binary classification algorithm, including logistic regression (LR), Gaussian Naive-Bayes (GaussianNB), and multilayer perceptron neural network (MLPNN). Technically, it mainly optimizes the empirical loss during training the classifiers. As a result, an automatic learning algorithm would be accomplished to draw a high classification performance, including accuracy (acc), precision (pre), recall (rec), and F1 score (f1). The results show that the combination of TFIDF and MLPNN always outperformed other combinations with all overall performance. In the case without any feature selection, the proposed framework yielded an overall classification performance with acc, pre, rec, and f1 of 84% and 82%, 85%, and 83%, respectively. Significantly, if the feature selection was well applied, the overall performance would finally improve up to 4% for each evaluation.
On Biasing Transformer Attention Towards Monotonicity
Authors: Annette Rios, Chantal Amrhein, Noëmi Aepli, Rico Sennrich
Date: 2021-04-08 00:00:00
Proceeding: NAACL 2021 4
Tasks: Grapheme-to-Phoneme Conversion, Morphological Inflection, Transliteration

Many sequence-to-sequence tasks in natural language processing are roughly monotonic in the alignment between source and target sequence, and previous work has facilitated or enforced learning of monotonic attention behavior via specialized attention functions or pretraining. In this work, we introduce a monotonicity loss function that is compatible with standard attention mechanisms and test it on several sequence-to-sequence tasks: grapheme-to-phoneme conversion, morphological inflection, transliteration, and dialect normalization. Experiments show that we can achieve largely monotonic behavior. Performance is mixed, with larger gains on top of RNN baselines. General monotonicity does not benefit transformer multihead attention, however, we see isolated improvements when only a subset of heads is biased towards monotonic behavior.
Towards Multi-Scale Style Control for Expressive Speech Synthesis
Authors: Xiang Li, Changhe Song, Jingbei Li, Zhiyong Wu, Jia Jia, Helen Meng
Date: 2021-04-08 00:00:00
Tasks: Expressive Speech Synthesis, Speech Synthesis, Style Transfer

This paper introduces a multi-scale speech style modeling method for end-to-end expressive speech synthesis. The proposed method employs a multi-scale reference encoder to extract both the global-scale utterance-level and the local-scale quasi-phoneme-level style features of the target speech, which are then fed into the speech synthesis model as an extension to the input phoneme sequence. During training time, the multi-scale style model could be jointly trained with the speech synthesis model in an end-to-end fashion. By applying the proposed method to style transfer task, experimental results indicate that the controllability of the multi-scale speech style model and the expressiveness of the synthesized speech are greatly improved. Moreover, by assigning different reference speeches to extraction of style on each scale, the flexibility of the proposed method is further revealed.
WNARS: WFST based Non-autoregressive Streaming End-to-End Speech Recognition
Authors: Zhichao Wang, Wenwen Yang, Pan Zhou, Wei Chen
Date: 2021-04-08 00:00:00
Tasks: Automatic Speech Recognition, Automatic Speech Recognition (ASR), Decoder, speech-recognition, Speech Recognition

Recently, attention-based encoder-decoder (AED) end-to-end (E2E) models have drawn more and more attention in the field of automatic speech recognition (ASR). AED models, however, still have drawbacks when deploying in commercial applications. Autoregressive beam search decoding makes it inefficient for high-concurrency applications. It is also inconvenient to integrate external word-level language models. The most important thing is that AED models are difficult for streaming recognition due to global attention mechanism. In this paper, we propose a novel framework, namely WNARS, using hybrid CTC-attention AED models and weighted finite-state transducers (WFST) to solve these problems together. We switch from autoregressive beam search to CTC branch decoding, which performs first-pass decoding with WFST in chunk-wise streaming way. The decoder branch then performs second-pass rescoring on the generated hypotheses non-autoregressively. On the AISHELL-1 task, our WNARS achieves a character error rate of 5.22% with 640ms latency, to the best of our knowledge, which is the state-of-the-art performance for online ASR. Further experiments on our 10,000-hour Mandarin task show the proposed method achieves more than 20% improvements with 50% latency compared to a strong TDNN-BLSTM lattice-free MMI baseline.
A Proposal for an Interactive Shell Based on a Typed Lambda Calculus
Authors: Kouji Matsui
Date: 2021-04-08 00:00:00
Tasks: Vocal Bursts Type Prediction

This paper presents Favalon, a functional programming language built on the premise of a lambda calculus for use as an interactive shell replacement. Favalon seamlessly integrates with typed versions of existing libraries and commands using type inference, flexible runtime type metadata, and the same techniques employed by shells to link commands together. Much of Favalon's syntax is customizable via user-defined functions, allowing it to be extended by anyone who is familiar with a command-line shell. Furthermore, Favalon's type inference engine can be separated from its runtime library and easily repurposed for other applications.
Enabling Cross-Domain Communication: How to Bridge the Gap between AI and HW Engineers
Authors: Michael J. Klaiber, Axel J. Acosta, Ingo Feldner, Falk Rehm
Date: 2021-04-08 00:00:00

A key issue in system design is the lack of communication between hardware, software and domain expert. Recent research work shows progress in automatic HW/SW co-design flows of neural accelerators that seems to make this kind of communication obsolete. Most real-world systems, however, are a composition of multiple processing units, communication networks and memories. A HW/SW co-design process of (reconfigurable) neural accelerators, therefore, is an important sub-problem towards a common co-design methodology. The ultimate challenge is to define the constraints for the design space exploration on system level - a task which requires deep knowledge and understanding of hardware architectures, mapping of workloads onto hardware and the application domain, e.g. artificial intelligence. For most projects, these skills are distributed among several people or even different teams which is one of the major reasons why there is no established end-to-end development methodology for digital systems. This position paper discusses possibilities how to establish such a methodology for systems that include (reconfigurable) dedicated accelerators and outlines the central role that languages and tools play in the process.
Heuristic Strategies for Solving Complex Interacting Large-Scale Stockpile Blending Problems
Authors: Yue Xie, Aneta Neumann, Frank Neumann
Date: 2021-04-08 00:00:00
Tasks: Scheduling

The Stockpile blending problem is an important component of mine production scheduling, where stockpiles are used to store and blend raw material. The goal of blending material from stockpiles is to create parcels of concentrate which contain optimal metal grades based on the material available. The volume of material that each stockpile provides to a given parcel is dependent on a set of mine schedule conditions and customer demands. Therefore, the problem can be formulated as a continuous optimization problem. In the real-world application, there are several constraints required to guarantee parcels that meet the demand of downstream customers. It is a challenge in solving the stockpile blending problems since its scale can be very large. We introduce two repaired operators for the problems to convert the infeasible solutions into the solutions without violating the two tight constraints. Besides, we introduce a multi-component fitness function for solving the large-scale stockpile blending problem which can maximize the volume of metal over the plan and maintain the balance between stockpiles according to the usage of metal. Furthermore, we investigate the well-known approach in this paper, which is used to solve optimization problems over continuous space, namely the differential evolution (DE) algorithm. The experimental results show that the DE algorithm combined with two proposed duration repair methods is significantly better in terms of the values of results than the results on real-world instances for both one-month problems and large-scale problems.
Weakly and Partially Supervised Learning Frameworks for Anomaly Detection
Authors: Bruno Degardin
Date: 2020-07-23 00:00:00
Tasks: Abnormal Event Detection In Video, Anomaly Detection, Anomaly Detection In Surveillance Videos, Multiple Instance Learning, Self-Supervised Learning

The main objective is to provide several solutions to the mentioned problems, by focusing on analyzing previous state-of-the-art methods and presenting an extensive overview to clarify the concepts employed on capturing normal and abnormal patterns. Also, by exploring different strategies, we were able to develop new approaches that consistently advance the state-of-the-art performance. Moreover, we announce the availability of a new large-scale first of its kind dataset fully annotated at the frame level, concerning a specific anomaly detection event with a wide diversity in fighting scenarios, that can be freely used by the research community. Along with this document with the purpose of requiring minimal supervision, two different proposals are described; the first method employs the recent technique of self-supervised learning to avoid the laborious task of annotation, where the training set is autonomously labeled using an iterative learning framework composed of two independent experts that feed data to each other through a Bayesian framework. The second proposal explores a new method to learn an anomaly ranking model in the multiple instance learning paradigm by leveraging weakly labeled videos, where the training labels are done at the video-level. The experiments were conducted in several well-known datasets, and our solutions solidly outperform the state-of-the-art. Additionally, as a proof-of-concept system, we also present the results of collected real-world simulations in different environments to perform a field test of our learned models.
Generative Neural Networks for Anomaly Detection in Crowded Scenes
Authors: Tian Wang, Meina Qiao, Zhiwei Lin, Ce Li, Hichem Snoussi, Zhe Liu, Chang Choi
Date: 2018-10-29 00:00:00
Tasks: Abnormal Event Detection In Video, Anomaly Detection, Semi-supervised Anomaly Detection

Security surveillance is critical to social harmony and people's peaceful life. It has a great impact on strengthening social stability and life safeguarding. Detecting anomaly timely, effectively and efficiently in video surveillance remains challenging. This paper proposes a new approach, called S 2 -VAE, for anomaly detection from video data. The S 2 -VAE consists of two proposed neural networks: a Stacked Fully Connected Variational AutoEncoder (S F -VAE) and a Skip Convolutional VAE (S C -VAE). The S F -VAE is a shallow generative network to obtain a model like Gaussian mixture to fit the distribution of the actual data. The S C -VAE, as a key component of S 2 -VAE, is a deep generative network to take advantages of CNN, VAE and skip connections. Both S F -VAE and S C -VAE are efficient and effective generative networks and they can achieve better performance for detecting both local abnormal events and global abnormal events. The proposed S 2 -VAE is evaluated using four public datasets. The experimental results show that the S 2 -VAE outperforms the state-of-the-art algorithms. The code is available publicly at https://github.com/tianwangbuaa/.
Optimization by Simulated Annealing
Authors: S. Kirkpatrick, C. D. Gelatt, M. P. Vecchi
Date: 1983-05-13 00:00:00
Tasks: Combinatorial Optimization

There is a deep and useful connection between statistical mechanics (the behavior of systems with many degrees of freedom in thermal equilibrium at a finite temperature) and multivariate or combinatorial optimization (finding the minimum of a given function depending on many parameters). A detailed analogy with annealing in solids provides a framework for optimization of the properties of very large and complex systems. This connection to statistical mechanics exposes new information and provides an unfamiliar perspective on traditional optimization problems and methods.
Chaos in the Classical Analogue of the Hofstadter Problem
Authors: M. del Mar Espinosa, Miguel A. Martin-Delgado, Alfonso Niella, David Paramo, Javier Rodriguez-Laguna
Date: 1998-08-11 00:00:00

The behaviour of an electron in a potential that resembles that of a bidimensional solid with a perpendicular magnetic field applied is studied from a classical point of view. This problem presents the standard features of chaos and some interesting new patterns. A new chaos indicator, called random walk indicator, is presented to describe som of these new patterns