pwc-archive/papers-with-abstracts huggingface dataset:
Character-level HyperNetworks for Hate Speech Detection
Authors: Tomer Wullach, Amir Adler, Einat Minkov
Date: 2021-11-11 00:00:00
Tasks: Data Augmentation, Hate Speech Detection

The massive spread of hate speech, hateful content targeted at specific subpopulations, is a problem of critical social importance. Automated methods of hate speech detection typically employ state-of-the-art deep learning (DL)-based text classifiers-large pretrained neural language models of over 100 million parameters, adapting these models to the task of hate speech detection using relevant labeled datasets. Unfortunately, there are only a few public labeled datasets of limited size that are available for this purpose. We make several contributions with high potential for advancing this state of affairs. We present HyperNetworks for hate speech detection, a special class of DL networks whose weights are regulated by a small-scale auxiliary network. These architectures operate at character-level, as opposed to word or subword-level, and are several orders of magnitude smaller compared to the popular DL classifiers. We further show that training hate detection classifiers using additional large amounts of automatically generated examples is beneficial in general, yet this practice especially boosts the performance of the proposed HyperNetworks. We report the results of extensive experiments, assessing the performance of multiple neural architectures on hate detection using five public datasets. The assessed methods include the pretrained language models of BERT, RoBERTa, ALBERT, MobileBERT and CharBERT, a variant of BERT that incorporates character alongside subword embeddings. In addition to the traditional setup of within-dataset evaluation, we perform cross-dataset evaluation experiments, testing the generalization of the various models in conditions of data shift. Our results show that the proposed HyperNetworks achieve performance that is competitive, and better in some cases, than these pretrained language models, while being smaller by orders of magnitude.
Self-Normalized Importance Sampling for Neural Language Modeling
Authors: Zijian Yang, Yingbo Gao, Alexander Gerstenberger, Jintao Jiang, Ralf Schlüter, Hermann Ney
Date: 2021-11-11 00:00:00
Tasks: Automatic Speech Recognition, Automatic Speech Recognition (ASR), Language Modeling, Language Modelling, speech-recognition, Speech Recognition

To mitigate the problem of having to traverse over the full vocabulary in the softmax normalization of a neural language model, sampling-based training criteria are proposed and investigated in the context of large vocabulary word-based neural language models. These training criteria typically enjoy the benefit of faster training and testing, at a cost of slightly degraded performance in terms of perplexity and almost no visible drop in word error rate. While noise contrastive estimation is one of the most popular choices, recently we show that other sampling-based criteria can also perform well, as long as an extra correction step is done, where the intended class posterior probability is recovered from the raw model outputs. In this work, we propose self-normalized importance sampling. Compared to our previous work, the criteria considered in this work are self-normalized and there is no need to further conduct a correction step. Through self-normalized language model training as well as lattice rescoring experiments, we show that our proposed self-normalized importance sampling is competitive in both research-oriented and production-oriented automatic speech recognition tasks.
Training Cross-Lingual embeddings for Setswana and Sepedi
Authors: Mack Makgatho, Vukosi Marivate, Tshephisho Sefara, Valencia Wagner
Date: 2021-11-11 00:00:00
Tasks: Cross-Lingual Transfer, Semantic Similarity, Semantic Textual Similarity, Word Embeddings

African languages still lag in the advances of Natural Language Processing techniques, one reason being the lack of representative data, having a technique that can transfer information between languages can help mitigate against the lack of data problem. This paper trains Setswana and Sepedi monolingual word vectors and uses VecMap to create cross-lingual embeddings for Setswana-Sepedi in order to do a cross-lingual transfer. Word embeddings are word vectors that represent words as continuous floating numbers where semantically similar words are mapped to nearby points in n-dimensional space. The idea of word embeddings is based on the distribution hypothesis that states, semantically similar words are distributed in similar contexts (Harris, 1954). Cross-lingual embeddings leverages monolingual embeddings by learning a shared vector space for two separately trained monolingual vectors such that words with similar meaning are represented by similar vectors. In this paper, we investigate cross-lingual embeddings for Setswana-Sepedi monolingual word vector. We use the unsupervised cross lingual embeddings in VecMap to train the Setswana-Sepedi cross-language word embeddings. We evaluate the quality of the Setswana-Sepedi cross-lingual word representation using a semantic evaluation task. For the semantic similarity task, we translated the WordSim and SimLex tasks into Setswana and Sepedi. We release this dataset as part of this work for other researchers. We evaluate the intrinsic quality of the embeddings to determine if there is improvement in the semantic representation of the word embeddings.
Multilingual and Multilabel Emotion Recognition using Virtual Adversarial Training
Authors: Vikram Gupta
Date: 2021-11-11 00:00:00
Proceeding: EMNLP (MRL) 2021 11
Tasks: Emotion Recognition, Multilabel Text Classification, text-classification, Text Classification

Virtual Adversarial Training (VAT) has been effective in learning robust models under supervised and semi-supervised settings for both computer vision and NLP tasks. However, the efficacy of VAT for multilingual and multilabel text classification has not been explored before. In this work, we explore VAT for multilabel emotion recognition with a focus on leveraging unlabelled data from different languages to improve the model performance. We perform extensive semi-supervised experiments on SemEval2018 multilabel and multilingual emotion recognition dataset and show performance gains of 6.2% (Arabic), 3.8% (Spanish) and 1.8% (English) over supervised learning with same amount of labelled data (10% of training data). We also improve the existing state-of-the-art by 7%, 4.5% and 1% (Jaccard Index) for Spanish, Arabic and English respectively and perform probing experiments for understanding the impact of different layers of the contextual models.
An Enactivist account of Mind Reading in Natural Language Understanding
Authors: Peter Wallis
Date: 2021-11-11 00:00:00
Tasks: Formal Logic, Natural Language Understanding

In this paper we apply our understanding of the radical enactivist agenda to the classic AI-hard problem of Natural Language Understanding. When Turing devised his famous test the assumption was that a computer could use language and the challenge would be to mimic human intelligence. It turned out playing chess and formal logic were easy compared to understanding what people say. The techniques of good old-fashioned AI (GOFAI) assume symbolic representation is the core of reasoning and by that paradigm human communication consists of transferring representations from one mind to another. However, one finds that representations appear in another's mind, without appearing in the intermediary language. People communicate by mind reading it seems. Systems with speech interfaces such as Alexa and Siri are of course common, but they are limited. Rather than adding mind reading skills, we introduced a "cheat" that enabled our systems to fake it. The cheat is simple and only slightly interesting to computer scientists and not at all interesting to philosophers. However, reading about the enactivist idea that we "directly perceive" the intentions of others, our cheat took on a new light and in this paper look again at how natural language understanding might actually work between humans.
From words to connections: Word use similarity as an honest signal conducive to employees' digital communication
Authors: A. Fronzetti Colladon, J. Saint-Charles, P. Mongeau
Date: 2021-11-11 00:00:00

Bringing together considerations from three research trends (honest signals of collaboration, socio-semantic networks and homophily theory), we hypothesise that word use similarity and having similar social network positions are linked with the level of employees' digital interaction. To verify our hypothesis, we analyse the communication of close to 1600 employees, interacting on the intranet communication forum of a large company. We study their social dynamics and the 'honest signals' that, in past research, proved to be conducive to employees' engagement and collaboration. We find that word use similarity is the main driver of interaction, much more than other language characteristics or similarity in network position. Our results suggest carefully choosing the language according to the target audience and have practical implications for both company managers and online community administrators. Understanding how to better use language could, for example, support the development of knowledge sharing practices or internal communication campaigns.
Improving Large-scale Language Models and Resources for Filipino
Authors: Jan Christian Blaise Cruz, Charibeth Cheng
Date: 2021-11-11 00:00:00
Proceeding: LREC 2022 6

In this paper, we improve on existing language resources for the low-resource Filipino language in two ways. First, we outline the construction of the TLUnified dataset, a large-scale pretraining corpus that serves as an improvement over smaller existing pretraining datasets for the language in terms of scale and topic variety. Second, we pretrain new Transformer language models following the RoBERTa pretraining technique to supplant existing models trained with small corpora. Our new RoBERTa models show significant improvements over existing Filipino models in three benchmark datasets with an average gain of 4.47% test accuracy across the three classification tasks of varying difficulty.
Scaling ASR Improves Zero and Few Shot Learning
Authors: Alex Xiao, Weiyi Zheng, Gil Keren, Duc Le, Frank Zhang, Christian Fuegen, Ozlem Kalinli, Yatharth Saraf, Abdelrahman Mohamed
Date: 2021-11-10 00:00:00
Tasks: Automatic Speech Recognition, Automatic Speech Recognition (ASR), Few-Shot Learning, speech-recognition, Speech Recognition

With 4.5 million hours of English speech from 10 different sources across 120 countries and models of up to 10 billion parameters, we explore the frontiers of scale for automatic speech recognition. We propose data selection techniques to efficiently scale training data to find the most valuable samples in massive datasets. To efficiently scale model sizes, we leverage various optimizations such as sparse transducer loss and model sharding. By training 1-10B parameter universal English ASR models, we push the limits of speech recognition performance across many domains. Furthermore, our models learn powerful speech representations with zero and few-shot capabilities on novel domains and styles of speech, exceeding previous results across multiple in-house and public benchmarks. For speakers with disorders due to brain damage, our best zero-shot and few-shot models achieve 22% and 60% relative improvement on the AphasiaBank test set, respectively, while realizing the best performance on public social media videos. Furthermore, the same universal model reaches equivalent performance with 500x less in-domain data on the SPGISpeech financial-domain dataset.
A Novel Corpus of Discourse Structure in Humans and Computers
Authors: Babak Hemmatian, Sheridan Feucht, Rachel Avram, Alexander Wey, Muskaan Garg, Kate Spitalnic, Carsten Eickhoff, Ellie Pavlick, Bjorn Sandstede, Steven Sloman
Date: 2021-11-10 00:00:00
Tasks: Text Generation

We present a novel corpus of 445 human- and computer-generated documents, comprising about 27,000 clauses, annotated for semantic clause types and coherence relations that allow for nuanced comparison of artificial and natural discourse modes. The corpus covers both formal and informal discourse, and contains documents generated using fine-tuned GPT-2 (Zellers et al., 2019) and GPT-3(Brown et al., 2020). We showcase the usefulness of this corpus for detailed discourse analysis of text generation by providing preliminary evidence that less numerous, shorter and more often incoherent clause relations are associated with lower perceived quality of computer-generated narratives and arguments.
Does the Brain Infer Invariance Transformations from Graph Symmetries?
Authors: Helmut Linde
Date: 2021-11-11 00:00:00

The invariance of natural objects under perceptual changes is possibly encoded in the brain by symmetries in the graph of synaptic connections. The graph can be established via unsupervised learning in a biologically plausible process across different perceptual modalities. This hypothetical encoding scheme is supported by the correlation structure of naturalistic audio and image data and it predicts a neural connectivity architecture which is consistent with many empirical observations about primary sensory cortex.
On Recovering the Best Rank-r Approximation from Few Entries
Authors: Shun Xu, Ming Yuan
Date: 2021-11-11 00:00:00

In this note, we investigate how well we can reconstruct the best rank-$r$ approximation of a large matrix from a small number of its entries. We show that even if a data matrix is of full rank and cannot be approximated well by a low-rank matrix, its best low-rank approximations may still be reliably computed or estimated from a small number of its entries. This is especially relevant from a statistical viewpoint: the best low-rank approximations to a data matrix are often of more interest than itself because they capture the more stable and oftentimes more reproducible properties of an otherwise complicated data-generating model. In particular, we investigate two agnostic approaches: the first is based on spectral truncation; and the second is a projected gradient descent based optimization procedure. We argue that, while the first approach is intuitive and reasonably effective, the latter has far superior performance in general. We show that the error depends on how close the matrix is to being of low rank. Both theoretical and numerical evidence is presented to demonstrate the effectiveness of the proposed approaches.
Machine Learning-Based Optimization of Chiral Photonic Nanostructures: Evolution- and Neural Network-Based Design
Authors: Oliver Mey, Arash Rahimi-Iman
Date: 2021-11-11 00:00:00

Chiral photonics opens new pathways to manipulate light-matter interactions and tailor the optical response of meta-surfaces and -materials by nanostructuring nontrivial patterns. Chirality of matter, such as that of molecules, and light, which in the simplest case is given by the handedness of circular polarization, have attracted much attention for applications in chemistry, nanophotonics and optical information processing. We report the design of chiral photonic structures using two machine learning methods, the evolutionary algorithm and neural network approach, for rapid and efficient optimization of optical properties for dielectric metasurfaces. The design recipes obtained for visible light in the range of transition-metal dichalcogenide exciton resonances show a frequency-dependent modification in the reflected light's degree of circular polarization, that is represented by the difference between left- and right-circularly polarized intensity. Our results suggest the facile fabrication and characterization of optical nanopatterned reflectors for chirality-sensitive light-matter coupling scenarios employing tungsten disulfide as possible active material with features such as valley Hall effect and optical valley coherence.
Adversarial sampling of unknown and high-dimensional conditional distributions
Authors: Malik Hassanaly, Andrew Glaws, Karen Stengel, Ryan N. King
Date: 2021-11-08 00:00:00
Tasks: Vocal Bursts Intensity Prediction

Many engineering problems require the prediction of realization-to-realization variability or a refined description of modeled quantities. In that case, it is necessary to sample elements from unknown high-dimensional spaces with possibly millions of degrees of freedom. While there exist methods able to sample elements from probability density functions (PDF) with known shapes, several approximations need to be made when the distribution is unknown. In this paper the sampling method, as well as the inference of the underlying distribution, are both handled with a data-driven method known as generative adversarial networks (GAN), which trains two competing neural networks to produce a network that can effectively generate samples from the training set distribution. In practice, it is often necessary to draw samples from conditional distributions. When the conditional variables are continuous, only one (if any) data point corresponding to a particular value of a conditioning variable may be available, which is not sufficient to estimate the conditional distribution. This work handles this problem using an a priori estimation of the conditional moments of a PDF. Two approaches, stochastic estimation, and an external neural network are compared here for computing these moments; however, any preferred method can be used. The algorithm is demonstrated in the case of the deconvolution of a filtered turbulent flow field. It is shown that all the versions of the proposed algorithm effectively sample the target conditional distribution with minimal impact on the quality of the samples compared to state-of-the-art methods. Additionally, the procedure can be used as a metric for the diversity of samples generated by a conditional GAN (cGAN) conditioned with continuous variables.
ARISE: ApeRIodic SEmi-parametric Process for Efficient Markets without Periodogram and Gaussianity Assumptions
Authors: Shao-Qun Zhang, Zhi-Hua Zhou
Date: 2021-11-08 00:00:00
Tasks: BIG-bench Machine Learning, Time Series, Time Series Analysis

Mimicking and learning the long-term memory of efficient markets is a fundamental problem in the interaction between machine learning and financial economics to sequential data. Despite the prominence of this issue, current treatments either remain largely limited to heuristic techniques or rely significantly on periodogram or Gaussianty assumptions. In this paper, we present the ApeRIodic SEmi-parametric (ARISE) process for investigating efficient markets. The ARISE process is formulated as an infinite-sum function of some known processes and employs the aperiodic spectrum estimation to determine the key hyper-parameters, thus possessing the power and potential of modeling the price data with long-term memory, non-stationarity, and aperiodic spectrum. We further theoretically show that the ARISE process has the mean-square convergence, consistency, and asymptotic normality without periodogram and Gaussianity assumptions. In practice, we apply the ARISE process to identify the efficiency of real-world markets. Besides, we also provide two alternative ARISE applications: studying the long-term memorability of various machine-learning models and developing a latent state-space model for inference and forecasting of time series. The numerical experiments confirm the superiority of our proposed approaches.
Data-Centric Engineering: integrating simulation, machine learning and statistics. Challenges and Opportunities
Authors: Indranil Pan, Lachlan Mason, Omar Matar
Date: 2021-11-07 00:00:00
Tasks: BIG-bench Machine Learning

Recent advances in machine learning, coupled with low-cost computation, availability of cheap streaming sensors, data storage and cloud technologies, has led to widespread multi-disciplinary research activity with significant interest and investment from commercial stakeholders. Mechanistic models, based on physical equations, and purely data-driven statistical approaches represent two ends of the modelling spectrum. New hybrid, data-centric engineering approaches, leveraging the best of both worlds and integrating both simulations and data, are emerging as a powerful tool with a transformative impact on the physical disciplines. We review the key research trends and application scenarios in the emerging field of integrating simulations, machine learning, and statistics. We highlight the opportunities that such an integrated vision can unlock and outline the key challenges holding back its realisation. We also discuss the bottlenecks in the translational aspects of the field and the long-term upskilling requirements of the existing workforce and future university graduates.
Occupational Income Inequality of Thailand: A Case Study of Exploratory Data Analysis beyond Gini Coefficient
Authors: Wanetha Sudswong, Anon Plangprasopchok, Chainarong Amornbunchornvej
Date: 2021-11-05 00:00:00

Income inequality is an important issue that has to be solved in order to make progress in our society. The study of income inequality is well received through the Gini coefficient, which is used to measure degrees of inequality in general. While this method is effective in several aspects, the Gini coefficient alone inevitably overlooks minority subpopulations (e.g. occupations) which results in missing undetected patterns of inequality in minority. In this study, the surveys of incomes and occupations from more than 12 millions households across Thailand have been analyzed by using both Gini coefficient and network densities of income domination networks to get insight regarding the degrees of general and occupational income inequality issues. The results show that, in agricultural provinces, there are less issues in both types of inequality (low Gini coefficients and network densities), while some non-agricultural provinces face an issue of occupational income inequality (high network densities) without any symptom of general income inequality (low Gini coefficients). Moreover, the results also illustrate the gaps of income inequality using estimation statistics, which not only support whether income inequality exists, but that we are also able to tell the magnitudes of income gaps among occupations. These results cannot be obtained via Gini coefficients alone. This work serves as a use case of analyzing income inequality from both general population and subpopulations perspectives that can be utilized in studies of other countries.
Joint Radar-Communications Processing from a Dual-Blind Deconvolution Perspective
Authors: Edwin Vargas, Kumar Vijay Mishra, Roman Jacome, Brian M. Sadler, Henry Arguello
Date: 2021-11-11 00:00:00

We consider a general spectral coexistence scenario, wherein the channels and transmit signals of both radar and communications systems are unknown at the receiver. In this \textit{dual-blind deconvolution} (DBD) problem, a common receiver admits the multi-carrier wireless communications signal that is overlaid with the radar signal reflected-off multiple targets. When the radar receiver is not collocated with the transmitter, such as in passive or multistatic radars, the transmitted signal is also unknown apart from the target parameters. Similarly, apart from the transmitted messages, the communications channel may also be unknown in dynamic environments such as vehicular networks. As a result, the estimation of unknown target and communications parameters in a DBD scenario is highly challenging. In this work, we exploit the sparsity of the channel to solve DBD by casting it as an atomic norm minimization problem. Our theoretical analyses and numerical experiments demonstrate perfect recovery of continuous-valued range-time and Doppler velocities of multiple targets as well as delay-Doppler communications channel parameters using uniformly-spaced time samples in the dual-blind receiver.