pwc-archive/papers-with-abstracts huggingface dataset:
The Information Bottleneck Problem and Its Applications in Machine Learning
Authors: Ziv Goldfeld, Yury Polyanskiy
Date: 2020-04-30 00:00:00
Tasks: BIG-bench Machine Learning, Dimensionality Reduction

Inference capabilities of machine learning (ML) systems skyrocketed in recent years, now playing a pivotal role in various aspect of society. The goal in statistical learning is to use data to obtain simple algorithms for predicting a random variable $Y$ from a correlated observation $X$. Since the dimension of $X$ is typically huge, computationally feasible solutions should summarize it into a lower-dimensional feature vector $T$, from which $Y$ is predicted. The algorithm will successfully make the prediction if $T$ is a good proxy of $Y$, despite the said dimensionality-reduction. A myriad of ML algorithms (mostly employing deep learning (DL)) for finding such representations $T$ based on real-world data are now available. While these methods are often effective in practice, their success is hindered by the lack of a comprehensive theory to explain it. The information bottleneck (IB) theory recently emerged as a bold information-theoretic paradigm for analyzing DL systems. Adopting mutual information as the figure of merit, it suggests that the best representation $T$ should be maximally informative about $Y$ while minimizing the mutual information with $X$. In this tutorial we survey the information-theoretic origins of this abstract principle, and its recent impact on DL. For the latter, we cover implications of the IB problem on DL theory, as well as practical algorithms inspired by it. Our goal is to provide a unified and cohesive description. A clear view of current knowledge is particularly important for further leveraging IB and other information-theoretic ideas to study DL models.
PeerNomination: Relaxing Exactness for Increased Accuracy in Peer Selection
Authors: Nicholas Mattei, Paolo Turrini, Stanislav Zhydkov
Date: 2020-04-30 00:00:00

In peer selection agents must choose a subset of themselves for an award or a prize. As agents are self-interested, we want to design algorithms that are impartial, so that an individual agent cannot affect their own chance of being selected. This problem has broad application in resource allocation and mechanism design and has received substantial attention in the artificial intelligence literature. Here, we present a novel algorithm for impartial peer selection, PeerNomination, and provide a theoretical analysis of its accuracy. Our algorithm possesses various desirable features. In particular, it does not require an explicit partitioning of the agents, as previous algorithms in the literature. We show empirically that it achieves higher accuracy than the exiting algorithms over several metrics.
Generative Adversarial Networks in Digital Pathology: A Survey on Trends and Future Potential
Authors: Maximilian Ernst Tschuchnig, Gertie Janneke Oostingh, Michael Gadermayr
Date: 2020-04-30 00:00:00

Image analysis in the field of digital pathology has recently gained increased popularity. The use of high-quality whole slide scanners enables the fast acquisition of large amounts of image data, showing extensive context and microscopic detail at the same time. Simultaneously, novel machine learning algorithms have boosted the performance of image analysis approaches. In this paper, we focus on a particularly powerful class of architectures, called Generative Adversarial Networks (GANs), applied to histological image data. Besides improving performance, GANs also enable application scenarios in this field, which were previously intractable. However, GANs could exhibit a potential for introducing bias. Hereby, we summarize the recent state-of-the-art developments in a generalizing notation, present the main applications of GANs and give an outlook of some chosen promising approaches and their possible future applications. In addition, we identify currently unavailable methods with potential for future applications.
Perceptual reasoning based solution methodology for linguistic optimization problems
Authors: Prashant K Gupta, Pranab K. Muhuri
Date: 2020-04-30 00:00:00
Tasks: Decision Making

Decision making in real-life scenarios may often be modeled as an optimization problem. It requires the consideration of various attributes like human preferences and thinking, which constrain achieving the optimal value of the problem objectives. The value of the objectives may be maximized or minimized, depending on the situation. Numerous times, the values of these problem parameters are in linguistic form, as human beings naturally understand and express themselves using words. These problems are therefore termed as linguistic optimization problems (LOPs), and are of two types, namely single objective linguistic optimization problems (SOLOPs) and multi-objective linguistic optimization problems (MOLOPs). In these LOPs, the value of the objective function(s) may not be known at all points of the decision space, and therefore, the objective function(s) as well as problem constraints are linked by the if-then rules. Tsukamoto inference method has been used to solve these LOPs; however, it suffers from drawbacks. As, the use of linguistic information inevitably calls for the utilization of computing with words (CWW), and therefore, 2-tuple linguistic model based solution methodologies were proposed for LOPs. However, we found that 2-tuple linguistic model based solution methodologies represent the semantics of the linguistic information using a combination of type-1 fuzzy sets and ordinal term sets. As, the semantics of linguistic information are best modeled using the interval type-2 fuzzy sets, hence we propose solution methodologies for LOPs based on CWW approach of perceptual computing, in this paper. The perceptual computing based solution methodologies use a novel design of CWW engine, called the perceptual reasoning (PR). PR in the current form is suitable for solving SOLOPs and, hence, we have also extended it to the MOLOPs.
Language Model Prior for Low-Resource Neural Machine Translation
Authors: Christos Baziotis, Barry Haddow, Alexandra Birch
Date: 2020-04-30 00:00:00
Proceeding: EMNLP 2020 11
Tasks: Knowledge Distillation, Language Modeling, Language Modelling, Low Resource Neural Machine Translation, Low-Resource Neural Machine Translation, Machine Translation, Translation

The scarcity of large parallel corpora is an important obstacle for neural machine translation. A common solution is to exploit the knowledge of language models (LM) trained on abundant monolingual data. In this work, we propose a novel approach to incorporate a LM as prior in a neural translation model (TM). Specifically, we add a regularization term, which pushes the output distributions of the TM to be probable under the LM prior, while avoiding wrong predictions when the TM "disagrees" with the LM. This objective relates to knowledge distillation, where the LM can be viewed as teaching the TM about the target language. The proposed approach does not compromise decoding speed, because the LM is used only at training time, unlike previous work that requires it during inference. We present an analysis of the effects that different methods have on the distributions of the TM. Results on two low-resource machine translation datasets show clear improvements even with limited monolingual data.
Addressing Zero-Resource Domains Using Document-Level Context in Neural Machine Translation
Authors: Dario Stojanovski, Alexander Fraser
Date: 2020-04-30 00:00:00
Proceeding: EACL (AdaptNLP) 2021 4
Tasks: Domain Adaptation, Machine Translation, Sentence, Translation

Achieving satisfying performance in machine translation on domains for which there is no training data is challenging. Traditional supervised domain adaptation is not suitable for addressing such zero-resource domains because it relies on in-domain parallel data. We show that when in-domain parallel data is not available, access to document-level context enables better capturing of domain generalities compared to only having access to a single sentence. Having access to more information provides a more reliable domain estimation. We present two document-level Transformer models which are capable of using large context sizes and we compare these models against strong Transformer baselines. We obtain improvements for the two zero resource domains we study. We additionally provide an analysis where we vary the amount of context and look at the case where in-domain data is available.
Bridging Linguistic Typology and Multilingual Machine Translation with Multi-View Language Representations
Authors: Arturo Oncevay, Barry Haddow, Alexandra Birch
Date: 2020-04-30 00:00:00
Proceeding: EMNLP 2020 11
Tasks: Clustering, Machine Translation, Translation

Sparse language vectors from linguistic typology databases and learned embeddings from tasks like multilingual machine translation have been investigated in isolation, without analysing how they could benefit from each other's language characterisation. We propose to fuse both views using singular vector canonical correlation analysis and study what kind of information is induced from each source. By inferring typological features and language phylogenies, we observe that our representations embed typology and strengthen correlations with language relationships. We then take advantage of our multi-view language vector space for multilingual machine translation, where we achieve competitive overall translation accuracy in tasks that require information about language similarities, such as language clustering and ranking candidates for multilingual transfer. With our method, which is also released as a tool, we can easily project and assess new languages without expensive retraining of massive multilingual or ranking models, which are major disadvantages of related approaches.
Tired of Topic Models? Clusters of Pretrained Word Embeddings Make for Fast and Good Topics too!
Authors: Suzanna Sia, Ayush Dalmia, Sabrina J. Mielke
Date: 2020-04-30 00:00:00
Proceeding: EMNLP 2020 11
Tasks: Clustering, Dimensionality Reduction, Reranking, Topic Models, Word Embeddings

Topic models are a useful analysis tool to uncover the underlying themes within document collections. The dominant approach is to use probabilistic topic models that posit a generative story, but in this paper we propose an alternative way to obtain topics: clustering pre-trained word embeddings while incorporating document information for weighted clustering and reranking top words. We provide benchmarks for the combination of different word embeddings and clustering algorithms, and analyse their performance under dimensionality reduction with PCA. The best performing combination for our approach performs as well as classical topic models, but with lower runtime and computational complexity.
Recipes for Adapting Pre-trained Monolingual and Multilingual Models to Machine Translation
Authors: Asa Cooper Stickland, Xi-An Li, Marjan Ghazvininejad
Date: 2020-04-30 00:00:00
Proceeding: EACL 2021 2
Tasks: Decoder, Machine Translation, Translation

There has been recent success in pre-training on monolingual data and fine-tuning on Machine Translation (MT), but it remains unclear how to best leverage a pre-trained model for a given MT task. This paper investigates the benefits and drawbacks of freezing parameters, and adding new ones, when fine-tuning a pre-trained model on MT. We focus on 1) Fine-tuning a model trained only on English monolingual data, BART. 2) Fine-tuning a model trained on monolingual data from 25 languages, mBART. For BART we get the best performance by freezing most of the model parameters, and adding extra positional embeddings. For mBART we match or outperform the performance of naive fine-tuning for most language pairs with the encoder, and most of the decoder, frozen. The encoder-decoder attention parameters are most important to fine-tune. When constraining ourselves to an out-of-domain training set for Vietnamese to English we see the largest improvements over the fine-tuning baseline.
You are right. I am ALARMED -- But by Climate Change Counter Movement
Authors: Shraey Bhatia, Jey Han Lau, Timothy Baldwin
Date: 2020-04-30 00:00:00
Tasks: Articles, Misinformation

The world is facing the challenge of climate crisis. Despite the consensus in scientific community about anthropogenic global warming, the web is flooded with articles spreading climate misinformation. These articles are carefully constructed by climate change counter movement (cccm) organizations to influence the narrative around climate change. We revisit the literature on climate misinformation in social sciences and repackage it to introduce in the community of NLP. Despite considerable work in detection of fake news, there is no misinformation dataset available that is specific to the domain.of climate change. We try to bridge this gap by scraping and releasing articles with known climate change misinformation.
Modelling Suspense in Short Stories as Uncertainty Reduction over Neural Representation
Authors: David Wilmot, Frank Keller
Date: 2020-04-30 00:00:00
Proceeding: ACL 2020 6
Tasks: Language Modeling, Language Modelling

Suspense is a crucial ingredient of narrative fiction, engaging readers and making stories compelling. While there is a vast theoretical literature on suspense, it is computationally not well understood. We compare two ways for modelling suspense: surprise, a backward-looking measure of how unexpected the current state is given the story so far; and uncertainty reduction, a forward-looking measure of how unexpected the continuation of the story is. Both can be computed either directly over story representations or over their probability distributions. We propose a hierarchical language model that encodes stories and computes surprise and uncertainty reduction. Evaluating against short stories annotated with human suspense judgements, we find that uncertainty reduction over representations is the best predictor, resulting in near-human accuracy. We also show that uncertainty reduction can be used to predict suspenseful events in movie synopses.
MLSUM: The Multilingual Summarization Corpus
Authors: Thomas Scialom, Paul-Alexis Dray, Sylvain Lamprier, Benjamin Piwowarski, Jacopo Staiano
Date: 2020-04-30 00:00:00
Proceeding: EMNLP 2020 11
Tasks: Text Summarization

We present MLSUM, the first large-scale MultiLingual SUMmarization dataset. Obtained from online newspapers, it contains 1.5M+ article/summary pairs in five different languages -- namely, French, German, Spanish, Russian, Turkish. Together with English newspapers from the popular CNN/Daily mail dataset, the collected data form a large scale multilingual dataset which can enable new research directions for the text summarization community. We report cross-lingual comparative analyses based on state-of-the-art systems. These highlight existing biases which motivate the use of a multi-lingual dataset.
Polarization Human Shape and Pose Dataset
Authors: Shihao Zou, Xinxin Zuo, Yiming Qian, Sen Wang, Chuan Guo, Chi Xu, Minglun Gong, Li Cheng
Date: 2020-04-30 00:00:00

Polarization images are known to be able to capture polarized reflected lights that preserve rich geometric cues of an object, which has motivated its recent applications in reconstructing detailed surface normal of the objects of interest. Meanwhile, inspired by the recent breakthroughs in human shape estimation from a single color image, we attempt to investigate the new question of whether the geometric cues from polarization camera could be leveraged in estimating detailed human body shapes. This has led to the curation of Polarization Human Shape and Pose Dataset (PHSPD), our home-grown polarization image dataset of various human shapes and poses.
MuSe 2020 -- The First International Multimodal Sentiment Analysis in Real-life Media Challenge and Workshop
Authors: Lukas Stappen, Alice Baird, Georgios Rizos, Panagiotis Tzirakis, Xinchen Du, Felix Hafner, Lea Schumann, Adria Mallol-Ragolta, Björn W. Schuller, Iulia Lefter, Erik Cambria, Ioannis Kompatsiaris
Date: 2020-04-30 00:00:00
Tasks: Emotion Recognition, Multimodal Sentiment Analysis, Sentiment Analysis

Multimodal Sentiment Analysis in Real-life Media (MuSe) 2020 is a Challenge-based Workshop focusing on the tasks of sentiment recognition, as well as emotion-target engagement and trustworthiness detection by means of more comprehensively integrating the audio-visual and language modalities. The purpose of MuSe 2020 is to bring together communities from different disciplines; mainly, the audio-visual emotion recognition community (signal-based), and the sentiment analysis community (symbol-based). We present three distinct sub-challenges: MuSe-Wild, which focuses on continuous emotion (arousal and valence) prediction; MuSe-Topic, in which participants recognise domain-specific topics as the target of 3-class (low, medium, high) emotions; and MuSe-Trust, in which the novel aspect of trustworthiness is to be predicted. In this paper, we provide detailed information on MuSe-CaR, the first of its kind in-the-wild database, which is utilised for the challenge, as well as the state-of-the-art features and modelling approaches applied. For each sub-challenge, a competitive baseline for participants is set; namely, on test we report for MuSe-Wild a combined (valence and arousal) CCC of .2568, for MuSe-Topic a score (computed as 0.34$\cdot$ UAR + 0.66$\cdot$F1) of 76.78 % on the 10-class topic and 40.64 % on the 3-class emotion prediction, and for MuSe-Trust a CCC of .4359.
An empirical study of computing with words approaches for multi-person and single-person systems
Authors: Prashant K Gupta, Pranab K. Muhuri
Date: 2020-04-30 00:00:00

Computing with words (CWW) has emerged as a powerful tool for processing the linguistic information, especially the one generated by human beings. Various CWW approaches have emerged since the inception of CWW, such as perceptual computing, extension principle based CWW approach, symbolic method based CWW approach, and 2-tuple based CWW approach. Furthermore, perceptual computing can use interval approach (IA), enhanced interval approach (EIA), or Hao-Mendel approach (HMA), for data processing. There have been numerous works in which HMA was shown to be better at word modelling than EIA, and EIA better than IA. But, a deeper study of these works reveals that HMA captures lesser fuzziness than the EIA or IA. Thus, we feel that EIA is more suited for word modelling in multi-person systems and HMA for single-person systems (as EIA is an improvement over IA). Furthermore, another set of works, compared the performances perceptual computing to the other above said CWW approaches. In all these works, perceptual computing was shown to be better than other CWW approaches. However, none of the works tried to investigate the reason behind this observed better performance of perceptual computing. Also, no comparison has been performed for scenarios where the inputs are differentially weighted. Thus, the aim of this work is to empirically establish that EIA is suitable for multi-person systems and HMA for single-person systems. Another dimension of this work is also to empirically prove that perceptual computing gives better performance than other CWW approaches based on extension principle, symbolic method and 2-tuple especially in scenarios where inputs are differentially weighted.
Few-Shot Learning for Opinion Summarization
Authors: Arthur Bražinskas, Mirella Lapata, Ivan Titov
Date: 2020-04-30 00:00:00
Proceeding: EMNLP 2020 11
Tasks: Few-Shot Learning, Informativeness, Language Modelling, Opinion Summarization

Opinion summarization is the automatic creation of text reflecting subjective information expressed in multiple documents, such as user reviews of a product. The task is practically important and has attracted a lot of attention. However, due to the high cost of summary production, datasets large enough for training supervised models are lacking. Instead, the task has been traditionally approached with extractive methods that learn to select text fragments in an unsupervised or weakly-supervised way. Recently, it has been shown that abstractive summaries, potentially more fluent and better at reflecting conflicting information, can also be produced in an unsupervised fashion. However, these models, not being exposed to actual summaries, fail to capture their essential properties. In this work, we show that even a handful of summaries is sufficient to bootstrap generation of the summary text with all expected properties, such as writing style, informativeness, fluency, and sentiment preservation. We start by training a conditional Transformer language model to generate a new product review given other available reviews of the product. The model is also conditioned on review properties that are directly related to summaries; the properties are derived from reviews with no manual effort. In the second stage, we fine-tune a plug-in module that learns to predict property values on a handful of summaries. This lets us switch the generator to the summarization mode. We show on Amazon and Yelp datasets that our approach substantially outperforms previous extractive and abstractive methods in automatic and human evaluation.
Distributed Stochastic Nonconvex Optimization and Learning based on Successive Convex Approximation
Authors: Paolo Di Lorenzo, Simone Scardapane
Date: 2020-04-30 00:00:00

We study distributed stochastic nonconvex optimization in multi-agent networks. We introduce a novel algorithmic framework for the distributed minimization of the sum of the expected value of a smooth (possibly nonconvex) function (the agents' sum-utility) plus a convex (possibly nonsmooth) regularizer. The proposed method hinges on successive convex approximation (SCA) techniques, leveraging dynamic consensus as a mechanism to track the average gradient among the agents, and recursive averaging to recover the expected gradient of the sum-utility function. Almost sure convergence to (stationary) solutions of the nonconvex problem is established. Finally, the method is applied to distributed stochastic training of neural networks. Numerical results confirm the theoretical claims, and illustrate the advantages of the proposed method with respect to other methods available in the literature.
PreCNet: Next-Frame Video Prediction Based on Predictive Coding
Authors: Zdenek Straka, Tomas Svoboda, Matej Hoffmann
Date: 2020-04-30 00:00:00
Tasks: Prediction, SSIM, Video Prediction

Predictive coding, currently a highly influential theory in neuroscience, has not been widely adopted in machine learning yet. In this work, we transform the seminal model of Rao and Ballard (1999) into a modern deep learning framework while remaining maximally faithful to the original schema. The resulting network we propose (PreCNet) is tested on a widely used next frame video prediction benchmark, which consists of images from an urban environment recorded from a car-mounted camera, and achieves state-of-the-art performance. Performance on all measures (MSE, PSNR, SSIM) was further improved when a larger training set (2M images from BDD100k), pointing to the limitations of the KITTI training set. This work demonstrates that an architecture carefully based in a neuroscience model, without being explicitly tailored to the task at hand, can exhibit exceptional performance.
Analyzing the Surprising Variability in Word Embedding Stability Across Languages
Authors: Laura Burdick, Jonathan K. Kummerfeld, Rada Mihalcea
Date: 2020-04-30 00:00:00
Proceeding: EMNLP 2021 11
Tasks: Word Embeddings

Word embeddings are powerful representations that form the foundation of many natural language processing architectures, both in English and in other languages. To gain further insight into word embeddings, we explore their stability (e.g., overlap between the nearest neighbors of a word in different embedding spaces) in diverse languages. We discuss linguistic properties that are related to stability, drawing out insights about correlations with affixing, language gender systems, and other features. This has implications for embedding use, particularly in research that uses them to study language trends.