pwc-archive/papers-with-abstracts huggingface dataset:
Easy Adaptation to Mitigate Gender Bias in Multilingual Text Classification
Authors: Xiaolei Huang
Date: 2022-04-12 00:00:00
Proceeding: NAACL 2022 7
Tasks: Domain Adaptation, Hate Speech Detection, Multilingual text classification, text-classification, Text Classification

Existing approaches to mitigate demographic biases evaluate on monolingual data, however, multilingual data has not been examined. In this work, we treat the gender as domains (e.g., male vs. female) and present a standard domain adaptation model to reduce the gender bias and improve performance of text classifiers under multilingual settings. We evaluate our approach on two text classification tasks, hate speech detection and rating prediction, and demonstrate the effectiveness of our approach with three fair-aware baselines.
A Call for Clarity in Beam Search: How It Works and When It Stops
Authors: Jungo Kasai, Keisuke Sakaguchi, Ronan Le Bras, Dragomir Radev, Yejin Choi, Noah A. Smith
Date: 2022-04-11 00:00:00
Tasks: Machine Translation, Text Generation, Text Summarization, Translation

Text generation with beam search has proven successful in a wide range of applications. We point out that, though largely overlooked in the literature, the commonly-used implementation of beam decoding (e.g., Hugging Face Transformers and fairseq) uses a first come, first served heuristic: it keeps a set of already completed sequences over time steps and stops when the size of this set reaches the beam size. Based on this finding, we introduce a patience factor, a simple modification to this beam decoding implementation, that generalizes the stopping criterion and provides flexibility to the depth of search. Empirical results demonstrate that adjusting this patience factor improves decoding performance of strong pretrained models on news text summarization and machine translation over diverse language pairs, with a negligible inference slowdown. Our approach only modifies one line of code and can be thus readily incorporated in any implementation. Further, we find that different versions of beam decoding result in large performance differences in summarization, demonstrating the need for clarity in specifying the beam search implementation in research work. Our code will be available upon publication.
Unified Speech-Text Pre-training for Speech Translation and Recognition
Authors: Yun Tang, Hongyu Gong, Ning Dong, Changhan Wang, Wei-Ning Hsu, Jiatao Gu, Alexei Baevski, Xian Li, Abdelrahman Mohamed, Michael Auli, Juan Pino
Date: 2022-04-11 00:00:00
Proceeding: ACL 2022 5
Tasks: Decoder, speech-recognition, Speech Recognition, Translation

We describe a method to jointly pre-train speech and text in an encoder-decoder modeling framework for speech translation and recognition. The proposed method incorporates four self-supervised and supervised subtasks for cross modality learning. A self-supervised speech subtask leverages unlabelled speech data, and a (self-)supervised text to text subtask makes use of abundant text training data. Two auxiliary supervised speech tasks are included to unify speech and text modeling space. Our contribution lies in integrating linguistic information from the text corpus into the speech pre-training. Detailed analysis reveals learning interference among subtasks. Two pre-training configurations for speech translation and recognition, respectively, are presented to alleviate subtask interference. Our experiments show the proposed method can effectively fuse speech and text information into one model. It achieves between 1.7 and 2.3 BLEU improvement above the state of the art on the MuST-C speech translation dataset and comparable WERs to wav2vec 2.0 on the Librispeech speech recognition task.
A Generative Language Model for Few-shot Aspect-Based Sentiment Analysis
Authors: Ehsan Hosseini-Asl, Wenhao Liu, Caiming Xiong
Date: 2022-04-11 00:00:00
Proceeding: Findings (NAACL) 2022 7
Tasks: Aspect-Based Sentiment Analysis, Aspect-Based Sentiment Analysis (ABSA), Intent Detection, Language Modeling, Language Modelling, Sentiment Analysis, SST-2, Text Generation

Sentiment analysis is an important task in natural language processing. In recent works, pre-trained language models are often used to achieve state-of-the-art results, especially when training data is scarce. It is common to fine-tune on the downstream task, usually by adding task-specific layers on top of the model. In this paper, we focus on aspect-based sentiment analysis, which involves extracting aspect term, category, and predicting their corresponding polarities. In particular, we are interested in few-shot settings. We propose to reformulate the extraction and prediction tasks into the sequence generation task, using a generative language model with unidirectional attention (GPT2 is used unless stated otherwise). This way, the model learns to accomplish the tasks via language generation without the need of training task-specific layers. Our evaluation results on the single-task polarity prediction show that our approach outperforms the previous state-of-the-art (based on BERT) on average performance by a large margins in few-shot and full-shot settings. More importantly, our generative approach significantly reduces the model variance caused by low-resource data. We further demonstrate that the proposed generative language model can handle joint and multi-task settings, unlike previous work. We observe that the proposed sequence generation method achieves further improved performances on polarity prediction when the model is trained via joint and multi-task settings. Further evaluation on similar sentiment analysis datasets, SST-2, SST- and OOS intent detection validates the superiority and noise robustness of generative language model in few-shot settings.
Large-Scale Streaming End-to-End Speech Translation with Neural Transducers
Authors: Jian Xue, Peidong Wang, Jinyu Li, Matt Post, Yashesh Gaur
Date: 2022-04-11 00:00:00
Tasks: Automatic Speech Recognition, Automatic Speech Recognition (ASR), Machine Translation, speech-recognition, Speech Recognition, Translation

Neural transducers have been widely used in automatic speech recognition (ASR). In this paper, we introduce it to streaming end-to-end speech translation (ST), which aims to convert audio signals to texts in other languages directly. Compared with cascaded ST that performs ASR followed by text-based machine translation (MT), the proposed Transformer transducer (TT)-based ST model drastically reduces inference latency, exploits speech information, and avoids error propagation from ASR to MT. To improve the modeling capacity, we propose attention pooling for the joint network in TT. In addition, we extend TT-based ST to multilingual ST, which generates texts of multiple languages at the same time. Experimental results on a large-scale 50 thousand (K) hours pseudo-labeled training set show that TT-based ST not only significantly reduces inference time but also outperforms non-streaming cascaded ST for English-German translation.
pysamoo: Surrogate-Assisted Multi-Objective Optimization in Python
Authors: Julian Blank, Kalyanmoy Deb
Date: 2022-04-12 00:00:00

Significant effort has been made to solve computationally expensive optimization problems in the past two decades, and various optimization methods incorporating surrogates into optimization have been proposed. However, most optimization toolboxes do not consist of ready-to-run algorithms for computationally expensive problems, especially in combination with other key requirements, such as handling multiple conflicting objectives or constraints. Thus, the lack of appropriate software packages has become a bottleneck for solving real-world applications. The proposed framework, pysamoo, addresses these shortcomings of existing optimization frameworks and provides multiple optimization methods for handling problems involving time-consuming evaluation functions. The framework extends the functionalities of pymoo, a popular and comprehensive toolbox for multi-objective optimization, and incorporates surrogates to support expensive function evaluations. The framework is available under the GNU Affero General Public License (AGPL) and is primarily designed for research purposes. For more information about pysamoo, readers are encouraged to visit: anyoptimization.com/projects/pysamoo.
Hard Problems are Easier for Success-based Parameter Control
Authors: Mario Alejandro Hevia Fajardo, Dirk Sudholt
Date: 2022-04-12 00:00:00
Tasks: Evolutionary Algorithms

Recent works showed that simple success-based rules for self-adjusting parameters in evolutionary algorithms (EAs) can match or outperform the best fixed parameters on discrete problems. Non-elitism in a (1,$\lambda$) EA combined with a self-adjusting offspring population size $\lambda$ outperforms common EAs on the multimodal Cliff problem. However, it was shown that this only holds if the success rate $s$ that governs self-adjustment is small enough. Otherwise, even on OneMax, the self-adjusting (1,$\lambda$) EA stagnates on an easy slope, where frequent successes drive down the offspring population size. We show that self-adjustment works as intended in the absence of easy slopes. We define everywhere hard functions, for which successes are never easy to find and show that the self-adjusting (1,$\lambda$) EA is robust with respect to the choice of success rates $s$. We give a general fitness-level upper bound on the number of evaluations and show that the expected number of generations is at most $O(d + \log(1/p_{\min}))$ where $d$ is the number of non-optimal fitness values and $p_{\min}$ is the smallest probability of finding an improvement from a non-optimal search point. We discuss implications for the everywhere hard function LeadingOnes and a new class OneMaxBlocks of everywhere hard functions with tunable difficulty.
Neural Network Pruning by Cooperative Coevolution
Authors: Haopu Shang, Jia-Liang Wu, Wenjing Hong, Chao Qian
Date: 2022-04-12 00:00:00
Tasks: Evolutionary Algorithms, Model Compression, Network Pruning

Neural network pruning is a popular model compression method which can significantly reduce the computing cost with negligible loss of accuracy. Recently, filters are often pruned directly by designing proper criteria or using auxiliary modules to measure their importance, which, however, requires expertise and trial-and-error. Due to the advantage of automation, pruning by evolutionary algorithms (EAs) has attracted much attention, but the performance is limited for deep neural networks as the search space can be quite large. In this paper, we propose a new filter pruning algorithm CCEP by cooperative coevolution, which prunes the filters in each layer by EAs separately. That is, CCEP reduces the pruning space by a divide-and-conquer strategy. The experiments show that CCEP can achieve a competitive performance with the state-of-the-art pruning methods, e.g., prune ResNet56 for $63.42\%$ FLOPs on CIFAR10 with $-0.24\%$ accuracy drop, and ResNet50 for $44.56\%$ FLOPs on ImageNet with $0.07\%$ accuracy drop.
Evolutionary Algorithms for Limiting the Effect of Uncertainty for the Knapsack Problem with Stochastic Profits
Authors: Aneta Neumann, Yue Xie, Frank Neumann
Date: 2022-04-12 00:00:00
Tasks: Evolutionary Algorithms, Stochastic Optimization

Evolutionary algorithms have been widely used for a range of stochastic optimization problems in order to address complex real-world optimization problems. We consider the knapsack problem where the profits involve uncertainties. Such a stochastic setting reflects important real-world scenarios where the profit that can be realized is uncertain. We introduce different ways of dealing with stochastic profits based on tail inequalities such as Chebyshev's inequality and Hoeffding bounds that allow to limit the impact of uncertainties. We examine simple evolutionary algorithms and the use of heavy tail mutation and a problem-specific crossover operator for optimizing uncertain profits. Our experimental investigations on different benchmarks instances show the results of different approaches based on tail inequalities as well as improvements achievable through heavy tail mutation and the problem specific crossover operator.
Coevolutionary Pareto Diversity Optimization
Authors: Aneta Neumann, Denis Antipov, Frank Neumann
Date: 2022-04-12 00:00:00
Tasks: Diversity

Computing diverse sets of high quality solutions for a given optimization problem has become an important topic in recent years. In this paper, we introduce a coevolutionary Pareto Diversity Optimization approach which builds on the success of reformulating a constrained single-objective optimization problem as a bi-objective problem by turning the constraint into an additional objective. Our new Pareto Diversity optimization approach uses this bi-objective formulation to optimize the problem while also maintaining an additional population of high quality solutions for which diversity is optimized with respect to a given diversity measure. We show that our standard co-evolutionary Pareto Diversity Optimization approach outperforms the recently introduced DIVEA algorithm which obtains its initial population by generalized diversifying greedy sampling and improving the diversity of the set of solutions afterwards. Furthermore, we study possible improvements of the Pareto Diversity Optimization approach. In particular, we show that the use of inter-population crossover further improves the diversity of the set of solutions.
Implementing Online Reinforcement Learning with Temporal Neural Networks
Authors: James E. Smith
Date: 2022-04-11 00:00:00
Tasks: Clustering, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

A Temporal Neural Network (TNN) architecture for implementing efficient online reinforcement learning is proposed and studied via simulation. The proposed T-learning system is composed of a frontend TNN that implements online unsupervised clustering and a backend TNN that implements online reinforcement learning. The reinforcement learning paradigm employs biologically plausible neo-Hebbian three-factor learning rules. As a working example, a prototype implementation of the cart-pole problem (balancing an inverted pendulum) is studied via simulation.
SATA: Sparsity-Aware Training Accelerator for Spiking Neural Networks
Authors: Ruokai Yin, Abhishek Moitra, Abhiroop Bhattacharjee, Youngeun Kim, Priyadarshini Panda
Date: 2022-04-11 00:00:00

Spiking Neural Networks (SNNs) have gained huge attention as a potential energy-efficient alternative to conventional Artificial Neural Networks (ANNs) due to their inherent high-sparsity activation. Recently, SNNs with backpropagation through time (BPTT) have achieved a higher accuracy result on image recognition tasks than other SNN training algorithms. Despite the success from the algorithm perspective, prior works neglect the evaluation of the hardware energy overheads of BPTT due to the lack of a hardware evaluation platform for this SNN training algorithm. Moreover, although SNNs have long been seen as an energy-efficient counterpart of ANNs, a quantitative comparison between the training cost of SNNs and ANNs is missing. To address the aforementioned issues, in this work, we introduce SATA (Sparsity-Aware Training Accelerator), a BPTT-based training accelerator for SNNs. The proposed SATA provides a simple and re-configurable systolic-based accelerator architecture, which makes it easy to analyze the training energy for BPTT-based SNN training algorithms. By utilizing the sparsity, SATA increases its computation energy efficiency by $5.58 \times$ compared to the one without using sparsity. Based on SATA, we show quantitative analyses of the energy efficiency of SNN training and compare the training cost of SNNs and ANNs. The results show that, on Eyeriss-like systolic-based architecture, SNNs consume $1.27\times$ more total energy with sparsities when compared to ANNs. We find that such high training energy cost is from time-repetitive convolution operations and data movements during backpropagation. Moreover, to propel the future SNN training algorithm design, we provide several observations on energy efficiency for different SNN-specific training parameters and propose an energy estimation framework for SNN training. Code for our framework is made publicly available.
Sparse Interaction Neighborhood Selection for Markov Random Fields via Reversible Jump and Pseudoposteriors
Authors: Victor Freguglia, Nancy Lopes Garcia
Date: 2022-04-12 00:00:00
Tasks: Model Selection

We consider the problem of estimating the interacting neighborhood of a Markov Random Field model with finite support and homogeneous pairwise interactions based on relative positions of a two-dimensional lattice. Using a Bayesian framework, we propose a Reversible Jump Monte Carlo Markov Chain algorithm that jumps across subsets of a maximal range neighborhood, allowing us to perform model selection based on a marginal pseudoposterior distribution of models. To show the strength of our proposed methodology we perform a simulation study and apply it to a real dataset from a discrete texture image analysis.
Robust online joint state/input/parameter estimation of linear systems
Authors: Jean-Sébastien Brouillon, Keith Moffat, Florian Dörfler, Giancarlo Ferrari-Trecate
Date: 2022-04-12 00:00:00
Tasks: parameter estimation, regression

This paper presents a method for jointly estimating the state, input, and parameters of linear systems in an online fashion. The method is specially designed for measurements that are corrupted with non-Gaussian noise or outliers, which are commonly found in engineering applications. In particular, it combines recursive, alternating, and iteratively-reweighted least squares into a single, one-step algorithm, which solves the estimation problem online and benefits from the robustness of least-deviation regression methods. The convergence of the iterative method is formally guaranteed. Numerical experiments show the good performance of the estimation algorithm in presence of outliers and in comparison to state-of-the-art methods.
Inference of hyperedges and overlapping communities in hypergraphs
Authors: Martina Contisciani, Federico Battiston, Caterina De Bacco
Date: 2022-04-12 00:00:00
Tasks: Hyperedge Prediction

Hypergraphs, encoding structured interactions among any number of system units, have recently proven a successful tool to describe many real-world biological and social networks. Here we propose a framework based on statistical inference to characterize the structural organization of hypergraphs. The method allows to infer missing hyperedges of any size in a principled way, and to jointly detect overlapping communities in presence of higher-order interactions. Furthermore, our model has an efficient numerical implementation, and it runs faster than dyadic algorithms on pairwise records projected from higher-order data. We apply our method to a variety of real-world systems, showing strong performance in hyperedge prediction tasks, detecting communities well aligned with the information carried by interactions, and robustness against addition of noisy hyperedges. Our approach illustrates the fundamental advantages of a hypergraph probabilistic model when modeling relational systems with higher-order interactions.
Identifying the Dynamics of a System by Leveraging Data from Similar Systems
Authors: Lei Xin, Lintao Ye, George Chiu, Shreyas Sundaram
Date: 2022-04-11 00:00:00

We study the problem of identifying the dynamics of a linear system when one has access to samples generated by a similar (but not identical) system, in addition to data from the true system. We use a weighted least squares approach and provide finite sample performance guarantees on the quality of the identified dynamics. Our results show that one can effectively use the auxiliary data generated by the similar system to reduce the estimation error due to the process noise, at the cost of adding a portion of error that is due to intrinsic differences in the models of the true and auxiliary systems. We also provide numerical experiments to validate our theoretical results. Our analysis can be applied to a variety of important settings. For example, if the system dynamics change at some point in time (e.g., due to a fault), how should one leverage data from the prior system in order to learn the dynamics of the new system? As another example, if there is abundant data available from a simulated (but imperfect) model of the true system, how should one weight that data compared to the real data from the system? Our analysis provides insights into the answers to these questions.
Do Loyal Users Enjoy Better Recommendations? Understanding Recommender Accuracy from a Time Perspective
Authors: Yitong Ji, Aixin Sun, Jie Zhang, Chenliang Li
Date: 2022-04-12 00:00:00
Tasks: Recommendation Systems

In academic research, recommender systems are often evaluated on benchmark datasets, without much consideration about the global timeline. Hence, we are unable to answer questions like: Do loyal users enjoy better recommendations than non-loyal users? Loyalty can be defined by the time period a user has been active in a recommender system, or by the number of historical interactions a user has. In this paper, we offer a comprehensive analysis of recommendation results along global timeline. We conduct experiments with five widely used models, i.e., BPR, NeuMF, LightGCN, SASRec and TiSASRec, on four benchmark datasets, i.e., MovieLens-25M, Yelp, Amazon-music, and Amazon-electronic. Our experiment results give an answer "No" to the above question. Users with many historical interactions suffer from relatively poorer recommendations. Users who stay with the system for a shorter time period enjoy better recommendations. Both findings are counter-intuitive. Interestingly, users who have recently interacted with the system, with respect to the time point of the test instance, enjoy better recommendations. The finding on recency applies to all users, regardless of users' loyalty. Our study offers a different perspective to understand recommender accuracy, and our findings could trigger a revisit of recommender model design. The code is available in \url{https://github.com/putatu/recommenderLoyalty.
Region-Adaptive Dense Network for Efficient Motion Deblurring
Authors: Kuldeep Purohit, A. N. Rajagopalan
Date: 2020-04-03 00:00:00
Proceeding: AAAI Conference on Artificial Intelligence 2020 4
Tasks: Deblurring, Decoder

In this paper, we address the problem of dynamic scene deblurring in the presence of motion blur. Restoration of images affected by severe blur necessitates a network design with a large receptive field, which existing networks attempt to achieve through simple increment in the number of generic convolution layers, kernel-size, or the scales at which the image is processed. However, these techniques ignore the non-uniform nature of blur, and they come at the expense of an increase in model size and inference time. We present a new architecture composed of region adaptive dense deformable modules that implicitly discover the spatially varying shifts responsible for non-uniform blur in the input image and learn to modulate the filters. This capability is complemented by a self-attentive module which captures non-local spatial relationships among the intermediate features and enhances the spatially varying processing capability. We incorporate these modules into a densely connected encoder-decoder design which utilizes pre-trained Densenet filters to further improve the performance. Our network facilitates interpretable modeling of the spatially-varying deblurring process while dispensing with multi-scale processing and large filters entirely. Extensive comparisons with prior art on benchmark dynamic scene deblurring datasets clearly demonstrate the superiority of the proposed networks via significant improvements in accuracy and speed, enabling almost real-time deblurring.