pwc-archive/papers-with-abstracts huggingface dataset:
Self-critical Sequence Training for Automatic Speech Recognition
Authors: Chen Chen, Yuchen Hu, Nana Hou, Xiaofeng Qi, Heqing Zou, Eng Siong Chng
Date: 2022-04-13 00:00:00
Tasks: Automatic Speech Recognition, Automatic Speech Recognition (ASR), Reinforcement Learning (RL), speech-recognition, Speech Recognition

Although automatic speech recognition (ASR) task has gained remarkable success by sequence-to-sequence models, there are two main mismatches between its training and testing that might lead to performance degradation: 1) The typically used cross-entropy criterion aims to maximize log-likelihood of the training data, while the performance is evaluated by word error rate (WER), not log-likelihood; 2) The teacher-forcing method leads to the dependence on ground truth during training, which means that model has never been exposed to its own prediction before testing. In this paper, we propose an optimization method called self-critical sequence training (SCST) to make the training procedure much closer to the testing phase. As a reinforcement learning (RL) based method, SCST utilizes a customized reward function to associate the training criterion and WER. Furthermore, it removes the reliance on teacher-forcing and harmonizes the model with respect to its inference procedure. We conducted experiments on both clean and noisy speech datasets, and the results show that the proposed SCST respectively achieves 8.7% and 7.8% relative improvements over the baseline in terms of WER.
Can Question Rewriting Help Conversational Question Answering?
Authors: Etsuko Ishii, Yan Xu, Samuel Cahyawijaya, Bryan Wilie
Date: 2022-04-13 00:00:00
Proceeding: insights (ACL) 2022 5
Tasks: Conversational Question Answering, Question Answering, Question Rewriting, reinforcement-learning, Reinforcement Learning (RL)

Question rewriting (QR) is a subtask of conversational question answering (CQA) aiming to ease the challenges of understanding dependencies among dialogue history by reformulating questions in a self-contained form. Despite seeming plausible, little evidence is available to justify QR as a mitigation method for CQA. To verify the effectiveness of QR in CQA, we investigate a reinforcement learning approach that integrates QR and CQA tasks and does not require corresponding QR datasets for targeted CQA. We find, however, that the RL method is on par with the end-to-end baseline. We provide an analysis of the failure and describe the difficulty of exploiting QR for CQA.
Probing for Constituency Structure in Neural Language Models
Authors: David Arps, Younes Samih, Laura Kallmeyer, Hassan Sajjad
Date: 2022-04-13 00:00:00
Tasks: Diagnostic

In this paper, we investigate to which extent contextual neural language models (LMs) implicitly learn syntactic structure. More concretely, we focus on constituent structure as represented in the Penn Treebank (PTB). Using standard probing techniques based on diagnostic classifiers, we assess the accuracy of representing constituents of different categories within the neuron activations of a LM such as RoBERTa. In order to make sure that our probe focuses on syntactic knowledge and not on implicit semantic generalizations, we also experiment on a PTB version that is obtained by randomly replacing constituents with each other while keeping syntactic structure, i.e., a semantically ill-formed but syntactically well-formed version of the PTB. We find that 4 pretrained transfomer LMs obtain high performance on our probing tasks even on manipulated data, suggesting that semantic and syntactic knowledge in their representations can be separated and that constituency information is in fact learned by the LM. Moreover, we show that a complete constituency tree can be linearly separated from LM representations.
Efficient Cluster-Based k-Nearest-Neighbor Machine Translation
Authors: Dexin Wang, Kai Fan, Boxing Chen, Deyi Xiong
Date: 2022-04-13 00:00:00
Proceeding: ACL 2022 5
Tasks: Contrastive Learning, Domain Adaptation, Machine Translation, NMT, Retrieval, Translation

k-Nearest-Neighbor Machine Translation (kNN-MT) has been recently proposed as a non-parametric solution for domain adaptation in neural machine translation (NMT). It aims to alleviate the performance degradation of advanced MT systems in translating out-of-domain sentences by coordinating with an additional token-level feature-based retrieval module constructed from in-domain data. Previous studies have already demonstrated that non-parametric NMT is even superior to models fine-tuned on out-of-domain data. In spite of this success, kNN retrieval is at the expense of high latency, in particular for large datastores. To make it practical, in this paper, we explore a more efficient kNN-MT and propose to use clustering to improve the retrieval efficiency. Concretely, we first propose a cluster-based Compact Network for feature reduction in a contrastive learning manner to compress context features into 90+% lower dimensional vectors. We then suggest a cluster-based pruning solution to filter out 10%-40% redundant nodes in large datastores while retaining translation quality. Our proposed methods achieve better or comparable performance while reducing up to 57% inference latency against the advanced non-parametric MT model on several machine translation benchmarks. Experimental results indicate that the proposed methods maintain the most useful information of the original datastore and the Compact Network shows good generalization on unseen domains.
HIT at SemEval-2022 Task 2: Pre-trained Language Model for Idioms Detection
Authors: Zheng Chu, Ziqing Yang, Yiming Cui, Zhigang Chen, Ming Liu
Date: 2022-04-13 00:00:00
Proceeding: SemEval (NAACL) 2022 7
Tasks: Language Modeling, Language Modelling, Sentence, Sentence Embedding, Sentence-Embedding, Task 2

The same multi-word expressions may have different meanings in different sentences. They can be mainly divided into two categories, which are literal meaning and idiomatic meaning. Non-contextual-based methods perform poorly on this problem, and we need contextual embedding to understand the idiomatic meaning of multi-word expressions correctly. We use a pre-trained language model, which can provide a context-aware sentence embedding, to detect whether multi-word expression in the sentence is idiomatic usage.
Impossible Triangle: What's Next for Pre-trained Language Models?
Authors: Chenguang Zhu, Michael Zeng
Date: 2022-04-13 00:00:00
Tasks: Data Augmentation, Few-Shot Learning, Generalized Few-Shot Learning, Knowledge Distillation, Prompt Learning

Recent development of large-scale pre-trained language models (PLM) have significantly improved the capability of models in various NLP tasks, in terms of performance after task-specific fine-tuning and zero-shot / few-shot learning. However, many of such models come with a dauntingly huge size that few institutions can afford to pre-train, fine-tune or even deploy, while moderate-sized models usually lack strong generalized few-shot learning capabilities. In this paper, we first elaborate the current obstacles of using PLM models in terms of the Impossible Triangle: 1) moderate model size, 2) state-of-the-art few-shot learning capability, and 3) state-of-the-art fine-tuning capability. We argue that all existing PLM models lack one or more properties from the Impossible Triangle. To remedy these missing properties of PLMs, various techniques have been proposed, such as knowledge distillation, data augmentation and prompt learning, which inevitably brings additional work to the application of PLMs in real scenarios. We then offer insights into future research directions of PLMs to achieve the Impossible Triangle, and break down the task into several key phases.
ASQA: Factoid Questions Meet Long-Form Answers
Authors: Ivan Stelmakh, Yi Luan, Bhuwan Dhingra, Ming-Wei Chang
Date: 2022-04-12 00:00:00
Tasks: Form, Question Answering

An abundance of datasets and availability of reliable evaluation metrics have resulted in strong progress in factoid question answering (QA). This progress, however, does not easily transfer to the task of long-form QA, where the goal is to answer questions that require in-depth explanations. The hurdles include (i) a lack of high-quality data, and (ii) the absence of a well-defined notion of the answer's quality. In this work, we address these problems by (i) releasing a novel dataset and a task that we call ASQA (Answer Summaries for Questions which are Ambiguous); and (ii) proposing a reliable metric for measuring performance on ASQA. Our task focuses on factoid questions that are ambiguous, that is, have different correct answers depending on interpretation. Answers to ambiguous questions should synthesize factual information from multiple sources into a long-form summary that resolves the ambiguity. In contrast to existing long-form QA tasks (such as ELI5), ASQA admits a clear notion of correctness: a user faced with a good summary should be able to answer different interpretations of the original ambiguous question. We use this notion of correctness to define an automated metric of performance for ASQA. Our analysis demonstrates an agreement between this metric and human judgments, and reveals a considerable gap between human performance and strong baselines.
CUNI-KIT System for Simultaneous Speech Translation Task at IWSLT 2022
Authors: Peter Polák, Ngoc-Quan Ngoc, Tuan-Nam Nguyen, Danni Liu, Carlos Mullov, Jan Niehues, Ondřej Bojar, Alexander Waibel
Date: 2022-04-12 00:00:00
Proceeding: IWSLT (ACL) 2022 5
Tasks: Translation

In this paper, we describe our submission to the Simultaneous Speech Translation at IWSLT 2022. We explore strategies to utilize an offline model in a simultaneous setting without the need to modify the original model. In our experiments, we show that our onlinization algorithm is almost on par with the offline setting while being $3\times$ faster than offline in terms of latency on the test set. We also show that the onlinized offline model outperforms the best IWSLT2021 simultaneous system in medium and high latency regimes and is almost on par in the low latency regime. We make our system publicly available.
Switching between Numerical Black-box Optimization Algorithms with Warm-starting Policies
Authors: Dominik Schröder, Diederick Vermetten, Hao Wang, Carola Doerr, Thomas Bäck
Date: 2022-04-13 00:00:00

When solving optimization problems with black-box approaches, the algorithms gather valuable information about the problem instance during the optimization process. This information is used to adjust the distributions from which new solution candidates are sampled. In fact, a key objective in evolutionary computation is to identify the most effective ways to collect and exploit instance knowledge. However, while considerable work is devoted to adjusting hyper-parameters of black-box optimization algorithms on the fly or exchanging some of its modular components, we barely know how to effectively switch between different black-box optimization algorithms. In this work, we build on the recent study of Vermetten et al. [GECCO 2020], who presented a data-driven approach to investigate promising switches between pairs of algorithms for numerical black-box optimization. We replicate their approach with a portfolio of five algorithms and investigate whether the predicted performance gains are realized when executing the most promising switches. Our results suggest that with a single switch between two algorithms, we outperform the best static choice among the five algorithms on 48 out of the 120 considered problem instances, the 24 BBOB functions in five different dimensions. We also show that for switching between BFGS and CMA-ES, a proper warm-starting of the parameters is crucial to realize high-performance gains. Lastly, with a sensitivity analysis, we find the actual performance gain per run is largely affected by the switching point, and in some cases, the switching point yielding the best actual performance differs from the one computed from the theoretical gain.
Evolving Modular Soft Robots without Explicit Inter-Module Communication using Local Self-Attention
Authors: Federico Pigozzi, Yujin Tang, Eric Medvet, David Ha
Date: 2022-04-13 00:00:00
Tasks: Inductive Bias

Modularity in robotics holds great potential. In principle, modular robots can be disassembled and reassembled in different robots, and possibly perform new tasks. Nevertheless, actually exploiting modularity is yet an unsolved problem: controllers usually rely on inter-module communication, a practical requirement that makes modules not perfectly interchangeable and thus limits their flexibility. Here, we focus on Voxel-based Soft Robots (VSRs), aggregations of mechanically identical elastic blocks. We use the same neural controller inside each voxel, but without any inter-voxel communication, hence enabling ideal conditions for modularity: modules are all equal and interchangeable. We optimize the parameters of the neural controller-shared among the voxels-by evolutionary computation. Crucially, we use a local self-attention mechanism inside the controller to overcome the absence of inter-module communication channels, thus enabling our robots to truly be driven by the collective intelligence of their modules. We show experimentally that the evolved robots are effective in the task of locomotion: thanks to self-attention, instances of the same controller embodied in the same robot can focus on different inputs. We also find that the evolved controllers generalize to unseen morphologies, after a short fine-tuning, suggesting that an inductive bias related to the task arises from true modularity.
Population Diversity Leads to Short Running Times of Lexicase Selection
Authors: Thomas Helmuth, Johannes Lengler, William La Cava
Date: 2022-04-13 00:00:00
Tasks: Diversity, Program Synthesis

In this paper we investigate why the running time of lexicase parent selection is empirically much lower than its worst-case bound of O(N*C). We define a measure of population diversity and prove that high diversity leads to low running times O(N + C) of lexicase selection. We then show empirically that genetic programming populations evolved under lexicase selection are diverse for several program synthesis problems, and explore the resulting differences in running time bounds.
Trajectory-based Algorithm Selection with Warm-starting
Authors: Anja Jankovic, Diederick Vermetten, Ana Kostovska, Jacob de Nobel, Tome Eftimov, Carola Doerr
Date: 2022-04-13 00:00:00
Tasks: regression

Landscape-aware algorithm selection approaches have so far mostly been relying on landscape feature extraction as a preprocessing step, independent of the execution of optimization algorithms in the portfolio. This introduces a significant overhead in computational cost for many practical applications, as features are extracted and computed via sampling and evaluating the problem instance at hand, similarly to what the optimization algorithm would perform anyway within its search trajectory. As suggested in Jankovic et al. (EvoAPPs 2021), trajectory-based algorithm selection circumvents the problem of costly feature extraction by computing landscape features from points that a solver sampled and evaluated during the optimization process. Features computed in this manner are used to train algorithm performance regression models, upon which a per-run algorithm selector is then built. In this work, we apply the trajectory-based approach onto a portfolio of five algorithms. We study the quality and accuracy of performance regression and algorithm selection models in the scenario of predicting different algorithm performances after a fixed budget of function evaluations. We rely on landscape features of the problem instance computed using one portion of the aforementioned budget of the same function evaluations. Moreover, we consider the possibility of switching between the solvers once, which requires them to be warm-started, i.e. when we switch, the second solver continues the optimization process already being initialized appropriately by making use of the information collected by the first solver. In this new context, we show promising performance of the trajectory-based per-run algorithm selection with warm-starting.
Sapinet: A sparse event-based spatiotemporal oscillator for learning in the wild
Authors: Ayon Borthakur
Date: 2022-04-13 00:00:00
Tasks: Classification, Denoising

We introduce Sapinet -- a spike timing (event)-based multilayer neural network for \textit{learning in the wild} -- that is: one-shot online learning of multiple inputs without catastrophic forgetting, and without the need for data-specific hyperparameter retuning. Key features of Sapinet include data regularization, model scaling, data classification, and denoising. The model also supports stimulus similarity mapping. We propose a systematic method to tune the network for performance. We studied the model performance on different levels of odor similarity, gaussian and impulse noise. Sapinet achieved high classification accuracies on standard machine olfaction datasets without the requirement of fine tuning for a specific dataset.
Swarm Modelling with Dynamic Mode Decomposition
Authors: Emma Hansen, Steven L. Brunton, Zhuoyuan Song
Date: 2022-04-08 00:00:00

Modelling biological or engineering swarms is challenging due to the inherently high dimension of the system, despite the often low-dimensional emergent dynamics. Most existing swarm modelling approaches are based on first principles and often result in swarm-specific parameterizations that do not generalize to a broad range of applications. In this work, we apply a purely data-driven method to (1) learn local interactions of homogeneous swarms through observation data and to (2) generate similar swarming behaviour using the learned model. In particular, a modified version of dynamic mode decomposition with control, called swarmDMD, is developed and tested on the canonical Vicsek swarm model. The goal is to use swarmDMD to learn inter-agent interactions that give rise to the observed swarm behaviour. We show that swarmDMD can faithfully reconstruct the swarm dynamics, and the model learned by swarmDMD provides a short prediction window for data extrapolation with a trade-off between prediction accuracy and prediction horizon. We also provide a comprehensive analysis on the efficacy of different observation data types on the modelling, where we find that inter-agent distance yields the most accurate models. We believe the proposed swarmDMD approach will be useful for studying multi-agent systems found in biology, physics, and engineering.
Features of the Earth's seasonal hydroclimate: Characterizations and comparisons across the Koppen-Geiger climates and across continents
Authors: Georgia Papacharalampous, Hristos Tyralis, Yannis Markonis, Petr Maca, Martin Hanel
Date: 2022-04-13 00:00:00
Tasks: Time Series, Time Series Analysis

Detailed investigations of time series features across climates, continents and variable types can progress our understanding and modelling ability of the Earth's hydroclimate and its dynamics. They can also improve our comprehension of the climate classification systems appearing in their core. Still, such investigations for seasonal hydroclimatic temporal dependence, variability and change are currently missing from the literature. Herein, we propose and apply at the global scale a methodological framework for filling this specific gap. We analyse over 13 000 earth-observed quarterly temperature, precipitation and river flow time series. We adopt the Koppen-Geiger climate classification system and define continental-scale geographical regions for conducting upon them seasonal hydroclimatic feature summaries. The analyses rely on three sample autocorrelation features, a temporal variation feature, a spectral entropy feature, a Hurst feature, a trend strength feature and a seasonality strength feature. We find notable differences to characterize the magnitudes of these features across the various Koppen-Geiger climate classes, as well as between continental-scale geographical regions. We, therefore, deem that the consideration of the comparative summaries could be beneficial in water resources engineering contexts. Lastly, we apply explainable machine learning to compare the investigated features with respect to how informative they are in distinguishing either the main Koppen-Geiger climates or the continental-scale regions. In this regard, the sample autocorrelation, temporal variation and seasonality strength features are found to be more informative than the spectral entropy, Hurst and trend strength features at the seasonal time scale.
Time series features for supporting hydrometeorological explorations and predictions in ungauged locations using large datasets
Authors: Georgia Papacharalampous, Hristos Tyralis
Date: 2022-04-13 00:00:00
Tasks: regression, Time Series, Time Series Analysis

Regression-based frameworks for streamflow regionalization are built around catchment attributes that traditionally originate from catchment hydrology, flood frequency analysis and their interplay. In this work, we deviated from this traditional path by formulating and extensively investigating the first regression-based streamflow regionalization frameworks that largely emerge from general-purpose time series features for data science and, more precisely, from a large variety of such features. We focused on 28 features that included (partial) autocorrelation, entropy, temporal variation, seasonality, trend, lumpiness, stability, nonlinearity, linearity, spikiness, curvature and others. We estimated these features for daily temperature, precipitation and streamflow time series from 511 catchments, and then merged them within regionalization contexts with traditional topographic, land cover, soil and geologic attributes. Precipitation and temperature features (e.g., the spectral entropy, seasonality strength and lag-1 autocorrelation of the precipitation time series, and the stability and trend strength of the temperature time series) were found to be useful predictors of many streamflow features. The same applies to traditional attributes, such as the catchment mean elevation. Relationships between predictor and dependent variables were also revealed, while the spectral entropy, the seasonality strength and several autocorrelation features of the streamflow time series were found to be more regionalizable than others.
Utilizing variational autoencoders in the Bayesian inverse problem of photoacoustic tomography
Authors: Teemu Sahlström, Tanja Tarvainen
Date: 2022-04-13 00:00:00
Tasks: BIG-bench Machine Learning, Image Reconstruction

There has been an increasing interest in utilizing machine learning methods in inverse problems and imaging. Most of the work has, however, concentrated on image reconstruction problems, and the number of studies regarding the full solution of the inverse problem is limited. In this work, we study a machine learning based approach for the Bayesian inverse problem of photoacoustic tomography. We develop an approach for estimating the posterior distribution in photoacoustic tomography using an approach based on the variational autoencoder. The approach is evaluated with numerical simulations and compared to the solution of the inverse problem using a Bayesian approach.