pwc-archive/papers-with-abstracts huggingface dataset:
Discovering Governing Equations by Machine Learning implemented with Invariance
Authors: Chao Chen, Xiaowei Jin, Hui Li
Date: 2022-03-29 00:00:00
Tasks: BIG-bench Machine Learning

The partial differential equation (PDE) plays a significantly important role in many fields of science and engineering. The conventional case of the derivation of PDE mainly relies on first principles and empirical observation. However, the development of machine learning technology allows us to mine potential control equations from the massive amounts of stored data in a fresh way. Although there has been considerable progress in the data-driven discovery of PDE, the extant literature mostly focuses on the improvements of discovery methods, without substantial breakthroughs in the discovery process itself, including the principles for the construction of candidates and how to incorporate physical priors. In this paper, through rigorous derivation of formulas, novel physically enhanced machining learning discovery methods for control equations: GSNN (Galileo Symbolic Neural Network) and LSNN (Lorentz Symbolic Neural Network) are firstly proposed based on Galileo invariance and Lorentz invariance respectively, setting forth guidelines for building the candidates of discovering equations. The adoption of mandatory embedding of physical constraints is fundamentally different from PINN in the form of the loss function, thus ensuring that the designed Neural Network strictly obeys the physical prior of invariance and enhancing the interpretability of the network. By comparing the results with PDE-NET in numerical experiments of Burgers equation and Sine-Gordon equation, it shows that the method presented in this study has better accuracy, parsimony, and interpretability.
Disentangling speech from surroundings with neural embeddings
Authors: Ahmed Omran, Neil Zeghidour, Zalán Borsos, Félix de Chaumont Quitry, Malcolm Slaney, Marco Tagliasacchi
Date: 2022-03-29 00:00:00
Tasks: Attribute

We present a method to separate speech signals from noisy environments in the embedding space of a neural audio codec. We introduce a new training procedure that allows our model to produce structured encodings of audio waveforms given by embedding vectors, where one part of the embedding vector represents the speech signal, and the rest represent the environment. We achieve this by partitioning the embeddings of different input waveforms and training the model to faithfully reconstruct audio from mixed partitions, thereby ensuring each partition encodes a separate audio attribute. As use cases, we demonstrate the separation of speech from background noise or from reverberation characteristics. Our method also allows for targeted adjustments of the audio output characteristics.
Quantum compiling with a variational instruction set for accurate and fast quantum computing
Authors: Ying Lu, Peng-Fei Zhou, Shao-Ming Fei, Shi-Ju Ran
Date: 2022-03-29 00:00:00

The quantum instruction set (QIS) is defined as the quantum gates that are physically realizable by controlling the qubits in quantum hardware. Compiling quantum circuits into the product of the gates in a properly defined QIS is a fundamental step in quantum computing. We here propose the quantum variational instruction set (QuVIS) formed by flexibly designed multi-qubit gates for higher speed and accuracy of quantum computing. The controlling of qubits for realizing the gates in a QuVIS is variationally achieved using the fine-grained time optimization algorithm. Significant reductions in both the error accumulation and time cost are demonstrated in realizing the swaps of multiple qubits and quantum Fourier transformations, compared with the compiling by a standard QIS such as the quantum microinstruction set (QuMIS, formed by several one- and two-qubit gates including one-qubit rotations and controlled-NOT gates). With the same requirement on quantum hardware, the time cost for QuVIS is reduced to less than one half of that for QuMIS. Simultaneously, the error is suppressed algebraically as the depth of the compiled circuit is reduced. As a general compiling approach with high flexibility and efficiency, QuVIS can be defined for different quantum circuits and be adapted to the quantum hardware with different interactions.
Invariance Learning based on Label Hierarchy
Authors: Shoji Toyota, Kenji Fukumizu
Date: 2022-03-29 00:00:00

Deep Neural Networks inherit spurious correlations embedded in training data and hence may fail to predict desired labels on unseen domains (or environments), which have different distributions from the domain used in training. Invariance Learning (IL) has been developed recently to overcome this shortcoming; using training data in many domains, IL estimates such a predictor that is invariant to a change of domain. However, the requirement of training data in multiple domains is a strong restriction of IL, since it often needs high annotation cost. We propose a novel IL framework to overcome this problem. Assuming the availability of data from multiple domains for a higher level of classification task, for which the labeling cost is low, we estimate an invariant predictor for the target classification task with training data in a single domain. Additionally, we propose two cross-validation methods for selecting hyperparameters of invariance regularization to solve the issue of hyperparameter selection, which has not been handled properly in existing IL methods. The effectiveness of the proposed framework, including the cross-validation, is demonstrated empirically, and the correctness of the hyperparameter selection is proved under some conditions.
Learning neural audio features without supervision
Authors: Sarthak Yadav, Neil Zeghidour
Date: 2022-03-29 00:00:00
Tasks: Audio Classification, Self-Supervised Learning

Deep audio classification, traditionally cast as training a deep neural network on top of mel-filterbanks in a supervised fashion, has recently benefited from two independent lines of work. The first one explores "learnable frontends", i.e., neural modules that produce a learnable time-frequency representation, to overcome limitations of fixed features. The second one uses self-supervised learning to leverage unprecedented scales of pre-training data. In this work, we study the feasibility of combining both approaches, i.e., pre-training learnable frontend jointly with the main architecture for downstream classification. First, we show that pretraining two previously proposed frontends (SincNet and LEAF) on Audioset drastically improves linear-probe performance over fixed mel-filterbanks, suggesting that learnable time-frequency representations can benefit self-supervised pre-training even more than supervised training. Surprisingly, randomly initialized learnable filterbanks outperform mel-scaled initialization in the self-supervised setting, a counter-intuitive result that questions the appropriateness of strong priors when designing learnable filters. Through exploratory analysis of the learned frontend components, we uncover crucial differences in properties of these frontends when used in a supervised and self-supervised setting, especially the affinity of self-supervised filters to diverge significantly from the mel-scale to model a broader range of frequencies.
Achieving Guidance in Applied Machine Learning through Software Engineering Techniques
Authors: Lars Reimann, Günter Kniesel-Wünsche
Date: 2022-03-29 00:00:00
Tasks: BIG-bench Machine Learning

Development of machine learning (ML) applications is hard. Producing successful applications requires, among others, being deeply familiar with a variety of complex and quickly evolving application programming interfaces (APIs). It is therefore critical to understand what prevents developers from learning these APIs, using them properly at development time, and understanding what went wrong when it comes to debugging. We look at the (lack of) guidance that currently used development environments and ML APIs provide to developers of ML applications, contrast these with software engineering best practices, and identify gaps in the current state of the art. We show that current ML tools fall short of fulfilling some basic software engineering gold standards and point out ways in which software engineering concepts, tools and techniques need to be extended and adapted to match the special needs of ML application development. Our findings point out ample opportunities for research on ML-specific software engineering.
Improving the Learnability of Machine Learning APIs by Semi-Automated API Wrapping
Authors: Lars Reimann, Günter Kniesel-Wünsche
Date: 2022-03-29 00:00:00
Tasks: BIG-bench Machine Learning

A major hurdle for students and professional software developers who want to enter the world of machine learning (ML), is mastering not just the scientific background but also the available ML APIs. Therefore, we address the challenge of creating APIs that are easy to learn and use, especially by novices. However, it is not clear how this can be achieved without compromising expressiveness. We investigate this problem for scikit-learn, a widely used ML API. In this paper, we analyze its use by the Kaggle community, identifying unused and apparently useless parts of the API that can be eliminated without affecting client programs. In addition, we discuss usability issues in the remaining parts, propose related design improvements and show how they can be implemented by semi-automated wrapping of the existing third-party API.
Over-the-Air Federated Learning via Second-Order Optimization
Authors: Peng Yang, Yuning Jiang, Ting Wang, Yong Zhou, Yuanming Shi, Colin N. Jones
Date: 2022-03-29 00:00:00
Tasks: Federated Learning

Federated learning (FL) is a promising learning paradigm that can tackle the increasingly prominent isolated data islands problem while keeping users' data locally with privacy and security guarantees. However, FL could result in task-oriented data traffic flows over wireless networks with limited radio resources. To design communication-efficient FL, most of the existing studies employ the first-order federated optimization approach that has a slow convergence rate. This however results in excessive communication rounds for local model updates between the edge devices and edge server. To address this issue, in this paper, we instead propose a novel over-the-air second-order federated optimization algorithm to simultaneously reduce the communication rounds and enable low-latency global model aggregation. This is achieved by exploiting the waveform superposition property of a multi-access channel to implement the distributed second-order optimization algorithm over wireless networks. The convergence behavior of the proposed algorithm is further characterized, which reveals a linear-quadratic convergence rate with an accumulative error term in each iteration. We thus propose a system optimization approach to minimize the accumulated error gap by joint device selection and beamforming design. Numerical results demonstrate the system and communication efficiency compared with the state-of-the-art approaches.
Gaussian Control Barrier Functions : A Non-Parametric Paradigm to Safety
Authors: Mouhyemen Khan, Tatsuya Ibuki, Abhijit Chatterjee
Date: 2022-03-29 00:00:00
Tasks: Collision Avoidance, Gaussian Processes

Inspired by the success of control barrier functions (CBFs) in addressing safety, and the rise of data-driven techniques for modeling functions, we propose a non-parametric approach for online synthesis of CBFs using Gaussian Processes (GPs). Mathematical constructs such as CBFs have achieved safety by designing a candidate function a priori. However, designing such a candidate function can be challenging. A practical example of such a setting would be to design a CBF in a disaster recovery scenario where safe and navigable regions need to be determined. The decision boundary for safety in such an example is unknown and cannot be designed a priori. In our approach, we work with safety samples or observations to construct the CBF online by assuming a flexible GP prior on these samples, and term our formulation as a Gaussian CBF. GPs have favorable properties, in addition to being non-parametric, such as analytical tractability and robust uncertainty estimation. This allows realizing the posterior components with high safety guarantees by incorporating variance estimation, while also computing associated partial derivatives in closed-form to achieve safe control. Moreover, the synthesized safety function from our approach allows changing the corresponding safe set arbitrarily based on the data, thus allowing non-convex safe sets. We validate our approach experimentally on a quadrotor by demonstrating safe control for fixed but arbitrary safe sets and collision avoidance where the safe set is constructed online. Finally, we juxtapose Gaussian CBFs with regular CBFs in the presence of noisy states to highlight its flexibility and robustness to noise. The experiment video can be seen at: https://youtu.be/HX6uokvCiGk.
Graph similarity learning for change-point detection in dynamic networks
Authors: Deborah Sulem, Henry Kenlay, Mihai Cucuringu, Xiaowen Dong
Date: 2022-03-29 00:00:00
Tasks: Change Point Detection, Fraud Detection, Graph Neural Network, Graph Similarity, Temporal Sequences

Dynamic networks are ubiquitous for modelling sequential graph-structured data, e.g., brain connectome, population flows and messages exchanges. In this work, we consider dynamic networks that are temporal sequences of graph snapshots, and aim at detecting abrupt changes in their structure. This task is often termed network change-point detection and has numerous applications, such as fraud detection or physical motion monitoring. Leveraging a graph neural network model, we design a method to perform online network change-point detection that can adapt to the specific network domain and localise changes with no delay. The main novelty of our method is to use a siamese graph neural network architecture for learning a data-driven graph similarity function, which allows to effectively compare the current graph and its recent history. Importantly, our method does not require prior knowledge on the network generative distribution and is agnostic to the type of change-points; moreover, it can be applied to a large variety of networks, that include for instance edge weights and node attributes. We show on synthetic and real data that our method enjoys a number of benefits: it is able to learn an adequate graph similarity function for performing online network change-point detection in diverse types of change-point settings, and requires a shorter data history to detect changes than most existing state-of-the-art baselines.
Machine Composition of Korean Music via Topological Data Analysis and Artificial Neural Network
Authors: Mai Lan Tran, Dongjin Lee, Jae-Hun Jung
Date: 2022-03-29 00:00:00
Tasks: Topological Data Analysis

Common AI music composition algorithms based on artificial neural networks are to train a machine by feeding a large number of music pieces and create artificial neural networks that can produce music similar to the input music data. This approach is a blackbox optimization, that is, the underlying composition algorithm is, in general, not known to users. In this paper, we present a way of machine composition that trains a machine the composition principle embedded in the given music data instead of directly feeding music pieces. We propose this approach by using the concept of {\color{black}{Overlap}} matrix proposed in \cite{TPJ}. In \cite{TPJ}, a type of Korean music, so-called the {\it Dodeuri} music such as Suyeonjangjigok has been analyzed using topological data analysis (TDA), particularly using persistent homology. As the raw music data is not suitable for TDA analysis, the music data is first reconstructed as a graph. The node of the graph is defined as a two-dimensional vector composed of the pitch and duration of each music note. The edge between two nodes is created when those nodes appear consecutively in the music flow. Distance is defined based on the frequency of such appearances. Through TDA on the constructed graph, a unique set of cycles is found for the given music. In \cite{TPJ}, the new concept of the {\it {\color{black}{Overlap}} matrix} has been proposed, which visualizes how those cycles are interconnected over the music flow, in a matrix form. In this paper, we explain how we use the {\color{black}{Overlap}} matrix for machine composition. The {\color{black}{Overlap}} matrix makes it possible to compose a new music piece algorithmically and also provide a seed music towards the desired artificial neural network. In this paper, we use the {\it Dodeuri} music and explain detailed steps.
Protein language models trained on multiple sequence alignments learn phylogenetic relationships
Authors: Umberto Lupo, Damiano Sgarbossa, Anne-Florence Bitbol
Date: 2022-03-29 00:00:00
Tasks: Prediction

Self-supervised neural language models with attention have recently been applied to biological sequence data, advancing structure, function and mutational effect prediction. Some protein language models, including MSA Transformer and AlphaFold's EvoFormer, take multiple sequence alignments (MSAs) of evolutionarily related proteins as inputs. Simple combinations of MSA Transformer's row attentions have led to state-of-the-art unsupervised structural contact prediction. We demonstrate that similarly simple, and universal, combinations of MSA Transformer's column attentions strongly correlate with Hamming distances between sequences in MSAs. Therefore, MSA-based language models encode detailed phylogenetic relationships. We further show that these models can separate coevolutionary signals encoding functional and structural constraints from phylogenetic correlations reflecting historical contingency. To assess this, we generate synthetic MSAs, either without or with phylogeny, from Potts models trained on natural MSAs. We find that unsupervised contact prediction is substantially more resilient to phylogenetic noise when using MSA Transformer versus inferred Potts models.
On Reinforcement Learning, Effect Handlers, and the State Monad
Authors: Ugo Dal Lago, Francesco Gavazzo, Alexis Ghyselen
Date: 2022-03-29 00:00:00
Tasks: Decision Making, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

We study the algebraic effects and handlers as a way to support decision-making abstractions in functional programs, whereas a user can ask a learning algorithm to resolve choices without implementing the underlying selection mechanism, and give a feedback by way of rewards. Differently from some recently proposed approach to the problem based on the selection monad [Abadi and Plotkin, LICS 2021], we express the underlying intelligence as a reinforcement learning algorithm implemented as a set of handlers for some of these algebraic operations, including those for choices and rewards. We show how we can in practice use algebraic operations and handlers -- as available in the programming language EFF -- to clearly separate the learning algorithm from its environment, thus allowing for a good level of modularity. We then show how the host language can be taken as a lambda-calculus with handlers, this way showing what the essential linguistic features are. We conclude by hinting at how type and effect systems could ensure safety properties, at the same time pointing at some directions for further work.
Physics-informed deep-learning applications to experimental fluid mechanics
Authors: Hamidreza Eivazi, Yuning Wang, Ricardo Vinuesa
Date: 2022-03-29 00:00:00
Tasks: Data Augmentation, Deep Learning, Super-Resolution

High-resolution reconstruction of flow-field data from low-resolution and noisy measurements is of interest due to the prevalence of such problems in experimental fluid mechanics, where the measurement data are in general sparse, incomplete and noisy. Deep-learning approaches have been shown suitable for such super-resolution tasks. However, a high number of high-resolution examples is needed, which may not be available for many cases. Moreover, the obtained predictions may lack in complying with the physical principles, e.g. mass and momentum conservation. Physics-informed deep learning provides frameworks for integrating data and physical laws for learning. In this study, we apply physics-informed neural networks (PINNs) for super-resolution of flow-field data both in time and space from a limited set of noisy measurements without having any high-resolution reference data. Our objective is to obtain a continuous solution of the problem, providing a physically-consistent prediction at any point in the solution domain. We demonstrate the applicability of PINNs for the super-resolution of flow-field data in time and space through three canonical cases: Burgers' equation, two-dimensional vortex shedding behind a circular cylinder and the minimal turbulent channel flow. The robustness of the models is also investigated by adding synthetic Gaussian noise. Furthermore, we show the capabilities of PINNs to improve the resolution and reduce the noise in a real experimental dataset consisting of hot-wire-anemometry measurements. Our results show the adequate capabilities of PINNs in the context of data augmentation for experiments in fluid mechanics.
Pareto Set Learning for Neural Multi-objective Combinatorial Optimization
Authors: Xi Lin, Zhiyuan Yang, Qingfu Zhang
Date: 2022-03-29 00:00:00
Tasks: Combinatorial Optimization, Traveling Salesman Problem

Multiobjective combinatorial optimization (MOCO) problems can be found in many real-world applications. However, exactly solving these problems would be very challenging, particularly when they are NP-hard. Many handcrafted heuristic methods have been proposed to tackle different MOCO problems over the past decades. In this work, we generalize the idea of neural combinatorial optimization, and develop a learning-based approach to approximate the whole Pareto set for a given MOCO problem without further search procedure. We propose a single preference-conditioned model to directly generate approximate Pareto solutions for any trade-off preference, and design an efficient multiobjective reinforcement learning algorithm to train this model. Our proposed method can be treated as a learning-based extension for the widely-used decomposition-based multiobjective evolutionary algorithm (MOEA/D). It uses a single model to accommodate all the possible preferences, whereas other methods use a finite number of solutions to approximate the Pareto set. Experimental results show that our proposed method significantly outperforms some other methods on the multiobjective traveling salesman problem, multiobjective vehicle routing problem, and multiobjective knapsack problem in terms of solution quality, speed, and model efficiency.
Spoofing-Aware Speaker Verification by Multi-Level Fusion
Authors: Haibin Wu, Lingwei Meng, Jiawen Kang, Jinchao Li, Xu Li, Xixin Wu, Hung-Yi Lee, Helen Meng
Date: 2022-03-29 00:00:00
Tasks: Speaker Verification

Recently, many novel techniques have been introduced to deal with spoofing attacks, and achieve promising countermeasure (CM) performances. However, these works only take the stand-alone CM models into account. Nowadays, a spoofing aware speaker verification (SASV) challenge which aims to facilitate the research of integrated CM and ASV models, arguing that jointly optimizing CM and ASV models will lead to better performance, is taking place. In this paper, we propose a novel multi-model and multi-level fusion strategy to tackle the SASV task. Compared with purely scoring fusion and embedding fusion methods, this framework first utilizes embeddings from CM models, propagating CM embeddings into a CM block to obtain a CM score. In the second-level fusion, the CM score and ASV scores directly from ASV systems will be concatenated into a prediction block for the final decision. As a result, the best single fusion system has achieved the SASV-EER of 0.97% on the evaluation set. Then by ensembling the top-5 fusion systems, the final SASV-EER reached 0.89%.