pwc-archive/papers-with-abstracts huggingface dataset:
Wasserstein Task Embedding for Measuring Task Similarities
Authors: Xinran Liu, Yikun Bai, Yuzhe Lu, Andrea Soltoggio, Soheil Kolouri
Date: 2022-08-24 00:00:00
Tasks: Meta-Learning

Measuring similarities between different tasks is critical in a broad spectrum of machine learning problems, including transfer, multi-task, continual, and meta-learning. Most current approaches to measuring task similarities are architecture-dependent: 1) relying on pre-trained models, or 2) training networks on tasks and using forward transfer as a proxy for task similarity. In this paper, we leverage the optimal transport theory and define a novel task embedding for supervised classification that is model-agnostic, training-free, and capable of handling (partially) disjoint label sets. In short, given a dataset with ground-truth labels, we perform a label embedding through multi-dimensional scaling and concatenate dataset samples with their corresponding label embeddings. Then, we define the distance between two datasets as the 2-Wasserstein distance between their updated samples. Lastly, we leverage the 2-Wasserstein embedding framework to embed tasks into a vector space in which the Euclidean distance between the embedded points approximates the proposed 2-Wasserstein distance between tasks. We show that the proposed embedding leads to a significantly faster comparison of tasks compared to related approaches like the Optimal Transport Dataset Distance (OTDD). Furthermore, we demonstrate the effectiveness of our proposed embedding through various numerical experiments and show statistically significant correlations between our proposed distance and the forward and backward transfer between tasks.
Prediction of the energy and exergy performance of F135 PW100 turbofan engine via deep learning
Authors: Mohammadreza Sabzehali, Amir Hossein Rabieeb, Mahdi Alibeigia, Amir Mosavi
Date: 2022-08-24 00:00:00

In the present study, the effects of flight-Mach number, flight altitude, fuel types, and intake air temperature on thrust specific fuel consumption, thrust, intake air mass flow rate, thermal and propulsive efficiecies, as well as the exergetic efficiency and the exergy destruction rate in F135 PW100 engine are investigated. Based on the results obtained in the first phase, to model the thermodynamic performance of the aforementioned engine cycle, Flight-Mach number and flight altitude are considered to be 2.5 and 30,000 m, respectively; due to the operational advantage of supersonic flying at high altitude flight conditions, and the higher thrust of hydrogen fuel. Accordingly, in the second phase, taking into account the mentioned flight conditions, an intelligent model has been obtained to predict output parameters (i.e., thrust, thrust specific fuel consumption, and overall exergetic efficiency) using the deep learning method. In the attained deep neural model, the pressure ratio of the high-pressure turbine, fan pressure ratio, turbine inlet temperature, intake air temperature, and bypass ratio are considered input parameters. The provided datasets are randomly divided into two sets: the first one contains 6079 samples for model training and the second set contains 1520 samples for testing. In particular, the Adam optimization algorithm, the cost function of the mean square error, and the active function of rectified linear unit are used to train the network. The results show that the error percentage of the deep neural model is equal to 5.02%, 1.43%, and 2.92% to predict thrust, thrust specific fuel consumption, and overall exergetic efficiency, respectively, which indicates the success of the attained model in estimating the output parameters of the present problem.
Ontology-Driven Self-Supervision for Adverse Childhood Experiences Identification Using Social Media Datasets
Authors: Jinge Wu, Rowena Smith, Honghan Wu
Date: 2022-08-24 00:00:00

Adverse Childhood Experiences (ACEs) are defined as a collection of highly stressful, and potentially traumatic, events or circumstances that occur throughout childhood and/or adolescence. They have been shown to be associated with increased risks of mental health diseases or other abnormal behaviours in later lives. However, the identification of ACEs from textual data with Natural Language Processing (NLP) is challenging because (a) there are no NLP ready ACE ontologies; (b) there are few resources available for machine learning, necessitating the data annotation from clinical experts; (c) costly annotations by domain experts and large number of documents for supporting large machine learning models. In this paper, we present an ontology-driven self-supervised approach (derive concept embeddings using an auto-encoder from baseline NLP results) for producing a publicly available resource that would support large-scale machine learning (e.g., training transformer based large language models) on social media corpus. This resource as well as the proposed approach are aimed to facilitate the community in training transferable NLP models for effectively surfacing ACEs in low-resource scenarios like NLP on clinical notes within Electronic Health Records. The resource including a list of ACE ontology terms, ACE concept embeddings and the NLP annotated corpus is available at https://github.com/knowlab/ACE-NLP.
Lifelong Learning for Neural powered Mixed Integer Programming
Authors: Sahil Manchanda, Sayan Ranu
Date: 2022-08-24 00:00:00
Tasks: Graph Attention, Knowledge Distillation, Lifelong learning

Mixed Integer programs (MIPs) are typically solved by the Branch-and-Bound algorithm. Recently, Learning to imitate fast approximations of the expert strong branching heuristic has gained attention due to its success in reducing the running time for solving MIPs. However, existing learning-to-branch methods assume that the entire training data is available in a single session of training. This assumption is often not true, and if the training data is supplied in continual fashion over time, existing techniques suffer from catastrophic forgetting. In this work, we study the hitherto unexplored paradigm of Lifelong Learning to Branch on Mixed Integer Programs. To mitigate catastrophic forgetting, we propose LIMIP, which is powered by the idea of modeling an MIP instance in the form of a bipartite graph, which we map to an embedding space using a bipartite Graph Attention Network. This rich embedding space avoids catastrophic forgetting through the application of knowledge distillation and elastic weight consolidation, wherein we learn the parameters key towards retaining efficacy and are therefore protected from significant drift. We evaluate LIMIP on a series of NP-hard problems and establish that in comparison to existing baselines, LIMIP is up to 50% better when confronted with lifelong learning.
Reduced-PINN: An Integration-Based Physics-Informed Neural Networks for Stiff ODEs
Authors: Pouyan Nasiri, Roozbeh Dargazany
Date: 2022-08-23 00:00:00

Physics-informed neural networks (PINNs) have recently received much attention due to their capabilities in solving both forward and inverse problems. For training a deep neural network associated with a PINN, one typically constructs a total loss function using a weighted sum of different loss terms and then tries to minimize that. This approach often becomes problematic for solving stiff equations since it cannot consider adaptive increments. Many studies reported the poor performance of the PINN and its challenges in simulating stiff chemical active issues with administering conditions of stiff ordinary differential conditions (ODEs). Studies show that stiffness is the primary cause of the failure of the PINN in simulating stiff kinetic systems. Here, we address this issue by proposing a reduced weak-form of the loss function, which led to a new PINN architecture, further named as Reduced-PINN, that utilizes a reduced-order integration method to enable the PINN to solve stiff chemical kinetics. The proposed Reduced-PINN can be applied to various reaction-diffusion systems involving stiff dynamics. To this end, we transform initial value problems (IVPs) to their equivalent integral forms and solve the resulting integral equations using physics-informed neural networks. In our derived integral-based optimization process, there is only one term without explicitly incorporating loss terms associated with ordinary differential equation (ODE) and initial conditions (ICs). To illustrate the capabilities of Reduced-PINN, we used it to simulate multiple stiff/mild second-order ODEs. We show that Reduced-PINN captures the solution accurately for a stiff scalar ODE. We also validated the Reduced-PINN against a stiff system of linear ODEs.
Training a T5 Using Lab-sized Resources
Authors: Manuel R. Ciosici, Leon Derczynski
Date: 2022-08-25 00:00:00
Tasks: Language Modeling, Language Modelling, Large Language Model

Training large neural language models on large datasets is resource- and time-intensive. These requirements create a barrier to entry, where those with fewer resources cannot build competitive models. This paper presents various techniques for making it possible to (a) train a large language model using resources that a modest research lab might have, and (b) train it in a reasonable amount of time. We provide concrete recommendations for practitioners, which we illustrate with a case study: a T5 model for Danish, the first for this language.
Kencorpus: A Kenyan Language Corpus of Swahili, Dholuo and Luhya for Natural Language Processing Tasks
Authors: Barack Wanjawa, Lilian Wanzare, Florence Indede, Owen McOnyango, Edward Ombui, Lawrence Muchemi
Date: 2022-08-25 00:00:00
Tasks: Machine Translation, Part-Of-Speech Tagging, POS, Question Answering, Speech-to-Text, Translation

Indigenous African languages are categorized as under-served in Natural Language Processing. They therefore experience poor digital inclusivity and information access. The processing challenge with such languages has been how to use machine learning and deep learning models without the requisite data. The Kencorpus project intends to bridge this gap by collecting and storing text and speech data that is good enough for data-driven solutions in applications such as machine translation, question answering and transcription in multilingual communities. The Kencorpus dataset is a text and speech corpus for three languages predominantly spoken in Kenya: Swahili, Dholuo and Luhya. Data collection was done by researchers from communities, schools, media, and publishers. The Kencorpus' dataset has a collection of 5,594 items - 4,442 texts (5.6M words) and 1,152 speech files (177hrs). Based on this data, Part of Speech tagging sets for Dholuo and Luhya (50,000 and 93,000 words respectively) were developed. We developed 7,537 Question-Answer pairs for Swahili and created a text translation set of 13,400 sentences from Dholuo and Luhya into Swahili. The datasets are useful for downstream machine learning tasks such as model training and translation. We also developed two proof of concept systems: for Kiswahili speech-to-text and machine learning system for Question Answering task, with results of 18.87% word error rate and 80% Exact Match (EM) respectively. These initial results give great promise to the usability of Kencorpus to the machine learning community. Kencorpus is one of few public domain corpora for these three low resource languages and forms a basis of learning and sharing experiences for similar works especially for low resource languages.
A logical theory for strong and weak ontic necessities in branching time
Authors: Fengkui Ju
Date: 2022-08-25 00:00:00

Ontic necessities are those modalities universally quantifying over domains of ontic possibilities, whose ``existence'' is independent of our knowledge. An ontic necessity, called the weak ontic necessity, causes interesting questions. An example for it is ``I should be dead by now''. A feature of this necessity is whether it holds at a state has nothing to do with whether its prejacent holds at the state. Is there a weak epistemic necessity expressed by ``should''? Is there a strong ontic necessity expressed by ``must''? How do we make sense of the strong and weak ontic necessities formally? In this paper, we do the following work. Firstly, we recognize strong/weak ontic/epistemic necessities and give our general ideas about them. Secondly, we present a complete logical theory for the strong and weak ontic necessities in branching time. This theory is based on the following approach. The weak ontic necessity quantifies over a domain of expected timelines, determined by the agent's system of ontic rules. The strong ontic necessity quantifies over a domain of accepted timelines, determined by undefeatable ontic rules.
A logical theory for conditional weak ontic necessity based on context update
Authors: Fengkui Ju
Date: 2022-08-25 00:00:00

Weak ontic necessity is the ontic necessity expressed by ``should'' or ``ought to'' in English. An example of it is ``I should be dead by now''. A feature of this necessity is whether it holds does not have anything to do with whether its prejacent holds. In this paper, we present a logical theory for conditional weak ontic necessity based on context update. A context is a set of ordered defaults, determining expected possible states of the present world. Sentences are evaluated with respect to contexts. When evaluating the conditional weak ontic necessity with respect to a context, we first update the context with the antecedent, then check whether the consequent holds with respect to the updated context. The logic is complete. Our theory combines premise semantics and update semantics for conditionals.
Addressing Token Uniformity in Transformers via Singular Value Transformation
Authors: Hanqi Yan, Lin Gui, Wenjie Li, Yulan He
Date: 2022-08-24 00:00:00
Tasks: Semantic Textual Similarity

Token uniformity is commonly observed in transformer-based models, in which different tokens share a large proportion of similar information after going through stacked multiple self-attention layers in a transformer. In this paper, we propose to use the distribution of singular values of outputs of each transformer layer to characterise the phenomenon of token uniformity and empirically illustrate that a less skewed singular value distribution can alleviate the `token uniformity' problem. Base on our observations, we define several desirable properties of singular value distributions and propose a novel transformation function for updating the singular values. We show that apart from alleviating token uniformity, the transformation function should preserve the local neighbourhood structure in the original embedding space. Our proposed singular value transformation function is applied to a range of transformer-based language models such as BERT, ALBERT, RoBERTa and DistilBERT, and improved performance is observed in semantic textual similarity evaluation and a range of GLUE tasks. Our source code is available at https://github.com/hanqi-qi/tokenUni.git.
Alleviating Search Bias in Bayesian Evolutionary Optimization with Many Heterogeneous Objectives
Authors: Xilu Wang, Yaochu Jin, Sebastian Schmitt, Markus Olhofer
Date: 2022-08-25 00:00:00
Tasks: Diversity

Multi-objective optimization problems whose objectives have different evaluation costs are commonly seen in the real world. Such problems are now known as multi-objective optimization problems with heterogeneous objectives (HE-MOPs). So far, however, only a few studies have been reported to address HE-MOPs, and most of them focus on bi-objective problems with one fast objective and one slow objective. In this work, we aim to deal with HE-MOPs having more than two black-box and heterogeneous objectives. To this end, we develop a multi-objective Bayesian evolutionary optimization approach to HE-MOPs by exploiting the different data sets on the cheap and expensive objectives in HE-MOPs to alleviate the search bias caused by the heterogeneous evaluation costs for evaluating different objectives. To make the best use of two different training data sets, one with solutions evaluated on all objectives and the other with those only evaluated on the fast objectives, two separate Gaussian process models are constructed. In addition, a new acquisition function that mitigates search bias towards the fast objectives is suggested, thereby achieving a balance between convergence and diversity. We demonstrate the effectiveness of the proposed algorithm by testing it on widely used multi-/many-objective benchmark problems whose objectives are assumed to be heterogeneously expensive.
CMOS-based area-and-power-efficient neuron and synapse circuits for time-domain analog spiking neural networks
Authors: Xiangyu Chen, Zolboo Byambadorj, Takeaki Yajima, Hisashi Inoue, Isao H. Inoue, Tetsuya Iizuka
Date: 2022-08-25 00:00:00

Conventional neural structures tend to communicate through analog quantities such as currents or voltages, however, as CMOS devices shrink and supply voltages decrease, the dynamic range of voltage/current-domain analog circuits becomes narrower, the available margin becomes smaller, and noise immunity decreases. More than that, the use of operational amplifiers (op-amps) and continuous-time or clocked comparators in conventional designs leads to high energy consumption and large chip area, which would be detrimental to building spiking neural networks. In view of this, we propose a neural structure for generating and transmitting time-domain signals, including a neuron module, a synapse module, and two weight modules. The proposed neural structure is driven by a leakage current of MOS transistors and uses an inverter-based comparator to realize a firing function, thus providing higher energy and area efficiency compared to conventional designs. The proposed neural structure is fabricated using TSMC 65 nm CMOS technology. The proposed neuron and synapse occupy the area of 127 {\mu}m^{ 2} and 231 {\mu}m^{ 2}, respectively, while achieving millisecond time constants. Actual chip measurements show that the proposed structure implements the temporal signal communication function with millisecond time constants, which is a critical step toward hardware reservoir computing for human-computer interaction. Simulation results of the spiking-neural network for reservoir computing with the behavioral model of the proposed neural structure demonstrate the learning function.
Adversarial Bayesian Simulation
Authors: Yuexi Wang, Veronika Ročková
Date: 2022-08-25 00:00:00

In the absence of explicit or tractable likelihoods, Bayesians often resort to approximate Bayesian computation (ABC) for inference. Our work bridges ABC with deep neural implicit samplers based on generative adversarial networks (GANs) and adversarial variational Bayes. Both ABC and GANs compare aspects of observed and fake data to simulate from posteriors and likelihoods, respectively. We develop a Bayesian GAN (B-GAN) sampler that directly targets the posterior by solving an adversarial optimization problem. B-GAN is driven by a deterministic mapping learned on the ABC reference by conditional GANs. Once the mapping has been trained, iid posterior samples are obtained by filtering noise at a negligible additional cost. We propose two post-processing local refinements using (1) data-driven proposals with importance reweighting, and (2) variational Bayes. We support our findings with frequentist-Bayesian results, showing that the typical total variation distance between the true and approximate posteriors converges to zero for certain neural network generators and discriminators. Our findings on simulated data show highly competitive performance relative to some of the most recent likelihood-free posterior simulators.
Development of Sleep State Trend (SST), a bedside measure of neonatal sleep state fluctuations based on single EEG channels
Authors: Saeed Montazeri Moghadam, Päivi Nevalainen, Nathan J. Stevenson, Sampsa Vanhatalo
Date: 2022-08-25 00:00:00
Tasks: EEG, Electroencephalogram (EEG)

Objective: To develop and validate an automated method for bedside monitoring of sleep state fluctuations in neonatal intensive care units. Methods: A deep learning -based algorithm was designed and trained using 53 EEG recordings from a long-term (a)EEG monitoring in 30 near-term neonates. The results were validated using an external dataset from 30 polysomnography recordings. In addition to training and validating a single EEG channel quiet sleep detector, we constructed Sleep State Trend (SST), a bedside-ready means for visualizing classifier outputs. Results: The accuracy of quiet sleep detection in the training data was 90%, and the accuracy was comparable (85-86%) in all bipolar derivations available from the 4-electrode recordings. The algorithm generalized well to an external dataset, showing 81% overall accuracy despite different signal derivations. SST allowed an intuitive, clear visualization of the classifier output. Conclusions: Fluctuations in sleep states can be detected at high fidelity from a single EEG channel, and the results can be visualized as a transparent and intuitive trend in the bedside monitors. Significance: The Sleep State Trend (SST) may provide caregivers a real-time view of sleep state fluctuations and its cyclicity.
Analysis of COVID-19 in India using vaccine epidemic model incorporating vaccine effectiveness and herd immunity
Authors: V. R. Saiprasad, R. Gopal, V. K. Chandrasekar, M. Lakshmanan
Date: 2022-08-25 00:00:00

COVID-19 will be a continuous threat to human population despite having a few vaccines at hand until we reach the endemic state through natural herd immunity and total immunization through universal vaccination. However, the vaccine acts as a practical tool for reducing the massive public health problem and the emerging economic consequences that the continuing COVID -19 epidemic is causing worldwide, while the vaccine efficacy wanes. In this work, we propose and analyze an epidemic model of Susceptible-Exposed-Infected-Recovered-Vaccinated (SEIRV) population taking into account the rate of vaccination and vaccine waning. The dynamics of the model has been investigated, and the condition for a disease-free endemic equilibrium state is obtained. Further, the analysis is extended to study the COVID-19 spread in India by considering the availability of vaccines and the related critical parameters such as vaccination rate, vaccine efficacy and waning of vaccine's impact on deciding the emerging fate of this epidemic. We have also discussed the conditions for herd immunity due to vaccinated individuals among the people. Our results highlight the importance of vaccines, the effectiveness of booster vaccination in protecting people from infection, and their importance in epidemic and pandemic modelling.
Rapid, remote and low-cost finger vasculature mapping for heart rate monitoring
Authors: Akhil Kallepalli, David B James, Mark A Richardson
Date: 2022-08-15 00:00:00
Tasks: Diagnostic

Today's diagnostics include devices such as pulse oximeters, blood pressure monitors, and temperature measurements. These devices provide vital information to medical personnel when making treatment decisions. Drawing inspiration from the fundamental utility of pulse oximeters, we present a methodology for a robust low-cost approach to imaging subsurface vasculature and monitoring heart rate. The approach uses off-the-shelf equipment, set up in free space without physical contact and exploits the nature of the interaction between light at near-infrared wavelengths with tissue. Image processing algorithms extract heart rate information from the snapshot and video sequence captured at a stand-off distance. The method can be applied in a room with ambient light and remains robust to scenarios comparable to medical situations. This research sets the platform for future diagnostic devices based on imaging systems and algorithms for non-contact point-of-care investigations.