pwc-archive/papers-with-abstracts huggingface dataset:
Improving Noise Robustness of Contrastive Speech Representation Learning with Speech Reconstruction
Authors: Heming Wang, Yao Qian, Xiaofei Wang, Yiming Wang, Chengyi Wang, Shujie Liu, Takuya Yoshioka, Jinyu Li, DeLiang Wang
Date: 2021-10-28 00:00:00
Tasks: Automatic Speech Recognition, Automatic Speech Recognition (ASR), Auxiliary Learning, Contrastive Learning, Data Augmentation, Denoising, Noisy Speech Recognition, Representation Learning, Speech Enhancement, speech-recognition, Speech Recognition, Speech Representation Learning

Noise robustness is essential for deploying automatic speech recognition (ASR) systems in real-world environments. One way to reduce the effect of noise interference is to employ a preprocessing module that conducts speech enhancement, and then feed the enhanced speech to an ASR backend. In this work, instead of suppressing background noise with a conventional cascaded pipeline, we employ a noise-robust representation learned by a refined self-supervised framework for noisy speech recognition. We propose to combine a reconstruction module with contrastive learning and perform multi-task continual pre-training on noisy data. The reconstruction module is used for auxiliary learning to improve the noise robustness of the learned representation and thus is not required during inference. Experiments demonstrate the effectiveness of our proposed method. Our model substantially reduces the word error rate (WER) for the synthesized noisy LibriSpeech test sets, and yields around 4.1/7.5% WER reduction on noisy clean/other test sets compared to data augmentation. For the real-world noisy speech from the CHiME-4 challenge (1-channel track), we have obtained the state of the art ASR performance without any denoising front-end. Moreover, we achieve comparable performance to the best supervised approach reported with only 16% of labeled data.
Guided Policy Search for Parameterized Skills using Adverbs
Authors: Benjamin A. Spiegel, George Konidaris
Date: 2021-10-23 00:00:00

We present a method for using adverb phrases to adjust skill parameters via learned adverb-skill groundings. These groundings allow an agent to use adverb feedback provided by a human to directly update a skill policy, in a manner similar to traditional local policy search methods. We show that our method can be used as a drop-in replacement for these policy search methods when dense reward from the environment is not available but human language feedback is. We demonstrate improved sample efficiency over modern policy search methods in two experiments.
Application of the Multi-label Residual Convolutional Neural Network text classifier using Content-Based Routing process
Authors: Tounsi Achraf, Elkefi Safa
Date: 2021-10-19 00:00:00
Tasks: text-classification, Text Classification

In this article, we will present an NLP application in text classifying process using the content-based router. The ultimate goal throughout this article is to predict the event described by a legal ad from the plain text of the ad. This problem is purely a supervised problem that will involve the use of NLP techniques and conventional modeling methodologies through the use of the Multi-label Residual Convolutional Neural Network for text classification. We will explain the approach put in place to solve the problem of classified ads, the difficulties encountered and the experimental results.
Natural Language Processing for Smart Healthcare
Authors: Binggui Zhou, Guanghua Yang, Zheng Shi, Shaodan Ma
Date: 2021-10-19 00:00:00
Tasks: Management

Smart healthcare has achieved significant progress in recent years. Emerging artificial intelligence (AI) technologies enable various smart applications across various healthcare scenarios. As an essential technology powered by AI, natural language processing (NLP) plays a key role in smart healthcare due to its capability of analysing and understanding human language. In this work, we review existing studies that concern NLP for smart healthcare from the perspectives of technique and application. We first elaborate on different NLP approaches and the NLP pipeline for smart healthcare from the technical point of view. Then, in the context of smart healthcare applications employing NLP techniques, we introduce representative smart healthcare scenarios, including clinical practice, hospital management, personal care, public health, and drug development. We further discuss two specific medical issues, i.e., the coronavirus disease 2019 (COVID-19) pandemic and mental health, in which NLP-driven smart healthcare plays an important role. Finally, we discuss the limitations of current works and identify the directions for future works.
BERMo: What can BERT learn from ELMo?
Authors: Sangamesh Kodge, Kaushik Roy
Date: 2021-10-18 00:00:00
Tasks: QQP, SST-2

We propose BERMo, an architectural modification to BERT, which makes predictions based on a hierarchy of surface, syntactic and semantic language features. We use linear combination scheme proposed in Embeddings from Language Models (ELMo) to combine the scaled internal representations from different network depths. Our approach has two-fold benefits: (1) improved gradient flow for the downstream task as every layer has a direct connection to the gradients of the loss function and (2) increased representative power as the model no longer needs to copy the features learned in the shallower layer which are necessary for the downstream task. Further, our model has a negligible parameter overhead as there is a single scalar parameter associated with each layer in the network. Experiments on the probing task from SentEval dataset show that our model performs up to $4.65\%$ better in accuracy than the baseline with an average improvement of $2.67\%$ on the semantic tasks. When subject to compression techniques, we find that our model enables stable pruning for compressing small datasets like SST-2, where the BERT model commonly diverges. We observe that our approach converges $1.67\times$ and $1.15\times$ faster than the baseline on MNLI and QQP tasks from GLUE dataset. Moreover, our results show that our approach can obtain better parameter efficiency for penalty based pruning approaches on QQP task.
Detecting Gender Bias in Transformer-based Models: A Case Study on BERT
Authors: Bingbing Li, Hongwu Peng, Rajat Sainju, Junhuan Yang, Lei Yang, Yueying Liang, Weiwen Jiang, Binghui Wang, Hang Liu, Caiwen Ding
Date: 2021-10-15 00:00:00
Tasks: Bias Detection, Gender Bias Detection

In this paper, we propose a novel gender bias detection method by utilizing attention map for transformer-based models. We 1) give an intuitive gender bias judgement method by comparing the different relation degree between the genders and the occupation according to the attention scores, 2) design a gender bias detector by modifying the attention module, 3) insert the gender bias detector into different positions of the model to present the internal gender bias flow, and 4) draw the consistent gender bias conclusion by scanning the entire Wikipedia, a BERT pretraining dataset. We observe that 1) the attention matrices, Wq and Wk introduce much more gender bias than other modules (including the embedding layer) and 2) the bias degree changes periodically inside of the model (attention matrix Q, K, V, and the remaining part of the attention layer (including the fully-connected layer, the residual connection, and the layer normalization module) enhance the gender bias while the averaged attentions reduces the bias).
Deep Learning for Bias Detection: From Inception to Deployment
Authors: Md Abul Bashar, Richi Nayak, Anjor Kothare, Vishal Sharma, Kesavan Kandadai
Date: 2021-10-12 00:00:00
Tasks: Bias Detection, Deep Learning, Language Modeling, Language Modelling, Transfer Learning

To create a more inclusive workplace, enterprises are actively investing in identifying and eliminating unconscious bias (e.g., gender, race, age, disability, elitism and religion) across their various functions. We propose a deep learning model with a transfer learning based language model to learn from manually tagged documents for automatically identifying bias in enterprise content. We first pretrain a deep learning-based language-model using Wikipedia, then fine tune the model with a large unlabelled data set related with various types of enterprise content. Finally, a linear layer followed by softmax layer is added at the end of the language model and the model is trained on a labelled bias dataset consisting of enterprise content. The trained model is thoroughly evaluated on independent datasets to ensure a general application. We present the proposed method and its deployment detail in a real-world application.
Batch-Softmax Contrastive Loss for Pairwise Sentence Scoring Tasks
Authors: Anton Chernyavskiy, Dmitry Ilvovsky, Pavel Kalinin, Preslav Nakov
Date: 2021-10-10 00:00:00
Proceeding: NAACL 2022 7
Tasks: Representation Learning, Sentence, Sentence Embeddings

The use of contrastive loss for representation learning has become prominent in computer vision, and it is now getting attention in Natural Language Processing (NLP). Here, we explore the idea of using a batch-softmax contrastive loss when fine-tuning large-scale pre-trained transformer models to learn better task-specific sentence embeddings for pairwise sentence scoring tasks. We introduce and study a number of variations in the calculation of the loss as well as in the overall training procedure; in particular, we find that data shuffling can be quite important. Our experimental results show sizable improvements on a number of datasets and pairwise sentence scoring tasks including classification, ranking, and regression. Finally, we offer detailed analysis and discussion, which should be useful for researchers aiming to explore the utility of contrastive loss in NLP.
Generational Frameshifts in Technology: Computer Science and Neurosurgery, The VR Use Case
Authors: Samuel R. Browd, Maya Sharma, Chetan Sharma
Date: 2021-10-08 00:00:00

We are at a unique moment in history where there is a confluence of technologies which will synergistically come together to transform the practice of neurosurgery. These technological transformations will be all-encompassing, including improved tools and methods for intraoperative performance of neurosurgery, scalable solutions for asynchronous neurosurgical training and simulation, as well as broad aggregation of operative data allowing fundamental changes in quality assessment, billing, outcome measures, and dissemination of surgical best practices. The ability to perform surgery more safely and more efficiently while capturing the operative details and parsing each component of the operation will open an entirely new epoch advancing our field and all surgical specialties. The digitization of all components within the operating room will allow us to leverage the various fields within computer and computational science to obtain new insights that will improve care and delivery of the highest quality neurosurgery regardless of location. The democratization of neurosurgery is at hand and will be driven by our development, extraction, and adoption of these tools of the modern world. Virtual reality provides a good example of how consumer-facing technologies are finding a clear role in industry and medicine and serves as a notable example of the confluence of various computer science technologies creating a novel paradigm for scaling human ability and interactions. The authors describe the technology ecosystem that has come and highlight a myriad of computational and data sciences that will be necessary to enable the operating room of the near future.
Cause-effect inference through spectral independence in linear dynamical systems: theoretical foundations
Authors: Michel Besserve, Naji Shajarisales, Dominik Janzing, Bernhard Schölkopf
Date: 2021-10-29 00:00:00
Tasks: Causal Discovery, Causal Inference, Time Series, Time Series Analysis

Distinguishing between cause and effect using time series observational data is a major challenge in many scientific fields. A new perspective has been provided based on the principle of Independence of Causal Mechanisms (ICM), leading to the Spectral Independence Criterion (SIC), postulating that the power spectral density (PSD) of the cause time series is uncorrelated with the squared modulus of the frequency response of the filter generating the effect. Since SIC rests on methods and assumptions in stark contrast with most causal discovery methods for time series, it raises questions regarding what theoretical grounds justify its use. In this paper, we provide answers covering several key aspects. After providing an information theoretic interpretation of SIC, we present an identifiability result that sheds light on the context for which this approach is expected to perform well. We further demonstrate the robustness of SIC to downsampling - an obstacle that can spoil Granger-based inference. Finally, an invariance perspective allows to explore the limitations of the spectral independence assumption and how to generalize it. Overall, these results support the postulate of Spectral Independence is a well grounded leading principle for causal inference based on empirical time series.
Adaptive Importance Sampling meets Mirror Descent: a Bias-variance tradeoff
Authors: Anna Korba, François Portier
Date: 2021-10-29 00:00:00

Adaptive importance sampling is a widely spread Monte Carlo technique that uses a re-weighting strategy to iteratively estimate the so-called target distribution. A major drawback of adaptive importance sampling is the large variance of the weights which is known to badly impact the accuracy of the estimates. This paper investigates a regularization strategy whose basic principle is to raise the importance weights at a certain power. This regularization parameter, that might evolve between zero and one during the algorithm, is shown (i) to balance between the bias and the variance and (ii) to be connected to the mirror descent framework. Using a kernel density estimate to build the sampling policy, the uniform convergence is established under mild conditions. Finally, several practical ways to choose the regularization parameter are discussed and the benefits of the proposed approach are illustrated empirically.
Data-driven Uncertainty Quantification in Computational Human Head Models
Authors: Kshitiz Upadhyay, Dimitris G. Giovanis, Ahmed Alshareef, Andrew K. Knutsen, Curtis L. Johnson, Aaron Carass, Philip V. Bayly, Michael D. Shields, K. T. Ramesh
Date: 2021-10-29 00:00:00
Tasks: Density Estimation, Dimensionality Reduction, Uncertainty Quantification

Computational models of the human head are promising tools for estimating the impact-induced response of brain, and thus play an important role in the prediction of traumatic brain injury. Modern biofidelic head model simulations are associated with very high computational cost, and high-dimensional inputs and outputs, which limits the applicability of traditional uncertainty quantification (UQ) methods on these systems. In this study, a two-stage, data-driven manifold learning-based framework is proposed for UQ of computational head models. This framework is demonstrated on a 2D subject-specific head model, where the goal is to quantify uncertainty in the simulated strain fields (i.e., output), given variability in the material properties of different brain substructures (i.e., input). In the first stage, a data-driven method based on multi-dimensional Gaussian kernel-density estimation and diffusion maps is used to generate realizations of the input random vector directly from the available data. Computational simulations of a small number of realizations provide input-output pairs for training data-driven surrogate models in the second stage. The surrogate models employ nonlinear dimensionality reduction using Grassmannian diffusion maps, Gaussian process regression to create a low-cost mapping between the input random vector and the reduced solution space, and geometric harmonics models for mapping between the reduced space and the Grassmann manifold. It is demonstrated that the surrogate models provide highly accurate approximations of the computational model while significantly reducing the computational cost. Monte Carlo simulations of the surrogate models are used for uncertainty propagation. UQ of strain fields highlight significant spatial variation in model uncertainty, and reveal key differences in uncertainty among commonly used strain-based brain injury predictor variables.
Optimal prediction for kernel-based semi-functional linear regression
Authors: Keli Guo, Jun Fan, Lixing Zhu
Date: 2021-10-29 00:00:00
Tasks: Prediction, regression

In this paper, we establish minimax optimal rates of convergence for prediction in a semi-functional linear model that consists of a functional component and a less smooth nonparametric component. Our results reveal that the smoother functional component can be learned with the minimax rate as if the nonparametric component were known. More specifically, a double-penalized least squares method is adopted to estimate both the functional and nonparametric components within the framework of reproducing kernel Hilbert spaces. By virtue of the representer theorem, an efficient algorithm that requires no iterations is proposed to solve the corresponding optimization problem, where the regularization parameters are selected by the generalized cross validation criterion. Numerical studies are provided to demonstrate the effectiveness of the method and to verify the theoretical analysis.
Dense Hierarchical Retrieval for Open-Domain Question Answering
Authors: Ye Liu, Kazuma Hashimoto, Yingbo Zhou, Semih Yavuz, Caiming Xiong, Philip S. Yu
Date: 2021-10-28 00:00:00
Proceeding: Findings (EMNLP) 2021 11
Tasks: Open-Domain Question Answering, Question Answering, Retrieval, Text Retrieval

Dense neural text retrieval has achieved promising results on open-domain Question Answering (QA), where latent representations of questions and passages are exploited for maximum inner product search in the retrieval process. However, current dense retrievers require splitting documents into short passages that usually contain local, partial, and sometimes biased context, and highly depend on the splitting process. As a consequence, it may yield inaccurate and misleading hidden representations, thus deteriorating the final retrieval result. In this work, we propose Dense Hierarchical Retrieval (DHR), a hierarchical framework that can generate accurate dense representations of passages by utilizing both macroscopic semantics in the document and microscopic semantics specific to each passage. Specifically, a document-level retriever first identifies relevant documents, among which relevant passages are then retrieved by a passage-level retriever. The ranking of the retrieved passages will be further calibrated by examining the document-level relevance. In addition, hierarchical title structure and two negative sampling strategies (i.e., In-Doc and In-Sec negatives) are investigated. We apply DHR to large-scale open-domain QA datasets. DHR significantly outperforms the original dense passage retriever and helps an end-to-end QA system outperform the strong baselines on multiple open-domain QA benchmarks.
Decision Attentive Regularization to Improve Simultaneous Speech Translation Systems
Authors: Mohd Abbas Zaidi, Beomseok Lee, Sangha Kim, Chanwoo Kim
Date: 2021-10-13 00:00:00
Tasks: Sentence, Simultaneous Speech-to-Text Translation, Speech-to-Text, Speech-to-Text Translation, Translation

Simultaneous translation systems start producing the output while processing the partial source sentence in the incoming input stream. These systems need to decide when to read more input and when to write the output. These decisions depend on the structure of source/target language and the information contained in the partial input sequence. Hence, read/write decision policy remains the same across different input modalities, i.e., speech and text. This motivates us to leverage the text transcripts corresponding to the speech input for improving simultaneous speech-to-text translation (SimulST). We propose Decision Attentive Regularization (DAR) to improve the decision policy of SimulST systems by using the simultaneous text-to-text translation (SimulMT) task. We also extend several techniques from the offline speech translation domain to explore the role of SimulMT task in improving SimulST performance. Overall, we achieve 34.66% / 4.5 BLEU improvement over the baseline model across different latency regimes for the MuST-C English-German (EnDe) SimulST task.
Anticipation-driven Adaptive Architecture for Assisted Living
Authors: Mihai Nadin, Asma Naz
Date: 2021-10-15 00:00:00

Anticipatory expression underlies human performance. Medical conditions and, especially, aging result in diminished anticipatory action. In order to mitigate the loss, means for engaging still available resources (capabilities) can be provided. In particular, anticipation-driven adaptive environments could be beneficial in medical care, as well as in assisted living for those seeking such assistance. These adaptive environments are conceived to be individualized and individualizable, in order to stimulate independent action instead of creating dependencies.
Low-Complexity Geometrical Shaping for 4D Modulation Formats via Amplitude Coding
Authors: Bin Chen, Wei Ling, Yunus Can Gültekin, Yi Lei, Chigo Okonkwo, Alex Alvarado
Date: 2021-10-29 00:00:00

Signal shaping is vital to approach Shannon's capacity, yet it is challenging to implement at very high speeds. For example, probabilistic shaping often requires arithmetic coding to realize the target distribution. Geometric shaping requires look-up tables to store the constellation points. In this paper, we propose a four-dimensional amplitude coding (4D-AC) geometrical shaper architecture. The proposed architecture can generate in real time geometrically shaped 4D formats via simple logic circuit operations and two conventional quadrature amplitude modulation (QAM) modulators. This paper describes the 4D-AC used in generating approximated versions of two recently proposed 4D orthant symmetric modulation formats with spectral efficiencies of 6 bit/4D-sym and 7 bit/4D-sym, respectively. Numerical results show losses below 0.05 dB when compared against the baseline formats.
Angle Diversity Trasmitter For High Speed Data Center Uplink Communications
Authors: Abrar S. Alhazmi, Sanaa H. Mohamed, Osama Z. Alsulami, T. E. H. El-Gorashi, Jaafar M. H. Elmirghani
Date: 2021-10-29 00:00:00
Tasks: Diversity, Vocal Bursts Intensity Prediction

This paper proposes an uplink optical wireless communication (OWC) link design that can be used by data centers to support communication in spine and leaf architectures between the top of rack leaf switches and large spine switches whose access points are mounted in the ceiling. The use of optical wireless links reduces cabling and allows easy reconfigurability for example when data centres expand. We consider three racks in a data center where each rack contains an Angle Diversity Transmitter (ADT) positioned on the top of the rack to realize the uplink function of a top-of-the-rack (ToR) or a leaf switch. Four receivers are considered to be installed on the ceiling where each is connected to a spine switch. Two types of optical receivers are studied which are a Wide Field-of-View Receiver (WFOVR) and an Angle Diversity Receiver (ADR). The performance of the proposed system is evaluated when the links run at data rates higher than 19 Gbps. The results indicate that the proposed approach achieves excellent performance using simple On-Off Keying (OOK)