pwc-archive/papers-with-abstracts huggingface dataset:
Extended Graph Temporal Classification for Multi-Speaker End-to-End ASR
Authors: Xuankai Chang, Niko Moritz, Takaaki Hori, Shinji Watanabe, Jonathan Le Roux
Date: 2022-03-01 00:00:00
Tasks: Automatic Speech Recognition, Automatic Speech Recognition (ASR), Pseudo Label, speech-recognition, Speech Recognition

Graph-based temporal classification (GTC), a generalized form of the connectionist temporal classification loss, was recently proposed to improve automatic speech recognition (ASR) systems using graph-based supervision. For example, GTC was first used to encode an N-best list of pseudo-label sequences into a graph for semi-supervised learning. In this paper, we propose an extension of GTC to model the posteriors of both labels and label transitions by a neural network, which can be applied to a wider range of tasks. As an example application, we use the extended GTC (GTC-e) for the multi-speaker speech recognition task. The transcriptions and speaker information of multi-speaker speech are represented by a graph, where the speaker information is associated with the transitions and ASR outputs with the nodes. Using GTC-e, multi-speaker ASR modelling becomes very similar to single-speaker ASR modeling, in that tokens by multiple speakers are recognized as a single merged sequence in chronological order. For evaluation, we perform experiments on a simulated multi-speaker speech dataset derived from LibriSpeech, obtaining promising results with performance close to classical benchmarks for the task.
RMBR: A Regularized Minimum Bayes Risk Reranking Framework for Machine Translation
Authors: Yidan Zhang, Yu Wan, Dayiheng Liu, Baosong Yang, Zhenan He
Date: 2022-03-01 00:00:00
Tasks: Machine Translation, NMT, Reranking, Translation

Beam search is the most widely used decoding method for neural machine translation (NMT). In practice, the top-1 candidate with the highest log-probability among the n candidates is selected as the preferred one. However, this top-1 candidate may not be the best overall translation among the n-best list. Recently, Minimum Bayes Risk (MBR) decoding has been proposed to improve the quality for NMT, which seeks for a consensus translation that is closest on average to other candidates from the n-best list. We argue that MBR still suffers from the following problems: The utility function only considers the lexical-level similarity between candidates; The expected utility considers the entire n-best list which is time-consuming and inadequate candidates in the tail list may hurt the performance; Only the relationship between candidates is considered. To solve these issues, we design a regularized MBR reranking framework (RMBR), which considers semantic-based similarity and computes the expected utility for each candidate by truncating the list. We expect the proposed framework to further consider the translation quality and model uncertainty of each candidate. Thus the proposed quality regularizer and uncertainty regularizer are incorporated into the framework. Extensive experiments on multiple translation tasks demonstrate the effectiveness of our method.
Paper Plain: Making Medical Research Papers Approachable to Healthcare Consumers with Natural Language Processing
Authors: Tal August, Lucy Lu Wang, Jonathan Bragg, Marti A. Hearst, Andrew Head, Kyle Lo
Date: 2022-02-28 00:00:00

When seeking information not covered in patient-friendly documents, like medical pamphlets, healthcare consumers may turn to the research literature. Reading medical papers, however, can be a challenging experience. To improve access to medical papers, we introduce a novel interactive interface-Paper Plain-with four features powered by natural language processing: definitions of unfamiliar terms, in-situ plain language section summaries, a collection of key questions that guide readers to answering passages, and plain language summaries of the answering passages. We evaluate Paper Plain, finding that participants who use Paper Plain have an easier time reading and understanding research papers without a loss in paper comprehension compared to those who use a typical PDF reader. Altogether, the study results suggest that guiding readers to relevant passages and providing plain language summaries, or "gists," alongside the original paper content can make reading medical papers easier and give readers more confidence to approach these papers.
Decisions over Sequences
Authors: Bhavook Bhardwaj, Siddharth Chatterjee
Date: 2022-02-28 00:00:00
Tasks: Decision Making

This paper introduces a class of objects called decision rules that map infinite sequences of alternatives to a decision space. These objects can be used to model situations where a decision maker encounters alternatives in a sequence such as receiving recommendations. Within the class of decision rules, we study natural subclasses: stopping and uniform stopping rules. Our main result establishes the equivalence of these two subclasses of decision rules. Next, we introduce the notion of computability of decision rules using Turing machines and show that computable rules can be implemented using a simpler computational device: a finite automaton. We further show that computability of choice rules -- an important subclass of decision rules -- is implied by their continuity with respect to a natural topology. Finally, we introduce some natural heuristics in this framework and provide their behavioral characterization.
An Empirical Study on Explanations in Out-of-Domain Settings
Authors: George Chrysostomou, Nikolaos Aletras
Date: 2022-02-28 00:00:00
Proceeding: ACL 2022 5

Recent work in Natural Language Processing has focused on developing approaches that extract faithful explanations, either via identifying the most important tokens in the input (i.e. post-hoc explanations) or by designing inherently faithful models that first select the most important tokens and then use them to predict the correct label (i.e. select-then-predict models). Currently, these approaches are largely evaluated on in-domain settings. Yet, little is known about how post-hoc explanations and inherently faithful models perform in out-of-domain settings. In this paper, we conduct an extensive empirical study that examines: (1) the out-of-domain faithfulness of post-hoc explanations, generated by five feature attribution methods; and (2) the out-of-domain performance of two inherently faithful models over six datasets. Contrary to our expectations, results show that in many cases out-of-domain post-hoc explanation faithfulness measured by sufficiency and comprehensiveness is higher compared to in-domain. We find this misleading and suggest using a random baseline as a yardstick for evaluating post-hoc explanation faithfulness. Our findings also show that select-then predict models demonstrate comparable predictive performance in out-of-domain settings to full-text trained models.
Towards Reducing the Need for Speech Training Data To Build Spoken Language Understanding Systems
Authors: Samuel Thomas, Hong-Kwang J. Kuo, Brian Kingsbury, George Saon
Date: 2022-02-26 00:00:00
Tasks: Spoken Language Understanding

The lack of speech data annotated with labels required for spoken language understanding (SLU) is often a major hurdle in building end-to-end (E2E) systems that can directly process speech inputs. In contrast, large amounts of text data with suitable labels are usually available. In this paper, we propose a novel text representation and training methodology that allows E2E SLU systems to be effectively constructed using these text resources. With very limited amounts of additional speech, we show that these models can be further improved to perform at levels close to similar systems built on the full speech datasets. The efficacy of our proposed approach is demonstrated on both intent and entity tasks using three different SLU datasets. With text-only training, the proposed system achieves up to 90% of the performance possible with full speech training. With just an additional 10% of speech data, these models significantly improve further to 97% of full performance.
Do Transformers know symbolic rules, and would we know if they did?
Authors: Tommi Gröndahl, Yujia Guo, N. Asokan
Date: 2022-02-19 00:00:00
Tasks: Zero-shot Generalization

To improve the explainability of leading Transformer networks used in NLP, it is important to tease apart genuine symbolic rules from merely associative input-output patterns. However, we identify several inconsistencies in how ``symbolicity'' has been construed in recent NLP literature. To mitigate this problem, we propose two criteria to be the most relevant, one pertaining to a system's internal architecture and the other to the dissociation between abstract rules and specific input identities. From this perspective, we critically examine prior work on the symbolic capacities of Transformers, and deem the results to be fundamentally inconclusive for reasons inherent in experiment design. We further maintain that there is no simple fix to this problem, since it arises -- to an extent -- in all end-to-end settings. Nonetheless, we emphasize the need for more robust evaluation of whether non-symbolic explanations exist for success in seemingly symbolic tasks. To facilitate this, we experiment on four sequence modelling tasks on the T5 Transformer in two experiment settings: zero-shot generalization, and generalization across class-specific vocabularies flipped between the training and test set. We observe that T5's generalization is markedly stronger in sequence-to-sequence tasks than in comparable classification tasks. Based on this, we propose a thus far overlooked analysis, where the Transformer itself does not need to be symbolic to be part of a symbolic architecture as the processor, operating on the input and output as external memory components.
A hardware-software co-design approach to minimize the use of memory resources in multi-core neuromorphic processors
Authors: Vanessa R. C. Leite, Zhe Su, Adrian M. Whatley, Giacomo Indiveri
Date: 2022-03-01 00:00:00

Both in electronics and biology, physical implementations of neural networks have severe energy and memory constraints. We propose a hardware-software co-design approach for minimizing the use of memory resources in multi-core neuromorphic processors, by taking inspiration from biological neural networks. We use this approach to design new routing schemes optimized for small-world networks and to provide guidelines for designing novel application-specific multi-core neuromorphic chips. Starting from the hierarchical routing scheme proposed, we present a hardware-aware placement algorithm that optimizes the allocation of resources for arbitrary network models. We validate the algorithm with a canonical small-world network and present preliminary results for other networks derived from it.
Making use of supercomputers in financial machine learning
Authors: Philippe Cotte, Pierre Lagier, Vincent Margot, Christophe Geissler
Date: 2022-03-01 00:00:00
Tasks: BIG-bench Machine Learning

This article is the result of a collaboration between Fujitsu and Advestis. This collaboration aims at refactoring and running an algorithm based on systematic exploration producing investment recommendations on a high-performance computer of the Fugaku, to see whether a very high number of cores could allow for a deeper exploration of the data compared to a cloud machine, hopefully resulting in better predictions. We found that an increase in the number of explored rules results in a net increase in the predictive performance of the final ruleset. Also, in the particular case of this study, we found that using more than around 40 cores does not bring a significant computation time gain. However, the origin of this limitation is explained by a threshold-based search heuristic used to prune the search space. We have evidence that for similar data sets with less restrictive thresholds, the number of cores actually used could very well be much higher, allowing parallelization to have a much greater effect.
Riemannian statistics meets random matrix theory: towards learning from high-dimensional covariance matrices
Authors: Salem Said, Simon Heuveline, Cyrus Mostajeran
Date: 2022-03-01 00:00:00

Riemannian Gaussian distributions were initially introduced as basic building blocks for learning models which aim to capture the intrinsic structure of statistical populations of positive-definite matrices (here called covariance matrices). While the potential applications of such models have attracted significant attention, a major obstacle still stands in the way of these applications: there seems to exist no practical method of computing the normalising factors associated with Riemannian Gaussian distributions on spaces of high-dimensional covariance matrices. The present paper shows that this missing method comes from an unexpected new connection with random matrix theory. Its main contribution is to prove that Riemannian Gaussian distributions of real, complex, or quaternion covariance matrices are equivalent to orthogonal, unitary, or symplectic log-normal matrix ensembles. This equivalence yields a highly efficient approximation of the normalising factors, in terms of a rather simple analytic expression. The error due to this approximation decreases like the inverse square of dimension. Numerical experiments are conducted which demonstrate how this new approximation can unlock the difficulties which have impeded applications to real-world datasets of high-dimensional covariance matrices. The paper then turns to Riemannian Gaussian distributions of block-Toeplitz covariance matrices. These are equivalent to yet another kind of random matrix ensembles, here called "acosh-normal" ensembles. Orthogonal and unitary "acosh-normal" ensembles correspond to the cases of block-Toeplitz with Toeplitz blocks, and block-Toeplitz (with general blocks) covariance matrices, respectively.
DynamicRetriever: A Pre-training Model-based IR System with Neither Sparse nor Dense Index
Authors: Yujia Zhou, Jing Yao, Zhicheng Dou, Ledell Wu, Ji-Rong Wen
Date: 2022-03-01 00:00:00
Tasks: Information Retrieval, Retrieval

Web search provides a promising way for people to obtain information and has been extensively studied. With the surgence of deep learning and large-scale pre-training techniques, various neural information retrieval models are proposed and they have demonstrated the power for improving search (especially, the ranking) quality. All these existing search methods follow a common paradigm, i.e. index-retrieve-rerank, where they first build an index of all documents based on document terms (i.e., sparse inverted index) or representation vectors (i.e., dense vector index), then retrieve and rerank retrieved documents based on similarity between the query and documents via ranking models. In this paper, we explore a new paradigm of information retrieval with neither sparse nor dense index but only a model. Specifically, we propose a pre-training model-based IR system called DynamicRetriever. As for this system, the training stage embeds the token-level and document-level information (especially, document identifiers) of the corpus into the model parameters, then the inference stage directly generates document identifiers for a given query. Compared with existing search methods, the model-based IR system has two advantages: i) it parameterizes the traditional static index with a pre-training model, which converts the document semantic mapping into a dynamic and updatable process; ii) with separate document identifiers, it captures both the term-level and document-level information for each document. Extensive experiments conducted on the public search benchmark MS MARCO verify the effectiveness and potential of our proposed new paradigm for information retrieval.
Results Merging in the Patent Domain
Authors: Vasileios Stamatis, Michail Salampasis
Date: 2022-03-01 00:00:00
Tasks: regression, Retrieval

In this paper, we test machine learning methods for results merging in patent document retrieval. Specifically, we examine random forest, decision tree, support vector machine (SVR), linear regression, polynomial regression, and deep neural networks (DNNs). We use two different methods for results merging, the multiple models (MM) method and the global model method (GM). Furthermore, we examine whether the ranking of the document's scores is linearly explainable. The CLEF-IP 2011 standard test collection was used in our experiments. The random forest produces the best results in comparison to all other models, and it fits the data better than linear and polynomial approaches.
Social Network Extraction Unsupervised
Authors: Mahyuddin K. M. Nasution, Rahmad Syah
Date: 2022-02-24 00:00:00

In the era of information technology, the two developing sides are data science and artificial intelligence. In terms of scientific data, one of the tasks is the extraction of social networks from information sources that have the nature of big data. Meanwhile, in terms of artificial intelligence, the presence of contradictory methods has an impact on knowledge. This article describes an unsupervised as a stream of methods for extracting social networks from information sources. There are a variety of possible approaches and strategies to superficial methods as a starting concept. Each method has its advantages, but in general, it contributes to the integration of each other, namely simplifying, enriching, and emphasizing the results.
PhyCOM: A Multi-Layer Parametric Network for Joint Linear Impairments Compensation and Symbol Detection
Authors: Vincent Choqueuse, Alexandru Frunza, Stéphane Azou, Pascal Morel
Date: 2022-03-01 00:00:00

In this paper, we focus on the joint impairments compensation and symbol detection problem in communication systems. First, we introduce a new multi-layer channel model that represents the underlying physics of multiple impairments in communication systems. This model is composed of widely linear parametric layers that describe the input-output relationship of the front-end impairments and channel effects. Using this particular model, we show that the joint compensation and zero-forcing detection problem can be solved by a particular feedforward network called PhyCOM. Because of the small number of network parameters, a PhyCOM network can be trained efficiently using sophisticated optimization algorithms and a limited number of pilot symbols. Numerical examples are provided to demonstrate the effectiveness of PhyCOM networks with communication systems corrupted by transmitter and receiver IQ imbalances, carrier frequency offset, finite impulse response channels, and transmitter and receiver phase noise distortions. Compared to conventional digital signal processing approaches, simulation results show that the proposed technique is much more flexible and offers better statistical performance both in terms of MSE and SER with a moderate increase of the computation complexity.
Improved iterative methods for solving risk parity portfolio
Authors: Jaehyuk Choi, Rong Chen
Date: 2022-02-28 00:00:00

Risk parity, also known as equal risk contribution, has recently gained increasing attention as a portfolio allocation method. However, solving portfolio weights must resort to numerical methods as the analytic solution is not available. This study improves two existing iterative methods: the cyclical coordinate descent (CCD) and Newton methods. We enhance the CCD method by simplifying the formulation using a correlation matrix and imposing an additional rescaling step. We also suggest an improved initial guess inspired by the CCD method for the Newton method. Numerical experiments show that the improved CCD method performs the best and is approximately three times faster than the original CCD method, saving more than 40% of the iterations.
An Overview of Microsoft Academic Service (MAS) and Applications
Authors: Arnab Sinha, Zhihong Shen, Yang song, Hao Ma, Darrin Eide, Bo-June (Paul) Hsu, Kuansan Wang
Date: 2015-05-18 00:00:00
Proceeding: WWW 2015 5

In this paper we describe a new release of a Web scale entity graph that serves as the backbone of Microsoft Academic Service (MAS), a major production effort with a broadened scope to the namesake vertical search engine that has been publicly available since 2008 as a research prototype. At the core of MAS is a heterogeneous entity graph comprised of six types of entities that model the scholarly activities: field of study, author, institution, paper, venue, and event. In addition to obtaining these entities from the publisher feeds as in the previous effort, we in this version include data mining results from the Web index and an in-house knowledge base from Bing, a major commercial search engine. As a result of the Bing integration, the new MAS graph sees significant increase in size, with fresh information streaming in automatically following their discoveries by the search engine. In addition, the rich entity relations included in the knowledge base provide additional signals to disambiguate and enrich the entities within and beyond the academic domain. The number of papers indexed by MAS, for instance, has grown from low tens of millions to 83 million while maintaining an above 95% accuracy based on test data sets derived from academic activities at Microsoft Research. Based on the data set, we demonstrate two scenarios in this work: a knowledge driven, highly interactive dialog that seamlessly combines reactive search and proactive suggestion experience, and a proactive heterogeneous entity recommendation.
A neural probabilistic model for context based citation recommendation
Authors: Wenyi Huang, Zhaohui Wu, Chen Liang, Prasenjit Mitra, C. Lee Giles
Date: 2015-02-19 00:00:00
Proceeding: AAAI 2015 2
Tasks: Citation Recommendation

Automatic citation recommendation can be very useful for authoring a paper and is an AI-complete problem due to the challenge of bridging the semantic gap between citation context and the cited paper. It is not always easy for knowledgeable researchers to give an accurate citation context for a cited paper or to find the right paper to cite given context. To help with this problem, we propose a novel neural probabilistic model that jointly learns the semantic representations of citation contexts and cited papers. The probability of citing a paper given a citation context is estimated by training a multi-layer neural network. We implement and evaluate our model on the entire CiteSeer dataset, which at the time of this work consists of 10,760,318 citation contexts from 1,017,457 papers. We show that the proposed model significantly outperforms other state-of-the-art models in recall, MAP, MRR, and nDCG.
Energy-Efﬁcient Over-the-Air Computation Scheme for Densely Deployed IoT Networks
Authors: Semiha Tedik Basaran, Student Member, Gunes Karabulut Kurt, and Periklis Chatzimisios, Senior Member, IEEE
Date: 2020-05-05 00:00:00
Proceeding: IEEE 2020 5

—In this article, we propose a spatial sampling
approach to reduce energy consumption for over-the-air
(function) computation (AirComp) scheme by utilizing the
cross-correlations among sensor readings. Since the con-
ventional AirComp scheme leads to a reduction in total
transmission time and latency thanks to the joint com-
munication and computation processes, it is especially
well-suited to Internet of Things (IoT) monitoring systems.
AirComp lets simultaneous transmissions of all nodes by
exploiting the superposition property of wireless channel;
however, it does not overcome the high energy consump-
tion paradigm, which is a fundamental problem of densely
deployed IoT monitoring systems. We present a minimum
mean square error (MMSE) estimation scheme while a small
number of observations are available for densely deployed
networks. The proposed MMSE estimator provides a signif-
icant mean squared error improvement with reducing en-
ergy consumption compared to the conventional estimator.
Since the network lifetime of IoT monitoring systems can be
almost doubled, the proposed estimator provides ﬂexibility
for the dense deployment of nodes. The simulation results
verify the theoretical expressions.