pwc-archive/papers-with-abstracts huggingface dataset:
A Unified Transformer-based Framework for Duplex Text Normalization
Authors: Tuan Manh Lai, Yang Zhang, Evelina Bakhturina, Boris Ginsburg, Heng Ji
Date: 2021-08-23 00:00:00
Tasks: Automatic Speech Recognition, Automatic Speech Recognition (ASR), Data Augmentation, Sentence, speech-recognition, Speech Recognition, Speech Synthesis, Text Normalization, text-to-speech, Text to Speech, Text-To-Speech Synthesis

Text normalization (TN) and inverse text normalization (ITN) are essential preprocessing and postprocessing steps for text-to-speech synthesis and automatic speech recognition, respectively. Many methods have been proposed for either TN or ITN, ranging from weighted finite-state transducers to neural networks. Despite their impressive performance, these methods aim to tackle only one of the two tasks but not both. As a result, in a complete spoken dialog system, two separate models for TN and ITN need to be built. This heterogeneity increases the technical complexity of the system, which in turn increases the cost of maintenance in a production setting. Motivated by this observation, we propose a unified framework for building a single neural duplex system that can simultaneously handle TN and ITN. Combined with a simple but effective data augmentation method, our systems achieve state-of-the-art results on the Google TN dataset for English and Russian. They can also reach over 95% sentence-level accuracy on an internal English TN dataset without any additional fine-tuning. In addition, we also create a cleaned dataset from the Spoken Wikipedia Corpora for German and report the performance of our systems on the dataset. Overall, experimental results demonstrate the proposed duplex text normalization framework is highly effective and applicable to a range of domains and languages
UzBERT: pretraining a BERT model for Uzbek
Authors: B. Mansurov, A. Mansurov
Date: 2021-08-22 00:00:00
Tasks: Language Modeling, Language Modelling, model, named-entity-recognition, Named Entity Recognition, Named Entity Recognition (NER), Part-Of-Speech Tagging, Question Answering

Pretrained language models based on the Transformer architecture have achieved state-of-the-art results in various natural language processing tasks such as part-of-speech tagging, named entity recognition, and question answering. However, no such monolingual model for the Uzbek language is publicly available. In this paper, we introduce UzBERT, a pretrained Uzbek language model based on the BERT architecture. Our model greatly outperforms multilingual BERT on masked language model accuracy. We make the model publicly available under the MIT open-source license.
How Cute is Pikachu? Gathering and Ranking Pokémon Properties from Data with Pokémon Word Embeddings
Authors: Mika Hämäläinen, Khalid Alnajjar, Niko Partanen
Date: 2021-08-21 00:00:00
Tasks: Descriptive, Word Embeddings

We present different methods for obtaining descriptive properties automatically for the 151 original Pok\'emon. We train several different word embeddings models on a crawled Pok\'emon corpus, and use them to rank automatically English adjectives based on how characteristic they are to a given Pok\'emon. Based on our experiments, it is better to train a model with domain specific data than to use a pretrained model. Word2Vec produces less noise in the results than fastText model. Furthermore, we expand the list of properties for each Pok\'emon automatically. However, none of the methods is spot on and there is a considerable amount of noise in the different semantic models. Our models have been released on Zenodo.
Metric Learning in Multilingual Sentence Similarity Measurement for Document Alignment
Authors: Charith Rajitha, Lakmali Piyarathne, Dilan Sachintha, Surangika Ranathunga
Date: 2021-08-21 00:00:00
Proceeding: RANLP 2021 9
Tasks: Metric Learning, Sentence, Sentence Similarity

Document alignment techniques based on multilingual sentence representations have recently shown state of the art results. However, these techniques rely on unsupervised distance measurement techniques, which cannot be fined-tuned to the task at hand. In this paper, instead of these unsupervised distance measurement techniques, we employ Metric Learning to derive task-specific distance measurements. These measurements are supervised, meaning that the distance measurement metric is trained using a parallel dataset. Using a dataset belonging to English, Sinhala, and Tamil, which belong to three different language families, we show that these task-specific supervised distance learning metrics outperform their unsupervised counterparts, for document alignment.
Yseop at FinSim-3 Shared Task 2021: Specializing Financial Domain Learning with Phrase Representations
Authors: Hanna Abi Akl, Dominique Mariko, Hugues de Mazancourt
Date: 2021-08-21 00:00:00
Proceeding: FinNLP 2021 8
Tasks: Sentence, Sentence Embeddings, Word Embeddings

In this paper, we present our approaches for the FinSim-3 Shared Task 2021: Learning Semantic Similarities for the Financial Domain. The aim of this shared task is to correctly classify a list of given terms from the financial domain into the most relevant hypernym (or top-level) concept in an external ontology. For our system submission, we evaluate two methods: a Sentence-RoBERTa (SRoBERTa) embeddings model pre-trained on a custom corpus, and a dual word-sentence embeddings model that builds on the first method by improving the proposed baseline word embeddings construction using the FastText model to boost the classification performance. Our system ranks 2nd overall on both metrics, scoring 0.917 on Average Accuracy and 1.141 on Mean Rank.
2020 U.S. presidential election in swing states: Gender differences in Twitter conversations
Authors: Amir Karami, Spring B. Clark, Anderson Mackenzie, Dorathea Lee, Michael Zhu, Hannah R. Boyajieff, Bailey Goldschmidt
Date: 2021-08-21 00:00:00

Social media is commonly used by the public during election campaigns to express their opinions regarding different issues. Among various social media channels, Twitter provides an efficient platform for researchers and politicians to explore public opinion regarding a wide range of topics such as the economy and foreign policy. Current literature mainly focuses on analyzing the content of tweets without considering the gender of users. This research collects and analyzes a large number of tweets and uses computational, human coding, and statistical analyses to identify topics in more than 300,000 tweets posted during the 2020 U.S. presidential election and to compare female and male users regarding the average weight of the discussed topics. Our findings are based upon a wide range of topics, such as tax, climate change, and the COVID-19 pandemic. Out of the topics, there exists a significant difference between female and male users for more than 70% of topics.
Decomposition Multi-Objective Evolutionary Optimization: From State-of-the-Art to Future Opportunities
Authors: Ke Li
Date: 2021-08-21 00:00:00
Tasks: Decision Making

Decomposition has been the mainstream approach in the classic mathematical programming for multi-objective optimization and multi-criterion decision-making. However, it was not properly studied in the context of evolutionary multi-objective optimization until the development of multi-objective evolutionary algorithm based on decomposition (MOEA/D). In this article, we present a comprehensive survey of the development of MOEA/D from its origin to the current state-of-the-art approaches. In order to be self-contained, we start with a step-by-step tutorial that aims to help a novice quickly get onto the working mechanism of MOEA/D. Then, selected major developments of MOEA/D are reviewed according to its core design components including weight vector settings, sub-problem formulations, selection mechanisms and reproduction operators. Besides, we also overviews some further developments for constraint handling, computationally expensive objective functions, preference incorporation, and real-world applications. In the final part, we shed some lights on emerging directions for future developments.
Natural Evolution Strategy for Unconstrained and Implicitly Constrained Problems with Ridge Structure
Authors: Masahiro Nomura, Isao Ono
Date: 2021-08-21 00:00:00

In this paper, we propose a new natural evolution strategy for unconstrained black-box function optimization (BBFO) problems and implicitly constrained BBFO problems. BBFO problems are known to be difficult because explicit representations of objective functions are not available. Implicit constraints make the problems more difficult because whether or not a solution is feasible is revealed when the solution is evaluated with the objective function. DX-NES-IC is one of the promising methods for implicitly constrained BBFO problems. DX-NES-IC has shown better performance than conventional methods on implicitly constrained benchmark problems. However, DX-NES-IC has a problem in that the moving speed of the probability distribution is slow on ridge structure. To address the problem, we propose the Fast Moving Natural Evolution Strategy (FM-NES) that accelerates the movement of the probability distribution on ridge structure by introducing the rank-one update into DX-NES-IC. The rank-one update is utilized in CMA-ES. Since naively introducing the rank-one update makes the search performance deteriorate on implicitly constrained problems, we propose a condition of performing the rank-one update. We also propose to reset the shape of the probability distribution when an infeasible solution is sampled at the first time. In numerical experiments using unconstrained and implicitly constrained benchmark problems, FM-NES showed better performance than DX-NES-IC on problems with ridge structure and almost the same performance as DX-NES-IC on the others. Furthermore, FM-NES outperformed xNES, CMA-ES, xNES with the resampling technique, and CMA-ES with the resampling technique.
Pattern Inversion as a Pattern Recognition Method for Machine Learning
Authors: Alexei Mikhailov, Mikhail Karavay
Date: 2021-08-15 00:00:00
Tasks: BIG-bench Machine Learning, Deep Learning

Artificial neural networks use a lot of coefficients that take a great deal of computing power for their adjustment, especially if deep learning networks are employed. However, there exist coefficients-free extremely fast indexing-based technologies that work, for instance, in Google search engines, in genome sequencing, etc. The paper discusses the use of indexing-based methods for pattern recognition. It is shown that for pattern recognition applications such indexing methods replace with inverse patterns the fully inverted files, which are typically employed in search engines. Not only such inversion provide automatic feature extraction, which is a distinguishing mark of deep learning, but, unlike deep learning, pattern inversion supports almost instantaneous learning, which is a consequence of absence of coefficients. The paper discusses a pattern inversion formalism that makes use on a novel pattern transform and its application for unsupervised instant learning. Examples demonstrate a view-angle independent recognition of three-dimensional objects, such as cars, against arbitrary background, prediction of remaining useful life of aircraft engines, and other applications. In conclusion, it is noted that, in neurophysiology, the function of the neocortical mini-column has been widely debated since 1957. This paper hypothesize that, mathematically, the cortical mini-column can be described as an inverse pattern, which physically serves as a connection multiplier expanding associations of inputs with relevant pattern classes.
Distilling Neuron Spike with High Temperature in Reinforcement Learning Agents
Authors: Ling Zhang, Jian Cao, Yuan Zhang, Bohan Zhou, Shuo Feng
Date: 2021-08-05 00:00:00
Tasks: reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL), Vocal Bursts Intensity Prediction

Spiking neural network (SNN), compared with depth neural network (DNN), has faster processing speed, lower energy consumption and more biological interpretability, which is expected to approach Strong AI. Reinforcement learning is similar to learning in biology. It is of great significance to study the combination of SNN and RL. We propose the reinforcement learning method of spike distillation network (SDN) with STBP. This method uses distillation to effectively avoid the weakness of STBP, which can achieve SOTA performance in classification, and can obtain a smaller, faster convergence and lower power consumption SNN reinforcement learning model. Experiments show that our method can converge faster than traditional SNN reinforcement learning and DNN reinforcement learning methods, about 1000 epochs faster, and obtain SNN 200 times smaller than DNN. We also deploy SDN to the PKU nc64c chip, which proves that SDN has lower power consumption than DNN, and the power consumption of SDN is more than 600 times lower than DNN on large-scale devices. SDN provides a new way of SNN reinforcement learning, and can achieve SOTA performance, which proves the possibility of further development of SNN reinforcement learning.
Time-Frequency Analysis based Deep Interference Classification for Frequency Hopping System
Authors: Changzhi Xu, Jingya Ren, Wanxin Yu, Yi Jin, Zhenxin Cao, Xiaogang Wu, Weiheng Jiang
Date: 2021-07-21 00:00:00
Tasks: Classification, Transfer Learning

It is known that, interference classification plays an important role in protecting the authorized communication system and avoiding its performance degradation in the hostile environment. In this paper, the interference classification problem for the frequency hopping communication system is discussed. Considering the possibility of presence multiple interferences in the frequency hopping system, in order to fully extract effective features of the interferences from the received signals, the linear and bilinear transform based composite time-frequency analysis method is adopted. Then the time-frequency spectrograms obtained from the time-frequency analysis are constructed as matching pairs and input to the deep neural network for classification. In particular, the Siamese neural network is used as the classifier, where the paired spectrograms are input into the two sub-networks of the deep networks, and these two sub-networks extract the features of the paired spectrograms for interference type classification. The simulation results confirm that the proposed algorithm can obtain higher classification accuracy than both traditional single time-frequency representation based approach and the AlexNet transfer learning or convolutional neural network based methods.
Network control by a constrained external agent as a continuous optimization problem
Authors: Jannes Nys, Milan van den Heuvel, Koen Schoors, Bruno Merlevede
Date: 2021-08-23 00:00:00

Social science studies dealing with control in networks typically resort to heuristics or describing the static control distribution. Optimal policies, however, require interventions that optimize control over a socioeconomic network subject to real-world constraints. We integrate optimisation tools from deep-learning with network science into a framework that is able to optimize such interventions in real-world networks. We demonstrate the framework in the context of corporate control, where it allows to characterize the vulnerability of strategically important corporate networks to sensitive takeovers, an important contemporaneous policy challenge. The framework produces insights that are relevant for governing real-world socioeconomic networks, and opens up new research avenues for improving our understanding and control of such complex systems.
Sequence-to-Sequence Learning on Keywords for Efficient FAQ Retrieval
Authors: Sourav Dutta, Haytham Assem, Edward Burgin
Date: 2021-08-23 00:00:00
Tasks: Keyword Extraction, Question Answering, Retrieval

Frequently-Asked-Question (FAQ) retrieval provides an effective procedure for responding to user's natural language based queries. Such platforms are becoming common in enterprise chatbots, product question answering, and preliminary technical support for customers. However, the challenge in such scenarios lies in bridging the lexical and semantic gap between varied query formulations and the corresponding answers, both of which typically have a very short span. This paper proposes TI-S2S, a novel learning framework combining TF-IDF based keyword extraction and Word2Vec embeddings for training a Sequence-to-Sequence (Seq2Seq) architecture. It achieves high precision for FAQ retrieval by better understanding the underlying intent of a user question captured via the representative keywords. We further propose a variant with an additional neural network module for guiding retrieval via relevant candidate identification based on similarity features. Experiments on publicly available dataset depict our approaches to provide around 92% precision-at-rank-5, exhibiting nearly 13% improvement over existing approaches.
Multi-Server Private Linear Transformation with Joint Privacy
Authors: Fatemeh Kazemi, Alex Sprintson
Date: 2021-08-22 00:00:00

This paper focuses on the Private Linear Transformation (PLT) problem in the multi-server scenario. In this problem, there are $N$ servers, each of which stores an identical copy of a database consisting of $K$ independent messages, and there is a user who wishes to compute $L$ independent linear combinations of a subset of $D$ messages in the database while leaking no information to the servers about the identity of the entire set of these $D$ messages required for the computation. We focus on the setting in which the coefficient matrix of the desired $L$ linear combinations generates a Maximum Distance Separable (MDS) code. We characterize the capacity of the PLT problem, defined as the supremum of all achievable download rates, for all parameters $N, K, D \geq 1$ and $L=1$, i.e., when the user wishes to compute one linear combination of $D$ messages. Moreover, we establish an upper bound on the capacity of PLT problem for all parameters $N, K, D, L \geq 1$, and leveraging some known capacity results, we show the tightness of this bound in the following regimes: (i) the case when there is a single server (i.e., $N=1$), (ii) the case when $L=1$, and (iii) the case when $L=D$.
Pre-training for Ad-hoc Retrieval: Hyperlink is Also You Need
Authors: Zhengyi Ma, Zhicheng Dou, Wei Xu, Xinyu Zhang, Hao Jiang, Zhao Cao, Ji-Rong Wen
Date: 2021-08-20 00:00:00
Tasks: Language Modeling, Language Modelling, Retrieval

Designing pre-training objectives that more closely resemble the downstream tasks for pre-trained language models can lead to better performance at the fine-tuning stage, especially in the ad-hoc retrieval area. Existing pre-training approaches tailored for IR tried to incorporate weak supervised signals, such as query-likelihood based sampling, to construct pseudo query-document pairs from the raw textual corpus. However, these signals rely heavily on the sampling method. For example, the query likelihood model may lead to much noise in the constructed pre-training data. \blfootnote{$\dagger$ This work was done during an internship at Huawei.} In this paper, we propose to leverage the large-scale hyperlinks and anchor texts to pre-train the language model for ad-hoc retrieval. Since the anchor texts are created by webmasters and can usually summarize the target document, it can help to build more accurate and reliable pre-training samples than a specific algorithm. Considering different views of the downstream ad-hoc retrieval, we devise four pre-training tasks based on the hyperlinks. We then pre-train the Transformer model to predict the pair-wise preference, jointly with the Masked Language Model objective. Experimental results on two large-scale ad-hoc retrieval datasets show the significant improvement of our model compared with the existing methods.
DO HIGH-FREQUENCY DATA IMPROVE HIGH-DIMENSIONALPORTFOLIO ALLOCATIONS
Authors: Nikolaus Hautsch, LADA M. KYJ AND PETER MALEC
Date: 2013-11-27 00:00:00
Proceeding: journal 2013 11
Tasks: Vocal Bursts Intensity Prediction

This paper addresses the debate about the usefulness of high-frequency (HF) data in large-scale portfolio
allocation.We construct global minimum variance portfolios based on the constituents of the S&P 500. HF-based
covariance matrix predictions are obtained by applying a blocked realized kernel estimator, different smoothing
windows, various regularization methods and two forecasting models.We show that HF-based predictions yield a
significantly lower portfolio volatility than methods employing daily returns. Particularly during the 2008 financial
crisis, these performance gains hold over longer horizons than previous studies have shown, translating into
substantial utility gains for an investor with pronounced risk aversion.
Finding essential parts of the brain in rs-fMRI can improve diagnosing ADHD by Deep Learning
Authors: Byunggun Kim, Jaeseon Park, Taehun Kim, Younghun Kwon
Date: 2021-08-14 00:00:00
Tasks: Diagnostic

Attention Deficit\Hyperactivity Disorder(ADHD) is considered a very common psychiatric disorder, but it is difficult to establish an accurate diagnostic method for ADHD. Recently, with the development of computing resources and machine learning methods, studies have been conducted to classify ADHD using resting-state functional magnetic resonance(rsfMRI) imaging data. However, most of them utilized all areas of the brain for training the models. In this study, as a different way from this approach, we conducted a study to classify ADHD by selecting areas that are essential for using a deep learning model. For the experiment, rsfMRI data provided by ADHD 200 global competition was used. To obtain an integrated result from the multiple sites, each region of the brain was evaluated with Leave one site out cross-validation. As a result, when we only used 15 important region of interest(ROIs) for training, an accuracy of 70.6% was obtained, significantly exceeding the existing results of 68.6% from all ROIs. In addition, to explore the new structure based on SCCNN-RNN, we performed the same experiment with three modified models: (1) Separate Channel CNN RNN with Attention (ASCRNN), (2) Separate Channel dilate CNN RNN with Attention (ASDRNN), (3) Separate Channel CNN slicing RNN with Attention (ASSRNN). As a result, the ASSRNN model provided a high accuracy of 70.46% when training with only 13 important region of interest (ROI). These results show that finding and using the crucial parts of the brain in diagnosing ADHD by Deep Learning can get better results than using all areas.