pwc-archive/papers-with-abstracts huggingface dataset:
Beamspace MIMO for Satellite Swarms
Authors: Maik Röper, Bho Matthiesen, Dirk Wübben, Petar Popovski, Armin Dekorsy
Date: 2021-12-16 00:00:00
Tasks: Position

Systems of small distributed satellites in low Earth orbit (LEO) transmitting cooperatively to a multiple antenna ground station (GS) are investigated. These satellite swarms have the benefit of much higher spatial separation in the transmit antennas than traditional big satellites with antenna arrays, promising a massive increase in spectral efficiency. However, this would require instantaneous perfect channel state information (CSI) and strong cooperation between satellites. In practice, orbital velocities around 7.5 km/s lead to very short channel coherence times on the order of fractions of the inter-satellite propagation delay, invalidating these assumptions. In this paper, we propose a distributed linear precoding scheme and a GS equalizer relying on local position information. In particular, each satellite only requires information about its own position and that of the GS, while the GS has complete positional information. Due to the deterministic nature of satellite movement this information is easily obtained and no inter-satellite information exchange is required during transmission. Based on the underlying geometrical channel approximation, the optimal inter-satellite distance is obtained analytically. Numerical evaluations show that the proposed scheme is, on average, within 99.8% of the maximum achievable rate for instantaneous CSI and perfect cooperation
Anomaly Detection using Capsule Networks for High-dimensional Datasets
Authors: Inderjeet Singh, Nandyala Hemachandra
Date: 2021-12-27 00:00:00
Tasks: Anomaly Detection, Binary Classification, Fraud Detection, Outlier Detection, Vocal Bursts Intensity Prediction

Anomaly detection is an essential problem in machine learning. Application areas include network security, health care, fraud detection, etc., involving high-dimensional datasets. A typical anomaly detection system always faces the class-imbalance problem in the form of a vast difference in the sample sizes of different classes. They usually have class overlap problems. This study used a capsule network for the anomaly detection task. To the best of our knowledge, this is the first instance where a capsule network is analyzed for the anomaly detection task in a high-dimensional complex data setting. We also handle the related novelty and outlier detection problems. The architecture of the capsule network was suitably modified for a binary classification task. Capsule networks offer a good option for detecting anomalies due to the effect of viewpoint invariance captured in its predictions and viewpoint equivariance captured in internal capsule architecture. We used six-layered under-complete autoencoder architecture with second and third layers containing capsules. The capsules were trained using the dynamic routing algorithm. We created $10$-imbalanced datasets from the original MNIST dataset and compared the performance of the capsule network with $5$ baseline models. Our leading test set measures are F1-score for minority class and area under the ROC curve. We found that the capsule network outperformed every other baseline model on the anomaly detection task by using only ten epochs for training and without using any other data level and algorithm level approach. Thus, we conclude that capsule networks are excellent in modeling complex high-dimensional imbalanced datasets for the anomaly detection task.
LINDA: Unsupervised Learning to Interpolate in Natural Language Processing
Authors: Yekyung Kim, Seohyeong Jeong, Kyunghyun Cho
Date: 2021-12-28 00:00:00
Tasks: Data Augmentation, text-classification, Text Classification

Despite the success of mixup in data augmentation, its applicability to natural language processing (NLP) tasks has been limited due to the discrete and variable-length nature of natural languages. Recent studies have thus relied on domain-specific heuristics and manually crafted resources, such as dictionaries, in order to apply mixup in NLP. In this paper, we instead propose an unsupervised learning approach to text interpolation for the purpose of data augmentation, to which we refer as "Learning to INterpolate for Data Augmentation" (LINDA), that does not require any heuristics nor manually crafted resources but learns to interpolate between any pair of natural language sentences over a natural language manifold. After empirically demonstrating the LINDA's interpolation capability, we show that LINDA indeed allows us to seamlessly apply mixup in NLP and leads to better generalization in text classification both in-domain and out-of-domain.
AET-SGD: Asynchronous Event-triggered Stochastic Gradient Descent
Authors: Nhuong Nguyen, Song Han
Date: 2021-12-27 00:00:00

Communication cost is the main bottleneck for the design of effective distributed learning algorithms. Recently, event-triggered techniques have been proposed to reduce the exchanged information among compute nodes and thus alleviate the communication cost. However, most existing event-triggered approaches only consider heuristic event-triggered thresholds. They also ignore the impact of computation and network delay, which play an important role on the training performance. In this paper, we propose an Asynchronous Event-triggered Stochastic Gradient Descent (SGD) framework, called AET-SGD, to i) reduce the communication cost among the compute nodes, and ii) mitigate the impact of the delay. Compared with baseline event-triggered methods, AET-SGD employs a linear increasing sample size event-triggered threshold, and can significantly reduce the communication cost while keeping good convergence performance. We implement AET-SGD and evaluate its performance on multiple representative data sets, including MNIST, FashionMNIST, KMNIST and CIFAR10. The experimental results validate the correctness of the design and show a significant communication cost reduction from 44x to 120x, compared to the state of the art. Our results also show that AET-SGD can resist large delay from the straggler nodes while obtaining a decent performance and a desired speedup ratio.
Deep Filtering with DNN, CNN and RNN
Authors: Bin Xie, Qing Zhang
Date: 2021-12-19 00:00:00

This paper is about a deep learning approach for linear and nonlinear filtering. The idea is to train a neural network with Monte Carlo samples generated from a nominal dynamic model. Then the network weights are applied to Monte Carlo samples from an actual dynamic model. A main focus of this paper is on the deep filters with three major neural network architectures (DNN, CNN, RNN). Our deep filter compares favorably to the traditional Kalman filter in linear cases and outperform the extended Kalman filter in nonlinear cases. Then a switching model with jumps is studied to show the adaptiveness and power of our deep filtering. Among the three major NNs, the CNN outperform the others on average. while the RNN does not seem to be suitable for the filtering problem. One advantage of the deep filter is its robustness when the nominal model and actual model differ. The other advantage of deep filtering is real data can be used directly to train the deep neutral network. Therefore, model calibration can be by-passed all together.
Generalized Difference Coder: A Novel Conditional Autoencoder Structure for Video Compression
Authors: Fabian Brand, Jürgen Seiler, André Kaup
Date: 2021-12-15 00:00:00
Tasks: Video Compression

Motion compensated inter prediction is a common component of all video coders. The concept was established in traditional hybrid coding and successfully transferred to learning-based video compression. To compress the residual signal after prediction, usually the difference of the two signals is compressed using a standard autoencoder. However, information theory tells us that a general conditional coder is more efficient. In this paper, we provide a solid foundation based on information theory and Shannon entropy to show the potentials but also the limits of conditional coding. Building on those results, we then propose the generalized difference coder, a special case of a conditional coder designed to avoid limiting bottlenecks. With this coder, we are able to achieve average rate savings of 27.8% compared to a standard autoencoder, by only adding a moderate complexity overhead of less than 7%.
Depth and Feature Learning are Provably Beneficial for Neural Network Discriminators
Authors: Carles Domingo-Enrich
Date: 2021-12-27 00:00:00

We construct pairs of distributions $\mu_d, \nu_d$ on $\mathbb{R}^d$ such that the quantity $|\mathbb{E}_{x \sim \mu_d} [F(x)] - \mathbb{E}_{x \sim \nu_d} [F(x)]|$ decreases as $\Omega(1/d^2)$ for some three-layer ReLU network $F$ with polynomial width and weights, while declining exponentially in $d$ if $F$ is any two-layer network with polynomial weights. This shows that deep GAN discriminators are able to distinguish distributions that shallow discriminators cannot. Analogously, we build pairs of distributions $\mu_d, \nu_d$ on $\mathbb{R}^d$ such that $|\mathbb{E}_{x \sim \mu_d} [F(x)] - \mathbb{E}_{x \sim \nu_d} [F(x)]|$ decreases as $\Omega(1/(d\log d))$ for two-layer ReLU networks with polynomial weights, while declining exponentially for bounded-norm functions in the associated RKHS. This confirms that feature learning is beneficial for discriminators. Our bounds are based on Fourier transforms.
Tracking Most Significant Arm Switches in Bandits
Authors: Joe Suk, Samory Kpotufe
Date: 2021-12-27 00:00:00

In bandit with distribution shifts, one aims to automatically adapt to unknown changes in reward distribution, and restart exploration when necessary. While this problem has been studied for many years, a recent breakthrough of Auer et al. (2018, 2019) provides the first adaptive procedure to guarantee an optimal (dynamic) regret $\sqrt{LT}$, for $T$ rounds, and an unknown number $L$ of changes. However, while this rate is tight in the worst case, it remained open whether faster rates are possible, without prior knowledge, if few changes in distribution are actually severe. To resolve this question, we propose a new notion of significant shift, which only counts very severe changes that clearly necessitate a restart: roughly, these are changes involving not only best arm switches, but also involving large aggregate differences in reward overtime. Thus, our resulting procedure adaptively achieves rates always faster (sometimes significantly) than $O(\sqrt{ST})$, where $S\ll L$ only counts best arm switches, while at the same time, always faster than the optimal $O(V^{\frac{1}{3}}T^{\frac{2}{3}})$ when expressed in terms of total variation $V$ (which aggregates differences overtime). Our results are expressed in enough generality to also capture non-stochastic adversarial settings.
Channel Parameter Estimation in the Presence of Phase Noise Based on Maximum Correntropy Criterion
Authors: Amir Alizadeh, Ghosheh Abed Hodtani
Date: 2021-12-15 00:00:00
Tasks: parameter estimation

Oscillator output generally has phase noise causing the output power spectral density (PSD) to disperse around a Dirac delta function. In this paper, the AWGN channel is considered, where the sent signal accompanying with phase noise is added to the channel Gaussian noise and received at the receiver. Conventional channel estimation algorithms such as least mean square (LMS) and mean MSE criterion are not suitable for this channel estimation. We (i) analyze this phase noise channel estimation with information theoretic learning (ITL) criterion, i.e., maximum correntropy criterion (MCC), leading to robustness in the channel estimator's steady state behavior; and (ii) improve the convergence rate by combining MSE and MCC as a novel mixed-LMS algorithm.
Unbiased Gradient Estimation in Unrolled Computation Graphs with Persistent Evolution Strategies
Authors: Paul Vicol, Luke Metz, Jascha Sohl-Dickstein
Date: 2021-12-27 00:00:00

Unrolled computation graphs arise in many scenarios, including training RNNs, tuning hyperparameters through unrolled optimization, and training learned optimizers. Current approaches to optimizing parameters in such computation graphs suffer from high variance gradients, bias, slow updates, or large memory usage. We introduce a method called Persistent Evolution Strategies (PES), which divides the computation graph into a series of truncated unrolls, and performs an evolution strategies-based update step after each unroll. PES eliminates bias from these truncations by accumulating correction terms over the entire sequence of unrolls. PES allows for rapid parameter updates, has low memory usage, is unbiased, and has reasonable variance characteristics. We experimentally demonstrate the advantages of PES compared to several other methods for gradient estimation on synthetic tasks, and show its applicability to training learned optimizers and tuning hyperparameters.
Last-Iterate Convergence of Saddle-Point Optimizers via High-Resolution Differential Equations
Authors: Tatjana Chavdarova, Michael I. Jordan, Manolis Zampetakis
Date: 2021-12-27 00:00:00

Several widely-used first-order saddle-point optimization methods yield an identical continuous-time ordinary differential equation (ODE) that is identical to that of the Gradient Descent Ascent (GDA) method when derived naively. However, the convergence properties of these methods are qualitatively different, even on simple bilinear games. Thus the ODE perspective, which has proved powerful in analyzing single-objective optimization methods, has not played a similar role in saddle-point optimization. We adopt a framework studied in fluid dynamics -- known as High-Resolution Differential Equations (HRDEs) -- to design differential equation models for several saddle-point optimization methods. Critically, these HRDEs are distinct for various saddle-point optimization methods. Moreover, in bilinear games, the convergence properties of the HRDEs match the qualitative features of the corresponding discrete methods. Additionally, we show that the HRDE of Optimistic Gradient Descent Ascent (OGDA) exhibits \emph{last-iterate convergence} for general monotone variational inequalities. Finally, we provide rates of convergence for the \emph{best-iterate convergence} of the OGDA method, relying solely on the first-order smoothness of the monotone operator.
A Survey on Using Gaze Behaviour for Natural Language Processing
Authors: Sandeep Mathias, Diptesh Kanojia, Abhijit Mishra, Pushpak Bhattacharyya
Date: 2021-12-21 00:00:00
Tasks: Complex Word Identification, Survey

Gaze behaviour has been used as a way to gather cognitive information for a number of years. In this paper, we discuss the use of gaze behaviour in solving different tasks in natural language processing (NLP) without having to record it at test time. This is because the collection of gaze behaviour is a costly task, both in terms of time and money. Hence, in this paper, we focus on research done to alleviate the need for recording gaze behaviour at run time. We also mention different eye tracking corpora in multiple languages, which are currently available and can be used in natural language processing. We conclude our paper by discussing applications in a domain - education - and how learning gaze behaviour can help in solving the tasks of complex word identification and automatic essay grading.
Toxicity Detection for Indic Multilingual Social Media Content
Authors: Manan Jhaveri, Devanshu Ramaiya, Harveen Singh Chadha
Date: 2022-01-03 00:00:00
Tasks: Abuse Detection, Sentence

Toxic content is one of the most critical issues for social media platforms today. India alone had 518 million social media users in 2020. In order to provide a good experience to content creators and their audience, it is crucial to flag toxic comments and the users who post that. But the big challenge is identifying toxicity in low resource Indic languages because of the presence of multiple representations of the same text. Moreover, the posts/comments on social media do not adhere to a particular format, grammar or sentence structure; this makes the task of abuse detection even more challenging for multilingual social media platforms. This paper describes the system proposed by team 'Moj Masti' using the data provided by ShareChat/Moj in \emph{IIIT-D Multilingual Abusive Comment Identification} challenge. We focus on how we can leverage multilingual transformer based pre-trained and fine-tuned models to approach code-mixed/code-switched classification tasks. Our best performing system was an ensemble of XLM-RoBERTa and MuRIL which achieved a Mean F-1 score of 0.9 on the test data/leaderboard. We also observed an increase in the performance by adding transliterated data. Furthermore, using weak metadata, ensembling and some post-processing techniques boosted the performance of our system, thereby placing us 1st on the leaderboard.
Which Student is Best? A Comprehensive Knowledge Distillation Exam for Task-Specific BERT Models
Authors: Made Nindyatama Nityasya, Haryo Akbarianto Wibowo, Rendi Chevi, Radityo Eko Prasojo, Alham Fikri Aji
Date: 2022-01-03 00:00:00
Tasks: CPU, Data Augmentation, Knowledge Distillation, text-classification, Text Classification, Word Embeddings

We perform knowledge distillation (KD) benchmark from task-specific BERT-base teacher models to various student models: BiLSTM, CNN, BERT-Tiny, BERT-Mini, and BERT-Small. Our experiment involves 12 datasets grouped in two tasks: text classification and sequence labeling in the Indonesian language. We also compare various aspects of distillations including the usage of word embeddings and unlabeled data augmentation. Our experiments show that, despite the rising popularity of Transformer-based models, using BiLSTM and CNN student models provide the best trade-off between performance and computational resource (CPU, RAM, and storage) compared to pruned BERT models. We further propose some quick wins on performing KD to produce small NLP models via efficient KD training mechanisms involving simple choices of loss functions, word embeddings, and unlabeled data preparation.
Learning with Latent Structures in Natural Language Processing: A Survey
Authors: Zhaofeng Wu
Date: 2022-01-03 00:00:00
Tasks: Survey

While end-to-end learning with fully differentiable models has enabled tremendous success in natural language process (NLP) and machine learning, there have been significant recent interests in learning with latent discrete structures to incorporate better inductive biases for improved end-task performance and better interpretability. This paradigm, however, is not straightforwardly amenable to the mainstream gradient-based optimization methods. This work surveys three main families of methods to learn such models: surrogate gradients, continuous relaxation, and marginal likelihood maximization via sampling. We conclude with a review of applications of these methods and an inspection of the learned latent structure that they induce.
Hypers at ComMA@ICON: Modelling Aggressiveness, Gender Bias and Communal Bias Identification
Authors: Sean Benhur, Roshan Nayak, Kanchana Sivanraju, Adeep Hande, Subalalitha Chinnaudayar Navaneethakrishnan, Ruba Priyadharshini, Bharathi Raja Chakravarthi
Date: 2021-12-31 00:00:00
Tasks: Sentence

Due to the exponentially increasing reach of social media, it is essential to focus on its negative aspects as it can potentially divide society and incite people into violence. In this paper, we present our system description of work on the shared task ComMA@ICON, where we have to classify how aggressive the sentence is and if the sentence is gender-biased or communal biased. These three could be the primary reasons to cause significant problems in society. As team Hypers we have proposed an approach that utilizes different pretrained models with Attention and mean pooling methods. We were able to get Rank 3 with 0.223 Instance F1 score on Bengali, Rank 2 with 0.322 Instance F1 score on Multi-lingual set, Rank 4 with 0.129 Instance F1 score on Meitei and Rank 5 with 0.336 Instance F1 score on Hindi. The source code and the pretrained models of this work can be found here.
Clustering Vietnamese Conversations From Facebook Page To Build Training Dataset For Chatbot
Authors: Trieu Hai Nguyen, Thi-Kim-Ngoan Pham, Thi-Hong-Minh Bui, Thanh-Quynh-Chau Nguyen
Date: 2021-12-31 00:00:00
Tasks: Chatbot, Clustering

The biggest challenge of building chatbots is training data. The required data must be realistic and large enough to train chatbots. We create a tool to get actual training data from Facebook messenger of a Facebook page. After text preprocessing steps, the newly obtained dataset generates FVnC and Sample dataset. We use the Retraining of BERT for Vietnamese (PhoBERT) to extract features of our text data. K-Means and DBSCAN clustering algorithms are used for clustering tasks based on output embeddings from PhoBERT$_{base}$. We apply V-measure score and Silhouette score to evaluate the performance of clustering algorithms. We also demonstrate the efficiency of PhoBERT compared to other models in feature extraction on the Sample dataset and wiki dataset. A GridSearch algorithm that combines both clustering evaluations is also proposed to find optimal parameters. Thanks to clustering such a number of conversations, we save a lot of time and effort to build data and storylines for training chatbot.
Establishing Strong Baselines for TripClick Health Retrieval
Authors: Sebastian Hofstätter, Sophia Althammer, Mete Sertkan, Allan Hanbury
Date: 2022-01-02 00:00:00
Tasks: Re-Ranking, Retrieval

We present strong Transformer-based re-ranking and dense retrieval baselines for the recently released TripClick health ad-hoc retrieval collection. We improve the - originally too noisy - training data with a simple negative sampling policy. We achieve large gains over BM25 in the re-ranking task of TripClick, which were not achieved with the original baselines. Furthermore, we study the impact of different domain-specific pre-trained models on TripClick. Finally, we show that dense retrieval outperforms BM25 by considerable margins, even with simple training procedures.
Challenges of sampling and how phylogenetic comparative methods help: With a case study of the Pama-Nyungan laminal contrast
Authors: Jayden L. Macklin-Cordes, Erich R. Round
Date: 2022-01-01 00:00:00

Phylogenetic comparative methods are new in our field and are shrouded, for most linguists, in at least a little mystery. Yet the path that led to their discovery in comparative biology is so similar to the methodological history of balanced sampling, that it is only an accident of history that they were not discovered by a typologist. Here we clarify the essential logic behind phylogenetic comparative methods and their fundamental relatedness to a deep intellectual tradition focussed on sampling. Then we introduce concepts, methods and tools which will enable typologists to use these methods in everyday typological research. The key commonality of phylogenetic comparative methods and balanced sampling is that they attempt to deal with statistical non-independence due to genealogy. Whereas sampling can never achieve independence and requires most comparative data to be discarded, phylogenetic comparative methods achieve independence while retaining and using all data. We discuss the essential notions of phylogenetic signal; uncertainty about trees; typological averages and proportions that are sensitive to genealogy; comparison across language families; and the effects of areality. Extensive supplementary materials illustrate computational tools for practical analysis and we illustrate the methods discussed with a typological case study of the laminal contrast in Pama-Nyungan.