pwc-archive/papers-with-abstracts huggingface dataset:
Improvement in Machine Translation with Generative Adversarial Networks
Authors: Jay Ahn, Hari Madhu, Viet Nguyen
Date: 2021-11-30 00:00:00
Tasks: Generative Adversarial Network, Machine Translation, NMT, Sentence, Text Generation, Translation

In this paper, we explore machine translation improvement via Generative Adversarial Network (GAN) architecture. We take inspiration from RelGAN, a model for text generation, and NMT-GAN, an adversarial machine translation model, to implement a model that learns to transform awkward, non-fluent English sentences to fluent ones, while only being trained on monolingual corpora. We utilize a parameter $\lambda$ to control the amount of deviation from the input sentence, i.e. a trade-off between keeping the original tokens and modifying it to be more fluent. Our results improved upon phrase-based machine translation in some cases. Especially, GAN with a transformer generator shows some promising results. We suggests some directions for future works to build upon this proof-of-concept.
Towards automatic identification of linguistic politeness in Hindi texts
Authors: Ritesh Kumar
Date: 2021-11-30 00:00:00

In this paper I present a classifier for automatic identification of linguistic politeness in Hindi texts. I have used the manually annotated corpus of over 25,000 blog comments to train an SVM. Making use of the discursive and interactional approaches to politeness the paper gives an exposition of the normative, conventionalised politeness structures of Hindi. It is seen that using these manually recognised structures as features in training the SVM significantly improves the performance of the classifier on the test set. The trained system gives a significantly high accuracy of over 77% which is within 2% of human accuracy.
Bilingual Topic Models for Comparable Corpora
Authors: Georgios Balikas, Massih-Reza Amini, Marianne Clausel
Date: 2021-11-30 00:00:00
Tasks: Cross-Lingual Word Embeddings, Retrieval, Semantic Similarity, Semantic Textual Similarity, Topic Models, Word Embeddings

Probabilistic topic models like Latent Dirichlet Allocation (LDA) have been previously extended to the bilingual setting. A fundamental modeling assumption in several of these extensions is that the input corpora are in the form of document pairs whose constituent documents share a single topic distribution. However, this assumption is strong for comparable corpora that consist of documents thematically similar to an extent only, which are, in turn, the most commonly available or easy to obtain. In this paper we relax this assumption by proposing for the paired documents to have separate, yet bound topic distributions. % a binding mechanism between the distributions of the paired documents. We suggest that the strength of the bound should depend on each pair's semantic similarity. To estimate the similarity of documents that are written in different languages we use cross-lingual word embeddings that are learned with shallow neural networks. We evaluate the proposed binding mechanism by extending two topic models: a bilingual adaptation of LDA that assumes bag-of-words inputs and a model that incorporates part of the text structure in the form of boundaries of semantically coherent segments. To assess the performance of the novel topic models we conduct intrinsic and extrinsic experiments on five bilingual, comparable corpora of English documents with French, German, Italian, Spanish and Portuguese documents. The results demonstrate the efficiency of our approach in terms of both topic coherence measured by the normalized point-wise mutual information, and generalization performance measured by perplexity and in terms of Mean Reciprocal Rank in a cross-lingual document retrieval task for each of the language pairs.
Challenges in Developing LRs for Non-Scheduled Languages: A Case of Magahi
Authors: Ritesh Kumar
Date: 2021-11-30 00:00:00
Tasks: POS

Magahi is an Indo-Aryan Language, spoken mainly in the Eastern parts of India. Despite having a significant number of speakers, there has been virtually no language resource (LR) or language technology (LT) developed for the language, mainly because of its status as a non-scheduled language. The present paper describes an attempt to develop an annotated corpus of Magahi. The data is mainly taken from a couple of blogs in Magahi, some collection of stories in Magahi and the recordings of conversation in Magahi and it is annotated at the POS level using BIS tagset.
Minor changes make a difference: a case study on the consistency of UD-based dependency parsers
Authors: Dmytro Kalpakchi, Johan Boye
Date: 2021-11-30 00:00:00
Proceeding: UDW (SyntaxFest) 2021 12
Tasks: Data Augmentation

Many downstream applications are using dependency trees, and are thus relying on dependency parsers producing correct, or at least consistent, output. However, dependency parsers are trained using machine learning, and are therefore susceptible to unwanted inconsistencies due to biases in the training data. This paper explores the effects of such biases in four languages - English, Swedish, Russian, and Ukrainian - though an experiment where we study the effect of replacing numerals in sentences. We show that such seemingly insignificant changes in the input can cause large differences in the output, and suggest that data augmentation can remedy the problems.
A Comparative Study of Transformers on Word Sense Disambiguation
Authors: Avi Chawla, Nidhi Mulay, Vikas Bishnoi, Gaurav Dhama, Dr. Anil Kumar Singh
Date: 2021-11-30 00:00:00
Tasks: Word Embeddings, Word Sense Disambiguation

Recent years of research in Natural Language Processing (NLP) have witnessed dramatic growth in training large models for generating context-aware language representations. In this regard, numerous NLP systems have leveraged the power of neural network-based architectures to incorporate sense information in embeddings, resulting in Contextualized Word Embeddings (CWEs). Despite this progress, the NLP community has not witnessed any significant work performing a comparative study on the contextualization power of such architectures. This paper presents a comparative study and an extensive analysis of nine widely adopted Transformer models. These models are BERT, CTRL, DistilBERT, OpenAI-GPT, OpenAI-GPT2, Transformer-XL, XLNet, ELECTRA, and ALBERT. We evaluate their contextualization power using two lexical sample Word Sense Disambiguation (WSD) tasks, SensEval-2 and SensEval-3. We adopt a simple yet effective approach to WSD that uses a k-Nearest Neighbor (kNN) classification on CWEs. Experimental results show that the proposed techniques also achieve superior results over the current state-of-the-art on both the WSD tasks
KARL-Trans-NER: Knowledge Aware Representation Learning for Named Entity Recognition using Transformers
Authors: Avi Chawla, Nidhi Mulay, Vikas Bishnoi, Gaurav Dhama
Date: 2021-11-30 00:00:00
Tasks: Machine Translation, named-entity-recognition, Named Entity Recognition, Named Entity Recognition (NER), NER, Representation Learning, Text Summarization, Triplet, World Knowledge

The inception of modeling contextual information using models such as BERT, ELMo, and Flair has significantly improved representation learning for words. It has also given SOTA results in almost every NLP task - Machine Translation, Text Summarization and Named Entity Recognition, to name a few. In this work, in addition to using these dominant context-aware representations, we propose a Knowledge Aware Representation Learning (KARL) Network for Named Entity Recognition (NER). We discuss the challenges of using existing methods in incorporating world knowledge for NER and show how our proposed methods could be leveraged to overcome those challenges. KARL is based on a Transformer Encoder that utilizes large knowledge bases represented as fact triplets, converts them to a graph context, and extracts essential entity information residing inside to generate contextualized triplet representation for feature augmentation. Experimental results show that the augmentation done using KARL can considerably boost the performance of our NER system and achieve significantly better results than existing approaches in the literature on three publicly available NER datasets, namely CoNLL 2003, CoNLL++, and OntoNotes v5. We also observe better generalization and application to a real-world setting from KARL on unseen entities.
SimpleTRON: Simple Transformer with O(N) Complexity
Authors: Uladzislau Yorsh, Alexander Kovalenko, Vojtěch Vančura, Daniel Vašata, Pavel Kordík, Tomáš Mikolov
Date: 2021-11-23 00:00:00
Tasks: Text Classification

In this paper, we propose that the dot product pairwise matching attention layer, which is widely used in Transformer-based models, is redundant for the model performance. Attention, in its original formulation, has to be seen rather as a human-level tool to explore and/or visualize relevancy scores in sequential data. However, the way how it is constructed leads to significant computational complexity. Instead, we present SimpleTRON: Simple Transformer with O(N) Complexity, a simple and fast alternative without any approximation that, unlike other approximation models, does not have any architecture-related overhead and therefore can be seen as a purely linear Transformer-like model. This architecture, to the best of our knowledge, outperforms existing sub-quadratic attention approximation models on several tasks from the Long-Range Arena benchmark. Moreover, we show, that SimpleTRON can benefit from weight transfer from pretrained large language models, as its parameters can be fully transferable.
Text Mining Drug/Chemical-Protein Interactions using an Ensemble of BERT and T5 Based Models
Authors: Virginia Adams, Hoo-chang Shin, Carol Anderson, Bo Liu, Anas Abidin
Date: 2021-11-30 00:00:00
Tasks: Relation, Relation Extraction, Sentence, Sentence Classification

In Track-1 of the BioCreative VII Challenge participants are asked to identify interactions between drugs/chemicals and proteins. In-context named entity annotations for each drug/chemical and protein are provided and one of fourteen different interactions must be automatically predicted. For this relation extraction task, we attempt both a BERT-based sentence classification approach, and a more novel text-to-text approach using a T5 model. We find that larger BERT-based models perform better in general, with our BioMegatron-based model achieving the highest scores across all metrics, achieving 0.74 F1 score. Though our novel T5 text-to-text method did not perform as well as most of our BERT-based models, it outperformed those trained on similar data, showing promising results, achieving 0.65 F1 score. We believe a text-to-text approach to relation extraction has some competitive advantages and there is a lot of room for research advancement.
Chemical Identification and Indexing in PubMed Articles via BERT and Text-to-Text Approaches
Authors: Virginia Adams, Hoo-chang Shin, Carol Anderson, Bo Liu, Anas Abidin
Date: 2021-11-30 00:00:00
Tasks: Articles, Chemical Indexing, Entity Linking, Metric Learning, named-entity-recognition, Named Entity Recognition, Named Entity Recognition (NER), NER, Word Embeddings

The Biocreative VII Track-2 challenge consists of named entity recognition, entity-linking (or entity-normalization), and topic indexing tasks -- with entities and topics limited to chemicals for this challenge. Named entity recognition is a well-established problem and we achieve our best performance with BERT-based BioMegatron models. We extend our BERT-based approach to the entity linking task. After the second stage of pretraining BioBERT with a metric-learning loss strategy called self-alignment pretraining (SAP), we link entities based on the cosine similarity between their SAP-BioBERT word embeddings. Despite the success of our named entity recognition experiments, we find the chemical indexing task generally more challenging. In addition to conventional NER methods, we attempt both named entity recognition and entity linking with a novel text-to-text or "prompt" based method that uses generative language models such as T5 and GPT. We achieve encouraging results with this new approach.
Automatic Extraction of Medication Names in Tweets as Named Entity Recognition
Authors: Carol Anderson, Bo Liu, Anas Abidin, Hoo-chang Shin, Virginia Adams
Date: 2021-11-30 00:00:00
Tasks: named-entity-recognition, Named Entity Recognition, Named Entity Recognition (NER)

Social media posts contain potentially valuable information about medical conditions and health-related behavior. Biocreative VII Task 3 focuses on mining this information by recognizing mentions of medications and dietary supplements in tweets. We approach this task by fine tuning multiple BERT-style language models to perform token-level classification, and combining them into ensembles to generate final predictions. Our best system consists of five Megatron-BERT-345M models and achieves a strict F1 score of 0.764 on unseen test data.
A Natural Language Processing and Deep Learning based Model for Automated Vehicle Diagnostics using Free-Text Customer Service Reports
Authors: Ali Khodadadi, Soroush Ghandiparsi, Chen-Nee Chuah
Date: 2021-11-29 00:00:00
Tasks: Fault Detection, Specificity, valid

Initial fault detection and diagnostics are imperative measures to improve the efficiency, safety, and stability of vehicle operation. In recent years, numerous studies have investigated data-driven approaches to improve the vehicle diagnostics process using available vehicle data. Moreover, data-driven methods are employed to enhance customer-service agent interactions. In this study, we demonstrate a machine learning pipeline to improve automated vehicle diagnostics. First, Natural Language Processing (NLP) is used to automate the extraction of crucial information from free-text failure reports (generated during customers' calls to the service department). Then, deep learning algorithms are employed to validate service requests and filter vague or misleading claims. Ultimately, different classification algorithms are implemented to classify service requests so that valid service requests can be directed to the relevant service department. The proposed model- Bidirectional Long Short Term Memory (BiLSTM) along with Convolution Neural Network (CNN)- shows more than 18\% accuracy improvement in validating service requests compared to technicians' capabilities. In addition, using domain-based NLP techniques at preprocessing and feature extraction stages along with CNN-BiLSTM based request validation enhanced the accuracy ($>25\%$), sensitivity ($>39\%$), specificity ($>11\%$), and precision ($>11\%$) of Gradient Tree Boosting (GTB) service classification model. The Receiver Operating Characteristic Area Under the Curve (ROC-AUC) reached 0.82.
Undecidability in Finite Transducers, Defense Systems and Finite Substitutions
Authors: Vesa Halava
Date: 2021-11-30 00:00:00

In this manuscript we present a detailed proof for undecidability of the equivalence of finite substitutions on regular language $b\{0,1\}^*c$. The proof is based on the works of Leonid P. Lisovik.
A Causal U-net based Neural Beamforming Network for Real-Time Multi-Channel Speech Enhancement
Authors: Xinlei Ren, Xu Zhang, LianWu Chen, Xiguang Zheng, Chen Zhang, Liang Guo, Bing Yu
Date: 2021-08-01 00:00:00
Proceeding: INTERSPEECH 2021 2021 8
Tasks: CPU, Speech Enhancement

People are meeting through video conferencing more often. While single channel speech enhancement techniques are useful for the individual participants, the speech quality will be significantly degraded in large meeting rooms where the far-field and reverberate conditions are introduced. Approaches based on microphone array signal processing are proposed to explore the inter-channel correlation among the individual microphone channels. In this work, a new causal U-net based multiple-in-multiple-out structure is proposed for real-time multi-channel speech enhancement. The proposed method incorporates the traditional beamforming structure with the multi-channel causal U-net by explicitly adding a beamforming operation at the end of the neural beamformer. The proposed method has entered the INTERSPEECH Far-field Multi-Channel Speech Enhancement Challenge for Video Conferencing. With 1.97M model parameters and 0.25 real-time factor on Intel Core i7 (2.6GHz) CPU, the proposed method has outperforms the baseline system of this challenge on PESQ, Si-SNR and STOI metrics.
Quantum Algorithm for Ensemble Learning
Authors: Antonio Macaluso, Stefano Lodi, Claudio Sartori
Date: 2020-09-16 00:00:00
Proceeding: Italian Conference on Theoretical Computer Science 2020 9
Tasks: Ensemble Learning

The idea of ensemble learning is to build a prediction model by combining the strengths of a collection of simpler base models. Although they are extensively used, ensemble methods have high requirements in terms of memory and computational time. In this work, we propose a quantum algorithm that allows reproducing ensemble classification using bagging strategy. The algorithm generates many sub-samples in superposition, in such a way that only a single execution of a quantum classifier is required. In particular, the entanglement between a quantum register and different training sub-samples in superposition allows obtaining a sum of individual results which gives rise to the ensemble prediction. When considering the overall temporal cost of the algorithm, the single base classifier impacts additively rather than multiplicatively, as it usually happens in ensemble framework. Furthermore, given that the number of base models scales exponentially with the number of qubits of the control register, our algorithm opens up the possibility of exponential speed-up for quantum ensemble.
Automating cell counting in fluorescent microscopy through deep learning with c-ResUnet
Authors: Morelli, Roberto and Clissa, Luca and Amici, Roberto and Cerri, Matteo and Hitrec, Timna and Luppi, Marco and Rinaldi, Lorenzo and Squarcio, Fabio and Zoccoli, Antonio
Date: 2021-11-25 00:00:00
Proceeding: Scientific Reports 2021 11
Tasks: 2D Semantic Segmentation, Object Counting

Counting cells in fluorescent microscopy is a tedious, time-consuming task that researchers have to accomplish to assess the effects of different experimental conditions on biological structures of interest. Although such objects are generally easy to identify, the process of manually annotating cells is sometimes subject to fatigue errors and suffers from arbitrariness due to the operator’s interpretation of the borderline cases. We propose a Deep Learning approach that exploits a fully-convolutional network in a binary segmentation fashion to localize the objects of interest. Counts are then retrieved as the number of detected items. Specifically, we introduce a Unet-like architecture, cell ResUnet (c-ResUnet), and compare its performance against 3 similar architectures. In addition, we evaluate through ablation studies the impact of two design choices, (i) artifacts oversampling and (ii) weight maps that penalize the errors on cells boundaries increasingly with overcrowding. In summary, the c-ResUnet outperforms the competitors with respect to both detection and counting metrics (respectively, F1 score = 0.81 and MAE = 3.09). Also, the introduction of weight maps contribute to enhance performances, especially in presence of clumping cells, artifacts and confounding biological structures. Posterior qualitative assessment by domain experts corroborates previous results, suggesting human-level performance inasmuch even erroneous predictions seem to fall within the limits of operator interpretation. Finally, we release the pre-trained model and the annotated dataset to foster research in this and related fields.
RPS: Portfolio Asset Selection using Graph based Representation Learning
Authors: Mohammadamin Fazli, Parsa Alian, Ali Owfi, Erfan Loghmani
Date: 2021-11-28 00:00:00
Tasks: Clustering, Portfolio Optimization, Representation Learning

Portfolio optimization is one of the essential fields of focus in finance. There has been an increasing demand for novel computational methods in this area to compute portfolios with better returns and lower risks in recent years. We present a novel computational method called Representation Portfolio Selection (RPS) by redefining the distance matrix of financial assets using Representation Learning and Clustering algorithms for portfolio selection to increase diversification. RPS proposes a heuristic for getting closer to the optimal subset of assets. Using empirical results in this paper, we demonstrate that widely used portfolio optimization algorithms, such as MVO, CLA, and HRP, can benefit from our asset subset selection.
Network Traffic Shaping for Enhancing Privacy in IoT Systems
Authors: Sijie Xiong, Anand D. Sarwate, Narayan B. Mandayam
Date: 2021-11-29 00:00:00

Motivated by privacy issues caused by inference attacks on user activities in the packet sizes and timing information of Internet of Things (IoT) network traffic, we establish a rigorous event-level differential privacy (DP) model on infinite packet streams. We propose a memoryless traffic shaping mechanism satisfying a first-come-first-served queuing discipline that outputs traffic dependent on the input using a DP mechanism. We show that in special cases the proposed mechanism recovers existing shapers which standardize the output independently from the input. To find the optimal shapers for given levels of privacy and transmission efficiency, we formulate the constrained problem of minimizing the expected delay per packet and propose using the expected queue size across time as a proxy. We further show that the constrained minimization is a convex program. We demonstrate the effect of shapers on both synthetic data and packet traces from actual IoT devices. The experimental results reveal inherent privacy-overhead tradeoffs: more shaping overhead provides better privacy protection. Under the same privacy level, there naturally exists a tradeoff between dummy traffic and delay. When dealing with heavier or less bursty input traffic, all shapers become more overhead-efficient. We also show that increased traffic from a larger number of IoT devices makes guaranteeing event-level privacy easier. The DP shaper offers tunable privacy that is invariant with the change in the input traffic distribution and has an advantage in handling burstiness over traffic-independent shapers. This approach well accommodates heterogeneous network conditions and enables users to adapt to their privacy/overhead demands.
Fitness landscape adaptation in open replicator systems with competition: application to cancer therapy
Authors: Igor Samokhin, Tatiana Yakushkina, Dmitry Markin, Alexander S. Bratus
Date: 2021-11-29 00:00:00

This study focuses on open quasispecies systems with competition and death flow, described by modified Eigen and Crow-Kimura models. We examine the evolutionary adaptation process as a reaction to changes in rates. One of the fundamental assumptions, which forms the basis of our mathematical model, is the existence of two different timescales: internal dynamics time and evolutionary time. The latter is much slower and exhibits significant adaptation events. These conditions allow us to represent the whole evolutionary process through a series of steady-state equations, where all the elements continuously depend on the evolutionary parameter.