pwc-archive/papers-with-abstracts huggingface dataset:
Transformer-based Korean Pretrained Language Models: A Survey on Three Years of Progress
Authors: Kichang Yang
Date: 2021-11-25 00:00:00
Tasks: Decoder, Language Modeling, Language Modelling, Translation

With the advent of Transformer, which was used in translation models in 2017, attention-based architectures began to attract attention. Furthermore, after the emergence of BERT, which strengthened the NLU-specific encoder part, which is a part of the Transformer, and the GPT architecture, which strengthened the NLG-specific decoder part, various methodologies, data, and models for learning the Pretrained Language Model began to appear. Furthermore, in the past three years, various Pretrained Language Models specialized for Korean have appeared. In this paper, we intend to numerically and qualitatively compare and analyze various Korean PLMs released to the public.
Does constituency analysis enhance domain-specific pre-trained BERT models for relation extraction?
Authors: Anfu Tang, Louise Deléger, Robert Bossy, Pierre Zweigenbaum, Claire Nédellec
Date: 2021-11-25 00:00:00
Tasks: DrugProt, Relation, Relation Extraction

Recently many studies have been conducted on the topic of relation extraction. The DrugProt track at BioCreative VII provides a manually-annotated corpus for the purpose of the development and evaluation of relation extraction systems, in which interactions between chemicals and genes are studied. We describe the ensemble system that we used for our submission, which combines predictions of fine-tuned bioBERT, sciBERT and const-bioBERT models by majority voting. We specifically tested the contribution of syntactic information to relation extraction with BERT. We observed that adding constituentbased syntactic information to BERT improved precision, but decreased recall, since relations rarely seen in the train set were less likely to be predicted by BERT models in which the syntactic information is infused. Our code is available online [https://github.com/Maple177/drugprot-relation-extraction].
Global alignment for relation extraction in Microbiology
Authors: Anfu Tang, Claire Nédellec, Pierre Zweigenbaum, Louise Deléger, Robert Bossy
Date: 2021-11-25 00:00:00
Tasks: Relation, Relation Extraction

We investigate a method to extract relations from texts based on global alignment and syntactic information. Combined with SVM, this method is shown to have a performance comparable or even better than LSTM on two RE tasks.
HTMOT : Hierarchical Topic Modelling Over Time
Authors: Judicael Poumay, Ashwin Ittoo
Date: 2021-11-22 00:00:00
Tasks: Topic Models

Over the years, topic models have provided an efficient way of extracting insights from text. However, while many models have been proposed, none are able to model topic temporality and hierarchy jointly. Modelling time provide more precise topics by separating lexically close but temporally distinct topics while modelling hierarchy provides a more detailed view of the content of a document corpus. In this study, we therefore propose a novel method, HTMOT, to perform Hierarchical Topic Modelling Over Time. We train HTMOT using a new implementation of Gibbs sampling, which is more efficient. Specifically, we show that only applying time modelling to deep sub-topics provides a way to extract specific stories or events while high level topics extract larger themes in the corpus. Our results show that our training procedure is fast and can extract accurate high-level topics and temporally precise sub-topics. We measured our model's performance using the Word Intrusion task and outlined some limitations of this evaluation method, especially for hierarchical models. As a case study, we focused on the various developments in the space industry in 2020.
Keyword Assisted Embedded Topic Model
Authors: Bahareh Harandizadeh, J. Hunter Priniski, Fred Morstatter
Date: 2021-11-22 00:00:00
Tasks: model, Topic Models, Word Embeddings

By illuminating latent structures in a corpus of text, topic models are an essential tool for categorizing, summarizing, and exploring large collections of documents. Probabilistic topic models, such as latent Dirichlet allocation (LDA), describe how words in documents are generated via a set of latent distributions called topics. Recently, the Embedded Topic Model (ETM) has extended LDA to utilize the semantic information in word embeddings to derive semantically richer topics. As LDA and its extensions are unsupervised models, they aren't defined to make efficient use of a user's prior knowledge of the domain. To this end, we propose the Keyword Assisted Embedded Topic Model (KeyETM), which equips ETM with the ability to incorporate user knowledge in the form of informative topic-level priors over the vocabulary. Using both quantitative metrics and human responses on a topic intrusion task, we demonstrate that KeyETM produces better topics than other guided, generative models in the literature.
Isomer: Transfer enhanced Dual-Channel Heterogeneous Dependency Attention Network for Aspect-based Sentiment Classification
Authors: Yukun Cao, Yijia Tang, Ziyue Wei, ChengKun Jin, Zeyu Miao, Yixin Fang, Haizhou Du, Feifei Xu
Date: 2021-11-21 00:00:00
Tasks: Sentence, Sentiment Analysis, Sentiment Classification

Aspect-based sentiment classification aims to predict the sentiment polarity of a specific aspect in a sentence. However, most existing methods attempt to construct dependency relations into a homogeneous dependency graph with the sparsity and ambiguity, which cannot cover the comprehensive contextualized features of short texts or consider any additional node types or semantic relation information. To solve those issues, we present a sentiment analysis model named Isomer, which performs a dual-channel attention on heterogeneous dependency graphs incorporating external knowledge, to effectively integrate other additional information. Specifically, a transfer-enhanced dual-channel heterogeneous dependency attention network is devised in Isomer to model short texts using heterogeneous dependency graphs. These heterogeneous dependency graphs not only consider different types of information but also incorporate external knowledge. Experiments studies show that our model outperforms recent models on benchmark datasets. Furthermore, the results suggest that our method captures the importance of various information features to focus on informative contextual words.
Weakly Supervised Prototype Topic Model with Discriminative Seed Words: Modifying the Category Prior by Self-exploring Supervised Signals
Authors: Bing Wang, Yue Wang, Ximing Li, Jihong Ouyang
Date: 2021-11-20 00:00:00
Tasks: text-classification, Text Classification, Weakly-supervised Learning

Dataless text classification, i.e., a new paradigm of weakly supervised learning, refers to the task of learning with unlabeled documents and a few predefined representative words of categories, known as seed words. The recent generative dataless methods construct document-specific category priors by using seed word occurrences only, however, such category priors often contain very limited and even noisy supervised signals. To remedy this problem, in this paper we propose a novel formulation of category prior. First, for each document, we consider its label membership degree by not only counting seed word occurrences, but also using a novel prototype scheme, which captures pseudo-nearest neighboring categories. Second, for each label, we consider its frequency prior knowledge of the corpus, which is also a discriminative knowledge for classification. By incorporating the proposed category prior into the previous generative dataless method, we suggest a novel generative dataless method, namely Weakly Supervised Prototype Topic Model (WSPTM). The experimental results on real-world datasets demonstrate that WSPTM outperforms the existing baseline methods.
How to Build Robust FAQ Chatbot with Controllable Question Generator?
Authors: Yan Pan, Mingyang Ma, Bernhard Pflugfelder, Georg Groh
Date: 2021-11-18 00:00:00
Tasks: Chatbot, Diversity, Passage Retrieval, Question Generation, Question-Generation, Retrieval, valid

Many unanswerable adversarial questions fool the question-answer (QA) system with some plausible answers. Building a robust, frequently asked questions (FAQ) chatbot needs a large amount of diverse adversarial examples. Recent question generation methods are ineffective at generating many high-quality and diverse adversarial question-answer pairs from unstructured text. We propose the diversity controllable semantically valid adversarial attacker (DCSA), a high-quality, diverse, controllable method to generate standard and adversarial samples with a semantic graph. The fluent and semantically generated QA pairs fool our passage retrieval model successfully. After that, we conduct a study on the robustness and generalization of the QA model with generated QA pairs among different domains. We find that the generated data set improves the generalizability of the QA model to the new target domain and the robustness of the QA model to detect unanswerable adversarial questions.
How News Evolves? Modeling News Text and Coverage using Graphs and Hawkes Process
Authors: Honggen Zhang, June Zhang
Date: 2021-11-18 00:00:00
Tasks: Articles, Time Series, Time Series Analysis

Monitoring news content automatically is an important problem. The news content, unlike traditional text, has a temporal component. However, few works have explored the combination of natural language processing and dynamic system models. One reason is that it is challenging to mathematically model the nuances of natural language. In this paper, we discuss how we built a novel dataset of news articles collected over time. Then, we present a method of converting news text collected over time to a sequence of directed multi-graphs, which represent semantic triples (Subject -> Predicate} ->Object). We model the dynamics of specific topological changes in these graphs using a set of multivariate count series, which we fit the discrete-time Hawkes process. With our real-world data, we show that the multivariate time series contain both dynamic information of how many articles/words were published each day and semantic information of the content of the articles. This yields novel insights into how news events are covered. We show with the experiment that our approach can be used to infer from a sequence of news articles if the articles were published by major or entertainment news outlets.
Intelligent Trading Systems: A Sentiment-Aware Reinforcement Learning Approach
Authors: Francisco Caio Lima Paiva, Leonardo Kanashiro Felizardo, Reinaldo Augusto da Costa Bianchi, Anna Helena Reali Costa
Date: 2021-11-14 00:00:00
Tasks: Algorithmic Trading, General Reinforcement Learning, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL), Sentiment Analysis, Time Series, Time Series Analysis

The feasibility of making profitable trades on a single asset on stock exchanges based on patterns identification has long attracted researchers. Reinforcement Learning (RL) and Natural Language Processing have gained notoriety in these single-asset trading tasks, but only a few works have explored their combination. Moreover, some issues are still not addressed, such as extracting market sentiment momentum through the explicit capture of sentiment features that reflect the market condition over time and assessing the consistency and stability of RL results in different situations. Filling this gap, we propose the Sentiment-Aware RL (SentARL) intelligent trading system that improves profit stability by leveraging market mood through an adaptive amount of past sentiment features drawn from textual news. We evaluated SentARL across twenty assets, two transaction costs, and five different periods and initializations to show its consistent effectiveness against baselines. Subsequently, this thorough assessment allowed us to identify the boundary between news coverage and market sentiment regarding the correlation of price-time series above which SentARL's effectiveness is outstanding.
Cross-validation for change-point regression: pitfalls and solutions
Authors: Florian Pein, Rajen D. Shah
Date: 2021-12-06 00:00:00
Tasks: regression

Cross-validation is the standard approach for tuning parameter selection in many non-parametric regression problems. However its use is less common in change-point regression, perhaps as its prediction error-based criterion may appear to permit small spurious changes and hence be less well-suited to estimation of the number and location of change-points. We show that in fact the problems of cross-validation with squared error loss are more severe and can lead to systematic under- or over-estimation of the number of change-points, and highly suboptimal estimation of the mean function in simple settings where changes are easily detectable. We propose two simple approaches to remedy these issues, the first involving the use of absolute error rather than squared error loss, and the second involving modifying the holdout sets used. For the latter, we provide conditions that permit consistent estimation of the number of change-points for a general change-point estimation procedure. We show these conditions are satisfied for least squares estimation using new results on its performance when supplied with the incorrect number of change-points. Numerical experiments show that our new approaches are competitive with common change-point methods using classical tuning parameter choices when error distributions are well-specified, but can substantially outperform these in misspecified models. An implementation of our methodology is available in the R package crossvalidationCP on CRAN.
Quantum Machine Learning for Radio Astronomy
Authors: Mohammad Kordzanganeh, Aydin Utting, Anna Scaife
Date: 2021-12-05 00:00:00
Tasks: Astronomy, BIG-bench Machine Learning, Classification, Quantum Machine Learning

In this work we introduce a novel approach to the pulsar classification problem in time-domain radio astronomy using a Born machine, often referred to as a quantum neural network. Using a single-qubit architecture, we show that the pulsar classification problem maps well to the Bloch sphere and that comparable accuracies to more classical machine learning approaches are achievable. We introduce a novel single-qubit encoding for the pulsar data used in this work and show that this performs comparably to a multi-qubit QAOA encoding.
DNN-based Policies for Stochastic AC OPF
Authors: Sarthak Gupta, Sidhant Misra, Deepjyoti Deka, Vassilis Kekatos
Date: 2021-12-04 00:00:00

A prominent challenge to the safe and optimal operation of the modern power grid arises due to growing uncertainties in loads and renewables. Stochastic optimal power flow (SOPF) formulations provide a mechanism to handle these uncertainties by computing dispatch decisions and control policies that maintain feasibility under uncertainty. Most SOPF formulations consider simple control policies such as affine policies that are mathematically simple and resemble many policies used in current practice. Motivated by the efficacy of machine learning (ML) algorithms and the potential benefits of general control policies for cost and constraint enforcement, we put forth a deep neural network (DNN)-based policy that predicts the generator dispatch decisions in real time in response to uncertainty. The weights of the DNN are learnt using stochastic primal-dual updates that solve the SOPF without the need for prior generation of training labels and can explicitly account for the feasibility constraints in the SOPF. The advantages of the DNN policy over simpler policies and their efficacy in enforcing safety limits and producing near optimal solutions are demonstrated in the context of a chance constrained formulation on a number of test cases.
Generalized Likelihood Ratio Test for Adversarially Robust Hypothesis Testing
Authors: Bhagyashree Puranik, Upamanyu Madhow, Ramtin Pedarsani
Date: 2021-12-04 00:00:00

Machine learning models are known to be susceptible to adversarial attacks which can cause misclassification by introducing small but well designed perturbations. In this paper, we consider a classical hypothesis testing problem in order to develop fundamental insight into defending against such adversarial perturbations. We interpret an adversarial perturbation as a nuisance parameter, and propose a defense based on applying the generalized likelihood ratio test (GLRT) to the resulting composite hypothesis testing problem, jointly estimating the class of interest and the adversarial perturbation. While the GLRT approach is applicable to general multi-class hypothesis testing, we first evaluate it for binary hypothesis testing in white Gaussian noise under $\ell_{\infty}$ norm-bounded adversarial perturbations, for which a known minimax defense optimizing for the worst-case attack provides a benchmark. We derive the worst-case attack for the GLRT defense, and show that its asymptotic performance (as the dimension of the data increases) approaches that of the minimax defense. For non-asymptotic regimes, we show via simulations that the GLRT defense is competitive with the minimax approach under the worst-case attack, while yielding a better robustness-accuracy tradeoff under weaker attacks. We also illustrate the GLRT approach for a multi-class hypothesis testing problem, for which a minimax strategy is not known, evaluating its performance under both noise-agnostic and noise-aware adversarial settings, by providing a method to find optimal noise-aware attacks, and heuristics to find noise-agnostic attacks that are close to optimal in the high SNR regime.
Data Fusion with Latent Map Gaussian Processes
Authors: Nicholas Oune, Jonathan Tammer Eweis-Labolle, Ramin Bostanabad
Date: 2021-12-04 00:00:00
Tasks: Gaussian Processes

Multi-fidelity modeling and calibration are data fusion tasks that ubiquitously arise in engineering design. In this paper, we introduce a novel approach based on latent-map Gaussian processes (LMGPs) that enables efficient and accurate data fusion. In our approach, we convert data fusion into a latent space learning problem where the relations among different data sources are automatically learned. This conversion endows our approach with attractive advantages such as increased accuracy, reduced costs, flexibility to jointly fuse any number of data sources, and ability to visualize correlations between data sources. This visualization allows the user to detect model form errors or determine the optimum strategy for high-fidelity emulation by fitting LMGP only to the subset of the data sources that are well-correlated. We also develop a new kernel function that enables LMGPs to not only build a probabilistic multi-fidelity surrogate but also estimate calibration parameters with high accuracy and consistency. The implementation and use of our approach are considerably simpler and less prone to numerical issues compared to existing technologies. We demonstrate the benefits of LMGP-based data fusion by comparing its performance against competing methods on a wide range of examples.
Distributed Adaptive Learning Under Communication Constraints
Authors: Marco Carpentiero, Vincenzo Matta, Ali H. Sayed
Date: 2021-12-03 00:00:00

This work examines adaptive distributed learning strategies designed to operate under communication constraints. We consider a network of agents that must solve an online optimization problem from continual observation of streaming data. The agents implement a distributed cooperative strategy where each agent is allowed to perform local exchange of information with its neighbors. In order to cope with communication constraints, the exchanged information must be unavoidably compressed. We propose a diffusion strategy nicknamed as ACTC (Adapt-Compress-Then-Combine), which relies on the following steps: i) an adaptation step where each agent performs an individual stochastic-gradient update with constant step-size; ii) a compression step that leverages a recently introduced class of stochastic compression operators; and iii) a combination step where each agent combines the compressed updates received from its neighbors. The distinguishing elements of this work are as follows. First, we focus on adaptive strategies, where constant (as opposed to diminishing) step-sizes are critical to respond in real time to nonstationary variations. Second, we consider the general class of directed graphs and left-stochastic combination policies, which allow us to enhance the interplay between topology and learning. Third, in contrast with related works that assume strong convexity for all individual agents' cost functions, we require strong convexity only at a network level, a condition satisfied even if a single agent has a strongly-convex cost and the remaining agents have non-convex costs. Fourth, we focus on a diffusion (as opposed to consensus) strategy. Under the demanding setting of compressed information, we establish that the ACTC iterates fluctuate around the desired optimizer, achieving remarkable savings in terms of bits exchanged between neighboring agents.
MatMat: Matrix Factorization by Matrix Fitting
Authors: Hao Wang
Date: 2021-12-06 00:00:00
Tasks: Fairness, Multi-Task Learning, Recommendation Systems

Matrix factorization is a widely adopted recommender system technique that fits scalar rating values by dot products of user feature vectors and item feature vectors. However, the formulation of matrix factorization as a scalar fitting problem is not friendly to side information incorporation or multi-task learning. In this paper, we replace the scalar values of the user rating matrix by matrices, and fit the matrix values by matrix products of user feature matrix and item feature matrix. Our framework is friendly to multitask learning and side information incorporation. We use popularity data as side information in our paper in particular to enhance the performance of matrix factorization techniques. In the experiment section, we prove the competence of our method compared with other approaches using both accuracy and fairness metrics. Our framework is an ideal substitute for tensor factorization in context-aware recommendation and many other scenarios.
ZeroMat: Solving Cold-start Problem of Recommender System with No Input Data
Authors: Hao Wang
Date: 2021-12-06 00:00:00
Tasks: Fairness, Recommendation Systems

Recommender system is an applicable technique in most E-commerce commercial product technical designs. However, nearly all recommender system faces a challenge called the cold-start problem. The problem is so notorious that almost every industrial practitioner needs to resolve this issue when building recommender systems. Most cold-start problem solvers need some kind of data input as the starter of the system. On the other hand, many real-world applications place popular items or random items as recommendation results. In this paper, we propose a new technique called ZeroMat that requries no input data at all and predicts the user item rating data that is competitive in Mean Absolute Error and fairness metric compared with the classic matrix factorization with affluent data, and much better performance than random placement.