pwc-archive/papers-with-abstracts huggingface dataset:
I still have Time(s): Extending HeidelTime for German Texts
Authors: Andy Lücking, Manuel Stoeckel, Giuseppe Abrami, Alexander Mehler
Date: 2022-04-19 00:00:00
Proceeding: LREC 2022 6

HeidelTime is one of the most widespread and successful tools for detecting temporal expressions in texts. Since HeidelTime's pattern matching system is based on regular expression, it can be extended in a convenient way. We present such an extension for the German resources of HeidelTime: HeidelTime-EXT . The extension has been brought about by means of observing false negatives within real world texts and various time banks. The gain in coverage is 2.7% or 8.5%, depending on the admitted degree of potential overgeneralization. We describe the development of HeidelTime-EXT, its evaluation on text samples from various genres, and share some linguistic observations. HeidelTime ext can be obtained from https://github.com/texttechnologylab/heideltime.
Impact of Tokenization on Language Models: An Analysis for Turkish
Authors: Cagri Toraman, Eyup Halit Yilmaz, Furkan Şahinuç, Oguzhan Ozcelik
Date: 2022-04-19 00:00:00

Tokenization is an important text preprocessing step to prepare input tokens for deep language models. WordPiece and BPE are de facto methods employed by important models, such as BERT and GPT. However, the impact of tokenization can be different for morphologically rich languages, such as Turkic languages, where many words can be generated by adding prefixes and suffixes. We compare five tokenizers at different granularity levels, i.e. their outputs vary from smallest pieces of characters to the surface form of words, including a Morphological-level tokenizer. We train these tokenizers and pretrain medium-sized language models using RoBERTa pretraining procedure on the Turkish split of the OSCAR corpus. We then fine-tune our models on six downstream tasks. Our experiments, supported by statistical tests, reveal that Morphological-level tokenizer has challenging performance with de facto tokenizers. Furthermore, we find that increasing the vocabulary size improves the performance of Morphological and Word-level tokenizers more than that of de facto tokenizers. The ratio of the number of vocabulary parameters to the total number of model parameters can be empirically chosen as 20% for de facto tokenizers and 40% for other tokenizers to obtain a reasonable trade-off between model size and performance.
Probing for the Usage of Grammatical Number
Authors: Karim Lasri, Tiago Pimentel, Alessandro Lenci, Thierry Poibeau, Ryan Cotterell
Date: 2022-04-19 00:00:00
Proceeding: ACL 2022 5

A central quest of probing is to uncover how pre-trained models encode a linguistic property within their representations. An encoding, however, might be spurious-i.e., the model might not rely on it when making predictions. In this paper, we try to find encodings that the model actually uses, introducing a usage-based probing setup. We first choose a behavioral task which cannot be solved without using the linguistic property. Then, we attempt to remove the property by intervening on the model's representations. We contend that, if an encoding is used by the model, its removal should harm the performance on the chosen behavioral task. As a case study, we focus on how BERT encodes grammatical number, and on how it uses this encoding to solve the number agreement task. Experimentally, we find that BERT relies on a linear encoding of grammatical number to produce the correct behavioral output. We also find that BERT uses a separate encoding of grammatical number for nouns and verbs. Finally, we identify in which layers information about grammatical number is transferred from a noun to its head verb.
Understanding Toxicity Triggers on Reddit in the Context of Singapore
Authors: Yun Yu Chong, Haewoon Kwak
Date: 2022-04-19 00:00:00

While the contagious nature of online toxicity sparked increasing interest in its early detection and prevention, most of the literature focuses on the Western world. In this work, we demonstrate that 1) it is possible to detect toxicity triggers in an Asian online community, and 2) toxicity triggers can be strikingly different between Western and Eastern contexts.
Generating Authentic Adversarial Examples beyond Meaning-preserving with Doubly Round-trip Translation
Authors: Siyu Lai, Zhen Yang, Fandong Meng, Xue Zhang, Yufeng Chen, Jinan Xu, Jie zhou
Date: 2022-04-19 00:00:00
Proceeding: NAACL 2022 7
Tasks: Machine Translation, NMT, Translation

Generating adversarial examples for Neural Machine Translation (NMT) with single Round-Trip Translation (RTT) has achieved promising results by releasing the meaning-preserving restriction. However, a potential pitfall for this approach is that we cannot decide whether the generated examples are adversarial to the target NMT model or the auxiliary backward one, as the reconstruction error through the RTT can be related to either. To remedy this problem, we propose a new criterion for NMT adversarial examples based on the Doubly Round-Trip Translation (DRTT). Specifically, apart from the source-target-source RTT, we also consider the target-source-target one, which is utilized to pick out the authentic adversarial examples for the target NMT model. Additionally, to enhance the robustness of the NMT model, we introduce the masked language models to construct bilingual adversarial pairs based on DRTT, which are used to train the NMT model directly. Extensive experiments on both the clean and noisy test sets (including the artificial and natural noise) show that our approach substantially improves the robustness of NMT models.
DecBERT: Enhancing the Language Understanding of BERT with Causal Attention Masks
Authors: Ziyang Luo, Yadong Xi, Jing Ma, Zhiwei Yang, Xiaoxi Mao, Changjie Fan, Rongsheng Zhang
Date: 2022-04-19 00:00:00
Proceeding: Findings (NAACL) 2022 7
Tasks: Decoder, Language Modeling, Language Modelling, Position

Since 2017, the Transformer-based models play critical roles in various downstream Natural Language Processing tasks. However, a common limitation of the attention mechanism utilized in Transformer Encoder is that it cannot automatically capture the information of word order, so explicit position embeddings are generally required to be fed into the target model. In contrast, Transformer Decoder with the causal attention masks is naturally sensitive to the word order. In this work, we focus on improving the position encoding ability of BERT with the causal attention masks. Furthermore, we propose a new pre-trained language model DecBERT and evaluate it on the GLUE benchmark. Experimental results show that (1) the causal attention mask is effective for BERT on the language understanding tasks; (2) our DecBERT model without position embeddings achieve comparable performance on the GLUE benchmark; and (3) our modification accelerates the pre-training process and DecBERT w/ PE achieves better overall performance than the baseline systems when pre-training with the same amount of computational resources.
Multilingual Syntax-aware Language Modeling through Dependency Tree Conversion
Authors: Shunsuke Kando, Hiroshi Noji, Yusuke Miyao
Date: 2022-04-19 00:00:00
Proceeding: spnlp (ACL) 2022 5
Tasks: Language Modeling, Language Modelling

Incorporating stronger syntactic biases into neural language models (LMs) is a long-standing goal, but research in this area often focuses on modeling English text, where constituent treebanks are readily available. Extending constituent tree-based LMs to the multilingual setting, where dependency treebanks are more common, is possible via dependency-to-constituency conversion methods. However, this raises the question of which tree formats are best for learning the model, and for which languages. We investigate this question by training recurrent neural network grammars (RNNGs) using various conversion methods, and evaluating them empirically in a multilingual setting. We examine the effect on LM performance across nine conversion methods and five languages through seven types of syntactic tests. On average, the performance of our best model represents a 19 \% increase in accuracy over the worst choice across all languages. Our best model shows the advantage over sequential/overparameterized LMs, suggesting the positive effect of syntax injection in a multilingual setting. Our experiments highlight the importance of choosing the right tree formalism, and provide insights into making an informed decision.
Imagination-Augmented Natural Language Understanding
Authors: Yujie Lu, Wanrong Zhu, Xin Eric Wang, Miguel Eckstein, William Yang Wang
Date: 2022-04-18 00:00:00
Proceeding: NAACL 2022 7
Tasks: Natural Language Understanding

Human brains integrate linguistic and perceptual information simultaneously to understand natural language, and hold the critical ability to render imaginations. Such abilities enable us to construct new abstract concepts or concrete objects, and are essential in involving practical knowledge to solve problems in low-resource scenarios. However, most existing methods for Natural Language Understanding (NLU) are mainly focused on textual signals. They do not simulate human visual imagination ability, which hinders models from inferring and learning efficiently from limited data samples. Therefore, we introduce an Imagination-Augmented Cross-modal Encoder (iACE) to solve natural language understanding tasks from a novel learning perspective -- imagination-augmented cross-modal understanding. iACE enables visual imagination with external knowledge transferred from the powerful generative and pre-trained vision-and-language models. Extensive experiments on GLUE and SWAG show that iACE achieves consistent improvement over visually-supervised pre-trained models. More importantly, results in extreme and normal few-shot settings validate the effectiveness of iACE in low-resource natural language understanding circumstances.
Co-evolutionary Probabilistic Structured Grammatical Evolution
Authors: Jessica Mégane, Nuno Lourenço, Penousal Machado
Date: 2022-04-19 00:00:00

This work proposes an extension to Structured Grammatical Evolution (SGE) called Co-evolutionary Probabilistic Structured Grammatical Evolution (Co-PSGE). In Co-PSGE each individual in the population is composed by a grammar and a genotype, which is a list of dynamic lists, each corresponding to a non-terminal of the grammar containing real numbers that correspond to the probability of choosing a derivation rule. Each individual uses its own grammar to map the genotype into a program. During the evolutionary process, both the grammar and the genotype are subject to variation operators. The performance of the proposed approach is compared to 3 different methods, namely, Grammatical Evolution (GE), Probabilistic Grammatical Evolution (PGE), and SGE on four different benchmark problems. The results show the effectiveness of the approach since Co-PSGE is able to outperform all the methods with statistically significant differences in the majority of the problems.
Multi-level Cross-view Contrastive Learning for Knowledge-aware Recommender System
Authors: Ding Zou, Wei Wei, Xian-Ling Mao, Ziyang Wang, Minghui Qiu, Feida Zhu, Xin Cao
Date: 2022-04-19 00:00:00
Tasks: Contrastive Learning, Data Augmentation, graph construction, Knowledge-Aware Recommendation, Recommendation Systems

Knowledge graph (KG) plays an increasingly important role in recommender systems. Recently, graph neural networks (GNNs) based model has gradually become the theme of knowledge-aware recommendation (KGR). However, there is a natural deficiency for GNN-based KGR models, that is, the sparse supervised signal problem, which may make their actual performance drop to some extent. Inspired by the recent success of contrastive learning in mining supervised signals from data itself, in this paper, we focus on exploring the contrastive learning in KG-aware recommendation and propose a novel multi-level cross-view contrastive learning mechanism, named MCCLK. Different from traditional contrastive learning methods which generate two graph views by uniform data augmentation schemes such as corruption or dropping, we comprehensively consider three different graph views for KG-aware recommendation, including global-level structural view, local-level collaborative and semantic views. Specifically, we consider the user-item graph as a collaborative view, the item-entity graph as a semantic view, and the user-item-entity graph as a structural view. MCCLK hence performs contrastive learning across three views on both local and global levels, mining comprehensive graph feature and structure information in a self-supervised manner. Besides, in semantic view, a k-Nearest-Neighbor (kNN) item-item semantic graph construction module is proposed, to capture the important item-item semantic relation which is usually ignored by previous work. Extensive experiments conducted on three benchmark datasets show the superior performance of our proposed method over the state-of-the-arts. The implementations are available at: https://github.com/CCIIPLab/MCCLK.
Enhancing CTR Prediction with Context-Aware Feature Representation Learning
Authors: Fangye Wang, Yingxu Wang, Dongsheng Li, Hansu Gu, Tun Lu, Peng Zhang, Ning Gu
Date: 2022-04-19 00:00:00
Tasks: Click-Through Rate Prediction, Representation Learning

CTR prediction has been widely used in the real world. Many methods model feature interaction to improve their performance. However, most methods only learn a fixed representation for each feature without considering the varying importance of each feature under different contexts, resulting in inferior performance. Recently, several methods tried to learn vector-level weights for feature representations to address the fixed representation issue. However, they only produce linear transformations to refine the fixed feature representations, which are still not flexible enough to capture the varying importance of each feature under different contexts. In this paper, we propose a novel module named Feature Refinement Network (FRNet), which learns context-aware feature representations at bit-level for each feature in different contexts. FRNet consists of two key components: 1) Information Extraction Unit (IEU), which captures contextual information and cross-feature relationships to guide context-aware feature refinement; and 2) Complementary Selection Gate (CSGate), which adaptively integrates the original and complementary feature representations learned in IEU with bit-level weights. Notably, FRNet is orthogonal to existing CTR methods and thus can be applied in many existing methods to boost their performance. Comprehensive experiments are conducted to verify the effectiveness, efficiency, and compatibility of FRNet.
Partial Relaxed Optimal Transport for Denoised Recommendation
Authors: Yanchao Tan, Carl Yang Member, Xiangyu Wei, Ziyue Wu, Xiaolin Zheng
Date: 2022-04-19 00:00:00
Tasks: Denoising, Recommendation Systems

The interaction data used by recommender systems (RSs) inevitably include noises resulting from mistaken or exploratory clicks, especially under implicit feedbacks. Without proper denoising, RS models cannot effectively capture users' intrinsic preferences and the true interactions between users and items. To address such noises, existing methods mostly rely on auxiliary data which are not always available. In this work, we ground on Optimal Transport (OT) to globally match a user embedding space and an item embedding space, allowing both non-deep and deep RS models to discriminate intrinsic and noisy interactions without supervision. Specifically, we firstly leverage the OT framework via Sinkhorn distance to compute the continuous many-to-many user-item matching scores. Then, we relax the regularization in Sinkhorn distance to achieve a closed-form solution with a reduced time complexity. Finally, to consider individual user behaviors for denoising, we develop a partial OT framework to adaptively relabel user-item interactions through a personalized thresholding mechanism. Extensive experiments show that our framework can significantly boost the performances of existing RS models.
Dynamic process fault prediction using canonical variable trend analysis
Authors: Yongping Hu, Xiaogang Deng, Yuping Cao, Xuemin Tian
Date: 2016-04-12 00:00:00
Proceeding: DV 2016 4
Tasks: Prediction, Time Series, Time Series Analysis, Time Series Prediction

Fault prediction technology is important to avoid 
serious process failure. This paper is concerned with the fault 
prediction of dynamic industrial process with incipient faults and 
proposes a canonical variable trend analysis (CVTA) based fault 
prediction method. In the proposed method, canonical variate 
analysis (CVA) algorithm is firstly applied to analyze the process 
dynamics and extract the uncorrelated latent features, called 
canonical variables. Furthermore, support vector machine is 
adopted to model the relationship between the historical and 
future values of the canonical variables, which leads to the time 
series prediction model for the canonical variables. Based on the 
predicted canonical variables, an overall monitoring statistic is 
used to forecast the change of the process status. Simulations on a 
continuous stirred tank reactor (CSTR) system demonstrate that 
the proposed method can indicate the trend of the incipient faults 
effectively.
Canonical Variate Dissimilarity Analysis for Process Incipient Fault Detection
Authors: Karl Ezra Salgado Pilarioand Yi Cao
Date: 2016-04-12 00:00:00
Proceeding: DV 2016 4
Tasks: Fault Detection

Early detection of incipient faults in industrial
processes is increasingly becoming important, as these
faults can slowly develop into serious abnormal events,
an emergency situation, or even failure of critical equip-
ment. Multivariate statistical process monitoring methods
are currently established for abrupt fault detection. Among
these, the canonical variate analysis (CVA) was proven to
be effective for dynamic process monitoring. However, the
traditional CVA indices may not be sensitive enough for
incipient faults. In this work, an extension of CVA, called
the canonical variate dissimilarity analysis (CVDA), is pro-
posed for process incipient fault detection in nonlinear dy-
namic processes under varying operating conditions. To
handle the non-Gaussian distributed data, the kernel den-
sity estimation was used for computing detection limits. A
CVA dissimilarity based index has been demonstrated to
outperform traditional CVA indices and other dissimilarity-
based indices, namely the dissimilarity analysis, recursive
dynamic transformed component statistical analysis, and
generalized canonical correlation analysis, in terms of sen-
sitivity when tested on slowly developing multiplicative and
additive faults in a continuous stirred-tank reactor under
closed-loop control and varying operating conditions.
m6 A-related lncRNAs are potential biomarkers for predicting prognoses and immune responses in patients with LUAD
Authors: Feng Xu, Xiaoling Huang, Yangyi Li, Yongsong Chen, and Ling Lin
Date: 2021-04-24 00:00:00
Proceeding: molecular therapy nucleic acids 2021 4

Lung adenocarcinoma (LUAD) is the most frequent subtype of lung cancer worldwide. However, the survival rate of LUAD patients remains low. N6-methyladenosine (m6A) and long noncoding RNAs (lncRNAs) play vital roles in the prognostic value and the immunotherapeutic response of LUAD. Thus, discerning lncRNAs associated with m6A in LUAD patients is
critical. In this study, m6A-related lncRNAs were analyzed and obtained by coexpression. Univariate, least absolute shrinkage and selection operator (LASSO), and multivariate Cox regression analyses were conducted to construct an m6A-related lncRNA model. Kaplan-Meier analysis, principalcomponent analysis (PCA), functional enrichment annotation, and nomogram were used to analyze the risk model. Finally, the potential immunotherapeutic signatures and drug sensitivity prediction targeting this model were also discussed. The risk model comprising 12 m6A-related lncRNAs was identified as
an independent predictor of prognoses. By regrouping the patients with this model, we can distinguish between them moreeffectively in terms of the immunotherapeutic response.
Finally, candidate compounds aimed at LUAD subtype differentiation were identified. This risk model based on the m6
A-based lncRNAs may be promising for the clinical prediction of prognoses and immunotherapeutic responses in LUAD patients.
End-to-end gener- ation of talking faces from noisy speech
Authors: Sefik Emre Eskimez; Ross K. Maddox; Chenliang Xu
Date: 2020-03-14 00:00:00
Proceeding: IEEE 2020 3
Tasks: Generative Adversarial Network

Acoustic cues are not the only component in speech communication; if the visual counterpart is present, it is shown to benefit speech comprehension. In this work, we propose an end-to-end (no pre- or post-processing) system that can generate talking faces from arbitrarily long noisy speech. We propose a mouth region mask to encourage the network to focus on mouth movements rather than speech irrelevant movements. In addition, we use generative adversarial network (GAN) training to improve the image quality and mouth-speech synchronization. Furthermore, we employ noise-resilient training to make our network robust to unseen non-stationary noise. We evaluate our system with image quality and mouth shape (landmark) measures on noisy speech utterances with five types of unseen non-stationary noise between -10 dB and 30 dB signal-to-noise ratio (SNR) with increments of 1 dB SNR. Results show that our system outperforms a state-of-the-art baseline system significantly, and our noise-resilient training improves performance for noisy speech in a wide range of SNR.
Active-learning-based non-intrusive Model Order Reduction
Authors: Qinyu Zhuang, Dirk Hartmann, Hans Joachim Bungartz, Juan Manuel Lorenzi
Date: 2022-04-08 00:00:00
Tasks: Active Learning, GPR, model, PAC learning

The Model Order Reduction (MOR) technique can provide compact numerical models for fast simulation. Different from the intrusive MOR methods, the non-intrusive MOR does not require access to the Full Order Models (FOMs), especially system matrices. Since the non-intrusive MOR methods strongly rely on the snapshots of the FOMs, constructing good snapshot sets becomes crucial. In this work, we propose a new active learning approach with two novelties. A novel idea with our approach is the use of single-time step snapshots from the system states taken from an estimation of the reduced-state space. These states are selected using a greedy strategy supported by an error estimator based Gaussian Process Regression (GPR). Additionally, we introduce a use case-independent validation strategy based on Probably Approximately Correct (PAC) learning. In this work, we use Artificial Neural Networks (ANNs) to identify the Reduced Order Model (ROM), however the method could be similarly applied to other ROM identification methods. The performance of the whole workflow is tested by a 2-D thermal conduction and a 3-D vacuum furnace model. With little required user interaction and a training strategy independent to a specific use case, the proposed method offers a huge potential for industrial usage to create so-called executable Digital Twins (DTs).