pwc-archive/papers-with-abstracts huggingface dataset:
DictBERT: Dictionary Description Knowledge Enhanced Language Model Pre-training via Contrastive Learning
Authors: Qianglong Chen, Feng-Lin Li, Guohai Xu, Ming Yan, Ji Zhang, Yin Zhang
Date: 2022-08-01 00:00:00
Tasks: Contrastive Learning, Language Modeling, Language Modelling, NER, Relation Extraction

Although pre-trained language models (PLMs) have achieved state-of-the-art performance on various natural language processing (NLP) tasks, they are shown to be lacking in knowledge when dealing with knowledge driven tasks. Despite the many efforts made for injecting knowledge into PLMs, this problem remains open. To address the challenge, we propose \textbf{DictBERT}, a novel approach that enhances PLMs with dictionary knowledge which is easier to acquire than knowledge graph (KG). During pre-training, we present two novel pre-training tasks to inject dictionary knowledge into PLMs via contrastive learning: \textit{dictionary entry prediction} and \textit{entry description discrimination}. In fine-tuning, we use the pre-trained DictBERT as a plugin knowledge base (KB) to retrieve implicit knowledge for identified entries in an input sequence, and infuse the retrieved knowledge into the input to enhance its representation via a novel extra-hop attention mechanism. We evaluate our approach on a variety of knowledge driven and language understanding tasks, including NER, relation extraction, CommonsenseQA, OpenBookQA and GLUE. Experimental results demonstrate that our model can significantly improve typical PLMs: it gains a substantial improvement of 0.5\%, 2.9\%, 9.0\%, 7.1\% and 3.3\% on BERT-large respectively, and is also effective on RoBERTa-large.
The impact of Twitter on political influence on the choice of a running mate: Social Network Analysis and Semantic Analysis -- A Review
Authors: Immaculate Wanza, Irad Kamuti, David Gichohi, Kinyua Gikunda
Date: 2022-07-31 00:00:00
Tasks: Decision Making

In this new era of social media, social networks are becoming increasingly important sources of user-generated content on the internet. These kinds of information resources, which include a lot of people's feelings, opinions, feedback, and reviews, are very useful for big businesses, markets, politics, journalism, and many other fields. Politics is one of the most talked-about and popular topics on social media networks right now. Many politicians use micro-blogging services like Twitter because they have a large number of followers and supporters on those networks. Politicians, political parties, political organizations, and foundations use social media networks to communicate with citizens ahead of time. Today, social media is used by hundreds of thousands of political groups and politicians. On these social media networks, every politician and political party has millions of followers, and politicians find new and innovative ways to urge individuals to participate in politics. Furthermore, social media assists politicians in various decision-making processes by providing recommendations, such as developing policies and strategies based on previous experiences, recommending and selecting suitable candidates for a particular constituency, recommending a suitable person for a particular position in the party, and launching a political campaign based on citizen sentiments on various issues and controversies, among other things. This research is a review on the use of social network analysis (SNA) and semantic analysis (SA) on the Twitter platform to study the supporters networks of political leaders because it can help in decision-making when predicting their political futures.
Mismatching-Aware Unsupervised Translation Quality Estimation For Low-Resource Languages
Authors: Fatemeh Azadi, Heshaam Faili, Mohammad Javad Dousti
Date: 2022-07-31 00:00:00
Tasks: Machine Translation, Translation

Translation Quality Estimation (QE) is the task of predicting the quality of machine translation (MT) output without any reference. This task has gained increasing attention as an important component in the practical applications of MT. In this paper, we first propose XLMRScore, which is a cross-lingual counterpart of BERTScore computed via the XLM-RoBERTa (XLMR) model. This metric can be used as a simple unsupervised QE method, nevertheless facing two issues: firstly, the untranslated tokens leading to unexpectedly high translation scores, and secondly, the issue of mismatching errors between source and hypothesis tokens when applying the greedy matching in XLMRScore. To mitigate these issues, we suggest replacing untranslated words with the unknown token and the cross-lingual alignment of the pre-trained model to represent aligned words closer to each other, respectively. We evaluate the proposed method on four low-resource language pairs of the WMT21 QE shared task, as well as a new English$\rightarrow$Persian (En-Fa) test dataset introduced in this paper. Experiments show that our method could get comparable results with the supervised baseline for two zero-shot scenarios, i.e., with less than 0.01 difference in Pearson correlation, while outperforming unsupervised rivals in all the low-resource language pairs for above 8%, on average.
The Who in Code-Switching: A Case Study for Predicting Egyptian Arabic-English Code-Switching Levels based on Character Profiles
Authors: Injy Hamed, Alia El Bolock, Cornelia Herbert, Slim Abdennadher, Ngoc Thang Vu
Date: 2022-07-31 00:00:00

Code-switching (CS) is a common linguistic phenomenon exhibited by multilingual individuals, where they tend to alternate between languages within one single conversation. CS is a complex phenomenon that not only encompasses linguistic challenges, but also contains a great deal of complexity in terms of its dynamic behaviour across speakers. Given that the factors giving rise to CS vary from one country to the other, as well as from one person to the other, CS is found to be a speaker-dependant behaviour, where the frequency by which the foreign language is embedded differs across speakers. While several researchers have looked into predicting CS behaviour from a linguistic point of view, research is still lacking in the task of predicting user CS behaviour from sociological and psychological perspectives. We provide an empirical user study, where we investigate the correlations between users' CS levels and character traits. We conduct interviews with bilinguals and gather information on their profiles, including their demographics, personality traits, and traveling experiences. We then use machine learning (ML) to predict users' CS levels based on their profiles, where we identify the main influential factors in the modeling process. We experiment with both classification as well as regression tasks. Our results show that the CS behaviour is affected by the relation between speakers, travel experiences as well as Neuroticism and Extraversion personality traits.
Neural Knowledge Bank for Pretrained Transformers
Authors: Damai Dai, Wenbin Jiang, Qingxiu Dong, Yajuan Lyu, Qiaoqiao She, Zhifang Sui
Date: 2022-07-31 00:00:00
Tasks: Language Modeling, Language Modelling, Machine Translation, Question Answering, Text Generation

The ability of pretrained Transformers to remember factual knowledge is essential but still limited for existing models. Inspired by existing work that regards Feed-Forward Networks (FFNs) in Transformers as key-value memories, we design a Neural Knowledge Bank (NKB) and a knowledge injection strategy to introduce extra factual knowledge for pretrained Transformers. The NKB is in the form of additional knowledgeable memory slots to the FFN and the memory-like architecture makes it highly interpretable and flexible. When injecting extra knowledge with the Salient Span Masking (SSM) pretraining objective, we fix the original pretrained model and train only the NKB. This training strategy makes sure the general language modeling ability of the original pretrained model is not influenced. By mounting the NKB onto the T5 model, we verify its strong ability to store extra factual knowledge based on three closed-book question answering datasets. Also, we prove that mounting the NKB will not degrade the general language modeling ability of T5 through two representative tasks, summarization and machine translation. Further, we thoroughly analyze the interpretability of the NKB and reveal the meaning of its keys and values in a human-readable way. Finally, we show the flexibility of the NKB by directly modifying its value vectors to update the factual knowledge stored in it.
Using Chatbots to Teach Languages
Authors: Yu Li, Chun-Yen Chen, Dian Yu, Sam Davidson, Ryan Hou, Xun Yuan, Yinghua Tan, Derek Pham, Zhou Yu
Date: 2022-07-31 00:00:00
Tasks: reinforcement-learning, Reinforcement Learning (RL)

This paper reports on progress towards building an online language learning tool to provide learners with conversational experience by using dialog systems as conversation practice partners. Our system can adapt to users' language proficiency on the fly. We also provide automatic grammar error feedback to help users learn from their mistakes. According to our first adopters, our system is entertaining and useful. Furthermore, we will provide the learning technology community a large-scale conversation dataset on language learning and grammar correction. Our next step is to make our system more adaptive to user profile information by using reinforcement learning algorithms.
Chinese grammatical error correction based on knowledge distillation
Authors: Peng Xia, Yuechi Zhou, Ziyan Zhang, Zecheng Tang, Juntao Li
Date: 2022-07-31 00:00:00
Tasks: Grammatical Error Correction, Knowledge Distillation

In view of the poor robustness of existing Chinese grammatical error correction models on attack test sets and large model parameters, this paper uses the method of knowledge distillation to compress model parameters and improve the anti-attack ability of the model. In terms of data, the attack test set is constructed by integrating the disturbance into the standard evaluation data set, and the model robustness is evaluated by the attack test set. The experimental results show that the distilled small model can ensure the performance and improve the training speed under the condition of reducing the number of model parameters, and achieve the optimal effect on the attack test set, and the robustness is significantly improved. Code is available at https://github.com/Richard88888/KD-CGEC.
PASTA: A Dataset for Modeling Participant States in Narratives
Authors: Sayontan Ghosh, Mahnaz Koupaee, Isabella Chen, Francis Ferraro, Nathanael Chambers, Niranjan Balasubramanian
Date: 2022-07-31 00:00:00
Tasks: Benchmarking, Common Sense Reasoning, counterfactual

The events in a narrative are understood as a coherent whole via the underlying states of their participants. Often, these participant states are not explicitly mentioned, instead left to be inferred by the reader. A model that understands narratives should likewise infer these implicit states, and even reason about the impact of changes to these states on the narrative. To facilitate this goal, we introduce a new crowdsourced English-language, Participant States dataset, PASTA. This dataset contains inferable participant states; a counterfactual perturbation to each state; and the changes to the story that would be necessary if the counterfactual were true. We introduce three state-based reasoning tasks that test for the ability to infer when a state is entailed by a story, to revise a story conditioned on a counterfactual state, and to explain the most likely state change given a revised story. Experiments show that today's LLMs can reason about states to some degree, but there is large room for improvement, especially in problems requiring access and ability to reason with diverse types of knowledge (e.g. physical, numerical, factual).
Smoothing Entailment Graphs with Language Models
Authors: Nick McKenna, Tianyi Li, Mark Johnson, Mark Steedman
Date: 2022-07-30 00:00:00
Tasks: Diversity, Explainable Models, Language Modeling, Language Modelling, Natural Language Inference, Relation Extraction

The diversity and Zipfian frequency distribution of natural language predicates in corpora leads to sparsity in Entailment Graphs (EGs) built by Open Relation Extraction (ORE). EGs are computationally efficient and explainable models of natural language inference, but as symbolic models, they fail if a novel premise or hypothesis vertex is missing at test-time. We present theory and methodology for overcoming such sparsity in symbolic models. First, we introduce a theory of optimal smoothing of EGs by constructing transitive chains. We then demonstrate an efficient, open-domain, and unsupervised smoothing method using an off-the-shelf Language Model to find approximations of missing premise predicates. This improves recall by 25.1 and 16.3 percentage points on two difficult directional entailment datasets, while raising average precision and maintaining model explainability. Further, in a QA task we show that EG smoothing is most useful for answering questions with lesser supporting text, where missing premise predicates are more costly. Finally, controlled experiments with WordNet confirm our theory and show that hypothesis smoothing is difficult, but possible in principle.
Cause-and-Effect Analysis of ADAS: A Comparison Study between Literature Review and Complaint Data
Authors: Jackie Ayoub, Zifei Wang, Meitang Li, Huizhong Guo, Rini Sherony, Shan Bao, Feng Zhou
Date: 2022-07-30 00:00:00

Advanced driver assistance systems (ADAS) are designed to improve vehicle safety. However, it is difficult to achieve such benefits without understanding the causes and limitations of the current ADAS and their possible solutions. This study 1) investigated the limitations and solutions of ADAS through a literature review, 2) identified the causes and effects of ADAS through consumer complaints using natural language processing models, and 3) compared the major differences between the two. These two lines of research identified similar categories of ADAS causes, including human factors, environmental factors, and vehicle factors. However, academic research focused more on human factors of ADAS issues and proposed advanced algorithms to mitigate such issues while drivers complained more of vehicle factors of ADAS failures, which led to associated top consequences. The findings from these two sources tend to complement each other and provide important implications for the improvement of ADAS in the future.
Masked Autoencoders As The Unified Learners For Pre-Trained Sentence Representation
Authors: Alexander Liu, Samuel Yang
Date: 2022-07-30 00:00:00
Tasks: Contrastive Learning, Natural Questions, Retrieval, Sentence, Sentence Embeddings, STS

Despite the progresses on pre-trained language models, there is a lack of unified frameworks for pre-trained sentence representation. As such, it calls for different pre-training methods for specific scenarios, and the pre-trained models are likely to be limited by their universality and representation quality. In this work, we extend the recently proposed MAE style pre-training strategy, RetroMAE, such that it may effectively support a wide variety of sentence representation tasks. The extended framework consists of two stages, with RetroMAE conducted throughout the process. The first stage performs RetroMAE over generic corpora, like Wikipedia, BookCorpus, etc., from which the base model is learned. The second stage takes place on domain-specific data, e.g., MS MARCO and NLI, where the base model is continuingly trained based on RetroMAE and contrastive learning. The pre-training outputs at the two stages may serve different applications, whose effectiveness are verified with comprehensive experiments. Concretely, the base model are proved to be effective for zero-shot retrieval, with remarkable performances achieved on BEIR benchmark. The continuingly pre-trained models further benefit more downstream tasks, including the domain-specific dense retrieval on MS MARCO, Natural Questions, and the sentence embeddings' quality for standard STS and transfer tasks in SentEval. The empirical insights of this work may inspire the future design of sentence representation pre-training. Our pre-trained models and source code will be released to the public communities.
ELF22: A Context-based Counter Trolling Dataset to Combat Internet Trolls
Authors: Huije Lee, Young Ju NA, Hoyun Song, Jisu Shin, Jong C. Park
Date: 2022-07-30 00:00:00
Proceeding: LREC 2022 6
Tasks: Response Generation, Sentence

Online trolls increase social costs and cause psychological damage to individuals. With the proliferation of automated accounts making use of bots for trolling, it is difficult for targeted individual users to handle the situation both quantitatively and qualitatively. To address this issue, we focus on automating the method to counter trolls, as counter responses to combat trolls encourage community users to maintain ongoing discussion without compromising freedom of expression. For this purpose, we propose a novel dataset for automatic counter response generation. In particular, we constructed a pair-wise dataset that includes troll comments and counter responses with labeled response strategies, which enables models fine-tuned on our dataset to generate responses by varying counter responses according to the specified strategy. We conducted three tasks to assess the effectiveness of our dataset and evaluated the results through both automatic and human evaluation. In human evaluation, we demonstrate that the model fine-tuned on our dataset shows a significantly improved performance in strategy-controlled sentence generation.
Dynamically Retrieving Knowledge via Query Generation for Informative Dialogue Generation
Authors: Zhongtian Hu, Lifang Wang, Yangqi Chen, Yushuang Liu, Ronghan Li, Meng Zhao, Xinyu Lu, Zejun Jiang
Date: 2022-07-30 00:00:00
Tasks: Dialogue Generation, Response Generation

Knowledge-driven dialog system has recently made remarkable breakthroughs. Compared with general dialog systems, superior knowledge-driven dialog systems can generate more informative and knowledgeable responses with pre-provided knowledge. However, in practical applications, the dialog system cannot be provided with corresponding knowledge in advance because it cannot know in advance the development of the conversation. Therefore, in order to make the knowledge dialogue system more practical, it is vital to find a way to retrieve relevant knowledge based on the dialogue history. To solve this problem, we design a knowledge-driven dialog system named DRKQG (Dynamically Retrieving Knowledge via Query Generation for informative dialog response). Specifically, the system can be divided into two modules: the query generation module and the dialog generation module. First, a time-aware mechanism is utilized to capture context information, and a query can be generated for retrieving knowledge through search engine. Then, we integrate the copy mechanism and transformers, which allows the response generation module to produce responses derived from the context and retrieved knowledge. Experimental results at LIC2022, Language and Intelligence Technology Competition, show that our module outperforms the baseline model by a large margin on automatic evaluation metrics, while human evaluation by the Baidu Linguistics team shows that our system achieves impressive results in Factually Correct and Knowledgeable.
CBAG: An Efficient Genetic Algorithm for the Graph Burning Problem
Authors: Mahdi Nazeri, Ali Mollahosseini, Iman Izadi
Date: 2022-08-01 00:00:00

Information spread is an intriguing topic to study in network science, which investigates how information, influence, or contagion propagate through networks. Graph burning is a simplified deterministic model for how information spreads within networks. The complicated NP-complete nature of the problem makes it computationally difficult to solve using exact algorithms. Accordingly, a number of heuristics and approximation algorithms have been proposed in the literature for the graph burning problem. In this paper, we propose an efficient genetic algorithm called Centrality BAsed Genetic-algorithm (CBAG) for solving the graph burning problem. Considering the unique characteristics of the graph burning problem, we introduce novel genetic operators, chromosome representation, and evaluation method. In the proposed algorithm, the well-known betweenness centrality is used as the backbone of our chromosome initialization procedure. The proposed algorithm is implemented and compared with previous heuristics and approximation algorithms on 15 benchmark graphs of different sizes. Based on the results, it can be seen that the proposed algorithm achieves better performance in comparison to the previous state-of-the-art heuristics. The complete source code is available online and can be used to find optimal or near-optimal solutions for the graph burning problem.
Assessing the robustness of critical behavior in stochastic cellular automata
Authors: Sidney Pontes-Filho, Pedro Lind, Stefano Nichele
Date: 2022-08-01 00:00:00

There is evidence that biological systems, such as the brain, work at a critical regime robust to noise, and are therefore able to remain in it under perturbations. In this work, we address the question of robustness of critical systems to noise. In particular, we investigate the robustness of stochastic cellular automata (CAs) at criticality. A stochastic CA is one of the simplest stochastic models showing criticality. The transition state of stochastic CA is defined through a set of probabilities. We systematically perturb the probabilities of an optimal stochastic CA known to produce critical behavior, and we report that such a CA is able to remain in a critical regime up to a certain degree of noise. We present the results using error metrics of the resulting power-law fitting, such as Kolmogorov-Smirnov statistic and Kullback-Leibler divergence. We discuss the implication of our results in regards to future realization of brain-inspired artificial intelligence systems.
Ultra-low Latency Adaptive Local Binary Spiking Neural Network with Accuracy Loss Estimator
Authors: Changqing Xu, Yijian Pei, Zili Wu, Yi Liu, YinTang Yang
Date: 2022-07-31 00:00:00
Tasks: Quantization

Spiking neural network (SNN) is a brain-inspired model which has more spatio-temporal information processing capacity and computational energy efficiency. However, with the increasing depth of SNNs, the memory problem caused by the weights of SNNs has gradually attracted attention. Inspired by Artificial Neural Networks (ANNs) quantization technology, binarized SNN (BSNN) is introduced to solve the memory problem. Due to the lack of suitable learning algorithms, BSNN is usually obtained by ANN-to-SNN conversion, whose accuracy will be limited by the trained ANNs. In this paper, we propose an ultra-low latency adaptive local binary spiking neural network (ALBSNN) with accuracy loss estimators, which dynamically selects the network layers to be binarized to ensure the accuracy of the network by evaluating the error caused by the binarized weights during the network learning process. Experimental results show that this method can reduce storage space by more than 20 % without losing network accuracy. At the same time, in order to accelerate the training speed of the network, the global average pooling(GAP) layer is introduced to replace the fully connected layers by the combination of convolution and pooling, so that SNNs can use a small number of time steps to obtain better recognition accuracy. In the extreme case of using only one time step, we still can achieve 92.92 %, 91.63 % ,and 63.54 % testing accuracy on three different datasets, FashionMNIST, CIFAR-10, and CIFAR-100, respectively.