pwc-archive/papers-with-abstracts huggingface dataset:
Attention-based Neural Load Forecasting: A Dynamic Feature Selection Approach
Authors: Jing Xiong, Pengyang Zhou, Alan Chen, Yu Zhang
Date: 2021-08-25 00:00:00
Tasks: Decoder, feature selection, Load Forecasting, Machine Translation, Time Series, Time Series Analysis, Time Series Forecasting, Translation

Encoder-decoder-based recurrent neural network (RNN) has made significant progress in sequence-to-sequence learning tasks such as machine translation and conversational models. Recent works have shown the advantage of this type of network in dealing with various time series forecasting tasks. The present paper focuses on the problem of multi-horizon short-term load forecasting, which plays a key role in the power system's planning and operation. Leveraging the encoder-decoder RNN, we develop an attention model to select the relevant features and similar temporal information adaptively. First, input features are assigned with different weights by a feature selection attention layer, while the updated historical features are encoded by a bi-directional long short-term memory (BiLSTM) layer. Then, a decoder with hierarchical temporal attention enables a similar day selection, which re-evaluates the importance of historical information at each time step. Numerical results tested on the dataset of the global energy forecasting competition 2014 show that our proposed model significantly outperforms some existing forecasting schemes.
Machine Learning for Sensor Transducer Conversion Routines
Authors: Thomas Newton, James T. Meech, Phillip Stanley-Marbell
Date: 2021-08-23 00:00:00
Tasks: BIG-bench Machine Learning

Sensors with digital outputs require software conversion routines to transform the unitless analogue-to-digital converter samples to physical quantities with correct units. These conversion routines are computationally complex given the limited computational resources of low-power embedded systems. This article presents a set of machine learning methods to learn new, less-complex conversion routines that do not sacrifice accuracy for the BME680 environmental sensor. We present a Pareto analysis of the tradeoff between accuracy and computational overhead for the models and models that reduce the computational overhead of the existing industry-standard conversion routines for temperature, pressure, and humidity by 62%, 71 %, and 18 % respectively. The corresponding RMS errors are 0.0114 degrees C, 0.0280 KPa, and 0.0337 %. These results show that machine learning methods for learning conversion routines can produce conversion routines with reduced computational overhead which maintain good accuracy.
Determining the origin of impulsive noise events using paired wireless sound sensors
Authors: Fabian Nemazi, Jon Nordby
Date: 2021-08-23 00:00:00

This work investigates how to identify the source of impulsive noise events using a pair of wireless noise sensors. One sensor is placed at a known noise source, and another sensor is placed at the noise receiver. Machine learning models receive data from the two sensors and estimate whether a given noise event originates from the known noise source or another source. To avoid privacy issues, the approach uses on-edge preprocessing that converts the sound into privacy compatible spectrograms. The system was evaluated at a shooting range and explosives training facility, using data collected during noise emission testing. The combination of convolutional neural networks with cross-correlation achieved the best results. We created multiple alternative models using different spectrogram representations. The best model detected 70.8\% of the impulsive noise events and correctly predicted 90.3\% of the noise events in the optimal trade-off between recall and precision.
Fast Accurate Defect Detection in Wafer Fabrication
Authors: Thomas Olschewski
Date: 2021-08-23 00:00:00
Tasks: Defect Detection

A generic fast method for object classification is proposed. In addition, a method for dimensional reduction is presented. The presented algorithms have been applied to real-world data from chip fabrication successfully to the task of predicting defect states of tens of thousands of chips of several products based on measurements or even just part of measurements. Unlike typical neural networks with a large number of weights to optimize over, the presented algorithm tries optimizing only over a very small number of variables in order to increase chances to find a global optimum. Our approach is interesting in that it is fast, led to good to very good performance with real-world wafer data, allows for short implementations and computes values which have a clear meaning easy to explain.
Training and Profiling a Pediatric Emotion Recognition Classifier on Mobile Devices
Authors: Agnik Banerjee, Peter Washington, Cezmi Mutlu, Aaron Kline, Dennis P. Wall
Date: 2021-08-22 00:00:00
Tasks: Diagnostic, Emotion Recognition, image-classification, Image Classification

Implementing automated emotion recognition on mobile devices could provide an accessible diagnostic and therapeutic tool for those who struggle to recognize emotion, including children with developmental behavioral conditions such as autism. Although recent advances have been made in building more accurate emotion classifiers, existing models are too computationally expensive to be deployed on mobile devices. In this study, we optimized and profiled various machine learning models designed for inference on edge devices and were able to match previous state of the art results for emotion recognition on children. Our best model, a MobileNet-V2 network pre-trained on ImageNet, achieved 65.11% balanced accuracy and 64.19% F1-score on CAFE, while achieving a 45-millisecond inference latency on a Motorola Moto G6 phone. This balanced accuracy is only 1.79% less than the current state of the art for CAFE, which used a model that contains 26.62x more parameters and was unable to run on the Moto G6, even when fully optimized. This work validates that with specialized design and optimization techniques, machine learning models can become lightweight enough for deployment on mobile devices and still achieve high accuracies on difficult image classification tasks.
Supervised Compression for Resource-Constrained Edge Computing Systems
Authors: Yoshitomo Matsubara, Ruihan Yang, Marco Levorato, Stephan Mandt
Date: 2021-08-21 00:00:00
Tasks: Data Compression, Edge-computing, Feature Compression, Image Compression, Knowledge Distillation

There has been much interest in deploying deep learning algorithms on low-powered devices, including smartphones, drones, and medical sensors. However, full-scale deep neural networks are often too resource-intensive in terms of energy and storage. As a result, the bulk part of the machine learning operation is therefore often carried out on an edge server, where the data is compressed and transmitted. However, compressing data (such as images) leads to transmitting information irrelevant to the supervised task. Another popular approach is to split the deep network between the device and the server while compressing intermediate features. To date, however, such split computing strategies have barely outperformed the aforementioned naive data compression baselines due to their inefficient approaches to feature compression. This paper adopts ideas from knowledge distillation and neural image compression to compress intermediate feature representations more efficiently. Our supervised compression approach uses a teacher model and a student model with a stochastic bottleneck and learnable prior for entropy coding (Entropic Student). We compare our approach to various neural image and feature compression baselines in three vision tasks and found that it achieves better supervised rate-distortion performance while maintaining smaller end-to-end latency. We furthermore show that the learned feature representations can be tuned to serve multiple downstream tasks.
SAUCE: Truncated Sparse Document Signature Bit-Vectors for Fast Web-Scale Corpus Expansion
Authors: Muntasir Wahed, Daniel Gruhl, Alfredo Alba, Anna Lisa Gentile, Petar Ristoski, Chad Deluca, Steve Welch, Ismini Lourentzou
Date: 2021-08-26 00:00:00
Tasks: Natural Language Understanding

Recent advances in text representation have shown that training on large amounts of text is crucial for natural language understanding. However, models trained without predefined notions of topical interest typically require careful fine-tuning when transferred to specialized domains. When a sufficient amount of within-domain text may not be available, expanding a seed corpus of relevant documents from large-scale web data poses several challenges. First, corpus expansion requires scoring and ranking each document in the collection, an operation that can quickly become computationally expensive as the web corpora size grows. Relying on dense vector spaces and pairwise similarity adds to the computational expense. Secondly, as the domain concept becomes more nuanced, capturing the long tail of domain-specific rare terms becomes non-trivial, especially under limited seed corpora scenarios. In this paper, we consider the problem of fast approximate corpus expansion given a small seed corpus with a few relevant documents as a query, with the goal of capturing the long tail of a domain-specific set of concept terms. To efficiently collect large-scale domain-specific corpora with limited relevance feedback, we propose a novel truncated sparse document bit-vector representation, termed Signature Assisted Unsupervised Corpus Expansion (SAUCE). Experimental results show that SAUCE can reduce the computational burden while ensuring high within-domain lexical coverage.
Position-Invariant Truecasing with a Word-and-Character Hierarchical Recurrent Neural Network
Authors: Hao Zhang, You-Chi Cheng, Shankar Kumar, Mingqing Chen, Rajiv Mathews
Date: 2021-08-26 00:00:00
Tasks: Language Modeling, Language Modelling, Machine Translation, named-entity-recognition, Named Entity Recognition, Named Entity Recognition (NER), Position, Sentence, speech-recognition, Speech Recognition, Translation

Truecasing is the task of restoring the correct case (uppercase or lowercase) of noisy text generated either by an automatic system for speech recognition or machine translation or by humans. It improves the performance of downstream NLP tasks such as named entity recognition and language modeling. We propose a fast, accurate and compact two-level hierarchical word-and-character-based recurrent neural network model, the first of its kind for this problem. Using sequence distillation, we also address the problem of truecasing while ignoring token positions in the sentence, i.e. in a position-invariant manner.
HAN: Higher-order Attention Network for Spoken Language Understanding
Authors: Dongsheng Chen, Zhiqi Huang, Yuexian Zou
Date: 2021-08-26 00:00:00
Tasks: Intent Detection, slot-filling, Slot Filling, Spoken Language Understanding

Spoken Language Understanding (SLU), including intent detection and slot filling, is a core component in human-computer interaction. The natural attributes of the relationship among the two subtasks make higher requirements on fine-grained feature interaction, i.e., the token-level intent features and slot features. Previous works mainly focus on jointly modeling the relationship between the two subtasks with attention-based models, while ignoring the exploration of attention order. In this paper, we propose to replace the conventional attention with our proposed Bilinear attention block and show that the introduced Higher-order Attention Network (HAN) brings improvement for the SLU task. Importantly, we conduct wide analysis to explore the effectiveness brought from the higher-order attention.
A Survey on Automated Fact-Checking
Authors: Zhijiang Guo, Michael Schlichtkrull, Andreas Vlachos
Date: 2021-08-26 00:00:00
Tasks: Fact Checking, Misinformation, Survey

Fact-checking has become increasingly important due to the speed with which both information and misinformation can spread in the modern media ecosystem. Therefore, researchers have been exploring how fact-checking can be automated, using techniques based on natural language processing, machine learning, knowledge representation, and databases to automatically predict the veracity of claims. In this paper, we survey automated fact-checking stemming from natural language processing, and discuss its connections to related tasks and disciplines. In this process, we present an overview of existing datasets and models, aiming to unify the various definitions given and identify common concepts. Finally, we highlight challenges for future research.
Probing Pre-trained Auto-regressive Language Models for Named Entity Typing and Recognition
Authors: Elena V. Epure, Romain Hennequin
Date: 2021-08-26 00:00:00
Proceeding: LREC 2022 6
Tasks: Entity Typing, Few-shot NER, Meta-Learning, Named Entity Recognition, Named Entity Recognition (NER)

Despite impressive results of language models for named entity recognition (NER), their generalization to varied textual genres, a growing entity type set, and new entities remains a challenge. Collecting thousands of annotations in each new case for training or fine-tuning is expensive and time-consuming. In contrast, humans can easily identify named entities given some simple instructions. Inspired by this, we challenge the reliance on large datasets and study pre-trained language models for NER in a meta-learning setup. First, we test named entity typing (NET) in a zero-shot transfer scenario. Then, we perform NER by giving few examples at inference. We propose a method to select seen and rare / unseen names when having access only to the pre-trained model and report results on these groups. The results show: auto-regressive language models as meta-learners can perform NET and NER fairly well especially for regular or seen names; name irregularity when often present for a certain entity type can become an effective exploitable cue; names with words foreign to the model have the most negative impact on results; the model seems to rely more on name than context cues in few-shot NER.
Alleviating Exposure Bias via Contrastive Learning for Abstractive Text Summarization
Authors: Shichao Sun, Wenjie Li
Date: 2021-08-26 00:00:00
Tasks: Abstractive Text Summarization, Contrastive Learning, Decoder, Text Summarization

Encoder-decoder models have achieved remarkable success in abstractive text summarization, which aims to compress one or more documents into a shorter version without the loss of the essential content. Unfortunately, these models mostly suffer a discrepancy between training and inference, i.e., the exposure bias problem. During the training stage, with teacher forcing these models are optimized to maximize the likelihood of the gold summary given the gold summary tokens as input to the decoder, while at inference the given tokens are replaced by the generated tokens. Consequently, low-quality summaries are very likely to be generated. To remedy this problem, we propose to leverage contrastive learning to decrease the likelihood of these low-quality summaries, and meanwhile increase the likelihood of the gold summary. Since our solution expands the states that the model perceives during training, we expect that the exposure bias problem can be alleviated. We experimentally demonstrate that our method effectively improves the performance of the state-of-the-art model on different datasets.
Just Say No: Analyzing the Stance of Neural Dialogue Generation in Offensive Contexts
Authors: Ashutosh Baheti, Maarten Sap, Alan Ritter, Mark Riedl
Date: 2021-08-26 00:00:00
Proceeding: EMNLP 2021 11
Tasks: Dialogue Generation, Text Generation

Dialogue models trained on human conversations inadvertently learn to generate toxic responses. In addition to producing explicitly offensive utterances, these models can also implicitly insult a group or individual by aligning themselves with an offensive statement. To better understand the dynamics of contextually offensive language, we investigate the stance of dialogue model responses in offensive Reddit conversations. Specifically, we create ToxiChat, a crowd-annotated dataset of 2,000 Reddit threads and model responses labeled with offensive language and stance. Our analysis reveals that 42% of human responses agree with toxic comments, whereas only 13% agree with safe comments. This undesirable behavior is learned by neural dialogue models, such as DialoGPT, which we show are two times more likely to agree with offensive comments. To enable automatic detection of offensive language, we fine-tuned transformer-based classifiers on ToxiChat that achieve 0.71 F1 for offensive labels and 0.53 Macro-F1 for stance labels. Finally, we quantify the effectiveness of controllable text generation (CTG) methods to mitigate the tendency of neural dialogue models to agree with offensive comments. Compared to the baseline, our best CTG model achieves a 19% reduction in agreement with offensive comments and produces 29% fewer offensive replies. Our work highlights the need for further efforts to characterize and analyze inappropriate behavior in dialogue models, in order to help make them safer. Our code and corpus are available at https://github.com/abaheti95/ToxiChat .
A Computational Approach to Measure Empathy and Theory-of-Mind from Written Texts
Authors: Yoon Kyung Lee, Inju Lee, Jae Eun Park, Yoonwon Jung, Jiwon Kim, Sowon Hahn
Date: 2021-08-26 00:00:00

Theory-of-mind (ToM), a human ability to infer the intentions and thoughts of others, is an essential part of empathetic experiences. We provide here the framework for using NLP models to measure ToM expressed in written texts. For this purpose, we introduce ToM-Diary, a crowdsourced 18,238 diaries with 74,014 Korean sentences annotated with different ToM levels. Each diary was annotated with ToM levels by trained psychology students and reviewed by selected psychology experts. The annotators first divided the diaries based on whether they mentioned other people: self-focused and other-focused. Examples of self-focused sentences are "I am feeling good". The other-focused sentences were further classified into different levels. These levels differ by whether the writer 1) mentions the presence of others without inferring their mental state(e.g., I saw a man walking down the street), 2) fails to take the perspective of others (e.g., I don't understand why they refuse to wear masks), or 3) successfully takes the perspective of others (It must have been hard for them to continue working). We tested whether state-of-the-art transformer-based models (e.g., BERT) could predict underlying ToM levels in sentences. We found that BERT more successfully detected self-focused sentences than other-focused ones. Sentences that successfully take the perspective of others (the highest ToM level) were the most difficult to predict. Our study suggests a promising direction for large-scale and computational approaches for identifying the ability of authors to empathize and take the perspective of others. The dataset is at [URL](https://github.com/humanfactorspsych/covid19-tom-empathy-diary)
A Statutory Article Retrieval Dataset in French
Authors: Antoine Louis, Gerasimos Spanakis
Date: 2021-08-26 00:00:00
Proceeding: ACL 2022 5
Tasks: Articles, Information Retrieval, Retrieval, Specificity

Statutory article retrieval is the task of automatically retrieving law articles relevant to a legal question. While recent advances in natural language processing have sparked considerable interest in many legal tasks, statutory article retrieval remains primarily untouched due to the scarcity of large-scale and high-quality annotated datasets. To address this bottleneck, we introduce the Belgian Statutory Article Retrieval Dataset (BSARD), which consists of 1,100+ French native legal questions labeled by experienced jurists with relevant articles from a corpus of 22,600+ Belgian law articles. Using BSARD, we benchmark several state-of-the-art retrieval approaches, including lexical and dense architectures, both in zero-shot and supervised setups. We find that fine-tuned dense retrieval models significantly outperform other systems. Our best performing baseline achieves 74.8% R@100, which is promising for the feasibility of the task and indicates there is still room for improvement. By the specificity of the domain and addressed task, BSARD presents a unique challenge problem for future research on legal information retrieval. Our dataset and source code are publicly available.
Data Augmentation for Low-Resource Named Entity Recognition Using Backtranslation
Authors: Usama Yaseen, Stefan Langer
Date: 2021-08-26 00:00:00
Proceeding: ICON 2021 12
Tasks: Data Augmentation, Low Resource Named Entity Recognition, named-entity-recognition, Named Entity Recognition, Named Entity Recognition (NER)

The state of art natural language processing systems relies on sizable training datasets to achieve high performance. Lack of such datasets in the specialized low resource domains lead to suboptimal performance. In this work, we adapt backtranslation to generate high quality and linguistically diverse synthetic data for low-resource named entity recognition. We perform experiments on two datasets from the materials science (MaSciP) and biomedical domains (S800). The empirical results demonstrate the effectiveness of our proposed augmentation strategy, particularly in the low-resource scenario.
Rethinking Why Intermediate-Task Fine-Tuning Works
Authors: Ting-Yun Chang, Chi-Jen Lu
Date: 2021-08-26 00:00:00
Proceeding: Findings (EMNLP) 2021 11

Supplementary Training on Intermediate Labeled-data Tasks (STILTs) is a widely applied technique, which first fine-tunes the pretrained language models on an intermediate task before on the target task of interest. While STILTs is able to further improve the performance of pretrained language models, it is still unclear why and when it works. Previous research shows that those intermediate tasks involving complex inference, such as commonsense reasoning, work especially well for RoBERTa. In this paper, we discover that the improvement from an intermediate task could be orthogonal to it containing reasoning or other complex skills -- a simple real-fake discrimination task synthesized by GPT2 can benefit diverse target tasks. We conduct extensive experiments to study the impact of different factors on STILTs. These findings suggest rethinking the role of intermediate fine-tuning in the STILTs pipeline.
Technological Approaches to Detecting Online Disinformation and Manipulation
Authors: Aleš Horák, Vít Baisa, Ondřej Herman
Date: 2021-08-26 00:00:00
Tasks: Benchmarking, Fact Checking

The move of propaganda and disinformation to the online environment is possible thanks to the fact that within the last decade, digital information channels radically increased in popularity as a news source. The main advantage of such media lies in the speed of information creation and dissemination. This, on the other hand, inevitably adds pressure, accelerating editorial work, fact-checking, and the scrutiny of source credibility. In this chapter, an overview of computer-supported approaches to detecting disinformation and manipulative techniques based on several criteria is presented. We concentrate on the technical aspects of automatic methods which support fact-checking, topic identification, text style analysis, or message filtering on social media channels. Most of the techniques employ artificial intelligence and machine learning with feature extraction combining available information resources. The following text firstly specifies the tasks related to computer detection of manipulation and disinformation spreading. The second section presents concrete methods of solving the tasks of the analysis, and the third sections enlists current verification and benchmarking datasets published and used in this area for evaluation and comparison.