pwc-archive/papers-with-abstracts huggingface dataset:
Explicit and Implicit Pattern Relation Analysis for Discovering Actionable Negative Sequences
Authors: Wei Wang, Longbing Cao
Date: 2022-04-04 00:00:00
Tasks: Decision Making, Diversity, Relation

Real-life events, behaviors and interactions produce sequential data. An important but rarely explored problem is to analyze those nonoccurring (also called negative) yet important sequences, forming negative sequence analysis (NSA). A typical NSA area is to discover negative sequential patterns (NSPs) consisting of important non-occurring and occurring elements and patterns. The limited existing work on NSP mining relies on frequentist and downward closure property-based pattern selection, producing large and highly redundant NSPs, nonactionable for business decision-making. This work makes the first attempt for actionable NSP discovery. It builds an NSP graph representation, quantify both explicit occurrence and implicit non-occurrence-based element and pattern relations, and then discover significant, diverse and informative NSPs in the NSP graph to represent the entire NSP set for discovering actionable NSPs. A DPP-based NSP representation and actionable NSP discovery method EINSP introduces novel and significant contributions for NSA and sequence analysis: (1) it represents NSPs by a determinantal point process (DPP) based graph; (2) it quantifies actionable NSPs in terms of their statistical significance, diversity, and strength of explicit/implicit element/pattern relations; and (3) it models and measures both explicit and implicit element/pattern relations in the DPP-based NSP graph to represent direct and indirect couplings between NSP items, elements and patterns. We substantially analyze the effectiveness of EINSP in terms of various theoretical and empirical aspects including complexity, item/pattern coverage, pattern size and diversity, implicit pattern relation strength, and data factors.
Delta Keyword Transformer: Bringing Transformers to the Edge through Dynamically Pruned Multi-Head Self-Attention
Authors: Zuzana Jelčicová, Marian Verhelst
Date: 2022-03-20 00:00:00
Tasks: Keyword Spotting

Multi-head self-attention forms the core of Transformer networks. However, their quadratically growing complexity with respect to the input sequence length impedes their deployment on resource-constrained edge devices. We address this challenge by proposing a dynamic pruning method, which exploits the temporal stability of data across tokens to reduce inference cost. The threshold-based method only retains significant differences between the subsequent tokens, effectively reducing the number of multiply-accumulates, as well as the internal tensor data sizes. The approach is evaluated on the Google Speech Commands Dataset for keyword spotting, and the performance is compared against the baseline Keyword Transformer. Our experiments show that we can reduce ~80% of operations while maintaining the original 98.4% accuracy. Moreover, a reduction of ~87-94% operations can be achieved when only degrading the accuracy by 1-4%, speeding up the multi-head self-attention inference by a factor of ~7.5-16.
Position-based Prompting for Health Outcome Generation
Authors: M. Abaho, D. Bollegala, P. Williamson, S. Dodd
Date: 2022-03-30 00:00:00
Tasks: Language Modelling, Position

Probing Pre-trained Language Models (PLMs) using prompts has indirectly implied that language models (LMs) can be treated as knowledge bases. To this end, this phenomena has been effective especially when these LMs are fine-tuned towards not just data of a specific domain, but also to the style or linguistic pattern of the prompts themselves. We observe that, satisfying a particular linguistic pattern in prompts is an unsustainable constraint that unnecessarily lengthens the probing task, especially because, they are often manually designed and the range of possible prompt template patterns can vary depending on the prompting objective and domain. We therefore explore an idea of using a position-attention mechanism to capture positional information of each word in a prompt relative to the mask to be filled, hence avoiding the need to re-construct prompts when the prompts linguistic pattern changes. Using our approach, we demonstrate the ability of eliciting answers to rare prompt templates (in a case study on health outcome generation) such as Postfix and Mixed patterns whose missing information is respectively at the start and in multiple random places of the prompt. More so, using various biomedical PLMs, our approach consistently outperforms a baseline in which the default mask language model (MLM) representation is used to predict masked tokens.
Survey on Automated Short Answer Grading with Deep Learning: from Word Embeddings to Transformers
Authors: Stefan Haller, Adina Aldea, Christin Seifert, Nicola Strisciuglio
Date: 2022-03-11 00:00:00
Tasks: Deep Learning, Representation Learning, Word Embeddings

Automated short answer grading (ASAG) has gained attention in education as a means to scale educational tasks to the growing number of students. Recent progress in Natural Language Processing and Machine Learning has largely influenced the field of ASAG, of which we survey the recent research advancements. We complement previous surveys by providing a comprehensive analysis of recently published methods that deploy deep learning approaches. In particular, we focus our analysis on the transition from hand engineered features to representation learning approaches, which learn representative features for the task at hand automatically from large corpora of data. We structure our analysis of deep learning methods along three categories: word embeddings, sequential models, and attention-based methods. Deep learning impacted ASAG differently than other fields of NLP, as we noticed that the learned representations alone do not contribute to achieve the best results, but they rather show to work in a complementary way with hand-engineered features. The best performance are indeed achieved by methods that combine the carefully hand-engineered features with the power of the semantic descriptions provided by the latest models, like transformers architectures. We identify challenges and provide an outlook on research direction that can be addressed in the future
Towards a Cognitive Computational Neuroscience of Auditory Phantom Perceptions
Authors: Patrick Krauss, Achim Schilling
Date: 2020-10-05 00:00:00

In order to gain a mechanistic understanding of how tinnitus emerges in the brain, we must build biologically plausible computational models that mimic both tinnitus development and perception, and test the tentative models with brain and behavioral experiments. With a special focus on tinnitus research, we review recent work at the intersection of artificial intelligence, psychology and neuroscience, indicating a new research agenda that follows the idea that experiments will yield theoretical insight only when employed to test brain-computational models. This view challenges the popular belief, that tinnitus research is primarily data limited, and that producing large, multi-modal, and complex datasets, analyzed with advanced data analysis algorithms, will finally lead to fundamental insights into how tinnitus emerges. However, there is converging evidence that although modern technologies allow assessing neural activity in unprecedentedly rich ways in both, animals and humans, empirical testing one verbally defined hypothesis about tinnitus after another, will never lead to a mechanistic understanding. Instead, hypothesis testing needs to be complemented with the construction of computational models that generate verifiable predictions. We argue, that even though, contemporary artificial intelligence and machine learning approaches largely lack biological plausibility, the models to be constructed will have to draw on concepts from these fields, since they have already proven to do well in modeling brain function. Nevertheless, biological fidelity will have to be increased successively, leading to ever better and fine-grained models, allowing at the end for even testing possible treatment strategies in silico, before application in animal or patient studies.
Deep Understanding based Multi-Document Machine Reading Comprehension
Authors: Feiliang Ren, Yongkang Liu, Bochao Li, Zhibo Wang, Yu Guo, Shilei Liu, Huimin Wu, Jiaqi Wang, Chunchao Liu, Bingchao Wang
Date: 2022-02-25 00:00:00
Tasks: Machine Reading Comprehension, Reading Comprehension, TriviaQA

Most existing multi-document machine reading comprehension models mainly focus on understanding the interactions between the input question and documents, but ignore following two kinds of understandings. First, to understand the semantic meaning of words in the input question and documents from the perspective of each other. Second, to understand the supporting cues for a correct answer from the perspective of intra-document and inter-documents. Ignoring these two kinds of important understandings would make the models oversee some important information that may be helpful for inding correct answers. To overcome this deiciency, we propose a deep understanding based model for multi-document machine reading comprehension. It has three cascaded deep understanding modules which are designed to understand the accurate semantic meaning of words, the interactions between the input question and documents, and the supporting cues for the correct answer. We evaluate our model on two large scale benchmark datasets, namely TriviaQA Web and DuReader. Extensive experiments show that our model achieves state-of-the-art results on both datasets.
Leveraging pre-trained language models for conversational information seeking from text
Authors: Patrizio Bellan, Mauro Dragoni, Chiara Ghidini
Date: 2022-03-31 00:00:00
Tasks: Few-Shot Learning, In-Context Learning

Recent advances in Natural Language Processing, and in particular on the construction of very large pre-trained language representation models, is opening up new perspectives on the construction of conversational information seeking (CIS) systems. In this paper we investigate the usage of in-context learning and pre-trained language representation models to address the problem of information extraction from process description documents, in an incremental question and answering oriented fashion. In particular we investigate the usage of the native GPT-3 (Generative Pre-trained Transformer 3) model, together with two in-context learning customizations that inject conceptual definitions and a limited number of samples in a few shot-learning fashion. The results highlight the potential of the approach and the usefulness of the in-context learning customizations, which can substantially contribute to address the "training data challenge" of deep learning based NLP techniques the BPM field. It also highlight the challenge posed by control flow relations for which further training needs to be devised.
LibCity: An Open Library for Traffic Prediction
Authors: Jingyuan Wang, Jiawei Jiang, Wenjun Jiang, Chao Li, Wayne Xin Zhao
Date: 2021-11-04 00:00:00
Proceeding: International Conference on Advances in Geographic Information Systems 2021 11
Tasks: Multivariate Time Series Forecasting, Prediction, Spatio-Temporal Forecasting, Time Series Analysis, Time Series Forecasting, Time Series Prediction, Traffic Prediction

With the increase of traffic prediction models, there has become an urgent need to develop a standardized framework to implement and evaluate these methods. This paper presents LibCity, a unified, comprehensive, and extensible library for traffic prediction, which provides researchers with a credible experimental tool and a convenient development framework. In this library, we reproduce 42 traffic prediction models and collect 29 spatial-temporal datasets, which allows researchers to conduct comprehensive experiments in a convenient way. To accelerate the development of new models, we design unified model interfaces based on unified data formats, which effectively encapsulate the details of the implementation. To verify the effectiveness of our implementations, we also report the reproducibility comparison results of LibCity, and set up a performance leaderboard for the four kinds of traffic prediction tasks. Our library will contribute to the standardization and reproducibility in the field of traffic prediction. The open source link of LibCity is https://github.com/LibCity/Bigscity-LibCity.
optimal sizing of capacitors placed on a radial distribution system
Authors: M. Baran, F.F. Wu
Date: 1989-01-01 00:00:00
Proceeding: IEEE Transactions on Power Delivery 1989 1

A capacitor sizing problem for capacitors placed on a radial distribution system is formulated as a nonlinear programming problem, and a solution algorithm is developed. The object is to find the optimal size of the capacitors so that the power losses will be minimized for a given load profile while considering the cost of the capacitors. The formulation also incorporates the AC power flow model for the system and the voltage constraints. The solution algorithm developed for the capacitor sizing problem is based on a Phase I-Phase II feasible directions approach. Novel power flow equations and a solution method, called DistFlow, for radial distribution systems are introduced. The method is computationally efficient and numerically robust, especially for distribution systems with large r/x ratio branches. DistFlow is used repeatedly as a subroutine in the optimization algorithm for the capacitor sizing problem. The test results for the algorithm indicate that the method is computationally efficient and has good convergence characteristics.
PlutoNet: An Efficient Polyp Segmentation Network with Modified Partial Decoder and Decoder Consistency Training
Authors: Tugberk Erol, Duygu Sarikaya
Date: 2022-04-06 00:00:00
Tasks: Decoder, Image Segmentation, Medical Image Segmentation, Semantic Segmentation

Deep learning models are used to minimize the number of polyps that goes unnoticed by the experts and to accurately segment the detected polyps during interventions. Although state-of-the-art models are proposed, it remains a challenge to define representations that are able to generalize well and that mediate between capturing low-level features and higher-level semantic details without being redundant. Another challenge with these models is that they require too many parameters, which can pose a problem with real-time applications. To address these problems, we propose PlutoNet for polyp segmentation which requires only 2,626,537 parameters, less than 10\% of the parameters required by its counterparts. With PlutoNet, we propose a novel \emph{decoder consistency training} approach that consists of a shared encoder, the modified partial decoder which is a combination of the partial decoder and full-scale connections that capture salient features at different scales without being redundant, and the auxiliary decoder which focuses on higher-level relevant semantic features. We train the modified partial decoder and the auxiliary decoder with a combined loss to enforce consistency, which helps improve the encoders representations. This way we are able to reduce uncertainty and false positive rates. We perform ablation studies and extensive experiments which show that PlutoNet performs significantly better than the state-of-the-art models, particularly on unseen datasets and datasets across different domains.
Vision Transformers for Single Image Dehazing
Authors: Yuda Song, Zhuqing He, Hui Qian, Xin Du
Date: 2022-04-08 00:00:00
Tasks: Image Dehazing, Single Image Dehazing

Image dehazing is a representative low-level vision task that estimates latent haze-free images from hazy images. In recent years, convolutional neural network-based methods have dominated image dehazing. However, vision Transformers, which has recently made a breakthrough in high-level vision tasks, has not brought new dimensions to image dehazing. We start with the popular Swin Transformer and find that several of its key designs are unsuitable for image dehazing. To this end, we propose DehazeFormer, which consists of various improvements, such as the modified normalization layer, activation function, and spatial information aggregation scheme. We train multiple variants of DehazeFormer on various datasets to demonstrate its effectiveness. Specifically, on the most frequently used SOTS indoor set, our small model outperforms FFA-Net with only 25% #Param and 5% computational cost. To the best of our knowledge, our large model is the first method with the PSNR over 40 dB on the SOTS indoor set, dramatically outperforming the previous state-of-the-art methods. We also collect a large-scale realistic remote sensing dehazing dataset for evaluating the method's capability to remove highly non-homogeneous haze.
BioBART: Pretraining and Evaluation of A Biomedical Generative Language Model
Authors: Hongyi Yuan, Zheng Yuan, Ruyi Gan, Jiaxing Zhang, Yutao Xie, Sheng Yu
Date: 2022-04-08 00:00:00
Proceeding: BioNLP (ACL) 2022 5
Tasks: Entity Linking, Language Modeling, Language Modelling, named-entity-recognition, Named Entity Recognition, Named Entity Recognition (NER), Natural Language Understanding, Nested Named Entity Recognition, Sentence, Text Generation

Pretrained language models have served as important backbones for natural language processing. Recently, in-domain pretraining has been shown to benefit various domain-specific downstream tasks. In the biomedical domain, natural language generation (NLG) tasks are of critical importance, while understudied. Approaching natural language understanding (NLU) tasks as NLG achieves satisfying performance in the general domain through constrained language generation or language prompting. We emphasize the lack of in-domain generative language models and the unsystematic generative downstream benchmarks in the biomedical domain, hindering the development of the research community. In this work, we introduce the generative language model BioBART that adapts BART to the biomedical domain. We collate various biomedical language generation tasks including dialogue, summarization, entity linking, and named entity recognition. BioBART pretrained on PubMed abstracts has enhanced performance compared to BART and set strong baselines on several tasks. Furthermore, we conduct ablation studies on the pretraining tasks for BioBART and find that sentence permutation has negative effects on downstream tasks.
Efficient Feedback and Partial Credit Grading for Proof Blocks Problems
Authors: Seth Poulsen, Shubhang Kulkarni, Geoffrey Herman, Matthew West
Date: 2022-04-08 00:00:00
Tasks: Management, Mathematical Proofs

Proof Blocks is a software tool that allows students to practice writing mathematical proofs by dragging and dropping lines instead of writing proofs from scratch. Proof Blocks offers the capability of assigning partial credit and providing solution quality feedback to students. This is done by computing the edit distance from a student's submission to some predefined set of solutions. In this work, we propose an algorithm for the edit distance problem that significantly outperforms the baseline procedure of exhaustively enumerating over the entire search space. Our algorithm relies on a reduction to the minimum vertex cover problem. We benchmark our algorithm on thousands of student submissions from multiple courses, showing that the baseline algorithm is intractable, and that our proposed algorithm is critical to enable classroom deployment. Our new algorithm has also been used for problems in many other domains where the solution space can be modeled as a DAG, including but not limited to Parsons Problems for writing code, helping students understand packet ordering in networking protocols, and helping students sketch solution steps for physics problems. Integrated into multiple learning management systems, the algorithm serves thousands of students each year.
GRAM: Fast Fine-tuning of Pre-trained Language Models for Content-based Collaborative Filtering
Authors: Yoonseok Yang, Kyu Seok Kim, Minsam Kim, Juneyoung Park
Date: 2022-04-08 00:00:00
Proceeding: NAACL 2022 7
Tasks: Collaborative Filtering, GPU, Knowledge Tracing, News Recommendation

Content-based collaborative filtering (CCF) predicts user-item interactions based on both users' interaction history and items' content information. Recently, pre-trained language models (PLM) have been used to extract high-quality item encodings for CCF. However, it is resource-intensive to train a PLM-based CCF model in an end-to-end (E2E) manner, since optimization involves back-propagating through every content encoding within a given user interaction sequence. To tackle this issue, we propose GRAM (GRadient Accumulation for Multi-modality in CCF), which exploits the fact that a given item often appears multiple times within a batch of interaction histories. Specifically, Single-step GRAM aggregates each item encoding's gradients for back-propagation, with theoretic equivalence to the standard E2E training. As an extension of Single-step GRAM, we propose Multi-step GRAM, which increases the gradient update latency, achieving a further speedup with drastically less GPU memory. GRAM significantly improves training efficiency (up to 146x) on five datasets from two task domains of Knowledge Tracing and News Recommendation. Our code is available at https://github.com/yoonseok312/GRAM.
Uncertain Case Identifiers in Process Mining: A User Study of the Event-Case Correlation Problem on Click Data
Authors: Marco Pegoraro, Merih Seran Uysal, Tom-Hendrik Hülsmann, Wil M. P. van der Aalst
Date: 2022-04-08 00:00:00

Among the many sources of event data available today, a prominent one is user interaction data. User activity may be recorded during the use of an application or website, resulting in a type of user interaction data often called click data. An obstacle to the analysis of click data using process mining is the lack of a case identifier in the data. In this paper, we show a case and user study for event-case correlation on click data, in the context of user interaction events from a mobility sharing company. To reconstruct the case notion of the process, we apply a novel method to aggregate user interaction data in separate user sessions-interpreted as cases-based on neural networks. To validate our findings, we qualitatively discuss the impact of process mining analyses on the resulting well-formed event log through interviews with process experts.
Process Mining on Uncertain Event Data
Authors: Marco Pegoraro
Date: 2022-04-08 00:00:00
Tasks: Attribute

With the widespread adoption of process mining in organizations, the field of process science is seeing an increase in the demand for ad-hoc analysis techniques of non-standard event data. An example of such data are uncertain event data: events characterized by a described and quantified attribute imprecision. This paper outlines a research project aimed at developing process mining techniques able to extract insights from uncertain data. We set the basis for this research topic, recapitulate the available literature, and define a future outlook.