pwc-archive/papers-with-abstracts huggingface dataset:
Evaluating Generative Patent Language Models
Authors: Jieh-Sheng Lee
Date: 2022-06-23 00:00:00
Tasks: Language Modelling

Generative language models are promising for assisting human writing in various domains. This manuscript aims to build generative language models in the patent domain and evaluate model performance from a human-centric perspective. The perspective is to measure the ratio of keystrokes that can be saved by autocompletion based on generative patent language models. A higher ratio means a more effective model which can save more keystrokes. This metric can be used to benchmark model performance. The metric is different from conventional machine-centric metrics that are token-based instead of keystroke-based. In terms of model size, the largest model built in this manuscript is 6B, which is state-of-the-art in the patent domain. Based on the metric, it is found that the largest model is not necessarily the best for the human-centric metric. The finding means that keeping increasing model sizes in the patent domain might be unnecessary if the purpose is to assist human writing with autocompletion. Several patent language models are pre-trained from scratch in this research. The pre-trained models are released for future researchers. Several visualization tools are also provided. The importance of building a generative language model in the patent domain is the potential to facilitate creativity and innovations in the future.
Using cognitive psychology to understand GPT-3
Authors: Marcel Binz, Eric Schulz
Date: 2022-06-21 00:00:00
Tasks: Decision Making, Language Modeling, Language Modelling, Large Language Model, Model-based Reinforcement Learning

We study GPT-3, a recent large language model, using tools from cognitive psychology. More specifically, we assess GPT-3's decision-making, information search, deliberation, and causal reasoning abilities on a battery of canonical experiments from the literature. We find that much of GPT-3's behavior is impressive: it solves vignette-based tasks similarly or better than human subjects, is able to make decent decisions from descriptions, outperforms humans in a multi-armed bandit task, and shows signatures of model-based reinforcement learning. Yet we also find that small perturbations to vignette-based tasks can lead GPT-3 vastly astray, that it shows no signatures of directed exploration, and that it fails miserably in a causal reasoning task. These results enrich our understanding of current large language models and pave the way for future investigations using tools from cognitive psychology to study increasingly capable and opaque artificial agents.
Why Robust Natural Language Understanding is a Challenge
Authors: Marco Casadio, Ekaterina Komendantskaya, Verena Rieser, Matthew L. Daggitt, Daniel Kienitz, Luca Arnaboldi, Wen Kokke
Date: 2022-06-21 00:00:00
Tasks: Natural Language Understanding

With the proliferation of Deep Machine Learning into real-life applications, a particular property of this technology has been brought to attention: robustness Neural Networks notoriously present low robustness and can be highly sensitive to small input perturbations. Recently, many methods for verifying networks' general properties of robustness have been proposed, but they are mostly applied in Computer Vision. In this paper we propose a Verification specification for Natural Language Understanding classification based on larger regions of interest, and we discuss the challenges of such task. We observe that, although the data is almost linearly separable, the verifier struggles to output positive results and we explain the problems and implications.
Knowledge Graph Fusion for Language Model Fine-tuning
Authors: Nimesh Bhana, Terence L. Van Zyl
Date: 2022-06-21 00:00:00
Tasks: Language Modeling, Language Modelling, model, Semantic Similarity, Semantic Textual Similarity

Language Models such as BERT have grown in popularity due to their ability to be pre-trained and perform robustly on a wide range of Natural Language Processing tasks. Often seen as an evolution over traditional word embedding techniques, they can produce semantic representations of text, useful for tasks such as semantic similarity. However, state-of-the-art models often have high computational requirements and lack global context or domain knowledge which is required for complete language understanding. To address these limitations, we investigate the benefits of knowledge incorporation into the fine-tuning stages of BERT. An existing K-BERT model, which enriches sentences with triplets from a Knowledge Graph, is adapted for the English language and extended to inject contextually relevant information into sentences. As a side-effect, changes made to K-BERT for accommodating the English language also extend to other word-based languages. Experiments conducted indicate that injected knowledge introduces noise. We see statistically significant improvements for knowledge-driven tasks when this noise is minimised. We show evidence that, given the appropriate task, modest injection with relevant, high-quality knowledge is most performant.
A hybrid level-based learning swarm algorithm with mutation operator for solving large-scale cardinality-constrained portfolio optimization problems
Authors: Massimiliano Kaucic, Filippo Piccotto, Gabriele Sbaiz, Giorgio Valentinuz
Date: 2022-06-29 00:00:00
Tasks: Portfolio Optimization

In this work, we propose a hybrid variant of the level-based learning swarm optimizer (LLSO) for solving large-scale portfolio optimization problems. Our goal is to maximize a modified formulation of the Sharpe ratio subject to cardinality, box and budget constraints. The algorithm involves a projection operator to deal with these three constraints simultaneously and we implicitly control transaction costs thanks to a rebalancing constraint. We also introduce a suitable exact penalty function to manage the turnover constraint. In addition, we develop an ad hoc mutation operator to modify candidate exemplars in the highest level of the swarm. The experimental results, using three large-scale data sets, show that the inclusion of this procedure improves the accuracy of the solutions. Then, a comparison with other variants of the LLSO algorithm and two state-of-the-art swarm optimizers points out the outstanding performance of the proposed solver in terms of exploration capabilities and solution quality. Finally, we assess the profitability of the portfolio allocation strategy in the last five years using an investible pool of 1119 constituents from the MSCI World Index.
Score Matching for Truncated Density Estimation on a Manifold
Authors: Daniel J. Williams, Song Liu
Date: 2022-06-29 00:00:00
Tasks: Density Estimation

When observations are truncated, we are limited to an incomplete picture of our dataset. Recent methods propose to use score matching for truncated density estimation, where the access to the intractable normalising constant is not required. We present a novel extension of truncated score matching to a Riemannian manifold with boundary. Applications are presented for the von Mises-Fisher and Kent distributions on a two dimensional sphere in $\mathbb{R}^3$, as well as a real-world application of extreme storm observations in the USA. In simulated data experiments, our score matching estimator is able to approximate the true parameter values with a low estimation error and shows improvements over a naive maximum likelihood estimator.
Treatment Effect Estimation with Observational Network Data using Machine Learning
Authors: Corinne Emmenegger, Meta-Lina Spohn, Timon Elmer, Peter Bühlmann
Date: 2022-06-29 00:00:00
Tasks: Causal Inference

Causal inference methods for treatment effect estimation usually assume independent units. However, this assumption is often questionable because units may interact, resulting in spillover effects between them. We develop augmented inverse probability weighting (AIPW) for estimation and inference of the expected average treatment effect (EATE) with observational data from a single (social) network with spillover effects. In contrast to overall effects such as the global average treatment effect (GATE), the EATE measures, in expectation and on average over all units, how the outcome of a unit is causally affected by its own treatment, marginalizing over the spillover effects from other units. We develop cross-fitting theory with plugin machine learning to obtain a semiparametric treatment effect estimator that converges at the parametric rate and asymptotically follows a Gaussian distribution. The asymptotics are developed using the dependency graph rather than the network graph, which makes explicit that we allow for spillover effects beyond immediate neighbors in the network. We apply our AIPW method to the Swiss StudentLife Study data to investigate the effect of hours spent studying on exam performance accounting for the students' social network.
How Train-Test Leakage Affects Zero-shot Retrieval
Authors: Maik Fröbe, Christopher Akiki, Martin Potthast, Matthias Hagen
Date: 2022-06-29 00:00:00
Tasks: Retrieval

Neural retrieval models are often trained on (subsets of) the millions of queries of the MS MARCO / ORCAS datasets and then tested on the 250 Robust04 queries or other TREC benchmarks with often only 50 queries. In such setups, many of the few test queries can be very similar to queries from the huge training data -- in fact, 69% of the Robust04 queries have near-duplicates in MS MARCO / ORCAS. We investigate the impact of this unintended train-test leakage by training neural retrieval models on combinations of a fixed number of MS MARCO / ORCAS queries that are highly similar to the actual test queries and an increasing number of other queries. We find that leakage can improve effectiveness and even change the ranking of systems. However, these effects diminish as the amount of leakage among all training instances decreases and thus becomes more realistic.
Supervised Deep Hashing for High-dimensional and Heterogeneous Case-based Reasoning
Authors: Qi Zhang, Liang Hu, Chongyang Shi, Ke Liu, Longbing Cao
Date: 2022-06-29 00:00:00
Tasks: Deep Hashing, Incremental Learning, Quantization, Retrieval, Vocal Bursts Intensity Prediction

Case-based Reasoning (CBR) on high-dimensional and heterogeneous data is a trending yet challenging and computationally expensive task in the real world. A promising approach is to obtain low-dimensional hash codes representing cases and perform a similarity retrieval of cases in Hamming space. However, previous methods based on data-independent hashing rely on random projections or manual construction, inapplicable to address specific data issues (e.g., high-dimensionality and heterogeneity) due to their insensitivity to data characteristics. To address these issues, this work introduces a novel deep hashing network to learn similarity-preserving compact hash codes for efficient case retrieval and proposes a deep-hashing-enabled CBR model HeCBR. Specifically, we introduce position embedding to represent heterogeneous features and utilize a multilinear interaction layer to obtain case embeddings, which effectively filtrates zero-valued features to tackle high-dimensionality and sparsity and captures inter-feature couplings. Then, we feed the case embeddings into fully-connected layers, and subsequently a hash layer generates hash codes with a quantization regularizer to control the quantization loss during relaxation. To cater to incremental learning of CBR, we further propose an adaptive learning strategy to update the hash function. Extensive experiments on public datasets show that HeCBR greatly reduces storage and significantly accelerates case retrieval. HeCBR achieves desirable performance compared with the state-of-the-art CBR methods and performs significantly better than hashing-based CBR methods in classification.
Minimalist and High-performance Conversational Recommendation with Uncertainty Estimation for User Preference
Authors: Yinan Zhang, Boyang Li, Yong liu, You Yuan, Chunyan Miao
Date: 2022-06-29 00:00:00
Tasks: Attribute, Conversational Recommendation, Reinforcement Learning (RL)

Conversational recommendation system (CRS) is emerging as a user-friendly way to capture users' dynamic preferences over candidate items and attributes. Multi-shot CRS is designed to make recommendations multiple times until the user either accepts the recommendation or leaves at the end of their patience. Existing works are trained with reinforcement learning (RL), which may suffer from unstable learning and prohibitively high demands for computing. In this work, we propose a simple and efficient CRS, MInimalist Non-reinforced Interactive COnversational Recommender Network (MINICORN). MINICORN models the epistemic uncertainty of the estimated user preference and queries the user for the attribute with the highest uncertainty. The system employs a simple network architecture and makes the query-vs-recommendation decision using a single rule. Somewhat surprisingly, this minimalist approach outperforms state-of-the-art RL methods on three real-world datasets by large margins. We hope that MINICORN will serve as a valuable baseline for future research.
Two-Stage Neural Contextual Bandits for Personalised News Recommendation
Authors: Mengyan Zhang, Thanh Nguyen-Tang, Fangzhao Wu, Zhenyu He, Xing Xie, Cheng Soon Ong
Date: 2022-06-26 00:00:00
Tasks: Computational Efficiency, Multi-Armed Bandits, News Recommendation, Vocal Bursts Valence Prediction

We consider the problem of personalised news recommendation where each user consumes news in a sequential fashion. Existing personalised news recommendation methods focus on exploiting user interests and ignores exploration in recommendation, which leads to biased feedback loops and hurt recommendation quality in the long term. We build on contextual bandits recommendation strategies which naturally address the exploitation-exploration trade-off. The main challenges are the computational efficiency for exploring the large-scale item space and utilising the deep representations with uncertainty. We propose a two-stage hierarchical topic-news deep contextual bandits framework to efficiently learn user preferences when there are many news items. We use deep learning representations for users and news, and generalise the neural upper confidence bound (UCB) policies to generalised additive UCB and bilinear UCB. Empirical results on a large-scale news recommendation dataset show that our proposed policies are efficient and outperform the baseline bandit policies.
Robust Online Voltage Control with an Unknown Grid Topology
Authors: Christopher Yeh, Jing Yu, Yuanyuan Shi, Adam Wierman
Date: 2022-06-29 00:00:00

Voltage control generally requires accurate information about the grid's topology in order to guarantee network stability. However, accurate topology identification is a challenging problem for existing methods, especially as the grid is subject to increasingly frequent reconfiguration due to the adoption of renewable energy. Further, running existing control mechanisms with incorrect network information may lead to unstable control. In this work, we combine a nested convex body chasing algorithm with a robust predictive controller to achieve provably finite-time convergence to safe voltage limits in the online setting where the network topology is initially unknown. Specifically, the online controller does not know the true network topology and line parameters, but instead must learn them over time by narrowing down the set of network topologies and line parameters that are consistent with its observations and adjusting reactive power generation accordingly to keep voltages within desired safety limits. We demonstrate the effectiveness of the approach using a case study, which shows that in practical settings the controller is indeed able to narrow the set of consistent topologies quickly enough to make control decisions that ensure stability.
Sequential 3D Human Pose Estimation Using Adaptive Point Cloud Sampling Strategy
Authors: Zihao Zhang, Lei Hu, Xiaoming Deng, Shihong Xia
Date: 2021-08-19 00:00:00
Proceeding: the Thirtieth International Joint Conference on Artificial Intelligence 2021 8
Tasks: 3D Human Pose Estimation, Pose Estimation

3D human pose estimation is a fundamental problem in artificial intelligence, and it has wide applications in AR/VR, HCI and robotics. However, human pose estimation from point clouds still suffers from noisy points and estimated jittery artifacts because of handcrafted-based point cloud sampling and single-frame-based estimation strategies. In this paper, we present a new perspective on the 3D human pose estimation method from point cloud sequences. To sample effective point clouds from input, we design a differentiable point cloud sampling method built on density-guided attention mechanism. To avoid the jitter caused by previous 3D human pose estimation problems, we adopt temporal information to obtain more stable results. Experiments on the ITOP dataset and the NTU-RGBD dataset demonstrate that all of our contributed components are effective, and our method can achieve state-of-the-art performance
f-AnoGAN: Fast Unsupervised Anomaly Detection with Generative Adversarial Networks
Authors: Thomas Schlegl，PhilippSeeböck，Sebastian M.Waldstein，GeorgLangs，UrsulaSchmidt-Erfurth
Date: 2019-01-30 00:00:00
Proceeding: Medical Image Analysis 2019 1
Tasks: Anomaly Classification, Anomaly Detection, Anomaly Segmentation, Generative Adversarial Network, Image Reconstruction, Unsupervised Anomaly Detection

Obtaining expert labels in clinical imaging is difficult since exhaustive annotation is time-consuming. Furthermore, not all possibly relevant markers may be known and sufficiently well described a priori to even guide annotation. While supervised learning yields good results if expert labeled training data is available, the visual variability, and thus the vocabulary of findings, we can detect and exploit, is limited to the annotated lesions. Here, we present fast AnoGAN (f-AnoGAN), a generative adversarial network (GAN) based unsupervised learning approach capable of identifying anomalous images and image segments, that can serve as imaging biomarker candidates. We build a generative model of healthy training data, and propose and evaluate a fast mapping technique of new data to the GAN’s latent space. The mapping is based on a trained encoder, and anomalies are detected via a combined anomaly score based on the building blocks of the trained model – comprising a discriminator feature residual error and an image reconstruction error. In the experiments on optical coherence tomography data, we compare the proposed method with alternative approaches, and provide comprehensive empirical evidence that f-AnoGAN outperforms alternative approaches and yields high anomaly detection accuracy. In addition, a visual Turing test with two retina experts showed that the generated images are indistinguishable from real normal retinal OCT images. The f-AnoGAN code is available at https://github.com/tSchlegl/f-AnoGAN.
Kepler-aSI at SemTab 2021
Authors: Wiem Baazouzi, Marouen Kachroudi, Sami Faiz
Date: 2021-10-01 00:00:00
Proceeding: SemTab@ISWC 2021 10
Tasks: Cell Entity Annotation, Column Type Annotation, Graph Matching, Knowledge Graphs, Table annotation

In this paper, we present our system Kepler-aSI, for the Semantic Web Challenge on Tabular Data to Knowledge Graph Matching (SemTab 2021). This system is participating for the second time in this campaign, bringing improvements and new technical aspects. Kepler-aSI analyzes tabular data to be able to detect correct matches in Wiki-data and DBPedia. It should be noted that each data resource, or each round of the campaign imposes a certain number of constraints, requiring advanced techniques. The aforementioned task turns out to be difficult for the machines, which requires an additional effort in order to deploy the cognitive capacity in the matching methods. Kepler-aSI still relies on the SPARQL query to semantically annotate tables in Knowledge Graphs (KG), in order to solve the critical problems of matching tasks. The results obtained during the evaluation phase are encouraging and show the strengths of the proposed system.
MAGIC: Mining an Augmented Graph using INK, starting from a CSV
Authors: Bram Steenwinckel, Filip De Turck, Femke Ongenae
Date: 2021-10-01 00:00:00
Proceeding: SemTab@ISWC 2021 10
Tasks: Cell Entity Annotation, Column Type Annotation, Graph Matching, Table annotation

A large portion of structured data does not yet reap the benefits of the Semantic Web. Therefore, The “Tabular Data to Knowledge Graph Matching” competition at ISWC tries to bridge this gap by evaluating and promoting the creation of such semantic annotations tools. Besides annotating data semantically, the system should also be able to further augment the datasets based on the provided annotations. In this paper, we propose a system that is capable of both annotating and augmenting a dataset by using the interpretable embedding technique INK. The “Tabular Data to Knowledge Graph Matching” competition was used to evaluate the proposed annotation capabilities of our proposed system.
JenTab Meets SemTab 2021's New Challenges
Authors: Nora Abdelmageed, Sirko Schindler
Date: 2021-10-01 00:00:00
Proceeding: SemTab@ISWC 2021 10
Tasks: Cell Entity Annotation, Column Type Annotation, Graph Matching, Table annotation

While tables are a rich source of structured information, their automated use is oftentimes prevented by the inherent ambiguity contained within. Issues ranging from mere typos over inconsistent naming conventions to homonymy among values pose substantial barriers to exploiting this source of knowledge. Although the Semantic Web can alleviate many of these issues, the actual annotation process remains challenging. To foster new ideas and the improvement of existing approaches, the Semantic Web Challenge on Tabular Data to Knowledge Graph Matching (SemTab) since 2019 hosts yearly competitions allowing systems to present their current capabilities. Datasets of different origins and characteristics highlight the various challenges present in this area. In this paper, we report on the evolution of our system, “JenTab”, during SemTab2021. We re-designed the system architecture, optimized individual modules, and developed various pipelines to target specific challenges posed throughout the challenge. JenTab is among the top 5 systems in the first two rounds of SemTab2021. The results demonstrate JenTab’s flexibility and its ability to quickly address new challenges.
Nonblind image deconvolution via leveraging model uncertainty in an untrained deep neural network
Authors: Mingqin Chen; Yuhui Quan; Tongyao Pang; Hui Ji
Date: 2022-05-18 00:00:00
Proceeding: International Journal of Computer Vision 2022 5
Tasks: Bayesian Inference, Image Deconvolution

Nonblind image deconvolution (NID) is about restoring the latent image with sharp details from
a noisy blurred one using a known blur kernel. This paper presents a dataset-free deep learning
approach for NID using untrained deep neural networks (DNNs), which does not require any external
training data with ground-truth images. Based on a spatially-adaptive dropout scheme, the proposed
approach learns a DNN with model uncertainty from the input blurred image, and the deconvolution result is obtained by aggregating the multiple predictions from the learned dropout DNN.
It is shown that the solution approximates a minimum-mean-squared-error estimator in Bayesian
inference. In addition, a self-supervised loss function for training is presented to efficiently handle
the noise in blurred images. Extensive experiments show that the proposed approach not only performs noticeably better than existing non-learning-based methods and unsupervised learning-based
methods, but also performs competitively against recent supervised learning-based methods.