pwc-archive/papers-with-abstracts huggingface dataset:
A Multi-Granularity Opinion Summarization Method
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Clustering, Opinion Mining, Opinion Summarization

Existing opinion mining (OM) is limited to applications on commercial reviews, with aspect and sentiment of the opinions in a coarse-grained form. In this paper, we further explore the definition of OM by extending the concepts of aspect and sentiment, and propose an opinion summarization method based on Multi-granularity Clustering and BERT (Jacob et al., 2018), i.e., MCB for emergent online discussion record in keeping with the further definition. A supporting Chinese corpus, ZH45 comprising 45 groups of discussion, and assorted metrics are also proposed.  Experiments based on ZH45 and the metrics demonstrate that MCB produces succinct and insightful opinion summaries.
Meta-CQG: A Meta-Learning Framework for Complex Question Generation over Knowledge Graph
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Decoder, Meta-Learning, Question Generation, Question-Generation, Relation

  Complex question generation (CQG) aims to generate questions involving multiple Knowledge Base (KB) relations or functional constraints. Existing methods train an encoder-decoder-based model to fit all questions. However, the questions in the real world exhibit an imbalanced distribution in many dimensions, such as question type, relation class, entity class, and query structure. This results in insufficient learning for minority class samples under different dimensions. To address this problem, we propose a meta-learning framework for complex question generation. It trains a unique generator for each sample via retrieving a few most related training samples, which can deeply and quickly dive into the content features (e.g. relation and entity) and structure features (e.g. query structure) of each sample. As retrieved samples directly determine the effectiveness of each unique generator, we design a self-supervised graph retriever to learn the potential features of samples and retrieve the most related samples according to multiple dimensions. We conduct experiments on both WebQuestionSP and ComplexWebQuestion, the results on the minority class of different dimensions have been significantly improved, which demonstrates the effectiveness of the proposed framework.
Evaluating the timing and magnitude of semantic change in diachronic word embedding models
Authors: Anonymous
Date: 2022-01-20 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Articles, Diachronic Word Embeddings, Word Embeddings

Recent studies have suggested that diachronic word embedding models are able to track the direction of changes in public perception. Building on these works, we evaluate the ability of diachronic word embedding models to accurately capture such changes both qualitatively and quantitatively, such as their timing and magnitudes. Using a longitudinal dataset on public perception of brands, we found that evolution of word meaning as captured by diachronic word embedding models, trained on New York Times articles, reflected the timing and magnitudes of general consumer awareness of companies. In contrast, this was not the case for other readily available characteristics, such as stock market prices. This comparison is enabled by a new feature extraction method which summarizes the semantic changes encoded in diachronic word embeddings.
Enhanced Knowledge Graphs Using Typed Entailment Graphs
Authors: Anonymous
Date: 2022-01-20 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Knowledge Graphs, Open Information Extraction, Question Answering

  Constructing knowledge graphs from open-domain corpora is a crucial stage in question answering. Most previous works are based on open information extraction methods, which extract relations by parsing sentences into triples <e1, r, e2>. These methods lack inference ability and are limited by corpus. When the query is different from the relations in the text-based knowledge graph, it is hard to return correct answers. In this paper, we propose a method to enhance knowledge graphs by using typed entailment graphs to add missing links. We construct the enhanced knowledge graph in both dynamical and offline ways. The experiment shows that our method outperforms the pre-trained language models in zero-shot cloze-style question answering. Furthermore, we find entailment graphs can significantly improve the recall and F-score of knowledge graphs.
Contrastive Event Extraction Using Video Enhancements
Authors: Anonymous
Date: 2022-01-20 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Event Extraction, Sentence

Event extraction aims to extract information of triggers associated with arguments from texts. Recent advanced methods consider the multi-modality to tackle the task by pairing the modalities without guaranteeing the alignment of event information across modalities, which negatively impacts on the model performances. To address the issue, we firstly constructed the Text Video Event Extraction (TVEE) dataset with an inner annotator agreement of 83.4\%, containing 7,598 pairs of text-videos, each of which is connected by event alignments.  To the best of our knowledge,  this is the first multimodal dataset with aligned event information in each sentence and video pair. Secondly, we present a \textbf{C}ontrastive \textbf{L}earning based \textbf{E}vent \textbf{E}xtraction model with enhancements from the \textbf{V}ideo modality (CLEEV) to pair videos and texts using event information. CLEEV constructs negative samples by measuring event weights based on occurrences of event types to enhance the contrast.We conducted experiments on the TVEE and VM2E2 datasets by incorporating modalities to assist the event extraction, outperforming SOTA methods with 1.0 and 1.2 point percentage improvements in terms of F-score, respectively.Our experimental results show that the multimedia information improves the event extraction from the textual modality\footnote{The dataset and code will be released based on acceptance.
Statistical word segmentation in spontaneous child-directed speech of Korean
Authors: Anonymous
Date: 2022-01-20 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Segmentation

The present study demonstrates advantages of child-directed speech (CDS) over adult-directed speech (ADS) in statistical word segmentation of spontaneous Korean. We derived phonetic input from phonemic corpus by applying a set of phonological rules. For modeling the statistical word segmentation based on transitional probability (TP), we used two syllable-based algorithms (i.e., Absolute and Relative) in two directions (i.e., Forward TP and Backward TP). Results show that (i) segmentation accuracy is greater with phonetic input than phonemic,  (ii) The model performs better when trained on CDS than ADS, and (iii) segmentation accuracy improves with child age.
EmpHi: Generating Empathetic Responses with Human-like Intents
Authors: Anonymous
Date: 2022-01-20 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Diversity

In empathetic conversations, humans express their empathy to others with empathetic intents. However, most existing empathetic conversational methods suffer from a lack of empathetic intents, which leads to monotonous empathy. To address the bias of the empathetic intents distribution between empathetic dialogue models and humans, we propose a novel model to generate empathetic responses with human-like empathetic intents, EmpHi for short. Precisely, EmpHi learns the distribution of potential empathetic intents with a discrete latent variable, then combines both implicit and explicit intent representation to generate responses with various empathetic intents. Experiments show that EmpHi outperforms state-of-the-art models in terms of empathy, relevance, and diversity on both automatic and human evaluation. Moreover, the case studies demonstrate the high interpretability and outstanding performance of our model.
Linearised Laplace Inference in Networks with Normalisation Layers and the Neural g-Prior
Authors: Javier Antoran, James Urquhart Allingham, David Janz, Erik Daxberger, Eric Nalisnick, José Miguel Hernández-Lobato
Date: 2021-11-22 00:00:00
Proceeding: pproximateinference AABI Symposium 2022 2
Tasks: image-classification, Image Classification, Model Selection, regression

We show that for neural networks (NN) with normalisation layers, i.e. batch norm, layer norm, or group norm, the Laplace model evidence does not approximate the volume of a posterior mode and is thus unsuitable for model selection. We instead propose to use the Laplace evidence of the linearized network, which is robust to the presence of these layers. We also identify heterogeneity in the scale of Jacobian entries corresponding to different weights. We ameliorate this issue by extending the scale-invariant g-prior to NNs. We demonstrate these methods on toy regression, and image classification with a CNN.
Learning Consistent Deep Generative Models from Sparsely Labeled Data
Authors: Gabriel Hope, Madina Abdrakhmanova, Xiaoyin Chen, Michael C Hughes, Erik B. Sudderth
Date: 2021-11-22 00:00:00
Proceeding: pproximateinference AABI Symposium 2022 2
Tasks: image-classification, Image Classification

We consider training deep generative models toward two simultaneous goals: discriminative classification and generative modeling using an explicit likelihood. While variational autoencoders (VAEs) offer a promising solution, we show that the dominant approach to training semi-supervised VAEs has several key weaknesses: it is fragile as generative modeling capacity increases, it is slow due to a required marginalization over labels, and it incoherently decouples into separate discriminative and generative models when all data is labeled. We remedy these concerns in a new proposed framework for semi-supervised VAE training that considers a more coherent downstream model architecture and a new objective which maximizes generative quality subject to a task-specific prediction constraint that ensures discriminative quality. We further enforce a consistency constraint, derived naturally from the generative model, that requires predictions on reconstructed data to match those on the original data. We show that our contributions -- a downstream architecture with prediction constraints and consistency constraints -- lead to improved generative samples as well as accurate image classification, with consistency particularly crucial for accuracy on sparsely-labeled datasets. Our approach enables advances in generative modeling to directly boost semi-supervised classification, an ability we demonstrate by learning a "very deep" prediction-constrained VAE with many layers of latent variables.
Improved Inverse-Free Variational Bounds for Sparse Gaussian Processes
Authors: Mark van der Wilk, Artem Artemev, James Hensman
Date: 2021-11-22 00:00:00
Proceeding: pproximateinference AABI Symposium 2022 2
Tasks: Gaussian Processes

The need for matrix decompositions (inverses) is often named as a major impediment to scaling Gaussian process (GP) models, even in efficient approximations. To address this, Van der Wilk et al. (2020) introduced a variational lower bound that can be computed without these costly operations. We improve this bound by 1) simplifying it by removing the need for iterative procedures, and 2) making it more numerically stable. While these improvements do not result in a procedure that is faster in wall-clock time than existing variational bounds, they are likely to be necessary steps along the way.
Can Sequential Bayesian Inference Solve Continual Learning?
Authors: Samuel Kessler, Adam D. Cobb, Stefan Zohren, Stephen J. Roberts
Date: 2021-11-22 00:00:00
Proceeding: pproximateinference AABI Symposium 2022 2
Tasks: Bayesian Inference, Continual Learning, Sequential Bayesian Inference

Previous work in Continual Learning (CL) has used sequential Bayesian inference to prevent forgetting and accumulate knowledge from previous tasks. A limiting factor to performing Bayesian CL has been exact inference in a Bayesian Neural Network (NN). We perform sequential Bayesian inference with a Bayesian NN using Hamiltonian Monte Carlo (HMC) and propagate the posterior as a prior for a new task by fitting a density estimator on HMC samples. We find that this approach fails to prevent forgetting. We propose an alternative view of the CL problem which directly models the data generating process and decomposes the CL problem in task specific and shared parameters. This method named Prototypical Bayesian CL and performs well compared to the latest Bayesian CL methods.
Meta-learning richer priors for VAEs
Authors: Marcello Massimo Negri, Vincent Fortuin, Jan Stuehmer
Date: 2021-11-22 00:00:00
Proceeding: pproximateinference AABI Symposium 2022 2
Tasks: Meta-Learning

Variational auto-encoders have proven to capture complicated data distributions and useful latent representations, while advances in meta-learning have made it possible to extract prior knowledge from data. We incorporate these two approaches and propose a novel flexible prior, namely the Pseudo-inputs prior, to obtain a richer latent space. We train VAEs using the Model-Agnostic Meta-Learning (MAML) algorithm and show that it achieves comparable reconstruction performance with standard training. However, we show that this MAML-VAE model learns richer latent representations, which we evaluate in terms of unsupervised few-shot classification as a downstream task. Moreover, we show that our proposed Pseudo-inputs prior outperforms baseline priors, including the VampPrior, in both models, while also encouraging high-level representations through its pseudo-inputs.
A Probabilistic Deep Image Prior over Image Space
Authors: Riccardo Barbano, Javier Antoran, José Miguel Hernández-Lobato, Bangti Jin
Date: 2021-11-22 00:00:00
Proceeding: pproximateinference AABI Symposium 2022 2
Tasks: Image Reconstruction

The deep image prior regularises under-specified image reconstruction problems by reparametrising the target image as the output of a CNN. We induce a prior over images by scoring CNN outputs using a classical image reconstruction regulariser. We translate this functional prior into weight space using a change of variables and propose an efficient linearised Laplace inference algorithm. Hyperparameters are optimised with Type-II MAP. We obtain pixel-wise uncertainty estimates, which we show to be calibrated to the reconstruction error.
Matrix Inversion free variational inference in Conditional Student's T Processes
Authors: Sebastian Popescu, Ben Glocker, Mark van der Wilk
Date: 2021-11-22 00:00:00
Proceeding: pproximateinference AABI Symposium 2022 2
Tasks: valid, Variational Inference

We propose a new variational lower bound for performing inference in sparse Student's T Processes that does not require computationally intensive operations such as matrix inversions or log determinants of matrices. We devise a mathematically valid and easy-to-sample from approximate posterior over the Inverse Wishart distributed covariance matrix encompassing both training data and inducing points. To deal with remaining log determinant terms we propose a conjugate gradient based lower bound that is tight when approximate posteriors are optimal. We demonstrate that our proposed model behaves similarly to matrix inversion dependent counterparts in terms of convergence of evidence lower bound and predictive capabilities at testing time on a wide array of toy experiments. Moreover, we test the validity of our method on medium and large scale datasets, showing encouraging results.
Shooting Schrödinger’s Cat
Authors: David Lopes Fernandes, Francisco Vargas, Carl Henrik Ek, Neill D. F. Campbell
Date: 2021-11-22 00:00:00
Proceeding: pproximateinference AABI Symposium 2022 2
Tasks: Variational Inference

We  present  a  variational  inference  scheme  to  learn  a  model  that  solves  the  Schrödinger Bridge Problem (SBP). In contrast to previous work, our approach is solver-agnostic and guarantees solutions that respect the prior beyond the first fitting iteration.  Having this solution allows us to generate new samples from one of the distributions by first sampling from the other one and then solving the dynamical system.  We show that our model is able to learn the transformation between the Gaussian distribution and arbitrary data, as well as learning dynamics that follow a potential function.
Variational Likelihood-Free Gradient Descent
Authors: Jack Simons, Song Liu, Mark Beaumont
Date: 2021-11-22 00:00:00
Proceeding: pproximateinference AABI Symposium 2022 2
Tasks: Density Ratio Estimation

In many scientific applications, we do not have explicit access to the likelihood function. However simulations of the process of interest, using different parameter settings, may give us access to the likelihood function implicitly. The methodology for approximating likelihoods and posterior distributions based on simulated observations can be described as simulation-based inference. In this paper, we propose a simulation-based inference algorithm in which we iteratively update particles to more closely resemble the posterior. Our approach utilises simulations to estimate a density ratio function at each iteration and then uses it to approximate the KL divergence between the particle density and the posterior density. By alternating between gradient descent and density ratio estimation, the approximated KL divergence is minimized. We benchmark the performance of our algorithm on a Gaussian mixture model and the M/G/1 queue process model and report promising results.
Probabilistic Deep Learning with Generalised Variational Inference
Authors: Giorgos Felekis, Theo Damoulas, Brooks Paige
Date: 2021-11-22 00:00:00
Proceeding: pproximateinference AABI Symposium 2022 2
Tasks: Bayesian Inference, Deep Learning, Probabilistic Deep Learning, Uncertainty Quantification, Variational Inference

We study probabilistic Deep Learning methods through the lens of Approximate Bayesian Inference. In particular, we examine Bayesian Neural Networks (BNNs), which usually suffer from multiple ill-posed assumptions such as prior and likelihood misspecification. In this direction, we investigate a recently proposed approximate inference framework called Generalised Variational Inference (GVI) in comparison to state-of-the-art methods including standard Variational Inference, Monte-Carlo Dropout, Stochastic gradient Langevin dynamics and Deep Ensembles. Also, we expand the original research around GVI by exploring a broader set of model architectures and mathematical settings on both real and synthetic data. Our experiments demonstrate that approximate posterior distributions derived from such a method offer attractive properties with respect to uncertainty quantification, prior specification robustness and predictive performance, especially in the case of BNNs.
Efficient Bayesian Inverse Reinforcement Learning via Conditional Kernel Density Estimation
Authors: Aishwarya Mandyam, Didong Li, Diana Cai, Andrew Jones, Barbara Engelhardt
Date: 2021-11-22 00:00:00
Proceeding: pproximateinference AABI Symposium 2022 2
Tasks: BIRL, Density Estimation, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

Inverse reinforcement learning (IRL) methods attempt to recover the reward function of an agent by observing its behavior. Given the large amount of uncertainty in the underlying reward function, it is often useful to model this function probabilistically, rather than estimate a single reward function. However, existing Bayesian approaches to IRL use a Q-value function to approximate the likelihood, leading to a computationally intractable and inflexible framework. Here, we introduce kernel density Bayesian IRL (KD-BIRL), a method that uses kernel density estimation to approximate the likelihood, or the probability of the observed states and actions given a reward function. This approximation allows for efficient posterior inference of the reward function given a sequence of agent observations. Empirically, using both linear and nonlinear reward functions in a Gridworld environment, we demonstrate that the KD-BIRL posterior centers around the true reward function.
PAC-Bayesian matrix completion with a spectral scaled Student prior
Authors: T Tien Mai
Date: 2021-11-22 00:00:00
Proceeding: pproximateinference AABI Symposium 2022 2
Tasks: Image Inpainting, Matrix Completion

We study the problem of matrix completion in this paper. A spectral scaled Student prior is exploited to favour the underlying low-rank structure of the data matrix. We provide a thorough theoretical investigation for our approach through PAC-Bayesian bounds. More precisely, our PAC-Bayesian approach enjoys a minimax-optimal oracle inequality which guarantees that our method works well under model misspecification and under general sampling distribution. Interestingly, we also provide efficient gradient-based sampling im- plementations for our approach by using Langevin Monte Carlo. More specifically, we show that our algorithms are significantly faster than Gibbs sampler in this problem. To illustrate the attractive features of our inference strategy, some numerical simulations are conducted and an application to image inpainting is demonstrated.
Structured Stochastic Gradient MCMC: a hybrid VI and MCMC approach
Authors: Antonios Alexos, Alex James Boyd, Stephan Mandt
Date: 2021-11-22 00:00:00
Proceeding: pproximateinference AABI Symposium 2022 2
Tasks: Bayesian Inference, Variational Inference

Stochastic gradient Markov chain Monte Carlo (SGMCMC) is considered the gold standard for Bayesian inference in large-scale models, such as Bayesian neural networks. Since practitioners face speed versus accuracy tradeoffs in these models, variational inference (VI) is often the preferable option. Unfortunately, VI makes strong assumptions on both the factorization and functional form of the posterior. In this work, we propose a new non-parametric variational approximation that makes no assumptions about the approximate posterior’s functional form and allows practitioners to specify the exact dependencies the algorithm should respect or break. The approach relies on a new Langevin-type algorithm that operates on a modified energy function, where parts of the latent variables are averaged over samples from earlier iterations of the Markov chain. This way, statistical dependencies can be broken in a controlled way, allowing the chain to mix faster. This scheme can be further modified in a “dropout” manner, leading to even more scalability. By implementing the scheme on a ResNet-20 architecture, we obtain better predictive likelihoods and larger effective sample sizes than full SGMCMC.
Towards automated Capability Assessment leveraging Deep Learning
Authors: Raoul Schönhof, Manuel Fechter
Date: 2022-01-28 00:00:00
Tasks: Deep Learning, Position

Aiming for a higher economic efficiency in manufacturing, an increased degree of automation is a key enabler. However, assessing the technical feasibility of an automated assembly solution for a dedicated process is difficult and often determined by the geometry of the given product parts. Among others, decisive criterions of the automation feasibility are the ability to separate and isolate single parts or the capability of component self-alignment in final position. To assess the feasibility, a questionnaire based evaluation scheme has been developed and applied by Fraunhofer researchers. However, the results strongly depend on the implicit knowledge and experience of the single engineer performing the assessment. This paper presents NeuroCAD, a software tool that automates the assessment using voxelization techniques. The approach enables the assessment of abstract and production relevant geometries features through deep-learning based on CAD files.