pwc-archive/papers-with-abstracts huggingface dataset:
Non-Monotonic Latent Alignments for CTC-Based Non-Autoregressive Machine Translation
Authors: Chenze Shao, Yang Feng
Date: 2022-10-08 00:00:00
Tasks: Machine Translation, Sentence, Translation

Non-autoregressive translation (NAT) models are typically trained with the cross-entropy loss, which forces the model outputs to be aligned verbatim with the target sentence and will highly penalize small shifts in word positions. Latent alignment models relax the explicit alignment by marginalizing out all monotonic latent alignments with the CTC loss. However, they cannot handle non-monotonic alignments, which is non-negligible as there is typically global word reordering in machine translation. In this work, we explore non-monotonic latent alignments for NAT. We extend the alignment space to non-monotonic alignments to allow for the global word reordering and further consider all alignments that overlap with the target sentence. We non-monotonically match the alignments to the target sentence and train the latent alignment model to maximize the F1 score of non-monotonic matching. Extensive experiments on major WMT benchmarks show that our method substantially improves the translation performance of CTC-based models. Our best model achieves 30.06 BLEU on WMT14 En-De with only one-iteration decoding, closing the gap between non-autoregressive and autoregressive models.
ConstGCN: Constrained Transmission-based Graph Convolutional Networks for Document-level Relation Extraction
Authors: Ji Qi, Bin Xu, Kaisheng Zeng, Jinxin Liu, Jifan Yu, Qi Gao, Juanzi Li, Lei Hou
Date: 2022-10-08 00:00:00
Tasks: Document-level Relation Extraction, graph construction, Relation, Relation Extraction

Document-level relation extraction with graph neural networks faces a fundamental graph construction gap between training and inference - the golden graph structure only available during training, which causes that most methods adopt heuristic or syntactic rules to construct a prior graph as a pseudo proxy. In this paper, we propose $\textbf{ConstGCN}$, a novel graph convolutional network which performs knowledge-based information propagation between entities along with all specific relation spaces without any prior graph construction. Specifically, it updates the entity representation by aggregating information from all other entities along with each relation space, thus modeling the relation-aware spatial information. To control the information flow passing through the indeterminate relation spaces, we propose to constrain the propagation using transmitting scores learned from the Noise Contrastive Estimation between fact triples. Experimental results show that our method outperforms the previous state-of-the-art (SOTA) approaches on the DocRE dataset.
Detecting Label Errors in Token Classification Data
Authors: Wei-Chen Wang, Jonas Mueller
Date: 2022-10-08 00:00:00
Tasks: General Classification, Token Classification

Mislabeled examples are a common issue in real-world data, particularly for tasks like token classification where many labels must be chosen on a fine-grained basis. Here we consider the task of finding sentences that contain label errors in token classification datasets. We study 11 different straightforward methods that score tokens/sentences based on the predicted class probabilities output by a (any) token classification model (trained via any procedure). In precision-recall evaluations based on real-world label errors in entity recognition data from CoNLL-2003, we identify a simple and effective method that consistently detects those sentences containing label errors when applied with different token classification models.
Improving End-to-End Text Image Translation From the Auxiliary Text Translation Task
Authors: Cong Ma, Yaping Zhang, Mei Tu, Xu Han, Linghui Wu, Yang Zhao, Yu Zhou
Date: 2022-10-08 00:00:00
Tasks: Multi-Task Learning, Translation

End-to-end text image translation (TIT), which aims at translating the source language embedded in images to the target language, has attracted intensive attention in recent research. However, data sparsity limits the performance of end-to-end text image translation. Multi-task learning is a non-trivial way to alleviate this problem via exploring knowledge from complementary related tasks. In this paper, we propose a novel text translation enhanced text image translation, which trains the end-to-end model with text translation as an auxiliary task. By sharing model parameters and multi-task training, our model is able to take full advantage of easily-available large-scale text parallel corpus. Extensive experimental results show our proposed method outperforms existing end-to-end methods, and the joint multi-task learning with both text translation and recognition tasks achieves better results, proving translation and recognition auxiliary tasks are complementary.
Don't Lose Yourself! Empathetic Response Generation via Explicit Self-Other Awareness
Authors: Weixiang Zhao, Yanyan Zhao, Xin Lu, Bing Qin
Date: 2022-10-08 00:00:00
Tasks: Empathetic Response Generation, Response Generation

As a critical step to achieve human-like chatbots, empathetic response generation has attained increasing interests. Previous attempts are incomplete and not sufficient enough to elicit empathy because they only focus on the initial aspect of empathy to automatically mimic the feelings and thoughts of the user via other-awareness. However, they ignore to maintain and take the own views of the system into account, which is a crucial process to achieve the empathy called self-other awareness. To this end, we propose to generate Empathetic response with explicit Self-Other Awareness (EmpSOA). Specifically, three stages, self-other differentiation, self-other modulation and self-other generation, are devised to clearly maintain, regulate and inject the self-other aware information into the process of empathetic response generation. Both automatic and human evaluations on the benchmark dataset demonstrate the superiority of EmpSOA to generate more empathetic responses.
Data-Efficiency with a Single GPU: An Exploration of Transfer Methods for Small Language Models
Authors: Alon Albalak, Akshat Shrivastava, Chinnadhurai Sankar, Adithya Sagar, Mike Ross
Date: 2022-10-08 00:00:00
Tasks: GPU, Multi-Task Learning

Multi-task learning (MTL), instruction tuning, and prompting have recently been shown to improve the generalizability of large language models to new tasks. However, the benefits of such methods are less well-documented in smaller language models, with some studies finding contradictory results. In this work, we explore and isolate the effects of (i) model size, (ii) general purpose MTL, (iii) in-domain MTL, (iv) instruction tuning, and (v) few-shot fine-tuning for models with fewer than 500 million parameters. Our experiments in the zero-shot setting demonstrate that models gain 31% relative improvement, on average, from general purpose MTL, with an additional 37.6% relative gain from in-domain MTL. Contradictory to prior works on large models, we find that instruction tuning provides a modest 2% performance improvement for small models.
ConvFinQA: Exploring the Chain of Numerical Reasoning in Conversational Finance Question Answering
Authors: Zhiyu Chen, Shiyang Li, Charese Smiley, Zhiqiang Ma, Sameena Shah, William Yang Wang
Date: 2022-10-07 00:00:00
Tasks: Conversational Question Answering, Question Answering

With the recent advance in large pre-trained language models, researchers have achieved record performances in NLP tasks that mostly focus on language pattern matching. The community is experiencing the shift of the challenge from how to model language to the imitation of complex reasoning abilities like human beings. In this work, we investigate the application domain of finance that involves real-world, complex numerical reasoning. We propose a new large-scale dataset, ConvFinQA, aiming to study the chain of numerical reasoning in conversational question answering. Our dataset poses great challenge in modeling long-range, complex numerical reasoning paths in real-world conversations. We conduct comprehensive experiments and analyses with both the neural symbolic methods and the prompting-based methods, to provide insights into the reasoning mechanisms of these two divisions. We believe our new dataset should serve as a valuable resource to push forward the exploration of real-world, complex reasoning tasks as the next research focus. Our dataset and code is publicly available at https://github.com/czyssrs/ConvFinQA.
Breaking BERT: Evaluating and Optimizing Sparsified Attention
Authors: Siddhartha Brahma, Polina Zablotskaia, David Mimno
Date: 2022-10-07 00:00:00

Transformers allow attention between all pairs of tokens, but there is reason to believe that most of these connections - and their quadratic time and memory - may not be necessary. But which ones? We evaluate the impact of sparsification patterns with a series of ablation experiments. First, we compare masks based on syntax, lexical similarity, and token position to random connections, and measure which patterns reduce performance the least. We find that on three common finetuning tasks even using attention that is at least 78% sparse can have little effect on performance if applied at later transformer layers, but that applying sparsity throughout the network reduces performance significantly. Second, we vary the degree of sparsity for three patterns supported by previous work, and find that connections to neighbouring tokens are the most significant. Finally, we treat sparsity as an optimizable parameter, and present an algorithm to learn degrees of neighboring connections that gives a fine-grained control over the accuracy-sparsity trade-off while approaching the performance of existing methods.
Retrieval Augmented Visual Question Answering with Outside Knowledge
Authors: Weizhe Lin, Bill Byrne
Date: 2022-10-07 00:00:00
Tasks: Answer Generation, Diagnostic, Passage Retrieval, Question Answering, Retrieval, Visual Question Answering, Visual Question Answering (VQA)

Outside-Knowledge Visual Question Answering (OK-VQA) is a challenging VQA task that requires retrieval of external knowledge to answer questions about images. Recent OK-VQA systems use Dense Passage Retrieval (DPR) to retrieve documents from external knowledge bases, such as Wikipedia, but with DPR trained separately from answer generation, introducing a potential limit on the overall system performance. Instead, we propose a joint training scheme which includes differentiable DPR integrated with answer generation so that the system can be trained in an end-to-end fashion. Our experiments show that our scheme outperforms recent OK-VQA systems with strong DPR for retrieval. We also introduce new diagnostic metrics to analyze how retrieval and generation interact. The strong retrieval ability of our model significantly reduces the number of retrieved documents needed in training, yielding significant benefits in answer quality and computation required for training.
Named Entity Recognition in Twitter: A Dataset and Analysis on Short-Term Temporal Shifts
Authors: Asahi Ushio, Leonardo Neves, Vitor Silva, Francesco Barbieri, Jose Camacho-Collados
Date: 2022-10-07 00:00:00
Tasks: Articles, Language Modeling, Language Modelling, Named Entity Recognition, Named Entity Recognition (NER)

Recent progress in language model pre-training has led to important improvements in Named Entity Recognition (NER). Nonetheless, this progress has been mainly tested in well-formatted documents such as news, Wikipedia, or scientific articles. In social media the landscape is different, in which it adds another layer of complexity due to its noisy and dynamic nature. In this paper, we focus on NER in Twitter, one of the largest social media platforms, and construct a new NER dataset, TweetNER7, which contains seven entity types annotated over 11,382 tweets from September 2019 to August 2021. The dataset was constructed by carefully distributing the tweets over time and taking representative trends as a basis. Along with the dataset, we provide a set of language model baselines and perform an analysis on the language model performance on the task, especially analyzing the impact of different time periods. In particular, we focus on three important temporal aspects in our analysis: short-term degradation of NER models over time, strategies to fine-tune a language model over different periods, and self-labeling as an alternative to lack of recently-labeled data. TweetNER7 is released publicly (https://huggingface.co/datasets/tner/tweetner7) along with the models fine-tuned on it.
Training Spiking Neural Networks with Local Tandem Learning
Authors: Qu Yang, Jibin Wu, Malu Zhang, Yansong Chua, Xinchao Wang, Haizhou Li
Date: 2022-10-10 00:00:00

Spiking neural networks (SNNs) are shown to be more biologically plausible and energy efficient over their predecessors. However, there is a lack of an efficient and generalized training method for deep SNNs, especially for deployment on analog computing substrates. In this paper, we put forward a generalized learning rule, termed Local Tandem Learning (LTL). The LTL rule follows the teacher-student learning approach by mimicking the intermediate feature representations of a pre-trained ANN. By decoupling the learning of network layers and leveraging highly informative supervisor signals, we demonstrate rapid network convergence within five training epochs on the CIFAR-10 dataset while having low computational complexity. Our experimental results have also shown that the SNNs thus trained can achieve comparable accuracies to their teacher ANNs on CIFAR-10, CIFAR-100, and Tiny ImageNet datasets. Moreover, the proposed LTL rule is hardware friendly. It can be easily implemented on-chip to perform fast parameter calibration and provide robustness against the notorious device non-ideality issues. It, therefore, opens up a myriad of opportunities for training and deployment of SNN on ultra-low-power mixed-signal neuromorphic computing chips.10
Motion Planning on Visual Manifolds
Authors: M Seetha Ramaiah
Date: 2022-10-08 00:00:00
Tasks: Motion Planning

In this thesis, we propose an alternative characterization of the notion of Configuration Space, which we call Visual Configuration Space (VCS). This new characterization allows an embodied agent (e.g., a robot) to discover its own body structure and plan obstacle-free motions in its peripersonal space using a set of its own images in random poses. Here, we do not assume any knowledge of geometry of the agent, obstacles or the environment. We demonstrate the usefulness of VCS in (a) building and working with geometry-free models for robot motion planning, (b) explaining how a human baby might learn to reach objects in its peripersonal space through motor babbling, and (c) automatically generating natural looking head motion animations for digital avatars in virtual environments. This work is based on the formalism of manifolds and manifold learning using the agent's images and hence we call it Motion Planning on Visual Manifolds.
Self-organizing nest migration dynamics synthesis for ant colony systems
Authors: Matin Macktoobian
Date: 2022-10-08 00:00:00

In this study, we synthesize a novel dynamical approach for ant colonies enabling them to migrate to new nest sites in a self-organizing fashion. In other words, we realize ant colony migration as a self-organizing phenotype-level collective behavior. For this purpose, we first segment the edges of the graph of ants' pathways. Then, each segment, attributed to its own pheromone profile, may host an ant. So, multiple ants may occupy an edge at the same time. Thanks to this segment-wise edge formulation, ants have more selection options in the course of their pathway determination, thereby increasing the diversity of their colony's emergent behaviors. In light of the continuous pheromone dynamics of segments, each edge owns a spatio-temporal piece-wise continuous pheromone profile in which both deposit and evaporation processes are unified. The passive dynamics of the proposed migration mechanism is sufficiently rich so that an ant colony can migrate to the vicinity of a new nest site in a self-organizing manner without any external supervision. In particular, we perform extensive simulations to test our migration dynamics applied to a colony including 500 ants traversing a pathway graph comprising 200 nodes and 4000 edges which are segmented based on various resolutions. The obtained results exhibit the effectiveness of our strategy.
An associative memory model with very high memory rate: Image storage by sequential addition learning
Authors: Hiroshi Inazawa
Date: 2022-10-08 00:00:00
Tasks: Memorization

In this paper, we present a neural network system related to about memory and recall that consists of one neuron group (the "cue ball") and a one-layer neural net (the "recall net"). This system realizes the bidirectional memorization learning between one cue neuron in the cue ball and the neurons in the recall net. It can memorize many patterns and recall these patterns or those that are similar at any time. Furthermore, the patterns are recalled at most the same time. This model's recall situation seems to resemble human recall of a variety of similar things almost simultaneously when one thing is recalled. It is also possible for additional learning to occur in the system without affecting the patterns memorized in advance. Moreover, the memory rate (the number of memorized patterns / the total number of neurons) is close to 100%; this system's rate is 0.987. Finally, pattern data constraints become an important aspect of this system.
On Parallel or Distributed Asynchronous Iterations with Unbounded Delays and Possible Out of Order Messages or Flexible Communication for Convex Optimization Problems and Machine Learning
Authors: Didier El Baz
Date: 2022-10-05 00:00:00
Tasks: Survey

We describe several features of parallel or distributed asynchronous iterative algorithms such as unbounded delays, possible out of order messages or flexible communication. We concentrate on the concept of macroiteration sequence which was introduced in order to study the convergence or termination of asynchronous iterations. A survey of asynchronous iterations for convex optimization problems is also presented. Finally, a new result of convergence for parallel or distributed asynchronous iterative algorithms with flexible communication for convex optimization problems and machine learning is proposed.
A copula-based boosting model for time-to-event prediction with dependent censoring
Authors: Alise Danielle Midtfjord, Riccardo De Bin, Arne Bang Huseby
Date: 2022-10-10 00:00:00
Tasks: Time-to-Event Prediction

A characteristic feature of time-to-event data analysis is possible censoring of the event time. Most of the statistical learning methods for handling censored data are limited by the assumption of independent censoring, even if this can lead to biased predictions when the assumption does not hold. This paper introduces Clayton-boost, a boosting approach built upon the accelerated failure time model, which uses a Clayton copula to handle the dependency between the event and censoring distributions. By taking advantage of a copula, the independent censoring assumption is not needed any more. During comparisons with commonly used methods, Clayton-boost shows a strong ability to remove prediction bias at the presence of dependent censoring and outperforms the comparing methods either if the dependency strength or percentage censoring are considerable. The encouraging performance of Clayton-boost shows that there is indeed reasons to be critical about the independent censoring assumption, and that real-world data could highly benefit from modelling the potential dependency.
Bayesian Sparse Regression for Mixed Multi-Responses with Application to Runtime Metrics Prediction in Fog Manufacturing
Authors: Xiaoyu Chen, Xiaoning Kang, Ran Jin, Xinwei Deng
Date: 2022-10-10 00:00:00
Tasks: Prediction, Uncertainty Quantification, Variable Selection

Fog manufacturing can greatly enhance traditional manufacturing systems through distributed Fog computation units, which are governed by predictive computational workload offloading methods under different Industrial Internet architectures. It is known that the predictive offloading methods highly depend on accurate prediction and uncertainty quantification of runtime performance metrics, containing multivariate mixed-type responses (i.e., continuous, counting, binary). In this work, we propose a Bayesian sparse regression for multivariate mixed responses to enhance the prediction of runtime performance metrics and to enable the statistical inferences. The proposed method considers both group and individual variable selection to jointly model the mixed types of runtime performance metrics. The conditional dependency among multiple responses is described by a graphical model using the precision matrix, where a spike-and-slab prior is used to enable the sparse estimation of the graph. The proposed method not only achieves accurate prediction, but also makes the predictive model more interpretable with statistical inferences on model parameters and prediction in the Fog manufacturing. A simulation study and a real case example in a Fog manufacturing are conducted to demonstrate the merits of the proposed model.
Low-power multi-mode fiber projector overcomes shallow neural networks classifiers
Authors: Daniele Ancora, Matteo Negri, Antonio Gianfrate, Dimitris Trypogeorgos, Lorenzo Dominici, Daniele Sanvitto, Federico Ricci-Tersenghi, Luca Leuzzi
Date: 2022-10-10 00:00:00

In the domain of disordered photonics, the characterization of optically opaque materials for light manipulation and imaging is a primary aim. Among various complex devices, multi-mode optical fibers stand out as cost-effective and easy-to-handle tools, making them attractive for several tasks. In this context, we cast these fibers into random hardware projectors, transforming an input dataset into a higher dimensional speckled image set. The goal of our study is to demonstrate that using such randomized data for classification by training a single logistic regression layer improves accuracy compared to training on direct raw images. Interestingly, we found that the classification accuracy achieved is higher than that obtained with the standard transmission matrix model, a widely accepted tool for describing light transmission through disordered devices. We conjecture that the reason for such improved performance could be due to the fact that the hardware classifier operates in a flatter region of the loss landscape when trained on fiber data, which aligns with the current theory of deep neural networks. These findings suggest that the class of random projections operated by multi-mode fibers generalize better to previously unseen data, positioning them as promising tools for optically-assisted neural networks. With this study, in fact, we want to contribute to advancing the knowledge and practical utilization of these versatile instruments, which may play a significant role in shaping the future of neuromorphic machine learning.
Identifying patterns of main causes of death in the young EU population
Authors: Simona Korenjak-Černe, Nataša Kejžar
Date: 2022-10-10 00:00:00

The study of mortality patterns is a popular research topic in many areas. We are particularly interested in mortality patterns among main causes of death associated with age-gender combinations. We use symbolic data analysis (SDA) and include three dimensions: age, gender, and patterns across main causes of death. In this study, we present an alternative method to identify clusters of EU countries with similar mortality patterns in the young population, while considering comprehensive information on the distribution of deaths among the main causes of death by different age-gender groups. We explore possible relationships between mortality patterns in the identified clusters and some other sociodemographic indicators. We use EU data of crude mortality rates from 2016, as the most recent complete data available.