pwc-archive/papers-with-abstracts huggingface dataset:
Biologically-Inspired Continual Learning of Human Motion Sequences
Authors: Joachim Ott, Shih-Chii Liu
Date: 2022-11-02 00:00:00
Tasks: Continual Learning, Temporal Sequences

This work proposes a model for continual learning on tasks involving temporal sequences, specifically, human motions. It improves on a recently proposed brain-inspired replay model (BI-R) by building a biologically-inspired conditional temporal variational autoencoder (BI-CTVAE), which instantiates a latent mixture-of-Gaussians for class representation. We investigate a novel continual-learning-to-generate (CL2Gen) scenario where the model generates motion sequences of different classes. The generative accuracy of the model is tested over a set of tasks. The final classification accuracy of BI-CTVAE on a human motion dataset after sequentially learning all action classes is 78%, which is 63% higher than using no-replay, and only 5.4% lower than a state-of-the-art offline trained GRU model.
MuMIC -- Multimodal Embedding for Multi-label Image Classification with Tempered Sigmoid
Authors: Fengjun Wang, Sarai Mizrachi, Moran Beladev, Guy Nadav, Gil Amsalem, Karen Lastmann Assaraf, Hadas Harush Boker
Date: 2022-11-02 00:00:00
Tasks: Classification, image-classification, Image Classification, Multi-Label Classification, MUlTI-LABEL-ClASSIFICATION, Multi-Label Image Classification, Representation Learning, Transfer Learning

Multi-label image classification is a foundational topic in various domains. Multimodal learning approaches have recently achieved outstanding results in image representation and single-label image classification. For instance, Contrastive Language-Image Pretraining (CLIP) demonstrates impressive image-text representation learning abilities and is robust to natural distribution shifts. This success inspires us to leverage multimodal learning for multi-label classification tasks, and benefit from contrastively learnt pretrained models. We propose the Multimodal Multi-label Image Classification (MuMIC) framework, which utilizes a hardness-aware tempered sigmoid based Binary Cross Entropy loss function, thus enables the optimization on multi-label objectives and transfer learning on CLIP. MuMIC is capable of providing high classification performance, handling real-world noisy data, supporting zero-shot predictions, and producing domain-specific image embeddings. In this study, a total of 120 image classes are defined, and more than 140K positive annotations are collected on approximately 60K Booking.com images. The final MuMIC model is deployed on Booking.com Content Intelligence Platform, and it outperforms other state-of-the-art models with 85.6% GAP@10 and 83.8% GAP on all 120 classes, as well as a 90.1% macro mAP score across 32 majority classes. We summarize the modeling choices which are extensively tested through ablation studies. To the best of our knowledge, we are the first to adapt contrastively learnt multimodal pretraining for real-world multi-label image classification problems, and the innovation can be transferred to other domains.
SimuShips -- A High Resolution Simulation Dataset for Ship Detection with Precise Annotations
Authors: Minahil Raza, Hanna Prokopova, Samir Huseynzade, Sepinoud Azimi, Sebastien Lafond
Date: 2022-09-22 00:00:00

Obstacle detection is a fundamental capability of an autonomous maritime surface vessel (AMSV). State-of-the-art obstacle detection algorithms are based on convolutional neural networks (CNNs). While CNNs provide higher detection accuracy and fast detection speed, they require enormous amounts of data for their training. In particular, the availability of domain-specific datasets is a challenge for obstacle detection. The difficulty in conducting onsite experiments limits the collection of maritime datasets. Owing to the logistic cost of conducting on-site operations, simulation tools provide a safe and cost-efficient alternative for data collection. In this work, we introduce SimuShips, a publicly available simulation-based dataset for maritime environments. Our dataset consists of 9471 high-resolution (1920x1080) images which include a wide range of obstacle types, atmospheric and illumination conditions along with occlusion, scale and visible proportion variations. We provide annotations in the form of bounding boxes. In addition, we conduct experiments with YOLOv5 to test the viability of simulation data. Our experiments indicate that the combination of real and simulated images improves the recall for all classes by 2.9%.
Computer-Aided Modelling of the Bilingual Word Indices to the Ninth-Century Uchitel'noe evangelie
Authors: Martin Ruskov, Lora Taseva
Date: 2022-10-25 00:00:00
Tasks: Translation

The development of bilingual dictionaries to medieval translations presents diverse difficulties. These result from two types of philological circumstances: a) the asymmetry between the source language and the target language; and b) the varying available sources of both the original and translated texts. In particular, the full critical edition of Tihova of Constantine of Preslav's Uchitel'noe evangelie ('Didactic Gospel') gives a relatively good idea of the Old Church Slavonic translation but not of its Greek source text. This is due to the fact that Cramer's edition of the catenae - used as the parallel text in it - is based on several codices whose text does not fully coincide with the Slavonic. This leads to the addition of the newly-discovered parallels from Byzantine manuscripts and John Chrysostom's homilies. Our approach to these issues is a step-wise process with two main goals: a) to facilitate the philological annotation of input data and b) to consider the manifestations of the mentioned challenges, first, separately in order to simplify their resolution, and, then, in their combination. We demonstrate how we model various types of asymmetric translation correlates and the variability resulting from the pluralism of sources. We also demonstrate how all these constructions are being modelled and processed into the final indices. Our approach is designed with generalisation in mind and is intended to be applicable also for other translations from Greek into Old Church Slavonic.
Towards automatic generation of Piping and Instrumentation Diagrams (P&IDs) with Artificial Intelligence
Authors: Edwin Hirtreiter, Lukas Schulze Balhorn, Artur M. Schweidtmann
Date: 2022-10-26 00:00:00
Tasks: Transfer Learning, Translation

Developing Piping and Instrumentation Diagrams (P&IDs) is a crucial step during the development of chemical processes. Currently, this is a tedious, manual, and time-consuming task. We propose a novel, completely data-driven method for the prediction of control structures. Our methodology is inspired by end-to-end transformer-based human language translation models. We cast the control structure prediction as a translation task where Process Flow Diagrams (PFDs) are translated to P&IDs. To use established transformer-based language translation models, we represent the P&IDs and PFDs as strings using our recently proposed SFILES 2.0 notation. Model training is performed in a transfer learning approach. Firstly, we pre-train our model using generated P&IDs to learn the grammatical structure of the process diagrams. Thereafter, the model is fine-tuned leveraging transfer learning on real P&IDs. The model achieved a top-5 accuracy of 74.8% on 10,000 generated P&IDs and 89.2% on 100,000 generated P&IDs. These promising results show great potential for AI-assisted process engineering. The tests on a dataset of 312 real P&IDs indicate the need of a larger P&IDs dataset for industry applications.
Using contradictions improves question answering systems
Authors: Étienne Fortier-Dubois, Domenic Rosati
Date: 2022-09-28 00:00:00
Tasks: Multiple-choice, Natural Language Inference, Question Answering

This work examines the use of contradiction in natural language inference (NLI) for question answering (QA). Typically, NLI systems help answer questions by determining if a potential answer is \emph{entailed} (supported) by some background context. But is it useful to also determine if an answer contradicts the context? We test this in two settings, multiple choice and extractive QA, and find that systems that incorporate contradiction can do slightly better than entailment-only systems on certain datasets. However, the best performances come from using contradiction, entailment, and QA model confidence scores together. This has implications for the deployment of QA systems in domains such as medicine and science where safety is an issue.
Moving beyond word lists: towards abstractive topic labels for human-like topics of scientific documents
Authors: Domenic Rosati
Date: 2022-10-28 00:00:00
Tasks: Document Summarization, Multi-Document Summarization, Topic Models

Topic models represent groups of documents as a list of words (the topic labels). This work asks whether an alternative approach to topic labeling can be developed that is closer to a natural language description of a topic than a word list. To this end, we present an approach to generating human-like topic labels using abstractive multi-document summarization (MDS). We investigate our approach with an exploratory case study. We model topics in citation sentences in order to understand what further research needs to be done to fully operationalize MDS for topic labeling. Our case study shows that in addition to more human-like topics there are additional advantages to evaluation by using clustering and summarization measures instead of topic model measures. However, we find that there are several developments needed before we can design a well-powered study to evaluate MDS for topic modeling fully. Namely, improving cluster cohesion, improving the factuality and faithfulness of MDS, and increasing the number of documents that might be supported by MDS. We present a number of ideas on how these can be tackled and conclude with some thoughts on how topic modeling can also be used to improve MDS in general.
Generating Clear Images From Images With Distortions Caused by Adverse Weather Using Generative Adversarial Networks
Authors: Nuriel Shalom Mor
Date: 2022-11-01 00:00:00
Tasks: Autonomous Driving, Autonomous Vehicles, Generative Adversarial Network, Image Reconstruction, Object Recognition

We presented a method for improving computer vision tasks on images affected by adverse weather conditions, including distortions caused by adherent raindrops. Overcoming the challenge of applying computer vision to images affected by adverse weather conditions is essential for autonomous vehicles utilizing RGB cameras. For this purpose, we trained an appropriate generative adversarial network and showed that it was effective at removing the effect of the distortions, in the context of image reconstruction and computer vision tasks. We showed that object recognition, a vital task for autonomous driving vehicles, is completely impaired by the distortions and occlusions caused by adherent raindrops and that performance can be restored by our de-raining model. The approach described in this paper could be applied to all adverse weather conditions.
Haven't I Seen You Before? Assessing Identity Leakage in Synthetic Irises
Authors: Patrick Tinsley, Adam Czajka, Patrick Flynn
Date: 2022-11-03 00:00:00

Generative Adversarial Networks (GANs) have proven to be a preferred method of synthesizing fake images of objects, such as faces, animals, and automobiles. It is not surprising these models can also generate ISO-compliant, yet synthetic iris images, which can be used to augment training data for iris matchers and liveness detectors. In this work, we trained one of the most recent GAN models (StyleGAN3) to generate fake iris images with two primary goals: (i) to understand the GAN's ability to produce "never-before-seen" irises, and (ii) to investigate the phenomenon of identity leakage as a function of the GAN's training time. Previous work has shown that personal biometric data can inadvertently flow from training data into synthetic samples, raising a privacy concern for subjects who accidentally appear in the training dataset. This paper presents analysis for three different iris matchers at varying points in the GAN training process to diagnose where and when authentic training samples are in jeopardy of leaking through the generative process. Our results show that while most synthetic samples do not show signs of identity leakage, a handful of generated samples match authentic (training) samples nearly perfectly, with consensus across all matchers. In order to prioritize privacy, security, and trust in the machine learning model development process, the research community must strike a delicate balance between the benefits of using synthetic data and the corresponding threats against privacy from potential identity leakage.
Rare Wildlife Recognition with Self-Supervised Representation Learning
Authors: Xiaochen Zheng
Date: 2022-10-29 00:00:00
Tasks: Contrastive Learning, Representation Learning

Automated animal censuses with aerial imagery are a vital ingredient towards wildlife conservation. Recent models are generally based on supervised learning and thus require vast amounts of training data. Due to their scarcity and minuscule size, annotating animals in aerial imagery is a highly tedious process. In this project, we present a methodology to reduce the amount of required training data by resorting to self-supervised pretraining. In detail, we examine a combination of recent contrastive learning methodologies like Momentum Contrast (MoCo) and Cross-Level Instance-Group Discrimination (CLD) to condition our model on the aerial images without the requirement for labels. We show that a combination of MoCo, CLD, and geometric augmentations outperforms conventional models pretrained on ImageNet by a large margin. Meanwhile, strategies for smoothing label or prediction distribution in supervised learning have been proven useful in preventing the model from overfitting. We combine the self-supervised contrastive models with image mixup strategies and find that it is useful for learning more robust visual representations. Crucially, our methods still yield favorable results even if we reduce the number of training animals to just 10%, at which point our best model scores double the recall of the baseline at similar precision. This effectively allows reducing the number of required annotations to a fraction while still being able to train high-accuracy models in such highly challenging settings.
Adversarial Multimodal Domain Transfer for Video-Level Sentiment Analysis
Authors: Wang Yanan; Wu Jianming; Furumai Kazuaki; Wada Shinya; Kurihara Satoshi
Date: 2022-05-11 00:00:00
Proceeding: IEEE Access 2022 5
Tasks: Multimodal Sentiment Analysis, Sentiment Analysis

Video-level sentiment analysis is a challenging task and requires systems to obtain discriminative multimodal representations that can capture difference in sentiments across various modalities.
However, due to diverse distributions of various modalities and the unified multimodal labels are not always
adaptable to unimodal learning, the distance difference between unimodal representations increases, and
prevents systems from learning discriminative multimodal representations. In this paper, to obtain more
discriminative multimodal representations that can further improve systems’ performance, we propose a
VAE-based adversarial multimodal domain transfer (VAE-AMDT) and jointly train it with a multi-attention
module to reduce the distance difference between unimodal representations. We first perform variational
autoencoder (VAE) to make visual, linguistic and acoustic representations follow a common distribution,
and then introduce adversarial training to transfer all unimodal representations to a joint embedding space.
As a result, we fuse various modalities on this joint embedding space via the multi-attention module,
which consists of self-attention, cross-attention and triple-attention for highlighting important sentimental
representations over time and modality. Our method improves F1-score of the state-of-the-art by 3.6%
on MOSI and 2.9% on MOSEI datasets, and prove its efficacy in obtaining discriminative multimodal
representations for video-level sentiment analysis.
Network Amplification With Efficient MACs Allocation
Authors: Chuanjian Liu, Kai Han, An Xiao, Ying Nie, Wei zhang, Yunhe Wang
Date: 2022-07-01 00:00:00
Proceeding: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops 2022 7

Recent studies on deep convolutional neural networks present a simple paradigm of architecture design, i.e., models with more MACs typically achieve better accuracies, such as EfficientNet and RegNet. These works try to enlarge the network architecture with one unified rule by sampling and statistical methods. However, the rule is not prospective to the design of large networks because it is obtained from the experience of researchers on small network architectures. In this paper, we propose to enlarge the capacity of CNN models by fine-grained MACs allocation for the width, depth and resolution on the stage level. In particular, starting from a base small model, we gradually add extra channels, layers or resolution by using a dynamic programming manner. With step-by-step modifying the computations on different stages, the enlarged network will be equipped with optimal allocation and utilization of MACs. On EfficientNet, our method consistently outperforms the performance of the original scaling method. In particular, the proposed method is used to enlarge models sourced by GhostNet, we achieve state-of-the-art 80.9% and 84.3% ImageNet top-1 accuracies under the setting of 600M and 4.4B MACs, respectively.
KAMEL : Knowledge Analysis with Multitoken Entities in Language Models
Authors: Jan-Christoph Kalo, Leandra Fichtel
Date: 2022-11-01 00:00:00
Proceeding: Automated Knowledge Base Construction 2022 11
Tasks: Knowledge Graphs, Probing Language Models

Large language models (LMs) have been shown to capture large amounts of relational
knowledge from the pre-training corpus. These models can be probed for this factual knowledge by using cloze-style prompts as demonstrated on the LAMA benchmark. However,
recent studies have uncovered that results only perform well, because the models are good
at performing educated guesses or recalling facts from the training data. We present a novel
Wikidata-based benchmark dataset, KAMEL , for probing relational knowledge in LMs.
In contrast to previous datasets, it covers a broader range of knowledge, probes for single-,
and multi-token entities, and contains facts with literal values. Furthermore, the evaluation
procedure is more accurate, since the dataset contains alternative entity labels and deals
with higher-cardinality relations. Instead of performing the evaluation on masked language
models, we present results for a variety of recent causal LMs in a few-shot setting. We show
that indeed novel models perform very well on LAMA, achieving a promising F1-score of
52.90%, while only achieving 17.62% on KAMEL. Our analysis shows that even large language models are far from being able to memorize all varieties of relational knowledge that
is usually stored knowledge graphs.
Align, Write, Re-order: Explainable End-to-End Speech Translation via Operation Sequence Generation
Authors: Motoi Omachi, Brian Yan, Siddharth Dalmia, Yuya Fujita, Shinji Watanabe
Date: 2022-11-11 00:00:00
Tasks: Automatic Speech Recognition, Automatic Speech Recognition (ASR), speech-recognition, Speech Recognition, Translation

The black-box nature of end-to-end speech translation (E2E ST) systems makes it difficult to understand how source language inputs are being mapped to the target language. To solve this problem, we would like to simultaneously generate automatic speech recognition (ASR) and ST predictions such that each source language word is explicitly mapped to a target language word. A major challenge arises from the fact that translation is a non-monotonic sequence transduction task due to word ordering differences between languages -- this clashes with the monotonic nature of ASR. Therefore, we propose to generate ST tokens out-of-order while remembering how to re-order them later. We achieve this by predicting a sequence of tuples consisting of a source word, the corresponding target words, and post-editing operations dictating the correct insertion points for the target word. We examine two variants of such operation sequences which enable generation of monotonic transcriptions and non-monotonic translations from the same speech input simultaneously. We apply our approach to offline and real-time streaming models, demonstrating that we can provide explainable translations without sacrificing quality or latency. In fact, the delayed re-ordering ability of our approach improves performance during streaming. As an added benefit, our method performs ASR and ST simultaneously, making it faster than using two separate systems to perform these tasks.
CoRAL: a Context-aware Croatian Abusive Language Dataset
Authors: Ravi Shekhar, Mladen Karan, Matthew Purver
Date: 2022-11-11 00:00:00
Tasks: Abusive Language

In light of unprecedented increases in the popularity of the internet and social media, comment moderation has never been a more relevant task. Semi-automated comment moderation systems greatly aid human moderators by either automatically classifying the examples or allowing the moderators to prioritize which comments to consider first. However, the concept of inappropriate content is often subjective, and such content can be conveyed in many subtle and indirect ways. In this work, we propose CoRAL -- a language and culturally aware Croatian Abusive dataset covering phenomena of implicitness and reliance on local and global context. We show experimentally that current models degrade when comments are not explicit and further degrade when language skill and context knowledge are required to interpret the comment.
OneFormer: One Transformer to Rule Universal Image Segmentation
Authors: Jitesh Jain, Jiachen Li, MangTik Chiu, Ali Hassani, Nikita Orlov, Humphrey Shi
Date: 2022-11-10 00:00:00
Proceeding: CVPR 2023 1
Tasks: Instance Segmentation, Panoptic Segmentation, Scene Parsing, Segmentation, Semantic Segmentation, Universal Segmentation

Universal Image Segmentation is not a new concept. Past attempts to unify image segmentation in the last decades include scene parsing, panoptic segmentation, and, more recently, new panoptic architectures. However, such panoptic architectures do not truly unify image segmentation because they need to be trained individually on the semantic, instance, or panoptic segmentation to achieve the best performance. Ideally, a truly universal framework should be trained only once and achieve SOTA performance across all three image segmentation tasks. To that end, we propose OneFormer, a universal image segmentation framework that unifies segmentation with a multi-task train-once design. We first propose a task-conditioned joint training strategy that enables training on ground truths of each domain (semantic, instance, and panoptic segmentation) within a single multi-task training process. Secondly, we introduce a task token to condition our model on the task at hand, making our model task-dynamic to support multi-task training and inference. Thirdly, we propose using a query-text contrastive loss during training to establish better inter-task and inter-class distinctions. Notably, our single OneFormer model outperforms specialized Mask2Former models across all three segmentation tasks on ADE20k, CityScapes, and COCO, despite the latter being trained on each of the three tasks individually with three times the resources. With new ConvNeXt and DiNAT backbones, we observe even more performance improvement. We believe OneFormer is a significant step towards making image segmentation more universal and accessible. To support further research, we open-source our code and models at https://github.com/SHI-Labs/OneFormer