pwc-archive/papers-with-abstracts huggingface dataset:
Error Bounds of Projection Models in Weakly Supervised 3D Human Pose Estimation
Authors: Nikolas Klug, Moritz Einfalt, Stephan Brehm, Rainer Lienhart
Date: 2020-10-23 00:00:00
Tasks: 3D Human Pose Estimation, Monocular 3D Human Pose Estimation, Pose Estimation, Position, Weakly-supervised 3D Human Pose Estimation

The current state-of-the-art in monocular 3D human pose estimation is heavily influenced by weakly supervised methods. These allow 2D labels to be used to learn effective 3D human pose recovery either directly from images or via 2D-to-3D pose uplifting. In this paper we present a detailed analysis of the most commonly used simplified projection models, which relate the estimated 3D pose representation to 2D labels: normalized perspective and weak perspective projections. Specifically, we derive theoretical lower bound errors for those projection models under the commonly used mean per-joint position error (MPJPE). Additionally, we show how the normalized perspective projection can be replaced to avoid this guaranteed minimal error. We evaluate the derived lower bounds on the most commonly used 3D human pose estimation benchmark datasets. Our results show that both projection models lead to an inherent minimal error between 19.3mm and 54.7mm, even after alignment in position and scale. This is a considerable share when comparing with recent state-of-the-art results. Our paper thus establishes a theoretical baseline that shows the importance of suitable projection models in weakly supervised 3D human pose estimation.
Matching the Clinical Reality: Accurate OCT-Based Diagnosis From Few Labels
Authors: Valentyn Melnychuk, Evgeniy Faerman, Ilja Manakov, Thomas Seidl
Date: 2020-10-23 00:00:00
Tasks: Diagnostic, Medical Image Analysis, Retinal OCT Disease Classification, Semi-Supervised Image Classification, Transfer Learning

Unlabeled data is often abundant in the clinic, making machine learning methods based on semi-supervised learning a good match for this setting. Despite this, they are currently receiving relatively little attention in medical image analysis literature. Instead, most practitioners and researchers focus on supervised or transfer learning approaches. The recently proposed MixMatch and FixMatch algorithms have demonstrated promising results in extracting useful representations while requiring very few labels. Motivated by these recent successes, we apply MixMatch and FixMatch in an ophthalmological diagnostic setting and investigate how they fare against standard transfer learning. We find that both algorithms outperform the transfer learning baseline on all fractions of labelled data. Furthermore, our experiments show that exponential moving average (EMA) of model parameters, which is a component of both algorithms, is not needed for our classification problem, as disabling it leaves the outcome unchanged. Our code is available online: https://github.com/Valentyn1997/oct-diagn-semi-supervised
Model-Based Machine Learning for Joint Digital Backpropagation and PMD Compensation
Authors: Rick M. Bütler, Christian Häger, Henry D. Pfister, Gabriele Liga, Alex Alvarado
Date: 2020-10-23 00:00:00
Tasks: BIG-bench Machine Learning

In this paper, we propose a model-based machine-learning approach for dual-polarization systems by parameterizing the split-step Fourier method for the Manakov-PMD equation. The resulting method combines hardware-friendly time-domain nonlinearity mitigation via the recently proposed learned digital backpropagation (LDBP) with distributed compensation of polarization-mode dispersion (PMD). We refer to the resulting approach as LDBP-PMD. We train LDBP-PMD on multiple PMD realizations and show that it converges within 1% of its peak dB performance after 428 training iterations on average, yielding a peak effective signal-to-noise ratio of only 0.30 dB below the PMD-free case. Similar to state-of-the-art lumped PMD compensation algorithms in practical systems, our approach does not assume any knowledge about the particular PMD realization along the link, nor any knowledge about the total accumulated PMD. This is a significant improvement compared to prior work on distributed PMD compensation, where knowledge about the accumulated PMD is typically assumed. We also compare different parameterization choices in terms of performance, complexity, and convergence behavior. Lastly, we demonstrate that the learned models can be successfully retrained after an abrupt change of the PMD realization along the fiber.
A Survey on Recent Approaches for Natural Language Processing in Low-Resource Scenarios
Authors: Michael A. Hedderich, Lukas Lange, Heike Adel, Jannik Strötgen, Dietrich Klakow
Date: 2020-10-23 00:00:00
Proceeding: NAACL 2021 4
Tasks: Data Augmentation, Survey, Transfer Learning

Deep neural networks and huge language models are becoming omnipresent in natural language applications. As they are known for requiring large amounts of training data, there is a growing body of work to improve the performance in low-resource settings. Motivated by the recent fundamental changes towards neural models and the popular pre-train and fine-tune paradigm, we survey promising approaches for low-resource natural language processing. After a discussion about the different dimensions of data availability, we give a structured overview of methods that enable learning when training data is sparse. This includes mechanisms to create additional labeled data like data augmentation and distant supervision as well as transfer learning settings that reduce the need for target supervision. A goal of our survey is to explain how these methods differ in their requirements as understanding them is essential for choosing a technique suited for a specific low-resource setting. Further key aspects of this work are to highlight open issues and to outline promising directions for future research.
Self-Learning Transformations for Improving Gaze and Head Redirection
Authors: Yufeng Zheng, Seonwook Park, Xucong Zhang, Shalini De Mello, Otmar Hilliges
Date: 2020-10-23 00:00:00
Proceeding: NeurIPS 2020 12
Tasks: Disentanglement, Gaze Estimation, Self-Learning

Many computer vision tasks rely on labeled data. Rapid progress in generative modeling has led to the ability to synthesize photorealistic images. However, controlling specific aspects of the generation process such that the data can be used for supervision of downstream tasks remains challenging. In this paper we propose a novel generative model for images of faces, that is capable of producing high-quality images under fine-grained control over eye gaze and head orientation angles. This requires the disentangling of many appearance related factors including gaze and head orientation but also lighting, hue etc. We propose a novel architecture which learns to discover, disentangle and encode these extraneous variations in a self-learned manner. We further show that explicitly disentangling task-irrelevant factors results in more accurate modelling of gaze and head orientation. A novel evaluation scheme shows that our method improves upon the state-of-the-art in redirection accuracy and disentanglement between gaze direction and head orientation changes. Furthermore, we show that in the presence of limited amounts of real-world training data, our method allows for improvements in the downstream task of semi-supervised cross-dataset gaze estimation. Please check our project page at: https://ait.ethz.ch/projects/2020/STED-gaze/
Network Classifiers Based on Social Learning
Authors: Virginia Bordignon, Stefan Vlaski, Vincenzo Matta, Ali H. Sayed
Date: 2020-10-23 00:00:00

This work proposes a new way of combining independently trained classifiers over space and time. Combination over space means that the outputs of spatially distributed classifiers are aggregated. Combination over time means that the classifiers respond to streaming data during testing and continue to improve their performance even during this phase. By doing so, the proposed architecture is able to improve prediction performance over time with unlabeled data. Inspired by social learning algorithms, which require prior knowledge of the observations distribution, we propose a Social Machine Learning (SML) paradigm that is able to exploit the imperfect models generated during the learning phase. We show that this strategy results in consistent learning with high probability, and it yields a robust structure against poorly trained classifiers. Simulations with an ensemble of feedforward neural networks are provided to illustrate the theoretical results.
FAME: Feature-Based Adversarial Meta-Embeddings for Robust Input Representations
Authors: Lukas Lange, Heike Adel, Jannik Strötgen, Dietrich Klakow
Date: 2020-10-23 00:00:00
Proceeding: EMNLP 2021 11
Tasks: NER, POS, POS Tagging, Sentence, Sentence Classification, Word Embeddings

Combining several embeddings typically improves performance in downstream tasks as different embeddings encode different information. It has been shown that even models using embeddings from transformers still benefit from the inclusion of standard word embeddings. However, the combination of embeddings of different types and dimensions is challenging. As an alternative to attention-based meta-embeddings, we propose feature-based adversarial meta-embeddings (FAME) with an attention function that is guided by features reflecting word-specific properties, such as shape and frequency, and show that this is beneficial to handle subword-based embeddings. In addition, FAME uses adversarial training to optimize the mappings of differently-sized embeddings to the same space. We demonstrate that FAME works effectively across languages and domains for sequence labeling and sentence classification, in particular in low-resource settings. FAME sets the new state of the art for POS tagging in 27 languages, various NER settings and question classification in different domains.
Learning Multi-layer Graphs and a Common Representation for Clustering
Authors: Sravanthi Gurugubelli, Sundeep Prabhakar Chepuri
Date: 2020-10-23 00:00:00
Tasks: Clustering, Graph Learning

In this paper, we focus on graph learning from multi-view data of shared entities for spectral clustering. We can explain interactions between the entities in multi-view data using a multi-layer graph with a common vertex set, which represents the shared entities. The edges of different layers capture the relationships of the entities. Assuming a smoothness data model, we jointly estimate the graph Laplacian matrices of the individual graph layers and low-dimensional embedding of the common vertex set. We constrain the rank of the graph Laplacian matrices to obtain multi-component graph layers for clustering. The low-dimensional node embeddings, common to all the views, assimilate the complementary information present in the views. We propose an efficient solver based on alternating minimization to solve the proposed multi-layer multi-component graph learning problem. Numerical experiments on synthetic and real datasets demonstrate that the proposed algorithm outperforms state-of-the-art multi-view clustering techniques.
Topic Space Trajectories: A case study on machine learning literature
Authors: Bastian Schäfermeier, Gerd Stumme, Tom Hanika
Date: 2020-10-23 00:00:00
Tasks: BIG-bench Machine Learning

The annual number of publications at scientific venues, for example, conferences and journals, is growing quickly. Hence, even for researchers it becomes harder and harder to keep track of research topics and their progress. In this task, researchers can be supported by automated publication analysis. Yet, many such methods result in uninterpretable, purely numerical representations. As an attempt to support human analysts, we present topic space trajectories, a structure that allows for the comprehensible tracking of research topics. We demonstrate how these trajectories can be interpreted based on eight different analysis approaches. To obtain comprehensible results, we employ non-negative matrix factorization as well as suitable visualization techniques. We show the applicability of our approach on a publication corpus spanning 50 years of machine learning research from 32 publication venues. Our novel analysis method may be employed for paper classification, for the prediction of future research topics, and for the recommendation of fitting conferences and journals for submitting unpublished work.
Linearly Converging Error Compensated SGD
Authors: Eduard Gorbunov, Dmitry Kovalev, Dmitry Makarenko, Peter Richtárik
Date: 2020-10-23 00:00:00
Proceeding: NeurIPS 2020 12
Tasks: Quantization

In this paper, we propose a unified analysis of variants of distributed SGD with arbitrary compressions and delayed updates. Our framework is general enough to cover different variants of quantized SGD, Error-Compensated SGD (EC-SGD) and SGD with delayed updates (D-SGD). Via a single theorem, we derive the complexity results for all the methods that fit our framework. For the existing methods, this theorem gives the best-known complexity results. Moreover, using our general scheme, we develop new variants of SGD that combine variance reduction or arbitrary sampling with error feedback and quantization and derive the convergence rates for these methods beating the state-of-the-art results. In order to illustrate the strength of our framework, we develop 16 new methods that fit this. In particular, we propose the first method called EC-SGD-DIANA that is based on error-feedback for biased compression operator and quantization of gradient differences and prove the convergence guarantees showing that EC-SGD-DIANA converges to the exact optimum asymptotically in expectation with constant learning rate for both convex and strongly convex objectives when workers compute full gradients of their loss functions. Moreover, for the case when the loss function of the worker has the form of finite sum, we modified the method and got a new one called EC-LSVRG-DIANA which is the first distributed stochastic method with error feedback and variance reduction that converges to the exact optimum asymptotically in expectation with a constant learning rate.
Exploring Common and Individual Characteristics of Students via Matrix Recovering
Authors: Zhen Wang, Ben Teng, Yun Zhou, Hanshuang Tong, Guangtong Liu
Date: 2020-10-23 00:00:00

Balancing group teaching and individual mentoring is an important issue in education area. The nature behind this issue is to explore common characteristics shared by multiple students and individual characteristics for each student. Biclustering methods have been proved successful for detecting meaningful patterns with the goal of driving group instructions based on students' characteristics. However, these methods ignore the individual characteristics of students as they only focus on common characteristics of students. In this article, we propose a framework to detect both group characteristics and individual characteristics of students simultaneously. We assume that the characteristics matrix of students' is composed of two parts: one is a low-rank matrix representing the common characteristics of students; the other is a sparse matrix representing individual characteristics of students. Thus, we treat the balancing issue as a matrix recovering problem. The experiment results show the effectiveness of our method. Firstly, it can detect meaningful biclusters that are comparable with the state-of-the-art biclutering algorithms. Secondly, it can identify individual characteristics for each student simultaneously. Both the source code of our algorithm and the real datasets are available upon request.
Graph-Homomorphic Perturbations for Private Decentralized Learning
Authors: Stefan Vlaski, Ali H. Sayed
Date: 2020-10-23 00:00:00
Tasks: Privacy Preserving, Stochastic Optimization

Decentralized algorithms for stochastic optimization and learning rely on the diffusion of information as a result of repeated local exchanges of intermediate estimates. Such structures are particularly appealing in situations where agents may be hesitant to share raw data due to privacy concerns. Nevertheless, in the absence of additional privacy-preserving mechanisms, the exchange of local estimates, which are generated based on private data can allow for the inference of the data itself. The most common mechanism for guaranteeing privacy is the addition of perturbations to local estimates before broadcasting. These perturbations are generally chosen independently at every agent, resulting in a significant performance loss. We propose an alternative scheme, which constructs perturbations according to a particular nullspace condition, allowing them to be invisible (to first order in the step-size) to the network centroid, while preserving privacy guarantees. The analysis allows for general nonconvex loss functions, and is hence applicable to a large number of machine learning and signal processing problems, including deep learning.
Unbiased Estimation Equation under $f$-Separable Bregman Distortion Measures
Authors: Masahiro Kobayashi, Kazuho Watanabe
Date: 2020-10-23 00:00:00

We discuss unbiased estimation equations in a class of objective function using a monotonically increasing function $f$ and Bregman divergence. The choice of the function $f$ gives desirable properties such as robustness against outliers. In order to obtain unbiased estimation equations, analytically intractable integrals are generally required as bias correction terms. In this study, we clarify the combination of Bregman divergence, statistical model, and function $f$ in which the bias correction term vanishes. Focusing on Mahalanobis and Itakura-Saito distances, we provide a generalization of fundamental existing results and characterize a class of distributions of positive reals with a scale parameter, which includes the gamma distribution as a special case. We discuss the possibility of latent bias minimization when the proportion of outliers is large, which is induced by the extinction of the bias correction term.
Pre-training Graph Transformer with Multimodal Side Information for Recommendation
Authors: Yong liu, Susen Yang, Chenyi Lei, Guoxin Wang, Haihong Tang, Juyong Zhang, Aixin Sun, Chunyan Miao
Date: 2020-10-23 00:00:00
Tasks: Recommendation Systems, Unsupervised Pre-training

Side information of items, e.g., images and text description, has shown to be effective in contributing to accurate recommendations. Inspired by the recent success of pre-training models on natural language and images, we propose a pre-training strategy to learn item representations by considering both item side information and their relationships. We relate items by common user activities, e.g., co-purchase, and construct a homogeneous item graph. This graph provides a unified view of item relations and their associated side information in multimodality. We develop a novel sampling algorithm named MCNSampling to select contextual neighbors for each item. The proposed Pre-trained Multimodal Graph Transformer (PMGT) learns item representations with two objectives: 1) graph structure reconstruction, and 2) masked node feature reconstruction. Experimental results on real datasets demonstrate that the proposed PMGT model effectively exploits the multimodality side information to achieve better accuracies in downstream tasks including item recommendation, item classification, and click-through ratio prediction. We also report a case study of testing the proposed PMGT model in an online setting with 600 thousand users.
ST-BERT: Cross-modal Language Model Pre-training For End-to-end Spoken Language Understanding
Authors: Minjeong Kim, Gyuwan Kim, Sang-Woo Lee, Jung-Woo Ha
Date: 2020-10-23 00:00:00
Tasks: cross-modal alignment, Language Modeling, Language Modelling, Masked Language Modeling, Spoken Language Understanding

Language model pre-training has shown promising results in various downstream tasks. In this context, we introduce a cross-modal pre-trained language model, called Speech-Text BERT (ST-BERT), to tackle end-to-end spoken language understanding (E2E SLU) tasks. Taking phoneme posterior and subword-level text as an input, ST-BERT learns a contextualized cross-modal alignment via our two proposed pre-training tasks: Cross-modal Masked Language Modeling (CM-MLM) and Cross-modal Conditioned Language Modeling (CM-CLM). Experimental results on three benchmarks present that our approach is effective for various SLU datasets and shows a surprisingly marginal performance degradation even when 1% of the training data are available. Also, our method shows further SLU performance gain via domain-adaptive pre-training with domain-specific speech-text pair data.
Pre-training with Meta Learning for Chinese Word Segmentation
Authors: Zhen Ke, Liang Shi, Songtao Sun, Erli Meng, Bin Wang, Xipeng Qiu
Date: 2020-10-23 00:00:00
Proceeding: NAACL 2021 4
Tasks: Chinese Word Segmentation, Language Modeling, Language Modelling, Meta-Learning, Segmentation

Recent researches show that pre-trained models (PTMs) are beneficial to Chinese Word Segmentation (CWS). However, PTMs used in previous works usually adopt language modeling as pre-training tasks, lacking task-specific prior segmentation knowledge and ignoring the discrepancy between pre-training tasks and downstream CWS tasks. In this paper, we propose a CWS-specific pre-trained model METASEG, which employs a unified architecture and incorporates meta learning algorithm into a multi-criteria pre-training task. Empirical results show that METASEG could utilize common prior segmentation knowledge from different existing criteria and alleviate the discrepancy between pre-trained models and downstream CWS tasks. Besides, METASEG can achieve new state-of-the-art performance on twelve widely-used CWS datasets and significantly improve model performance in low-resource settings.
Reducing Bias in Modeling Real-world Password Strength via Deep Learning and Dynamic Dictionaries
Authors: Dario Pasquini, Marco Cianfriglia, Giuseppe Ateniese, Massimo Bernaschi
Date: 2020-10-23 00:00:00
Tasks: Security Studies

Password security hinges on an in-depth understanding of the techniques adopted by attackers. Unfortunately, real-world adversaries resort to pragmatic guessing strategies such as dictionary attacks that are inherently difficult to model in password security studies. In order to be representative of the actual threat, dictionary attacks must be thoughtfully configured and tuned. However, this process requires a domain-knowledge and expertise that cannot be easily replicated. The consequence of inaccurately calibrating dictionary attacks is the unreliability of password security analyses, impaired by a severe measurement bias. In the present work, we introduce a new generation of dictionary attacks that is consistently more resilient to inadequate configurations. Requiring no supervision or domain-knowledge, this technique automatically approximates the advanced guessing strategies adopted by real-world attackers. To achieve this: (1) We use deep neural networks to model the proficiency of adversaries in building attack configurations. (2) Then, we introduce dynamic guessing strategies within dictionary attacks. These mimic experts' ability to adapt their guessing strategies on the fly by incorporating knowledge on their targets. Our techniques enable more robust and sound password strength estimates within dictionary attacks, eventually reducing overestimation in modeling real-world threats in password security. Code available: https://github.com/TheAdamProject/adams