pwc-archive/papers-with-abstracts huggingface dataset:
MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformers
Authors: Wenhui Wang, Hangbo Bao, Shaohan Huang, Li Dong, Furu Wei
Date: 2020-12-31 00:00:00
Proceeding: Findings (ACL) 2021 8
Tasks: Relation, XLM-R

We generalize deep self-attention distillation in MiniLM (Wang et al., 2020) by only using self-attention relation distillation for task-agnostic compression of pretrained Transformers. In particular, we define multi-head self-attention relations as scaled dot-product between the pairs of query, key, and value vectors within each self-attention module. Then we employ the above relational knowledge to train the student model. Besides its simplicity and unified principle, more favorably, there is no restriction in terms of the number of student's attention heads, while most previous work has to guarantee the same head number between teacher and student. Moreover, the fine-grained self-attention relations tend to fully exploit the interaction knowledge learned by Transformer. In addition, we thoroughly examine the layer selection strategy for teacher models, rather than just relying on the last layer as in MiniLM. We conduct extensive experiments on compressing both monolingual and multilingual pretrained models. Experimental results demonstrate that our models distilled from base-size and large-size teachers (BERT, RoBERTa and XLM-R) outperform the state-of-the-art.
A CNN Approach to Simultaneously Count Plants and Detect Plantation-Rows from UAV Imagery
Authors: Lucas Prado Osco, Mauro dos Santos de Arruda, Diogo Nunes Gonçalves, Alexandre Dias, Juliana Batistoti, Mauricio de Souza, Felipe David Georges Gomes, Ana Paula Marques Ramos, Lúcio André de Castro Jorge, Veraldo Liesenberg, Jonathan Li, Lingfei Ma, José Marcato Junior, Wesley Nunes Gonçalves
Date: 2020-12-31 00:00:00

In this paper, we propose a novel deep learning method based on a Convolutional Neural Network (CNN) that simultaneously detects and geolocates plantation-rows while counting its plants considering highly-dense plantation configurations. The experimental setup was evaluated in a cornfield with different growth stages and in a Citrus orchard. Both datasets characterize different plant density scenarios, locations, types of crops, sensors, and dates. A two-branch architecture was implemented in our CNN method, where the information obtained within the plantation-row is updated into the plant detection branch and retro-feed to the row branch; which are then refined by a Multi-Stage Refinement method. In the corn plantation datasets (with both growth phases, young and mature), our approach returned a mean absolute error (MAE) of 6.224 plants per image patch, a mean relative error (MRE) of 0.1038, precision and recall values of 0.856, and 0.905, respectively, and an F-measure equal to 0.876. These results were superior to the results from other deep networks (HRNet, Faster R-CNN, and RetinaNet) evaluated with the same task and dataset. For the plantation-row detection, our approach returned precision, recall, and F-measure scores of 0.913, 0.941, and 0.925, respectively. To test the robustness of our model with a different type of agriculture, we performed the same task in the citrus orchard dataset. It returned an MAE equal to 1.409 citrus-trees per patch, MRE of 0.0615, precision of 0.922, recall of 0.911, and F-measure of 0.965. For citrus plantation-row detection, our approach resulted in precision, recall, and F-measure scores equal to 0.965, 0.970, and 0.964, respectively. The proposed method achieved state-of-the-art performance for counting and geolocating plants and plant-rows in UAV images from different types of crops.
Educational Content Linking for Enhancing Learning Need Remediation in MOOCs
Authors: Shang-Wen Li
Date: 2020-12-31 00:00:00

Since its introduction in 2011, there have been over 4000 MOOCs on various subjects on the Web, serving over 35 million learners. MOOCs have shown the ability to democratize knowledge dissemination and bring the best education in the world to every learner. However, the disparate distances between participants, the size of the learner population, and the heterogeneity of the learners' backgrounds make it extremely difficult for instructors to interact with the learners in a timely manner, which adversely affects learning experience. To address the challenges, in this thesis, we propose a framework: educational content linking. By linking and organizing pieces of learning content scattered in various course materials into an easily accessible structure, we hypothesize that this framework can provide learners guidance and improve content navigation. Since most instruction and knowledge acquisition in MOOCs takes place when learners are surveying course materials, better content navigation may help learners find supporting information to resolve their confusion and thus improve learning outcome and experience. To support our conjecture, we present end-to-end studies to investigate our framework around two research questions: 1) can manually generated linking improve learning? 2) can learning content be generated with machine learning methods? For studying the first question, we built an interface that present learning materials and visualize the linking among them simultaneously. We found the interface enables users to search for desired course materials more efficiently, and retain more concepts more readily. For the second question, we propose an automatic content linking algorithm based on conditional random fields. We demonstrate that automatically generated linking can still lead to better learning, although the magnitude of the improvement over the unlinked interface is smaller.
Binary Graph Neural Networks
Authors: Mehdi Bahri, Gaétan Bahl, Stefanos Zafeiriou
Date: 2020-12-31 00:00:00
Proceeding: CVPR 2021 1
Tasks: Binarization, Graph Neural Network, Representation Learning

Graph Neural Networks (GNNs) have emerged as a powerful and flexible framework for representation learning on irregular data. As they generalize the operations of classical CNNs on grids to arbitrary topologies, GNNs also bring much of the implementation challenges of their Euclidean counterparts. Model size, memory footprint, and energy consumption are common concerns for many real-world applications. Network binarization allocates a single bit to parameters and activations, thus dramatically reducing the memory requirements (up to 32x compared to single-precision floating-point numbers) and maximizing the benefits of fast SIMD instructions on modern hardware for measurable speedups. However, in spite of the large body of work on binarization for classical CNNs, this area remains largely unexplored in geometric deep learning. In this paper, we present and evaluate different strategies for the binarization of graph neural networks. We show that through careful design of the models, and control of the training process, binary graph neural networks can be trained at only a moderate cost in accuracy on challenging benchmarks. In particular, we present the first dynamic graph neural network in Hamming space, able to leverage efficient k-NN search on binary vectors to speed-up the construction of the dynamic graph. We further verify that the binary models offer significant savings on embedded devices. Our code is publicly available on Github.
Neural Network Approximations for Calabi-Yau Metrics
Authors: Vishnu Jejjala, Damian Kaloni Mayorga Pena, Challenger Mishra
Date: 2020-12-31 00:00:00

Ricci flat metrics for Calabi-Yau threefolds are not known analytically. In this work, we employ techniques from machine learning to deduce numerical flat metrics for the Fermat quintic, for the Dwork quintic, and for the Tian-Yau manifold. This investigation employs a single neural network architecture that is capable of approximating Ricci flat Kaehler metrics for several Calabi-Yau manifolds of dimensions two and three. We show that measures that assess the Ricci flatness of the geometry decrease after training by three orders of magnitude. This is corroborated on the validation set, where the improvement is more modest. Finally, we demonstrate that discrete symmetries of manifolds can be learned in the process of learning the metric.
Transcription-dependent spatial organization of a gene locus
Authors: Lev Barinov, Sergey Ryabichko, William Bialek, Thomas Gregor
Date: 2020-12-31 00:00:00

There is growing appreciation that gene function is connected to the dynamic structure of the chromosome. Here we explore the interplay between three-dimensional structure and transcriptional activity at the single cell level. We show that inactive loci are spatially more compact than active ones, and that within active loci the enhancer driving transcription is closest to the promoter. On the other hand, even this shortest distance is too long to support direct physical contact between the enhancer-promoter pair when the locus is transcriptionally active. Artificial manipulation of genomic separations between enhancers and the promoter produces changes in physical distance and transcriptional activity, recapitulating the correlation seen in wild-type embryos, but disruption of topological domain boundaries has no effect. Our results suggest a complex interdependence between transcription and the spatial organization of cis-regulatory elements.
Trading bits in the readout from a genetic network
Authors: Marianne Bauer, Mariela D. Petkova, Thomas Gregor, Eric F. Wieschaus, William Bialek
Date: 2020-12-31 00:00:00
Tasks: Position

In genetic networks, information of relevance to the organism is represented by the concentrations of transcription factor molecules. In order to extract this information the cell must effectively "measure"' these concentrations, but there are physical limits to the precision of these measurements. We explore this trading between bits of precision in measuring concentration and bits of relevant information that can be extracted, using the gap gene network in the early fly embryo as an example. We argue that cells in the embryo can extract all the available information about their position, but only if the concentration measurements approach the physical limits to information capacity. These limits necessitate the observed proliferation of enhancer elements with sensitivities to combinations of transcription factors, but fine tuning of the parameters of these multiple enhancers is not required.
Fairness in Machine Learning
Authors: Luca Oneto, Silvia Chiappa
Date: 2020-12-31 00:00:00
Tasks: BIG-bench Machine Learning, Fairness

Machine learning based systems are reaching society at large and in many aspects of everyday life. This phenomenon has been accompanied by concerns about the ethical issues that may arise from the adoption of these technologies. ML fairness is a recently established area of machine learning that studies how to ensure that biases in the data and model inaccuracies do not lead to models that treat individuals unfavorably on the basis of characteristics such as e.g. race, gender, disabilities, and sexual or political orientation. In this manuscript, we discuss some of the limitations present in the current reasoning about fairness and in methods that deal with it, and describe some work done by the authors to address them. More specifically, we show how causal Bayesian networks can play an important role to reason about and deal with fairness, especially in complex unfairness scenarios. We describe how optimal transport theory can be used to develop methods that impose constraints on the full shapes of distributions corresponding to different sensitive attributes, overcoming the limitation of most approaches that approximate fairness desiderata by imposing constraints on the lower order moments or other functions of those distributions. We present a unified framework that encompasses methods that can deal with different settings and fairness criteria, and that enjoys strong theoretical guarantees. We introduce an approach to learn fair representations that can generalize to unseen tasks. Finally, we describe a technique that accounts for legal restrictions about the use of sensitive attributes.
Universal Adaptive Control of Nonlinear Systems
Authors: Brett T. Lopez, Jean-Jacques E. Slotine
Date: 2020-12-31 00:00:00
Tasks: Motion Planning, parameter estimation, Transfer Learning

This work develops a new direct adaptive control framework that extends the certainty equivalence principle to general nonlinear systems with unmatched model uncertainties. The approach adjusts the rate of adaptation online to eliminate the effects of parameter estimation transients on closed-loop stability. The method can be immediately combined with a previously designed or learned feedback policy if a corresponding model-parameterized Lyapunov function or contraction metric is known. Simulation results of various nonlinear systems with unmatched uncertainties demonstrates the approach.
Language-Mediated, Object-Centric Representation Learning
Authors: Ruocheng Wang, Jiayuan Mao, Samuel J. Gershman, Jiajun Wu
Date: 2020-12-31 00:00:00
Proceeding: Findings (ACL) 2021 8
Tasks: Object, Object Discovery, Referring Expression, Referring Expression Comprehension, Representation Learning, Semantic Segmentation, Unsupervised Object Segmentation

We present Language-mediated, Object-centric Representation Learning (LORL), a paradigm for learning disentangled, object-centric scene representations from vision and language. LORL builds upon recent advances in unsupervised object discovery and segmentation, notably MONet and Slot Attention. While these algorithms learn an object-centric representation just by reconstructing the input image, LORL enables them to further learn to associate the learned representations to concepts, i.e., words for object categories, properties, and spatial relationships, from language input. These object-centric concepts derived from language facilitate the learning of object-centric representations. LORL can be integrated with various unsupervised object discovery algorithms that are language-agnostic. Experiments show that the integration of LORL consistently improves the performance of unsupervised object discovery methods on two datasets via the help of language. We also show that concepts learned by LORL, in conjunction with object discovery methods, aid downstream tasks such as referring expression comprehension.
UCCA's Foundational Layer: Annotation Guidelines v2.1
Authors: Omri Abend, Nathan Schneider, Dotan Dvir, Jakob Prange, Ari Rappoport
Date: 2020-12-31 00:00:00

This is the annotation manual for Universal Conceptual Cognitive Annotation (UCCA; Abend and Rappoport, 2013), specifically the Foundational Layer. UCCA is a graph-based semantic annotation scheme based on typological linguistic principles. It has been applied to several languages; for ease of exposition these guidelines give examples mainly in English. New annotators may wish to start with the tutorial on the UCCA framework (Abend et al., 2020). Further resources are available at the project homepage: https://universalconceptualcognitiveannotation.github.io
The Sample Complexity of Robust Covariance Testing
Authors: Ilias Diakonikolas, Daniel M. Kane
Date: 2020-12-31 00:00:00

We study the problem of testing the covariance matrix of a high-dimensional Gaussian in a robust setting, where the input distribution has been corrupted in Huber's contamination model. Specifically, we are given i.i.d. samples from a distribution of the form $Z = (1-\epsilon) X + \epsilon B$, where $X$ is a zero-mean and unknown covariance Gaussian $\mathcal{N}(0, \Sigma)$, $B$ is a fixed but unknown noise distribution, and $\epsilon>0$ is an arbitrarily small constant representing the proportion of contamination. We want to distinguish between the cases that $\Sigma$ is the identity matrix versus $\gamma$-far from the identity in Frobenius norm. In the absence of contamination, prior work gave a simple tester for this hypothesis testing task that uses $O(d)$ samples. Moreover, this sample upper bound was shown to be best possible, within constant factors. Our main result is that the sample complexity of covariance testing dramatically increases in the contaminated setting. In particular, we prove a sample complexity lower bound of $\Omega(d^2)$ for $\epsilon$ an arbitrarily small constant and $\gamma = 1/2$. This lower bound is best possible, as $O(d^2)$ samples suffice to even robustly {\em learn} the covariance. The conceptual implication of our result is that, for the natural setting we consider, robust hypothesis testing is at least as hard as robust estimation.
Promoting Graph Awareness in Linearized Graph-to-Text Generation
Authors: Alexander Hoyle, Ana Marasović, Noah Smith
Date: 2020-12-31 00:00:00
Proceeding: Findings (ACL) 2021 8
Tasks: Denoising, Text Generation

Generating text from structured inputs, such as meaning representations or RDF triples, has often involved the use of specialized graph-encoding neural networks. However, recent applications of pretrained transformers to linearizations of graph inputs have yielded state-of-the-art generation results on graph-to-text tasks. Here, we explore the ability of these linearized models to encode local graph structures, in particular their invariance to the graph linearization strategy and their ability to reconstruct corrupted inputs. Our findings motivate solutions to enrich the quality of models' implicit graph encodings via scaffolding. Namely, we use graph-denoising objectives implemented in a multi-task text-to-text framework. We find that these denoising scaffolds lead to substantial improvements in downstream generation in low-resource settings.
Evidence-based Factual Error Correction
Authors: James Thorne, Andreas Vlachos
Date: 2020-12-31 00:00:00
Tasks: Fact Checking, Fact Verification, Misinformation

This paper introduces the task of factual error correction: performing edits to a claim so that the generated rewrite is better supported by evidence. This extends the well-studied task of fact verification by providing a mechanism to correct written texts that are refuted or only partially supported by evidence. We demonstrate that it is feasible to train factual error correction systems from existing fact checking datasets which only contain labeled claims accompanied by evidence, but not the correction. We achieve this by employing a two-stage distant supervision approach that incorporates evidence into masked claims when generating corrections. Our approach, based on the T5 transformer and using retrieved evidence, achieved better results than existing work which used a pointer copy network and gold evidence, producing accurate factual error corrections for 5x more instances in human evaluation and a .125 increase in SARI score. The evaluation is conducted on a dataset of 65,000 instances based on a recent fact verification shared task and we release it to enable further work on the task.
Conditional Generation of Temporally-ordered Event Sequences
Authors: Shih-ting Lin, Nathanael Chambers, Greg Durrett
Date: 2020-12-31 00:00:00
Proceeding: ACL 2021 5
Tasks: Denoising, Story Completion

Models of narrative schema knowledge have proven useful for a range of event-related tasks, but they typically do not capture the temporal relationships between events. We propose a single model that addresses both temporal ordering, sorting given events into the order they occurred, and event infilling, predicting new events which fit into an existing temporally-ordered sequence. We use a BART-based conditional generation model that can capture both temporality and common event co-occurrence, meaning it can be flexibly applied to different tasks in this space. Our model is trained as a denoising autoencoder: we take temporally-ordered event sequences, shuffle them, delete some events, and then attempt to recover the original event sequence. This task teaches the model to make inferences given incomplete knowledge about the events in an underlying scenario. On the temporal ordering task, we show that our model is able to unscramble event sequences from existing datasets without access to explicitly labeled temporal training data, outperforming both a BERT-based pairwise model and a BERT-based pointer network. On event infilling, human evaluation shows that our model is able to generate events that fit better temporally into the input events when compared to GPT-2 story completion models.
Understanding Politics via Contextualized Discourse Processing
Authors: Rajkumar Pujari, Dan Goldwasser
Date: 2020-12-31 00:00:00
Proceeding: EMNLP 2021 11
Tasks: Articles

Politicians often have underlying agendas when reacting to events. Arguments in contexts of various events reflect a fairly consistent set of agendas for a given entity. In spite of recent advances in Pretrained Language Models (PLMs), those text representations are not designed to capture such nuanced patterns. In this paper, we propose a Compositional Reader model consisting of encoder and composer modules, that attempts to capture and leverage such information to generate more effective representations for entities, issues, and events. These representations are contextualized by tweets, press releases, issues, news articles, and participating entities. Our model can process several documents at once and generate composed representations for multiple entities over several issues or events. Via qualitative and quantitative empirical analysis, we show that these representations are meaningful and effective.
iGOS++: Integrated Gradient Optimized Saliency by Bilateral Perturbations
Authors: Saeed Khorram, Tyler Lawson, Fuxin Li
Date: 2020-12-31 00:00:00

The black-box nature of the deep networks makes the explanation for "why" they make certain predictions extremely challenging. Saliency maps are one of the most widely-used local explanation tools to alleviate this problem. One of the primary approaches for generating saliency maps is by optimizing a mask over the input dimensions so that the output of the network is influenced the most by the masking. However, prior work only studies such influence by removing evidence from the input. In this paper, we present iGOS++, a framework to generate saliency maps that are optimized for altering the output of the black-box system by either removing or preserving only a small fraction of the input. Additionally, we propose to add a bilateral total variation term to the optimization that improves the continuity of the saliency map especially under high resolution and with thin object parts. The evaluation results from comparing iGOS++ against state-of-the-art saliency map methods show significant improvement in locating salient regions that are directly interpretable by humans. We utilized iGOS++ in the task of classifying COVID-19 cases from x-ray images and discovered that sometimes the CNN network is overfitted to the characters printed on the x-ray images when performing classification. Fixing this issue by data cleansing significantly improved the precision and recall of the classifier.