pwc-archive/papers-with-abstracts huggingface dataset:
Vision-Language Models as Success Detectors
Authors: Yuqing Du, Ksenia Konyushkova, Misha Denil, Akhil Raju, Jessica Landon, Felix Hill, Nando de Freitas, Serkan Cabi
Date: 2023-03-13 00:00:00
Tasks: Question Answering, Visual Question Answering, Visual Question Answering (VQA)

Detecting successful behaviour is crucial for training intelligent agents. As such, generalisable reward models are a prerequisite for agents that can learn to generalise their behaviour. In this work we focus on developing robust success detectors that leverage large, pretrained vision-language models (Flamingo, Alayrac et al. (2022)) and human reward annotations. Concretely, we treat success detection as a visual question answering (VQA) problem, denoted SuccessVQA. We study success detection across three vastly different domains: (i) interactive language-conditioned agents in a simulated household, (ii) real world robotic manipulation, and (iii) "in-the-wild" human egocentric videos. We investigate the generalisation properties of a Flamingo-based success detection model across unseen language and visual changes in the first two domains, and find that the proposed method is able to outperform bespoke reward models in out-of-distribution test scenarios with either variation. In the last domain of "in-the-wild" human videos, we show that success detection on unseen real videos presents an even more challenging generalisation task warranting future work. We hope our initial results encourage further work in real world success detection and reward modelling.
Breaking Common Sense: WHOOPS! A Vision-and-Language Benchmark of Synthetic and Compositional Images
Authors: Nitzan Bitton-Guetta, Yonatan Bitton, Jack Hessel, Ludwig Schmidt, Yuval Elovici, Gabriel Stanovsky, Roy Schwartz
Date: 2023-03-13 00:00:00
Proceeding: ICCV 2023 1
Tasks: Common Sense Reasoning, Explanation Generation, Image Captioning, Image Generation, Image-to-Text Retrieval, Question Answering, Visual Commonsense Reasoning, Visual Question Answering, Visual Question Answering (VQA)

Weird, unusual, and uncanny images pique the curiosity of observers because they challenge commonsense. For example, an image released during the 2022 world cup depicts the famous soccer stars Lionel Messi and Cristiano Ronaldo playing chess, which playfully violates our expectation that their competition should occur on the football field. Humans can easily recognize and interpret these unconventional images, but can AI models do the same? We introduce WHOOPS!, a new dataset and benchmark for visual commonsense. The dataset is comprised of purposefully commonsense-defying images created by designers using publicly-available image generation tools like Midjourney. We consider several tasks posed over the dataset. In addition to image captioning, cross-modal matching, and visual question answering, we introduce a difficult explanation generation task, where models must identify and explain why a given image is unusual. Our results show that state-of-the-art models such as GPT3 and BLIP2 still lag behind human performance on WHOOPS!. We hope our dataset will inspire the development of AI models with stronger visual commonsense reasoning abilities. Data, models and code are available at the project website: whoops-benchmark.github.io
A Survey of Graph Prompting Methods: Techniques, Applications, and Challenges
Authors: Xuansheng Wu, Kaixiong Zhou, Mingchen Sun, Xin Wang, Ninghao Liu
Date: 2023-03-13 00:00:00
Tasks: Prompt Learning, Survey

The recent "pre-train, prompt, predict training" paradigm has gained popularity as a way to learn generalizable models with limited labeled data. The approach involves using a pre-trained model and a prompting function that applies a template to input samples, adding indicative context and reformulating target tasks as the pre-training task. However, the design of prompts could be a challenging and time-consuming process in complex tasks. The limitation can be addressed by using graph data, as graphs serve as structured knowledge repositories by explicitly modeling the interaction between entities. In this survey, we review prompting methods from the graph perspective, where prompting functions are augmented with graph knowledge. In particular, we introduce the basic concepts of graph prompt learning, organize the existing work of designing graph prompting functions, and describe their applications and future challenges. This survey will bridge the gap between graphs and prompt design to facilitate future methodology development.
Can Workers Meaningfully Consent to Workplace Wellbeing Technologies?
Authors: Shreya Chowdhary, Anna Kawakami, Mary L. Gray, Jina Suh, Alexandra Olteanu, Koustuv Saha
Date: 2023-03-13 00:00:00

Sensing technologies deployed in the workplace can unobtrusively collect detailed data about individual activities and group interactions that are otherwise difficult to capture. A hopeful application of these technologies is that they can help businesses and workers optimize productivity and wellbeing. However, given the workplace's inherent and structural power dynamics, the prevalent approach of accepting tacit compliance to monitor work activities rather than seeking workers' meaningful consent raises privacy and ethical concerns. This paper unpacks the challenges workers face when consenting to workplace wellbeing technologies. Using a hypothetical case to prompt reflection among six multi-stakeholder focus groups involving 15 participants, we explored participants' expectations and capacity to consent to these technologies. We sketched possible interventions that could better support meaningful consent to workplace wellbeing technologies by drawing on critical computing and feminist scholarship -- which reframes consent from a purely individual choice to a structural condition experienced at the individual level that needs to be freely given, reversible, informed, enthusiastic, and specific (FRIES). The focus groups revealed how workers are vulnerable to "meaningless" consent -- as they may be subject to power dynamics that minimize their ability to withhold consent and may thus experience an erosion of autonomy, also undermining the value of data gathered in the name of "wellbeing." To meaningfully consent, participants wanted changes to the technology and to the policies and practices surrounding the technology. Our mapping of what prevents workers from meaningfully consenting to workplace wellbeing technologies (challenges) and what they require to do so (interventions) illustrates how the lack of meaningful consent is a structural problem requiring socio-technical solutions.
Intersection Warning System for Occlusion Risks using Relational Local Dynamic Maps
Authors: Florian Damerow, Yuda Li, Tim Puphal, Benedict Flade, Julian Eggert
Date: 2023-03-13 00:00:00
Tasks: Survival Analysis

This work addresses the task of risk evaluation in traffic scenarios with limited observability due to restricted sensorial coverage. Here, we concentrate on intersection scenarios that are difficult to access visually. To identify the area of sight, we employ ray casting on a local dynamic map providing geometrical information and road infrastructure. Based on the area with reduced visibility, we first model scene entities that pose a potential risk without being visually perceivable yet. Then, we predict a worst-case trajectory in the survival analysis for collision risk estimation. Resulting risk indicators are utilized to evaluate the driver's current behavior, to warn the driver in critical situations, to give suggestions on how to act safely or to plan safe trajectories. We validate our approach by applying the resulting intersection warning system on real world scenarios. The proposed system's behavior reveals to mimic the general behavior of a correctly acting human driver.
Parallel Vertex Diffusion for Unified Visual Grounding
Authors: Zesen Cheng, Kehan Li, Peng Jin, Xiangyang Ji, Li Yuan, Chang Liu, Jie Chen
Date: 2023-03-13 00:00:00
Tasks: Visual Grounding

Unified visual grounding pursues a simple and generic technical route to leverage multi-task data with less task-specific design. The most advanced methods typically present boxes and masks as vertex sequences to model referring detection and segmentation as an autoregressive sequential vertex generation paradigm. However, generating high-dimensional vertex sequences sequentially is error-prone because the upstream of the sequence remains static and cannot be refined based on downstream vertex information, even if there is a significant location gap. Besides, with limited vertexes, the inferior fitting of objects with complex contours restricts the performance upper bound. To deal with this dilemma, we propose a parallel vertex generation paradigm for superior high-dimension scalability with a diffusion model by simply modifying the noise dimension. An intuitive materialization of our paradigm is Parallel Vertex Diffusion (PVD) to directly set vertex coordinates as the generation target and use a diffusion model to train and infer. We claim that it has two flaws: (1) unnormalized coordinate caused a high variance of loss value; (2) the original training objective of PVD only considers point consistency but ignores geometry consistency. To solve the first flaw, Center Anchor Mechanism (CAM) is designed to convert coordinates as normalized offset values to stabilize the training loss value. For the second flaw, Angle summation loss (ASL) is designed to constrain the geometry difference of prediction and ground truth vertexes for geometry-level consistency. Empirical results show that our PVD achieves state-of-the-art in both referring detection and segmentation, and our paradigm is more scalable and efficient than sequential vertex generation with high-dimension data.
Optimization of the location and design of urban green spaces
Authors: Caroline Leboeuf, Margarida Carvalho, Yan Kestens, Benoît Thierry
Date: 2023-03-13 00:00:00
Tasks: Decision Making

The recent promotion of sustainable urban planning combined with a growing need for public interventions to improve well-being and health have led to an increased collective interest for green spaces in and around cities. In particular, parks have proven a wide range of benefits in urban areas. This also means inequities in park accessibility may contribute to health inequities. In this work, we showcase the application of classic tools from Operations Research to assist decision-makers to improve parks' accessibility, distribution and design. Given the context of public decision-making, we are particularly concerned with equity and environmental justice, and are focused on an advanced assessment of users' behavior through a spatial interaction model. We present a two-stage fair facility location and design model, which serves as a template model to assist public decision-makers at the city-level for the planning of urban green spaces. The first-stage of the optimization model is about the optimal city-budget allocation to neighborhoods based on a data exposing inequality attributes. The second-stage seeks the optimal location and design of parks for each neighborhood, and the objective consists of maximizing the total expected probability of individuals visiting parks. We show how to reformulate the latter as a mixed-integer linear program. We further introduce a clustering method to reduce the size of the problem and determine a close to optimal solution within reasonable time. The model is tested using the case study of the city of Montreal and comparative results are discussed in detail to justify the performance of the model.
Joint Behavior and Common Belief
Authors: Meir Friedenberg, Joseph Y. Halpern
Date: 2023-03-13 00:00:00

For over 25 years, common belief has been widely viewed as necessary for joint behavior. But this is not quite correct. We show by example that what can naturally be thought of as joint behavior can occur without common belief. We then present two variants of common belief that can lead to joint behavior, even without standard common belief ever being achieved, and show that one of them, action-stamped common belief, is in a sense necessary and sufficient for joint behavior. These observations are significant because, as is well known, common belief is quite difficult to achieve in practice, whereas these variants are more easily achievable.
Traffic Prediction with Transfer Learning: A Mutual Information-based Approach
Authors: Yunjie Huang, Xiaozhuang Song, Yuanshao Zhu, Shiyao Zhang, James J. Q. Yu
Date: 2023-03-13 00:00:00
Tasks: Graph Reconstruction, Management, Traffic Prediction, Transfer Learning

In modern traffic management, one of the most essential yet challenging tasks is accurately and timely predicting traffic. It has been well investigated and examined that deep learning-based Spatio-temporal models have an edge when exploiting Spatio-temporal relationships in traffic data. Typically, data-driven models require vast volumes of data, but gathering data in small cities can be difficult owing to constraints such as equipment deployment and maintenance costs. To resolve this problem, we propose TrafficTL, a cross-city traffic prediction approach that uses big data from other cities to aid data-scarce cities in traffic prediction. Utilizing a periodicity-based transfer paradigm, it identifies data similarity and reduces negative transfer caused by the disparity between two data distributions from distant cities. In addition, the suggested method employs graph reconstruction techniques to rectify defects in data from small data cities. TrafficTL is evaluated by comprehensive case studies on three real-world datasets and outperforms the state-of-the-art baseline by around 8 to 25 percent.
Probabilistic Uncertainty-Aware Risk Spot Detector for Naturalistic Driving
Authors: Tim Puphal, Malte Probst, Julian Eggert
Date: 2023-03-13 00:00:00
Tasks: Autonomous Vehicles, Survival Analysis

Risk assessment is a central element for the development and validation of Autonomous Vehicles (AV). It comprises a combination of occurrence probability and severity of future critical events. Time Headway (TH) as well as Time-To-Contact (TTC) are commonly used risk metrics and have qualitative relations to occurrence probability. However, they lack theoretical derivations and additionally they are designed to only cover special types of traffic scenarios (e.g. following between single car pairs). In this paper, we present a probabilistic situation risk model based on survival analysis considerations and extend it to naturally incorporate sensory, temporal and behavioral uncertainties as they arise in real-world scenarios. The resulting Risk Spot Detector (RSD) is applied and tested on naturalistic driving data of a multi-lane boulevard with several intersections, enabling the visualization of road criticality maps. Compared to TH and TTC, our approach is more selective and specific in predicting risk. RSD concentrates on driving sections of high vehicle density where large accelerations and decelerations or approaches with high velocity occur.
Evaluating Visual Number Discrimination in Deep Neural Networks
Authors: Ivana Kajić, Aida Nematzadeh
Date: 2023-03-13 00:00:00

The ability to discriminate between large and small quantities is a core aspect of basic numerical competence in both humans and animals. In this work, we examine the extent to which the state-of-the-art neural networks designed for vision exhibit this basic ability. Motivated by studies in animal and infant numerical cognition, we use the numerical bisection procedure to test number discrimination in different families of neural architectures. Our results suggest that vision-specific inductive biases are helpful in numerosity discrimination, as models with such biases have lowest test errors on the task, and often have psychometric curves that qualitatively resemble those of humans and animals performing the task. However, even the strongest models, as measured on standard metrics of performance, fail to discriminate quantities in transfer experiments with differing training and testing conditions, indicating that such inductive biases might not be sufficient.
Computing equivariant matrices on homogeneous spaces for Geometric Deep Learning and Automorphic Lie Algebras
Authors: Vincent Knibbeler
Date: 2023-03-13 00:00:00

We develop an elementary method to compute spaces of equivariant maps from a homogeneous space $G/H$ of a Lie group $G$ to a module of this group. The Lie group is not required to be compact. More generally, we study spaces of invariant sections in homogeneous vector bundles, and take a special interest in the case where the fibres are algebras. These latter cases have a natural global algebra structure. We classify these automorphic algebras for the case where the homogeneous space has compact stabilisers. This work has applications in the theoretical development of geometric deep learning and also in the theory of automorphic Lie algebras.
NeuroQL: A Neuro-Symbolic Language and Dataset for Inter-Subjective Reasoning
Authors: Nick Papoulias
Date: 2023-03-13 00:00:00
Tasks: Information Retrieval, Retrieval

We present a new AI task and baseline solution for Inter-Subjective Reasoning. We define inter-subjective information, to be a mixture of objective and subjective information possibly shared by different parties. Examples may include commodities and their objective properties as reported by IR (Information Retrieval) systems, that need to be cross-referenced with subjective user reviews from an online forum. For an AI system to successfully reason about both, it needs to be able to combine symbolic reasoning of objective facts with the shared consensus found on subjective user reviews. To this end we introduce the NeuroQL dataset and DSL (Domain-specific Language) as a baseline solution for this problem. NeuroQL is a neuro-symbolic language that extends logical unification with neural primitives for extraction and retrieval. It can function as a target for automatic translation of inter-subjective questions (posed in natural language) into the neuro-symbolic code that can answer them.
VMCDL: Vulnerability Mining Based on Cascaded Deep Learning Under Source Control Flow
Authors: Wen Zhou
Date: 2023-03-13 00:00:00
Tasks: Binary Classification

With the rapid development of the computer industry and computer software, the risk of software vulnerabilities being exploited has greatly increased. However, there are still many shortcomings in the existing mining techniques for leakage source research, such as high false alarm rate, coarse-grained detection, and dependence on expert experience. In this paper, we mainly use the c/c++ source code data of the SARD dataset, process the source code of CWE476, CWE469, CWE516 and CWE570 vulnerability types, test the Joern vulnerability scanning function of the cutting-edge tool, and propose a new cascading deep learning model VMCDL based on source code control flow to effectively detect vulnerabilities. First, this paper uses joern to locate and extract sensitive functions and statements to form a sensitive statement library of vulnerable code. Then, the CFG flow vulnerability code snippets are generated by bidirectional breadth-first traversal, and then vectorized by Doc2vec. Finally, the cascade deep learning model based on source code control flow is used for classification to obtain the classification results. In the experimental evaluation, we give the test results of Joern on specific vulnerabilities, and give the confusion matrix and label data of the binary classification results of the model algorithm on single vulnerability type source code, and compare and verify the five indicators of FPR, FNR, ACC, P and F1, respectively reaching 10.30%, 5.20%, 92.50%,85.10% and 85.40%,which shows that it can effectively reduce the false alarm rate of static analysis.
Modality-Agnostic Debiasing for Single Domain Generalization
Authors: Sanqing Qu, Yingwei Pan, Guang Chen, Ting Yao, Changjun Jiang, Tao Mei
Date: 2023-03-13 00:00:00
Proceeding: CVPR 2023 1
Tasks: Data Augmentation, Domain Generalization, Semantic Segmentation

Deep neural networks (DNNs) usually fail to generalize well to outside of distribution (OOD) data, especially in the extreme case of single domain generalization (single-DG) that transfers DNNs from single domain to multiple unseen domains. Existing single-DG techniques commonly devise various data-augmentation algorithms, and remould the multi-source domain generalization methodology to learn domain-generalized (semantic) features. Nevertheless, these methods are typically modality-specific, thereby being only applicable to one single modality (e.g., image). In contrast, we target a versatile Modality-Agnostic Debiasing (MAD) framework for single-DG, that enables generalization for different modalities. Technically, MAD introduces a novel two-branch classifier: a biased-branch encourages the classifier to identify the domain-specific (superficial) features, and a general-branch captures domain-generalized features based on the knowledge from biased-branch. Our MAD is appealing in view that it is pluggable to most single-DG models. We validate the superiority of our MAD in a variety of single-DG scenarios with different modalities, including recognition on 1D texts, 2D images, 3D point clouds, and semantic segmentation on 2D images. More remarkably, for recognition on 3D point clouds and semantic segmentation on 2D images, MAD improves DSU by 2.82\% and 1.5\% in accuracy and mIOU.
Upcycling Models under Domain and Category Shift
Authors: Sanqing Qu, Tianpei Zou, Florian Roehrbein, Cewu Lu, Guang Chen, DaCheng Tao, Changjun Jiang
Date: 2023-03-13 00:00:00
Proceeding: CVPR 2023 1
Tasks: Clustering, Domain Adaptation, Source-Free Domain Adaptation, Universal Domain Adaptation, Unsupervised Domain Adaptation

Deep neural networks (DNNs) often perform poorly in the presence of domain shift and category shift. How to upcycle DNNs and adapt them to the target task remains an important open problem. Unsupervised Domain Adaptation (UDA), especially recently proposed Source-free Domain Adaptation (SFDA), has become a promising technology to address this issue. Nevertheless, existing SFDA methods require that the source domain and target domain share the same label space, consequently being only applicable to the vanilla closed-set setting. In this paper, we take one step further and explore the Source-free Universal Domain Adaptation (SF-UniDA). The goal is to identify "known" data samples under both domain and category shift, and reject those "unknown" data samples (not present in source classes), with only the knowledge from standard pre-trained source model. To this end, we introduce an innovative global and local clustering learning technique (GLC). Specifically, we design a novel, adaptive one-vs-all global clustering algorithm to achieve the distinction across different target classes and introduce a local k-NN clustering strategy to alleviate negative transfer. We examine the superiority of our GLC on multiple benchmarks with different category shift scenarios, including partial-set, open-set, and open-partial-set DA. Remarkably, in the most challenging open-partial-set DA scenario, GLC outperforms UMAD by 14.8\% on the VisDA benchmark. The code is available at https://github.com/ispc-lab/GLC.