pwc-archive/papers-with-abstracts huggingface dataset:
Object as Query: Lifting any 2D Object Detector to 3D Detection
Authors: Zitian Wang, Zehao Huang, Jiahui Fu, Naiyan Wang, Si Liu
Date: 2023-01-06 00:00:00
Proceeding: ICCV 2023 1
Tasks: 3D Object Detection, Object, object-detection, Object Detection

3D object detection from multi-view images has drawn much attention over the past few years. Existing methods mainly establish 3D representations from multi-view images and adopt a dense detection head for object detection, or employ object queries distributed in 3D space to localize objects. In this paper, we design Multi-View 2D Objects guided 3D Object Detector (MV2D), which can lift any 2D object detector to multi-view 3D object detection. Since 2D detections can provide valuable priors for object existence, MV2D exploits 2D detectors to generate object queries conditioned on the rich image semantics. These dynamically generated queries help MV2D to recall objects in the field of view and show a strong capability of localizing 3D objects. For the generated queries, we design a sparse cross attention module to force them to focus on the features of specific objects, which suppresses interference from noises. The evaluation results on the nuScenes dataset demonstrate the dynamic object queries and sparse feature aggregation can promote 3D detection capability. MV2D also exhibits a state-of-the-art performance among existing methods. We hope MV2D can serve as a new baseline for future research. Code is available at \url{https://github.com/tusen-ai/MV2D}.
Text2Poster: Laying out Stylized Texts on Retrieved Images
Authors: Chuhao Jin, Hongteng Xu, Ruihua Song, Zhiwu Lu
Date: 2023-01-06 00:00:00
Tasks: Image Retrieval, Layout Design, multimodal generation

Poster generation is a significant task for a wide range of applications, which is often time-consuming and requires lots of manual editing and artistic experience. In this paper, we propose a novel data-driven framework, called \textit{Text2Poster}, to automatically generate visually-effective posters from textual information. Imitating the process of manual poster editing, our framework leverages a large-scale pretrained visual-textual model to retrieve background images from given texts, lays out the texts on the images iteratively by cascaded auto-encoders, and finally, stylizes the texts by a matching-based method. We learn the modules of the framework by weakly- and self-supervised learning strategies, mitigating the demand for labeled data. Both objective and subjective experiments demonstrate that our Text2Poster outperforms state-of-the-art methods, including academic research and commercial software, on the quality of generated posters.
A survey on Organoid Image Analysis Platforms
Authors: Alireza Ranjbaran, Azadeh Nazemi
Date: 2023-01-06 00:00:00
Tasks: Cultural Vocal Bursts Intensity Prediction, Drug Discovery, Survey

An in-vitro cell culture system is used for biological discoveries and hypothesis-driven research on a particular cell type to understand mechanistic or test pharmaceutical drugs. Conventional in-vitro cultures have been applied to primary cells and immortalised cell lines plated on 2D surfaces. However, they are unreliable in complex physiological environments and can not always predict in-vivo behaviour correctly. Organoids are multicellular spheroids of a primary donor or stem cells that are replaced in vitro cell culture systems and are widely used in biological, biomedical and translational studies. Native heterogeneity, microanatomy, and functionality of an organ or diseased tissue can be represented by three-dimensional in-vitro tissue models such as organoids. Organoids are essential in in-vitro models for drug discovery and personalised drug screening. Many imaging artefacts such as organoid occlusion, overlap, out-of-focus spheroids and considerable heterogeneity in size cause difficulty in conventional image processing. Despite the power of organoid models for biology, their size and shape have mostly not been considered. Drug responses depend on dynamic changes in individual organoid morphology, number and size, which means differences in organoid shape and size, movement through focal planes, and live-cell staining with limited options cause challenges for drug response and growth analysis. This study primarily introduces the importance of the role of the organoid culture system in different disciplines of medical science and various scopes of utilising organoids. Then studies the challenges of operating organoids, followed by reviewing image analysis systems or platforms applied to organoids to address organoid utilising challenges.
Convolutional XGBoost (C-XGBOOST) Model for Brain Tumor Detection
Authors: Muyiwa Babayomi, Oluwatosin Atinuke Olagbaju, Abdulrasheed Adedolapo Kadiri
Date: 2023-01-05 00:00:00
Tasks: image-classification, Image Classification, Medical Image Classification, Prognosis

Brain tumors are masses or abnormal growths of cells within the brain or the central spinal canal with symptoms such as headaches, seizures, weakness or numbness in the arms or legs, changes in personality or behaviour, nausea, vomiting, vision or hearing problems and dizziness. Conventional diagnosis of brain tumour involves some tests and procedure which may include the consideration of medical history, physical examination, imaging tests (such as CT or MRI scans), and biopsy (removal and examination of a small piece of the tumor tissue). These procedures, while effective, are mentally strenuous and time demanding due to the manual examination of the brain scans and the thorough evaluation of test results. It has been established in lots of medical research that brain tumours diagnosed and treated early generally tends to have a better prognosis. Deep learning techniques have evolved over the years and have demonstrated impressive and faster outcomes in the classification of brain tumours in medical imaging, with very little to no human interference. This study proposes a model for the early detection of brain tumours using a combination of convolutional neural networks (CNNs) and extreme gradient boosting (XGBoost). The proposed model, named C-XGBoost has a lower model complexity compared to purely CNNs, making it easier to train and less prone to overfitting. It is also better able to handle imbalanced and unstructured data, which are common issues in real-world medical image classification tasks. To evaluate the effectiveness of the proposed model, we employed a dataset of brain MRI images with and without tumours.
TempSAL -- Uncovering Temporal Information for Deep Saliency Prediction
Authors: Bahar Aydemir, Ludo Hoffstetter, Tong Zhang, Mathieu Salzmann, Sabine Süsstrunk
Date: 2023-01-05 00:00:00
Tasks: Object, Object Recognition, Prediction, Saliency Prediction

Deep saliency prediction algorithms complement the object recognition features, they typically rely on additional information, such as scene context, semantic relationships, gaze direction, and object dissimilarity. However, none of these models consider the temporal nature of gaze shifts during image observation. We introduce a novel saliency prediction model that learns to output saliency maps in sequential time intervals by exploiting human temporal attention patterns. Our approach locally modulates the saliency predictions by combining the learned temporal maps. Our experiments show that our method outperforms the state-of-the-art models, including a multi-duration saliency model, on the SALICON benchmark. Our code will be publicly available on GitHub.
HierVL: Learning Hierarchical Video-Language Embeddings
Authors: Kumar Ashutosh, Rohit Girdhar, Lorenzo Torresani, Kristen Grauman
Date: 2023-01-05 00:00:00
Proceeding: CVPR 2023 1
Tasks: Action Classification, Action Recognition, Long Term Action Anticipation, Long Term Anticipation, Multi-Instance Retrieval

Video-language embeddings are a promising avenue for injecting semantics into visual representations, but existing methods capture only short-term associations between seconds-long video clips and their accompanying text. We propose HierVL, a novel hierarchical video-language embedding that simultaneously accounts for both long-term and short-term associations. As training data, we take videos accompanied by timestamped text descriptions of human actions, together with a high-level text summary of the activity throughout the long video (as are available in Ego4D). We introduce a hierarchical contrastive training objective that encourages text-visual alignment at both the clip level and video level. While the clip-level constraints use the step-by-step descriptions to capture what is happening in that instant, the video-level constraints use the summary text to capture why it is happening, i.e., the broader context for the activity and the intent of the actor. Our hierarchical scheme yields a clip representation that outperforms its single-level counterpart as well as a long-term video representation that achieves SotA results on tasks requiring long-term video modeling. HierVL successfully transfers to multiple challenging downstream tasks (in EPIC-KITCHENS-100, Charades-Ego, HowTo100M) in both zero-shot and fine-tuned settings.
PressureVision++: Estimating Fingertip Pressure from Diverse RGB Images
Authors: Patrick Grady, Jeremy A. Collins, Chengcheng Tang, Christopher D. Twigg, Kunal Aneja, James Hays, Charles C. Kemp
Date: 2023-01-05 00:00:00
Tasks: Mixed Reality

Touch plays a fundamental role in manipulation for humans; however, machine perception of contact and pressure typically requires invasive sensors. Recent research has shown that deep models can estimate hand pressure based on a single RGB image. However, evaluations have been limited to controlled settings since collecting diverse data with ground-truth pressure measurements is difficult. We present a novel approach that enables diverse data to be captured with only an RGB camera and a cooperative participant. Our key insight is that people can be prompted to apply pressure in a certain way, and this prompt can serve as a weak label to supervise models to perform well under varied conditions. We collect a novel dataset with 51 participants making fingertip contact with diverse objects. Our network, PressureVision++, outperforms human annotators and prior work. We also demonstrate an application of PressureVision++ to mixed reality where pressure estimation allows everyday surfaces to be used as arbitrary touch-sensitive interfaces. Code, data, and models are available online.
What You Say Is What You Show: Visual Narration Detection in Instructional Videos
Authors: Kumar Ashutosh, Rohit Girdhar, Lorenzo Torresani, Kristen Grauman
Date: 2023-01-05 00:00:00

Narrated ''how-to'' videos have emerged as a promising data source for a wide range of learning problems, from learning visual representations to training robot policies. However, this data is extremely noisy, as the narrations do not always describe the actions demonstrated in the video. To address this problem we introduce the novel task of visual narration detection, which entails determining whether a narration is visually depicted by the actions in the video. We propose What You Say is What You Show (WYS^2), a method that leverages multi-modal cues and pseudo-labeling to learn to detect visual narrations with only weakly labeled data. Our model successfully detects visual narrations in in-the-wild videos, outperforming strong baselines, and we demonstrate its impact for state-of-the-art summarization and temporal alignment of instructional videos.
Filtering, Distillation, and Hard Negatives for Vision-Language Pre-Training
Authors: Filip Radenovic, Abhimanyu Dubey, Abhishek Kadian, Todor Mihaylov, Simon Vandenhende, Yash Patel, Yi Wen, Vignesh Ramanathan, Dhruv Mahajan
Date: 2023-01-05 00:00:00
Proceeding: CVPR 2023 1
Tasks: Contrastive Learning, Text Spotting, Zero-Shot Learning

Vision-language models trained with contrastive learning on large-scale noisy data are becoming increasingly popular for zero-shot recognition problems. In this paper we improve the following three aspects of the contrastive pre-training pipeline: dataset noise, model initialization and the training objective. First, we propose a straightforward filtering strategy titled Complexity, Action, and Text-spotting (CAT) that significantly reduces dataset size, while achieving improved performance across zero-shot vision-language tasks. Next, we propose an approach titled Concept Distillation to leverage strong unimodal representations for contrastive training that does not increase training complexity while outperforming prior work. Finally, we modify the traditional contrastive alignment objective, and propose an importance-sampling approach to up-sample the importance of hard-negatives without adding additional complexity. On an extensive zero-shot benchmark of 29 tasks, our Distilled and Hard-negative Training (DiHT) approach improves on 20 tasks compared to the baseline. Furthermore, for few-shot linear probing, we propose a novel approach that bridges the gap between zero-shot and few-shot performance, substantially improving over prior work. Models are available at https://github.com/facebookresearch/diht.
Restarts subject to approximate sharpness: A parameter-free and optimal scheme for first-order methods
Authors: Ben Adcock, Matthew J. Colbrook, Maksym Neyra-Nesterenko
Date: 2023-01-05 00:00:00

Sharpness is an almost generic assumption in continuous optimization that bounds the distance from minima by objective function suboptimality. It facilitates the acceleration of first-order methods through restarts. However, sharpness involves problem-specific constants that are typically unknown, and restart schemes typically reduce convergence rates. Moreover, these schemes are challenging to apply in the presence of noise or with approximate model classes (e.g., in compressive imaging or learning problems), and they generally assume that the first-order method used produces feasible iterates. We consider the assumption of approximate sharpness, a generalization of sharpness that incorporates an unknown constant perturbation to the objective function error. This constant offers greater robustness (e.g., with respect to noise or relaxation of model classes) for finding approximate minimizers. By employing a new type of search over the unknown constants, we design a restart scheme that applies to general first-order methods and does not require the first-order method to produce feasible iterates. Our scheme maintains the same convergence rate as when the constants are known. The convergence rates we achieve for various first-order methods match the optimal rates or improve on previously established rates for a wide range of problems. We showcase our restart scheme in several examples and highlight potential future applications and developments of our framework and theory.
GeoDE: a Geographically Diverse Evaluation Dataset for Object Recognition
Date: 2023-01-05 00:00:00
Proceeding: NeurIPS 2023 11
Tasks: Object, Object Recognition

Current dataset collection methods typically scrape large amounts of data from the web. While this technique is extremely scalable, data collected in this way tends to reinforce stereotypical biases, can contain personally identifiable information, and typically originates from Europe and North America. In this work, we rethink the dataset collection paradigm and introduce GeoDE, a geographically diverse dataset with 61,940 images from 40 classes and 6 world regions, and no personally identifiable information, collected through crowd-sourcing. We analyse GeoDE to understand differences in images collected in this manner compared to web-scraping. Despite the smaller size of this dataset, we demonstrate its use as both an evaluation and training dataset, highlight shortcomings in current models, as well as show improved performances when even small amounts of GeoDE (1000 - 2000 images per region) are added to a training dataset. We release the full dataset and code at https://geodiverse-data-collection.cs.princeton.edu/
Super Sparse 3D Object Detection
Authors: Lue Fan, Yuxue Yang, Feng Wang, Naiyan Wang, Zhaoxiang Zhang
Date: 2023-01-05 00:00:00
Tasks: 3D Object Detection, Autonomous Driving, Object, object-detection, Object Detection

As the perception range of LiDAR expands, LiDAR-based 3D object detection contributes ever-increasingly to the long-range perception in autonomous driving. Mainstream 3D object detectors often build dense feature maps, where the cost is quadratic to the perception range, making them hardly scale up to the long-range settings. To enable efficient long-range detection, we first propose a fully sparse object detector termed FSD. FSD is built upon the general sparse voxel encoder and a novel sparse instance recognition (SIR) module. SIR groups the points into instances and applies highly-efficient instance-wise feature extraction. The instance-wise grouping sidesteps the issue of the center feature missing, which hinders the design of the fully sparse architecture. To further enjoy the benefit of fully sparse characteristic, we leverage temporal information to remove data redundancy and propose a super sparse detector named FSD++. FSD++ first generates residual points, which indicate the point changes between consecutive frames. The residual points, along with a few previous foreground points, form the super sparse input data, greatly reducing data redundancy and computational overhead. We comprehensively analyze our method on the large-scale Waymo Open Dataset, and state-of-the-art performance is reported. To showcase the superiority of our method in long-range detection, we also conduct experiments on Argoverse 2 Dataset, where the perception range ($200m$) is much larger than Waymo Open Dataset ($75m$). Code is open-sourced at https://github.com/tusen-ai/SST.
Does compressing activations help model parallel training?
Authors: Song Bian, Dacheng Li, Hongyi Wang, Eric P. Xing, Shivaram Venkataraman
Date: 2023-01-06 00:00:00
Tasks: model, Quantization

Large-scale Transformer models are known for their exceptional performance in a range of tasks, but training them can be difficult due to the requirement for communication-intensive model parallelism. One way to improve training speed is to compress the message size in communication. Previous approaches have primarily focused on compressing gradients in a data parallelism setting, but compression in a model-parallel setting is an understudied area. We have discovered that model parallelism has fundamentally different characteristics than data parallelism. In this work, we present the first empirical study on the effectiveness of compression methods for model parallelism. We implement and evaluate three common classes of compression algorithms - pruning-based, learning-based, and quantization-based - using a popular Transformer training framework. We evaluate these methods across more than 160 settings and 8 popular datasets, taking into account different hyperparameters, hardware, and both fine-tuning and pre-training stages. We also provide analysis when the model is scaled up. Finally, we provide insights for future development of model parallelism compression algorithms.
A Data-Driven Gaussian Process Filter for Electrocardiogram Denoising
Authors: Mircea Dumitru, Qiao Li, Erick Andres Perez Alday, Ali Bahrami Rad, Gari D. Clifford, Reza Sameni
Date: 2023-01-06 00:00:00
Tasks: Denoising, Gaussian Processes

Objective: Gaussian Processes (GP)-based filters, which have been effectively used for various applications including electrocardiogram (ECG) filtering can be computationally demanding and the choice of their hyperparameters is typically ad hoc. Methods: We develop a data-driven GP filter to address both issues, using the notion of the ECG phase domain -- a time-warped representation of the ECG beats onto a fixed number of samples and aligned R-peaks, which is assumed to follow a Gaussian distribution. Under this assumption, the computation of the sample mean and covariance matrix is simplified, enabling an efficient implementation of the GP filter in a data-driven manner, with no ad hoc hyperparameters. The proposed filter is evaluated and compared with a state-of-the-art wavelet-based filter, on the PhysioNet QT Database. The performance is evaluated by measuring the signal-to-noise ratio (SNR) improvement of the filter at SNR levels ranging from -5 to 30dB, in 5dB steps, using additive noise. For a clinical evaluation, the error between the estimated QT-intervals of the original and filtered signals is measured and compared with the benchmark filter. Results: It is shown that the proposed GP filter outperforms the benchmark filter for all the tested noise levels. It also outperforms the state-of-the-art filter in terms of QT-interval estimation error bias and variance. Conclusion: The proposed GP filter is a versatile technique for preprocessing the ECG in clinical and research applications, is applicable to ECG of arbitrary lengths and sampling frequencies, and provides confidence intervals for its performance.
SEQUENT: Towards Traceable Quantum Machine Learning using Sequential Quantum Enhanced Training
Authors: Philipp Altmann, Leo Sünkel, Jonas Stein, Tobias Müller, Christoph Roch, Claudia Linnhoff-Popien
Date: 2023-01-06 00:00:00
Tasks: Hybrid Machine Learning, image-classification, Image Classification, Quantum Machine Learning, Transfer Learning

Applying new computing paradigms like quantum computing to the field of machine learning has recently gained attention. However, as high-dimensional real-world applications are not yet feasible to be solved using purely quantum hardware, hybrid methods using both classical and quantum machine learning paradigms have been proposed. For instance, transfer learning methods have been shown to be successfully applicable to hybrid image classification tasks. Nevertheless, beneficial circuit architectures still need to be explored. Therefore, tracing the impact of the chosen circuit architecture and parameterization is crucial for the development of beneficially applicable hybrid methods. However, current methods include processes where both parts are trained concurrently, therefore not allowing for a strict separability of classical and quantum impact. Thus, those architectures might produce models that yield a superior prediction accuracy whilst employing the least possible quantum impact. To tackle this issue, we propose Sequential Quantum Enhanced Training (SEQUENT) an improved architecture and training process for the traceable application of quantum computing methods to hybrid machine learning. Furthermore, we provide formal evidence for the disadvantage of current methods and preliminary experimental results as a proof-of-concept for the applicability of SEQUENT.
Evaluating counterfactual explanations using Pearl's counterfactual method
Authors: Bevan I. Smith
Date: 2023-01-06 00:00:00
Tasks: counterfactual, Counterfactual Explanation

Counterfactual explanations (CEs) are methods for generating an alternative scenario that produces a different desirable outcome. For example, if a student is predicted to fail a course, then counterfactual explanations can provide the student with alternate ways so that they would be predicted to pass. The applications are many. However, CEs are currently generated from machine learning models that do not necessarily take into account the true causal structure in the data. By doing this, bias can be introduced into the CE quantities. I propose in this study to test the CEs using Judea Pearl's method of computing counterfactuals which has thus far, surprisingly, not been seen in the counterfactual explanation (CE) literature. I furthermore evaluate these CEs on three different causal structures to show how the true underlying causal structure affects the CEs that are generated. This study presented a method of evaluating CEs using Pearl's method and it showed, (although using a limited sample size), that thirty percent of the CEs conflicted with those computed by Pearl's method. This shows that we cannot simply trust CEs and it is vital for us to know the true causal structure before we blindly compute counterfactuals using the original machine learning model.