pwc-archive/papers-with-abstracts huggingface dataset:
Improving fine-grained understanding in image-text pre-training
Authors: Ioana Bica, Anastasija Ilić, Matthias Bauer, Goker Erdogan, Matko Bošnjak, Christos Kaplanis, Alexey A. Gritsenko, Matthias Minderer, Charles Blundell, Razvan Pascanu, Jovana Mitrović
Date: 2024-01-18 00:00:00
Tasks: object-detection, Object Detection

We introduce SPARse Fine-grained Contrastive Alignment (SPARC), a simple method for pretraining more fine-grained multimodal representations from image-text pairs. Given that multiple image patches often correspond to single words, we propose to learn a grouping of image patches for every token in the caption. To achieve this, we use a sparse similarity metric between image patches and language tokens and compute for each token a language-grouped vision embedding as the weighted average of patches. The token and language-grouped vision embeddings are then contrasted through a fine-grained sequence-wise loss that only depends on individual samples and does not require other batch samples as negatives. This enables more detailed information to be learned in a computationally inexpensive manner. SPARC combines this fine-grained loss with a contrastive loss between global image and text embeddings to learn representations that simultaneously encode global and local information. We thoroughly evaluate our proposed method and show improved performance over competing approaches both on image-level tasks relying on coarse-grained information, e.g. classification, as well as region-level tasks relying on fine-grained information, e.g. retrieval, object detection, and segmentation. Moreover, SPARC improves model faithfulness and captioning in foundational vision-language models.
Evolutionary Multi-Objective Optimization of Large Language Model Prompts for Balancing Sentiments
Authors: Jill Baumann, Oliver Kramer
Date: 2024-01-18 00:00:00
Tasks: Evolutionary Algorithms, Language Modeling, Language Modelling, Large Language Model, Prompt Engineering, Sentiment Analysis

The advent of large language models (LLMs) such as ChatGPT has attracted considerable attention in various domains due to their remarkable performance and versatility. As the use of these models continues to grow, the importance of effective prompt engineering has come to the fore. Prompt optimization emerges as a crucial challenge, as it has a direct impact on model performance and the extraction of relevant information. Recently, evolutionary algorithms (EAs) have shown promise in addressing this issue, paving the way for novel optimization strategies. In this work, we propose a evolutionary multi-objective (EMO) approach specifically tailored for prompt optimization called EMO-Prompts, using sentiment analysis as a case study. We use sentiment analysis capabilities as our experimental targets. Our results demonstrate that EMO-Prompts effectively generates prompts capable of guiding the LLM to produce texts embodying two conflicting emotions simultaneously.
Temporal Insight Enhancement: Mitigating Temporal Hallucination in Multimodal Large Language Models
Authors: Li Sun, Liuan Wang, Jun Sun, Takayuki Okatani
Date: 2024-01-18 00:00:00
Tasks: Hallucination

Recent advancements in Multimodal Large Language Models (MLLMs) have significantly enhanced the comprehension of multimedia content, bringing together diverse modalities such as text, images, and videos. However, a critical challenge faced by these models, especially when processing video inputs, is the occurrence of hallucinations - erroneous perceptions or interpretations, particularly at the event level. This study introduces an innovative method to address event-level hallucinations in MLLMs, focusing on specific temporal understanding in video content. Our approach leverages a novel framework that extracts and utilizes event-specific information from both the event query and the provided video to refine MLLMs' response. We propose a unique mechanism that decomposes on-demand event queries into iconic actions. Subsequently, we employ models like CLIP and BLIP2 to predict specific timestamps for event occurrences. Our evaluation, conducted using the Charades-STA dataset, demonstrates a significant reduction in temporal hallucinations and an improvement in the quality of event-related responses. This research not only provides a new perspective in addressing a critical limitation of MLLMs but also contributes a quantitatively measurable method for evaluating MLLMs in the context of temporal-related questions.
Enhancing the Fairness and Performance of Edge Cameras with Explainable AI
Authors: Truong Thanh Hung Nguyen, Vo Thanh Khang Nguyen, Quoc Hung Cao, Van Binh Truong, Quoc Khanh Nguyen, Hung Cao
Date: 2024-01-18 00:00:00
Tasks: Diagnostic, Fairness, Human Detection

The rising use of Artificial Intelligence (AI) in human detection on Edge camera systems has led to accurate but complex models, challenging to interpret and debug. Our research presents a diagnostic method using Explainable AI (XAI) for model debugging, with expert-driven problem identification and solution creation. Validated on the Bytetrack model in a real-world office Edge network, we found the training dataset as the main bias source and suggested model augmentation as a solution. Our approach helps identify model biases, essential for achieving fair and trustworthy models.
Next-Generation Simulation Illuminates Scientific Problems of Organised Complexity
Authors: Cheng Wang, Chuwen Wang, Wang Zhang, Shirong Zeng, Yu Zhao, Ronghui Ning, Changjun Jiang
Date: 2024-01-18 00:00:00
Tasks: Weather Forecasting

As artificial intelligence becomes increasingly prevalent in scientific research, data-driven methodologies appear to overshadow traditional approaches in resolving scientific problems. In this Perspective, we revisit a classic classification of scientific problems and acknowledge that a series of unresolved problems remain. Throughout the history of researching scientific problems, scientists have continuously formed new paradigms facilitated by advances in data, algorithms, and computational power. To better tackle unresolved problems, especially those of organised complexity, a novel paradigm is necessitated. While recognising that the strengths of new paradigms have expanded the scope of resolvable scientific problems, we aware that the continued advancement of data, algorithms, and computational power alone is hardly to bring a new paradigm. We posit that the integration of paradigms, which capitalises on the strengths of each, represents a promising approach. Specifically, we focus on next-generation simulation (NGS), which can serve as a platform to integrate methods from different paradigms. We propose a methodology, sophisticated behavioural simulation (SBS), to realise it. SBS represents a higher level of paradigms integration based on foundational models to simulate complex systems, such as social systems involving sophisticated human strategies and behaviours. NGS extends beyond the capabilities of traditional mathematical modelling simulations and agent-based modelling simulations, and therefore, positions itself as a potential solution to problems of organised complexity in complex systems.
Slicer Networks
Authors: Hang Zhang, Xiang Chen, Rongguang Wang, Renjiu Hu, Dongdong Liu, Gaolei Li
Date: 2024-01-18 00:00:00
Tasks: Image Registration, Lesion Segmentation, Medical Image Analysis, Segmentation

In medical imaging, scans often reveal objects with varied contrasts but consistent internal intensities or textures. This characteristic enables the use of low-frequency approximations for tasks such as segmentation and deformation field estimation. Yet, integrating this concept into neural network architectures for medical image analysis remains underexplored. In this paper, we propose the Slicer Network, a novel architecture designed to leverage these traits. Comprising an encoder utilizing models like vision transformers for feature extraction and a slicer employing a learnable bilateral grid, the Slicer Network strategically refines and upsamples feature maps via a splatting-blurring-slicing process. This introduces an edge-preserving low-frequency approximation for the network outcome, effectively enlarging the effective receptive field. The enhancement not only reduces computational complexity but also boosts overall performance. Experiments across different medical imaging applications, including unsupervised and keypoints-based image registration and lesion segmentation, have verified the Slicer Network's improved accuracy and efficiency.
PPNet: A Two-Stage Neural Network for End-to-end Path Planning
Authors: Qinglong Meng, Chongkun Xia, Xueqian Wang, Songping Mai, Bin Liang
Date: 2024-01-18 00:00:00

The classical path planners, such as sampling-based path planners, can provide probabilistic completeness guarantees in the sense that the probability that the planner fails to return a solution if one exists, decays to zero as the number of samples approaches infinity. However, finding a near-optimal feasible solution in a given period is challenging in many applications such as the autonomous vehicle. To achieve an end-to-end near-optimal path planner, we first divide the path planning problem into two subproblems, which are path space segmentation and waypoints generation in the given path's space. We further propose a two-stage neural network named Path Planning Network (PPNet) each stage solves one of the subproblems abovementioned. Moreover, we propose a novel efficient data generation method for path planning named EDaGe-PP. EDaGe-PP can generate data with continuous-curvature paths with analytical expression while satisfying the clearance requirement. The results show the total computation time of generating random 2D path planning data is less than 1/33 and the success rate of PPNet trained by the dataset that is generated by EDaGe-PP is about 2 times compared to other methods. We validate PPNet against state-of-the-art path planning methods. The results show that PPNet can find a near-optimal solution in 15.3ms, which is much shorter than the state-of-the-art path planners.
A Comparative Analysis on Metaheuristic Algorithms Based Vision Transformer Model for Early Detection of Alzheimer's Disease
Authors: Anuvab Sen, Udayon Sen, Subhabrata Roy
Date: 2024-01-18 00:00:00

A number of life threatening neuro-degenerative disorders had degraded the quality of life for the older generation in particular. Dementia is one such symptom which may lead to a severe condition called Alzheimer's disease if not detected at an early stage. It has been reported that the progression of such disease from a normal stage is due to the change in several parameters inside the human brain. In this paper, an innovative metaheuristic algorithms based ViT model has been proposed for the identification of dementia at different stage. A sizeable number of test data have been utilized for the validation of the proposed scheme. It has also been demonstrated that our model exhibits superior performance in terms of accuracy, precision, recall as well as F1-score.
A Semantic Approach for Big Data Exploration in Industry 4.0
Authors: Idoia Berges, Víctor Julio Ramírez-Durán, Arantza Illarramendi
Date: 2024-01-18 00:00:00
Tasks: Cloud Computing

The growing trends in automation, Internet of Things, big data and cloud computing technologies have led to the fourth industrial revolution (Industry 4.0), where it is possible to visualize and identify patterns and insights, which results in a better understanding of the data and can improve the manufacturing process. However, many times, the task of data exploration results difficult for manufacturing experts because they might be interested in analyzing also data that does not appear in pre-designed visualizations and therefore they must be assisted by Information Technology experts. In this paper, we present a proposal materialized in a semantic-based visual query system developed for a real Industry 4.0 scenario that allows domain experts to explore and visualize data in a friendly way. The main novelty of the system is the combined use that it makes of captured data that are semantically annotated first, and a 2D customized digital representation of a machine that is also linked with semantic descriptions. Those descriptions are expressed using terms of an ontology, where, among others, the sensors that are used to capture indicators about the performance of a machine that belongs to a Industry 4.0 scenario have been modeled. Moreover, this semantic description allows to: formulate queries at a higher level of abstraction, provide customized graphical visualizations of the results based on the format and nature of the data, and download enriched data enabling further types of analysis.
Querying Easily Flip-flopped Samples for Deep Active Learning
Authors: Seong Jin Cho, Gwangsu Kim, Junghyun Lee, Jinwoo Shin, Chang D. Yoo
Date: 2024-01-18 00:00:00
Tasks: Active Learning

Active learning is a machine learning paradigm that aims to improve the performance of a model by strategically selecting and querying unlabeled data. One effective selection strategy is to base it on the model's predictive uncertainty, which can be interpreted as a measure of how informative a sample is. The sample's distance to the decision boundary is a natural measure of predictive uncertainty, but it is often intractable to compute, especially for complex decision boundaries formed in multiclass classification tasks. To address this issue, this paper proposes the {\it least disagree metric} (LDM), defined as the smallest probability of disagreement of the predicted label, and an estimator for LDM proven to be asymptotically consistent under mild assumptions. The estimator is computationally efficient and can be easily implemented for deep learning models using parameter perturbation. The LDM-based active learning is performed by querying unlabeled data with the smallest LDM. Experimental results show that our LDM-based active learning algorithm obtains state-of-the-art overall performance on all considered datasets and deep architectures.
Adaptive Self-training Framework for Fine-grained Scene Graph Generation
Authors: Kibum Kim, Kanghoon Yoon, Yeonjun In, Jinyoung Moon, Donghyun Kim, Chanyoung Park
Date: 2024-01-18 00:00:00
Tasks: Graph Generation, Scene Graph Generation

Scene graph generation (SGG) models have suffered from inherent problems regarding the benchmark datasets such as the long-tailed predicate distribution and missing annotation problems. In this work, we aim to alleviate the long-tailed problem of SGG by utilizing unannotated triplets. To this end, we introduce a Self-Training framework for SGG (ST-SGG) that assigns pseudo-labels for unannotated triplets based on which the SGG models are trained. While there has been significant progress in self-training for image recognition, designing a self-training framework for the SGG task is more challenging due to its inherent nature such as the semantic ambiguity and the long-tailed distribution of predicate classes. Hence, we propose a novel pseudo-labeling technique for SGG, called Class-specific Adaptive Thresholding with Momentum (CATM), which is a model-agnostic framework that can be applied to any existing SGG models. Furthermore, we devise a graph structure learner (GSL) that is beneficial when adopting our proposed self-training framework to the state-of-the-art message-passing neural network (MPNN)-based SGG models. Our extensive experiments verify the effectiveness of ST-SGG on various SGG models, particularly in enhancing the performance on fine-grained predicate classes.
SEINE: Structure Encoding and Interaction Network for Nuclei Instance Segmentation
Authors: Ye Zhang, Linghan Cai, Ziyue Wang, Yongbing Zhang
Date: 2024-01-18 00:00:00
Tasks: Instance Segmentation, Semantic Segmentation

Nuclei instance segmentation in histopathological images is of great importance for biological analysis and cancer diagnosis but remains challenging for two reasons. (1) Similar visual presentation of intranuclear and extranuclear regions of chromophobe nuclei often causes under-segmentation, and (2) current methods lack the exploration of nuclei structure, resulting in fragmented instance predictions. To address these problems, this paper proposes a structure encoding and interaction network, termed SEINE, which develops the structure modeling scheme of nuclei and exploits the structure similarity between nuclei to improve the integrality of each segmented instance. Concretely, SEINE introduces a contour-based structure encoding (SE) that considers the correlation between nuclei structure and semantics, realizing a reasonable representation of the nuclei structure. Based on the encoding, we propose a structure-guided attention (SGA) module that takes the clear nuclei as prototypes to enhance the structure learning for the fuzzy nuclei. To strengthen the structural learning ability, a semantic feature fusion (SFF) is presented to boost the semantic consistency of semantic and structure branches. Furthermore, a position enhancement (PE) method is applied to suppress incorrect nuclei boundary predictions. Extensive experiments demonstrate the superiority of our approaches, and SEINE achieves state-of-the-art (SOTA) performance on four datasets. The code is available at https://github.com/zhangye-zoe/SEINE.
A Survey on Learning from Graphs with Heterophily: Recent Advances and Future Directions
Authors: Chenghua Gong, Yao Cheng, Jianxiang Yu, Can Xu, Caihua Shan, Siqiang Luo, Xiang Li
Date: 2024-01-18 00:00:00
Tasks: Graph Learning, Survey

Graphs are structured data that models complex relations between real-world entities. Heterophilic graphs, where linked nodes are prone to be with different labels or dissimilar features, have recently attracted significant attention and found many real-world applications. Meanwhile, increasing efforts have been made to advance learning from graphs with heterophily. Various graph heterophily measures, benchmark datasets, and learning paradigms are emerging rapidly. In this survey, we comprehensively review existing works on learning from graphs with heterophily. First, we overview over 500 publications, of which more than 340 are directly related to heterophilic graphs. After that, we survey existing metrics of graph heterophily and list recent benchmark datasets. Further, we systematically categorize existing methods based on a hierarchical taxonomy including GNN models, learning paradigms and practical applications. In addition, broader topics related to graph heterophily are also included. Finally, we discuss the primary challenges of existing studies and highlight promising avenues for future research.
CLIP Model for Images to Textual Prompts Based on Top-k Neighbors
Authors: Xin Zhang, YeMing Cai, Tianzhi Jia
Date: 2024-01-18 00:00:00
Tasks: Image Generation, multimodal generation

Text-to-image synthesis, a subfield of multimodal generation, has gained significant attention in recent years. We propose a cost-effective approach for image-to-prompt generation that leverages generative models to generate textual prompts without the need for large amounts of annotated data. We divide our method into two stages: online stage and offline stage. We use a combination of the CLIP model and K-nearest neighbors (KNN) algorithm. The proposed system consists of two main parts: an offline task and an online task. Our method owns the highest metric 0.612 among these models, which is 0.013, 0.055, 0.011 higher than Clip, Clip + KNN(top 10) respectively.
Cooperative Tri-Point Model-Based Ground-to-Air Coverage Extension in Beyond 5G Networks
Authors: Ziwei Cai, Min Sheng, Junju Liu, Chenxi Zhao, Jiandong Li
Date: 2024-01-18 00:00:00

The utilization of existing terrestrial infrastructures to provide coverage for aerial users is a potentially low-cost solution. However, the already deployed terrestrial base stations (TBSs) result in weak ground-to-air (G2A) coverage due to the down-tilted antennas. Furthermore, achieving optimal coverage across the entire airspace through antenna adjustment is challenging due to the complex signal coverage requirements in three-dimensional space, especially in the vertical direction. In this paper, we propose a cooperative tri-point (CoTP) model-based method that utilizes cooperative beams to enhance the G2A coverage extension. To utilize existing TBSs for establishing effective cooperation, we prove that the cooperation among three TBSs can ensure G2A coverage with a minimum coverage overlap, and design the CoTP model to analyze the G2A coverage extension. Using the model, a cooperative coverage structure based on Delaunay triangulation is designed to divide triangular prism-shaped subspaces and corresponding TBS cooperation sets. To enable TBSs in the cooperation set to cover different height subspaces while maintaining ground coverage, we design a cooperative beam generation algorithm to maximize the coverage in the triangular prism-shaped airspace. The simulation results and field trials demonstrate that the proposed method can efficiently enhance the G2A coverage extension while guaranteeing ground coverage.
Explaining Drift using Shapley Values
Authors: Narayanan U. Edakunni, Utkarsh Tekriwal, Anukriti Jain
Date: 2024-01-18 00:00:00

Machine learning models often deteriorate in their performance when they are used to predict the outcomes over data on which they were not trained. These scenarios can often arise in real world when the distribution of data changes gradually or abruptly due to major events like a pandemic. There have been many attempts in machine learning research to come up with techniques that are resilient to such Concept drifts. However, there is no principled framework to identify the drivers behind the drift in model performance. In this paper, we propose a novel framework - DBShap that uses Shapley values to identify the main contributors of the drift and quantify their respective contributions. The proposed framework not only quantifies the importance of individual features in driving the drift but also includes the change in the underlying relation between the input and output as a possible driver. The explanation provided by DBShap can be used to understand the root cause behind the drift and use it to make the model resilient to the drift.
Bootstrapping OTS-Funcimg Pre-training Model (Botfip) -- A Comprehensive Symbolic Regression Framework
Authors: Tianhao Chen, Pengbo Xu, Haibiao Zheng
Date: 2024-01-18 00:00:00
Tasks: regression, Symbolic Regression

In the field of scientific computing, many problem-solving approaches tend to focus only on the process and final outcome, even in AI for science, there is a lack of deep multimodal information mining behind the data, missing a multimodal framework akin to that in the image-text domain. In this paper, we take Symbolic Regression(SR) as our focal point and, drawing inspiration from the BLIP model in the image-text domain, propose a scientific computing multimodal framework based on Function Images (Funcimg) and Operation Tree Sequence (OTS), named Bootstrapping OTS-Funcimg Pre-training Model (Botfip). In SR experiments, we validate the advantages of Botfip in low-complexity SR problems, showcasing its potential. As a MED framework, Botfip holds promise for future applications in a broader range of scientific computing problems.