pwc-archive/papers-with-abstracts huggingface dataset:
Joint Searching and Grounding: Multi-Granularity Video Content Retrieval
Authors: Zhiguo Chen, Xun Jiang, Xing Xu, Zuo Cao, Yijun Mo, and Heng Tao Shen
Date: 2023-10-23 00:00:00
Proceeding: Conference 2023 10
Tasks: Contrastive Learning, Retrieval, Video Retrieval

Text-based video retrieval is a well-studied task aimed at retrieving relevant videos from a large collection in response to a given text query. Most existing TVR works assume that videos are already trimmed and fully relevant to the query thus ignoring that most videos in real-world scenarios are untrimmed and contain massive irrelevant video content. Moreover, as users' queries are only relevant to video events rather than complete videos, it is also more practical to provide specific video events rather than an untrimmed video list. In this paper, we introduce a challenging but more realistic task called Multi-Granularity Video Content Retrieval (MGVCR), which involves retrieving both video files and specific video content with their temporal locations. This task presents significant challenges since it requires identifying and ranking the partial relevance between long videos and text queries under the lack of temporal alignment supervision between the query and relevant moments. To this end, we propose a novel unified framework, termed, Joint Searching and Grounding (JSG). It consists of two branches: (1) a glance branch that coarsely aligns the query and moment proposals using inter-video contrastive learning, and (2) a gaze branch that finely aligns two modalities using both inter- and intra-video contrastive learning. Based on the glance-to-gaze design, our JSG method learns two separate joint embedding spaces for moments and text queries using a hybrid synergistic contrastive learning strategy. Extensive experiments on three public benchmarks, i.e., Charades-STA, DiDeMo, and ActivityNet-Captions demonstrate the superior performance of our JSG method on both video-level retrieval and event-level retrieval subtasks. Our open-source implementation code is available at https://github.com/CFM-MSG/Code_JSG.
Multi-Grained Attention Network With Mutual Exclusion for Composed Query-Based Image Retrieval
Authors: Shenshen Li; Xing Xu; Xun Jiang; Fumin Shen; Xin Liu; Heng Tao Shen
Date: 2023-08-21 00:00:00
Proceeding: journal 2023 8
Tasks: Image Retrieval, Retrieval

The Composed Query-Based Image Retrieval (CQBIR) task aims to precisely obtain the preserved and modified parts, based on the multi-grained semantics learned from the composed query. Since the composed query includes a reference image and the modification text, not just a single modality, this task is more challenging than the general image retrieval tasks. Most previous methods attempt to learn preserved and modified parts via different attention modules and fuse them as a unified representation. However, these methods have two intrinsic drawbacks: 1) The different granular semantic information of the composed query is neglected, which results in the fact that learned preserved and modified parts are irrelevant to correct semantics. 2) The preserved and modified parts learned by previous methods have obvious overlaps, which may lead the model to obtain sub-optimal preserved and modified regions. To this end, we propose a novel method termed Multi-Grained Attention Network with Mutual Exclusion (MANME) to address the above problems. Our MANME method mainly consists of two components: 1) A multi-grained semantic construction for obtaining various textual and visual semantic information. 2) An attention with mutual exclusion constraint for reducing the degree of overlap between preserved and modified parts. It adequately utilizes the various granular semantic information and effectively refines the learned preserved and modified parts. Extensive experiments and further analyses on three widely used CQBIR datasets demonstrate that our proposed MANME method achieves new state-of-the-art performance on the CQBIR task.
Privacy-Preserved Evolutionary Graph Modeling via Gromov-Wasserstein Autoregression
Authors: Yue Xiang，Dixin Luo，Hongteng Xu
Date: 2023-06-26 00:00:00
Proceeding: Conference 2023 6
Tasks: Privacy Preserving

Real-world graphs like social networks are often evolutionary over time, whose observations at different timestamps lead to graph sequences. Modeling such evolutionary graphs is important for many applications, but solving this problem often requires the correspondence between the graphs at different timestamps, which may leak private node information, e.g., the temporal behavior patterns of the nodes. We proposed a Gromov-Wasserstein Autoregressive (GWAR) model to capture the generative mechanisms of evolutionary graphs, which does not require the correspondence information and thus preserves the privacy of the graphs' nodes. This model consists of two autoregressions, predicting the number of nodes and the probabilities of nodes and edges, respectively. The model takes observed graphs as its input and predicts future graphs via solving a joint graph alignment and merging task. This task leads to a fused Gromov-Wasserstein (FGW) barycenter problem, in which we approximate the alignment of the graphs based on a novel inductive fused Gromov-Wasserstein (IFGW) distance. The IFGW distance is parameterized by neural networks and can be learned under mild assumptions, thus, we can infer the FGW barycenters without iterative optimization and predict future graphs efficiently. Experiments show that our GWAR achieves encouraging performance in modeling evolutionary graphs in privacy-preserving scenarios.
Robust Generalization against Corruptions via Worst-Case Sharp ness Minimization
Authors: Zhuo Huang, Xiaobo Xia, Li Shen, Jun Yu, Chen Gong, Bo Han, Tongliang Liu
Date: 2023-02-14 00:00:00
Proceeding: journal 2023 2

Robust generalization aims to deal with the most challenging data distributions which are rarely presented in training set and contain severe noise corruptions. Common solutions such as distributionally robust optimization (DRO) focus on the worst-case empirical risk to ensure low training error on the uncommon noisy distributions. However, due to the over-parameterized model being optimized on scarce worst-case data, DRO fails to produce a smooth loss landscape, thus struggling on generalizing well to the test set. Therefore, instead of focusing on the worst-case risk minimization, we propose SharpDRO by penalizing the sharpness of the worst-case distribution, which measures the loss changes around the neighbor of learning parameters. Through worst-case sharpness minimization, the proposed method successfully produces a flat loss curve on the corrupted distributions, thus achieving robust generalization. Moreover, by considering whether the distribution annotation is available, we apply SharpDRO to two problem settings and design a worst-case selection process for robust generalization. Through simulating real-world noisy distributions using CIFAR10/100 and ImageNet30 datasets, we show that SharpDRO exhibits strong generalization ability against severe corruptions and exceeds well-known baseline methods with large performance gains.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
Efficient Personalized Text-to-image Generation by Leveraging Textual Subspace
Authors: Shian Du, Xiaotian Cheng, Qi Qian, Henglu Wei, Yi Xu, Xiangyang Ji
Date: 2024-06-30 00:00:00
Tasks: Image Generation, Representation Learning, Text to Image Generation, Text-to-Image Generation

Personalized text-to-image generation has attracted unprecedented attention in the recent few years due to its unique capability of generating highly-personalized images via using the input concept dataset and novel textual prompt. However, previous methods solely focus on the performance of the reconstruction task, degrading its ability to combine with different textual prompt. Besides, optimizing in the high-dimensional embedding space usually leads to unnecessary time-consuming training process and slow convergence. To address these issues, we propose an efficient method to explore the target embedding in a textual subspace, drawing inspiration from the self-expressiveness property. Additionally, we propose an efficient selection strategy for determining the basis vectors of the textual subspace. The experimental evaluations demonstrate that the learned embedding can not only faithfully reconstruct input image, but also significantly improves its alignment with novel input textual prompt. Furthermore, we observe that optimizing in the textual subspace leads to an significant improvement of the robustness to the initial word, relaxing the constraint that requires users to input the most relevant initial word. Our method opens the door to more efficient representation learning for personalized text-to-image generation.
Training morphological neural networks with gradient descent: some theoretical insights
Authors: Samy Blusseau
Date: 2024-02-05 00:00:00

Morphological neural networks, or layers, can be a powerful tool to boost the progress in mathematical morphology, either on theoretical aspects such as the representation of complete lattice operators, or in the development of image processing pipelines. However, these architectures turn out to be difficult to train when they count more than a few morphological layers, at least within popular machine learning frameworks which use gradient descent based optimization algorithms. In this paper we investigate the potential and limitations of differentiation based approaches and back-propagation applied to morphological networks, in light of the non-smooth optimization concept of Bouligand derivative. We provide insights and first theoretical guidelines, in particular regarding initialization and learning rates.
Mimicking User Data: On Mitigating Fine-Tuning Risks in Closed Large Language Models
Authors: Francisco Eiras, Aleksandar Petrov, Phillip H. S. Torr, M. Pawan Kumar, Adel Bibi
Date: 2024-06-12 00:00:00
Tasks: Instruction Following, Safety Alignment

Fine-tuning large language models on small, high-quality datasets can enhance their performance on specific downstream tasks. Recent research shows that fine-tuning on benign, instruction-following data can inadvertently undo the safety alignment process and increase a model's propensity to comply with harmful queries. Although critical, understanding and mitigating safety risks in well-defined tasks remains distinct from the instruction-following context due to structural differences in the data. Our work addresses the gap in our understanding of these risks across diverse types of data in closed models - where providers control how user data is utilized in the fine-tuning process. We demonstrate how malicious actors can subtly manipulate the structure of almost any task-specific dataset to foster significantly more dangerous model behaviors, while maintaining an appearance of innocuity and reasonable downstream task performance. To address this issue, we propose a novel mitigation strategy that mixes in safety data which mimics the task format and prompting style of the user data, showing this is more effective than existing baselines at re-establishing safety alignment while maintaining similar task performance.
Unveiling Themes in Judicial Proceedings: A Cross-Country Study Using Topic Modeling on Legal Documents from India and the UK
Authors: Krish Didwania, Dr. Durga Toshniwal, Amit Agarwal
Date: 2024-05-27 00:00:00

Legal documents are indispensable in every country for legal practices and serve as the primary source of information regarding previous cases and employed statutes. In today's world, with an increasing number of judicial cases, it is crucial to systematically categorize past cases into subgroups, which can then be utilized for upcoming cases and practices. Our primary focus in this endeavor was to annotate cases using topic modeling algorithms such as Latent Dirichlet Allocation, Non-Negative Matrix Factorization, and Bertopic for a collection of lengthy legal documents from India and the UK. This step is crucial for distinguishing the generated labels between the two countries, highlighting the differences in the types of cases that arise in each jurisdiction. Furthermore, an analysis of the timeline of cases from India was conducted to discern the evolution of dominant topics over the years.
Moderate Coreset: A Universal Method of Data Selection for Real-world Data-efficient Deep Learning
Authors: Xiaobo Xia · Jiale Liu · Jun Yu · Xu Shen · Bo Han · Tongliang Liu
Date: 2023-02-28 00:00:00
Proceeding: journal 2023 2

Deep learning methods nowadays rely on massive data, resulting in substantial costs of data storage and model training. Data selection is a useful tool to alleviate such costs, where a coreset of massive data is extracted to practically perform on par with full data. Based on carefully-designed score criteria, existing methods first count the score of each data point and then select the data points whose scores lie in a certain range to construct a coreset. These methods work well in their respective preconceived scenarios but are not robust to the change of scenarios, since the optimal range of scores varies as the scenario changes. The issue limits the application of these methods, because realistic scenarios often mismatch preconceived ones, and it is inconvenient or unfeasible to tune the criteria and methods accordingly. In this paper, to address the issue, a concept of the moderate coreset is discussed. Specifically, given any score criterion of data selection, different scenarios prefer data points with scores in different intervals. As the score median is a proxy of the score distribution in statistics, the data points with scores close to the score median can be seen as a proxy of full data and generalize different scenarios, which are used to construct the moderate coreset. As a proof-of-concept, a universal method that inherits the moderate coreset and uses the distance of a data point to its class center as the score criterion, is proposed to meet complex realistic scenarios. Extensive experiments confirm the advance of our method over prior state-of-the-art methods, leading to a strong baseline for future research. The implementation is available at https://github.com/tmllab/Moderate-DS.
HJE: Joint Convolutional Representation Learning for Knowledge Hypergraph Completion
Authors: Zhao Li; Chenxu Wang; Xin Wang; Zirui Chen; Jianxin Li
Date: 2024-02-13 00:00:00
Proceeding: journal 2024 2
Tasks: Knowledge Graphs, Representation Learning

Knowledge hypergraph representation learning , which projects entities and n -ary relations into a low-dimensional vector space, remains a challenging area to be explored despite the ubiquity of n -ary relational facts in the real world. Current methods are always extensions of those used for knowledge graphs with shallow or deep structures. However, shallow and linear models limit the extraction capacity of the latent knowledge, while deep and non-linear models lead to the overabundance of parameters. In this paper, we propose a novel knowledge hypergraph completion model called HJE, which utilizes the powerful capability of convolutional neural networks for efficient representation learning. Interaction-enhanced 3D convolution and relation-aware 2D convolution are jointly utilized by HJE to extract explicit and implicit global knowledge and semantic information effectively without compromising the translation property of the model. Moreover, HJE constructs a unified learnable embedding matrix to capture entity position information in knowledge tuples. The entity mask mechanism can naturally couple the multilinear scoring approach for n -ary facts to speed up the training convergence of the model. Extensive experimental results on real datasets of knowledge hypergraphs and knowledge graphs demonstrate the superior performance of HJE compared with state-of-the-art baselines.
HyConvE: A Novel Embedding Model for Knowledge Hypergraph Link Prediction with Convolutional Neural Networks
Authors: Chenxu Wang, Xin Wang, Zhao Li, Zirui Chen, and Jianxin Li
Date: 2023-04-01 00:00:00
Proceeding: journal 2023 4
Tasks: hypergraph embedding, Knowledge Graphs, Link Prediction, Position, Prediction

Knowledge hypergraph embedding, which projects entities and n-ary relations into a low-dimensional continuous vector space to predict missing links, remains a challenging area to be explored despite the ubiquity of n-ary relational facts in the real world. Currently, knowledge hypergraph link prediction methods are essentially simple extensions of those used in knowledge graphs, where n-ary relational facts are decomposed into different subelements. Convolutional neural networks have been shown to have remarkable information extraction capabilities in previous work on knowledge graph link prediction. In this paper, we propose a novel embedding-based knowledge hypergraph link prediction model named HyConvE, which exploits the powerful learning ability of convolutional neural networks for effective link prediction. Specifically, we employ 3D convolution to capture the deep interactions of entities and relations to efficiently extract explicit and implicit knowledge in each n-ary relational fact without compromising its translation property. In addition, appropriate relation and position-aware filters are utilized sequentially to perform two-dimensional convolution operations to capture the intrinsic patterns and position information in each n-ary relation, respectively. Extensive experimental results on real datasets of knowledge hypergraphs and knowledge graphs demonstrate the superior performance of HyConvE compared with state-of-the-art baselines.
EnhancE：Enhanced Entity and Relation Embedding for Knowledge Hypergraph Link Prediction
Authors: Chenxu Wang, Zhao Li, Xin Wang, and Zirui Chen
Date: 2023-04-30 00:00:00
Proceeding: journal 2023 4
Tasks: Knowledge Graphs, Link Prediction, Position, Prediction, Relation

Knowledge Hypergraphs, as the generalization of knowledge graphs, have attracted increasingly widespread attention due to their friendly compatibility with real-world facts. However, link prediction in knowledge hypergraph is still an underexplored field despite the ubiquity of n-ary facts in the real world. Several recent representative embedding-based knowledge hypergraph link prediction methods have proven to be effective in a series of benchmarks, however, they only consider the position (or role) information, ignoring the neighborhood structure among entities and rich semantic information within each fact. To this end, we propose a model named EnhancE for effective link prediction in knowledge hypergraphs. On the one hand, a more expressive entity representation is obtained with both position and neighborhood information added to the initial embedding. On the other hand, rich semantic information of the involved entities within each tuple is incorporated into relation embedding for enhanced representation. Extensive experimental results over real datasets of both knowledge hypergraph and knowledge graph demonstrate the excellent performance of EnhancE compared with a variety of state-of-the-art baselines.
RAC-BERT: Character Radical Enhanced BERT for Ancient Chinese
Authors: Lifan Han, Xin Wang, Meng Wang, Zhao Li, Heyi Zhang, Zirui Chen, Xiaowang Zhang
Date: 2023-10-08 00:00:00
Proceeding: journal 2023 10
Tasks: Natural Language Understanding, Text Generation

In recent years, Chinese pre-training language models have achieved significant improvements in the fields, such as natural language understanding (NLU) and text generation. However, most of these existing pre-trained language models focus on modern Chinese but ignore the rich semantic information embedded for Chinese characters, especially the radical information. To this end, we present RAC-BERT, a language-specific BERT model for ancient Chinese. Specifically, we propose two new radical-based pre-training tasks, which are: (1) replacing the masked tokens with random words of the same radical, that can mitigate the gap between the pre-training and fine-tuning stages; (2) predicting the radical of the masked token, not the original word, that reduces the computational effort. Extensive experiments were conducted on two ancient Chinese NLP datasets. The results show that our model significantly outperforms the state-of-the-art models on most tasks. And we conducted ablation experiments to demonstrate the effectiveness of our approach. The pre-trained model are publicly available at https://github.com/CubeHan/RAC-BERT

Access provided by Huawei Technologi
CycMuNet+: Cycle-Projected Mutual Learning for Spatial-Temporal Video Super-Resolution
Authors: Mengshun Hu; Kui Jiang; Zheng Wang; Xiang Bai; Ruimin Hu
Date: 2023-07-23 00:00:00
Proceeding: journal 2023 7
Tasks: Super-Resolution, Video Reconstruction, Video Super-Resolution

Spatial-Temporal Video Super-Resolution (ST-VSR) aims to generate high-quality videos with higher resolution (HR) and higher frame rate (HFR). Quite intuitively, pioneering two-stage based methods complete ST-VSR by directly combining two sub-tasks: Spatial Video Super-Resolution (S-VSR) and Temporal Video Super-Resolution (T-VSR) but ignore the reciprocal relations among them. 1) T-VSR to S-VSR: temporal correlations help accurate spatial detail representation; 2) S-VSR to T-VSR: abundant spatial information contributes to the refinement of temporal prediction. To this end, we propose a one-stage based Cycle-projected Mutual learning network (CycMuNet) for ST-VSR, which makes full use of spatial-temporal correlations via the mutual learning between S-VSR and T-VSR. Specifically, we propose to exploit the mutual information among them via iterative up- and down projections, where spatial and temporal features are fully fused and distilled, helping high-quality video reconstruction. In addition, we also show interesting extensions for efficient network design (CycMuNet+), such as parameter sharing and dense connection on projection units and feedback mechanism in CycMuNet. Besides extensive experiments on benchmark datasets, we also compare our proposed CycMuNet (+) with S-VSR and T-VSR tasks, demonstrating that our method significantly outperforms the state-of-the-art methods.
Store and Fetch Immediately: Everything Is All You Need for Space-Time Video Super-resolution
Authors: Mengshun Hu，Kui Jiang，Zhixiang Nie，Jiahuan Zhou，Zheng Wang
Date: 2023-06-26 00:00:00
Proceeding: Conference 2023 6
Tasks: All, Space-time Video Super-resolution, Super-Resolution, Video Super-Resolution

Existing space-time video super-resolution (ST-VSR) methods fail to achieve high-quality reconstruction since they fail to fully explore the spatial-temporal correlations, long-range components in particular. Although the recurrent structure for ST-VSR adopts bidirectional propagation to aggregate information from the entire video, collecting the temporal information between the past and future via one-stage representations inevitably loses the long-range relations. To alleviate the limitation, this paper proposes an immediate storeand-fetch network to promote long-range correlation learning, where the stored information from the past and future can be refetched to help the representation of the current frame. Specifically, the proposed network consists of two modules: a backward recurrent module (BRM) and a forward recurrent module (FRM). The former first performs backward inference from future to past, while storing future super-resolution (SR) information for each frame. Following that, the latter performs forward inference from past to future to super-resolve all frames, while storing past SR information for each frame. Since FRM inherits SR information from BRM, therefore, spatial and temporal information from the entire video sequence is immediately stored and fetched, which allows drastic improvement for ST-VSR. Extensive experiments both on ST-VSR and space video super-resolution (S-VSR) as well as time video super-resolution (T-VSR) have demonstrated the effectiveness of our proposed method over other state-of-the-art methods on public datasets. Code is available https://github.com/hhhhhumengshun/SFI-STVR