pwc-archive/papers-with-abstracts huggingface dataset:
LoGoPrompt: Synthetic Text Images Can Be Good Visual Prompts for Vision-Language Models
Authors: Cheng Shi, Sibei Yang
Date: 2023-09-03 00:00:00
Proceeding: ICCV 2023 1
Tasks: Domain Generalization, Few-Shot Learning, Prompt Engineering, Prompt Learning, Visual Prompt Tuning

Prompt engineering is a powerful tool used to enhance the performance of pre-trained models on downstream tasks. For example, providing the prompt "Let's think step by step" improved GPT-3's reasoning accuracy to 63% on MutiArith while prompting "a photo of" filled with a class name enables CLIP to achieve $80$\% zero-shot accuracy on ImageNet. While previous research has explored prompt learning for the visual modality, analyzing what constitutes a good visual prompt specifically for image recognition is limited. In addition, existing visual prompt tuning methods' generalization ability is worse than text-only prompting tuning. This paper explores our key insight: synthetic text images are good visual prompts for vision-language models! To achieve that, we propose our LoGoPrompt, which reformulates the classification objective to the visual prompt selection and addresses the chicken-and-egg challenge of first adding synthetic text images as class-wise visual prompts or predicting the class first. Without any trainable visual prompt parameters, experimental results on 16 datasets demonstrate that our method consistently outperforms state-of-the-art methods in few-shot learning, base-to-new generalization, and domain generalization.
EdaDet: Open-Vocabulary Object Detection Using Early Dense Alignment
Authors: Cheng Shi, Sibei Yang
Date: 2023-09-03 00:00:00
Proceeding: ICCV 2023 1
Tasks: Object, object-detection, Object Detection, Open-vocabulary object detection, Open Vocabulary Object Detection, Zero-Shot Learning

Vision-language models such as CLIP have boosted the performance of open-vocabulary object detection, where the detector is trained on base categories but required to detect novel categories. Existing methods leverage CLIP's strong zero-shot recognition ability to align object-level embeddings with textual embeddings of categories. However, we observe that using CLIP for object-level alignment results in overfitting to base categories, i.e., novel categories most similar to base categories have particularly poor performance as they are recognized as similar base categories. In this paper, we first identify that the loss of critical fine-grained local image semantics hinders existing methods from attaining strong base-to-novel generalization. Then, we propose Early Dense Alignment (EDA) to bridge the gap between generalizable local semantics and object-level prediction. In EDA, we use object-level supervision to learn the dense-level rather than object-level alignment to maintain the local fine-grained semantics. Extensive experiments demonstrate our superior performance to competing approaches under the same strict setting and without using external training resources, i.e., improving the +8.4% novel box AP50 on COCO and +3.9% rare mask AP on LVIS.
Tight Bounds for Machine Unlearning via Differential Privacy
Authors: Yiyang Huang, Clément L. Canonne
Date: 2023-09-02 00:00:00
Tasks: Machine Unlearning

We consider the formulation of "machine unlearning" of Sekhari, Acharya, Kamath, and Suresh (NeurIPS 2021), which formalizes the so-called "right to be forgotten" by requiring that a trained model, upon request, should be able to "unlearn" a number of points from the training data, as if they had never been included in the first place. Sekhari et al. established some positive and negative results about the number of data points that can be successfully unlearnt by a trained model without impacting the model's accuracy (the "deletion capacity"), showing that machine unlearning could be achieved by using differentially private (DP) algorithms. However, their results left open a gap between upper and lower bounds on the deletion capacity of these algorithms: our work fully closes this gap, obtaining tight bounds on the deletion capacity achievable by DP-based machine unlearning algorithms.
Storytelling from an Image Stream Using Scene Graphs
Authors: Ruize Wang, Zhongyu Wei, Piji Li, Qi Zhang, Xuanjing Huang
Date: 2020-04-03 00:00:00
Proceeding: The Thirty-Fourth AAAI Conference on Artificial Intelligence 2020 4
Tasks: Story Generation, Visual Storytelling

Visual storytelling aims at generating a story from an image stream. Most existing methods tend to represent images directly with the extracted high-level features, which is not intuitive and difficult to interpret. We argue that translating each image into a graph-based semantic representation, i.e., scene graph, which explicitly encodes the objects and relationships detected within image, would benefit representing and describing images. To this end, we propose a novel graph-based architecture for visual storytelling by modeling the two-level relationships on scene graphs. In particular, on the within-image level, we employ a Graph Convolution Network (GCN) to enrich local fine-grained region representations of objects on scene graphs. To further model the interaction among images, on the cross-images level, a Temporal Convolution Network (TCN) is utilized to refine the region representations along the temporal dimension. Then the relation-aware representations are fed into the Gated Recurrent Unit (GRU) with attention mechanism for story generation. Experiments are conducted on the public visual storytelling dataset. Automatic and human evaluation results indicate that our method achieves state-of-the-art.
Two Heads are Better Than One: Hypergraph-Enhanced Graph Reasoning for Visual Event Ratiocination
Authors: Wenbo Zheng, Lan Yan, Chao Gou, Fei-Yue Wang
Date: 2021-07-18 00:00:00
Proceeding: International Conference on Machine Learning 2021 7
Tasks: Visual Storytelling

Even with a still image, humans can ratiocinate various visual cause-and-effect descriptions before, at present, and after, as well as beyond the given image. However, it is challenging for models to achieve such task–the visual event ratiocination, owing to the limitations of time and space. To this end, we propose a novel multi-modal model, Hypergraph-Enhanced Graph Reasoning. First it represents the contents from the same modality as a semantic graph and mines the intra-modality relationship, therefore breaking the limitations in the spatial domain. Then, we introduce the Graph Self-Attention Enhancement. On the one hand, this enables semantic graph representations from different modalities to enhance each other and captures the inter-modality relationship along the line. On the other hand, it utilizes our built multi-modal hypergraphs in different moments to boost individual semantic graph representations, and breaks the limitations in the temporal domain. Our method illustrates the case of "two heads are better than one" in the sense that semantic graph representations with the help of the proposed enhancement mechanism are more robust than those without. Finally, we re-project these representations and leverage their outcomes to generate textual cause-and-effect descriptions. Experimental results show that our model achieves significantly higher performance in comparison with other state-of-the-arts.
Imagine, Reason and Write: Visual Storytelling with Graph Knowledge and Relational Reasoning
Authors: Chunpu Xu, Min Yang, Chengming Li, Ying Shen, Xiang Ao, and Ruifeng Xu
Date: 2021-05-18 00:00:00
Proceeding: The Thirty-Fifth AAAI Conference on Artificial Intelligence 2021 5
Tasks: Diversity, Informativeness, Relational Reasoning, Visual Storytelling

Visual storytelling is a task of creating a short story based on photo streams. Different from visual captions, stories contain not only factual descriptions, but also imaginary concepts that do not appear in the images. In this paper, we propose a novel imagine-reason-write generation framework (IRW) for visual storytelling, inspired by the logic of humans when they write the story. First, an imagine module is leveraged to learn the imaginative storyline explicitly, improving the coherence and reasonability of the generated story. Second, we employ a reason module to fully exploit the external knowledge (commonsense knowledge base) and task-specific knowledge (scene graph and event graph) with relational reasoning method based on the storyline. In this way, we can effectively capture the most informative commonsense and visual relationships among objects in images, which enhances the diversity and informativeness of the generated story. Finally, we integrate the imaginary concepts and relational knowledge to generate human-like story based on the original semantics of images. Extensive experiments on a benchmark dataset (i.e., VIST) demonstrate that the proposed IRW framework significantly outperforms the state-of-the-art methods across multiple evaluation metrics.
AOG-LSTM: An adaptive attention neural network for visual storytelling
Authors: Hanqing Liu, Jiacheng Yang, Chia-Hao Chang, Wei Wang, Hai-Tao Zheng, Yong Jiang, Hui Wang, Rui Xie, and Wei Wu
Date: 2023-06-26 00:00:00
Proceeding: Neurocomputing 2023 6
Tasks: Decoder, Visual Storytelling

Visual storytelling is the task of generating a related story for a given image sequence, which has received significant attention. However, using general RNNs (such as LSTM and GRU) as the decoder limit the performance of the models in this task. This is because they can not differentiate different types of information representations. In addition, optimizing the probabilities of subsequent words conditioned on the previous ground-truth sequences can cause error accumulation during inference. Moreover, the existing method of alleviating error accumulation based on replacing reference words does not take into account the different effects of each word. To address the above problems, we propose a modified neural network named AOG-LSTM and a modified training strategy named ARS, respectively. AOG-LSTM can adaptatively pay appropriate attention to different information representations within it when predicting different words. During training, ARS replaces some words in the reference sentences with model predictions similar to the existing method. However, we utilize the selection network and selection strategy to select more appropriate words for the replacement to better improve the model. Experiments on the VIST Dataset demonstrate that our model outperforms several strong baselines on the most commonly used metrics.
Anytime Active Learning
Authors: Maria E. Ramirez-Loaiza, Aron Culotta, Mustafa Bilgic
Date: 2014-07-27 00:00:00
Proceeding: AAAI - Association for the Advancement of Artificial Intelligence 2014 7
Tasks: Active Learning, Document Classification, Text Classification

A common bottleneck in deploying supervised learning systems is collecting human-annotated examples. In many domains, annotators form an opinion about the label of an example incrementally--e.g., each additional word read from a document or each additional minute spent inspecting a video helps inform the annotation. In this paper, we investigate whether we can train learning systems more efficiently by requesting an annotation before inspection is fully complete--e.g., after reading only 25 words of a document. While doing so may reduce the overall annotation time, it also introduces the risk that the annotator might not be able to provide a label if interrupted too early. We propose an anytime active learning approach that optimizes the annotation time and response rate simultaneously. We conduct user studies on two document classification datasets and develop simulated annotators that mimic the users. Our simulated experiments show that anytime active learning outperforms several baselines on these two datasets. For example, with an annotation budget of one hour, training a classifier by annotating the first 25 words of each document reduces classification error by 17% over annotating the first 100 words of each document.
Coherent Visual Storytelling via Parallel Top-Down Visual and Topic Attention
Authors: Jinjing Gu, Hanli Wang
Date: 2022-08-17 00:00:00
Proceeding: IEEE Transactions on Circuits and Systems for Video Technology 2022 8
Tasks: Diversity, Sentence, Text Generation, Visual Storytelling

Visual storytelling aims at producing a narrative paragraph for a given photo album automatically. It introduces more new challenges than individual image paragraph descriptions, mainly due to the difficulty in preserving coherent topics and in generating diverse phrases to depict the rich content of a photo album. Existing attention-based models that lack higher-level guiding information always result in a deviation between the generated sentence and the topic expressed by the image. In addition, these widely applied language generation approaches employing standard beam search tend to produce monotonous descriptions. In this work, a coherent visual storytelling (CoVS) framework is designed to address the above-mentioned problems. Specifically, in the encoding phase, an image sequence encoder is designed to efficiently extract visual features of the input photo album. Then, the novel parallel top-down visual and topic attention (PTDVTA) decoder is constructed via a topic-aware neural network, a parallel top-down attention model, and a coherent language generator. Concretely, visual attention focuses on the attributes and the relationships of the objects, while topic attention integrating a topic-aware neural network could improve the coherence of generated sentences. Eventually, a phrase beam search algorithm with n -gram hamming diversity is further designed to optimize the expression diversity of the generated story. To justify the proposed CoVS framework, extensive experiments are conducted on the VIST dataset, which shows that CoVS can automatically generate coherent and diverse stories in a more natural way. Moreover, CoVS obtains better performance than state-of-the-art baselines on BLEU-4 and METEOR scores, while maintaining good CIDEr and ROUGH_L scores. The source code of this work can be found in https://mic.tongji.edu.cn .
SentiStory: A Multi-Layered Sentiment-Aware Generative Model for Visual Storytelling
Authors: Wei Chen, Xuefeng Liu, Jianwei Niu
Date: 2022-06-16 00:00:00
Proceeding: IEEE Transactions on Circuits and Systems for Video Technology 2022 6
Tasks: Visual Storytelling

The visual storytelling (VIST) task aims at generating reasonable, human-like and coherent stories with the image streams as input. Although many deep learning models have achieved promising results, most of them do not directly leverage the sentiment information of stories. In this paper, we propose a sentiment-aware generative model for VIST called SentiStory. The key of SentiStory is a multi-layered sentiment extraction module (MLSEM). For a given image stream, the higher layer gives coarse-grained but accurate sentiments, while the lower layer of the MLSEM extracts fine-grained but usually unreliable ones. The two layers are combined strategically to generate coherent and rich visual sentiment concepts for the VIST task. Results from both automatic and human evaluations demonstrate that with the help of the MLSEM, SentiStory achieves improvement in generating more coherent and human-like stories.
Visual Storytelling with Hierarchical BERT Semantic Guidance
Authors: Ruichao Fan, Hanli Wang, Jinjing Gu, and Xianhui Liu
Date: 2022-01-10 00:00:00
Proceeding: ACM Multimedia Asia 2022 1
Tasks: Sentence, Text Generation, Visual Storytelling

Visual storytelling, which aims at automatically producing a narrative paragraph for photo album, remains quite challenging due to the complexity and diversity of photo album content. In addition, open-domain photo albums cover a broad range of topics and this results in highly variable vocabularies and expression styles to describe photo albums. In this work, a novel teacher-student visual storytelling framework with hierarchical BERT semantic guidance (HBSG) is proposed to address the above-mentioned challenges. The proposed teacher module consists of two joint tasks, namely, word-level latent topic generation and semantic-guided sentence generation. The first task aims to predict the latent topic of the story. As there is no ground-truth topic information, a pre-trained BERT model based on visual contents and annotated stories is utilized to mine topics. Then the topic vector is distilled to a designed image-topic prediction model. In the semantic-guided sentence generation task, HBSG is introduced for two purposes. The first is to narrow down the language complexity across topics, where the co-attention decoder with vision and semantic is designed to leverage the latent topics to induce topic-related language models. The second is to employ sentence semantic as an online external linguistic knowledge teacher module. Finally, an auxiliary loss is devised to transform linguistic knowledge into the language generation model. Extensive experiments are performed to demonstrate the effectiveness of HBSG framework, which surpasses the state-of-the-art approaches evaluated on the VIST test set.
An Interval-Valued Time Series Forecasting Scheme With Probability Distribution Features for Electric Power Generation Prediction
Authors: TING-JEN CHANG, SAMANTHA LEE, Jonathan Lee, AND CHI-JIE LU
Date: 2022-01-11 00:00:00
Proceeding: journal 2022 1
Tasks: Descriptive, regression, Time Series, Time Series Forecasting

Developing an effective interval-valued time series (ITS) forecasting scheme for electric
power generation is an important issue for energy operators and governments when making energy strategic
decisions. The existing studies for ITS forecasting only consider basic descriptive information such as
center, radius, upper and lower bounds, and overlooks the distribution information within the data interval.
In this study, an interval-valued time series forecasting scheme based on probability distribution information
features of interval-valued data with machine learning algorithms is proposed to enhance electric power
generation forecasting. In the proposed scheme, the central tendency features and dispersion features from
the interval-valued data are designed as integrated features sets (IFS) and used as predictor variables.
Three methods including supper vector regression and extreme learning machine and multivariate adaptive
regression splines based on the IFS are utilized to develop ITS forecasting models. The daily time series
of the metered generation from the Australian Energy Market Operator is used to illustrate the proposed
scheme. Empirical results show that the proposed ITS forecasting schemes with IFS outperform the eight
benchmark models and thus validate that the proposed scheme is an effective alternative for interval-valued
electric power generation forecasting.
Temporal Attention Bottleneck is informative? Interpretability through Disentangled Generative Representations for Energy Time Series Disaggregation
Authors: Khalid Oublal, Said Ladjal, David Benhaiem, Emmanuel le-borgne, François Roueff
Date: 2023-06-23 00:00:00
Proceeding: ICML 2023 6
Tasks: Time Series

Generative models have garnered significant attention for their ability to address the challenge of source separation in disaggregation tasks. This approach holds promise for promoting energy conservation by enabling homeowners to obtain detailed information on their energy consumption solely through the analysis of aggregated load curves. Nevertheless, the model's ability to generalize and its interpretability remain two major challenges. To tackle these challenges, we deploy a generative model called TAB-VAE (Temporal Attention Bottleneck for Variational Auto-encoder), based on hierarchical architecture, addresses signature variability, and provides a robust, interpretable separation through the design of its informative representation of latent space. Our implementation and evaluation guidelines are available at https://github.com/oublalkhalid/TAB-VAE.
Bayesian topology inference on partially known networks from input-output pairs
Authors: Martín Sevilla, Santiago Segarra
Date: 2023-09-12 00:00:00

We propose a sampling algorithm to perform system identification from a set of input-output graph signal pairs. The dynamics of the systems we study are given by a partially known adjacency matrix and a generic parametric graph filter of unknown parameters. The methodology we employ is built upon the principles of annealed Langevin diffusion. This enables us to draw samples from the posterior distribution instead of following the classical approach of point estimation using maximum likelihood. We investigate how to harness the prior information inherent in a dataset of graphs of different sizes through the utilization of graph neural networks. We demonstrate, via numerical experiments involving both real-world and synthetic networks, that integrating prior knowledge into the estimation process enhances estimation performance.
Uncertainty-aware Traffic Prediction under Missing Data
Authors: Hao Mei, Junxian Li, Zhiming Liang, Guanjie Zheng, Bin Shi, Hua Wei
Date: 2023-09-13 00:00:00
Tasks: Decision Making, Graph Neural Network, Prediction, Traffic Prediction, Uncertainty Quantification

Traffic prediction is a crucial topic because of its broad scope of applications in the transportation domain. Recently, various studies have achieved promising results. However, most studies assume the prediction locations have complete or at least partial historical records and cannot be extended to non-historical recorded locations. In real-life scenarios, the deployment of sensors could be limited due to budget limitations and installation availability, which makes most current models not applicable. Though few pieces of literature tried to impute traffic states at the missing locations, these methods need the data simultaneously observed at the locations with sensors, making them not applicable to prediction tasks. Another drawback is the lack of measurement of uncertainty in prediction, making prior works unsuitable for risk-sensitive tasks or involving decision-making. To fill the gap, inspired by the previous inductive graph neural network, this work proposed an uncertainty-aware framework with the ability to 1) extend prediction to missing locations with no historical records and significantly extend spatial coverage of prediction locations while reducing deployment of sensors and 2) generate probabilistic prediction with uncertainty quantification to help the management of risk and decision making in the down-stream tasks. Through extensive experiments on real-life datasets, the result shows our method achieved promising results on prediction tasks, and the uncertainty quantification gives consistent results which highly correlated with the locations with and without historical data. We also show that our model could help support sensor deployment tasks in the transportation field to achieve higher accuracy with a limited sensor deployment budget.