Generating Hashtags for Short-form Videos with Guided Signals (2023)

Abstract: Short-form video hashtag recommendation (SVHR) aims to recommend hashtags to content creators from videos and corresponding descriptions. Most prior studies regard SVHR as a classification or ranking problem and select hashtags from a set of limited candidates. However, in reality, users can create new hashtags, and trending hashtags change rapidly over time on social media. Both of these properties cannot be easily modeled with classification approaches. To bridge this gap, we formulate SVHR as a generation task that better represents how hashtags are created naturally. Additionally, we propose the Guided Generative Model (GGM) where we augment the input features by retrieving relevant hashtags from a large-scale hashtag pool as extra guidance signals. Experimental results on two short-form video datasets show that our generative models outperform strong classification baselines, and the guidance signals further boost the performance by 8.11 and 2.17 absolute ROUGE-1 scores on average, respectively. We also perform extensive analyses including human evaluation, demonstrating that our generative model can create meaningful and relevant novel hashtags while achieving state-of-the-art performance on known hashtags

## introduction short-form videos on social media are increasingly popular thanks to the proliferation of multimedia technologies and portable devices @xcite . to highlight the topics and salient information of the videos, hashtags -words or unspaced phrases prefixed with a "#" -have been widely used. proper use of hashtags can also increase the probability of the videos being discovered @xcite . in light of this, short-form video hashtag recommendation figure 1 : video frames and the video description are the inputs. the guided generative model (ggm) generates hashtags related to video frames (e.g., #winter and #cold-weather) as well as video description (e.g., #snowstorm and #toronto). the generated novel hashtag that never appears in the training set is highlighted. we use mock-up video frames in the paper. to fill this research gap, we formulate svhr as a generation task that better represents the process through which hashtags are created by content creators. figure 1 shows an example of the generation results that include a novel hashtag (#cold-weather). the generative model learns to generate hashtags related to video frames as well as video description. additionally, we propose to retrieve hashtags from a large hashtag pool to augment input features and use the retrieved hashtags to guide hashtag generation. inspired by the effectiveness of visionlanguage models (vlms) @xcite @xcite in video-text retrieval tasks, we construct our hashtag retriever based on vlm. then, we build a multimodal hashtag generator to generate hashtags from the retrieved hashtags, video frames, and user-written video descriptions. to leverage multimodal inputs, we introduce a cross-modal attention mechanism (cam) to fuse information from different modalities. we name the whole architecture guided generative model (ggm). we conduct experiments to evaluate strong classification baselines, generative models and the proposed ggm on two well-known short-form video datasets: sfvd1 and sfvd2. for the classification models, we regard svhr as a multi-label classification problem and compute probabilities over all hashtags that appear in training set by a softmax activation so the models can capture the interaction between the labels of each video. experimental results demonstrate that the generative models outperform the classification models, and the guidance signals further boost the performance by 8.11 and 2.17 rouge-1 scores on average. we further create an unseen test set @xcite for sfvd1 and analyze the models' performance on it. results show that generative models are able to generate unseen hashtags that the classification models can never predict. in addition, we assess the generated hashtags with human evaluation since the automatic metrics might underestimate our models' ability to create novel hashtags. the results from our human evaluations show that ggm is able to create meaningful novel hashtags that are statistically comparable to the ground truth hashtags. our contributions are summarized as follows: keyphrase and microblog hashtag generation. keyphrase generation (kpg) aims to generate phrases that highlight salient information for a piece of text. according to @xcite , existing kpg methods can be divided into one2one @xcite which generates one keyphrase at a time and one2seq @xcite which generates a sequence of keyphrases at once. in this work, we apply one2seq framework and randomly shuffle the target hashtags in each batch to mitigate the effect of order when fine-tuning the model. different from kpg, svhr requires the model to process multimodal inputs. recently, @xcite introduced the microblog hashtag generation task, which can be viewed as a variation of the kpg for the social media domain. additionally, topic-aware models @xcite and news articles @xcite are leveraged to improve hashtag generation. to our knowledge, we are the first to generate hashtags for short-form videos. retrieval-augmented generation. retrieval-augmented generation (rag) has been widely used in nlp tasks, such as neural machine transla- video description: there is a short [...] self attention feed forward audio feature cross attention bi self-att causal self-att cross attention feed forward feed forward cross attention n× n× start of sequence token: <@xmath0: #kitty #cat #white #kitties video encoder text decoder multimodal fusion multimodal fusion guidance signal: #cat #cats #kitten ... + vlm-based hashtag retriever text decoder n× figure 2: an overview of our proposed framework. the vlm-based hashtag retriever retrieves the relevant hashtags as extra features to guide the generative model. information from different modalities is fused by crossmodal multi-head attention. ## methodology in this section, we first introduce our problem definition of the svhr task. then, we present our guided generative model (ggm) which generates the hashtag from multimodal inputs. ## conclusion and future work in this paper, we formulate the short-form video hashtag recommendation (svhr) as a generation task, and we propose a guided generative model (ggm) that generates hashtags from multimodal inputs and guided signals from a vlm-based hashtag retriever. our work benchmarks classification and generative models on svhr datasets and highlights the advantage of using generative models. our ggm achieves state-of-the-art performance, and the human evaluation results show that ggm is able to generate meaningful novel hashtags compa-rable to ground-truth hashtags. we hope our work can catalyze research on using generative models for svhr. for future work, since the hashtag recommendation is a highly concurrent task in the real-world application, we believe improving computational efficiency to strike a balance between accuracy and efficiency is one of the important directions. besides, the popular trend of short-form videos changes rapidly on the internet, so it's important for systems to accurately generate hashtags for new trending videos.