T ext2 T raj2 T ext: Learning-by-Synthesis Framework for Contextual Captioning of Human Movement Trajectories (2024)

Abstract: This paper presents Text2Traj2Text, a novel learning-by-synthesis framework for captioning possible contexts behind shopper’s trajectory data in retail stores. Our work will impact various retail applications that need better customer understanding, such as targeted advertising and inventory management. The key idea is leveraging large language models to synthesize a diverse and realistic collection of contextual captions as well as the corresponding movement trajectories on a store map. Despite learned from fully synthesized data, the captioning model can generalize well to trajectories/captions created by real human subjects. Our systematic evaluation confirmed the effectiveness of the proposed framework over competitive approaches in terms of ROUGE and BERT Score metrics.

## introduction retail is an essential industry that is closely tied to our daily lives. imagine a customer visiting a supermarket. the customer first goes to the fruit section and compares various products. next, they go to the fish section, where they compare two products. afterward, they pass by the processed food section and head to the checkout, purchasing discounted organic strawberries and fish. from these movements and purchases, one can guess that "the customer is budget-conscious and interested in healthy meals." such profiling and verbalization of possible contexts behind shopping behaviors is vital for retailers to improve customer understanding and customer experience. we are interested in automating this intelligent activity, with recent advances in largescale language modeling. doing so would help facilitate and scale up retailer's operations beyond the number of experts, and can also enhance several applications such as targeted advertising @xcite ## text2traj2text fig. 2 illustrates the overview of the proposed framework, text2traj2text, which consists of text2traj data synthesis phase and traj2text model fine-tuning phase. ## experiments we conducted systematic experiments to evaluate the effectiveness of the text2traj2text framework. through the experiments, we aim to answer the following questions: baseline models. we compared our captioning model against the following baselines: (a) t5-small and t5-base @xcite fine-tuned without paraphrasing-based data augmentation; evaluation metrics. we employed rouge (r-1, r-2, r-l) @xcite and bert score (bsprecision, recall, f1 score) @xcite ) as evaluation metrics. rouge score captures lexical overlap by comparing n-grams and word sequences between generated and reference texts, while bert score, which utilizes bert embeddings, measures semantic similarity. ## related work human movement analysis. studies on human movements can be found in various research contexts, such as urban engineering @xcite , traffic simulation @xcite , autonomous driving @xcite , tourism @xcite , and public health @xcite . concrete techniques include pattern mining @xcite , semantic mining @xcite , trajectory prediction @xcite , and crowd analysis @xcite . compared to these prior arts, our work is the first to explore the potential of recent progress in large language modeling to empower human movement analysis and its application to retail scenarios. ## conclusion we presented a new task named contextual captioning of human movement trajectories, and a dedicated learning-by-synthesis framework, i.e., text2traj2text, with a particular focus on retail scenarios. we leverage llms to synthesize realistic and diverse collection of contextual captions as well as concrete trajectories on store maps. our captioning model fine-tuned on these synthesized data demonstrated equal or even better performance than existing llms with a higher number of parameters. moreover, the model well generalizes to human-created trajectories and captions. although this work focused exclusively on retail scenarios, we believe that the proposed task and framework would open up a new opportunity for adopting neural language generation techniques to various applications that need automated human activity understanding. this also raises new technical challenges such as effective encoding of very long trajectory data as input to language models and efficient inference of learned models to enable online captioning.