Location-Aware Visual Question Generation with Lightweight Models (2023)

Abstract: This work introduces a novel task, location-aware visual question generation (LocaVQG), which aims to generate engaging questions from data relevant to a particular geographical location. Specifically, we represent such location-aware information with surrounding images and a GPS coordinate. To tackle this task, we present a dataset generation pipeline that leverages GPT-4 to produce diverse and sophisticated questions. Then, we aim to learn a lightweight model that can address the LocaVQG task and fit on an edge device, such as a mobile phone. To this end, we propose a method which can reliably generate engaging questions from location-aware information. Our proposed method outperforms baselines regarding human evaluation (e.g., engagement, grounding, coherence) and automatic evaluation metrics (e.g., BERTScore, ROUGE-2). Moreover, we conduct extensive ablation studies to justify our proposed techniques for both generating the dataset and solving the task.

## introduction driving is an integral part of our daily routines, playing a significant role in our lives. whether commuting to work, running errands, or embarking on exciting adventures, we heavily rely on automobiles to get us from one place to another. despite its undeniable convenience, driving requires constant focus on the road, the need to remain alert, and the mental strain of navigating through traffic. hence, staying behind the wheel after long working hours or during a long-distance trip can give rise to hazardous circumstances. to combat this, passengers often engage in conversation to keep the driver awake and attentive. project page can be found at https://github.com/ academiasinicanlplab/locavqg streetview images north east south west gps coordinate 40°75'79.05"n -73°95'70.14"w ## locavqg: location-aware visual question generation we introduce a novel task, location-aware visual question generation (locavqg). this section formally defines this task and describes how we collect data to construct the locavqg task tuples. ## generating locavqg dataset our goal is to train lightweight models to address the locavqg task. therefore, we aim to "label" the task tuples described in section 3. annotating the task tuples with engaging questions requires creativity and location-specific domain knowledge, which can be challenging for human annotators. in this work, we propose automatically generating questions from locavqg task tuples by leveraging the recent development of llms. an overview of the proposed dataset generation pipeline is depicted in figure 2 . ## learning and evaluating lightweight models we aim to train and evaluate lightweight models learning from the proposed locavqg dataset. ## conclusion in this work, we propose a novel task, locationaware visual question generation (locavqg), which aims to generate engaging questions from data relevant to a particular geographical location. specifically, we represent location-aware information using four directional street view images and a gps coordinate. to address this task, we introduce a dataset generation pipeline that leverages the recent advances of large language models (i.e.,