S ign2 V is: Automated Data Visualization from Sign Language (2025)

Abstract: Data visualizations, such as bar charts and histograms, are essential for analyzing and exploring data, enabling the effective communication of insights. While existing methods have been proposed to translate natural language descriptions into visualization queries, they focus solely on spoken languages, overlooking sign languages, which comprise about 200 variants used by 70 million Deaf and Hard-of-Hearing (DHH) individuals. To fill this gap, this paper proposes Sign2Vis, a sign language interface that enables the DHH community to engage more fully with data analysis. We first construct a paired dataset that includes sign language pose videos and their corresponding visualization queries. Using this dataset, we evaluate a variety of models, including both pipeline-based and end-to-end approaches. Extensive experiments, along with a user study involving 15 participants, demonstrate the effectiveness of Sign2Vis. Finally, we share key insights from our evaluation and highlight the need for more accessible and user-centered tools to support the DHH community in interactive data analytics.

## sign2vis : task formulation the sign2vis is formulated as a task of sequenceto-sequence learning, where the source sequence is the sign-language pose stream and the target sequence is the vql query. suppose that we have a sign-language corpus of n instances, i.e., the sign2vis problem can be formulated as follows: given a pair of sign-language pose video v , as well the table schema s, the goal is to learn a model f that can map the input {v, s} into a vql query q, i.e., @xmath0, s). ## sign2vis: the dataset to the best of our knowledge, the study of translation between sign language and visualization queries remains unexplored, and currently, no paired dataset of signs and visualization queries is available. therefore, it is imperative for us to construct a sign2vis dataset to facilitate further research in this domain. ## sign2visnet: a reference approach figure 4 gives an overview of our proposed sign2visnet, which is mainly based on the transformer network @xcite . ## discussion in this paper, we formulate and investigate a new but important problem of sign2vis. it opens a new direction to synthesize visualization queries from sign language, as a complementary interface for interacting with the database. our work aims to design a novel human-centered interface for a large number of dhh individuals to perform data visualization. in addition to the impact on the nlp community, our work also has impacts on other related communities, including the communities of human-computer interaction, visualization, database, computer vision, and programming languages. to the communities of human-computer interaction, visualization and database, our work can inspire researchers from these communities to design better human-centered interfaces to cater to more people who are physically disabled. to the communities of computer vision, and programming languages, the created dataset and built benchmark can facilitate researchers from these communities to design better encoder networks to understand sign language and design better generators to synthesize visualization queries with higher quality. ## conclusion in this paper, we have identified a new research problem of sign2vis to aid dhh individuals in data visualization, opening a path for synthesizing vql queries from sign language as a complementary interface. we present a parallel dataset of sign language pose videos and vql queries to support model development, along with a benchmark system featuring an end-to-end model and pipeline-based baselines for sign2vis. our work lays the groundwork for improving the accessibility of interactive data analytics for individuals with hearing impairments. we believe that our work will have a broader impact on communities of database, human-computer interaction, and programming languages. we also expect that our exploratory study on sign2vis would inspire further research on this topic.