Multimodal approaches where interactive visualization and natural language are used in tandem are emerging as promising techniques for data analysis. A significant barrier in effectively designing such multimodal techniques is the lack of a systematicunderstanding of how people verbalize visual representations of data . Motivated by these gaps, this paper devises and applies a transferable, semi-automated methodology to systematically study the relation between visualization and natural language through two crowd-sourced experiments and natural language analysis. We describe these experiments, analyze the resulting corpus of utterances with natural language processing techniques and derive an empirically supported semantic lexicon for aligning visualizations and verbalizations of data. Our results indicate a wide range of vocabulary used to describe visualizations and led to a number of high level concepts to categorize the space of words and related utterances . We discuss how our findings can be used for natural language generation, also reflecting on the design of the experiments and the semi-automated methodology used in the analysis. We discuss further research directions and argue for a role for such multimodal experiments in advancing our understanding of how people work with visualizations and also data at large .
Natural language and visualization are being increasingly deployed together for supporting data analysis in different ways, from multimodal interaction to enriched data summaries and insights. Yet, researchers still lack systematic, foundational knowledge on how people relate visual and textual information in data analysis contexts. We describe two studies aimed at developing a broader understanding about the relevant characteristics of data and charts when people read and reason about data visualizations. The first study asks participants to verbalize what they see in scatterplots that depict various levels of correlations. The second study then asks participants to choose visualizations that, in their interpretation, match verbal descriptions of correlation. Using natural language processing techniques and qualitative analysis, we extract key URLanize them in a taxonomy and analyze the responses. We observe that responses involve multiple themes with distinct occurrence profiles across scatterplots, and a comparison between studies reveals similarities and differences between the processes of producing and interpreting the verbal descriptions . We discuss how the results could inform the design of multimodal representations aligned with the data and analytical tasks, and present a research roadmap on the expansion of the taxonomy to deepen our understanding on the combination of visualizations and natural language .
R
Multimodal approaches where interactive visualization and natural language are used in tandem are emerging as promising techniques for data analysis. A significant barrier in effectively designing such multimodal techniques is the lack of a systematicunderstanding of how people verbalize visual representations of data . Motivated by these gaps, this paper devises and applies a transferable, semi-automated methodology to systematically study the relation between visualization and natural language through two crowd-sourced experiments and natural language analysis. We describe these experiments, analyze the resulting corpus of utterances with
Natural language and visualization are being increasingly deployed together for supporting data analysis in different ways, from multimodal interaction to enriched data summaries and insights. Yet, researchers still lack systematic, foundational knowledge on how people relate visual and textual information in data analysis contexts. We describe two studies aimed at developing a broader understanding about the relevant characteristics of data and charts when people read and reason about data visualizations. The first study asks participants to verbalize what they see in scatterplots that depict various levels of correlations. The second study then asks participants to choose visualizations that, in their interpretation, match verbal descriptions of correlation. Using
R
derive an empirically supported semantic lexicon for aligning visualizations and verbalizations of data. Our results indicate a wide range of vocabulary used to describe visualizations and led to a number of high level concepts to categorize the space of words and related utterances
qualitative analysis, we extract key URLanize them in a taxonomy and analyze the responses. We observe that responses involve multiple themes with distinct occurrence profiles across scatterplots, and a comparison between studies reveals similarities and differences between the processes of producing and interpreting the verbal descriptions
D
our findings can be used for natural language generation, also reflecting on the design of
R
experiments and the semi-automated methodology used in the analysis. We discuss further research directions and argue for a role for such multimodal experiments in advancing our understanding of how people work with visualizations and also data at large
results could inform the design of multimodal representations aligned with the data and analytical tasks, and present a research roadmap on the expansion of the taxonomy to deepen our understanding on the combination of visualizations and natural language