Natural language and visualization are being increasingly deployed together for supporting data analysis in different ways, from multimodal interaction to enriched data summaries and insights. Yet, researchers still lack systematic , foundational knowledge on how people relate visual and textual information in data analysis contexts. We describe two studies aimed at developing a broader understanding about the relevant characteristics of data and charts when people read and reason about data visualizations . The first study asks participants to verbalize what they see in scatterplots that depict various levels of correlations. The second study then asks participants to choose visualizations that , in their interpretation, match verbal descriptions of correlation. Using natural language processing techniques and qualitative analysis, we extract key URLanize them in a taxonomy and analyze the  responses. We observe that responses involve multiple themes with distinct occurrence profiles across scatterplots, and a comparison between studies reveals similarities and differences between the processes of producing and interpreting the verbal descriptions . We discuss how the results could inform the design of multimodal representations aligned with the data and analytical tasks, and present a research roadmap on the expansion of the taxonomy to deepen our understanding on the combination of visualizations and natural language.
Natural language and visualization are being increasingly deployed together for supporting data analysis in different ways, from multimodal interaction to enriched data summaries and insights. Yet, researchers still lack systematic knowledge on how viewers verbalize their interpretations of visualizations, and how they interpret verbalizations of visualizations in such contexts. We describe two studies aimed at identifying characteristics of data and charts that are relevant in such tasks . The first study asks participants to verbalize what they see in scatterplots that depict various levels of correlations. The second study then asks participants to choose visualizations that match a given verbal description of correlation. We extract key concepts from URLanize them in a taxonomy and analyze the categorized responses. We observe that participants use a wide range of vocabulary across all scatterplots, but particular concepts are preferred for higher levels of correlation. A comparison between the studies reveals the ambiguity of some of the concepts . We discuss how the results could inform the design of multimodal representations aligned with the data and analytical tasks, and present a research roadmap to deepen the understanding about visualizations and natural language.
D
, foundational
R
people relate visual and textual information in data analysis
viewers verbalize their interpretations of visualizations, and how they interpret verbalizations of visualizations in such
R
developing a broader understanding about the relevant
identifying
R
when people read and reason about data visualizations
that are relevant in such tasks
R
, in their interpretation, match verbal descriptions
match a given verbal description
R
Using natural language processing techniques and qualitative analysis, we extract key
We extract key concepts from
A
categorized
R
responses involve multiple themes with distinct occurrence profiles across scatterplots, and a comparison between studies reveals similarities and differences between the processes of producing and interpreting the verbal descriptions
participants use a wide range of vocabulary across all scatterplots, but particular concepts are preferred for higher levels of correlation. A comparison between the studies reveals the ambiguity of some of the concepts
R
on the expansion of the taxonomy to deepen our understanding on the combination of
to deepen the understanding about