A Sentence-Level Visualization of Attention in Large Language Models (2025)

Abstract: We introduce SAVIS, a sentence-level attention visualization tool that enhances the interpretability of long documents processed by Large Language Models (LLMs). By computing inter-sentence attention (ISA) through token-level attention aggregation, SAVIS reduces the complexity of attention analysis, enabling users to identify meaningful document-level patterns. The tool offers an interactive interface for exploring how sentences relate to each other in model processing. Our comparative analysis with existing visualization tools demonstrates that SAVIS improves task accuracy and reduces error identification time. We demonstrate its effectiveness for text analysis applications through case studies on various analysis tasks. Our open-source tool is available at https://pypi.org/project/savis with a screencast video at https://youtu.be/fTZZPHA55So.

## introduction attention mechanisms in language models enable interpretation of how models process text by showing weights assigned to different input elements @xcite . recent large language models (llms) like @xcite and palm 2 @xcite have achieved stateof-the-art performance across various natural language processing (nlp) tasks, using these attention mechanisms as core components. however, interpreting attention patterns becomes increasingly challenging as documents grow longer, particularly when analyzing how models process documentlevel context. the challenge of interpretation scales with both model size and input length. transformer-based models like bert contain 144 attention heads across 12 layers @xcite , each learning distinct patterns ranging from syntactic dependen-* corresponding author. cies to broader semantic relationships. for practitioners without deep nlp expertise, these patterns are complicated to interpret as they must examine hundreds of attention patterns simultaneously. studies show that only certain heads serve important functions, with one study finding that only 10 out of 48 encoder heads are sufficient to maintain translation quality in machine translation models @xcite . @xcite found that up to 60% of attention heads can be pruned without significantly impacting model performance. various approaches have been proposed to visualize attention in llms, from early attentionmatrix heatmaps @xcite to bipartite graph representations @xcite . more recent tools include bertviz @xcite for multi-scale visualization, lit @xcite and @xcite for interactive analysis, and knowl-edgevis @xcite for semantic exploration. however, these token-level visualization methods face a fundamental scalability challenge: for a document with n tokens across l layers and h attention heads per layer, practitioners must examine l × h × n 2 attention connections. to address this challenge, we introduce savis, an attention visualization tool that enhances the interpretability of long documents processed by llms. through case studies and comparative analysis, we demonstrate that our sentence-level approach significantly reduces the time required to identify attention patterns while maintaining interpretability. our contributions are (1) a method for aggregating token-level attention at the sentence level for a more straightforward interpretation of document-level patterns and (2) an interactive visualization tool for exploring sentence relationships, with features designed to reveal how models process sequential information. ## inter-sentence attention processing long documents with transformer-based models requires understanding the relationships between sentences. however, interpreting these relationships through token-level attention patterns is challenging due to their complexity. we introduce inter-sentence attention (isa), which captures sentence-level relationships by aggregating tokenlevel attention patterns. given two sentences s a and s b with token indices [i a , i a+1 ) and [i b , i b+1 ) respectively, we compute their inter-sentence attention through a threestep process. first, we integrate attention patterns across layers by taking the maximum attention score at each position: a(i, j) = max @xmath0, j), where α l (i, j) = softmax(q l,i k t l,j / √ d k )v l,j is the standard scaled dot-product attention at layer l. here, q l,i and k l,j are query and key vectors for tokens i and j in layer l, v l,j is the value vector, and d k is the dimension of key vectors. we then compute attention scores between sentence pairs by taking the maximum attention score between any token pair in two sentences: β h (s a , s b ) = max (i,j)∈sa×s b a(i, j) for each attention head h. finally, we aggregate these scores across attention heads by selecting the maximum value: isa(s a , s b ) = max @xmath1, s b ), where h is the set of attention heads. while prior work has used averaging across attention heads (abnar and zuidema, 2020), we opt for taking the maximum attention score to preserve strong signals from individual heads. this choice is motivated by findings that different attention heads often specialize in capturing specific linguistic patterns @xcite . this aggregation approach first integrates attention patterns across layers and then computes sentence-level attention by identifying relationships between sentence pairs. the process reduces the computational complexity from o(n 2 ) to o(m 2 ) for a document with n tokens and m sentences, where n ≫ m since each sentence typically consists of multiple tokens. this reduction enables analysis of document-level attention patterns while maintaining the most salient connections between sentences. ## case studies to evaluate savis's effectiveness in real-world scenarios, we conducted two case studies with public relations (pr) practitioners. using the gemma 7b model @xcite , we analyzed how pr practitioners utilized our tool to improve their analysis of public communications. ## comparative analysis we evaluated savis against bertviz @xcite , @xcite , and a baseline approach without visualization. twenty participants analyzed the nepa public comments (@xmath5. using savis, participants identified more errors (nepa: 7.2, amazon: 7.8) than with other tools, except when using the combination of savis+bertviz (nepa: 8.8, amazon: 9.0). participants also identified errors faster with savis (nepa: 7.9 minutes, amazon: 7.4 minutes) compared to bertviz (nepa: 11.5 minutes, amazon: 11.6 minutes) and other approaches. they needed fewer prompt revisions with savis (nepa: 2.3, amazon: 3.4) and achieved higher accuracy. starting from the initial values (nepa: 77.5%, amazon: 82.5%), participants using savis improved accuracy to 88.5% (+11 percentage points (pp)) for nepa and 84.95% (+2.45pp) for amazon. those using savis+bertviz showed slightly higher improvements (nepa: 91.75%, +14.25pp; amazon: 87.72%, +5.22pp). dodrio and bertviz also enabled improvements, while the baseline approach had minimal improvement for both datasets. figure 5 shows performance over time and across revisions. participants using savis and savis+bertviz detected errors faster, needed fewer revisions, and reached peak accuracy in fewer iterations compared to other approaches. ## limitations and future work while our visualization tool demonstrates effectiveness for pr tasks, several limitations should be acknowledged. our approach has been primarily evaluated with a single model architecture, and testing across diverse architectures would be necessary to confirm broader applicability. while savis's sentence-level visualization effectively enables pr practitioners to identify patterns quickly, token-level visualizations can complement it for detailed analysis. our comparative analysis shows that while savis achieves strong performance independently, combining it with token-level visualizations like bertviz provided additional benefits, suggesting potential value in multi-level visualization approaches. for future work, we plan to develop a unified interface that dynamically combines sentence-level and token-level views, allowing users to begin with simplified patterns and drill down to details where needed. we aim to explore adaptive visualization techniques that automatically suggest appropriate levels of detail for different analysis tasks. ## conclusion this paper introduced savis, a novel visualization tool for interpreting llm text generation at the sentence level. we defined inter-sentence attention (isa) to analyze sentence relationships and implemented a visualization approach combining interactive features and attention patterns. savis is open-source and available at https://pypi.org/ project/savis under mit license. our case studies on nepa public comments and amazon product reviews demonstrated savis's effectiveness in identifying and addressing llm output inaccuracies, particularly in handling complex linguistic structures such as reversed conditions. comparative analysis validated our approach, showing that practitioners using savis achieved faster error detection and higher accuracy with fewer revisions than existing tools. this approach offers practitioners an efficient tool for understanding llm behavior in various text analysis tasks.