LAWCAT : Efficient Distillation from Quadratic to Linear Attention with Convolution across Tokens for Long Context Modeling (2025)

Abstract: Although transformer architectures have achieved state-of-the-art performance across diverse domains, their quadratic computational complexity with respect to sequence length remains a significant bottleneck, particularly for latency-sensitive long-context applications. While recent linear-complexity alternatives are increasingly powerful, effectively training them from scratch is still resource-intensive. To overcome these limitations, we propose LAWCAT (Linear Attention with Convolution Across Time), a novel linearization framework designed to efficiently transfer the capabilities of pretrained transformers into a performant linear attention architecture. LAWCAT integrates causal Conv1D layers to enhance local dependency modeling and employs normalized gated linear attention to improve generalization across varying context lengths. Our comprehensive evaluations demonstrate that, distilling Mistral-7B with only 1K-length sequences yields over 90% passkey retrieval accuracy up to 22K tokens, significantly extending its effective context window. Similarly, Llama3.2-1B LAWCAT variant achieves competitive performance on S-NIAH 1&2&3 tasks (1K-8K context length) and BABILong benchmark (QA2&QA3, 0K-16K context length), requiring less than 0.1% pre-training tokens compared with pre-training models. Furthermore, LAWCAT exhibits faster prefill speeds than FlashAttention-2 for sequences exceeding 8K tokens. LAWCAT thus provides an efficient pathway to high-performance, long-context linear models suitable for edge deployment, reducing reliance on extensive long-sequence training data and computational resources.

## introduction the widespread adoption and significant success of transformer-based architectures have driven remarkable advancements across various natural language processing (nlp) @xcite @xcite , reasoning @xcite , and computer vision (cv) tasks @xcite . transformers leverage self-attention mechanisms to effectively model long-range dependencies, achieving state-of-the-art (sota) performance on diverse benchmarks @xcite . however, their quadratic computational complexity with respect to sequence length remains a critical bottleneck, limiting their applicability to latency-sensitive edge based applications that are constrained in terms of training data and cost. to address this limitation, modern recurrent models have emerged as efficient alternatives, approximating or reformulating attention mechanisms to achieve linear complexity. recent studies, such as mamba-2 @xcite , gated linear attention (gla) @xcite , and hierarchically gated linear rnn2 (hgrn2) @xcite , demonstrate the feasibility of state update mechanisms that substantially reduce computational costs while maintaining competitive performance. despite these advances, training these models from scratch often requires substantial computational resources and extensive data, hindering rapid development and experimentation. on the other hand, knowledge distillation, a powerful technique for transferring knowledge from a large, pretrained teacher model to a smaller and more efficient student model, offers an effective strategy to alleviate these challenges. recently, some research like mambainllama @xcite and lolcats @xcite directly distill transformer models into recurrent architectures. however, mambainllama still requires extensive training data (∼20b tokens) and retains 50% self-attention layers to achieve strong performance up to 38k tokens, and lolcats exhibits limited generalization to sequences significantly longer than those seen during distillation. to address this problem, we introduce lawcat (linear attention with convolution across time), a novel distillation framework that efficiently transfers and enhances the rich representational capabilities of pre-trained transformer models into modern linear attention models. lawcat is particularly well-suited for edge applications that require longcontext modeling under tight computational constraints with limited training data. lawcat integrates a depth-separable convolution module, consisting of a depth-wise convolution layer followed by a linear layer (similar to mobilenet (howard, 2017), but we use causal conv1d across tokens), and a gated linear attention with normalization to effectively bridge the gap between quadratic and linear attention mechanisms. the main contributions of this paper are summarized as follows: self-attention the majority of the powerful llm models like llama 3 @xcite , mis-tral @xcite , and phi 4 (abdin et al., 2024) adopt the multi-head scaled dot-product softmax attention (sdpa). given a query q @xmath0, keys {k i } t @xmath1, and values {v i } t @xmath2, a single-head sdpa computes output o t as (omitting the scaling factor √ d for simplicity): this can be equivalently expressed as o @xmath3, where w ti denotes the attention weight assigned to value v i at output position t. the softmax in the numerator/denominator ensures context-dependent normalization -each weight w ti depends not only on the query-key similarity q t k t i but also on all other similarities {q t k t j } @xmath4. this coupling gives softmax attention a powerful ability to focus on a few relevant tokens while normalizing out less relevant ones, but also yields the o(n 2 ) computation complexity with respect to the sequence length n . kernel-based linear attention to mitigate this quadratic computation complexity, many linear attention methods aim to find a kernel function whose structure approximates attention scores through inner products of transformed query and key representations. assuming the kernel κ is positive semidefinite (psd), then there exists a feature mapping ϕ satisfying: κ(q, k) = ϕ(q)ϕ(k) t thus, the attention output at position t can be computed as: similar to the eq. 1, we can simplify is as o @xmath5, and wti can be regarded as the attention weights of the linear attention. this is analogous to softmax attention but uses ϕ(q)ϕ(k) t to approxiamte exp(qk t ) for all queries/keys, at least in expectation. if ϕ can perfectly realize exp as an inner product in a higher-dimensional feature space, then the linear attention exactly equals softmax attention. in practice, ϕ might be a simple nonlinear function such as 1 + elu @xcite , or a positive random feature map as in the performer @xcite ) that aims to fit the exponential. such linear attention schemes run in o(n ) time, but still suffer a noticeable performance gap between transformer models. ## algorithmic motivation a fundamental distinction between softmax attention and its linear approximations lies in the normalization mechanism and the role of the exponential function. softmax attention weights, {w ti ∝ exp(q t k t i )}, are normalized by a sum over all keys for a given query. this inherently creates a competitive dynamic: the exp function sharply amplifies scores for keys highly similar to the query relative to others, effectively focusing attention mass on the most relevant tokens @xcite . although self-attention is renowned for modeling global dependencies, much of its effectiveness also stems from capturing local context (e.g., adjacent words or neighboring pixels). the softmax mechanism excels here as its exponential weighting amplifies locally coherent patterns (like increasing the relevance of "new york" when queried with "york"), effectively capturing correlations abundant in natural sequences. empirical evidence @xcite , including the disproportionate performance drop observed when masking local versus random tokens, further demonstrates that softmax models leverage local context more effectively than typical linear variants. in contrast, linear attention often employs feature maps ϕ applied independently to queries and keys, yielding attention weights wti ∝ ϕ(q t )ϕ(k i ) t . the normalization term j ϕ(q t )ϕ(k j ) t simply aggregates transformed key features without the strong competitive effect of the softmax denominator. this can lead to limitations such as attention dilution or oversmoothing, where attention distributions become less peaked @xcite . more critically, this formulation struggles to replicate the strong local modeling capabilities inherent in softmax attention @xcite . this weaker local modeling is a key drawback of naive linear attention. meanwhile, the potentially low-rank nature of the ϕ(q)ϕ(k) t kernel can make it insensitive to crucial local variations and contextual nuances. two distinct queries q and q ′ , even if originating from different local contexts, might yield similar ϕ(q) ≈ ϕ(q ′ ) and thus nearly identical attention patterns, a failure mode less likely in softmax due to its context-sensitive normalization. ## conclusions we introduced lawcat (linear attention with convolution across time), a novel and efficient distillation framework that strategically integrates a causal conv1d layer to bolster local dependency modeling and employs a normalized gated linear attention mechanism for robust generalization across varying sequence lengths. our comprehensive experiments validate the effectiveness of lawcat. distilling models like mistral v0.1 7b and llama3.2 1b yielded linear attention variants capable of remarkable long-context performance, achieving over 90% accuracy on passkey retrieval tasks up to 22k tokens and matching sota recurrent models on niah benchmarks, all while using minimal training data and shorter sequence lengths during distillation. furthermore, the resulting lawcat models demonstrate practical efficiency gains, exhibiting faster prefill latency than highly optimized flashattention-2 implemen-tations for sequences exceeding 8k tokens. lawcat offers a practical and resourceefficient pathway for developing and deploying long-context models by leveraging existing pretrained transformers. this approach helps democratize access to moderately long-sequence modeling capabilities, particularly beneficial for resourceconstrained edge environments or rapid prototyping. furthermore, due to its causal and linear nature, lawcat is particularly attractive for streaming inference, opening new opportunities for lowlatency, on-device reasoning.