Does Self-Attention Need Separate Weights in Transformers? (2025)

Abstract: Self-attention has revolutionized natural language processing by capturing long-range dependencies and improving context understanding. However, it comes with high computational costs and struggles with sequential data’s inherent directionality. This paper investigates and presents a simplified approach called “shared weight self-attention,” where a single weight matrix is used for Keys, Queries, and Values instead of separate matrices for each. This approach cuts training parameters by more than half and significantly reduces training time. Our method not only improves efficiency but also achieves strong performance on tasks from the GLUE benchmark, even outperforming the standard BERT baseline in handling noisy and out-of-domain data. Experimental results show a 66.53% reduction in parameter size within the attention block and competitive accuracy improvements of 3.55% and 0.89% over symmetric and pairwise attention-based BERT models, respectively.

## introduction natural language processing (nlp) has seen remarkable progress with the advent of transformerbased architectures @xcite . these models have revolutionized tasks such as machine translation @xcite , language modeling @xcite , and question answering @xcite , achieving better accuracy and performance. central to the success of these models is the self-attention mechanism @xcite , which allows them to weigh the importance of different words in a sentence dynamically. self-attention's main challenges include computational inefficiency with quadratic complexity, difficulty in handling long-term dependencies effectively, and the lack of inherent directionality in capturing sequential relationships. while the attention mechanism itself has been extensively investigated @xcite @xcite , and improvements in computational complexity have been proposed @xcite @xcite , the primary method retains the same architecture in using separate trainable weight matrices to compute keys, queries, and values, which leads to a high parameter count and significant complexity for computing attention. we would like to ask: "do we need the three weight matrix representations of (key, query, value) for learning self-attention scores?" to address this question, we revisit the concept of self-attention and propose a novel shared weight self-attention mechanism that employs a single weight matrix for all three representations to reduce the parameter size and the time and memory complexity. our shared weight matrix enables the model to efficiently capture the essential features needed for understanding semantics without the overhead of managing multiple matrices. the shared matrix is a regularization to capture the common weights learned from each representation. this simplification reduces the model's computational footprint, retains the ability to focus on relevant parts of the input data effectively, and enhances prediction generalization for noisy input and out-of-domain test data. in this work, we explore alternative compatibility functions within the self-attention mechanism of transformer-based encoder models, particularly bert @xcite . by utilizing a shared representation for (key, query, value), our approach achieves substantial improvements in efficiency while maintaining the model's performance without any compromise on accuracy. our contributions can be summarized as follows: ## ablation study parameter analysis: this study explores the efficiency of using shared weights in the self-attention mechanism. by implementing a shared transformation, s(x), along with separate diagonal matrices d q , d k , and d v for queries, keys, and values, the model requires fewer parameters, totaling (d 2 +3d). this setup results in a 66.53% reduction in parameters compared to the traditional (3d 2 ) needed by the standard self-attention in bert, as highlighted in ## related work the introduction of self-attention in transformer architecture in 2017 by @xcite marked a significant turning point by enabling models to dynamically concentrate on relevant parts of input sequences, building upon earlier work by @xcite , who applied attention mechanisms within recurrent neural networks (rnns) for machine translation and thus improved translation accuracy. according to @xcite , self-attention mechanisms were enhanced to better model complex data dependencies, which contributed to the development of more advanced attention models. subsequently, @xcite delved deeper into self-attention mechanisms, resulting in the creation of models like bert @xcite . this model utilized bidirectional training of transformers to capture context from both directions in a sequence, leading to state-of-the-art performance across a range of nlp tasks. reviews conducted by @xcite and @xcite highlighted the significant role of weighted dot-product attention in contemporary models. @xcite assessed the versatility of self-attention mechanisms in computer vision, parameter reduction 0% 33% 30-35% 33% 66.67% computational complexity high moderate high moderate low diagonal scaling matrices no no no no yes expressiveness high reduced q-k diversity enhanced (pairwise) moderate retained (via scaling) training speed baseline 10-15% faster 5-10% slower 10-15% faster 15-20% faster memory usage high moderate high moderate low implementation simplicity complex simple complex (u matrix) simple simplest our proposed method builds upon these advancements by utilizing a single shared weight matrix, w s , for a unified representation. keys, queries, and values are derived through diagonal matrix multiplication with specific vectors, resulting in a 66.53% reduction in parameters within the selfattention block. despite this significant reduction, our method maintains robust performance across bert configurations, demonstrating the potential for more efficient yet powerful nlp models. ## limitations our work mainly focused on studying an alternative compatibility function with the self-attention mechanism in transformer-based encoder models, particularly those evaluated using nlu. while we show good performance in this setting, our results do not necessarily translate to decoder models, pure language modeling tasks, or machine translation. for many applications, the cross-attention mechanism is crucial for achieving high accuracy on these tasks and does not completely align with our use case, where we support shared representations through a trainable matrix. in our model, we use a single shared weight matrix w s for the unified representation, reducing the number of parameters in the self-attention block by 66. 67% compared to the baseline models. although this reduction is significant, its impact on broader applications requires further analysis. due to the resource restriction, we only observed improved training efficiency for smaller bert-like models with approximately 100 million parameters in one of our experiments. however, these findings may not generalize well to much larger models, such as those of an order of magnitude larger. one limitation of our approach is its reliance on a single softmax weight, which may not exhibit optimal behavior for more complex datasets, suggesting the need for multiple weights or alternative strategies. we also recognize the importance of decoder components in text-generation tasks, which we have yet to fully explore. overcoming these challenges through future investigations will contribute to the generalization and scalability of our approach in diverse nlp frameworks. ## conclusions the shared weight self-attention mechanism presented simplifies the traditional self-attention model by using a single shared matrix with elementwise scaling for keys, queries, and values. this approach reduces parameter complexity while maintaining high performance. extensive experiments on the glue benchmark datasets demonstrate that the shared weight self-attention-based bert model performs comparably to traditional bert models on clean data and shows superior robustness under various noise conditions. the empirical results highlight the model's ability to capture essential features more effectively and maintain stability even with noisy inputs. this makes the shared weight self-attention mechanism particularly suitable for applications in environments with noisy or imperfect data. additionally, the significant reduction in learnable parameters leads to more efficient models that are easier to deploy in resource-constrained settings.