Layer Duplication in LLM s (2025)

Abstract: We investigate the effect of duplicating multihead self-attention layers in large language models (LLMs) across a range of language tasks, with and without fine-tuning. The results demonstrate that duplicating the initial layers once or twice often yields a significant performance boost. Attention analysis uncovered the underlying mechanisms driving the improvement when performing layer duplication. This method enhances LLM capabilities with or without additional training or labeled data.

## introduction large language models (llms) have become a cornerstone of natural language processing (nlp), achieving state-of-the-art results across a wide range of tasks, such as question answering, language generation, and classification. these models, characterized by their deep architectures, large number of parameters, and attention mechanisms @xcite , show improvements with increased scale @xcite . however, this growth in model size also leads to increased computational training costs, raising challenges in optimizing performance while maintaining efficiency. common strategies for enhancing llms include data augmentation @xcite , prompt engineering @xcite , and scaling models prior to pretraining @xcite . in addition, few-shot and in-context learning @xcite have enabled models to generalize effectively without further training. another, less explored, technique is layer duplication, a method that modifies the model architecture posttraining to improve its performance by increasing its effective depth without changing its learned weights. layer duplication is a relatively unexplored method that has shown promise for improving llm performance both with and without finetuning. at the heart of llms is the transformer architecture @xcite , which uses stacked multi-head self-attention (mhsa) layers to build contextual understanding. the idea behind layer duplication is to replicate certain mhsa layers in the architecture, enabling the model to pass information through the same transformation multiple times. this reprocessing may allow the model to refine representations more deeply, enhancing its performance on various tasks. importantly, this approach requires no additional training data and minimal architectural changes, making it especially appealing for scenarios with limited resources. despite significant progress in model design and training efficiency, there remains a trade-off between performance gains and computational cost. increasing model size often results in diminishing returns @xcite . to address this, parameter-efficient fine-tuning (peft) techniques @xcite , such as low-rank adaptation (lora) @xcite , have been developed to reduce fine-tuning costs by adapting only a subset of parameters. layer duplication offers a complementary strategy: it alters the flow of computation to extract more value from existing weights. this study is motivated by the need to improve llm performance, with or without finetuning. we hypothesize that layer duplication is particularly effective in nlp models because language understanding may benefit from repeated contextualization over the full sequence, whereas vision tasks, which often rely on localized spatial features, are less likely to gain from repeating the same layer multiple times. for this reason, our work focuses on applying layer duplication in llms. our contributions are as follows: (1) we demonstrate that duplicating one of the initial layers one or two times can be expected to significantly boost llm performance in both few-shot and fine-tuned settings, and provide theoretical and empirical motivation for why the initial layers are particularly effective to duplicate. (2) we show that performance gains depend more on the task than on model size. ## related work while increasing model depth and width is a well-established strategy for boosting performance @xcite , the idea of expanding existing pretrained models through layer duplication with or without finetuning remain largely unexplored. relaxed recursive transformers @xcite and dynamic layer operation @xcite (dlo) explore layer reuse and vertical scaling, but differ from our method in motivation, complexity, and practicality. relaxed recursive trans-formers compress models by looping over a fixed block of layers at inference and using lora to relax weight sharing. while effective, this adds architectural complexity and degrades performance. dlo scales depth by partitioning a model (e.g., 32 layers into 4 groups of 8) and adding 10 new layers per group, partly initialized from the group's final layer. it manages compute via a routing policy trained during supervised fine-tuning. in contrast, we do not compress the model or rely on dynamic control. we propose a simple, static, and efficient technique, namely, layer duplication. we empirically and theoretically identify which layers to replicate and how many times, showing strong gains even without fine-tuning. our approach avoids fixed architectural templates, offering a flexible, low-overhead way to improve existing models, and provides attention-based analyses and theoretical insights into why and where duplication helps. another related approach, pondernet @xcite , introduces adaptive computation by dynamically adjusting the number of processing steps based on task complexity. this method focuses on optimizing computational efficiency by varying the depth of computation as needed. while pondernet is dynamic and adapts to specific tasks, layer duplication offers a more straightforward, static alternative. by duplicating mhsa layers, the model reuses the learned representations from those layers, allowing it to process information again. this enables the model to build on the contextualized effects of the original layer, further enhancing its ability to make logical connections between words and concepts, especially in tasks requiring reasoning and deep semantic understanding. layer skipping is another approach to manipulating a given model's architecture. techniques such as layerskip @xcite , unified layer skipping @xcite , and adainfer @xcite selectively bypass certain layers to improve inference efficiency while trying to best maintain model accuracy. unlike layer duplication, which aims to enhance model accuracy through duplication of specific layers, these methods focus on identifying and skipping layers that are of less importance. unlike prior work that focuses on efficiency or adaptiveness, our approach statically reuses existing layers to enhance performance. this is the first systematic demonstration showing that duplicat-ing early layers improves llm performance. we motivate this both empirically and theoretically. ## qualitative analysis to further investigate the impact of layer duplication, we analyzed its effects on attention mechanisms. this analyses aims to give further motivation why duplication of early layers improves performance. ## conclusion we have demonstrated the potential of layer duplication in transformer-based llms across fewshot multiple-choice task settings and fine-tuned question answering settings. our results show that duplicating any of the initial layers once or twice, whether or not additional fine-tuning is applied, can significantly improve model performance. we therefore recommend this strategy to practitioners. our experiments suggest that this approach has a high likelihood of delivering meaningful performance gains. beyond empirical performance gains, we theoretically motivated why duplicating the initial layers is the best approach based upon previous work and the learned weights of the weighter. furthermore we showed an attention analysis that revealed that duplication of the first layer modifies attention patterns in a way that enhances contextual understanding. these findings suggest that early layer duplication effectively leverages pretrained weights, improving performance without additional training.