R ing F ormer: Rethinking Recurrent Transformer with Adaptive Level Signals (2025)

Abstract: Transformers have achieved great success in effectively processing sequential data such as text. Their architecture consisting of several attention and feedforward blocks can model relations between elements of a sequence in parallel manner, which makes them very efficient to train and effective in sequence modeling. Even though they have shown strong performance in processing sequential data, the size of their parameters is considerably larger when compared to other architectures such as RNN and CNN based models. Therefore, several approaches have explored parameter sharing and recurrence in Transformer models to address their computational demands. However, such methods struggle to maintain high performance compared to the original transformer model. To address this challenge, we propose our novel approach, RingFormer, which employs one Transformer layer that processes input repeatedly in a circular, ring-like manner, while utilizing low-rank matrices to generate input-dependent level signals. This allows us to reduce the model parameters substantially while maintaining high performance in a variety of tasks such as translation and image classification, as validated in the experiments.

## introduction transformer models, since their introduction @xcite , have dramatically transformed the landscape of deep learning, particularly excelling in tasks involving sequential data such as natural language processing @xcite and machine translation @xcite . not long after their inception, they have also shown strong performance in various other domains such as reinforcement learning @xcite , image classification @xcite @xcite , object detection @xcite and image generation @xcite @xcite . their core architecture, characterized by self-attention mechanisms and feedforward neural networks, enables effective handling of long-range dependencies and parallel processing of input sequences. the ability of this architecture to model intricate relationships within data has led to significant breakthroughs, making it a foundation model across many modern largescale ai systems @xcite @xcite . however, the impressive capabilities of transformer models come with substantial computational and memory costs @xcite . the standard transformer architecture consists of multiple layers, each containing millions of parameters that need to be trained and stored. this results in high memory usage and significant computational demands, often requiring specialized hardware. moreover, deploying these models in resource-constrained environments, such as mobile devices or edge computing scenarios, becomes challenging due to their size and complexity. these limitations have spurred a growing interest in developing more parameter-efficient transformer architectures @xcite that can retain their powerful performance while being less resource intensive. in this paper, we introduce a transformer architecture that recurrently leverages a single shared transformer block in a novel way by integrating input-dependent level signals at each block iteration, which are shown to be crucial for adapting the shared block to different stages of the model. the level signals are generated by depth-specific low-rank transformations applied to the input in the attention and feedforward layers within the transformer block. our ringformer model can also be viewed as stacking transformer layers whose parameters combine (1) a set of global parameters shared across all transformer layers and (2) a set of local low-rank layer-dependent parameters. this simple design effectively addresses the trade-off between reducing the number of model parameters and limiting the model's capacity to capture complex patterns. we validate our model through experiments and analysis on machine translation and image classification. the results of experiments and analysis demonstrate that our model closely replicates the behavior of the original transformer model, and it performs better against existing parameter-matched recurrence-based transformer models, underscoring the effectiveness of our approach in maintaining high performance with fewer parameters. the contributions of this paper are summarized as follows: ## experiments we evaluate ringformer and baseline models on two tasks: wmt-14 german-english translation @xcite and imagenet-based classification @xcite . these benchmarks are widely used, large enough for reliable performance, and feasible given our computational constraints. as ringformer uses recursive parameter sharing, we compare it to models with similar strategies. due to limited research in this area, we selected two representative methods: universal transformer @xcite (fully recurrent) and one wide feedforward (pires et al., 2023) (partially reccurent, with shared feedforward layer). these models capture the main variants of parameter sharing and offer meaningful comparisons to ring-former and the standard transformer @xcite . ## conclusion in this paper, we introduce ringformer, a parameter-efficient recurrent transformer architecture that employs a single transformer layer recurrently while integrating input-dependent signal vectors created using low-rank matrices for each level. this approach significantly reduces the number of parameters while maintaining high performance in tasks such as machine translation and image classification. we hope that our research on enhancing recurrent transformer with adaptive level signals can enable smaller organizations and research institutions to train powerful models without the need for extensive computational resources, thus democratizing access to advanced ai capabilities.