ARXSA : A General Negative Feedback Control Theory in Vision-Language Models (2025)

Abstract: The Transformer model has been increasingly applied across various domains, driven by the self-attention mechanism, which offers robust data processing capabilities and has substantially contributed to the advancement of the model. In the self-attention mechanism, three core matrices from the same data batch are computed together to determine correlations between input elements. Drawing inspiration from the efficiency and stability conferred by negative feedback structures in predictive control systems, the concept of vertical training was introduced to integrate data from multiple batches. Accordingly, this paper proposes an autoregressive with exogenous inputs (ARX) approach for the self-attention mechanism, transforming the Encoder block into a negative feedback predictive control system. A network architecture based on this method is also proposed, enabling the autoregressive with exogenous inputs for self-attention to transmit data from batches at previous time points. The effectiveness of the proposed approach is validated through comparative experimental evaluations.

## introduction in recent years, neural network research had gradually become central to advancements in artificial intelligence @xcite . computer vision (cv) and natural language processing (nlp) had also assumed significant roles in daily life @xcite . among the developments in these fields, the emergence of self-attention (sa) mechanisms and transformer networks had addressed the limitations imposed by short memory in traditional attention mechanisms @xcite . the application of sa to the cv domain had similarly enhanced network performance @xcite . however, existing research on sa mechanisms primarily concentrated on improving computational efficiency and optimizing performance @xcite . for instance, the swin transformer significantly reduced computational overhead by minimizing less relevant correlations in the sa mechanism, yielding improved performance on several benchmark datasets @xcite . these computational optimizations were generally predicated on the importance of pixel correlation calculations. while these methods reduced computational load for distant pixel relationships, they concurrently strengthen the correlation between neighboring pixels. some approaches also lowered the computational complexity by reducing the dimensionality of low-rank core matrices, effectively adjusting the image size @xcite . although these techniques had undeniably optimized the sa mechanism, there remained a notable lack of a robust theoretical foundation for sa and transformer models. as a result, the tuning of transformer parameters and architectures continued to rely heavily on empirical experimentation. to enhance the controllability of neural networks, several studies had explored the integration of neural networks with control theory, beginning with common residual blocks, in an effort to uncover underlying patterns @xcite . control theory, having evolved rapidly over the past decades from a branch of mathematics into a distinct and widely applied discipline, was capable of integrating principles from numerous fields. predictive control theory, in particular, offered the advantage of improving both control performance and efficiency by forecasting future system behaviors and optimizing control inputs @xcite . in pursuit of imparting controllability to neural networks, this study examines the self-attention mechanism from the perspective of predictive control systems. it is observed that the modified sa mechanism can be conceptualized as an autoregressive with exogenous inputs (arx) model @xcite . to introduce predictability into certain neural networks, this paper re-evaluates the connections between the self-attention mechanism and various blocks within transformers. by integrating the residual structure in the encoder block with a modified sa mechanism, a method has been developed that enhances training accuracy with minimal additional computational cost. specifically, the proposed method, termed autoregressive with exogenous inputs for self-attention (arxsa), gains temporal characteristics by constructing the selfattention mechanism through core matrices generated at different time points. these matrices are combined with the current matrix through a recurrent process, imparting temporal characteristics to the exogenous inputs as well. to assess the feasibility of this approach, the autocorrelation function (acf) is employed to validate its autoregressive properties @xcite . the results confirm that the arxsa method exhibits autocorrelation, aligning it with the arx model. this suggests that arxsa can adjust network outputs based on historical data. to distinguish between conventional training methods and those incorporating historical data, the concept of unit time in the training process is introduced, defined based on batches. consequently, training methods are classified into horizontal and vertical training, depending on the unit time. by employing the vertical training concept, the arxsa method transforms the encoder block into a negative feedback predictive control system. the feedback path of arxsa improves both the efficiency and robustness of the overall network @xcite @xcite . furthermore, arxsa is integrated into transformers to enhance their operational efficiency. the resulting transformer can be interpreted as a predictive control system with autoregressive properties. extensive experiments on several widely-used benchmark datasets demonstrate the efficacy of the proposed transformer in image classification and object detection tasks. additionally, its generality is validated in networks similar to swin transformer. the contributions are summarized as follows: ## method this section will systematically introduce predictive control theory, the arx model, and the acf, detailing the relationship between predictive control and the arx model through illustrative examples. next, the concept of unit time, which is defined in relation to batch processing in neural network training, will be introduced. the unit time will be followed by a description of two distinct training methods based on the unit time concept. subsequently, the integration of the arx model with the self-attention mechanism to form the arxsa method will be explained, and its autocorrelation properties will be validated by acf. lastly, a comprehensive overview of the network architecture incorporating the arxsa method into the transformer will be provided. ## experiments in this section, we selected several popular vision benchmarks to verify the effectiveness of the proposed arxformer, including image classification, object detection and text classification. each set of experiments is validated multiple times and we will evaluate it by the average performance and the single best performance. since the arxsa method does not increase the number of parameters, the number of parameters and flops of the same network will not change regardless of whether the arxsa method is used or not. ## conclusion with the recent prevailing trend of the transformer, we envisioned that integrating negative feedback predictive control theory into the self-attention mechanism to enhance the overall efficiency and stability of the network. thus, the concept of vertical training and employed it to define the unit time for network training was introduced. consequently, we proposed and validated the arxsa method, which transforms the encoder block in the network into a negative feedback predictive control system. finally, the arxsa method was integrated into different network frameworks to verify the applicability and stability. in future work, we will study the performance of arxsa under different ar orders and study the extension of such methods to learning methods of non-sa mechanisms.