Unleashing the Potential of Large Language Models through Spectral Modulation (2024)

Abstract: Large Language Models (LLMs) have demonstrated impressive capabilities across various domains, garnering significant attention from both academia and industry. However, enhancing the performance of LLMs typically requires scaling up model sizes or fine-tuning with additional datasets, which results in substantial computational costs. This paper poses an intriguing question: Can we improve the performance of LLMs without additional training? Drawing inspiration from signal processing principles, which suggest that noise often resides in high-frequency components while low-frequency components carry the essence of signals, we propose uncovering untapped potential in LLMs from a frequency perspective. We hypothesize that the high-frequency components in the weight matrices of LLMs’ linear layers may conceal noise that interferes with predictive accuracy. Therefore, we propose conducting spectral modulation in the parameter space of LLMs, which can seamlessly integrate with various models in a plug-and-play manner. Extensive experiments have demonstrated the superiority of our approach, with spectral modulation yielding an average performance improvement of up to 10.12%.

## introduction the emergence of large language models (llms) @xcite @xcite has revolutionized text generation quality across various generative tasks, becoming a central and widely discussed topic in the field of artificial intelligence. this advancement can be attributed to the exponential growth in both training data and model parameters, exemplified by models like @xcite with its staggering 175 billion parameters. however, achieving better performance often requires scaling up model size or increasing training data * corresponding author: yao zhu. @xcite , which raises the question of whether it is possible to further enhance the capabilities of pre-trained llms without the need for full-model fine-tuning or parameterefficient fine-tuning methods @xcite . in the field of signal processing @xcite , signals are usually analyzed within the frequency domain. here, noise, which is typically interwoven with the signal, is found mostly in high-frequency components, whereas low-frequency components constitute the essence of the signal. to improve signal quality, researchers leverage spectral modulation as a denoising technique. turning to large language models (llms), could the process of training their vast number of parameters inadvertently introduce performance-constraining noise into the parameter space, akin to the inevitable introduction of noise during signal acquisition in signal processing? drawing inspiration from this analogy, this paper proposes to explore the potential of llms from a novel frequency-domain perspective. the transformer-based large language models are primarily composed of self-attention and feedforward modules, which are mainly constructed by linear layers. in this paper, we treat the weight matrices of the linear layers in the model as twodimensional matrices, and utilize two-dimensional fourier transform to obtain their spectra. we hypothesize that the low-frequency components within weight matrices contain the critical information pivotal to determining model performance, whereas high-frequency components may conceal noise detrimental to model performance. consequently, modulating these high-frequency components may be beneficial in unlocking the untapped potential of large language models (llms). motivated by this hypothesis, this paper introduces a spectral modulation method applied to weight matrices, specifically aimed at preserving low-frequency components while reducing highfrequency ones, thereby enhancing model performance. this method involves two hyperparameters for each weight matrix: a decay factor for reducing high-frequency components and a protection threshold for preserving low-frequency components. the visual impact of spectral modulation on the weight matrices is provided in appendix h. recognizing that different weight matrices in llms may require different optimal hyperparameters for effective joint spectral modulation, we sample a small validation dataset and employ bayesian optimization to determine the hyperparameters and report their performance on a held-out test set. the optimization objective is to maximize the model's prediction accuracy on the validation dataset. experimentally, we conduct a comprehensive evaluation of three types of llms across ten datasets and achieve superior performance compared to the vanilla model. we analyze the outputs of the model from a lexical composition perspective and find that the answers generated by our spectrally modulated model exhibit higher informativeness. meanwhile, we discussed the impact of spectral modulation on model interpretability and found that spectral modulation also improves the model in this aspect. furthermore, we investigate how the spectral modulation method generalizes to tasks beyond linguistic ones. in summary, the proposed method is surprisingly simple yet effective, and, to the best of our knowledge, it represents the first attempt to discuss constraints in llms' performance from a frequency perspective and to perform spectral modulation on the weight matrices of llms. extensive experiments on three existing open-source large language models across 10 datasets demonstrate the effectiveness of our proposed method, which leads to average performance improvements of up to 10.12% compared to the vanilla models. ## related work frequency domain analysis occupies a critical position in the field of signal processing. this technique facilitates the elucidation of the frequency composition of signals, the identification of their intrinsic characteristics, and the analysis of noise components within these signals. taking image signal processing as an example, frequency domain analysis has many typical applications, including image compression @xcite @xcite and image denoising @xcite @xcite @xcite , among others. due to page length limitations, a more detailed review of related literature on the application of frequency domain analysis is provided in appendix f. in recent years, there has been a growing emergence of endeavors integrating frequency analysis into deep learning. @xcite introduce a learning-based frequency selection method aimed at identifying trivial frequency components that could be eliminated without sacrificing accuracy, which can enhance the precision of image classification. @xcite extend the concept of channel attention to the frequency domain, proposing the multi-spectral channel attention framework. this framework exhibited promising performance across tasks such as image classification, object detection, and instance segmentation. patro and agneeswaran (2024) propose a scattering vision transformer to more effectively capture finegrained information within the input. this model first uses fourier transform operations to implement a frequency-domain layer for feature extraction in the shallow layers of the network, and subsequently employs multi-head self-attention modules in the deeper layers for feature modeling. @xcite propose the frequency domain kernelization method, which uses the discrete cosine transform (dct) to map the transformer's query and key to the frequency domain. this approach effectively transforms the complexity of attention computation to linear time, thereby reducing training costs and improving inference speed. our approach diverges from previous studies as we investigate the effects of frequency domain processing on the parameter space rather than on the input space or the feature space. specifically, our method achieves enhanced performance by processing model parameters in the frequency domain without requiring additional retraining phases. ## experiments in this section, we first introduce the experimental setup and then investigate the impact of hyperparameters on model performance with single-step spectral modulation. next, we further enhance the performance of llms through multi-step spectral modulation. we then discuss the impact of spectral modulation on lexical composition to gain a deeper understanding of our approach. following this, we compare spectral modulation with the existing method and explore its generalization performance on visual tasks. additionally, we discuss the effect of spectral modulation on model's interpretability in appendix i. ## conclusions this paper revisits the factors limiting the performance of large language models (llms) from a frequency perspective, hypothesizing that highfrequency components in the weight matrices of llms may carry noise detrimental to model performance. to address this, we propose a straightforward, retraining-free spectral modulation method that enhances model performance by reducing high-frequency components while preserving lowfrequency components in the weight matrices. extensive experiments conducted on ten different datasets and three distinct llms demonstrate the effectiveness of our method. moreover, we analyzed the lexical composition in the generated outputs from the spectrally modulated models and their high-frequency counterparts, which suggests that spectral modulation contributes to more specific and complete outputs. additionally, experiments on the classic imagenet classification task demonstrate the effectiveness of the method's generalization to non-linguistic tasks. we hope that our findings can inspire further research into the internal mechanisms of llms from the perspective of spectral analysis, ultimately leading to the development of more reliable and effective llms.