Hypernetworks for Perspectivist Adaptation (2025)

Abstract: The task of perspective-aware classification introduces a bottleneck in terms of parametric efficiency that did not get enough recognition in existing studies. In this article, we aim to address this issue by applying an existing architecture, the hypernetwork+adapters combination, to perspectivist classification. Ultimately, we arrive at a solution that can compete with specialized models in adopting user perspectives on hate speech and toxicity detection, while also making use of considerably fewer parameters. Our solution is architecture-agnostic and can be applied to a wide range of base models out of the box.

## introduction in the recent years, perspective-aware approach to subjective linguistic tasks has been gaining prominence in nlp. this approach suggests that for tasks that involve subjectivity, dataset designers should collect multiple labels from different annotators for each data instance; these labels need to be retained and used in model training @xcite @xcite . this policy is warranted in tasks like hate speech detection, where multiple labels assigned by annotators with diverse backgrounds can be equally applicable. this notion contrasts the popular practice of designating a single supposedly correct label for each bit of data while discarding conflicting annotator judgments. likewise, nlp researchers have argued that subjective tasks require perspective-aware machine learning methods, i.e., methods that can capture diverse opinions based on unaggregated labels @xcite . we are particularly interested in the paradigm of strong perspectivism @xcite . the latter suggests that much like model personalization, separate models or representations should be trained on labels produced by individual annotators. in practice, this entails one of the following: (1) training a full language model checkpoint on each annotator's labels; (2) when using peft methods @xcite , training a separate adaptation component for each perspective; (3) using a specialized language model architecture. 1 we argue that both the first and the second strategy require a prohibitive amount of trainable parameters that increases with model size and number of modeling targets (see discussion in section 6). to tackle this problem, we make use of an architecture that employs a small trainable module and adapts a model to diverse perspectives while keeping most parameters frozen. the core of our solution is the hypernetwork+adapters combination. a hypernetwork is a neural architecture in which a source neural network is trained to predict the weights of a target network; these weights are subsequently used in inference @xcite . importantly, hypernetworks can also be used to predict weights of adapters, including lora adapters @xcite , rather than weights of entire models. adapters can be defined as small trainable modules designed for parameter-efficient tuning of nlp models including large language models; compatibility with adapters makes hypernetworks suitable for the same task @xcite @xcite . we provide more details on this architecture in section 3. the novel aspect of our work is that we attempt to repurpose the hypernetwork+adapters combination for perspectivist modeling and, by extension, model personalization. we posit its strengths, i.e., the reduction of trainable parameters, as a way to address the mentioned bottleneck issue. to our knowledge, this direction has not been thoroughly explored in previous studies. our article makes the following con-tributions. first, we consider whether hypernetwork+adapters setting is generally applicable to annotator-aware text classification. our results suggest that hypernetworks are well fit for this task, albeit not unconditionally. second, in our experiments, foot_1 we show that our hypernetwork-based architecture performs on par with recent perspectivist model architecturesparticularly, annotator-aware representations for texts @xcite , and annotator embeddings @xcite . at the same time, we note the limitations of the proposed architecture and attempt to address them. third, we demonstrate that the proposed architecture offers a better trade-off in terms of parameter efficiency than the baseline models. at the same time, it also demonstrates a greater degree of versatility: since the weights of the base model are not being affected, there is no risk of forgetting @xcite or degradation of the base model's fundamental capabilities. these properties of our method make us believe that it has promise in modeling subjective tasks. perspectivist learning: modeling human variation in labeling based on annotator perspectives has been advocated in several recent studies @xcite . in terms of modeling techniques, various recipes have been explored, including trainable crowd layers @xcite ; trainable tokens @xcite ; multi-task classification with annotatorspecific heads @xcite , and others. recently, studies have focused on active learning @xcite and few-shot perspective modeling @xcite as two ways to deal with data sparsity. strong perspectivism brings together perspectivist learning and model personalization, which is why recent methods from the latter domain are also relevant to our research. in particular, studies by @xcite and @xcite show that adapters and lora adapters can rival full model finetuning on the llm personalization task, even when the number of trainable parameters is reduced to a fraction of the original model size. finally, in our study, we are particularly interested in recent works that optimize encoder-based classifiers against pluralistic labels and make use of annotator embeddings @xcite . in this work, we consider both of these as our baselines. hypernetworks: the idea of hypernetworks has its roots in an earlier concept of weight generators @xcite and aims to constrain the search space when modeling complex objectives through searching in the limited weight space. practically, this means generating the weights of a target model by means of a separate generator model (hypernetwork). the title paper, published in 2017 @xcite , had two additional objectives: reduction of trainable parameters (1) and regularized training (2). the study proposed an implementation of two network types: a cnn network generating weights for another cnn network and an rnn network producing weights for a target rnn. their results showed that the system achieves competitive performance and reduces the trainable parameter count, effectively fulfilling its purpose. the authors' assumptions were further scrutinized in follow-up works. as an example, a study by soydaner (2020) applied hypernetworks to convolutional autoencoder models and showed that the weights of an autoencoder can be closely approximated by a much smaller hypernetwork, resulting in significant reduction of trainable parameters. moreover, the ml community recognized early the potential of hypernetworks in multitask learning. for instance, a study by @xcite adapts a transformer model to a multitude of tasks by predicting task-specific weights for the model's feed-forward layer using a hypernetwork. this adaptation strategy is of particular interest to us due to @xcite 's success in learning perspectives through multi-task classification. the above architectures leveraged hypernetworks to predict the model's own weights. however, the parameter-efficient finetuning paradigm brought about an alternative approach, which is to infer the weights of small trainable submodules while leaving the weights of the main model intact. adapters are a specific instance of these submodules @xcite ; modeling them with hypernetworks has led to impactful results in several machine learning applications. specifically, in image generation, hypernetworks have been used to personalize diffusion models achieving a considerable speed-up compared to other methods @xcite . in a different vein, in speech recognition, müller-eberstein et al. (2024) employed hypernetworks to adapt asr models to individual speakers with atypical speech patterns. their hypernetwork obtains a 75% relative reduction in word error rate using only 0.1% of the model parameters. in nlp, hypernetworks have been utilized for multi-task and multilingual adaptation of larger transformer models, such as t5 @xcite . @xcite propose a single hypernetwork that produces adapter weights for multiple languages and tasks simultaneously, eliminating the need for separate language-specific task adapters. karimi @xcite and @xcite leverage a shared hypernetwork to effectively train adapter modules for a range of nlp tasks. finally, in 2025, @xcite used hypernetworks to generate task-sepecific lora parameters for llms on the fly given a textual task description, thus enabling zero-shot adaptation; their results show that hypernetworks are combinable with llms, while their approach may also be repurposed for user personalization in the future. ## results in our experiments, the proposed hypernetworkbased architecture demonstrates generally strong performance across all datasets when compared to both aart and ae baselines, while using substantially fewer trainable parameters. tables 2 and 3 report the mean and standard deviation over ten runs for each metric suite as defined in section 4.3. on the d mda dataset, our model achieves an annotator-level f1 of 70.24 ± 0.9 and a globallevel f1 of 78.11 ± 0.2, thus surpassing both the aart baseline (annotator f1 69.72 ± 1.1, global f1 77.38 ± 0.4) and the ae baseline (global accuracy 75.14, global f1 73.60). this result indicates that our architecture is effective at capturing individual annotator tendencies and producing coherent overall predictions in this scenario. for d rb , our approach again outperforms aart in annotator-level performance (73.81 ± 2.0 vs. 71.10 ± 3.2) and yields a competitive global-level f1 of 76.17 ± 1.8 (aart: 79.96 ± 1.9), suggesting that our hypernetwork is capable of modeling both base and rare annotator behaviors even when the dataset exhibits varied disagreement patterns. interestingly, on the more challenging d epic dataset, our model gives lower annotator-level f1 (53.16 ± 1.6) and global-level f1 (65.11 ± 1.2) compared to aart's 59.67 ± 0.9 and 66.16 ± 1.4, respectively. we believe this shortcoming stems from the higher complexity of data in epic, 3 which may require more specialized regularization strategies; still, our model attains a higher item-level disagreement correlation (0.28±0.03 vs. 0.20±0.06), thus showing that these two perspectivist metrics have a potential discrepancy. on d hs-brexit , the comparison with ae yields a global accuracy of 86.49 (ae: 87.03) and a global f1 of 58.30 (ae: 60.36), which is only slightly below the baseline. given the dataset size, it translates into a difference of 8 misclassified instances. beyond model generalizability, hypernetwork+adapters solution demonstrates remarkably better parameter efficiency, as it requires only ≈ 5.6 * 10 6 trainable parameters compared to ≈ 124.9 * 10 6 for aart and ≈ 125.5 * 10 6 for ae. this advantage is especially important within the strong perspectivist paradigm, where an ideal model should be able to scale up to hundreds of perspectives and, at the same time, take up a reasonable amount of memory and disk space. ## discussion two potentially advantageous aspects of the hypernetwork+adapters setting are (1) keeping the base model's weights intact and (2) adapting the model to all targets by training just one component. in what follows, we analyze why these aspects are especially relevant perspectivist learning. preserving the base model's original state could be beneficial for two reasons. concerning inherently multi-purpose models, such as t5 @xcite , it means that the base model's performance on other tasks will not be affected. this effect is especially important when dealing with large language models: they are often used for a wide range of tasks, and updating their weights can lead to catastrophic forgetting @xcite or degradation of their performance on previously learned tasks. for instance, if a model is fine-tuned for a particular task like hate speech detection, preserving the base model's original state ensures that its performance on other tasks remains intact. moreover, when adapting a model that has al3 for example, as per @xcite , gpt-3.5-turbo yields an f1 of 48.1 on epic in zero-shot settings; this differs from its zero-shot performance on other language understanding tasks, such as sentiment analysis or natural language inference (f1s of 91.13 and 67.87 respectively, @xcite ). ready been fine-tuned for some task irrespective of perspectives, our method can preserve this original, 'perspective-neutral' model. in a realistic setting of personalized content moderation, this model can be used as a fallback option. for example, if some user's view is not covered by the perspectivist model, the original, perspective-neutral model can be used to provide a fallback response. another benefit of modeling adapters and keeping the main classifier frozen is the mitigation of potential negative outcomes associated with perspectivist learning. in particular, when a roberta model that is fully trainable is finetuned for perspectivist labels (as is the case in @xcite , it is taught to associate the same text with varying labels (one to many). this can cause conflicting gradient steps leading the model away from the optimum. incorporating annotator features through embeddings is intended to resolve this issue by converting the task back to a one-to-one association. however, it is unclear whether these features receive sufficient weight in the classifier to achieve that. in our framework, there is no discrepancy between inputs and outputs, as the only trainable component is the hypernetwork, which learns a one-to-one correspondence between annotators and their respective adapters. this approach does not expose the base model's weights to controversial targets and thus avoids the associated difficulties. using a hypernetwork to learn adapter weights offers an additional advantage. training a separate lora-style adapter for each annotator is possible but involves training a large number of parameters that grows substantially with each added perspective. as shown in in contrast, our method stands out as the more cost-effective solution in all settings, except for d hs-brexit , which has an exceptionally small number of annotators foot_2 . like other approaches that make use of annotator embeddings, adding a new annotator requires 1×hdim parameters + rescaling the model to a new embedding. given the small size of the hypernetwork, all of this sums up to just 2 × hdim, allowing for very cheap scaling to additional perspectives. this property makes it especially attractive for perspectivist learning.