Stop Jostling: Adaptive Negative Sampling Reduces the Marginalization of Low-Resource Language Tokens by Cross-Entropy Loss (2025)

Abstract: Neural language models often struggle with low-resource languages due to the limited availability of training data, making tokens from these languages rare in the training set. This paper addresses a specific challenge during training: rare tokens are disproportionately affected by marginalization, which prevents them from learning effectively. We propose a thresholding technique that reduces the impact of this marginalization, allowing rare tokens to benefit from more meaningful alignment. Through experiments with a character-level language model, we demonstrate that this method significantly improves performance on low-resource language validation data. This work is the first to show how negative sampling can be applied to improve the representation of rare tokens by limiting the harmful influence of excessive marginalization, offering a new approach to enhancing language model performance for underrepresented languages.

## introduction neural language models have revolutionized natural language processing (nlp), providing stateof-the-art results in a wide range of tasks, such as machine translation, text generation, and sentiment analysis. however, the effectiveness of these models heavily relies on the availability of large, highquality datasets for pre-training. this dependency presents a significant challenge for low-resource languages, which often lack the extensive corpora needed for effective language model training. one of the main issues faced by multilingual language models is the difficulty in learning effective representations for tokens from low-resource languages. these tokens, which occur infrequently during training, tend to receive less alignment and are more responsive to noise from irrelevant contexts. recent studies have highlighted how this imbalance can negatively impact model performance non-target marginalization (red): move non-relevant w bob and w cat away from g θ (. . . , w happy , . . . ). the proposed method prevents marginalization of embeddings under the threshold. @xcite . existing solutions often focus on improving the general quality of embeddings @xcite or limiting the influence of rare tokens on the overall training process @xcite . in this paper, we identify a specific source of noise that affects rare tokens, which we call marginalization. marginalization by crossentropy loss pushes non-target embeddings away from irrelevant contexts and disproportionately impacts rare tokens, preventing them from learning meaningful representations. unlike previous methods that address the impact of rare tokens on the overall model, we focus on reducing the negative impact on rare tokens themselves, which is a less explored but equally important problem. to address this issue, we propose a simple yet effective adaptive negative sampling technique, which we call thresholding. by applying a threshold to the logits, the model effectively ignores nonrelevant tokens during the training process, allowing those non-relevant tokens to receive more meaningful updates. this approach is novel in its use of negative sampling to improve the representations of unselected negative samples, rather than focusing solely on training efficiency or contrastive learning. we validate thresholding effectiveness through experiments with a character-level multilingual language model trained on simulated mixed lowresource and high-resource language data. the results demonstrate that the proposed technique improves the representation and performance of rare tokens, making it particularly valuable for enhancing language models in low-resource settings. the main contributions of this paper are as follows: by addressing the challenges faced by tokens in low-resource languages, this paper presents a novel approach to improving multilingual language model performance, contributing to more balanced progress in nlp for underrepresented languages. ## experiments the code is available on github 1 . for this project, we used the nanogpt implementation by karpathy foot_1 . we conducted experiments on a small dataset of shakespeare's texts and trained a character-level language model. the dataset contains 65 unique characters, with a highly imbalanced distribution (figure 2 ). ## conclusion in this paper, we propose a method to improve the performance of language models in low-resource languages by reducing the impact of marginalization through logit thresholding. the experimental results demonstrate significant improvements. the language modeling accuracy for the low-resource language increased from 0.31 with baseline to 0.49, which is close to the accuracy for the high-resource language (0.53). additionally, the ppl best for the low-resource language was reduced from 10.63 to 6.11, almost reaching the ppl best for the high-resource language (4.97). the proposed approach not only improves performance metrics but also helps the model learn better representations, as evidenced by the alignment of "translations" of the same characters across different languages. furthermore, while previous work on negative sampling has primarily focused on enhancing training efficiency or improving the representation of positive examples, this method is, to the best of our knowledge, the first to show how negative sampling can directly improve the representation of non-sampled tokens.