Uncertainty in Semantic Language Modeling with PIXELS (2025)

Abstract: Pixel-based language models aim to solve the vocabulary bottleneck problem in language modeling, but the challenge of uncertainty quantification remains open. The novelty of this work consists of analysing uncertainty and confidence in pixel-based language models across 18 languages and 7 scripts, all part of 3 semantically challenging tasks. This is achieved through several methods such as Monte Carlo Dropout, Transformer Attention, and Ensemble Learning. The results suggest that pixel-based models underestimate uncertainty when reconstructing patches. The uncertainty is also influenced by the script, with Latin languages displaying lower uncertainty. The findings on ensemble learning show better performance when applying hyperparameter tuning during the named entity recognition and question-answering tasks across 16 languages.

## introduction after the release of chatgpt in 2022, the number of papers published every day on the topic of large language models (llms) has increased more than 20-fold @xcite . the number of parameters in these models jumped from 340 millions in implementations such as bert @xcite to billions of parameters in models like gpt-3 @xcite or llama @xcite . despite their obvious popularity, one of the central limitations of llms remains their uncertainty and lack of trustworthiness @xcite . as these models are being applied more and more to high-stakes scenarios, such as medicine @xcite or security @xcite , it is critical that their predictions can be trusted. generally, the research on the explainability and interpretability of llms is focused on traditional tokenizer-based methods, that split text into smaller units. they produce overconfident responses even when the predictions are likely incorrect @xcite . ## state of the art the first study to use visual features of text in order to create embeddings was applied to chinese and used linearizing bitmaps of characters or words @xcite . by using shared character components from chinese or korean, it becomes easier to generalize to new and less frequent characters. different studies @xcite @xcite used rendering techniques to obtain images of text. in this context, text rendering involves converting character codes into glyph indices, which are then used to generate the corresponding glyph images, while applying various styles, fonts, sizes, and colors. a glyph often contains one character only, but it can also represent accents or multiple characters in languages where ligatures are common, like arabic. dai and cai (2017) used text rendering in chinese, japanese, and korean, and extracted visual features from a convolutional neural network (cnn) to perform text classification. similarly, @xcite applied convolutions to squared rendered images to perform sentiment analysis in chinese and english. in the context of machine translation, @xcite suggested a very robust approach based on a variation of the vit. the training data is rendered into gray-scale images using the pygame backend and a slicing window is applied to create patches, which act as tokens. then, a 2d convolutional block followed by linear projection is used to create embeddings, which serve as input for the transformer encoder. the translation happens directly from pixel representations, without any word preprocessing. after training on seven language pairs, the approach matches the performance of traditional language models, with additional advantages. it is more robust to character permutations or substitutions, and it does not rely on text preprocessing steps, such as tokenization or segmentation. as of to date, systematic investigations into the uncertainty and calibration of pixel-based language models remain limited. @xcite showed that pixel is robust when it comes to characterlevel perturbations and code-switching. in this analysis, relevancy heatmaps were used to depict visual explanations of correct predictions, and there is evidence to suggest that these outputs are interpretable when identifying contradictions and entailment relationships. however, during semantic tasks like named entity recognition, sequence classification, and question answering, pixel is struggling to retain semantic knowledge and transfer it across scripts. reasons for this might include a lack of multilingual pretraining, as well as a limited ability to capture contextual information due to the use of unigram patch embeddings. while raw performance is desirable, it is crucial to have models that are reliable and explainable. glue the sequence classification (sc) task relies on the the general language understanding evaluation (glue) benchmark @xcite . it involves nine sentence-level understanding tasks (cola, sst-2, mrpc, qqp, sts-b mnli-m/mm, qnli, rte, wnli) in english, across three categories: single-sentence tasks, similarity and paraphrase tasks, and inference tasks. tydiqa-goldp to assess the ability of the model to perform question answering (qa), the tydiqa-goldp dataset was selected @xcite . it contains nine typologically diverse languages (english, arabic, bengali, finnish, indonesian, korean, russian, swahili, telugu). the dataset contains questions written by native speakers, passages with relevant information, and answers provided as short spans of text within the passage. unlike the primary task, the gold passage task focuses more on locating the exact answer within a given context. ## discussion this work showed that it is possible to integrate uncertainty quantification methods and measure calibration in the context of visual text models. these methods include monte carlo dropout at the patch level, with the observation that more work should be directed towards finding more effective ways of aggregating and visualizing uncertainty across longer patch sequences. attention based methods can also be used to gain insights into how these models encode information, but there remains the debate about whether or not attention counts as an explanation @xcite . still, this debate falls outside the scope of this research. ensemble learning with a low number of individual learners can also be used successfully to improve both performance and confidence. the results in the mc uncertainty experiment generally indicate high uncertainty for a high mask ratio. still, the most optimal value is a mask ratio of 50%, representing a reasonable trade-off between telugu english arabic bengali finnish indonesian swahili korean russian language 0.0 0.2 0.4 0.6 0.8 1.0 confidence validation uncertainty in tydiqa-goldp uncertainty and loss. scripts such as latin are less uncertain, indicating that multilingual pretraining is necessary. instead of language, one can focus on introducing a new script, as evidence suggests that there exists knowledge transfer between scripts like latin and cyrillic. for example, finetuning on one language such as chinese might benefit performance in other languages like korean or amharic. this approach is more robust than traditional llms, where the transfer of learning happens under stricter conditions, for instance when languages share syntactic structures or when there is a significant overlap between vocabularies. ensemble learning can be applied successfully to improve performance and calibration in pixelbased language models. the evaluation shows higher f1 scores for 17 of the 19 tested languages across two tasks. the models become more robust and can overcome individual weaknesses by aggregating predictions from multiple learners using hyperparameter tuning. additionally, ensemble learning improves calibration through better error diversification and data representation. ## conclusions and future work the findings of this study indicate that pixel-based language models represent a viable and lightweight solution to traditional language modeling, even for tasks that require semantic understanding of text. their reliability and explainability can also be improved through uncertainty quantification methods, as shown during the experiments. future research should focus on perfecting the existing techniques and exploring new ways of understanding the inner workings of models that enccde text as visual representation. one point to be explored in future works on text reconstruction is the idea of pixels-as-tokens in the context of the pixel transformer (pit) model, introduced by @xcite . instead of training the model to perform patch reconstruction, pit treats each pixel as a token and the reconstruction happens at the pixel level. evidence suggests amh hau ibo kin lug luo pcm swa wol yor avg pixel 47.7 82.4 79.9 64.2 76.5 66.6 78.7 79.8 59.7 70.7 70.7 ensemble 90.2 97.1 96.1 93.9 95.5 93.1 97.1 96.1 95.8 95.2 95