Transferring Extreme Subword Style Using Ngram Model-Based Logit Scaling (2025)

Abstract: We present an ngram model-based logit scaling technique that effectively transfers extreme subword stylistic variation to large language models at inference time. We demonstrate its efficacy by tracking the perplexity of generated text with respect to the ngram interpolated and original versions of an evaluation model. Minimizing the former measure while the latter approaches the perplexity of a text produced by a target author or character lets us select a sufficient degree of adaptation while retaining fluency.

## introduction text style transfer (tst) aims to reformulate a source text using the stylistic attributes of a given target text. authors vary a blend of these attributes to achieve a literary effect, with some modifications being more conspicuous than others. stylistic modification of subword units like characters or phonemes can prove especially noticeable. 1 one such subword style is orthographic variation, a phenomenon common in forms of "dialect literature" present throughout history but especially popular in the 19th century united states @xcite . these works utilize context and readerly knowledge to render their orthographic innovations legible despite their extreme modification of orthographic norms @xcite . we present a subword-level ngram-based logit scaling method that effectively transfers this form of extreme style at decoding time. we accomplish this by combining the next-token information derived from a large language model (llm) with information obtained from ngram models trained on a single-author/character corpus. ngram models are quick to train, data-efficient, and interpretable. training ngram models on small 1 in this paper, all fundamental units are subwords, as such references to tokens refers to subword tokens produced by subword tokenization methods. we introduce a scaled interpolation method that combines weighted ngram model predictions with those from pretrained llms to generate fluent stories that match the extreme subword style of particular dialect authors and characters. we also demonstrate how to tune and evaluate these transfers using perplexity measures. ## related work techniques like finetuning on further data @xcite , prompt editing @xcite and in-context learning @xcite have been used to achieve tst. while potentially effective, these avenues require further computation and additional training data. mechanical interpretation approaches can provoke style at inference time by intervening on model weights @xcite . however, this approach requires the target style to be in-distribution and suitably represented in the model. other recent works have re-evaluated lm approaches previously considered obsolete in the light of computational and theoretical advances. ngram modeling has been revisited for lm smoothing @xcite and in "infinite" form as a interpolation component used to complement llms @xcite . ## methodology we achieve subword tst by applying an ngram model-derived scaling factor to the output logits of an llm prior to softmaxing and sampling. vitally, information from the ngram model must contribute to the next-token probability without warping the llm's ability to produce fluent text. we ensure both fluency and transfer by scaling the ngram model-provided next token prediction with an additional factor f . given a vocabulary v , we calculate the scaling factor using equation 1 inspired by temperature decoding methods, the addition of the parameter f uniformly increases the scaling factor as f increases, leading to a higher proportion of generation information being derived from the ngram model. this also renders the scaling mixture "tunable". we generate story prompts and establish standard english baseline scores using the writing-prompts (wp) dataset @xcite . we sample 50 prompts from the dataset to guide story generation. we modify each original prompt with a brief instruction stem in order to produce the scaled generation prompt set. additionally, we apply three character/author templates to each prompt in order to produce a control prompt set. the first indicates that the model should act as a storyteller, and includes a description of its era and position. the second adds information about the target author, and the third the target character. see appendix b for more prompt creation details. models. for generation of control and scaled texts, we use mistralai's mistral-instruct-7b-v0.2 (mistral) @xcite ) and meta's llama3.2-3b-instruct (llama) @xcite pretrained instruction-tuned models. for perplexity evaluation, we use openai's gpt2-large (gpt2) @xcite . we obtain model checkpoints via huggingface. using the wordpieced target texts, we train a set of ngram models, {m n , m n-1 , ..., m 1 }, with @xmath0. when scaling generated logits, we employ the model set in a backoff configuration. if m 4 cannot make a 4gram next token prediction, we use a trigram prediction from m 3 , and so on. if no model can make a prediction, no scaling is performed. this is essentially a modification of stupid backoff @xcite . evaluation. we concatenate the tokens produced by each scaled or control generation and estimate their gpt2 perplexity using a sliding context window of 32 tokens with stride of 1. we do the same for the wp test-set baseline and gb target texts. when scoring general model performance, low perplexity is considered preferable. for our purposes, near-equal target (p p l()) and generation (gp p l()) perplexities indicate successful subword style transfer. we also measure the perplexity of the original texts using the interpolation of gpt2 and each target text's scaled ngram models, rp p l(). combining these two sources of information allows us to select an optimal schedule of scalings for subword style transfer by maximizing abs(p p l() -gp p l()) while minimizing rp p l(). intuitively, the first measure acts as an early stopping criterion, while the second measure indicates whether the gp p l() at a given scaling is produced by transfer and not chaotic. ## conclusions and further work our ngram scaling method produces plausible story generations that bear features of the extreme subword style of their target author or character in a compute and data-efficient manner. further work can be performed to test the method on other forms of subword variation, and to characterize the specific subword features that were transferred relative to the subword tokenization system used by a given llm and ngram model. additional work could also include increasing the precision of our method for determining scaling optimality, further characterizing a scale of subword-style extremity in order to help determine what forms of style are likely candidates for transfer by this method, and experimenting with hybrid generation across multiple author ngram models.