Empowering Character-level Text Infilling by Eliminating Sub-Tokens (2024)

Abstract: In infilling tasks, sub-tokens, representing instances where a complete token is segmented into two parts, often emerge at the boundaries of prefixes, middles, and suffixes. Traditional methods focused on training models at the token level, leading to sub-optimal performance in character-level infilling tasks during the inference stage. Alternately, some approaches considered character-level infilling, but they relied on predicting sub-tokens in inference, yet this strategy diminished ability in character-level infilling tasks due to the large perplexity of the model on sub-tokens. In this paper, we introduce FIM-SE, which stands for Fill-In-the-Middle with both Starting and Ending character constraints. The proposed method addresses character-level infilling tasks by utilizing a line-level format to avoid predicting any sub-token in inference. In addition, we incorporate two special tokens to signify the rest of the incomplete lines, thereby enhancing generation guidance. Extensive experiments demonstrate that our proposed approach surpasses previous methods, offering a significant advantage. Code is available at https://github.com/SenseLLM/FIM-SE.

## introduction the transformer @xcite decoderonly architecture has proven highly effective in various natural language processing (nlp) tasks. this success has paved the way for the development of advanced causal decoder-only models like @xcite , @xcite , llama @xcite , and falcon @xcite . these innovative models excel at generating coherent and contextually relevant responses to natural language prompts, showcasing state-of-the-art performance across various tasks, including question answering @xcite , ## preliminaries in this section, we provide a straightforward introduction to the fim @xcite ) method and conduct a theoretical analysis of how inconsistent labeling affects the model's perplexity. ## methodology in this section, we introduce the proposed method. we begin by outlining the training process with fim-se, followed by an explanation of the inference procedure. finally, we delve into more training details and highlight the distinctions between our approach and the traditional fim method. ## experiments in this section, we construct experiments to demonstrate the effectiveness of our method. due to space limitations, we have constructed more experiments in appendix b. ## conclusion in this paper, we showed that traditional infilling techniques struggle with managing the boundaries of prefixes and suffixes. to address this, we introduced a novel approach, referred to as fim-se. our method transforms the random-span mode to multiline mode by removing the l-prefix and f-suffix. we further incorporated two special tokens to delineate the two incomplete lines, thereby guiding the generation. extensive experiments reveal that our approach surpasses existing baselines with a clear edge. in future work, we plan to explore the adaptation of our method to the variant spm mode, which holds the promise of even better performance.