Fast Whitespace Correction with Encoder-Only Transformers (2023)

Abstract: The goal of whitespace correction is to fix space errors in arbitrary given text. For example, given the text “whi te space correctio nwithTransf or mers”, produce “whitespace correction with Transformers”. We compare two Transformer-based models, a character-level encoder-decoder model and a byte-level encoder-only model. We find that the encoder-only model is both faster and achieves higher quality. We provide an easy-to-use tool that is over 900 times faster than the previous best tool, with the same high quality. Our tool repairs text at a rate of over 200 kB/s on GPU, with a sequence-averaged F1-score ranging from 87.5% for hard-to-correct text up to 99% for text without any spaces.

## introduction most natural language processing applications assume a segmentation of the text into words. in english (and many other languages), this segmentation is typically achieved by splitting the text at space characters (and a few additional rules). however, many texts contain a significant amount of space errors, that is, spurious spaces or missing spaces. these can be due to ocr errors, imperfect extraction from pdf files, or typing errors. see @xcite for a more thorough discussion of the sources of such errors. we consider the following whitespace correction problem: given a text in natural language, with an arbitrary amount of missing or spurious spaces, compute a variant of the text with correct spacing. the text may contain spelling errors, which make the task more difficult, but it's not part of the problem to correct them. however, as shown in @xcite , with spaces repaired, spellingcorrection algorithms do a much better job. the best previous methods for whitespace correction achieve high f 1 -scores, however, at the price of very slow processing speeds; see section 2. this is a major obstacle for the practical use of such systems for large amounts of text. our goal in this work is to provide a practical tool with a much higher speed, without sacrificing the high quality. ## related work recent work on spelling correction @xcite @xcite and ocr postcorrection @xcite word for every input token, without addressing space errors. to use these systems for text with a combination of spelling or ocr errors and whitespace errors, it is necessary to correct the spaces first. other ocr postprocessing systems also try to correct space errors, but with limited success @xcite @xcite . @xcite showed that space errors can be corrected separately with models that are robust against ocr and spelling errors. previous work on whitespace correction uses autoregressive models to determine the most likely segmentation of a given text into words. the used models are n-gram models, character-level recurrent neural network language models or characterlevel neural machine translation (nmt) models. @xcite use a beam search with an ngram language model to split unknown tokens in ocr'ed croatian text into multiple words. @xcite use a character-level gru nmt model to segment texts from the acl anthology corpus. @xcite use n-gram statistics to split tokens in digitized historic english newspapers. @xcite use a beam search with a character lstm language model or n-gram language model for english word segmentation. @xcite use a beam search with a combination of a unidirectional character lstm language model and a bidirectional lstm whitespace classification model and introduce penalty terms to make use of existing spaces in the input text. in contrast to previous work, our best approach does not use an autoregressive model, but instead addresses the task with a sequence-labeling classification model. we make use of the transformer architecture, which improved the state of the art in many nlp tasks, including machine translation @xcite , language modeling @xcite , language understanding @xcite , and chinese word segmentation @xcite . we compare our approach with the previous best results from @xcite and other baselines. ## approach we compare two approaches for the whitespace correction problem: a character-level encoderdecoder (ed) model and a byte-level encoder-only model (eo). both models respect existing whitespace information in the input text. we pre-process the input text by removing duplicate, leading, and trailing whitespaces, and applying nfkc normalization foot_1 . ## demo we provide open access to all of the eo and ed models in form of a python package which can be installed via pip 12 or from source. the package comes with a command line tool and a python api both of which enable the user to correct whitespace errors in arbitrary text. additionally, the command line tool provides an option for running a whitespace correction json api and thereby enables access to our models not only from python code but also from other programming and development environments. in addition to the python package, we also provide a web interface at https://whitespacecorrection.cs.uni-freiburg.de that allows users to correct whitespaces in arbitrary text; see figure 1 . in particular, the web app allows to evaluate the output against an error-free ground truth and provides quick access to all of the used benchmarks. therefore, the easiest way to reproduce the results of our models on all benchmarks as presented in this paper is through the web interface. for more information visit our github repositories at https://github.com/ad-freiburg/whitespacecorrection and https://github.com/ad-freiburg/textcorrection-benchmarks . ## conclusion we have shown that carefully trained encoder-only transformers perform whitespace correction with the same high quality as the best previous work, but over 900 times faster. a classical encoder-decoder transformer can achieve the same quality, but with little to no gains in runtime speed. our software is open source and accessible as a python package as well as via a dedicated website. a logical next step in this line of work is to combine whitespace correction with spelling correction. recent large language models like gpt-3 @xcite can solve both tasks at once with high quality, but they are slow and expensive to run due to their size and autoregressive decoder architecture. by separating the tasks, as advocated by @xcite , each can be solved efficiently with specialized models, like our eo models for whitespace correction. it remains an interesting open question whether a single model can achieve both high quality and reasonably cheap inference. due to working directly with bytes, our eo models could be trained and applied across multiple languages. however, because datasets and benchmarks for whitespace correction in non-english languages are yet to be created, we leave the development of multilingual models for future work.