Restoring Missing Spaces in Scraped H ebrew Social Media (2025)

Abstract: A formidable challenge regarding scraped corpora of social media is the omission of whitespaces, causing pairs of words to be conflated together as one. In order for the text to be properly parsed and analyzed, these missing spaces must be detected and restored. However, it is particularly hard to restore whitespace in languages such as Hebrew which are written without vowels, because a conflated form can often be split into multiple different pairs of valid words. Thus, a simple dictionary lookup is not feasible. In this paper, we present and evaluate a series of neural approaches to restore missing spaces in scraped Hebrew social media. Our best all-around method involved pretraining a new character-based BERT model for Hebrew, and then fine-tuning a space restoration model on top of this new BERT model. This method is blazing fast, high-performing, and open for unrestricted use, providing a practical solution to process huge Hebrew social media corpora with a consumer-grade GPU. We release the new BERT model and the fine-tuned space-restoration model to the NLP community.

## introduction scraped corpora of social media tend to contain many instances of missing spaces, where two or more words have been run together as one. this phenomenon is likely due to the fact that the html source of internet pages often encodes different parts of the text in distinct html tags, without explicit indication of whether two consecutive tags contain a single word or two separate words. scraping algorithms exercise various heuristics to decide whether to add a space or not; however, these heuristics do not always succeed. in practice, the nlp researcher is often faced with digital corpora of scraped social media in which a substantial number of lines are corrupted with conflated words. these missing spaces can impair downstream tasks such as parsing, segmentation, and information retrieval. additionally, when these corpora are used to train language models, the errors are propagated forward into the model. thus, it is essential to restore missing spaces wherever possible. the problem of missing spaces is particularly acute in languages such as hebrew, in which words are generally written as consonants alone. the omission of vowels results in extreme ambiguity, such that a given sequence of letters can generally be interpreted as multiple different words @xcite . crucially, this means that when two words are run together, they generally cannot be separated by means of a simple dictionary lookup, because there are multiples ways of splitting the conflated sequence into two valid hebrew words. for instance, here is an actual line contained in the hebrew section of the public oscar corpus @xcite : ‫אחד‬ ‫קושיכל‬ ‫רמת‬ ‫יכול‬ ("level of difficulty everyone can do it"). the words ‫קושי‬ ("difficulty") and ‫כל‬ ("every") are conflated together in the corpus as a single string. however, there is more than one way to split this string; if we were to apply a dictionary lookup, we could also split it into the two words ‫קו‬ ("line") and ‫שיכל‬ ("transposed"). another line in the same corpus contains the conflated string ‫,קריאהבמה‬ which can be split into ‫קריאה‬ ("reading") and ‫במה‬ ("in what"), or into ‫קריא‬ ("readable") and ‫הבמה‬ ("the stage"). hundreds of thousands of additional sentences within the hebrew portion of the oscar corpus are similarly corrupted. yet, the oscar internet crawl is the primary component of virtually all publically-available hebrew bert models, including hebert @xcite , ale-phbert @xcite , and alephbertgimmel @xcite . an efficient context-aware method is needed to fix this. in this paper, we present and evaluate a series of neural approaches for the restoration of the missing spaces within social-media corpora. ## task definition we formalize the space-restoration task as follows: given an input string s with characters c 1 ...c n where |s| = n, our goal is to predict a binary label for each c i , indicating whether a space should appear before the character at that position. this formulation treats the problem as a character-level sequence labeling task, which is particularly suitable for languages like hebrew where subword boundaries must be handled carefully. ## neural models for space restoration in this study, we develop and evaluate a series of neural models for the restoration of the missing spaces. ## results we evaluate each model's ability to accurately restore all missing spaces across the sentences in our test corpus. we compute precision, recall, and f1 score for restoring a space (a positive label). results are presented in ## conclusion almost all of the methods presented here provide decent accuracy on the space restoration task. however, because scraped social media corpora tend to be huge, the decoder-based methods are largely impractical, due to issues of speed (thus for the open dicta-lm model), or cost (thus regarding the com-mercial llm models). fortunately, the fine-tuned character-bert model which we presented here provides a practical solution: it is blazing fast, free for unrestricted use, and achieves accuracy which rivals the other methods. we are thus pleased to hereby release this model to the nlp community. furthermore, the character-based hebrew bert model which we pretrained as part of this project is released here as well, so that nlp researchers can continue to fine-tune it for other character-level nlp tasks as well. failures due to proper nouns: our model does not always recognize proper nouns for what they are, and sometimes attempts to divide them into two, especially when one (or both) of the resulting pieces is a common hebrew word. failures due to unusual grammatical suffixes: when the input text contains relatively long words which also contain a relatively rare grammatical suffix, our model is sometimes fooled and attempts to add a space before the grammatical suffix, as in the following examples: in light of these failures, practical use of the model would entail use of additional filters in order to restrain the model from splitting too eagerly. for instance, an ner model could be used to identify proper names in the text, and to restrain the space restoration model from splitting those names. similarly, in order to avoid the issue with grammatical suffixes, a script could check whether the letters after a word-split form of the few dozen sequences of letters which comprise hebrew grammatical suffixes; in such cases, it would be wise to ignore the additional space predicted by the model. your output must be the exact same sentence, in hebrew, with the only difference being the addition of the missing spaces. if there are no missing spaces, return the exact input sentence verbatim.