Canine: Pre-training an Efficient Tokenization-Free Encoder for Language Representation (2022)

Abstract: Pipelined NLP systems have largely been superseded by end-to-end neural modeling, yet nearly all commonly used models still require an explicit tokenization step. While recent tokenization approaches based on data-derived subword lexicons are less brittle than manually engineered tokenizers, these techniques are not equally suited to all languages, and the use of any fixed vocabulary may limit a model’s ability to adapt. In this paper, we present Canine, a neural encoder that operates directly on character sequences—without explicit tokenization or vocabulary—and a pre-training strategy that operates either directly on characters or optionally uses subwords as a soft inductive bias. To use its finer-grained input effectively and efficiently, Canine combines downsampling, which reduces the input sequence length, with a deep transformer stack, which encodes context. Canine outperforms a comparable mBert model by 5.7 F1 on TyDi QA, a challenging multilingual benchmark, despite having fewer model parameters.

## introduction end-to-end neural models have generally replaced the traditional nlp pipeline, and with it, the error cascades and feature engineering common to such systems, preferring instead to let the model automatically induce its own sophisticated representations. tokenization, however, is one of the few holdovers from that era, with nearly all commonly used models today requiring an explicit preprocessing stage to segment a raw text string into a sequence of discrete model inputs. broadly speaking, tokenizers are generally either carefully constructed systems of language-specific rules, which are costly, requiring both manual feature engineering and linguistic expertise, or datadriven algorithms such as byte pair encoding @xcite , wordpiece @xcite , or sentencepiece @xcite that split strings based on frequencies in a corpus, which are less brittle and easier to scale, but are ultimately too simplistic to properly handle the wide range of linguistic phenomena that can't be captured by mere string-splitting ( §2.1). the degree of sophistication required to accurately capture the full breadth of linguistic phenomena, along with the infeasibility of writing such rules by hand across all languages and domains, suggests that explicit tokenization itself is problematic. in contrast, an end-to-end model that operates directly on raw text strings would avoid these issues, instead learning to compose individual characters into its own arbitrarily complex features, with potential benefits for both accuracy and ease of use. while this change is conceptually very simple-one could replace the subword vocabulary in a model like bert @xcite with a vocabulary made solely of individual characters-doing so leads to two immediate problems. first, the computational complexity of a transformer @xcite , the main component in bert as well as other models such as gpt @xcite and t5 @xcite , grows quadratically with the length of the input. since standard subword models have roughly four characters per subword on average, the 4x increase in input sequence length would result is a significantly slower model. second, simply switching to a character vocabulary yields empirically poor results ( §4.2). in order to enable tokenization-free modeling that overcomes these obstacles, we present canine. canine is a large language encoder with a deep transformer stack at its core. inputs to the model are sequences of unicode characters. 1 to represent the full space of unicode characters foot_1 without a vocabulary, we use a hashing strategy. to avoid the slowdown from increasing the sequence length, canine uses strided convolutions to downsample input sequences to a shorter length before the deep transformer stack. like bert, we pre-train canine on the masked language model (mlm) and next sentence prediction (nsp) tasks. for the mlm task, canine offers two options: in this article, we contribute: ## canine canine consists of three primary components: ## conclusion in this article, we described canine, which is, to our knowledge, the first pre-trained deep encoder for language understanding that uses a tokenization-free, vocabulary-free model, while surpassing the quality of models built on top of heuristic tokenizers. canine eliminates many engineering pitfalls for practitioners and opens up new research directions for the community.