MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformers
Abstract
Autoregressive transformers are spectacular models for short sequences but scale poorly to long sequences such as high-resolution images, podcasts, code, or books. We proposed Megabyte, a multi-scale decoder architecture that enables end-to-end differentiable modeling of sequences of over one million bytes. Megabyte segments sequences into patches and uses a local submodel within patches and a global model between patches. This enables sub-quadratic self-attention, much larger feedforward layers for the same compute, and improved parallelism during decoding -- unlocking better performance at reduced cost for both training and generation. Extensive experiments show that Megabyte allows byte-level models to perform competitively with subword models on long context language modeling, achieve state-of-the-art density estimation on ImageNet, and model audio from raw files. Together, these results establish the viability of tokenization-free autoregressive sequence modeling at scale.
Community
๐๐คฉ
๐๐ฆ
Reminds me pretty strongly of the Hierarchical Perceiver architecture.
It would be interesting to see tests with long-range dependencies, rare tokens, or complex patterns in the data.
I have read this paper and run the demo, It seem this model only get a good performence in char level. I doubt the performence in downstream task
This paper was cited by @karpathy in the Tokenizer Tutorial: https://youtu.be/zduSFxRajkE?si=qthqUglGA0qc8Z0F
Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper