GPT -2 Metadata Pretraining Towards Instruction Finetuning for U krainian (2023)

Abstract: We explore pretraining unidirectional language models on 4B tokens from the largest curated corpus of Ukrainian, UberText 2.0. We enrich document text by surrounding it with weakly structured metadata, such as title, tags, and publication year, enabling metadata-conditioned text generation and text-conditioned metadata prediction at the same time. We pretrain GPT-2 Small, Medium and Large models each on single GPU, reporting training times, BPC on BrUK and BERTScore on titles for 1000 News from the Future. Next, we venture to formatting POS and NER datasets as instructions, and train low-rank attention adapters, performing these tasks as constrained text generation. We release our models for the community at https://github.com/proger/uk4b .

## introduction large language models provide a text-based user interface to perform multiple language processing tasks. the release of ubertext 2.0 @xcite is a milestone that unlocks pretraining experiments of language models on curated ukrainian texts. coupled with recent improvements to hardware and software, we can train larger models on a single consumer gpu from scratch. our contributions are: it's not only compute that's important. while the work of @xcite discovered prompts that drove the model to perform tasks like sum-marization, @xcite introduce pattern-exploiting training that reformulates sentences into cloze tasks on purpose. it is beneficial to curate examples of natural language instructions to save compute. instruction finetuning datasets, such as the flan collection, released by @xcite , curate massive amounts of task-specific datasets and provide a pipeline to reformulate tasks into natural language using seqio introduced in @xcite . flan t5 demonstrates that you can achieve higher performance on multiple nlp tasks at once with smaller models in 1.5b-11b range using such data curation methods. these ideas inspire us to leverage metadata and attempt to formulate nlp tasks using natural language. techniques like sequence length warmup @xcite , gradient clipping @xcite enable training stability. @xcite enable memory savings by quantizing gradient statistics. @xcite explore a recurrent formulation of attention with lower computational complexity, and @xcite view it as fast weight programmers improving capacity of attention in the recurrent setting. @xcite provide a programming language to implement high performing kernels quickly. @xcite demonstrate how to significantly speed up computation of self-attention and allow much larger context sizes than 1024 or 2048 tokens. finally, geiping and goldstein (2022) demonstrate achieving competitive pretraining speed and performance on a single gpu in 24 hours with a bert-like model. notably, these two advancements, the release of py-torch 2.0 and andrej karpathy's nanogpt tweets, encouraged us to try pretraining from scratch. low-rank adaptation methods presented in @xcite and extended in @xcite enable finetuning of large pretrained models on consumer hardware by updating only a small fraction of extra parameters, suggesting we can efficiently maintain adapters for many tasks in memory at once and achieve better finetuning performance. @xcite observe that smaller models optimize faster in the beginning of training and propose grafting parameters of a smaller network onto a larger one to continue training after some time. we keep this idea in mind for the future. ## discussion we are excited to release a new decoder-only monolingual model trained on curated ukrainian data to the community. it took us over a month to pretrain the first large model successfully and in the process we became aware of possible improvements to the model, such as removing biases. these improvements resulted in a narrow visual gap between medium and large, 7 we choose _ at random 1 . we were able to report a much lower validation loss on large due to a spike towards the end of training. loss curves in figure 1 , bpc values in while aiming towards a general purpose language agent trained on a single gpu, we are lured by simplicity of formatting tasks as instructions. during our experiments, we observed the model drifting away from the ner task into text generation on long inputs, requiring us to use constrained decoding to "remind" it what the model is supposed to be doing. we achieve a competitive result this way, however would still choose a more traditional approach to solve ner, as confirmed by our measurements in it is suprising to find that pos could be solved by "filling in blanks" by picking maximum probability tokens in parallel. we used that result in there is room for more data to faithfully leverage the prediction of the number of tokens we need to train for to optimally utilize compute. there is more available data in @xcite , @xcite and @xcite . we leave filtering this data to future work.