How to Train Data-Efficient LLMs

Published on Feb 15
· Featured in Daily Papers on Feb 16


The training of large language models (LLMs) is expensive. In this paper, we study data-efficient approaches for pre-training LLMs, i.e., techniques that aim to optimize the Pareto frontier of model quality and training resource/data consumption. We seek to understand the tradeoffs associated with data selection routines based on (i) expensive-to-compute data-quality estimates, and (ii) maximization of coverage and diversity-based measures in the feature space. Our first technique, Ask-LLM, leverages the zero-shot reasoning capabilities of instruction-tuned LLMs to directly assess the quality of a training example. To target coverage, we propose Density sampling, which models the data distribution to select a diverse sample. In our comparison of 19 samplers, involving hundreds of evaluation tasks and pre-training runs, we find that Ask-LLM and Density are the best methods in their respective categories. Coverage sampling can recover the performance of the full data, while models trained on Ask-LLM data consistently outperform full-data training -- even when we reject 90% of the original dataset, while converging up to 70% faster.


This is an automated message from the Librarian Bot. I found the following papers similar to this paper.

The following papers were recommended by the Semantic Scholar API

Please give a thumbs up to this comment if you found it helpful!

If you want recommendations for any Paper on Hugging Face checkout this Space

You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend

This was a fascinating read. Amazing research

@Noveen , have you considered fine-tuning an encoder based model with the data from "ASK-LLM Sampling" for cheaper classification? Also will you publish that dataset?

Really cool research! We need more papers like this targeting efficiencies and using simple techniques to do so.

Paper author

Thanks for reading and your kind words, Derek!

Good idea about fine-tuning a smaller model on these Ask-LLM scores - we haven't explored this as of yet. Regarding open-sourcing the Ask-LLM scores - I'm not sure whether that would be possible but I'll try to explore this further.

Sign up or log in to comment

Models citing this paper 0

No model linking this paper

Cite in a model to link it from this page.

Datasets citing this paper 1

Spaces citing this paper 0

No Space linking this paper

Cite in a Space to link it from this page.

Collections including this paper 20