Repository: EleutherAI/lm-evaluation-harness
Issue: Is there a way to cache the building of datasets?
Root cause: Unimplemented Feature Gap

Hey all, once again, excellent framework.

I'm using the framework programatically.

I wrote code to do this myself, but I want to make sure I'm not being stupid and wanted to check that the package doesn't already support this.

I know it supports the caching of results from LLMs that have already been run, but don't see caching for anything else. Although there's a lot of code here to understand. So with that said, I've got code that caches the building/registration of tasks, groups, datasets before preprocessing, and datasets after post processing (request dicts).

Reduces the spin up time to start testing a model to about a second or so and that's a big win for my use case.

I know that PRing can be a bit of a hassle at times, but if this isn't already part of the package, if you all think it would be worthwhile for me to PR, would love to.

The basics of my proposal are to just add a .cache directory, a caching module, and then to store all of the cached files in the .cache directory. (They'd all be pickles)

All functions involving a cache then have a kwarg on them to allow for rewriting the cache if necessary.

A future enhancement would be to also rewrite the cache automatically if any of the target entities change (tasks, groups) and any subsequent caches that depend on them, using a hash reduced from the last time a file was modified, if any file changed, full logic runs and the cache is refreshed, or whatever the fastest/most robust approach would be.

But the ideal would be the user doesn't have to manage that themselves because I can see a few shooting oneself in the foot scenarios happening.

Hi! Thank you for using the framework and for being interested in contributing!

I think it'd certainly be worthwhile to put that caching code in a fork and we can look to see if it makes sense to PR it. 

I think this would be a huge improvement for the mentioned reasons (minimizing non-inference runtimes and making rerunning evals much faster) but that we'd want to be very careful to minimize the amount that this causes bugs or prevents a user from observing an expected change when debugging things. One way to minimize shooting oneself in the foot would be to potentially make this, like model output caching, strictly opt-in

I also outlined a potential less invasive or less codechange-intensive version of this in #1286 , where we could piggyback off of HF datasets' existing caching tools to cache at least some intermediate results ( a large amount of slowness is due to frequent Jinja2 template rendering)-- I think this might achieve a good portion of the benefits while still getting a significant speedup.



In a similar vein, I'm curious if this still works (from the main README.md):

`python write_out.py \
    --tasks all_tasks \
    --num_fewshot 5 \
    --num_examples 10 \
    --output_base_path /path/to/output/folder`

I'm trying to a variant of this command to run+download all the benchmarks for local offline usage later, but this functionality seems to be faulty now, depsite being in the example usage. The error I get is:

`File "/home/user/.conda/envs/conda_env/lib/python3.11/site-packages/lm_eval/tasks/__init__.py", line 221, in get_task
    raise KeyError(f"Missing task {task_name}")
KeyError: 'Missing task all_tasks'`

Would this be a viable way to get all of them downloaded?



> I'm curious if this still works (from the main README.md):

Ah sorry, I will clarify this in the README--`all_tasks` is a placeholder for `<the list of task names you care about>`.

If you truly want to download *all* the tasks, I think you can do so via `--tasks *` to pattern-match to everything. I'd recommend using this to download just the tasks you care about as it'll save storage + time, and then to use HF datasets and transformers offline settings to run: https://huggingface.co/docs/datasets/v2.16.1/loading#offline

> Ah sorry, I will clarify this in the README--`all_tasks` is a placeholder for `<the list of task names you care about>`.

Oh, thank you for that clarification. I guess I'm used to seeing placeholders marked \<like so\> haha. Your tip looks great though. I will go with that, and make and edit here if I hit any bumps. Thanks for the timely response!





Aight peeps, I submitted the PR, may our odds ever be in the favor of our wonderful maintainers!

I think the cache is a great thing but it missed the biggest problem. The lm-evaluation-harness takes longer to "redownload" and "review" the remote datasets for somethig like arc_challenge than to evaluate the model itself.
It's a huge burden if you just want to do a quick evaluation of a couple hundred questions - 3 minutes of "dataset viewer" and 10 seconds of evaluation time.