Repository: huggingface/lighteval
Issue: [BUG] Python API docs generating splits forever
Root cause: Documentation Deficiency

## Describe the bug

Seen when running https://github.com/huggingface/lighteval/blob/v0.9.2/docs/source/using-the-python-api.mdx:

```none
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 8348.54 examples/s]
Generating auxiliary_train split: 100%|██████████| 99842/99842 [00:00<00:00, 734635.07 examples/s]
Generating test split: 100%|██████████| 102/102 [00:00<00:00, 53517.51 examples/s]
Generating validation split: 100%|██████████| 11/11 [00:00<00:00, 11630.29 examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 7508.60 examples/s]
Generating auxiliary_train split: 100%|██████████| 99842/99842 [00:00<00:00, 695671.32 examples/s]
Generating test split: 100%|██████████| 100/100 [00:00<00:00, 116282.34 examples/s]
Generating validation split: 100%|██████████| 11/11 [00:00<00:00, 13007.43 examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 7937.74 examples/s]
Generating auxiliary_train split: 100%|██████████| 99842/99842 [00:00<00:00, 823297.30 examples/s]
Generating test split: 100%|██████████| 235/235 [00:00<00:00, 252410.10 examples/s]
Generating validation split: 100%|██████████| 26/26 [00:00<00:00, 34174.84 examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 8760.03 examples/s]
Generating auxiliary_train split: 100%|██████████| 99842/99842 [00:00<00:00, 793201.77 examples/s]
Generating test split: 100%|██████████| 114/114 [00:00<00:00, 125301.53 examples/s]
Generating validation split: 100%|██████████| 12/12 [00:00<00:00, 16442.88 examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 8727.22 examples/s]
Generating auxiliary_train split: 100%|██████████| 99842/99842 [00:00<00:00, 546932.61 examples/s]
Generating test split: 100%|██████████| 145/145 [00:00<00:00, 103925.85 examples/s]
Generating validation split: 100%|██████████| 16/16 [00:00<00:00, 15917.66 examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 5739.33 examples/s]
Generating auxiliary_train split: 100%|██████████| 99842/99842 [00:00<00:00, 560104.46 examples/s]
Generating test split: 100%|██████████| 378/378 [00:00<00:00, 382127.48 examples/s]
Generating validation split: 100%|██████████| 41/41 [00:00<00:00, 53538.75 examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 8198.41 examples/s]
Generating auxiliary_train split: 100%|██████████| 99842/99842 [00:00<00:00, 658148.99 examples/s]
Generating test split: 100%|██████████| 126/126 [00:00<00:00, 95756.90 examples/s]
Generating validation split: 100%|██████████| 14/14 [00:00<00:00, 25137.10 examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 8504.27 examples/s]
Generating auxiliary_train split: 100%|██████████| 99842/99842 [00:00<00:00, 793643.73 examples/s]
Generating test split: 100%|██████████| 100/100 [00:00<00:00, 114786.64 examples/s]
Generating validation split: 100%|██████████| 10/10 [00:00<00:00, 16783.93 examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 8680.26 examples/s]
Generating auxiliary_train split: 100%|██████████| 99842/99842 [00:00<00:00, 730517.51 examples/s]
Generating test split: 100%|██████████| 310/310 [00:00<00:00, 309358.61 examples/s]
Generating validation split: 100%|██████████| 32/32 [00:00<00:00, 45206.38 examples/s]
```

## To Reproduce

Run the script without HF cache

## Expected behavior

Not to generate splits forever.

## Version info

`lighteval[math]==0.9.2`

Hi! This is not a bug, it's because the mmlu task has a lot of subsets and splits, and loads them all for evaluation (I think around 50 splits for which we load the different test/dev configs). 

Just change the evaluation to one with less subsets (or select a single subset from mmlu) and you should be good to go

Ah I see, thanks for clarifying, and appreciate the response!

What do you think of using a task in the API docs that is not so verbose/time intensive to download?

Ideally "getting started" docs can be run with minimal resources (including download time).

Sgtm! We could replace `mmlu` with `mmlu:abstract_algebra` (the first subset)