Repository: EleutherAI/lm-evaluation-harness
Issue: How to run bbh locally?
Root cause: Configuration Error

[BBH](https://huggingface.co/datasets/lukaemon/bbh) is a script-based dataset, but there is also a[ Parquet branch](https://huggingface.co/datasets/lukaemon/bbh/tree/refs%2Fconvert%2Fparquet).

However, using the Parquet branch will break the dataset's subsets. Originally, BBH contains many sub-tasks, but after loading the dataset, only the 'default' subset is available.

My expected attempt is that modifying the dataset_path in the following path will make it work properly.
`lm-evaluation-harness/lm_eval/tasks/bbh/cot_fewshot/_cot_fewshot_template_yaml`
```yaml
dataset_path: /mnt/dolphinfs/ssd_pool/docker/user/hadoop-aipnlp/BERT_TRAINING_SERVICE/platform/dataset/lukaemon/bbh/refs/convert/parquet
output_type: generate_until
test_split: test
doc_to_target: "{{target}}"
metric_list:
  - metric: exact_match
    aggregation: mean
    higher_is_better: true
    # ignore_case: true
    # ignore_punctuation: true
generation_kwargs:
  max_gen_toks: 1024
  until:
    - "
"
    - "Q"
    - "\n\n"
  do_sample: false
  temperature: 0.0
filter_list:
  - name: "get-answer"
    filter:
      - function: "regex"
        regex_pattern: "(?<=the answer is )(.*)(?=.)"
      - function: "take_first"
num_fewshot: 3
metadata:
  version: 3.0

```

However, all sub-tasks fail to load because the corresponding subsets cannot be retrieved.
```yaml
dataset_name: "boolean_expressions"
# dataset_name: "default"
description: "Evaluate the result of a random Boolean expression.\n\n"
doc_to_text: "Q: {{input}}\nA: Let's think step by step.\n"
include: "_cot_fewshot_template_yaml"
task: "bbh_cot_fewshot_boolean_expressions"
fewshot_config:
  sampler: first_n
  samples: [
    {
      "input": "not ( ( not not True ) ) is",
      "target": "Remember that (i) expressions inside brackets are always evaluated first and that (ii) the order of operations from highest priority to lowest priority is \"not\", \"and\", \"or\", respectively.\nWe first simplify this expression \"Z\" as follows: \"Z = not ( ( not not True ) ) = not ( ( A ) )\" where \"A = not not True\".\nLet's evaluate A: A = not not True = not (not True) = not False = True.\nPlugging in A, we get: Z = not ( ( A ) ) = not ( ( True ) ) = not True = False. So the answer is False."
    },
    {
      "input": "True and False and not True and True is",
      "target": "Remember that (i) expressions inside brackets are always evaluated first and that (ii) the order of operations from highest priority to lowest priority is \"not\", \"and\", \"or\", respectively.\nWe first simplify this expression \"Z\" as follows: \"Z = True and False and not True and True = A and B\" where \"A = True and False\" and \"B = not True and True\".\nLet's evaluate A: A = True and False = False.\nLet's evaluate B: B = not True and True = not (True and True) = not (True) = False.\nPlugging in A and B, we get: Z = A and B = False and False = False. So the answer is False."
    },
    {
      "input": "not not ( not ( False ) ) is",
      "target": "Remember that (i) expressions inside brackets are always evaluated first and that (ii) the order of operations from highest priority to lowest priority is \"not\", \"and\", \"or\", respectively.\nWe first simplify this expression \"Z\" as follows: \"Z = not not ( not ( False ) ) = not not ( A )\" where \"A = not ( False )\".\nLet's evaluate A: A = not ( False ) = not False = True.\nPlugging in A, we get: Z = not not ( A ) = not not (True) = not not False = True. So the answer is True."
    }
  ]


```

For someone like me, who is only concerned with the total score, I also tried a different approach: I commented out all the tasks, leaving only one sub-task, and then changed the dataset_name of that sub-task to 'default'. This allowed it to run, but the score was surprisingly low, and it seems like there are some parts I don't fully understand.
```yaml
group: bbh
task:
  # - bbh_cot_fewshot_boolean_expressions
  # - bbh_cot_fewshot_causal_judgement
  # - bbh_cot_fewshot_date_understanding
  # - bbh_cot_fewshot_disambiguation_qa
  # - bbh_cot_fewshot_dyck_languages
  # - bbh_cot_fewshot_formal_fallacies
  # - bbh_cot_fewshot_geometric_shapes
  # - bbh_cot_fewshot_hyperbaton
  # - bbh_cot_fewshot_logical_deduction_five_objects
  # - bbh_cot_fewshot_logical_deduction_seven_objects
  # - bbh_cot_fewshot_logical_deduction_three_objects
  # - bbh_cot_fewshot_movie_recommendation
  # - bbh_cot_fewshot_multistep_arithmetic_two
  # - bbh_cot_fewshot_navigate
  # - bbh_cot_fewshot_object_counting
  # - bbh_cot_fewshot_penguins_in_a_table
  # - bbh_cot_fewshot_reasoning_about_colored_objects
  # - bbh_cot_fewshot_ruin_names
  # - bbh_cot_fewshot_salient_translation_error_detection
  # - bbh_cot_fewshot_snarks
  # - bbh_cot_fewshot_sports_understanding
  # - bbh_cot_fewshot_temporal_sequences
  # - bbh_cot_fewshot_tracking_shuffled_objects_five_objects
  # - bbh_cot_fewshot_tracking_shuffled_objects_seven_objects
  # - bbh_cot_fewshot_tracking_shuffled_objects_three_objects
  # - bbh_cot_fewshot_web_of_lies
  - bbh_cot_fewshot_word_sorting
aggregate_metric_list:
  - metric: exact_match
    aggregation: mean
    weight_by_size: true
    filter_list: get-answer
metadata:
  version: 3.0

```


switchover to https://huggingface.co/datasets/SaylorTwift/bbh which has individual parquet files for each subset!