Repository: stanford-crfm/helm
Issue: Missing expand function in ModelRunExpander and NumTrainTrialsRunExpander, results to empty run_specs
Root cause: Documentation Deficiency

Hello team!

I get

```
16 entries produced 0 run specs
There were no RunSpecs or they got filtered out.
```

when executing the config for OpinionsQA. 
I want to query models myself, so I want to find out how we can get non-empty `run_specs`. Can you help me this this? I've described the details below.

__________
My goal is to execute the experiments on the paper "Whose Opinions Do Language Models Reflect?", by querying models myself.

The paper references the code: https://github.com/tatsu-lab/opinions_qa
It provides us precomputed results so that we can try using their metrics.

For me to actually compute the results, it refers us to: https://github.com/stanford-crfm/helm
and tells us to run
```
helm-run -c src/helm/benchmark/presentation/run_specs_opinions_qa_openai_default.conf --max-eval-instances 500 --suite $SUITE
helm-run -c src/helm/benchmark/presentation/run_specs_opinions_qa_ai21_default.conf --max-eval-instances 500 --suite $SUITE
helm-run -c src/helm/benchmark/presentation/run_specs_opinions_qa_openai_steer.conf --max-eval-instances 50000 --suite $SUITE
helm-run -c src/helm/benchmark/presentation/run_specs_opinions_qa_ai21_steer.conf --max-eval-instances 50000 --suite $SUITE

```
So, I ran one line of that to begin with.  (I changed a portion of the file name specs --> entries, reason explained at the very bottom)
`helm-run -c src/helm/benchmark/presentation/run_specs_opinions_qa_openai_default.conf --max-eval-instances 500 --suite $SUITE`

run_spec_factory.py has a portion:
https://github.com/stanford-crfm/helm/blob/799b7e3c640723da6783a5010da5ee3b4fce7989/src/helm/benchmark/run_spec_factory.py#L68-L72

```
    print("run_specs before expander", run_specs)    # I added this line

    # Apply expanders
    for expander in expanders:
        print("name:", expander.name) # I added this line
        run_specs = [
            child_run_spec for parent_run_spec in run_specs for child_run_spec in expander.expand(parent_run_spec)
        ]
  
    print("run_specs after expander", run_specs) # I added this line
```

`run_specs` has class object before the for loop, 

> run_specs before expander [RunSpec(name='opinions_qa:survey=Pew_American_Trends_Panel_W82,num_logprobs=100,context=default,num_train_trials=1', scenario_spec=ScenarioSpec(class_name='helm.benchmark.scenarios.opinions_qa_scenario.OpinionsQAScenario', args={'survey_type': 'Pew_American_Trends_Panel_W82', 'context': 'default'}), adapter_spec=AdapterSpec(method='multiple_choice_joint', global_prefix='', global_suffix='', instructions='', input_prefix='Question: ', input_suffix='\n', reference_prefix='A. ', reference_suffix='\n', output_prefix='Answer: ', output_suffix='\n', instance_prefix='\n', substitutions=[], max_train_instances=0, max_eval_instances=None, num_outputs=100, num_train_trials=1, num_trials=1, sample_train=False, model_deployment='', model='', temperature=0.0, max_tokens=1, stop_sequences=['\n'], random=None, multi_label=False, image_generation_parameters=None, eval_splits=None), metric_specs=[], data_augmenter_spec=DataAugmenterSpec(perturbation_specs=[], should_augment_train_instances=False, should_include_original_train=False, should_skip_unchanged_train=False, should_augment_eval_instances=False, should_include_original_eval=False, should_skip_unchanged_eval=False, seeds_per_instance=1), groups=['opinions_qa'], annotators=None)]


but it is empty after the for loop.
>run_specs after expander []

We know that we do go into the for loop because the expanders have 2 objects 
>list of expanders: [<helm.benchmark.run_expander.ModelRunExpander object at 0x10801e3a0>, <helm.benchmark.run_expander.NumTrainTrialsRunExpander object at 0x10801e5b0>]

and because we can print expander.name 
>name: model

>name: num_train_trials

Something wrong is happening when computing `expander.expand(parent_run_spec)`.


A few lines prior, expander is defined as 
`expanders = [RUN_EXPANDERS[key](value) for key, value in args.items() if key in RUN_EXPANDERS]`
https://github.com/stanford-crfm/helm/blob/799b7e3c640723da6783a5010da5ee3b4fce7989/src/helm/benchmark/run_spec_factory.py#L63


`RUN_EXPANDERS` comes from run_expander.py in the same directory
It is defined as a list of subclasses
https://github.com/stanford-crfm/helm/blob/799b7e3c640723da6783a5010da5ee3b4fce7989/src/helm/benchmark/run_expander.py#L1575-L1605

The subclasses we are using are: `ModelRunExpander` and `NumTrainTrialsRunExpander`.
We know this from the names I printed before. `“model”`, `“num_train_trials”`

The problem is, that there is **no expand function defined in those classes**. There is no other place in the code that defined expand function.
https://github.com/stanford-crfm/helm/blob/799b7e3c640723da6783a5010da5ee3b4fce7989/src/helm/benchmark/run_expander.py#L573-L636
https://github.com/stanford-crfm/helm/blob/799b7e3c640723da6783a5010da5ee3b4fce7989/src/helm/benchmark/run_expander.py#L516-L526

That is why, the code does not proceed to the next step, ending the run with a message: 
>16 entries produced 0 run specs
>There were no RunSpecs or they got filtered out.



Can you help me solve this?

__________
Similar to this issue: https://github.com/stanford-crfm/helm/issues/2948
This issue solves the problem by adding:
```
register_builtin_configs_from_helm_package()
register_configs_from_directory(args.local_path)
```
The current version already has the above included, but the problem still exists.
https://github.com/stanford-crfm/helm/blob/799b7e3c640723da6783a5010da5ee3b4fce7989/src/helm/benchmark/run.py#L273-L274
__________

Here is what I did before running the config line, aligning with the document and the current versions of what needs to be installed.
https://crfm-helm.readthedocs.io/en/latest/

```
# Create a virtual environment
python3 -m pip install virtualenv
python3 -m virtualenv -p python3.9 helm-venv

# Activate
source helm-venv/bin/activate. 

# install HELM
pip install crfm-helm

# install wget
brew install wget

# install urllib3
pip install urllib3==1.26.6

```
Make sure that the config file is in the right directory
https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/presentation/run_entries_opinions_qa_openai_default.conf

```
# Run Opinions api config file
helm-run -c src/helm/benchmark/presentation/run_entries_opinions_qa_openai_default.conf --max-eval-instances 10 --suite my-suite
```
The config line is mentioned in https://github.com/tatsu-lab/opinions_qa?tab=readme-ov-file, but the file name change is requested on a pull request https://github.com/tatsu-lab/opinions_qa/pull/4/files. The updated config files are in https://github.com/stanford-crfm/helm/tree/main/src/helm/benchmark/presentation 

__________
Thank you team for all your work! I enjoyed reading the paper and following the code! 


@carolina-museum 

> The subclasses we are using are: ModelRunExpander and NumTrainTrialsRunExpander.
> The problem is, that there is no expand function defined in those classes.

They both inherit from [ReplaceValueRunExpander](https://github.com/stanford-crfm/helm/blob/799b7e3c640723da6783a5010da5ee3b4fce7989/src/helm/benchmark/run_expander.py#L45), which defines the `expand()` method used by its subclasses. Please refer to the class definition for its concrete implementation.

If `expand()` were somehow undefined, calling it should raise an error rather than resulting in an empty list. Since that isn't happening, we can reasonably assume that the subclasses do implement the `expand()` method.

```python
class A:
    pass


a = A()
a.expand()  # AttributeError: 'A' object has no attribute 'expand'
```

That being said, I don't believe the issue is due to a missing `expand()` method. Could you please double-check?

Hi @carolina-museum, the main issue with is the `model=opinions_qa_openai` and `model=opinions_qa_ai21` strings in the run entries file. `opinions_qa_openai` and `opinions_qa_ai21` both [refer to groups of models](https://github.com/stanford-crfm/helm/blob/baf792b58baf1fc7ae280d6bd80e547d348a5b1a/src/helm/benchmark/run_expander.py#L605-L613) that consist entirely of models that have been deprecated by OpenAI and AI21 and are no longer accessible. As such, those models are no longer usable from HELM. We should probably return a more helpful error message in this situation.

To get this to work, change `model=opinions_qa_openai` or `model=opinions_qa_ai21` to specify the model you want to evaluate, e.g. `model=openai/gpt2`.

In general, the OpinionQA code in this repository is unmaintained and I would not expect it to work out of the box. Some potential issues include:

1. The original OpinionsQA code was specialized to only work on specific OpenAI and AI21 models (which explains the the buggy run entries files).
2. Some of the logprobs handling code in HELM has changed; in particular, some clients (e.g. `OpenAIClient`) have incorrect logprobs handling.
3. If you are working with API models, most of them do not provide logprobs, so a sampling technique is required instead, which is not supported by the existing code.

I'd recommend contacting the OpinionQA lead author @ShibaniSanturkar for advice.

@yifanmai Thank you for your quick response! This is super helpful.
I am working on it right now, so I will update you once I make some progress. Thank you!

@huyfififi Thank you for looking into this and giving me a clear explanation. That helps me a lot!

Thank you for all your help!
I tried, but it was very difficult for me to reach my goal😢 but this process was really helpful for me, so I'd like to share what I tried today.
_____________

Here are the changes I made after getting the replies.

In this portion of opinions_qa_scenario.py, I added one line that includes a folder "data" in the end, after downloading the data, because path seemed to be inconsistent.

https://github.com/stanford-crfm/helm/blob/0bc1f42e11c43ae9bea601354f9635d39ca2947f/src/helm/benchmark/scenarios/opinions_qa_scenario.py#L130-L144
```
def get_instances(self, output_path: str) -> List[Instance]:
        
        self.download_data(output_path)
    
        # Read all the instances
        instances: List[Instance] = []
        splits: Dict[str, str] = {
            "dev": TRAIN_SPLIT,
            "test": TEST_SPLIT,
        }

        output_path = os.path.join(output_path, "data")  # I added this line!
        all_splits = ["dev", "test"] if self.context == "steer-qa" else ["test"]
        csv_dict = {
            "dev": os.path.join(output_path, f"{self.context}.csv"),
            "test": os.path.join(output_path, f"{self.survey_type}.csv"),
        }

```

<br>

In this portion of run_expander.py, I commented out all the models and inserted `model=openai/gpt2` as @yifanmai suggested.
https://github.com/stanford-crfm/helm/blob/0bc1f42e11c43ae9bea601354f9635d39ca2947f/src/helm/benchmark/run_expander.py#L605-L612
```
 "opinions_qa_openai": [
                "openai/gpt2",  # This line is added
                # "openai/ada",
                # "openai/davinci",
                # "openai/text-ada-001",
                # "openai/text-davinci-001",
                # "openai/text-davinci-002",
                # "openai/text-davinci-003",
            ],
```

Also, because I am using gpt2 model, I now run:
`helm-run -c venv-0218/helm-main/src/helm/benchmark/presentation/run_entries_opinions_qa_openai_default.conf --max-eval-instances 10 --suite my-suite`

I also had to install gdown.
So, install it before running the command above.
`pip install gdown`
gdown is used in opinions_qa_scenario.py, but I still had to install it even though it is listed in requirements.txt.
https://github.com/stanford-crfm/helm/blob/0bc1f42e11c43ae9bea601354f9635d39ca2947f/requirements.txt


Then, I was able to get to the point where I got this log message (below is the last few lines of the message):
```
    Writing 1490 characters to benchmark_output/runs/my-suite/opinions_qa:survey=Pew_American_Trends_Panel_W92,num_logprobs=100,context=default,num_train_trials=1,model=openai_gpt2,num_train_trials=1/run_spec.json
    Writing 290 characters to benchmark_output/runs/my-suite/opinions_qa:survey=Pew_American_Trends_Panel_W92,num_logprobs=100,context=default,num_train_trials=1,model=openai_gpt2,num_train_trials=1/scenario.json
    Writing 24142 characters to benchmark_output/runs/my-suite/opinions_qa:survey=Pew_American_Trends_Panel_W92,num_logprobs=100,context=default,num_train_trials=1,model=openai_gpt2,num_train_trials=1/scenario_state.json
    Writing 2 characters to benchmark_output/runs/my-suite/opinions_qa:survey=Pew_American_Trends_Panel_W92,num_logprobs=100,context=default,num_train_trials=1,model=openai_gpt2,num_train_trials=1/stats.json
    Writing 2 characters to benchmark_output/runs/my-suite/opinions_qa:survey=Pew_American_Trends_Panel_W92,num_logprobs=100,context=default,num_train_trials=1,model=openai_gpt2,num_train_trials=1/per_instance_stats.json
    CacheStats.print_status {
      prod_env/cache/huggingface.sqlite: 798 queries, 790 computes
    } [0.0s]
  } [0.638s]
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 16/16 [00:17<00:00,  1.07s/it]
  Done.
```
The problem is the number of characters is only 2 for stats.json and per_instance_stats.jon. and when I look into it, the two characters are --> [].
This means that I am not able to get the results that I want to use for evaluating representativeness, steerability, and consistency. 

______________

As suggested in @yifanmai 's reply and #2444, it is very difficult to reproduce the results of the paper. 

My options for the next step is to write my own scenarios and run using helm, or make something simpler that imitates the experiments in the paper "Whose Opinions Do Language Models Reflect?".

Also, my initial thought that the expand function is missing was not accurate.
So, I'm closing this issue 👍

_______________


Thank you all for supporting me to go through this project. I learned a lot and gained good experience!

Hi @carolina-museum, thanks for following up with your observations - this was very helpful!