Repository: stanford-crfm/helm
Issue: How to run benchmarks on local lora model
Root cause: Environment Incompatibility

Hi, 

I would like to run local trained unmerged lora model. Previously I had working setup for full model and now when I use it for lora trained model (adapter_model.safetensors) it does not work.
`OSError: Error no file named pytorch_model.bin, model.safetensors, tf_model.h5, model.ckpt.index or flax_model.msgpack found in directory /absolute_path_to_model/lora-out.
`
 I checked the documentation and I didn't find anything useful. Can you please help me to figure it out ? Thanks

model_deployments.yaml
```
model_deployments:
  - name: huggingface/lora-out
    model_name: psa/lora-out
    tokenizer_name: psa/lora-out
    max_sequence_length: 131072
    client_spec:
      class_name: "helm.clients.huggingface_client.HuggingFaceClient"
      args:
        pretrained_model_name_or_path: /absolute_path_to_model/lora-out 
        device_map: auto
```
model_metadata.yaml
```
models:
  - name: psa/lora-out
    display_name: lora-out # Main model name in benchmark
    description: something
    creator_organization_name: psa 
    access: Limited
    num_parameters: 8043892736
    release_date: 2024-08-13
    tags: [TEXT_MODEL_TAG, PARTIAL_FUNCTIONALITY_TEXT_MODEL_TAG]
```
tokenizer_configs.yaml
```
tokenizer_configs:
  - name: psa/lora-out
    tokenizer_spec:
      class_name: "helm.tokenizers.huggingface_tokenizer.HuggingFaceTokenizer"
      args:
        pretrained_model_name_or_path: /absolute_path_to_model/lora-out
```

```
export SCHEMA_PATH=src/helm/benchmark/static/schema_lite.yaml
export RUN_ENTRIES_CONF_PATH=src/helm/benchmark/presentation/run_entries_lite_20240424.conf
export NUM_TRAIN_TRIALS=1
export MAX_EVAL_INSTANCES=1000
export PRIORITY=2
export SUITE_NAME=my-suite
export MODELS_TO_RUN=psa/lora-out

helm-run --conf-paths $RUN_ENTRIES_CONF_PATH --num-train-trials $NUM_TRAIN_TRIALS --max-eval-instances $MAX_EVAL_INSTANCES --priority $PRIORITY --suite $SUITE_NAME --models-to-run $MODELS_TO_RUN
```

Hmm, I don't think I've encountered this error before. Are you able to load your model using `transformers` in your Python environment, or do you get the same errors?

```python3
from transformers import AutoModelForCausalLM

AutoModelForCausalLM.from_pretrained("/absolute_path_to_model/lora-out", device_map="auto")
```

Hi Yifanmai,

thank you for your reply. For sure it works in python/transformers like a charm. 
Please note it is local model on the harddrive not in the huggingfacehub nor cache.


Could you also post your full stack trace?

Are you using the same Python environment for trying it out in Python / transformers and for running it in HELM?

In general, we support loading local Hugging Face (not Hub) models in HELM, so this is not expected.

Hi yifanmai, 

I know local model works, but what about local adapter model with local base model.
There is an output:
```
export MODELS_TO_RUN=psa/lora-out
export SCHEMA_PATH=src/helm/benchmark/static/schema_lite.yaml
export RUN_ENTRIES_CONF_PATH=src/helm/benchmark/presentation/run_entries_lite_20240424.conf
export NUM_TRAIN_TRIALS=1
export MAX_EVAL_INSTANCES=1000
export PRIORITY=2
export SUITE_NAME=my-suite


helm-run --conf-paths $RUN_ENTRIES_CONF_PATH --num-train-trials $NUM_TRAIN_TRIALS --max-eval-instances $MAX_EVAL_INSTANCES --priority $PRIORITY --suite $SUITE_NAME --models-to-run $MODELS_TO_RUN
main {
  Reading tokenizer configs from /home/user/miniforge3/envs/helm/lib/python3.9/site-packages/helm/config/tokenizer_configs.yaml...
  Reading model deployments from /home/user/miniforge3/envs/helm/lib/python3.9/site-packages/helm/config/model_deployments.yaml...
  Reading tokenizer configs from prod_env/tokenizer_configs.yaml...
  Reading model deployments from prod_env/model_deployments.yaml...
  Read 108 run entries from src/helm/benchmark/presentation/run_entries_lite_20240424.conf
  ensure_file_downloaded {
    Not downloading https://raw.githubusercontent.com/HazyResearch/legalbench/main/helm_prompt_settings.jsonl because benchmark_output/scenarios/legalbench/prompt_construction_settings.json already exists
  } [0.0s]
  ensure_file_downloaded {
    Not downloading https://raw.githubusercontent.com/HazyResearch/legalbench/main/helm_prompt_settings.jsonl because benchmark_output/scenarios/legalbench/prompt_construction_settings.json already exists
  } [0.0s]
  ensure_file_downloaded {
    Not downloading https://raw.githubusercontent.com/HazyResearch/legalbench/main/helm_prompt_settings.jsonl because benchmark_output/scenarios/legalbench/prompt_construction_settings.json already exists
  } [0.0s]
  ensure_file_downloaded {
    Not downloading https://raw.githubusercontent.com/HazyResearch/legalbench/main/helm_prompt_settings.jsonl because benchmark_output/scenarios/legalbench/prompt_construction_settings.json already exists
  } [0.0s]
  ensure_file_downloaded {
    Not downloading https://raw.githubusercontent.com/HazyResearch/legalbench/main/helm_prompt_settings.jsonl because benchmark_output/scenarios/legalbench/prompt_construction_settings.json already exists
  } [0.0s]
  ensure_file_downloaded {
    Not downloading https://raw.githubusercontent.com/HazyResearch/legalbench/main/helm_prompt_settings.jsonl because benchmark_output/scenarios/legalbench/prompt_construction_settings.json already exists
  } [0.0s]
  ensure_file_downloaded {
    Not downloading https://raw.githubusercontent.com/HazyResearch/legalbench/main/helm_prompt_settings.jsonl because benchmark_output/scenarios/legalbench/prompt_construction_settings.json already exists
  } [0.0s]
  ensure_file_downloaded {
    Not downloading https://raw.githubusercontent.com/HazyResearch/legalbench/main/helm_prompt_settings.jsonl because benchmark_output/scenarios/legalbench/prompt_construction_settings.json already exists
  } [0.0s]
  ensure_file_downloaded {
    Not downloading https://raw.githubusercontent.com/HazyResearch/legalbench/main/helm_prompt_settings.jsonl because benchmark_output/scenarios/legalbench/prompt_construction_settings.json already exists
  } [0.0s]
  ensure_file_downloaded {
    Not downloading https://raw.githubusercontent.com/HazyResearch/legalbench/main/helm_prompt_settings.jsonl because benchmark_output/scenarios/legalbench/prompt_construction_settings.json already exists
  } [0.0s]
  108 entries produced 28 run specs
  run_specs {
    RunSpec(name='narrative_qa:model=psa_lora-out', scenario_spec=ScenarioSpec(class_name='helm.benchmark.scenarios.narrativeqa_scenario.NarrativeQAScenario', args={}), adapter_spec=AdapterSpec(method='generation', global_prefix='', global_suffix='', instructions='', input_prefix='Passage: ', input_suffix='\n', reference_prefix='A. ', reference_suffix='\n', output_prefix='Answer: ', output_suffix='\n', instance_prefix='\n', substitutions=[], max_train_instances=5, max_eval_instances=1000, num_outputs=1, num_train_trials=1, num_trials=1, sample_train=True, model_deployment='huggingface/lora-out', model='psa/lora-out', temperature=0.0, max_tokens=100, stop_sequences=['\n'], random=None, multi_label=False, image_generation_parameters=None, eval_splits=None), metric_specs=[MetricSpec(class_name='helm.benchmark.metrics.basic_metrics.BasicGenerationMetric', args={'names': ['exact_match', 'quasi_exact_match', 'f1_score', 'rouge_l', 'bleu_1', 'bleu_4']}), MetricSpec(class_name='helm.benchmark.metrics.basic_metrics.BasicReferenceMetric', args={}), MetricSpec(class_name='helm.benchmark.metrics.basic_metrics.InstancesPerSplitMetric', args={})], data_augmenter_spec=DataAugmenterSpec(perturbation_specs=[], should_augment_train_instances=False, should_include_original_train=False, should_skip_unchanged_train=False, should_augment_eval_instances=False, should_include_original_eval=False, should_skip_unchanged_eval=False, seeds_per_instance=1), groups=['narrative_qa'], annotators=None)
    RunSpec(name='natural_qa:mode=openbook_longans,model=psa_lora-out', scenario_spec=ScenarioSpec(class_name='helm.benchmark.scenarios.natural_qa_scenario.NaturalQAScenario', args={'mode': 'openbook_longans'}), adapter_spec=AdapterSpec(method='generation', global_prefix='', global_suffix='', instructions='', input_prefix='', input_suffix='\n', reference_prefix='A. ', reference_suffix='\n', output_prefix='Answer: ', output_suffix='\n', instance_prefix='\n', substitutions=[], max_train_instances=5, max_eval_instances=1000, num_outputs=1, num_train_trials=1, num_trials=1, sample_train=True, model_deployment='huggingface/lora-out', model='psa/lora-out', temperature=0.0, max_tokens=300, stop_sequences=['\n'], random=None, multi_label=False, image_generation_parameters=None, eval_splits=None), metric_specs=[MetricSpec(class_name='helm.benchmark.metrics.basic_metrics.BasicGenerationMetric', args={'names': ['exact_match', 'quasi_exact_match', 'f1_score']}), MetricSpec(class_name='helm.benchmark.metrics.basic_metrics.BasicReferenceMetric', args={}), MetricSpec(class_name='helm.benchmark.metrics.basic_metrics.InstancesPerSplitMetric', args={})], data_augmenter_spec=DataAugmenterSpec(perturbation_specs=[], should_augment_train_instances=False, should_include_original_train=False, should_skip_unchanged_train=False, should_augment_eval_instances=False, should_include_original_eval=False, should_skip_unchanged_eval=False, seeds_per_instance=1), groups=['natural_qa_openbook_longans'], annotators=None)
    RunSpec(name='natural_qa:mode=closedbook,model=psa_lora-out', scenario_spec=ScenarioSpec(class_name='helm.benchmark.scenarios.natural_qa_scenario.NaturalQAScenario', args={'mode': 'closedbook'}), adapter_spec=AdapterSpec(method='generation', global_prefix='', global_suffix='', instructions='', input_prefix='Question: ', input_suffix='\n', reference_prefix='A. ', reference_suffix='\n', output_prefix='Answer: ', output_suffix='\n', instance_prefix='\n', substitutions=[], max_train_instances=5, max_eval_instances=1000, num_outputs=1, num_train_trials=1, num_trials=1, sample_train=True, model_deployment='huggingface/lora-out', model='psa/lora-out', temperature=0.0, max_tokens=300, stop_sequences=['\n'], random=None, multi_label=False, image_generation_parameters=None, eval_splits=None), metric_specs=[MetricSpec(class_name='helm.benchmark.metrics.basic_metrics.BasicGenerationMetric', args={'names': ['exact_match', 'quasi_exact_match', 'f1_score']}), MetricSpec(class_name='helm.benchmark.metrics.basic_metrics.BasicReferenceMetric', args={}), MetricSpec(class_name='helm.benchmark.metrics.basic_metrics.InstancesPerSplitMetric', args={})], data_augmenter_spec=DataAugmenterSpec(perturbation_specs=[], should_augment_train_instances=False, should_include_original_train=False, should_skip_unchanged_train=False, should_augment_eval_instances=False, should_include_original_eval=False, should_skip_unchanged_eval=False, seeds_per_instance=1), groups=['natural_qa_closedbook'], annotators=None)
    RunSpec(name='commonsense:dataset=openbookqa,method=multiple_choice_joint,model=psa_lora-out', scenario_spec=ScenarioSpec(class_name='helm.benchmark.scenarios.commonsense_scenario.OpenBookQA', args={}), adapter_spec=AdapterSpec(method='multiple_choice_joint', global_prefix='', global_suffix='', instructions='The following are multiple choice questions (with answers) about common sense.\n', input_prefix='Question: ', input_suffix='\n', reference_prefix='A. ', reference_suffix='\n', output_prefix='Answer: ', output_suffix='\n', instance_prefix='\n', substitutions=[], max_train_instances=5, max_eval_instances=1000, num_outputs=5, num_train_trials=1, num_trials=1, sample_train=True, model_deployment='huggingface/lora-out', model='psa/lora-out', temperature=0.0, max_tokens=1, stop_sequences=['\n'], random=None, multi_label=False, image_generation_parameters=None, eval_splits=None), metric_specs=[MetricSpec(class_name='helm.benchmark.metrics.basic_metrics.BasicGenerationMetric', args={'names': ['exact_match', 'quasi_exact_match', 'prefix_exact_match', 'quasi_prefix_exact_match']}), MetricSpec(class_name='helm.benchmark.metrics.basic_metrics.BasicReferenceMetric', args={}), MetricSpec(class_name='helm.benchmark.metrics.basic_metrics.InstancesPerSplitMetric', args={})], data_augmenter_spec=DataAugmenterSpec(perturbation_specs=[], should_augment_train_instances=False, should_include_original_train=False, should_skip_unchanged_train=False, should_augment_eval_instances=False, should_include_original_eval=False, should_skip_unchanged_eval=False, seeds_per_instance=1), groups=['openbookqa'], annotators=None)
    RunSpec(name='mmlu:subject=abstract_algebra,method=multiple_choice_joint,model=psa_lora-out', scenario_spec=ScenarioSpec(class_name='helm.benchmark.scenarios.mmlu_scenario.MMLUScenario', args={'subject': 'abstract_algebra'}), adapter_spec=AdapterSpec(method='multiple_choice_joint', global_prefix='', global_suffix='', instructions='The following are multiple choice questions (with answers) about abstract algebra.\n', input_prefix='Question: ', input_suffix='\n', reference_prefix='A. ', reference_suffix='\n', output_prefix='Answer: ', output_suffix='\n', instance_prefix='\n', substitutions=[], max_train_instances=5, max_eval_instances=1000, num_outputs=5, num_train_trials=1, num_trials=1, sample_train=True, model_deployment='huggingface/lora-out', model='psa/lora-out', temperature=0.0, max_tokens=1, stop_sequences=['\n'], random=None, multi_label=False, image_generation_parameters=None, eval_splits=None), metric_specs=[MetricSpec(class_name='helm.benchmark.metrics.basic_metrics.BasicGenerationMetric', args={'names': ['exact_match', 'quasi_exact_match', 'prefix_exact_match', 'quasi_prefix_exact_match']}), MetricSpec(class_name='helm.benchmark.metrics.basic_metrics.BasicReferenceMetric', args={}), MetricSpec(class_name='helm.benchmark.metrics.basic_metrics.InstancesPerSplitMetric', args={})], data_augmenter_spec=DataAugmenterSpec(perturbation_specs=[], should_augment_train_instances=False, should_include_original_train=False, should_skip_unchanged_train=False, should_augment_eval_instances=False, should_include_original_eval=False, should_skip_unchanged_eval=False, seeds_per_instance=1), groups=['mmlu'], annotators=None)
    RunSpec(name='mmlu:subject=college_chemistry,method=multiple_choice_joint,model=psa_lora-out', scenario_spec=ScenarioSpec(class_name='helm.benchmark.scenarios.mmlu_scenario.MMLUScenario', args={'subject': 'college_chemistry'}), adapter_spec=AdapterSpec(method='multiple_choice_joint', global_prefix='', global_suffix='', instructions='The following are multiple choice questions (with answers) about college chemistry.\n', input_prefix='Question: ', input_suffix='\n', reference_prefix='A. ', reference_suffix='\n', output_prefix='Answer: ', output_suffix='\n', instance_prefix='\n', substitutions=[], max_train_instances=5, max_eval_instances=1000, num_outputs=5, num_train_trials=1, num_trials=1, sample_train=True, model_deployment='huggingface/lora-out', model='psa/lora-out', temperature=0.0, max_tokens=1, stop_sequences=['\n'], random=None, multi_label=False, image_generation_parameters=None, eval_splits=None), metric_specs=[MetricSpec(class_name='helm.benchmark.metrics.basic_metrics.BasicGenerationMetric', args={'names': ['exact_match', 'quasi_exact_match', 'prefix_exact_match', 'quasi_prefix_exact_match']}), MetricSpec(class_name='helm.benchmark.metrics.basic_metrics.BasicReferenceMetric', args={}), MetricSpec(class_name='helm.benchmark.metrics.basic_metrics.InstancesPerSplitMetric', args={})], data_augmenter_spec=DataAugmenterSpec(perturbation_specs=[], should_augment_train_instances=False, should_include_original_train=False, should_skip_unchanged_train=False, should_augment_eval_instances=False, should_include_original_eval=False, should_skip_unchanged_eval=False, seeds_per_instance=1), groups=['mmlu'], annotators=None)
    RunSpec(name='mmlu:subject=computer_security,method=multiple_choice_joint,model=psa_lora-out', scenario_spec=ScenarioSpec(class_name='helm.benchmark.scenarios.mmlu_scenario.MMLUScenario', args={'subject': 'computer_security'}), adapter_spec=AdapterSpec(method='multiple_choice_joint', global_prefix='', global_suffix='', instructions='The following are multiple choice questions (with answers) about computer security.\n', input_prefix='Question: ', input_suffix='\n', reference_prefix='A. ', reference_suffix='\n', output_prefix='Answer: ', output_suffix='\n', instance_prefix='\n', substitutions=[], max_train_instances=5, max_eval_instances=1000, num_outputs=5, num_train_trials=1, num_trials=1, sample_train=True, model_deployment='huggingface/lora-out', model='psa/lora-out', temperature=0.0, max_tokens=1, stop_sequences=['\n'], random=None, multi_label=False, image_generation_parameters=None, eval_splits=None), metric_specs=[MetricSpec(class_name='helm.benchmark.metrics.basic_metrics.BasicGenerationMetric', args={'names': ['exact_match', 'quasi_exact_match', 'prefix_exact_match', 'quasi_prefix_exact_match']}), MetricSpec(class_name='helm.benchmark.metrics.basic_metrics.BasicReferenceMetric', args={}), MetricSpec(class_name='helm.benchmark.metrics.basic_metrics.InstancesPerSplitMetric', args={})], data_augmenter_spec=DataAugmenterSpec(perturbation_specs=[], should_augment_train_instances=False, should_include_original_train=False, should_skip_unchanged_train=False, should_augment_eval_instances=False, should_include_original_eval=False, should_skip_unchanged_eval=False, seeds_per_instance=1), groups=['mmlu'], annotators=None)
    RunSpec(name='mmlu:subject=econometrics,method=multiple_choice_joint,model=psa_lora-out', scenario_spec=ScenarioSpec(class_name='helm.benchmark.scenarios.mmlu_scenario.MMLUScenario', args={'subject': 'econometrics'}), adapter_spec=AdapterSpec(method='multiple_choice_joint', global_prefix='', global_suffix='', instructions='The following are multiple choice questions (with answers) about econometrics.\n', input_prefix='Question: ', input_suffix='\n', reference_prefix='A. ', reference_suffix='\n', output_prefix='Answer: ', output_suffix='\n', instance_prefix='\n', substitutions=[], max_train_instances=5, max_eval_instances=1000, num_outputs=5, num_train_trials=1, num_trials=1, sample_train=True, model_deployment='huggingface/lora-out', model='psa/lora-out', temperature=0.0, max_tokens=1, stop_sequences=['\n'], random=None, multi_label=False, image_generation_parameters=None, eval_splits=None), metric_specs=[MetricSpec(class_name='helm.benchmark.metrics.basic_metrics.BasicGenerationMetric', args={'names': ['exact_match', 'quasi_exact_match', 'prefix_exact_match', 'quasi_prefix_exact_match']}), MetricSpec(class_name='helm.benchmark.metrics.basic_metrics.BasicReferenceMetric', args={}), MetricSpec(class_name='helm.benchmark.metrics.basic_metrics.InstancesPerSplitMetric', args={})], data_augmenter_spec=DataAugmenterSpec(perturbation_specs=[], should_augment_train_instances=False, should_include_original_train=False, should_skip_unchanged_train=False, should_augment_eval_instances=False, should_include_original_eval=False, should_skip_unchanged_eval=False, seeds_per_instance=1), groups=['mmlu'], annotators=None)
    RunSpec(name='mmlu:subject=us_foreign_policy,method=multiple_choice_joint,model=psa_lora-out', scenario_spec=ScenarioSpec(class_name='helm.benchmark.scenarios.mmlu_scenario.MMLUScenario', args={'subject': 'us_foreign_policy'}), adapter_spec=AdapterSpec(method='multiple_choice_joint', global_prefix='', global_suffix='', instructions='The following are multiple choice questions (with answers) about us foreign policy.\n', input_prefix='Question: ', input_suffix='\n', reference_prefix='A. ', reference_suffix='\n', output_prefix='Answer: ', output_suffix='\n', instance_prefix='\n', substitutions=[], max_train_instances=5, max_eval_instances=1000, num_outputs=5, num_train_trials=1, num_trials=1, sample_train=True, model_deployment='huggingface/lora-out', model='psa/lora-out', temperature=0.0, max_tokens=1, stop_sequences=['\n'], random=None, multi_label=False, image_generation_parameters=None, eval_splits=None), metric_specs=[MetricSpec(class_name='helm.benchmark.metrics.basic_metrics.BasicGenerationMetric', args={'names': ['exact_match', 'quasi_exact_match', 'prefix_exact_match', 'quasi_prefix_exact_match']}), MetricSpec(class_name='helm.benchmark.metrics.basic_metrics.BasicReferenceMetric', args={}), MetricSpec(class_name='helm.benchmark.metrics.basic_metrics.InstancesPerSplitMetric', args={})], data_augmenter_spec=DataAugmenterSpec(perturbation_specs=[], should_augment_train_instances=False, should_include_original_train=False, should_skip_unchanged_train=False, should_augment_eval_instances=False, should_include_original_eval=False, should_skip_unchanged_eval=False, seeds_per_instance=1), groups=['mmlu'], annotators=None)
    RunSpec(name='math:subject=number_theory,level=1,use_official_examples=False,use_chain_of_thought=True,model=psa_lora-out', scenario_spec=ScenarioSpec(class_name='helm.benchmark.scenarios.math_scenario.MATHScenario', args={'subject': 'number_theory', 'level': 1, 'use_official_examples': False, 'use_chain_of_thought': True}), adapter_spec=AdapterSpec(method='generation', global_prefix='', global_suffix='', instructions='Given a mathematics problem, determine the answer. Simplify your answer as much as possible.\n', input_prefix='Problem: ', input_suffix='\n', reference_prefix='A. ', reference_suffix='\n', output_prefix='Answer: ', output_suffix='\n', instance_prefix='###\n', substitutions=[], max_train_instances=8, max_eval_instances=1000, num_outputs=1, num_train_trials=1, num_trials=1, sample_train=True, model_deployment='huggingface/lora-out', model='psa/lora-out', temperature=0.0, max_tokens=400, stop_sequences=['###'], random=None, multi_label=False, image_generation_parameters=None, eval_splits=None), metric_specs=[MetricSpec(class_name='helm.benchmark.metrics.basic_metrics.BasicGenerationMetric', args={'names': ['math_equiv_chain_of_thought']}), MetricSpec(class_name='helm.benchmark.metrics.basic_metrics.BasicReferenceMetric', args={}), MetricSpec(class_name='helm.benchmark.metrics.basic_metrics.InstancesPerSplitMetric', args={})], data_augmenter_spec=DataAugmenterSpec(perturbation_specs=[], should_augment_train_instances=False, should_include_original_train=False, should_skip_unchanged_train=False, should_augment_eval_instances=False, should_include_original_eval=False, should_skip_unchanged_eval=False, seeds_per_instance=1), groups=['math_chain_of_thought'], annotators=None)
    RunSpec(name='math:subject=intermediate_algebra,level=1,use_official_examples=False,use_chain_of_thought=True,model=psa_lora-out', scenario_spec=ScenarioSpec(class_name='helm.benchmark.scenarios.math_scenario.MATHScenario', args={'subject': 'intermediate_algebra', 'level': 1, 'use_official_examples': False, 'use_chain_of_thought': True}), adapter_spec=AdapterSpec(method='generation', global_prefix='', global_suffix='', instructions='Given a mathematics problem, determine the answer. Simplify your answer as much as possible.\n', input_prefix='Problem: ', input_suffix='\n', reference_prefix='A. ', reference_suffix='\n', output_prefix='Answer: ', output_suffix='\n', instance_prefix='###\n', substitutions=[], max_train_instances=8, max_eval_instances=1000, num_outputs=1, num_train_trials=1, num_trials=1, sample_train=True, model_deployment='huggingface/lora-out', model='psa/lora-out', temperature=0.0, max_tokens=400, stop_sequences=['###'], random=None, multi_label=False, image_generation_parameters=None, eval_splits=None), metric_specs=[MetricSpec(class_name='helm.benchmark.metrics.basic_metrics.BasicGenerationMetric', args={'names': ['math_equiv_chain_of_thought']}), MetricSpec(class_name='helm.benchmark.metrics.basic_metrics.BasicReferenceMetric', args={}), MetricSpec(class_name='helm.benchmark.metrics.basic_metrics.InstancesPerSplitMetric', args={})], data_augmenter_spec=DataAugmenterSpec(perturbation_specs=[], should_augment_train_instances=False, should_include_original_train=False, should_skip_unchanged_train=False, should_augment_eval_instances=False, should_include_original_eval=False, should_skip_unchanged_eval=False, seeds_per_instance=1), groups=['math_chain_of_thought'], annotators=None)
    RunSpec(name='math:subject=algebra,level=1,use_official_examples=False,use_chain_of_thought=True,model=psa_lora-out', scenario_spec=ScenarioSpec(class_name='helm.benchmark.scenarios.math_scenario.MATHScenario', args={'subject': 'algebra', 'level': 1, 'use_official_examples': False, 'use_chain_of_thought': True}), adapter_spec=AdapterSpec(method='generation', global_prefix='', global_suffix='', instructions='Given a mathematics problem, determine the answer. Simplify your answer as much as possible.\n', input_prefix='Problem: ', input_suffix='\n', reference_prefix='A. ', reference_suffix='\n', output_prefix='Answer: ', output_suffix='\n', instance_prefix='###\n', substitutions=[], max_train_instances=8, max_eval_instances=1000, num_outputs=1, num_train_trials=1, num_trials=1, sample_train=True, model_deployment='huggingface/lora-out', model='psa/lora-out', temperature=0.0, max_tokens=400, stop_sequences=['###'], random=None, multi_label=False, image_generation_parameters=None, eval_splits=None), metric_specs=[MetricSpec(class_name='helm.benchmark.metrics.basic_metrics.BasicGenerationMetric', args={'names': ['math_equiv_chain_of_thought']}), MetricSpec(class_name='helm.benchmark.metrics.basic_metrics.BasicReferenceMetric', args={}), MetricSpec(class_name='helm.benchmark.metrics.basic_metrics.InstancesPerSplitMetric', args={})], data_augmenter_spec=DataAugmenterSpec(perturbation_specs=[], should_augment_train_instances=False, should_include_original_train=False, should_skip_unchanged_train=False, should_augment_eval_instances=False, should_include_original_eval=False, should_skip_unchanged_eval=False, seeds_per_instance=1), groups=['math_chain_of_thought'], annotators=None)
    RunSpec(name='math:subject=prealgebra,level=1,use_official_examples=False,use_chain_of_thought=True,model=psa_lora-out', scenario_spec=ScenarioSpec(class_name='helm.benchmark.scenarios.math_scenario.MATHScenario', args={'subject': 'prealgebra', 'level': 1, 'use_official_examples': False, 'use_chain_of_thought': True}), adapter_spec=AdapterSpec(method='generation', global_prefix='', global_suffix='', instructions='Given a mathematics problem, determine the answer. Simplify your answer as much as possible.\n', input_prefix='Problem: ', input_suffix='\n', reference_prefix='A. ', reference_suffix='\n', output_prefix='Answer: ', output_suffix='\n', instance_prefix='###\n', substitutions=[], max_train_instances=8, max_eval_instances=1000, num_outputs=1, num_train_trials=1, num_trials=1, sample_train=True, model_deployment='huggingface/lora-out', model='psa/lora-out', temperature=0.0, max_tokens=400, stop_sequences=['###'], random=None, multi_label=False, image_generation_parameters=None, eval_splits=None), metric_specs=[MetricSpec(class_name='helm.benchmark.metrics.basic_metrics.BasicGenerationMetric', args={'names': ['math_equiv_chain_of_thought']}), MetricSpec(class_name='helm.benchmark.metrics.basic_metrics.BasicReferenceMetric', args={}), MetricSpec(class_name='helm.benchmark.metrics.basic_metrics.InstancesPerSplitMetric', args={})], data_augmenter_spec=DataAugmenterSpec(perturbation_specs=[], should_augment_train_instances=False, should_include_original_train=False, should_skip_unchanged_train=False, should_augment_eval_instances=False, should_include_original_eval=False, should_skip_unchanged_eval=False, seeds_per_instance=1), groups=['math_chain_of_thought'], annotators=None)
    RunSpec(name='math:subject=geometry,level=1,use_official_examples=False,use_chain_of_thought=True,model=psa_lora-out', scenario_spec=ScenarioSpec(class_name='helm.benchmark.scenarios.math_scenario.MATHScenario', args={'subject': 'geometry', 'level': 1, 'use_official_examples': False, 'use_chain_of_thought': True}), adapter_spec=AdapterSpec(method='generation', global_prefix='', global_suffix='', instructions='Given a mathematics problem, determine the answer. Simplify your answer as much as possible.\n', input_prefix='Problem: ', input_suffix='\n', reference_prefix='A. ', reference_suffix='\n', output_prefix='Answer: ', output_suffix='\n', instance_prefix='###\n', substitutions=[], max_train_instances=8, max_eval_instances=1000, num_outputs=1, num_train_trials=1, num_trials=1, sample_train=True, model_deployment='huggingface/lora-out', model='psa/lora-out', temperature=0.0, max_tokens=400, stop_sequences=['###'], random=None, multi_label=False, image_generation_parameters=None, eval_splits=None), metric_specs=[MetricSpec(class_name='helm.benchmark.metrics.basic_metrics.BasicGenerationMetric', args={'names': ['math_equiv_chain_of_thought']}), MetricSpec(class_name='helm.benchmark.metrics.basic_metrics.BasicReferenceMetric', args={}), MetricSpec(class_name='helm.benchmark.metrics.basic_metrics.InstancesPerSplitMetric', args={})], data_augmenter_spec=DataAugmenterSpec(perturbation_specs=[], should_augment_train_instances=False, should_include_original_train=False, should_skip_unchanged_train=False, should_augment_eval_instances=False, should_include_original_eval=False, should_skip_unchanged_eval=False, seeds_per_instance=1), groups=['math_chain_of_thought'], annotators=None)
    RunSpec(name='math:subject=counting_and_probability,level=1,use_official_examples=False,use_chain_of_thought=True,model=psa_lora-out', scenario_spec=ScenarioSpec(class_name='helm.benchmark.scenarios.math_scenario.MATHScenario', args={'subject': 'counting_and_probability', 'level': 1, 'use_official_examples': False, 'use_chain_of_thought': True}), adapter_spec=AdapterSpec(method='generation', global_prefix='', global_suffix='', instructions='Given a mathematics problem, determine the answer. Simplify your answer as much as possible.\n', input_prefix='Problem: ', input_suffix='\n', reference_prefix='A. ', reference_suffix='\n', output_prefix='Answer: ', output_suffix='\n', instance_prefix='###\n', substitutions=[], max_train_instances=8, max_eval_instances=1000, num_outputs=1, num_train_trials=1, num_trials=1, sample_train=True, model_deployment='huggingface/lora-out', model='psa/lora-out', temperature=0.0, max_tokens=400, stop_sequences=['###'], random=None, multi_label=False, image_generation_parameters=None, eval_splits=None), metric_specs=[MetricSpec(class_name='helm.benchmark.metrics.basic_metrics.BasicGenerationMetric', args={'names': ['math_equiv_chain_of_thought']}), MetricSpec(class_name='helm.benchmark.metrics.basic_metrics.BasicReferenceMetric', args={}), MetricSpec(class_name='helm.benchmark.metrics.basic_metrics.InstancesPerSplitMetric', args={})], data_augmenter_spec=DataAugmenterSpec(perturbation_specs=[], should_augment_train_instances=False, should_include_original_train=False, should_skip_unchanged_train=False, should_augment_eval_instances=False, should_include_original_eval=False, should_skip_unchanged_eval=False, seeds_per_instance=1), groups=['math_chain_of_thought'], annotators=None)
    RunSpec(name='math:subject=precalculus,level=1,use_official_examples=False,use_chain_of_thought=True,model=psa_lora-out', scenario_spec=ScenarioSpec(class_name='helm.benchmark.scenarios.math_scenario.MATHScenario', args={'subject': 'precalculus', 'level': 1, 'use_official_examples': False, 'use_chain_of_thought': True}), adapter_spec=AdapterSpec(method='generation', global_prefix='', global_suffix='', instructions='Given a mathematics problem, determine the answer. Simplify your answer as much as possible.\n', input_prefix='Problem: ', input_suffix='\n', reference_prefix='A. ', reference_suffix='\n', output_prefix='Answer: ', output_suffix='\n', instance_prefix='###\n', substitutions=[], max_train_instances=8, max_eval_instances=1000, num_outputs=1, num_train_trials=1, num_trials=1, sample_train=True, model_deployment='huggingface/lora-out', model='psa/lora-out', temperature=0.0, max_tokens=400, stop_sequences=['###'], random=None, multi_label=False, image_generation_parameters=None, eval_splits=None), metric_specs=[MetricSpec(class_name='helm.benchmark.metrics.basic_metrics.BasicGenerationMetric', args={'names': ['math_equiv_chain_of_thought']}), MetricSpec(class_name='helm.benchmark.metrics.basic_metrics.BasicReferenceMetric', args={}), MetricSpec(class_name='helm.benchmark.metrics.basic_metrics.InstancesPerSplitMetric', args={})], data_augmenter_spec=DataAugmenterSpec(perturbation_specs=[], should_augment_train_instances=False, should_include_original_train=False, should_skip_unchanged_train=False, should_augment_eval_instances=False, should_include_original_eval=False, should_skip_unchanged_eval=False, seeds_per_instance=1), groups=['math_chain_of_thought'], annotators=None)
    RunSpec(name='gsm:model=psa_lora-out', scenario_spec=ScenarioSpec(class_name='helm.benchmark.scenarios.gsm_scenario.GSM8KScenario', args={}), adapter_spec=AdapterSpec(method='generation', global_prefix='', global_suffix='', instructions='', input_prefix='Q: ', input_suffix='\n', reference_prefix='A. ', reference_suffix='\n', output_prefix='A: ', output_suffix='\n', instance_prefix='\n', substitutions=[], max_train_instances=5, max_eval_instances=1000, num_outputs=1, num_train_trials=1, num_trials=1, sample_train=True, model_deployment='huggingface/lora-out', model='psa/lora-out', temperature=0.0, max_tokens=400, stop_sequences=['\n\n'], random=None, multi_label=False, image_generation_parameters=None, eval_splits=None), metric_specs=[MetricSpec(class_name='helm.benchmark.metrics.basic_metrics.BasicGenerationMetric', args={'names': ['exact_match_indicator', 'final_number_exact_match']}), MetricSpec(class_name='helm.benchmark.metrics.basic_metrics.InstancesPerSplitMetric', args={})], data_augmenter_spec=DataAugmenterSpec(perturbation_specs=[], should_augment_train_instances=False, should_include_original_train=False, should_skip_unchanged_train=False, should_augment_eval_instances=False, should_include_original_eval=False, should_skip_unchanged_eval=False, seeds_per_instance=1), groups=['gsm'], annotators=None)
    RunSpec(name='legalbench:subset=abercrombie,model=psa_lora-out', scenario_spec=ScenarioSpec(class_name='helm.benchmark.scenarios.legalbench_scenario.LegalBenchScenario', args={'subset': 'abercrombie'}), adapter_spec=AdapterSpec(method='generation', global_prefix='', global_suffix='', instructions='A mark is generic if it is the common name for the product. A mark is descriptive if it describes a purpose, nature, or attribute of the product. A mark is suggestive if it suggests or implies a quality or characteristic of the product. A mark is arbitrary if it is a real English word that has no relation to the product. A mark is fanciful if it is an invented word. Label the type of mark for the following products.\n', input_prefix='', input_suffix='\n', reference_prefix='A. ', reference_suffix='\n', output_prefix='Type: ', output_suffix='\n', instance_prefix='\n', substitutions=[], max_train_instances=5, max_eval_instances=1000, num_outputs=1, num_train_trials=1, num_trials=1, sample_train=True, model_deployment='huggingface/lora-out', model='psa/lora-out', temperature=0.0, max_tokens=30, stop_sequences=['\n'], random=None, multi_label=False, image_generation_parameters=None, eval_splits=None), metric_specs=[MetricSpec(class_name='helm.benchmark.metrics.basic_metrics.BasicGenerationMetric', args={'names': ['exact_match', 'quasi_exact_match', 'prefix_exact_match', 'quasi_prefix_exact_match']}), MetricSpec(class_name='helm.benchmark.metrics.basic_metrics.BasicReferenceMetric', args={}), MetricSpec(class_name='helm.benchmark.metrics.basic_metrics.InstancesPerSplitMetric', args={})], data_augmenter_spec=DataAugmenterSpec(perturbation_specs=[], should_augment_train_instances=False, should_include_original_train=False, should_skip_unchanged_train=False, should_augment_eval_instances=False, should_include_original_eval=False, should_skip_unchanged_eval=False, seeds_per_instance=1), groups=['legalbench'], annotators=None)
    RunSpec(name='legalbench:subset=corporate_lobbying,model=psa_lora-out', scenario_spec=ScenarioSpec(class_name='helm.benchmark.scenarios.legalbench_scenario.LegalBenchScenario', args={'subset': 'corporate_lobbying'}), adapter_spec=AdapterSpec(method='generation', global_prefix='', global_suffix='', instructions="You are a lobbyist analyzing Congressional bills for their impacts on companies. Given the title and summary of the bill, plus information on the company from its 10K SEC filing, it is your job to determine if a bill is at least somewhat relevant to a company in terms of whether it could impact the company's bottom-line if it was enacted (by saying Yes or No).\n", input_prefix='', input_suffix='\n', reference_prefix='A. ', reference_suffix='\n', output_prefix='Label: ', output_suffix='\n', instance_prefix='\n', substitutions=[], max_train_instances=5, max_eval_instances=1000, num_outputs=1, num_train_trials=1, num_trials=1, sample_train=True, model_deployment='huggingface/lora-out', model='psa/lora-out', temperature=0.0, max_tokens=30, stop_sequences=['\n'], random=None, multi_label=False, image_generation_parameters=None, eval_splits=None), metric_specs=[MetricSpec(class_name='helm.benchmark.metrics.basic_metrics.BasicGenerationMetric', args={'names': ['exact_match', 'quasi_exact_match', 'prefix_exact_match', 'quasi_prefix_exact_match']}), MetricSpec(class_name='helm.benchmark.metrics.basic_metrics.BasicReferenceMetric', args={}), MetricSpec(class_name='helm.benchmark.metrics.basic_metrics.InstancesPerSplitMetric', args={})], data_augmenter_spec=DataAugmenterSpec(perturbation_specs=[], should_augment_train_instances=False, should_include_original_train=False, should_skip_unchanged_train=False, should_augment_eval_instances=False, should_include_original_eval=False, should_skip_unchanged_eval=False, seeds_per_instance=1), groups=['legalbench'], annotators=None)
    RunSpec(name='legalbench:subset=international_citizenship_questions,model=psa_lora-out', scenario_spec=ScenarioSpec(class_name='helm.benchmark.scenarios.legalbench_scenario.LegalBenchScenario', args={'subset': 'international_citizenship_questions'}), adapter_spec=AdapterSpec(method='generation', global_prefix='', global_suffix='', instructions='Answer the following questions considering the state of international law on January 1st, 2020. Answer Yes or No.\n', input_prefix='', input_suffix='\n', reference_prefix='A. ', reference_suffix='\n', output_prefix='Answer: ', output_suffix='\n', instance_prefix='\n', substitutions=[], max_train_instances=5, max_eval_instances=1000, num_outputs=1, num_train_trials=1, num_trials=1, sample_train=True, model_deployment='huggingface/lora-out', model='psa/lora-out', temperature=0.0, max_tokens=30, stop_sequences=['\n'], random=None, multi_label=False, image_generation_parameters=None, eval_splits=None), metric_specs=[MetricSpec(class_name='helm.benchmark.metrics.basic_metrics.BasicGenerationMetric', args={'names': ['exact_match', 'quasi_exact_match', 'prefix_exact_match', 'quasi_prefix_exact_match']}), MetricSpec(class_name='helm.benchmark.metrics.basic_metrics.BasicReferenceMetric', args={}), MetricSpec(class_name='helm.benchmark.metrics.basic_metrics.InstancesPerSplitMetric', args={})], data_augmenter_spec=DataAugmenterSpec(perturbation_specs=[], should_augment_train_instances=False, should_include_original_train=False, should_skip_unchanged_train=False, should_augment_eval_instances=False, should_include_original_eval=False, should_skip_unchanged_eval=False, seeds_per_instance=1), groups=['legalbench'], annotators=None)
    RunSpec(name='legalbench:subset=function_of_decision_section,model=psa_lora-out', scenario_spec=ScenarioSpec(class_name='helm.benchmark.scenarios.legalbench_scenario.LegalBenchScenario', args={'subset': 'function_of_decision_section'}), adapter_spec=AdapterSpec(method='generation', global_prefix='', global_suffix='', instructions='Classify the following text using the following definitions.\n\n- Facts: The paragraph describes the factual background that led up to the present lawsuit.\n- Procedural History: The paragraph describes the course of litigation that led to the current proceeding before the court.\n- Issue: The paragraph describes the legal or factual issue that must be resolved by the court.\n- Rule: The paragraph describes a rule of law relevant to resolving the issue.\n- Analysis: The paragraph analyzes the legal issue by applying the relevant legal principles to the facts of the present dispute.\n- Conclusion: The paragraph presents a conclusion of the court.\n- Decree: The paragraph constitutes a decree resolving the dispute.\n', input_prefix='', input_suffix='\n', reference_prefix='A. ', reference_suffix='\n', output_prefix='Function: ', output_suffix='\n', instance_prefix='\n', substitutions=[], max_train_instances=5, max_eval_instances=1000, num_outputs=1, num_train_trials=1, num_trials=1, sample_train=True, model_deployment='huggingface/lora-out', model='psa/lora-out', temperature=0.0, max_tokens=30, stop_sequences=['\n'], random=None, multi_label=False, image_generation_parameters=None, eval_splits=None), metric_specs=[MetricSpec(class_name='helm.benchmark.metrics.basic_metrics.BasicGenerationMetric', args={'names': ['exact_match', 'quasi_exact_match', 'prefix_exact_match', 'quasi_prefix_exact_match']}), MetricSpec(class_name='helm.benchmark.metrics.basic_metrics.BasicReferenceMetric', args={}), MetricSpec(class_name='helm.benchmark.metrics.basic_metrics.InstancesPerSplitMetric', args={})], data_augmenter_spec=DataAugmenterSpec(perturbation_specs=[], should_augment_train_instances=False, should_include_original_train=False, should_skip_unchanged_train=False, should_augment_eval_instances=False, should_include_original_eval=False, should_skip_unchanged_eval=False, seeds_per_instance=1), groups=['legalbench'], annotators=None)
    RunSpec(name='legalbench:subset=proa,model=psa_lora-out', scenario_spec=ScenarioSpec(class_name='helm.benchmark.scenarios.legalbench_scenario.LegalBenchScenario', args={'subset': 'proa'}), adapter_spec=AdapterSpec(method='generation', global_prefix='', global_suffix='', instructions='A private right of action is when a regular person, a private citizen, is legally entitled to enforce their rights under a given statute. Does the clause specify a private right of action? Answer Yes or No.\n', input_prefix='', input_suffix='\n', reference_prefix='A. ', reference_suffix='\n', output_prefix='Answer: ', output_suffix='\n', instance_prefix='\n', substitutions=[], max_train_instances=5, max_eval_instances=1000, num_outputs=1, num_train_trials=1, num_trials=1, sample_train=True, model_deployment='huggingface/lora-out', model='psa/lora-out', temperature=0.0, max_tokens=30, stop_sequences=['\n'], random=None, multi_label=False, image_generation_parameters=None, eval_splits=None), metric_specs=[MetricSpec(class_name='helm.benchmark.metrics.basic_metrics.BasicGenerationMetric', args={'names': ['exact_match', 'quasi_exact_match', 'prefix_exact_match', 'quasi_prefix_exact_match']}), MetricSpec(class_name='helm.benchmark.metrics.basic_metrics.BasicReferenceMetric', args={}), MetricSpec(class_name='helm.benchmark.metrics.basic_metrics.InstancesPerSplitMetric', args={})], data_augmenter_spec=DataAugmenterSpec(perturbation_specs=[], should_augment_train_instances=False, should_include_original_train=False, should_skip_unchanged_train=False, should_augment_eval_instances=False, should_include_original_eval=False, should_skip_unchanged_eval=False, seeds_per_instance=1), groups=['legalbench'], annotators=None)
    RunSpec(name='med_qa:model=psa_lora-out', scenario_spec=ScenarioSpec(class_name='helm.benchmark.scenarios.med_qa_scenario.MedQAScenario', args={}), adapter_spec=AdapterSpec(method='multiple_choice_joint', global_prefix='', global_suffix='', instructions='The following are multiple choice questions (with answers) about medicine.\n', input_prefix='Question: ', input_suffix='\n', reference_prefix='A. ', reference_suffix='\n', output_prefix='Answer: ', output_suffix='\n', instance_prefix='\n', substitutions=[], max_train_instances=5, max_eval_instances=1000, num_outputs=5, num_train_trials=1, num_trials=1, sample_train=True, model_deployment='huggingface/lora-out', model='psa/lora-out', temperature=0.0, max_tokens=1, stop_sequences=['\n'], random=None, multi_label=False, image_generation_parameters=None, eval_splits=None), metric_specs=[MetricSpec(class_name='helm.benchmark.metrics.basic_metrics.BasicGenerationMetric', args={'names': ['exact_match', 'quasi_exact_match', 'prefix_exact_match', 'quasi_prefix_exact_match']}), MetricSpec(class_name='helm.benchmark.metrics.basic_metrics.BasicReferenceMetric', args={}), MetricSpec(class_name='helm.benchmark.metrics.basic_metrics.InstancesPerSplitMetric', args={})], data_augmenter_spec=DataAugmenterSpec(perturbation_specs=[], should_augment_train_instances=False, should_include_original_train=False, should_skip_unchanged_train=False, should_augment_eval_instances=False, should_include_original_eval=False, should_skip_unchanged_eval=False, seeds_per_instance=1), groups=['med_qa'], annotators=None)
    RunSpec(name='wmt_14:language_pair=cs-en,model=psa_lora-out', scenario_spec=ScenarioSpec(class_name='helm.benchmark.scenarios.wmt_14_scenario.WMT14Scenario', args={'source_language': 'cs', 'target_language': 'en'}), adapter_spec=AdapterSpec(method='generation', global_prefix='', global_suffix='', instructions='Translate the following sentences from Czech to English.', input_prefix='Czech: ', input_suffix='\n', reference_prefix='A. ', reference_suffix='\n', output_prefix='English: ', output_suffix='\n', instance_prefix='\n', substitutions=[], max_train_instances=1, max_eval_instances=1000, num_outputs=1, num_train_trials=1, num_trials=1, sample_train=True, model_deployment='huggingface/lora-out', model='psa/lora-out', temperature=0.0, max_tokens=100, stop_sequences=['\n\n'], random=None, multi_label=False, image_generation_parameters=None, eval_splits=None), metric_specs=[MetricSpec(class_name='helm.benchmark.metrics.basic_metrics.BasicGenerationMetric', args={'names': ['exact_match', 'quasi_exact_match', 'f1_score', 'rouge_l', 'bleu_1', 'bleu_4']}), MetricSpec(class_name='helm.benchmark.metrics.basic_metrics.BasicReferenceMetric', args={}), MetricSpec(class_name='helm.benchmark.metrics.basic_metrics.InstancesPerSplitMetric', args={})], data_augmenter_spec=DataAugmenterSpec(perturbation_specs=[], should_augment_train_instances=False, should_include_original_train=False, should_skip_unchanged_train=False, should_augment_eval_instances=False, should_include_original_eval=False, should_skip_unchanged_eval=False, seeds_per_instance=1), groups=['wmt_14'], annotators=None)
    RunSpec(name='wmt_14:language_pair=de-en,model=psa_lora-out', scenario_spec=ScenarioSpec(class_name='helm.benchmark.scenarios.wmt_14_scenario.WMT14Scenario', args={'source_language': 'de', 'target_language': 'en'}), adapter_spec=AdapterSpec(method='generation', global_prefix='', global_suffix='', instructions='Translate the following sentences from German to English.', input_prefix='German: ', input_suffix='\n', reference_prefix='A. ', reference_suffix='\n', output_prefix='English: ', output_suffix='\n', instance_prefix='\n', substitutions=[], max_train_instances=1, max_eval_instances=1000, num_outputs=1, num_train_trials=1, num_trials=1, sample_train=True, model_deployment='huggingface/lora-out', model='psa/lora-out', temperature=0.0, max_tokens=100, stop_sequences=['\n\n'], random=None, multi_label=False, image_generation_parameters=None, eval_splits=None), metric_specs=[MetricSpec(class_name='helm.benchmark.metrics.basic_metrics.BasicGenerationMetric', args={'names': ['exact_match', 'quasi_exact_match', 'f1_score', 'rouge_l', 'bleu_1', 'bleu_4']}), MetricSpec(class_name='helm.benchmark.metrics.basic_metrics.BasicReferenceMetric', args={}), MetricSpec(class_name='helm.benchmark.metrics.basic_metrics.InstancesPerSplitMetric', args={})], data_augmenter_spec=DataAugmenterSpec(perturbation_specs=[], should_augment_train_instances=False, should_include_original_train=False, should_skip_unchanged_train=False, should_augment_eval_instances=False, should_include_original_eval=False, should_skip_unchanged_eval=False, seeds_per_instance=1), groups=['wmt_14'], annotators=None)
    RunSpec(name='wmt_14:language_pair=fr-en,model=psa_lora-out', scenario_spec=ScenarioSpec(class_name='helm.benchmark.scenarios.wmt_14_scenario.WMT14Scenario', args={'source_language': 'fr', 'target_language': 'en'}), adapter_spec=AdapterSpec(method='generation', global_prefix='', global_suffix='', instructions='Translate the following sentences from French to English.', input_prefix='French: ', input_suffix='\n', reference_prefix='A. ', reference_suffix='\n', output_prefix='English: ', output_suffix='\n', instance_prefix='\n', substitutions=[], max_train_instances=1, max_eval_instances=1000, num_outputs=1, num_train_trials=1, num_trials=1, sample_train=True, model_deployment='huggingface/lora-out', model='psa/lora-out', temperature=0.0, max_tokens=100, stop_sequences=['\n\n'], random=None, multi_label=False, image_generation_parameters=None, eval_splits=None), metric_specs=[MetricSpec(class_name='helm.benchmark.metrics.basic_metrics.BasicGenerationMetric', args={'names': ['exact_match', 'quasi_exact_match', 'f1_score', 'rouge_l', 'bleu_1', 'bleu_4']}), MetricSpec(class_name='helm.benchmark.metrics.basic_metrics.BasicReferenceMetric', args={}), MetricSpec(class_name='helm.benchmark.metrics.basic_metrics.InstancesPerSplitMetric', args={})], data_augmenter_spec=DataAugmenterSpec(perturbation_specs=[], should_augment_train_instances=False, should_include_original_train=False, should_skip_unchanged_train=False, should_augment_eval_instances=False, should_include_original_eval=False, should_skip_unchanged_eval=False, seeds_per_instance=1), groups=['wmt_14'], annotators=None)
    RunSpec(name='wmt_14:language_pair=hi-en,model=psa_lora-out', scenario_spec=ScenarioSpec(class_name='helm.benchmark.scenarios.wmt_14_scenario.WMT14Scenario', args={'source_language': 'hi', 'target_language': 'en'}), adapter_spec=AdapterSpec(method='generation', global_prefix='', global_suffix='', instructions='Translate the following sentences from Hindi to English.', input_prefix='Hindi: ', input_suffix='\n', reference_prefix='A. ', reference_suffix='\n', output_prefix='English: ', output_suffix='\n', instance_prefix='\n', substitutions=[], max_train_instances=1, max_eval_instances=1000, num_outputs=1, num_train_trials=1, num_trials=1, sample_train=True, model_deployment='huggingface/lora-out', model='psa/lora-out', temperature=0.0, max_tokens=100, stop_sequences=['\n\n'], random=None, multi_label=False, image_generation_parameters=None, eval_splits=None), metric_specs=[MetricSpec(class_name='helm.benchmark.metrics.basic_metrics.BasicGenerationMetric', args={'names': ['exact_match', 'quasi_exact_match', 'f1_score', 'rouge_l', 'bleu_1', 'bleu_4']}), MetricSpec(class_name='helm.benchmark.metrics.basic_metrics.BasicReferenceMetric', args={}), MetricSpec(class_name='helm.benchmark.metrics.basic_metrics.InstancesPerSplitMetric', args={})], data_augmenter_spec=DataAugmenterSpec(perturbation_specs=[], should_augment_train_instances=False, should_include_original_train=False, should_skip_unchanged_train=False, should_augment_eval_instances=False, should_include_original_eval=False, should_skip_unchanged_eval=False, seeds_per_instance=1), groups=['wmt_14'], annotators=None)
    RunSpec(name='wmt_14:language_pair=ru-en,model=psa_lora-out', scenario_spec=ScenarioSpec(class_name='helm.benchmark.scenarios.wmt_14_scenario.WMT14Scenario', args={'source_language': 'ru', 'target_language': 'en'}), adapter_spec=AdapterSpec(method='generation', global_prefix='', global_suffix='', instructions='Translate the following sentences from Russian to English.', input_prefix='Russian: ', input_suffix='\n', reference_prefix='A. ', reference_suffix='\n', output_prefix='English: ', output_suffix='\n', instance_prefix='\n', substitutions=[], max_train_instances=1, max_eval_instances=1000, num_outputs=1, num_train_trials=1, num_trials=1, sample_train=True, model_deployment='huggingface/lora-out', model='psa/lora-out', temperature=0.0, max_tokens=100, stop_sequences=['\n\n'], random=None, multi_label=False, image_generation_parameters=None, eval_splits=None), metric_specs=[MetricSpec(class_name='helm.benchmark.metrics.basic_metrics.BasicGenerationMetric', args={'names': ['exact_match', 'quasi_exact_match', 'f1_score', 'rouge_l', 'bleu_1', 'bleu_4']}), MetricSpec(class_name='helm.benchmark.metrics.basic_metrics.BasicReferenceMetric', args={}), MetricSpec(class_name='helm.benchmark.metrics.basic_metrics.InstancesPerSplitMetric', args={})], data_augmenter_spec=DataAugmenterSpec(perturbation_specs=[], should_augment_train_instances=False, should_include_original_train=False, should_skip_unchanged_train=False, should_augment_eval_instances=False, should_include_original_eval=False, should_skip_unchanged_eval=False, seeds_per_instance=1), groups=['wmt_14'], annotators=None)
  } [0.0s]
  Running in local mode with base path: prod_env
Looking in path: prod_env
  AutoTokenizer: cache_backend_config = SqliteCacheBackendConfig(path='prod_env/cache')
  AutoClient: file_storage_path = prod_env/cache
  AutoClient: cache_backend_config = SqliteCacheBackendConfig(path='prod_env/cache')
  AutoTokenizer: cache_backend_config = SqliteCacheBackendConfig(path='prod_env/cache')
  Found 1 account(s).
Looking in path: prod_env
  AnnotatorFactory: file_storage_path = prod_env/cache
  AnnotatorFactory: cache_backend_config = SqliteCacheBackendConfig(path='prod_env/cache')
  0%|                                                                                                                                                                                        | 0/28 [00:00<?, ?it/s]  Running narrative_qa:model=psa_lora-out {
    scenario.get_instances {
      ensure_file_downloaded {
        Not downloading https://github.com/deepmind/narrativeqa/archive/master.zip because benchmark_output/scenarios/narrativeqa/data/narrativeqa-master already exists
      } [0.0s]
    } [1.022s]
    1572 instances, 1102 train instances, 470/470 eval instances
    DataPreprocessor.preprocess {
    } [0.0s]
    GenerationAdapter.adapt {
      1572 instances, choosing 5/1102 train instances, 470 eval instances
      Adapting with train_trial_index=0 {
        Sampled 5 examples for trial #0.
        Parallelizing computation on 470 items over 4 threads {
                                                                                                                                                                                                                             Created cache with config: SqliteCacheConfig(path='prod_env/cache/psa.sqlite')                                                                                                     | 0/470 [00:00<?, ?it/s]
          Created cache with config: SqliteCacheConfig(path='prod_env/cache/psa.sqlite')
          Created cache with config: SqliteCacheConfig(path='prod_env/cache/psa.sqlite')
          Created cache with config: SqliteCacheConfig(path='prod_env/cache/psa.sqlite')
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 470/470 [00:04<00:00, 106.03it/s]
        } [4.437s]████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████       | 451/470 [00:04<00:00, 173.41it/s]
        Sample prompts {
          reference index = None, request_mode = None {
            Passage: The Little White Bird is a series of short episodes, including both accounts of the narrator's day-to-day activities in contemporary London and fanciful tales set in Kensington Gardens and elsewhere.The story is set in several locations; the earlier chapters are set in the town of London, contemporaneous to the time of Barrie's writing, and involving some time travel of a few years, and other fantasy elements, while remaining within the London setting. The middle chapters that later became Peter Pan in Kensington Gardens are set in London's famous Kensington Gardens, introduced by the statement that "All perambulators lead to Kensington Gardens". The Kensington Gardens chapters include detailed descriptions of the features of the Gardens, along with fantasy names given to the locations by the story's characters, especially after "Lock-Out Time", described by Barrie as the time at the end of the day when the park gates are closed to the public, and the fairies and other magical inhabitants of the park can move about more freely than during the daylight, when they must hide from ordinary people. The third section of the book, following the Kensington Gardens chapters, are again set generally in London, though there are some short returns to the Gardens that are not part of the Peter Pan stories. In a two-page diversion in chapter 24, Barrie brings the story to Patagonia, and a journey by ship returning to England at the "white cliffs of Albion".
            Question: Where  was the ship in Pentagonia returned to?
            Answer: England
            
            Passage: The film explores several flashbacks and present timelines to show how Dean and Cindy became involved. Dean is a young high school dropout, working for a moving company in Brooklyn. Cindy is a pre-med student living with her constantly fighting parents and also caring for her grandmother in Pennsylvania. Cindy and Dean meet at Cindy's grandmother's nursing home while Dean is delivering a new resident's furniture and they begin dating afterwards.
            Cindy discovers she is pregnant, and tells Dean that the baby is most likely not his, as her ex-boyfriend Bobby didn't use protection. Dean asks Cindy whether or not she wants to keep the baby. At an abortion clinic, Cindy decides at the last moment to cancel the procedure, and on a bus ride home, Dean tells her he doesn't mind if the child is not his, and that he wants to begin a family with her. Before the wedding, Bobby finds out about Dean, and beats him up.
            Five years later, the couple lives in rural Pennsylvania with their daughter Frankie. Dean works at painting houses while Cindy is a nurse at a clinic. One evening, Dean insists on taking Cindy out for a romantic getaway at a motel so they can have some time off from their preoccupied lives, much to Cindy's reluctance.
            While buying wine in a liquor store, Cindy sees Bobby, who asks Cindy if she has ever cheated on her husband. She hesitates, but eventually says no. In the car, Cindy and Dean get into an argument when she mentions seeing Bobby again. At the motel, they continue fighting during sex. Cindy is called away early in the morning to work at the clinic, and she leaves a note for Dean. At the clinic, Cindy's boss, Dr. Feinberg, talks to her about a position he had offered her, and asks if she would move closer to work, suggesting that they would be able to spend time together on weekends. Visibly upset, Cindy says she previously thought he was offering her the position because she was good at her job.
            Angered that Cindy left the motel without waking him, Dean shows up drunk at the clinic, leading to a violent altercation with Dr. Feinberg. Cindy says she wants a divorce after Dr. Feinberg fires her. After leaving the clinic, Dean tries to persuade Cindy to give the marriage another chance, asking if she wants their daughter to grow up in a broken home. Cindy says she does not want Frankie to grow up with parents who are so hateful to each other.
            Dean reminds Cindy of their wedding vows, and the two apologize to each other. Dean is seen walking away from the house, with Frankie running after him. Dean tells Frankie to go back to her mom despite Frankie begging him to stay. Dean tricks Frankie by challenging her to a race in an attempt to send her back to Cindy, and he continues walking away while Cindy picks up an upset Frankie, who cries "I love him".
            Question: Who beats up Dean before the wedding?
            Answer: Bobby
            
            Passage: Prince Amerigo, an impoverished but charismatic Italian nobleman, is in London for his marriage to Maggie Verver, only child of the widower Adam Verver, the fabulously wealthy American financier and art collector. While there, he re-encounters Charlotte Stant, another young American and a former mistress from his days in Rome; they met in Mrs. Assingham's drawing room. Charlotte is not wealthy, which is one reason they did not marry. Maggie and Charlotte have been dear friends since childhood, although Maggie doesn't know of Charlotte and Amerigo's past relationship. Charlotte and Amerigo go shopping together for a wedding present for Maggie. They find a curiosity shop where the shopkeeper offers them an antique gilded crystal bowl. The Prince declines to purchase it, as he suspects it contains a hidden flaw.
            After Maggie's marriage, she is afraid that her father has become lonely, as they had been close for years. She persuades him to propose to Charlotte, who accepts Adam's proposal. Soon after their wedding, Charlotte and Amerigo are thrown together because their respective spouses seem more interested in their father-daughter relationship than in their marriages. Amerigo and Charlotte finally consummate an adulterous affair.
            Maggie begins to suspect the pair. She happens to go to the same shop and buys the golden bowl they had rejected. Regretting the high price he charged her, the shopkeeper visits Maggie and confesses to overcharging. At her home, he sees photographs of Amerigo and Charlotte. He tells Maggie of the pair's shopping trip on the eve of her marriage and their intimate conversation in his shop. (They had spoken Italian, but he understands the language.)
            Maggie confronts Amerigo. She begins a secret campaign to separate him and Charlotte while never revealing their affair to her father. Also concealing her knowledge from Charlotte and denying any change to their friendship, she gradually persuades her father to return to America with his wife. After previously regarding Maggie as a na誰ve, immature American, the Prince seems impressed by his wife's delicate diplomacy. The novel ends with Adam and Charlotte Verver about to depart for the United States. Amerigo says he can "see nothing but" Maggie and embraces her.
            Question: Where is Prince and Maggie to be married?
            Answer: London
            
            Passage: The film has a straightforward chronology with events spanning from 1937 to 1959; the setting is Tibet, except for brief sequences in China and India. It begins with the search for the 14th mindstream emanation of the Dalai Lama. After a vision by Reting Rinpoche (the regent of Tibet) several lamas disguised as servants discover a promising candidate: a child born to a farming family in the province of Amdo, near the Chinese border.
            These and other lamas administer a test to the child in which he must select from various objects the ones that belonged to the previous Dalai Lama. The child passes the test, and he and his family are brought to Potala Palace in Lhasa, where he will be installed as Dalai Lama when he comes of age.
            During the journey, the child becomes homesick and frightened, but is comforted by Reting, who tells him the story of the first Dalai Lama â whom the lamas called "Kundun." As the film progresses, the boy matures in both age and learning. After a brief power struggle in which Reting is imprisoned and dies, the Dalai Lama begins taking a more active role in governance and religious leadership.
            Meanwhile, the Chinese communists, recently victorious in their revolution, are proclaiming Tibet a traditional part of Imperial China and express their desire to reincorporate it with the newly formed People's Republic of China. Eventually, despite Tibet's pleas to the United Nations, the United States, the United Kingdom, and India for intervention, Chinese Communist forces invade Tibet. The Chinese are initially helpful, but when the Tibetans resist Communist reorganization and reeducation of their society, the Chinese become oppressive.
            Following a series of atrocities suffered by his people, the Dalai Lama resolves to meet with Chairman Mao Zedong in Beijing. While Mao initially expresses his sympathies to the Tibetan people and the Dalai Lama and insists that changes must be made as the Dalai Lama sees fit, relations inevitably deteriorate. During their face-to-face meeting on the final day of the Dalai Lama's visit, Mao makes clear his view that "religion is poison" and that the Tibetans are "poisoned and inferior" because of it.
            Upon his return to Tibet, the Dalai Lama learns of more horrors perpetrated against his people, who have by now repudiated their treaty with China and begun guerrilla action against the Chinese. After the Chinese make clear their intention to kill him, the Dalai Lama is convinced by his family and his Lord Chamberlain to flee to India.
            After consulting the Nechung Oracle about the proper escape route, the Dalai Lama and his staff put on disguises and slip out of Lhasa under cover of darkness. During an arduous journey, throughout which they are pursued by the Chinese, the Dalai Lama becomes very ill and experiences several visions of the past and future. The group eventually makes it to a small mountain pass on the Indian border. As the Dalai Lama walks to the guard post, an Indian guard approaches him, salutes, and inquires: "May I ask, are you the Lord Buddha?" The Dalai Lama replies with the film's final line: "I think that I am a reflection, like the moon on water. When you see me, and I try to be a good man, you see yourself."
            Once the Dalai Lama arrives at his new residence, he unpacks his telescope and steps outside. Erecting it and removing his spectacles, he gazes through it toward the Himalayas â and toward Tibet.
            The film concludes with two lines printed on screen:
            The Dalai Lama has not yet returned to Tibet. He hopes one day to make the journey.
            Question: What country is oppressing the Tibetians?
            Answer: China
            
            Passage: Protagonist Jack Holloway lives a solitary life in a wilderness of planet Zarathustra, itself "owned" by the Chartered Zarathustra Corporation (under Victor Grego), which installed basic services and colonial outposts initially, and now reaps the benefits of new discoveries, such as the valuable 'sunstones' mined by Holloway until he befriends a tiny, feline-like humanoid, nicknamed 'Little Fuzzy', and its traveling-companions. Upon discovery that the 'Fuzzies' intelligence may qualify them as a sapient species, the Company intervenes against them, on grounds that such a discovery would declare the planet a protected aboriginal zone, and deprive the Chartered Zarathustra Company of rights to the resources there. Leonard Kellogg, one of Grego's staff, kills a Fuzzy and thus provokes a court case to decide whether the Fuzzies are sapient. In the midst of the proceedings, the Terran Navy commander reveals that his people have been studying Fuzzies, and prove that Fuzzies have at least the mental capacity of a ten-year-old human child. At once, the Charter of the Zarathustra Company is invalidated, and Kellogg commits suicide in his cell.
            The second book, Fuzzy Sapiens, deals with the newly 'Charterless Zarathustra Company' and its gradual cooperation with the planet's new governor to ensure control over the planet, while the Fuzzies attach themselves to individual human guardians including the Company's staff. It becomes clear that criminals are using the irregular status of the government and of the company to attack it and steal sunstones.
            The third book, Fuzzy Bones by William Tuning, suggests that the remarkable demand by all Fuzzies for the ration-pack 'Extraterrestrial Type Three' (aka 'Extee 3' or 'estefee') does not coincide with the composition of Zarathustran soil. A third significant Fuzzy character is developed called Starwatcher. Little Fuzzy, Diamond, and Starwatcher become the clear leaders in working with humans.
            Golden Dream fits with these three books in terms of the general plot and relationships.
            After these two official sequels, the original third book by Piper himself offers an alternative future, wherein Little Fuzzy himself is separated from Jack Holloway and introduces a band of wild Fuzzies to the combined society.
            Question: Why are the government of Zarathustra and the now Charterless Zarathustra Company cooperating with each other?
            Answer: Criminals are taking advantage of the unstable government and company to attack and steal sunstones.
            
            Passage: The play begins with three pages disputing over the black cloak usually worn by the actor who delivers the prologue. They draw lots for the cloak, and one of the losers, Anaides, starts telling the audience what happens in the play to come; the others try to suppress him, interrupting him and putting their hands over his mouth. Soon they are fighting over the cloak and criticizing the author and the spectators as well.
            In the play proper, the goddess Diana, also called Cynthia, has ordained a "solemn revels" in the valley of Gargaphie in Greece. The gods Cupid and Mercury appear, and they too start to argue. Mercury has awakened Echo, who weeps for Narcissus, and states that a drink from Narcissus's spring causes the drinkers to "Grow dotingly enamored of themselves." The courtiers and ladies assembled for the Cynthia's revels all drink from the spring.
            Asotus, a foolish spendthrift who longs to become a courtier and a master of fashion and manners, also drinks from the spring; emboldened by vanity and self-love, he challenges all comers to a competition of "court compliment." The competition is held, in four phases, and the courtiers are beaten. Two symbolic masques are performed within the play for the assembled revelers. At their conclusion, Cynthia (representing Queen Elizabeth) has the dancers unmask and shows that vices have masqueraded as virtues. She sentences them to make reparation and to purify themselves by bathing in the spring at Mount Helicon.
            The figure of Actaeon in the play may represent Robert Devereux, 2nd Earl of Essex, while Cynthia's lady in waiting Arete may be Lucy, Countess of Bedford, one of Elizabeth's ladies in waiting as well as Jonson's patroness.
            The play is notably rich in music, as is typical for the theatre of the boys' companies, which originated as church choirs.
            Question: WHAT NAME WAS CYNTHIA MORE FAMOUSLY KNOWN BY?
            Answer:
          } [0.0s]
        } [0.0s]
      } [4.44s]
      470 requests
    } [4.44s]
    Executor.execute {
      Parallelizing computation on 470 items over 4 threads {
                                                                                                                                                                                                                           Created cache with config: SqliteCacheConfig(path='prod_env/cache/psa.sqlite')                                                                                                       | 0/470 [00:00<?, ?it/s]
        Created cache with config: SqliteCacheConfig(path='prod_env/cache/psa.sqlite')
        Created cache with config: SqliteCacheConfig(path='prod_env/cache/huggingface.sqlite')
        Created cache with config: SqliteCacheConfig(path='prod_env/cache/huggingface.sqlite')
        Loading /absolute_path_to_model/lora-out (kwargs={}) for HELM tokenizer psa/lora-out with Hugging Face Transformers {
        Created cache with config: SqliteCacheConfig(path='prod_env/cache/huggingface.sqlite')
        Created cache with config: SqliteCacheConfig(path='prod_env/cache/huggingface.sqlite')
        } [0.436s]
        Loading /absolute_path_to_model/lora-out (kwargs={'device_map': 'auto'}) for HELM model psa/lora-out with Hugging Face Transformers {
          Hugging Face device_map set to "auto".
          Loading Hugging Face model /absolute_path_to_model/lora-out {
2025-04-17 11:51:59.023065: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:477] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1744890719.033217  868717 cuda_dnn.cc:8310] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
E0000 00:00:1744890719.036273  868717 cuda_blas.cc:1418] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-04-17 11:51:59.047666: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
Unused kwargs: ['_load_in_4bit', '_load_in_8bit', 'quant_method']. These kwargs are not used in <class 'transformers.utils.quantization_config.BitsAndBytesConfig'>.
          } [3.402s]
        } [3.402s]
        
        Loading /absolute_path_to_model/lora-out (kwargs={'device_map': 'auto'}) for HELM model psa/lora-out with Hugging Face Transformers {
          Hugging Face device_map set to "auto".
          Loading Hugging Face model /absolute_path_to_model/lora-out {
Unused kwargs: ['_load_in_4bit', '_load_in_8bit', 'quant_method']. These kwargs are not used in <class 'transformers.utils.quantization_config.BitsAndBytesConfig'>.
            Traceback (most recent call last):
  File "/home/user/miniforge3/envs/helm/lib/python3.9/site-packages/retrying.py", line 251, in call
    attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
  File "/home/user/miniforge3/envs/helm/lib/python3.9/site-packages/helm/clients/auto_client.py", line 116, in make_request_with_retry
    return client.make_request(request)
  File "/home/user/miniforge3/envs/helm/lib/python3.9/site-packages/helm/clients/huggingface_client.py", line 301, in make_request
    huggingface_model: HuggingFaceServer = HuggingFaceServerFactory.get_server(
  File "/home/user/miniforge3/envs/helm/lib/python3.9/site-packages/helm/clients/huggingface_client.py", line 229, in get_server
    HuggingFaceServerFactory._servers[helm_model_name] = HuggingFaceServer(
  File "/home/user/miniforge3/envs/helm/lib/python3.9/site-packages/helm/clients/huggingface_client.py", line 104, in __init__
    self.model = AutoModelForCausalLM.from_pretrained(pretrained_model_name_or_path, **kwargs)
  File "/home/user/miniforge3/envs/helm/lib/python3.9/site-packages/transformers/models/auto/auto_factory.py", line 564, in from_pretrained
    return model_class.from_pretrained(
  File "/home/user/miniforge3/envs/helm/lib/python3.9/site-packages/transformers/modeling_utils.py", line 3730, in from_pretrained
    raise EnvironmentError(
OSError: Error no file named pytorch_model.bin, model.safetensors, tf_model.h5, model.ckpt.index or flax_model.msgpack found in directory /absolute_path_to_model/lora-out.

            Request failed. Retrying (attempt #2) in 10 seconds... (See above for error details)
          } [0.003s]
        } [0.003s]
        
        Loading /absolute_path_to_model/lora-out (kwargs={'device_map': 'auto'}) for HELM model psa/lora-out with Hugging Face Transformers {
          Hugging Face device_map set to "auto".
          Loading Hugging Face model /absolute_path_to_model/lora-out {
          Traceback (most recent call last):
  File "/home/user/miniforge3/envs/helm/lib/python3.9/site-packages/retrying.py", line 251, in call
    attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
  File "/home/user/miniforge3/envs/helm/lib/python3.9/site-packages/helm/clients/auto_client.py", line 116, in make_request_with_retry
    return client.make_request(request)
  File "/home/user/miniforge3/envs/helm/lib/python3.9/site-packages/helm/clients/huggingface_client.py", line 301, in make_request
    huggingface_model: HuggingFaceServer = HuggingFaceServerFactory.get_server(
  File "/home/user/miniforge3/envs/helm/lib/python3.9/site-packages/helm/clients/huggingface_client.py", line 229, in get_server
    HuggingFaceServerFactory._servers[helm_model_name] = HuggingFaceServer(
  File "/home/user/miniforge3/envs/helm/lib/python3.9/site-packages/helm/clients/huggingface_client.py", line 104, in __init__
    self.model = AutoModelForCausalLM.from_pretrained(pretrained_model_name_or_path, **kwargs)
  File "/home/user/miniforge3/envs/helm/lib/python3.9/site-packages/transformers/models/auto/auto_factory.py", line 564, in from_pretrained
    return model_class.from_pretrained(
  File "/home/user/miniforge3/envs/helm/lib/python3.9/site-packages/transformers/modeling_utils.py", line 3730, in from_pretrained
    raise EnvironmentError(
OSError: Error no file named pytorch_model.bin, model.safetensors, tf_model.h5, model.ckpt.index or flax_model.msgpack found in directory /absolute_path_to_model/lora-out.

            Request failed. Retrying (attempt #2) in 10 seconds... (See above for error details)
Unused kwargs: ['_load_in_4bit', '_load_in_8bit', 'quant_method']. These kwargs are not used in <class 'transformers.utils.quantization_config.BitsAndBytesConfig'>.
          } [0.002s]
        } [0.002s]
        
        Loading /absolute_path_to_model/lora-out (kwargs={'device_map': 'auto'}) for HELM model psa/lora-out with Hugging Face Transformers {
          Hugging Face device_map set to "auto".
          Loading Hugging Face model /absolute_path_to_model/lora-out {
            Traceback (most recent call last):
  File "/home/user/miniforge3/envs/helm/lib/python3.9/site-packages/retrying.py", line 251, in call
    attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
  File "/home/user/miniforge3/envs/helm/lib/python3.9/site-packages/helm/clients/auto_client.py", line 116, in make_request_with_retry
    return client.make_request(request)
  File "/home/user/miniforge3/envs/helm/lib/python3.9/site-packages/helm/clients/huggingface_client.py", line 301, in make_request
    huggingface_model: HuggingFaceServer = HuggingFaceServerFactory.get_server(
  File "/home/user/miniforge3/envs/helm/lib/python3.9/site-packages/helm/clients/huggingface_client.py", line 229, in get_server
    HuggingFaceServerFactory._servers[helm_model_name] = HuggingFaceServer(
  File "/home/user/miniforge3/envs/helm/lib/python3.9/site-packages/helm/clients/huggingface_client.py", line 104, in __init__
    self.model = AutoModelForCausalLM.from_pretrained(pretrained_model_name_or_path, **kwargs)
  File "/home/user/miniforge3/envs/helm/lib/python3.9/site-packages/transformers/models/auto/auto_factory.py", line 564, in from_pretrained
    return model_class.from_pretrained(
  File "/home/user/miniforge3/envs/helm/lib/python3.9/site-packages/transformers/modeling_utils.py", line 3730, in from_pretrained
    raise EnvironmentError(
OSError: Error no file named pytorch_model.bin, model.safetensors, tf_model.h5, model.ckpt.index or flax_model.msgpack found in directory /absolute_path_to_model/lora-out.

            Request failed. Retrying (attempt #2) in 10 seconds... (See above for error details)
Unused kwargs: ['_load_in_4bit', '_load_in_8bit', 'quant_method']. These kwargs are not used in <class 'transformers.utils.quantization_config.BitsAndBytesConfig'>.
          } [0.002s]
        } [0.002s]
```

See this line of stack trace:

```raw
  File "/home/user/miniforge3/envs/helm/lib/python3.9/site-packages/helm/clients/huggingface_client.py", line 104, in __init__
    self.model = AutoModelForCausalLM.from_pretrained(pretrained_model_name_or_path, **kwargs)
```

This is equivalent to calling:

```python3
AutoModelForCausalLM.from_pretrained(pretrained_model_name_or_path="/absolute_path_to_model/lora-out", device_map="auto")
```

You mentioned earlier that this call works, but HELM does not, which is surprising to me.

You mentioned you have a unmerged LoRA model - can you give me a complete example script of how you are loading this and running inference in `transformers` outside of HELM?

Hi @yifanmai , sorry for delayed answer, but I had lot of on my plate in work.

I found that indeed loading .from_pretrained does not work in then conda environment where I have helm installed. 

`OSError: Error no file named pytorch_model.bin, model.safetensors, tf_model.h5, model.ckpt.index or flax_model.msgpack found in directory /absolute_path_to_model/lora-out.`

But it works perfectly in the environment where I have axolotl installed. 

So I checked requirements file quickly and I see some differences, in helm there is not tlr and deepspeed. 
I tried to install tlr from source, and It didn't help me. 
Do you have any clue what should I do ?

Thanks

Could you try installing these additional requirements (tlr, deepspeed etc.) in your HELM virtual environment? Basically, you need to set up your environment to be able to `AutoModelForCausalLM.from_pretrained()` on your model.

Hi @yifanmai ,

it didn't help installing axolotl dependencies to helm environment.

But I tried to create new environment, installed axolotl and deps. there and then installed helm and it works now. 
Thank you for your efforts to help me, I appreciate it.