Repository: stanford-crfm/helm
Issue: HELM Lite results didn't display correctly with helm-server
Root cause: Configuration Error

I have run the helm lite eval on my model using the command mentioned in this issue
https://github.com/stanford-crfm/helm/issues/2262

I proceeded with ` helm-summarize --suite lite-eval-1000 --schema-file schema_lite.yaml` and `helm-server`.

However, the results cannot be displayed correctly. 

upon inspection, I noticed that the runSpecs.json is empty in the chrome console.

<img width="1512" alt="SCR-20240205-npfc" src="https://github.com/stanford-crfm/helm/assets/16046667/ef21c2dc-ba58-406f-bb5a-fcf323a795d5">


```
  Reading contamination information from contamination.yaml...
  validate_contamination {
    WARNING: model together/bloom not defined in schema
    WARNING: group the_pile not defined in schema
    WARNING: model together/gpt-j-6b not defined in schema
    WARNING: group the_pile not defined in schema
    WARNING: model together/gpt-neox-20b not defined in schema
    WARNING: group the_pile not defined in schema
    WARNING: model together/opt-66b not defined in schema
    WARNING: model together/opt-175b not defined in schema
    WARNING: group the_pile not defined in schema
    WARNING: group the_pile not defined in schema
    WARNING: group the_pile not defined in schema
    WARNING: model together/yalm not defined in schema
    WARNING: group the_pile not defined in schema
    WARNING: model together/t0pp not defined in schema
    WARNING: group hellaswag not defined in schema
    WARNING: group boolq not defined in schema
    WARNING: group summarization_xsum not defined in schema
    WARNING: group summarization_cnndm not defined in schema
    WARNING: group imdb not defined in schema
    WARNING: group hellaswag not defined in schema
    WARNING: group boolq not defined in schema
    WARNING: group quac not defined in schema
  } [0.0s]
0it [00:00, ?it/s]
  Summarizer.check_metrics_defined {
  } [0.0s]
  Parallelizing computation on 0 items over 8 threads {
0it [00:00, ?it/s]
  } [0.0s]
  Directory benchmark_output/runs/lite-eval-1000/data_overlap not found; skipped producing instance ids file.
  Directory benchmark_output/runs/lite-eval-1000/data_overlap not found; skipped import of overlap results.
  Writing 48710 characters to benchmark_output/runs/lite-eval-1000/schema.json
  Summarizer.write_executive_summary {
    Writing 99 characters to benchmark_output/runs/lite-eval-1000/summary.json
  } [0.004s]
  Writing 2 characters to benchmark_output/runs/lite-eval-1000/runs.json
  Writing 2 characters to benchmark_output/runs/lite-eval-1000/run_specs.json
  Writing 2 characters to benchmark_output/runs/lite-eval-1000/runs_to_run_suites.json
  Writing 2526 characters to benchmark_output/runs/lite-eval-1000/groups.json
  Writing 5038 characters to benchmark_output/runs/lite-eval-1000/groups_metadata.json
  Summarizer.write_cost_report {
    Writing 2 characters to benchmark_output/runs/lite-eval-1000/costs.json
  } [0.004s]
  Symlinking benchmark_output/runs/lite-eval-1000 to latest.
  Done.
} [0.68s]
```

The run_specs.json seems to be not populating correctly.

May i check with you on how to debug this issue?

Thank you


Hi, could you provide the exact `helm-run` commands that you used before running `helm-summarize`?

Closing due to staleness. Please reopen this issue if you are still running into this problem.