Repository: stanford-crfm/helm
Issue: Split code copyright from copyright to avoid merging in main table
GitHub labels: p2

Visualization

We have discussed this, and came up during working session today. Leads to confusion, since people interpret the code and non-code as being the same in the primary table, which then leads to the incorrect interpretation that code models copy more.

We could break this up into two groups, or we can keep it as one group and have a flag in the group that determines whether we want to aggregate (or how - we might imagine also wanting to report differences for performance disparities).  Perhaps we should do the latter.  Edit: this comment really applies to #904, which is a more general issue.

@lxuechen Could you code from non-code for now?

Let me do the latter now. 

As opposed to myself doing zero-shot learning here, could @percyliang point me toward an example that groups things together properly. I'm reading through the `presentation` folder, but things are going very slowly. 

In `get_copyright_spec` in `run_specs.py` you could set the `groups` to be `copyright_code` depending on the parameters.

After #906, when I run the following:

```
benchmark-present --conf src/benchmark/presentation/run_specs_small.conf --local --max-eval-instances 10 --suite v1
benchmark-summarize --suite v1
```

I get:

```
Traceback (most recent call last):
  File "/home/yifanmai/oss/benchmarking/venv/bin/benchmark-summarize", line 8, in <module>
    sys.exit(main())
  File "/home/yifanmai/oss/benchmarking/src/common/hierarchical_logger.py", line 104, in wrapper
    return fn(*args, **kwargs)
  File "/home/yifanmai/oss/benchmarking/src/benchmark/presentation/summarize.py", line 592, in main
    summarizer.write_groups()
  File "/home/yifanmai/oss/benchmarking/src/benchmark/presentation/summarize.py", line 522, in write_groups
    json.dumps(list(map(asdict_without_nones, self.create_index_tables()))),
  File "/home/yifanmai/oss/benchmarking/src/benchmark/presentation/summarize.py", line 281, in create_index_tables
    for subgroup in self.expand_subgroups(group)
  File "/home/yifanmai/oss/benchmarking/src/benchmark/presentation/summarize.py", line 259, in expand_subgroups
    return expand_subgroups_(group, visited=set())
  File "/home/yifanmai/oss/benchmarking/src/benchmark/presentation/summarize.py", line 253, in expand_subgroups_
    return [group] + [
  File "/home/yifanmai/oss/benchmarking/src/benchmark/presentation/summarize.py", line 256, in <listcomp>
    for subsubgroup in expand_subgroups_(self.schema.name_to_run_group[subgroup], visited)
KeyError: 'copyright'
```

Is there something I need to do to get the copyright group to work?

> After #906, when I run the following:
> 
> ```
> benchmark-present --conf src/benchmark/presentation/run_specs_small.conf --local --max-eval-instances 10 --suite v1
> benchmark-summarize --suite v1
> ```
> 
> I get:
> 
> ```
> Traceback (most recent call last):
>   File "/home/yifanmai/oss/benchmarking/venv/bin/benchmark-summarize", line 8, in <module>
>     sys.exit(main())
>   File "/home/yifanmai/oss/benchmarking/src/common/hierarchical_logger.py", line 104, in wrapper
>     return fn(*args, **kwargs)
>   File "/home/yifanmai/oss/benchmarking/src/benchmark/presentation/summarize.py", line 592, in main
>     summarizer.write_groups()
>   File "/home/yifanmai/oss/benchmarking/src/benchmark/presentation/summarize.py", line 522, in write_groups
>     json.dumps(list(map(asdict_without_nones, self.create_index_tables()))),
>   File "/home/yifanmai/oss/benchmarking/src/benchmark/presentation/summarize.py", line 281, in create_index_tables
>     for subgroup in self.expand_subgroups(group)
>   File "/home/yifanmai/oss/benchmarking/src/benchmark/presentation/summarize.py", line 259, in expand_subgroups
>     return expand_subgroups_(group, visited=set())
>   File "/home/yifanmai/oss/benchmarking/src/benchmark/presentation/summarize.py", line 253, in expand_subgroups_
>     return [group] + [
>   File "/home/yifanmai/oss/benchmarking/src/benchmark/presentation/summarize.py", line 256, in <listcomp>
>     for subsubgroup in expand_subgroups_(self.schema.name_to_run_group[subgroup], visited)
> KeyError: 'copyright'
> ```
> 
> Is there something I need to do to get the copyright group to work?

You mean after #1055? I think I should have fixed the schema thing. Is this run with latest commit on main? 

Yes, this is on main `7ba6024`.

Given the implementation of this is fixed, can we close this issue now? Or are we waiting for the next batch of results before closing this? @yifanmai @teetone 

This is updated now.