Repository: stanford-crfm/helm
Issue: Language modeling for GooseAI models: `gooseai/gpt-j-6b` and `gooseai/gpt-neo-20b`
Root cause: External Dependency Breakage

When we set `max_tokens=0` and `echo=True` in the request, we do not get any log probabilities for the original prompt in the response. This behavior is different from the OpenAI API. 

GooseAI is working on fixing this issue, but we don't have a concrete ETA. We may need a workaround.

Error from the big run:

```
  Error when running wikitext_103:model=default:
Traceback (most recent call last):
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/presentation/present.py", line 118, in run
    new_run_specs = run_benchmarking(
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/run.py", line 69, in run_benchmarking
    runner.run_all()
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/runner.py", line 75, in run_all
    self.run_one(run_spec)
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/runner.py", line 99, in run_one
    scenario_state = self.executor.execute(scenario_state)
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/common/hierarchical_logger.py", line 104, in wrapper
    return fn(*args, **kwargs)
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/executor.py", line 84, in execute
    request_states = list(
  File "/u/nlp/anaconda/main/anaconda3/envs/crfm_benchmarking/lib/python3.8/site-packages/tqdm/std.py", line 1195, in __iter__
    for obj in iterable:
  File "/u/nlp/anaconda/main/anaconda3/envs/crfm_benchmarking/lib/python3.8/concurrent/futures/_base.py", line 619, in result_iterator
    yield fs.pop().result()
  File "/u/nlp/anaconda/main/anaconda3/envs/crfm_benchmarking/lib/python3.8/concurrent/futures/_base.py", line 444, in result
    return self.__get_result()
  File "/u/nlp/anaconda/main/anaconda3/envs/crfm_benchmarking/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
    raise self._exception
  File "/u/nlp/anaconda/main/anaconda3/envs/crfm_benchmarking/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/executor.py", line 77, in process
    result: RequestResult = self.remote_service.make_request(self.execution_spec.auth, state.request)
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/proxy/remote_service.py", line 47, in make_request
    RemoteService._check_response(response)
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/proxy/remote_service.py", line 29, in _check_response
    raise RemoteServiceError(response["error"])
proxy.remote_service.RemoteServiceError: EXCEPTION: 'token_logprobs
```

@teetone It is hard to develop a workaround without making some big changes to the infra and breaking some abstractions. The current LM pipeline was designed assuming `max_tokens=0` is allowed for every model. Some issues I can think of:

1. In `run_specs.py`, `max_tokens=0` is set for every LM-related scenario regardless of model type. A possible solution is to override it in `adapter.py`.
3. If `max_tokens==1`, we need to exclude the last token when computing perplexity. This is not supported by the current infra. Changes need to be made in `basic_metrics.py` and `metrics.py` (and also `adapter.py`)
4. If the model needs to generate 1 token, then the max prompt length will be 1 token shorter (e.g. 2047 instead of 2048). This makes the comparison across models unfair. Also, a model's property (i.e. max_length) should not depend on whether it is hosted by GooseAI.
5. Due to the changes in max_lengths, `adapt_language_modeling()` and `adapt_language_modeling_minimal_pairs()` need to handle gooseai models and all other models separately.
6. It involves many stuffs so there may be other issues I haven't thought of.

The first two problems may be solved by adding `if model.startswith("gooseai")` here and there. It should be able to work but it will break the abstractions and is inelegant. The third issue cannot be solved. Also, if GooseAi fixes this in the future, then we will have to rerun everything (wasting quota) due to the change in prompt lengths and completely revert the workaround PR. Is it possible to follow up with GooseAI and check whether there is any progress?


We can't evaluate GooseAI models because of these bugs. We will rely on offline batch evaluations moving forward.