Repository: stanford-crfm/helm
Issue: AutoClient `AttributeError: 'tuple' object has no attribute 'error'`
Root cause: Validation Gap

Possibly related to #1533. This issue is minor because it only shows up when the run would have failed anyway.

When running `narrative_qa:model=writer_palmyra-e,data_augmentation=canonical`:

```
Traceback (most recent call last):
  File "/u/scr/maiyifan/miniconda3/envs/helm-2023-04-29/lib/python3.8/site-packages/helm/proxy/clients/auto_client.py", line 155, in make_request
    return make_request_with_retry(client=client, request=request)
  File "/u/scr/maiyifan/miniconda3/envs/helm-2023-04-29/lib/python3.8/site-packages/helm/proxy/retry.py", line 68, in <lambda>
    return lambda f: lambda *args, **kwargs: _retrying.call(f, *args, **kwargs)
  File "/sailhome/maiyifan/.local/lib/python3.8/site-packages/retrying.py", line 268, in call
    raise RetryError(attempt)
retrying.RetryError: RetryError[Attempts: 8, Error:
  File "/sailhome/maiyifan/.local/lib/python3.8/site-packages/retrying.py", line 251, in call
    attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
  File "/u/scr/maiyifan/miniconda3/envs/helm-2023-04-29/lib/python3.8/site-packages/helm/proxy/clients/auto_client.py", line 150, in make_request_with_retry
    return client.make_request(request)
  File "/u/scr/maiyifan/miniconda3/envs/helm-2023-04-29/lib/python3.8/site-packages/helm/proxy/clients/palmyra_client.py", line 113, in make_request
    raise ValueError(f"Invalid response: {response}")
]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/u/scr/maiyifan/miniconda3/envs/helm-2023-04-29/lib/python3.8/site-packages/helm/benchmark/runner.py", line 121, in run_all
    self.run_one(run_spec)
  File "/u/scr/maiyifan/miniconda3/envs/helm-2023-04-29/lib/python3.8/site-packages/helm/benchmark/runner.py", line 195, in run_one
    scenario_state = self.executor.execute(scenario_state)
  File "/u/scr/maiyifan/miniconda3/envs/helm-2023-04-29/lib/python3.8/site-packages/helm/common/hierarchical_logger.py", line 104, in wrapper
    return fn(*args, **kwargs)
  File "/u/scr/maiyifan/miniconda3/envs/helm-2023-04-29/lib/python3.8/site-packages/helm/benchmark/executor.py", line 73, in execute
    request_states = parallel_map(
  File "/u/scr/maiyifan/miniconda3/envs/helm-2023-04-29/lib/python3.8/site-packages/helm/common/general.py", line 211, in parallel_map
    results = list(tqdm(executor.map(process, items), total=len(items), disable=None))
  File "/sailhome/maiyifan/.local/lib/python3.8/site-packages/tqdm/std.py", line 1183, in __iter__
    for obj in iterable:
  File "/u/scr/maiyifan/miniconda3/envs/helm-2023-04-29/lib/python3.8/concurrent/futures/_base.py", line 619, in result_iterator
    yield fs.pop().result()
  File "/u/scr/maiyifan/miniconda3/envs/helm-2023-04-29/lib/python3.8/concurrent/futures/_base.py", line 444, in result
    return self.__get_result()
  File "/u/scr/maiyifan/miniconda3/envs/helm-2023-04-29/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
    raise self._exception
  File "/u/scr/maiyifan/miniconda3/envs/helm-2023-04-29/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/u/scr/maiyifan/miniconda3/envs/helm-2023-04-29/lib/python3.8/site-packages/helm/benchmark/executor.py", line 83, in process
    result: RequestResult = self.service.make_request(self.execution_spec.auth, state.request)
  File "/u/scr/maiyifan/miniconda3/envs/helm-2023-04-29/lib/python3.8/site-packages/helm/proxy/services/server_service.py", line 103, in make_request
    request_result: RequestResult = self.client.make_request(request)
  File "/u/scr/maiyifan/miniconda3/envs/helm-2023-04-29/lib/python3.8/site-packages/helm/proxy/clients/auto_client.py", line 164, in make_request
    return replace(last_attempt.value, error=f"{retry_error}. Error: {last_attempt.value.error}")
AttributeError: 'tuple' object has no attribute 'error'
```