Repository: stanford-crfm/helm
Issue: Google Authentication Error
Root cause: Environment Incompatibility

When I try to run the following configuration, 

`entries: [{description: "summarization_xsum:temperature=0.3,model=huggingface/gpt2", priority: 1}]`

I get the following error

`ToxicityMetric() {
        Parallelizing computation on 1 items over 4 threads {
                                                                                                                                                                                                          WARNING:google.auth.compute_engine._metadata:Compute Engine Metadata server unavailable on attempt 1 of 3. Reason: timed out                                                          | 0/1 [00:00<?, ?it/s]
WARNING:google.auth.compute_engine._metadata:Compute Engine Metadata server unavailable on attempt 2 of 3. Reason: [Errno 113] No route to host
WARNING:google.auth.compute_engine._metadata:Compute Engine Metadata server unavailable on attempt 3 of 3. Reason: timed out
WARNING:google.auth._default:Authentication failed using Compute Engine authentication due to unavailable metadata server.
          Request failed. Retrying (attempt #2) in 6 seconds...
WARNING:google.auth.compute_engine._metadata:Compute Engine Metadata server unavailable on attempt 1 of 3. Reason: timed out
WARNING:google.auth.compute_engine._metadata:Compute Engine Metadata server unavailable on attempt 2 of 3. Reason: [Errno 113] No route to host
WARNING:google.auth.compute_engine._metadata:Compute Engine Metadata server unavailable on attempt 3 of 3. Reason: timed out
WARNING:google.auth._default:Authentication failed using Compute Engine authentication due to unavailable metadata server.
          Request failed. Retrying (attempt #3) in 22 seconds...
WARNING:google.auth.compute_engine._metadata:Compute Engine Metadata server unavailable on attempt 1 of 3. Reason: timed out
WARNING:google.auth.compute_engine._metadata:Compute Engine Metadata server unavailable on attempt 2 of 3. Reason: [Errno 113] No route to host
WARNING:google.auth.compute_engine._metadata:Compute Engine Metadata server unavailable on attempt 3 of 3. Reason: timed out
WARNING:google.auth._default:Authentication failed using Compute Engine authentication due to unavailable metadata server.
          Request failed. Retrying (attempt #4) in 48 seconds...`


Could you please help me figure out what the issue might be or what I could do to solve this? Thank you!

Could you share the command that you're trying to run?

Does the machine you're running on have network access to GCP? This looks like an authentication error, except it seems to be complaining about network unreachability, rather than the usual missing credentials error.

This also looks like it could be related to Perspective API; I'd suggest [setting up the Perspective API key](https://crfm-helm.readthedocs.io/en/latest/benchmark/#perspective-api) just in case. Alternatively, you can remove usages of the Perspective API client from the code and see if it works (if you're running from `main`, the Perspective API client will only instantiated if toxicity metrics are used, whereas in the older releases, it is instantiated eagerly).

@yifanmai Could you explain how to turn off Perspective API? I think for most of the users, we do not want to use a service that only process one query/sec ^^

The out of quota error I encounter with Perspective API:
`          Request failed. Retrying (attempt #2) in 0 seconds...
          Error was thrown when making a request to Perspective API: <HttpError 429 when requesting https://commentanalyzer.googleapis.com/v1alpha1/comments:analyze?key=AIzaSyDx43YapHL_xs_8N6sEZAwOsi1EfYBF9R4&alt=json returned "Quota exceeded for quota metric 'Analysis requests (AnalyzeComment)' and limit 'Analysis requests (AnalyzeComment) per minute' of service 'commentanalyzer.googleapis.com' for consumer 'project_number:401526159181'.". Details: "[{'@type': 'type.googleapis.com/google.rpc.ErrorInfo', 'reason': 'RATE_LIMIT_EXCEEDED', 'domain': 'googleapis.com', 'metadata': {'quota_location': 'global', 'quota_limit_value': '60', 'service': 'commentanalyzer.googleapis.com', 'quota_limit': 'AnalyzeRequestsPerMinutePerProject', 'consumer': 'projects/401526159181', 'quota_metric': 'CommentAnalyzerService/analyze_requests'}}, {'@type': 'type.googleapis.com/google.rpc.Help', 'links': [{'description': 'Request a higher quota limit.', 'url': 'https://cloud.google.com/docs/quota#requesting_higher_quota'}]}]">
          Request failed. Retrying (attempt #2) in 0 seconds...`

@yifanmai The command I am running: 

`helm-run --conf-paths run_specs.conf --local --suite v2 --max-eval-instances 10`

The `run_specs.conf` has the following configuration:

`entries: [{description: "summarization_xsum:model=huggingface/gpt2,temperature=0.3,device=cuda", priority: 1}]`

I haven't set up Perspective API, I will try it and get back to you. 

@shalakasatheesh hello, I have the same problem. Have you solved it？

> 

the simplest way is returning the 0-value from the toxicity metric, replace `evaluate_generation` in `toxicity_metrics.py` with the following.

```python
    def evaluate_generation(
        self,
        adapter_spec: AdapterSpec,
        request_state: RequestState,
        metric_service: MetricService,
        eval_cache_path: str,
    ) -> List[Stat]:
        stats: List[Stat] = [
            # Metrics from the RealToxicityPrompts paper
            Stat(MetricName("expected_max_toxicity")).add(0),
            Stat(MetricName("max_toxicity_probability")).add(True),
            # Additional metrics we added
            Stat(MetricName("toxic_frac")).add(0),
        ]
        return stats
```

Should be fixed after #1554 is merged

Apparently this is still broken...

Command: `venv/bin/helm-run -r self_instruct:model=anthropic/claude-v1.3 -m 10 -t 1 --suite 2`
Error Message: 
```
WARNING:google.auth.compute_engine._metadata:Compute Engine Metadata server unavailable on attempt 1 of 3. Reason: timed out
WARNING:google.auth.compute_engine._metadata:Compute Engine Metadata server unavailable on attempt 1 of 3. Reason: timed out
WARNING:google.auth.compute_engine._metadata:Compute Engine Metadata server unavailable on attempt 1 of 3. Reason: timed out
WARNING:google.auth.compute_engine._metadata:Compute Engine Metadata server unavailable on attempt 1 of 3. Reason: timed out
WARNING:google.auth.compute_engine._metadata:Compute Engine Metadata server unavailable on attempt 2 of 3. Reason: timed out
WARNING:google.auth.compute_engine._metadata:Compute Engine Metadata server unavailable on attempt 3 of 3. Reason: [Errno 64] Host is down
WARNING:google.auth._default:Authentication failed using Compute Engine authentication due to unavailable metadata server.
          
          Traceback (most recent call last):
  File "HELM/venv/lib/python3.8/site-packages/retrying.py", line 200, in call
    attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
  File "HELM/src/helm/proxy/services/server_service.py", line 128, in get_toxicity_scores_with_retry
    return self.perspective_api_client.get_toxicity_scores(request)
  File "HELM/src/helm/proxy/clients/perspective_api_client.py", line 89, in get_toxicity_scores
    client = self._get_or_initialize_client()
  File "HELM/src/helm/proxy/clients/perspective_api_client.py", line 71, in _get_or_initialize_client
    self.client = discovery.build(
  File "HELM/venv/lib/python3.8/site-packages/googleapiclient/_helpers.py", line 130, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "HELM/venv/lib/python3.8/site-packages/googleapiclient/discovery.py", line 298, in build
    service = build_from_document(
  File "HELM/venv/lib/python3.8/site-packages/googleapiclient/_helpers.py", line 130, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "HELM/venv/lib/python3.8/site-packages/googleapiclient/discovery.py", line 580, in build_from_document
    credentials = _auth.default_credentials(
  File "HELM/venv/lib/python3.8/site-packages/googleapiclient/_auth.py", line 57, in default_credentials
    credentials, _ = google.auth.default(
  File "HELM/venv/lib/python3.8/site-packages/google/auth/_default.py", line 616, in default
    raise exceptions.DefaultCredentialsError(_HELP_MESSAGE)
google.auth.exceptions.DefaultCredentialsError: Could not automatically determine credentials. Please set GOOGLE_APPLICATION_CREDENTIALS or explicitly create credentials and re-run the application. For more information, please see https://cloud.google.com/docs/authentication/getting-started

```