Repository: stanford-crfm/helm
Issue: Yi-large Model deployment not found / Qwen2-72b-instruct tokenizer issues
Root cause: Unimplemented Feature Gap

Got this issue when running with yi-large:

`"helm.proxy.services.remote_service.RemoteServiceError: EXCEPTION: Model deployment 01-ai/yi-large not found"
`
And got this issue when running with qwen2-72b-instruct:

```
File "/home/benchmarking/src/helm/tokenizers/auto_tokenizer.py", line 56, in _get_tokenizer
    assert isinstance(tokenizer, Tokenizer)  # To make mypy happy
```

I'm wondering what's the model deployment for yi-large (also what's the difference between yi-large and yi-large-preview) and the tokenizer for qwen2-72b-instruct (i've been using "tokenizer": "qwen/qwen2-72b-instruct" which is found in the model_deployments.yaml file).
Thanks!

Hi @Joey-Ji,

> `"helm.proxy.services.remote_service.RemoteServiceError: EXCEPTION: Model deployment 01-ai/yi-large not found"`

We just released a package (v0.5.2) with `01-ai/yi-large` and `01-ai/yi-large-preview` today. The older versions of the package do not have these models. You can either use that package (`pip install --upgrade crfm-helm`), or install from the GitHub repository main branch.

> And got this issue when running with qwen2-72b-instruct

Could you post the logs for the error, including the error message?

> also what's the difference between yi-large and yi-large-preview

These are two _different_ models provided by 01.AI. I wasn't able to find much public information about these models. You could try contacting 01.AI with your question directly.

> the tokenizer for qwen2-72b-instruct

It should be OK to use this tokenizer.

@yifanmai Thanks for the reply! I think the root cause is that i was still using 0.5.0 and I think the upgrade will solve those two issues. I will try running again later and if there's any more errors, i will post it here. Thanks!

@yifanmai

Bumped to `crfm-helm==0.5.2` but still receiving an error on `TokenizationRequest` with `tokenizer_name="qwen/qwen2-72b-instruct"`.

## Code Snippet
```python
request = TokenizationRequest(tokenizer=tokenizer_name, text=model_input)
        tokenization_result: TokenizationRequestResult = self.crfm_service.tokenize(
            self.crfm_auth, request
        )
```

## Exception
```bash
Exception has occurred: RemoteServiceError
EXCEPTION: 
  File "/home/benchmarking/src/helm/proxy/server.py", line 67, in safe_call
    result = func(params)

  File "/home/benchmarking/src/helm/proxy/server.py", line 219, in perform
    return dataclasses.asdict(service.tokenize(auth, request))

  File "/home/benchmarking/src/helm/proxy/services/server_service.py", line 156, in tokenize
    return self.tokenizer.tokenize(request)

  File "/home/benchmarking/src/helm/tokenizers/auto_tokenizer.py", line 68, in tokenize
    tokenizer: Tokenizer = self._get_tokenizer(request.tokenizer)

  File "/home/benchmarking/src/helm/tokenizers/auto_tokenizer.py", line 56, in _get_tokenizer
    assert isinstance(tokenizer, Tokenizer)  # To make mypy happy
 Request: {"text": "<REDACTED>", "tokenizer": "qwen/qwen2-72b-instruct", "encode": false, "truncation": false, "max_length": 2048}
  File "/Users/justinwlin/Projects/research/REDACTED/agent/agent.py", line 408, in _get_num_tokens_from_helm
    tokenization_result: TokenizationRequestResult = self.crfm_service.tokenize(
  File "/Users/justinwlin/Projects/research/REDACTED/agent/agent.py", line 445, in _truncate_input_to_max_tokens
    num_input_tokens = self._get_num_tokens_from_helm(model_input)
  File "/Users/justinwlin/Projects/research/REDACTED/agent/agent.py", line 503, in _process_iteration
    truncated_input = self._truncate_input_to_max_tokens(str(self.chat_chain))
  File "/Users/justinwlin/Projects/research/REDACTED/agent/agent.py", line 617, in _process_subtask
    iteration_result = self._process_iteration(subtask_index, iteration)
  File "/Users/justinwlin/Projects/research/REDACTED/agent/agent.py", line 700, in run
    subtask_completion = self._process_subtask(
  File "/Users/justinwlin/Projects/research/REDACTED/run_challenge.py", line 401, in run_challenge
    for subtask_index, subtask_completion in enumerate(
  File "/Users/justinwlin/Projects/research/REDACTED/run_challenge.py", line 597, in main
    runner.run_challenge(
  File "/Users/justinwlin/Projects/research/REDACTED/run_challenge.py", line 612, in <module>
    main()
helm.proxy.services.remote_service.RemoteServiceError: EXCEPTION: 
  File "/home/benchmarking/src/helm/proxy/server.py", line 67, in safe_call
    result = func(params)

  File "/home/benchmarking/src/helm/proxy/server.py", line 219, in perform
    return dataclasses.asdict(service.tokenize(auth, request))

  File "/home/benchmarking/src/helm/proxy/services/server_service.py", line 156, in tokenize
    return self.tokenizer.tokenize(request)

  File "/home/benchmarking/src/helm/tokenizers/auto_tokenizer.py", line 68, in tokenize
    tokenizer: Tokenizer = self._get_tokenizer(request.tokenizer)

  File "/home/benchmarking/src/helm/tokenizers/auto_tokenizer.py", line 56, in _get_tokenizer
    assert isinstance(tokenizer, Tokenizer)  # To make mypy happy
 Request: {"text": "<REDACTED>", "tokenizer": "qwen/qwen2-72b-instruct", "encode": false, "truncation": false, "max_length": 2048}
 ```

`qwen/qwen2-72b-instruct` is currently not supported by the remote service. I can upgrade the service later this week.

Thanks Yifan!

I added `qwen/qwen2-72b-instruct` to the remote service. Let me know if it works for you.

It's working -- thanks @yifanmai!

It also works for me, thanks! @yifanmai