Issue with vLLM
When used with vLLM, I get the following error
AttributeError: 'Arcade100kTokenizer' object has no attribute 'errors'
I tried to modify the tokenizer source code, to set 'errors' attribute. It leads to the following error
File "/home/priyansk/rtp2-src/generations_batch.py", line 405, in <module> main() File "/home/priyansk/rtp2-src/generations_batch.py", line 356, in main outputs = llm.generate(input_prompts_trimmed, sampling_params)
File "/home/priyansk/miniforge-pypy3/envs/vllm/lib/python3.10/site-packages/vllm/entrypoints/llm.py", line 172, in generate return self._run_engine(use_tqdm)File "/home/priyansk/miniforge-pypy3/envs/vllm/lib/python3.10/site-packages/vllm/entrypoints/llm.py", line 196, in _run_engine step_outputs = self.llm_engine.step()File "/home/priyansk/miniforge-pypy3/envs/vllm/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 756, in step return self._process_model_outputs(output, scheduler_outputs)File "/home/priyansk/miniforge-pypy3/envs/vllm/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 661, in _process_model_outputs self._process_sequence_group_outputs(seq_group, outputs)File "/home/priyansk/miniforge-pypy3/envs/vllm/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 532, in _process_sequence_group_outputs self._decode_sequence(seq, seq_group.sampling_params)
File "/home/priyansk/miniforge-pypy3/envs/vllm/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 837, in _decode_sequence read_offset) = detokenize_incrementally(
File "/home/priyansk/miniforge-pypy3/envs/vllm/lib/python3.10/site-packages/vllm/transformers_utils/tokenizer.py", line 131, in detokenize_incrementally new_tokens = tokenizer.convert_ids_to_tokens(
File "/home/priyansk/miniforge-pypy3/envs/vllm/lib/python3.10/site-packages/transformers/tokenization_utils.py", line 982, in convert_ids_to_tokens tokens.append(self._convert_id_to_token(index))
File "/home/priyansk/.cache/huggingface/modules/transformers_modules/stabilityai/stablelm-2-zephyr-1_6b/8e5b1aa67b32d39c29c489497a8194a3270b55a4/tokenization_arcade100k.py", line 261, in _convert_id_to_token raise ValueError(f"unknown ids {index}")ValueError: unknown ids
Hi,
@kpriyanshu256
! Thanks for reporting this. I've exposed the errors field in the tokenizer. If you'd like to try with VLLM support, you can give this branch a try until the PR is merged.