Extend Nemo AutoTokenizer & SentencePieceTokenizer API for TensorRT-LLM & AMMO evaluation scripts usage
State: closed | Pull request: yes
Labels: name: stale

name: common

# What does this PR do ?

Extending NeMo's AutoTokenizer and SentencePieceTokenizer for consistency with HuggingFace tokenizers API as it is used throughout TensorRT-LLM and AMMO in evaluation tools.

This is necessary to seamlessly enable model validation -- including quantized variants -- with the current scripts and to avoid code duplication.

Example use-cases to support:
- overall API [link1](https://github.com/NVIDIA/TensorRT-LLM/blob/118b3d7e7bab720d8ea9cd95338da60f7512c93a/tensorrt_llm/hlapi/tokenizer.py#L14-L46)
- `encode`: [link1](https://github.com/NVIDIA/TensorRT-LLM/blob/035b99e0d09d4f2dfdb949810cf7245112aa4165/examples/gpt/run_hf.py#L92) and [link2](https://github.com/NVIDIA/TensorRT-LLM/blob/118b3d7e7bab720d8ea9cd95338da60f7512c93a/examples/summarize.py#L162)
- `decode`: [link1](https://github.com/NVIDIA/TensorRT-LLM/blob/035b99e0d09d4f2dfdb949810cf7245112aa4165/examples/mmlu.py#L295)
- `batch_decode`: [link1](https://github.com/NVIDIA/TensorRT-LLM/blob/118b3d7e7bab720d8ea9cd95338da60f7512c93a/examples/summarize.py#L208)
- `batch_encode_plus` [link2](https://github.com/NVIDIA/TensorRT-LLM/blob/035b99e0d09d4f2dfdb949810cf7245112aa4165/tensorrt_llm/quantization/quantize_by_ammo.py#L204)
- setting  `pad_token_id`: [link1](https://github.com/NVIDIA/TensorRT-LLM/blob/118b3d7e7bab720d8ea9cd95338da60f7512c93a/examples/utils.py#L118-L119)

**Collection**: NLP

# Changelog 

- adding different variants of encode/decode methods
- adding `pad_token_id` (with setter enabled for HF AutoTokenizer wrapper) and `eos_token_id` property attributes for HF tokenizer

# Usage
See examples listed above.

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.


jenkins

jenkins

LGTM, but can we check if autotokenizer and local sentencepiece outputs match for the new methods implemented before we merge?

it is probably fine if they're not identical, but we need to know where the differences arise and also document them if any

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

This PR was closed because it has been inactive for 7 days since being marked as stale.