Repository: transformers
Issue: Missing `vocab_file` Attribute When Using Custom SentencePiece Models
Labels: Feature request,Good Difficult Issue

### System Info

- `transformers` version: 4.36.2
- Platform: Linux-6.1.69-1-lts-x86_64-with-glibc2.38
- Python version: 3.11.6
- Huggingface_hub version: 0.19.4
- Safetensors version: 0.4.0
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): 2.1.2+cpu (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: NA
- Using distributed or parallel set-up in script?: NA

### Who can help?

@ArthurZucker 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Steps to Reproduce

1. Import the SentencePiece library:

   ```python
   import sentencepiece as spm
   ```

2. Verify the SentencePiece version:

   ```python
   spm.__version__  # Ensure it is 0.1.98 or later
   ```

3. Attempt to create a `LlamaConverter` instance with a loaded SentencePiece tokenizer:

   ```python
   from transformers.convert_slow_tokenizer import LlamaConverter

   # Define the path to your SentencePiece model
   spm_model_path = "path/to/your/tokenizer.model"

   # Load the SentencePiece tokenizer
   spm_tokenizer = spm.SentencePieceProcessor()
   spm_tokenizer.Load(spm_model_path)

   # Create a LlamaConverter instance with the loaded tokenizer
   llama_converter = LlamaConverter(spm_tokenizer)
   ```

4. Observe the error raised due to the missing `vocab_file` attribute.

### Expected Behavior

To address this issue and make it easier for users to work with custom SentencePiece models, the following modifications are proposed:

1. **Modify `Converter` Class**:
   - Update the `Converter` class to accept a file path to the SentencePiece model when initializing an instance of the class.
   - Instantiate the SentencePiece tokenizer and load the model from the provided file path within the `Converter` class.

   ```python
   import sentencepiece as spm

   class Converter:
       def __init__(self, file_path_to_tokenizer):
           self.file_path = file_path_to_tokenizer
           self.original_tokenizer = spm.SentencePieceProcessor()
           self.original_tokenizer.Load(self.file_path)

       def converted(self) -> Tokenizer:
           raise NotImplementedError()
   ```

2. **Update `SpmConverter` Class**:
   - Update the `SpmConverter` class to use the `file_path` attribute instead of the non-existent `model_file` attribute when loading the SentencePiece model.
   - Ensure compatibility with the modifications made in the `Converter` class.

   ```python
   from transformers.convert_slow_tokenizer import Converter

   class SpmConverter(Converter):
       def __init__(self, file_path_to_tokenizer):
           requires_backends(self, "protobuf")
           super().__init__(file_path_to_tokenizer)

           # Load the SentencePiece model using the file path
           with open(self.file_path, "rb") as f:
               m.ParseFromString(f.read())
   ```

### Expected Outcome

With these proposed modifications, users should be able to create `Converter` instances with their custom SentencePiece models by providing the file path, and the `SpmConverter` class should correctly load the model using the specified file path, resolving the issue of missing attributes and ensuring compatibility with Transformers.