Text Generation
Transformers
PyTorch
llama
Inference Endpoints
text-generation-inference
bianchidotdev commited on
Commit
8f47b0f
1 Parent(s): bce5d60

Add tokenizer.json

Browse files
Files changed (2) hide show
  1. README.md +3 -1
  2. tokenizer.json +0 -0
README.md CHANGED
@@ -18,7 +18,9 @@ In this repo, we present a permissively licensed open source reproduction of Met
18
  We release the weights in two formats: an EasyLM format to be use with our [EasyLM framework](https://github.com/young-geng/EasyLM), and a PyTorch format to be used with the [Hugging Face transformers](https://huggingface.co/docs/transformers/index) library. Both our training framework EasyLM and the checkpoint weights are licensed permissively under the Apache 2.0 license.
19
 
20
  ### Loading the Weights with Hugging Face Transformers
21
- Preview checkpoints can be directly loaded from Hugging Face Hub. **Please note that it is advised to avoid using the Hugging Face fast tokenizer for now, as we’ve observed that** [**the auto-converted fast tokenizer sometimes gives incorrect tokenizations**](https://github.com/huggingface/transformers/issues/24233)**.** This can be achieved by directly using the `LlamaTokenizer` class, or passing in the `use_fast=False` option for the `AutoTokenizer` class. See the following example for usage.
 
 
22
 
23
  ```python
24
  import torch
 
18
  We release the weights in two formats: an EasyLM format to be use with our [EasyLM framework](https://github.com/young-geng/EasyLM), and a PyTorch format to be used with the [Hugging Face transformers](https://huggingface.co/docs/transformers/index) library. Both our training framework EasyLM and the checkpoint weights are licensed permissively under the Apache 2.0 license.
19
 
20
  ### Loading the Weights with Hugging Face Transformers
21
+ Preview checkpoints can be directly loaded from Hugging Face Hub.
22
+
23
+ See the following example for usage.
24
 
25
  ```python
26
  import torch
tokenizer.json ADDED
The diff for this file is too large to render. See raw diff