Spaces:

nroggendorff
/

train-llama

Paused

nroggendorff commited on 30 days ago

Commit

0d3291b

•

1 Parent(s): f9b4329

Update train.py

Files changed (1) hide show

train.py CHANGED Viewed

@@ -214,9 +214,9 @@ def main(push_to_hub=True, is_inst_finetune=False):
         tokenizer = load_tokenizer()
         print("Loaded Tokenizer.")
-        print("Adding Tokens..")
-        num_new_tokens = update_tokenizer(tokenizer, dataset)
-        print(f"Added {num_new_tokens} new tokens to the vocabulary")
     if INIT == 0:

         tokenizer = load_tokenizer()
         print("Loaded Tokenizer.")
+        # print("Adding Tokens..")
+        # num_new_tokens = update_tokenizer(tokenizer, dataset)
+        # print(f"Added {num_new_tokens} new tokens to the vocabulary")
     if INIT == 0: