module:cuda/12.1/12.1.1 is provided for EXPERIMENTAL use You are using the default legacy behaviour of the . This is expected, and simply means that the `legacy` (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set `legacy=False`. This should only be set if you understand what it means, and thouroughly read the reason why this was added as explained in https://github.com/huggingface/transformers/pull/24565 /home/ace14459tv/cuda121/lib/python3.11/site-packages/transformers/convert_slow_tokenizer.py:473: UserWarning: The sentencepiece tokenizer that you are converting to a fast tokenizer uses the byte fallback option which is not implemented in the fast tokenizers. In practice this means that the fast version of the tokenizer can produce unknown tokens whereas the sentencepiece version would have converted these unknown tokens into a sequence of byte tokens matching the original piece of text. warnings.warn( Map: 0%| | 0/8939 [00:00 trainer.train() File "/home/ace14459tv/cuda121/lib/python3.11/site-packages/transformers/trainer.py", line 1537, in train return inner_training_loop( ^^^^^^^^^^^^^^^^^^^^ File "/home/ace14459tv/cuda121/lib/python3.11/site-packages/transformers/trainer.py", line 1929, in _inner_training_loop self._maybe_log_save_evaluate(tr_loss, model, trial, epoch, ignore_keys_for_eval) File "/home/ace14459tv/cuda121/lib/python3.11/site-packages/transformers/trainer.py", line 2279, in _maybe_log_save_evaluate self._save_checkpoint(model, trial, metrics=metrics) File "/home/ace14459tv/cuda121/lib/python3.11/site-packages/transformers/trainer.py", line 2359, in _save_checkpoint self._save_optimizer_and_scheduler(staging_output_dir) File "/home/ace14459tv/cuda121/lib/python3.11/site-packages/transformers/trainer.py", line 2462, in _save_optimizer_and_scheduler torch.save(self.optimizer.state_dict(), os.path.join(output_dir, OPTIMIZER_NAME)) File "/home/ace14459tv/cuda121/lib/python3.11/site-packages/torch/serialization.py", line 618, in save with _open_zipfile_writer(f) as opened_zipfile: File "/home/ace14459tv/cuda121/lib/python3.11/site-packages/torch/serialization.py", line 466, in __exit__ self.file_like.write_end_of_file() RuntimeError: [enforce fail at inline_container.cc:424] . unexpected pos 567552064 vs 567551952 wandb: - 0.016 MB of 0.016 MB uploaded wandb: \ 0.061 MB of 0.061 MB uploaded wandb: | 0.061 MB of 0.061 MB uploaded wandb: wandb: Run history: wandb: eval/accuracy ▂▁▁▁▇▇▇▇▇▇▇▇█▇▇█▇██▇████████████▇▇█████▇ wandb: eval/loss ▂▁▁▁▁▁▁▁▂▂▂▂▃▄▄▄▄▄▅▅▅▅▅▆▆▅▅▅▆▆▆▆▆▆▇▇▇▆██ wandb: eval/runtime ▂▂▂▁▂▂▁▁▁▁▃▂▂▂▃▃▂▂▁▂▂▅▂▃▂▂▃▂▁▁▁█▆▂▁▁▁▃▁▁ wandb: eval/samples_per_second ▆▇▇█▇▇████▆▇▇▇▅▆▇▇▇▇▆▄▇▆▇▇▆▇█▇█▁▃▇█▇█▆█▇ wandb: eval/steps_per_second ▆▇▇█▇▇████▆▇▇▇▅▆▇▇▇▇▆▄▇▆▇▇▆▇█▇█▁▃▇█▇█▆█▇ wandb: train/epoch ▁▁▁▁▂▂▂▂▂▃▃▃▃▃▃▄▄▄▄▄▅▅▅▅▅▆▆▆▆▆▆▇▇▇▇▇▇███ wandb: train/global_step ▁▁▁▁▂▂▂▂▂▃▃▃▃▃▃▄▄▄▄▄▅▅▅▅▅▆▆▆▆▆▆▇▇▇▇▇▇███ wandb: train/learning_rate ▁▂▃▄▆▆████▇▇▇▇▇▇▇▇▆▆▆▆▆▆▆▆▅▅▅▅▅▅▅▅▄▄▄▄▄▄ wandb: train/loss █▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁ wandb: wandb: Run summary: wandb: eval/accuracy 47.81 wandb: eval/loss 3.03867 wandb: eval/runtime 3.4147 wandb: eval/samples_per_second 327.698 wandb: eval/steps_per_second 20.499 wandb: train/epoch 60.0 wandb: train/global_step 33540 wandb: train/learning_rate 2e-05 wandb: train/loss 0.0138 wandb: wandb: 🚀 View run chromatic-wonton-107 at: https://wandb.ai/shion46/huggingface/runs/np2744rt wandb: Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s) wandb: Find logs at: ./wandb/run-20240224_072910-np2744rt/logs 60%|██████ | 33540/55900 [39:28<23:15, 16.02it/s]