Repository: transformers
Issue: Early stopping required metric_for_best_model, but did not find eval_f1 so early stopping is disabled
Labels: Good Second Issue,bug

### System Info

- `transformers` version: 4.35.2
- Platform: Linux-3.10.0-1160.49.1.el7.x86_64-x86_64-with-glibc2.17
- Python version: 3.8.18
- Huggingface_hub version: 0.19.4
- Safetensors version: 0.4.1
- Accelerate version: 0.25.0
- Accelerate config:    not found
- PyTorch version (GPU?): 1.13.1+cu116 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

### Who can help?

@muellerzr @pacman100

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

```python
import os
from transformers import AutoTokenizer
from transformers import AutoModelForSequenceClassification
from transformers import TrainingArguments, Trainer
from transformers import EarlyStoppingCallback, IntervalStrategy
import numpy as np
import evaluate

os.environ["CUDA_VISIBLE_DEVICES"]=str(gpu_id)
from datasets import Dataset, DatasetDict
train_k = pd.read_csv('train.csv', usecols=["text", "k"])
train_k.rename(columns={"text":"text", "k":"label"}, inplace=True)
val_k = pd.read_csv('val.csv', usecols=["text", "k"])
val_k.rename(columns={"text":"text", "k":"label"}, inplace=True)
test_k = pd.read_csv('test.csv', usecols=["text", "k"])
test_k.rename(columns={"text":"text", "k":"label"}, inplace=True)

train_k = Dataset.from_pandas(train_k)
val_k = Dataset.from_pandas(val_k)
test_k = Dataset.from_pandas(test_k)

ds = DatasetDict()
ds['train'] = train_k
ds['val'] = val_k
ds['test'] = test_k

tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
def tokenize_function(examples):
    return tokenizer(str(examples['text']), padding="max_length", truncation=True)

tokenized_datasets = ds.map(tokenize_function)
tokenized_train_k = tokenized_datasets["train"]
tokenized_val_k = tokenized_datasets["val"]

model_k = AutoModelForSequenceClassification.from_pretrained("bert-base-uncased", num_labels=6)
training_args = TrainingArguments(output_dir="trained_k_predictors", evaluation_strategy="steps", eval_steps=100, metric_for_best_model = 'f1', learning_rate=1e-3, num_train_epochs=5, weight_decay=0.01, load_best_model_at_end=True, per_device_train_batch_size = 16, per_device_eval_batch_size = 32, save_total_limit = 3, optim="adafactor", label_names=['label'], remove_unused_columns=False,)
metric = evaluate.load("f1")
def compute_metrics(eval_pred):
    logits, labels = eval_pred
    predictions = np.argmax(logits, axis=-1)
    return {'f1': metric.compute(predictions=predictions, references=labels)}
trainer = Trainer(model=model_k, args=training_args, train_dataset=tokenized_train_k, eval_dataset=tokenized_val_k, compute_metrics=compute_metrics, callbacks = [EarlyStoppingCallback(early_stopping_patience=3)])
trainer.train()
```

## Error message:
{'eval_runtime': 21.6631, 'eval_samples_per_second': 208.926, 'eval_steps_per_second': 3.277, 'epoch': 0.47}                                                                                                       
  9%|████████████████                                                                                                                                                           | 5                       [26/1960]
00/5305 [06:15<41:00,  1.95it/s]
100%|████████████████████████████████████████████████████████████████████�     $                                                                                                                          [24/1960]
�█████████████████████████████████████████████�early stopping required metric_for_best_model, but did not find eval_f1 so                                                                                 [23/1960]
 early stopping is disabled██████████████████████████| 71/71 [00:21<00:00,  3.42it/s]
Traceback (most recent call last):                                                                                                                                                                                 
  File "/scratch/manish/apl/apl_env/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/scratch/manish/apl/apl_env/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/scratch/manish/apl/src/apl.py", line 386, in <module>
    main()
  File "/scratch/manish/apl/src/apl.py", line 139, in main
    trainer.train()
  File "/scratch/manish/apl/apl_env/lib/python3.8/site-packages/transformers/trainer.py", line 1555, in train
    return inner_training_loop(
  File "/scratch/manish/apl/apl_env/lib/python3.8/site-packages/transformers/trainer.py", line 1922, in _inner_training_loop
    self._maybe_log_save_evaluate(tr_loss, model, trial, epoch, ignore_keys_for_eval)
  File "/scratch/manish/apl/apl_env/lib/python3.8/site-packages/transformers/trainer.py", line 2282, in _maybe_log_save_evaluate
    self._save_checkpoint(model, trial, metrics=metrics)
  File "/scratch/manish/apl/apl_env/lib/python3.8/site-packages/transformers/trainer.py", line 2407, in _save_checkpoint
    metric_value = metrics[metric_to_check]
KeyError: 'eval_f1'
  9%|███████████████▉                                                                                                                                                         | 500                        [3/1960]
/5305 [06:18<1:00:34,  1.32it/s]



### Expected behavior

Train the model with early stopping enabled.