Repository: huggingface/evaluate
Issue: Trainer Custom_loss - Empty metrics in Evaluate
Root cause: Documentation Deficiency

Hello everyone,

I am following the guide for [Token Classification from HF here.](https://huggingface.co/docs/transformers/tasks/token_classification)
It's working perfectly with my dataset. Unfortunately, I have an unbalanced dataset. So I am following the override with CustomTrainer for a loss function.

```
from torch import nn
from transformers import Trainer

class CustomTrainer(Trainer):
    def compute_loss(self, model, inputs, return_outputs=False):
        labels = inputs.get("labels")

        outputs = model(**inputs)
        logits = outputs.get("logits")

        # Class_weights are computed as a torch tensor (Similar to Lewis Tunstall tutorial)
        loss_fct = nn.CrossEntropyLoss(weight=class_weights)
        loss = loss_fct(logits.view(-1, self.model.config.num_labels), labels.view(-1))
        return (loss, outputs) if return_outputs else loss
````
And I don't forget to add label_names in the training_args.
```
training_args = TrainingArguments(
    ....
    label_names=["labels"],
    ....
)
```
However, Evaluate gave me weird metrics... No Training Loss, negative Validation loss.... no F1 score, and identical Accuracy.
```
Epoch	Training Loss	Validation Loss	Precision	Recall	F1	Accuracy
1	No log	-0.188098	0.000000	0.000000	0.000000	0.758095
2	No log	-0.195945	0.000000	0.000000	0.000000	0.758095

/usr/local/lib/python3.10/dist-packages/seqeval/metrics/v1.py:57: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.
  _warn_prf(average, modifier, msg_start, len(result))
/usr/local/lib/python3.10/dist-packages/seqeval/metrics/v1.py:57: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 due to no predicted samples. Use `zero_division` parameter to control this behavior.
  _warn_prf(average, modifier, msg_start, len(result))
TrainOutput(global_step=198, training_loss=0.12444540466925111, metrics={'train_runtime': 3245.6095, 'train_samples_per_second': 0.975, 'train_steps_per_second': 0.061, 'total_flos': 56929757663484.0, 'train_loss': 0.12444540466925111, 'epoch': 2.0})

```
I believe that @sgugger and @lewtun might have the solution. And I am willing to pay for a fresh beer or Pastis! : )

Thank you in advance,

Akim 

If you have no log for the training loss, it probably means your `logging_steps` training argument is higher than your `evaluation_steps`. As for the message form evaluate, it seems you have some labels that are not predicted by the model, which would make sense, at least at the beginning of training.

You should double-check your weights for the loss though, as having a negative validation loss seems weird to me. Also make sure to ask this on the [forums](https://discuss.huggingface.co/) so the community at large may help you :-)

Thank you for your help sgugger 

Unfortunately, I posted on the forum last week and updated it a few times but without a response.

**Update - Solution:** 
I blindly followed Lewis' tutorial for Sentiment analysis ... He was dividing by the ` / len(emotion_df)`. So my outlier the label `O`  was negative... 

As I am doing Token Classification:
I leverage the ` sum.()`  function to divide my class_weight.

````
class_weights1 = (1 - ( weight_labelsss / sum_labels )).values
class_weights1
````

Hopefully, it will help somebody...