Repository: pytorch
Issue: torch.nn.CrossEntropyLoss: class weighting changes label_smoothing
Labels: module: nn,module: loss,triaged

### 🐛 Describe the bug

The class weighting in torch.nn.CrossEntropyLoss changes the probabilities that classification networks predict if use together with nonzero label smoothing. This leads to networks learning to predict wrong uncertainties in unbalanced datasets.
In extreme cases / datasets with rare classes the predicted probabilities are so wrong that the wrong class gets predicted.

I wrote a toy example below to reproduce this issue. It is a simple 2 layer network that gets as input a one hot vector with a class label and is supposed to predict the same class. As you can imagine it gets 100 % training accuracy with standard setting and predicts uncertainties according to the correct label smoothed probabilities.
The code snippet contains 4 tests with different class weighting.

When upweighting one of the classes (together with label smoothing) the training accuracy drops to 0.0008 because the network always puts 0.9147% probability on the rare class. This is because the label smoothing on the incorrect class is increased by the class weighting.

Training without label smoothing but with class weighting again gives 100% accuracy.

My expectation would be that class weighting does not change how much probability the label smoothing puts on the wrong classes.

Example:
(I use sklearn to automatically compute balanced class weights here but one can of course also set them manually.)

```Python
import torch
from torch import nn
from torch import optim
import torch.nn.functional as F
import numpy as np
from sklearn.utils.class_weight import compute_class_weight


num_classes = 4

# Test 1: balanced dataset + label smoothing -> balance class weights -> no problem
#probabilities = [0.25, 0.25, 0.25, 0.25]
#label_smoothing = 0.1

# Test 2: imbalanced dataset + label smoothing -> one class has 8x higher weight -> wrong uncertainties
#probabilities = [0.32, 0.04, 0.32, 0.32]
#label_smoothing = 0.1

# Test 3: rare class + label smoothing -> one class 333x higher weight -> wrong uncertainties + wrong predictions
probabilities = [0.333, 0.001, 0.333, 0.333]
label_smoothing = 0.1

# Test 4: rare class no label smoothing -> one class ~300x higher weight -> no problem
#probabilities = [0.333, 0.001, 0.333, 0.333]
#label_smoothing = 0.0

model = torch.nn.Sequential(nn.Linear(num_classes, 32),
                            nn.ReLU(inplace=True),
                            nn.Linear(32, num_classes))
optimizer = optim.AdamW(model.parameters(), lr=0.0003)

dataset_size = 10000
options = np.arange(num_classes)
# Normalize to make sure we have valid probabilities
probabilities = [float(i)/sum(probabilities) for i in probabilities]
print('Class probabilities in dataset', probabilities)
inputs = []
labels = []
labels_numpy = []

for _ in range(dataset_size):
  sample = np.random.choice(options, p=probabilities)
  labels_numpy.append(sample)
  tensor_sample = torch.from_numpy(np.array(sample))
  labels.append(tensor_sample)
  inputs.append(F.one_hot(tensor_sample, num_classes).to(torch.float32))



inputs = torch.stack(inputs)
labels = torch.stack(labels)
model.train()
class_weights = torch.tensor(compute_class_weight(class_weight='balanced', classes=np.arange(4), y=labels_numpy),
                             dtype=torch.float32)
print('Class weights', class_weights)
ce_loss = nn.CrossEntropyLoss(weight=class_weights, label_smoothing=label_smoothing)
optimizer.zero_grad()

for epoch in range(5000):
  prediction = model(inputs)
  loss = ce_loss(prediction, labels)
  if epoch % 100 == 0:
    print('Epoch:', epoch, loss)
  loss.backward()
  optimizer.step()
  optimizer.zero_grad()


final_logits = model(inputs)
final_prob = torch.nn.functional.softmax(final_logits, dim=1)
final_pred = torch.argmax(final_prob, dim=1)
print(final_prob[:10])
print(final_pred[:10])
print(labels[:10])
test = labels == final_pred
print('Accuracy:', sum(test) / len(test))
```

### Versions

[pip3] numpy==1.21.6
[pip3] pytorch-lightning==0.7.1
[pip3] torch==1.12.1+cu113
[pip3] torchaudio==0.12.1+cu113
[pip3] torchmetrics==0.11.0
[pip3] torchvision==0.13.1+cu113
[conda] numpy                     1.21.6                   pypi_0    pypi
[conda] pytorch-lightning         0.7.1                    pypi_0    pypi
[conda] torch                     1.12.1+cu113             pypi_0    pypi
[conda] torchaudio                0.12.1+cu113             pypi_0    pypi
[conda] torchmetrics              0.11.0                   pypi_0    pypi
[conda] torchvision               0.13.1+cu113             pypi_0    pypi


cc @albanD @mruberry @jbschlosser @walterddr @mikaylagawarecki