Repository: pytorch
Issue: ignore_index does not work as described with nn.CrossEntropyLoss
Labels: module: nn,triaged

## 🐛 Bug

Ignored targets/indices impact the output of nn.CrossEntropyLoss/torch.nn.functional.cross_entropy, have gradients associated with them, and impact the gradients of all other classes. This is a result of "ignore_index" only being used with nll_loss in the underlying implementation and not being used in the log_softmax. The log_softmax is over all the classes, including any classes that the loss function has been instructed to ignore, which means the ignored targets appear in the denominator for all the other classes.

If ignore_index is negative then the loss works as intended, but if the ignore_index is a valid index then it will not be ignored and it will impact the input gradient.
A similar issue exists with "weight" and zero-weight targets will also impact the gradient/loss - this is unavoidable but it probably should be clearly stated in the documentation as it could result in undesirable behavior. 

## To Reproduce

Repo case showing the impact on the loss (l2 should produce the same loss for all three examples as it ignores index 0):
```python
>>> import torch
>>> x1 = torch.as_tensor([[1., 1., 1.], [1., 1., 1.]])
>>> x2 = torch.as_tensor([[0., 1., 1.], [0., 1., 1.]])
>>> x3 = torch.as_tensor([[float("-inf"), 1., 1.], [float("-inf"), 1., 1.]])
>>> y = torch.as_tensor([1, 2])
>>> l1 = torch.nn.CrossEntropyLoss(reduction="none")
>>> l2 = torch.nn.CrossEntropyLoss(ignore_index=0, reduction="none")
>>> l3 = torch.nn.CrossEntropyLoss(weight=torch.as_tensor([0., 1., 1.]), reduction="none")
>>> l1(x1,y)
tensor([1.0986, 1.0986])
>>> l1(x2,y)
tensor([0.8620, 0.8620])
>>> l1(x3,y)
tensor([0.6931, 0.6931])
>>> l2(x1,y)
tensor([1.0986, 1.0986])
>>> l2(x2,y)
tensor([0.8620, 0.8620])
>>> l2(x3,y)
tensor([0.6931, 0.6931])
>>> l3(x1,y)
tensor([1.0986, 1.0986])
>>> l3(x2,y)
tensor([0.8620, 0.8620])
>>> l3(x3,y)
tensor([0.6931, 0.6931])
```
Repo case showing that "ignore_index" has a gradient associated with it (the gradient for the first column should be 0):
```python
>>> x = y = torch.as_tensor([1,2,3])
>>> m = torch.nn.Embedding(4,4,padding_idx=0)
>>> crit = torch.nn.CrossEntropyLoss(ignore_index=0)
>>> y_hat = m(x)
>>> loss = crit(y_hat,y)
>>> loss.backward()
>>> m.weight.grad
tensor([[ 0.0000,  0.0000,  0.0000,  0.0000],
        [ 0.0436, -0.1709,  0.0204,  0.1069],
        [ 0.0729,  0.1171, -0.2597,  0.0698],
        [ 0.1109,  0.0582,  0.0490, -0.2181]])
```


## Expected behavior
Expected outputs for the first repo case showing the impact on the loss:
```python
>>> import torch
>>> x1 = torch.as_tensor([[1., 1., 1.], [1., 1., 1.]])
>>> x2 = torch.as_tensor([[0., 1., 1.], [0., 1., 1.]])
>>> x3 = torch.as_tensor([[float("-inf"), 1., 1.], [float("-inf"), 1., 1.]])
>>> y = torch.as_tensor([1, 2])
>>> l1 = torch.nn.CrossEntropyLoss(reduction="none")
>>> l2 = torch.nn.CrossEntropyLoss(ignore_index=0, reduction="none")
>>> l3 = torch.nn.CrossEntropyLoss(weight=torch.as_tensor([0., 1., 1.]), reduction="none")
>>> l1(x1,y)
tensor([1.0986, 1.0986])
>>> l1(x2,y)
tensor([0.8620, 0.8620])
>>> l1(x3,y)
tensor([0.6931, 0.6931])
>>> l2(x1,y)
tensor([0.6931, 0.6931])
>>> l2(x2,y)
tensor([0.6931, 0.6931])
>>> l2(x3,y)
tensor([0.6931, 0.6931])
>>> l3(x1,y)
tensor([1.0986, 1.0986])
>>> l3(x2,y)
tensor([0.8620, 0.8620])
>>> l3(x3,y)
tensor([0.6931, 0.6931])
```
Repo case showing that "ignore_index" has a gradient associated with it (the gradient for the first column should be 0):
```python
>>> x = y = torch.as_tensor([1,2,3])
>>> m = torch.nn.Embedding(4,4,padding_idx=0)
>>> crit = torch.nn.CrossEntropyLoss(ignore_index=0)
>>> y_hat = m(x)
>>> loss = crit(y_hat,y)
>>> loss.backward()
>>> m.weight.grad
tensor([[ 0.0000,  0.0000,  0.0000,  0.0000],
        [ 0.0000, -0.1709,  0.0204,  0.1069],
        [ 0.0000,  0.1171, -0.2597,  0.0698],
        [ 0.0000,  0.0582,  0.0490, -0.2181]])
```

## Environment
N/A

## Additional context
I'm currently working around this issue by using a boolean ignore mask (vector of length num_classes): 
```python
class CrossEntropyLoss(torch.nn.CrossEntropyLoss):
    def __init__(self, weight=None, ignore_index=-100, ignore_mask=None, reduction='mean'):
        self.ignore_mask = ignore_mask
        if ignore_mask is not None and weight is None:
            weight = ignore_mask.eq(False).float()
        super().__init__(weight=weight, ignore_index=ignore_index, reduction=reduction)

    def forward(self, input: Tensor, target: Tensor) -> Tensor:
        masked = input.clone()
        masked[:, self.ignore_mask] = float("-inf")
```
There's some previous discussion of this issue in #25166, but it was closed with a workaround.


cc @albanD @mruberry @jbschlosser