Repository: transformers
Issue: DETR: IndexError: Caught IndexError in replica 0 on device 0. IndexError: index 8 is out of bounds for dimension 0 with size 8
Labels: Good Second Issue,Vision

### System Info

- `transformers` version: 4.37.1
- Platform: Linux-6.2.0-32-generic-x86_64-with-glibc2.37
- Python version: 3.10.12
- Huggingface_hub version: 0.20.3
- Safetensors version: 0.4.2
- Accelerate version: 0.26.1
- Accelerate config: 	not found
- PyTorch version (GPU?): 2.1.2+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: Not explicitly, but Trainer is picking up 2 GPUs

### Who can help?

@amyeroberts Hi. I'm getting the error in the title trying to reproduce [this example](https://huggingface.co/docs/transformers/tasks/object_detection). The error is real. I don't know what caused it, but I've narrowed the cause to DETR receiving `BatchSize x NumGPUs` number of targets, but expecting only `BatchSize` which causes the overflow. If I limit the amount of GPUs to 1 (via `CUDA_VISIBLE_DEVICES=0`, for example), it runs ok.

Here's the stack trace:
```
Traceback (most recent call last):
  File "/home/mgruner/cellphones-in-the-wild/./train.py", line 116, in <module>
    trainer.train()
  File "/opt/pyenv/versions/cellphones-in-the-wild/lib/python3.10/site-packages/transformers/trainer.py", line 1539, in train
    return inner_training_loop(
  File "/opt/pyenv/versions/cellphones-in-the-wild/lib/python3.10/site-packages/transformers/trainer.py", line 1869, in _inner_training_loop
    tr_loss_step = self.training_step(model, inputs)
  File "/opt/pyenv/versions/cellphones-in-the-wild/lib/python3.10/site-packages/transformers/trainer.py", line 2768, in training_step
    loss = self.compute_loss(model, inputs)
  File "/opt/pyenv/versions/cellphones-in-the-wild/lib/python3.10/site-packages/transformers/trainer.py", line 2791, in compute_loss
    outputs = model(**inputs)
  File "/opt/pyenv/versions/cellphones-in-the-wild/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/opt/pyenv/versions/cellphones-in-the-wild/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/pyenv/versions/cellphones-in-the-wild/lib/python3.10/site-packages/torch/nn/parallel/data_parallel.py", line 185, in forward
    outputs = self.parallel_apply(replicas, inputs, module_kwargs)
  File "/opt/pyenv/versions/cellphones-in-the-wild/lib/python3.10/site-packages/torch/nn/parallel/data_parallel.py", line 200, in parallel_apply
    return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
  File "/opt/pyenv/versions/cellphones-in-the-wild/lib/python3.10/site-packages/torch/nn/parallel/parallel_apply.py", line 110, in parallel_apply
    output.reraise()
  File "/opt/pyenv/versions/cellphones-in-the-wild/lib/python3.10/site-packages/torch/_utils.py", line 694, in reraise
    raise exception
IndexError: Caught IndexError in replica 0 on device 0.
Original Traceback (most recent call last):
  File "/opt/pyenv/versions/cellphones-in-the-wild/lib/python3.10/site-packages/torch/nn/parallel/parallel_apply.py", line 85, in _worker
    output = module(*input, **kwargs)
  File "/opt/pyenv/versions/cellphones-in-the-wild/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/opt/pyenv/versions/cellphones-in-the-wild/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/pyenv/versions/cellphones-in-the-wild/lib/python3.10/site-packages/transformers/models/detr/modeling_detr.py", line 1603, in forward
    loss_dict = criterion(outputs_loss, labels)
  File "/opt/pyenv/versions/cellphones-in-the-wild/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/opt/pyenv/versions/cellphones-in-the-wild/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/pyenv/versions/cellphones-in-the-wild/lib/python3.10/site-packages/transformers/models/detr/modeling_detr.py", line 2202, in forward
    indices = self.matcher(outputs_without_aux, targets)
  File "/opt/pyenv/versions/cellphones-in-the-wild/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/opt/pyenv/versions/cellphones-in-the-wild/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/pyenv/versions/cellphones-in-the-wild/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/opt/pyenv/versions/cellphones-in-the-wild/lib/python3.10/site-packages/transformers/models/detr/modeling_detr.py", line 2330, in forward
    indices = [linear_sum_assignment(c[i]) for i, c in enumerate(cost_matrix.split(sizes, -1))]
  File "/opt/pyenv/versions/cellphones-in-the-wild/lib/python3.10/site-packages/transformers/models/detr/modeling_detr.py", line 2330, in <listcomp>
    indices = [linear_sum_assignment(c[i]) for i, c in enumerate(cost_matrix.split(sizes, -1))]
IndexError: index 8 is out of bounds for dimension 0 with size 8
```

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1. Follow this tutorial: https://huggingface.co/docs/transformers/tasks/object_detection


### Expected behavior

I expect the model to train.