Repository: transformers
Issue: SAM mask-generation - crops_n_layers
Labels: bug

### System Info

If I increase in a mask-generation pipeline the "crops_n_layers" to > 0. I run into issues with the batch size:

```python
from PIL import Image
from transformers import pipeline

relative_image_path = "data\image.png"
raw_image = Image.open(relative_image_path)

generator = pipeline("mask-generation", model="facebook/sam-vit-base", device=0)
outputs = generator(raw_image, points_per_batch=64, crops_n_layers=1)
``` 

Error message:
```
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
File c:\Users\LabUser\Documents\repos\test-classification\main.py:4
      [1](file:///C:/Users/LabUser/Documents/repos/test-classification/main.py:1) # %%
      [3](file:///C:/Users/LabUser/Documents/repos/test-classification/main.py:3) generator = pipeline("mask-generation", model="facebook/sam-vit-huge", device=0)
----> [4](file:///C:/Users/LabUser/Documents/repos/test-classification/main.py:4) outputs = generator([raw_image], points_per_batch=64, crops_n_layers=1)

File c:\Users\LabUser\Documents\repos\test-classification\env\lib\site-packages\transformers\pipelines\mask_generation.py:166, in MaskGenerationPipeline.__call__(self, image, num_workers, batch_size, *args, **kwargs)
    [128](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/pipelines/mask_generation.py:128) def __call__(self, image, *args, num_workers=None, batch_size=None, **kwargs):
    [129](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/pipelines/mask_generation.py:129)     """
    [130](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/pipelines/mask_generation.py:130)     Generates binary segmentation masks
    [131](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/pipelines/mask_generation.py:131) 
   (...)
    [164](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/pipelines/mask_generation.py:164) 
    [165](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/pipelines/mask_generation.py:165)     """
--> [166](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/pipelines/mask_generation.py:166)     return super().__call__(image, *args, num_workers=num_workers, batch_size=batch_size, **kwargs)

File c:\Users\LabUser\Documents\repos\test-classification\env\lib\site-packages\transformers\pipelines\base.py:1282, in Pipeline.__call__(self, inputs, num_workers, batch_size, *args, **kwargs)
   [1278](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/pipelines/base.py:1278) if can_use_iterator:
   [1279](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/pipelines/base.py:1279)     final_iterator = self.get_iterator(
   [1280](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/pipelines/base.py:1280)         inputs, num_workers, batch_size, preprocess_params, forward_params, postprocess_params
   [1281](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/pipelines/base.py:1281)     )
-> [1282](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/pipelines/base.py:1282)     outputs = list(final_iterator)
   [1283](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/pipelines/base.py:1283)     return outputs
   [1284](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/pipelines/base.py:1284) else:

File c:\Users\LabUser\Documents\repos\test-classification\env\lib\site-packages\transformers\pipelines\pt_utils.py:124, in PipelineIterator.__next__(self)
    [121](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/pipelines/pt_utils.py:121)     return self.loader_batch_item()
    [123](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/pipelines/pt_utils.py:123) # We're out of items within a batch
--> [124](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/pipelines/pt_utils.py:124) item = next(self.iterator)
    [125](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/pipelines/pt_utils.py:125) processed = self.infer(item, **self.params)
    [126](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/pipelines/pt_utils.py:126) # We now have a batch of "inferred things".

File c:\Users\LabUser\Documents\repos\test-classification\env\lib\site-packages\transformers\pipelines\pt_utils.py:269, in PipelinePackIterator.__next__(self)
    [266](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/pipelines/pt_utils.py:266)             return accumulator
    [268](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/pipelines/pt_utils.py:268) while not is_last:
--> [269](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/pipelines/pt_utils.py:269)     processed = self.infer(next(self.iterator), **self.params)
    [270](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/pipelines/pt_utils.py:270)     if self.loader_batch_size is not None:
    [271](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/pipelines/pt_utils.py:271)         if isinstance(processed, torch.Tensor):

File c:\Users\LabUser\Documents\repos\test-classification\env\lib\site-packages\transformers\pipelines\base.py:1208, in Pipeline.forward(self, model_inputs, **forward_params)
   [1206](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/pipelines/base.py:1206)     with inference_context():
   [1207](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/pipelines/base.py:1207)         model_inputs = self._ensure_tensor_on_device(model_inputs, device=self.device)
-> [1208](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/pipelines/base.py:1208)         model_outputs = self._forward(model_inputs, **forward_params)
   [1209](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/pipelines/base.py:1209)         model_outputs = self._ensure_tensor_on_device(model_outputs, device=torch.device("cpu"))
   [1210](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/pipelines/base.py:1210) else:

File c:\Users\LabUser\Documents\repos\test-classification\env\lib\site-packages\transformers\pipelines\mask_generation.py:233, in MaskGenerationPipeline._forward(self, model_inputs, pred_iou_thresh, stability_score_thresh, mask_threshold, stability_score_offset)
    [230](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/pipelines/mask_generation.py:230) original_sizes = model_inputs.pop("original_sizes").tolist()
    [231](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/pipelines/mask_generation.py:231) reshaped_input_sizes = model_inputs.pop("reshaped_input_sizes").tolist()
--> [233](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/pipelines/mask_generation.py:233) model_outputs = self.model(**model_inputs)
    [235](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/pipelines/mask_generation.py:235) # post processing happens here in order to avoid CPU GPU copies of ALL the masks
    [236](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/pipelines/mask_generation.py:236) low_resolution_masks = model_outputs["pred_masks"]

File c:\Users\LabUser\Documents\repos\test-classification\env\lib\site-packages\torch\nn\modules\module.py:1736, in Module._wrapped_call_impl(self, *args, **kwargs)
   [1734](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/torch/nn/modules/module.py:1734)     return self._compiled_call_impl(*args, **kwargs)  # type: ignore[misc]
   [1735](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/torch/nn/modules/module.py:1735) else:
-> [1736](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/torch/nn/modules/module.py:1736)     return self._call_impl(*args, **kwargs)

File c:\Users\LabUser\Documents\repos\test-classification\env\lib\site-packages\torch\nn\modules\module.py:1747, in Module._call_impl(self, *args, **kwargs)
   [1742](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/torch/nn/modules/module.py:1742) # If we don't have any hooks, we want to skip the rest of the logic in
   [1743](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/torch/nn/modules/module.py:1743) # this function, and just call forward.
   [1744](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/torch/nn/modules/module.py:1744) if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks
   [1745](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/torch/nn/modules/module.py:1745)         or _global_backward_pre_hooks or _global_backward_hooks
   [1746](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/torch/nn/modules/module.py:1746)         or _global_forward_hooks or _global_forward_pre_hooks):
-> [1747](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/torch/nn/modules/module.py:1747)     return forward_call(*args, **kwargs)
   [1749](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/torch/nn/modules/module.py:1749) result = None
   [1750](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/torch/nn/modules/module.py:1750) called_always_called_hooks = set()

File c:\Users\LabUser\Documents\repos\test-classification\env\lib\site-packages\transformers\models\sam\modeling_sam.py:1371, in SamModel.forward(self, pixel_values, input_points, input_labels, input_boxes, input_masks, image_embeddings, multimask_output, attention_similarity, target_embedding, output_attentions, output_hidden_states, return_dict, **kwargs)
   [1368](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/models/sam/modeling_sam.py:1368)     input_labels = torch.ones_like(input_points[:, :, :, 0], dtype=torch.int, device=input_points.device)
   [1370](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/models/sam/modeling_sam.py:1370) if input_points is not None and image_embeddings.shape[0] != input_points.shape[0]:
-> [1371](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/models/sam/modeling_sam.py:1371)     raise ValueError(
   [1372](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/models/sam/modeling_sam.py:1372)         "The batch size of the image embeddings and the input points must be the same. ",
   [1373](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/models/sam/modeling_sam.py:1373)         "Got {} and {} respectively.".format(image_embeddings.shape[0], input_points.shape[0]),
   [1374](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/models/sam/modeling_sam.py:1374)         " if you want to pass multiple points for the same image, make sure that you passed ",
   [1375](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/models/sam/modeling_sam.py:1375)         " input_points of shape (batch_size, point_batch_size, num_points_per_image, 3) and ",
   [1376](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/models/sam/modeling_sam.py:1376)         " input_labels of shape (batch_size, point_batch_size, num_points_per_image)",
   [1377](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/models/sam/modeling_sam.py:1377)     )
   [1379](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/models/sam/modeling_sam.py:1379) sparse_embeddings, dense_embeddings = self.prompt_encoder(
   [1380](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/models/sam/modeling_sam.py:1380)     input_points=input_points,
   [1381](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/models/sam/modeling_sam.py:1381)     input_labels=input_labels,
   [1382](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/models/sam/modeling_sam.py:1382)     input_boxes=input_boxes,
   [1383](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/models/sam/modeling_sam.py:1383)     input_masks=input_masks,
   [1384](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/models/sam/modeling_sam.py:1384) )
   [1386](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/models/sam/modeling_sam.py:1386) low_res_masks, iou_predictions, mask_decoder_attentions = self.mask_decoder(
   [1387](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/models/sam/modeling_sam.py:1387)     image_embeddings=image_embeddings,
   [1388](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/models/sam/modeling_sam.py:1388)     image_positional_embeddings=image_positional_embeddings,
   (...)
   [1394](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/models/sam/modeling_sam.py:1394)     output_attentions=output_attentions,
   [1395](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/models/sam/modeling_sam.py:1395) )

ValueError: ('The batch size of the image embeddings and the input points must be the same. ', 'Got 5 and 1 respectively.', ' if you want to pass multiple points for the same image, make sure that you passed ', ' input_points of shape (batch_size, point_batch_size, num_points_per_image, 3) and ', ' input_labels of shape (batch_size, point_batch_size, num_points_per_image)')
```

@Rocketknight1 @amyeroberts @qubvel 

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Run:
```python
from PIL import Image
from transformers import pipeline

relative_image_path = "data\image.png"
raw_image = Image.open(relative_image_path)

generator = pipeline("mask-generation", model="facebook/sam-vit-base", device=0)
outputs = generator(raw_image, points_per_batch=64, crops_n_layers=1)
``` 

### Expected behavior

It should support this change in parameter