Repository: transformers
Issue: DPT normalization causes contouring when there are significant disparities in depth values between adjacent areas
Labels: Good Second Issue

### System Info

Python 3.10.12
transformers-4.36.2

### Who can help?

@stevhliu @NielsRogge

### Information

- [X] The official example scripts
- [x] My own modified scripts

### Tasks

- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

```
from transformers import DPTImageProcessor, DPTForDepthEstimation
import torch
import numpy as np
from PIL import Image
import requests

url = "https://images.unsplash.com/photo-1605146768851-eda79da39897?q=80&w=2970&auto=format&fit=crop&ixlib=rb-4.0.3&ixid=M3wxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8fA%3D%3D"
image = Image.open(requests.get(url, stream=True).raw)

processor = DPTImageProcessor.from_pretrained("Intel/dpt-large")
model = DPTForDepthEstimation.from_pretrained("Intel/dpt-large")

# prepare image for the model
inputs = processor(images=image, return_tensors="pt")

with torch.no_grad():
    outputs = model(**inputs)
    predicted_depth = outputs.predicted_depth

# interpolate to original size
prediction = torch.nn.functional.interpolate(
    predicted_depth.unsqueeze(1),
    size=image.size[::-1],
    mode="bicubic",
    align_corners=False,
)

# visualize the prediction
output = prediction.squeeze().cpu().numpy()
formatted = (output * 255 / np.max(output)).astype("uint8")
depth = Image.fromarray(formatted)
display(depth)
```

>
Some weights of DPTForDepthEstimation were not initialized from the model checkpoint at Intel/dpt-large and are newly initialized: ['neck.fusion_stage.layers.0.residual_layer1.convolution2.bias', 'neck.fusion_stage.layers.0.residual_layer1.convolution1.weight', 'neck.fusion_stage.layers.0.residual_layer1.convolution2.weight', 'neck.fusion_stage.layers.0.residual_layer1.convolution1.bias']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.

![edge_effects_depth (1)](https://github.com/huggingface/transformers/assets/90732384/054a7e81-9611-4418-9706-29ada47c64a1)

### Expected behavior

Anecdotally, the local scaling methodology used by get_depth_map at https://huggingface.co/diffusers/controlnet-depth-sdxl-1.0 seems to work better for models that perform better at identifying close-range depth. The global scaling methodology seems to work better for models that perform better at identifying far-range depth. I combined them below:

```
def get_depth_map(image, feature_extractor, depth_estimator, scale_local):
    inputs = feature_extractor(images=image, return_tensors="pt").pixel_values.to("cuda")
    with torch.no_grad(), torch.autocast("cuda"):
        depth_map = depth_estimator(inputs).predicted_depth

    depth_map = torch.nn.functional.interpolate(
        depth_map.unsqueeze(1),
        size=image.size[::-1],
        mode="bicubic",
        align_corners=False,
    )

    if scale_local:
        depth_min = torch.amin(depth_map, dim=[1, 2, 3], keepdim=True)
        depth_max = torch.amax(depth_map, dim=[1, 2, 3], keepdim=True)
        depth_map = (depth_map - depth_min) / (depth_max - depth_min)
        image = torch.cat([depth_map] * 3, dim=1)
    
        image = image.permute(0, 2, 3, 1).cpu().numpy()[0]
        image = Image.fromarray((image * 255.0).clip(0, 255).astype(np.uint8))
        return image
    
    output = depth_map.squeeze().cpu().numpy()
    formatted = (output * 255 / np.max(output)).astype("uint8")
    return Image.fromarray(formatted)


depth_estimator_hybrid = DPTForDepthEstimation.from_pretrained("Intel/dpt-hybrid-midas").to("cuda")
depth_estimator_dinov2_nyu = DPTForDepthEstimation.from_pretrained("facebook/dpt-dinov2-giant-nyu").to("cuda")
    
image_processor_hybrid = AutoImageProcessor.from_pretrained("Intel/dpt-hybrid-midas")
image_processor_dinov2_nyu = AutoImageProcessor.from_pretrained("facebook/dpt-dinov2-giant-nyu")

# Close range depth
bad_close_result = get_depth_map(image, image_processor_hybrid, depth_estimator_hybrid, False)
good_close_result = get_depth_map(image, image_processor_hybrid, depth_estimator_hybrid, True)

# Far range depth
downscaled_image = image.resize((1024, 1024))  # This image is too big for my GPU to processdpt-dinov2-giant-nyu so I downscaled it
good_far_result = get_depth_map(downscaled_image, image_processor_dinov2_nyu, depth_estimator_dinov2_nyu, False)
bad_far_result = get_depth_map(downscaled_image, image_processor_dinov2_nyu, depth_estimator_dinov2_nyu, True)
```

`display(bad_close_result)`
![globally_scaled_depth_close](https://github.com/huggingface/transformers/assets/90732384/980dcd22-7e63-4842-b330-637025a9ce8d)

`display(good_close_result)`
![locally_scaled_depth_close](https://github.com/huggingface/transformers/assets/90732384/970081fc-418d-459e-9576-84316ef6362e)

`display(good_far_result)`
![globally_scaled_depth_far](https://github.com/huggingface/transformers/assets/90732384/9a265c67-51da-47a1-bd3c-4a989480eb97)
`display(bad_far_result)`
![locally_scaled_depth_far](https://github.com/huggingface/transformers/assets/90732384/8aa280aa-8166-45be-89a0-0c1463337a90)

Sufficiently blurring the image prior to detecting depth also gets rid of this, ie:

```
blurred_image = image.filter(ImageFilter.GaussianBlur(radius=5))
display(get_depth_map(blurred_image, image_processor_hybrid, depth_estimator_hybrid, False))
```

![blurred_depth](https://github.com/huggingface/transformers/assets/90732384/0f447c9b-0ad3-44f0-9c0e-4813be913899)