Repository: opencv
Issue: DNN: Overflow Errors with large image sizes
Labels: bug,RFC,category: dnn,confirmed,effort: ∞

### System Information

OpenCV version: 4.9.0
Operating System / Platform: Windows 10
Compiler & compiler version: Microsoft Visual Studio Professional 2022 (64-bit) - Version 17.1.0


### Detailed description

When inferencing with a MATLAB generated semantic segmentation ONNX model there are several places in the code where type "int" overflows causing exceptions. The model is based off of deeplabv3plus.

### Steps to reproduce

The trained ONNX model is based off of this Mathworks link: [https://www.mathworks.com/help/vision/ref/deeplabv3pluslayers.html](https://www.mathworks.com/help/vision/ref/deeplabv3pluslayers.html). There is a small change to the example (due to project needs and obfuscated below) where we've changed the number of training inputs from two to thirty like:
```
classNames = [ "x1","x2", "x3", "x4", "x5", "x6", ...
    "x7","x8", "x9", "x10", "x11", "x12", ...
    "x13","x14", "x15", "x16", "x17", "x18", ...
    "x19","x20", "x21", "x22", "x23", "x24", ...
    "x25","x26", "x27", "x28", "x29", "x30"];
labelIDs   = [1:1:30];
```
When inferencing with certain size inputs the overflow will occur. As an example, you can use an image of size 4864x4864 to demonstrate the overflow. The generated ONNX file works correctly when used in MATLAB applications for inferencing.

There are two sections of code that I've tracked down that probably need addressed.

1. [https://github.com/opencv/opencv/blob/4.9.0/modules/dnn/src/net_impl.cpp](https://github.com/opencv/opencv/blob/4.9.0/modules/dnn/src/net_impl.cpp) Lines 1189-1193
```
        for (int i = 0; i < ints.size(); i++)
            CV_CheckGT(total(ints[i]), 0, "");

        for (int i = 0; i < os.size(); i++)
            CV_CheckGT(total(os[i]), 0, "");
```
should probably be changed to:
```
       for (int i = 0; i < ints.size(); i++)        
            CV_CheckGT((size_t)total(ints[i]), (size_t)0, "");
        

        for (int i = 0; i < os.size(); i++)
            CV_CheckGT((size_t)total(os[i]), (size_t)0, "");
```

Without this change I receive the following exception:
```
OpenCV(4.9.0) Error: Unspecified error (>  (expected: 'total(ints[i]) > 0'), where
>     'total(ints[i])' is -1455947776
> must be greater than
>     '0' is 0
) in void __cdecl cv::dnn::dnn4_v20231225::Net::Impl::getLayerShapesRecursively(int,class std::map<int,struct cv::dnn::dnn4_v20231225::detail::LayerShapes,struct std::less<int>,class std::allocator<struct std::pair<int const ,struct cv::dnn::dnn4_v20231225::detail::LayerShapes> > > &), file C:\opencv\opencv-4.9.0\modules\dnn\src\net_impl.cpp, line 1190
[ERROR:0@3.284] global net_impl.cpp:1202 cv::dnn::dnn4_v20231225::Net::Impl::getLayerShapesRecursively OPENCV/DNN: [Deconvolution]:(onnx_node!dec_upsample2): getMemoryShapes() post validation failed. inputs=1 outputs=1/1 blobs=2 inplace=0
[ERROR:0@3.284] global net_impl.cpp:1205 cv::dnn::dnn4_v20231225::Net::Impl::getLayerShapesRecursively     input[0] = [ 1 30 1216 1216 ]
[ERROR:0@3.285] global net_impl.cpp:1209 cv::dnn::dnn4_v20231225::Net::Impl::getLayerShapesRecursively     output[0] = [ 1 30 4864 4864 ]
[ERROR:0@3.285] global net_impl.cpp:1213 cv::dnn::dnn4_v20231225::Net::Impl::getLayerShapesRecursively     blobs[0] = CV_32FC1 [ 30 30 8 8 ]
[ERROR:0@3.285] global net_impl.cpp:1213 cv::dnn::dnn4_v20231225::Net::Impl::getLayerShapesRecursively     blobs[1] = CV_32FC1 [ 30 1 ]
[ERROR:0@3.286] global net_impl.cpp:1215 cv::dnn::dnn4_v20231225::Net::Impl::getLayerShapesRecursively Exception message: OpenCV(4.9.0) C:\opencv\opencv-4.9.0\modules\dnn\src\net_impl.cpp:1190: error: (-2:Unspecified error) in function 'void __cdecl cv::dnn::dnn4_v20231225::Net::Impl::getLayerShapesRecursively(int,class std::map<int,struct cv::dnn::dnn4_v20231225::detail::LayerShapes,struct std::less<int>,class std::allocator<struct std::pair<int const ,struct cv::dnn::dnn4_v20231225::detail::LayerShapes> > > &)'
>  (expected: 'total(ints[i]) > 0'), where
>     'total(ints[i])' is -1455947776
> must be greater than
>     '0' is 0

Exception: OpenCV(4.9.0) C:\opencv\opencv-4.9.0\modules\dnn\src\net_impl.cpp:1190: error: (-2:Unspecified error) in function 'void __cdecl cv::dnn::dnn4_v20231225::Net::Impl::getLayerShapesRecursively(int,class std::map<int,struct cv::dnn::dnn4_v20231225::detail::LayerShapes,struct std::less<int>,class std::allocator<struct std::pair<int const ,struct cv::dnn::dnn4_v20231225::detail::LayerShapes> > > &)'
>  (expected: 'total(ints[i]) > 0'), where
>     'total(ints[i])' is -1455947776
> must be greater than
>     '0' is 0
```
2. After making the change in 1) there is another error in [https://github.com/opencv/opencv/blob/4.9.0/modules/dnn/src/layers/convolution_layer.cpp](https://github.com/opencv/opencv/blob/4.9.0/modules/dnn/src/layers/convolution_layer.cpp) on Line 1841

I believe this:
```
    int offset = (c * kernel_h * kernel_w + h * kernel_w + w) * plane_size_col;
```
should be changed to:
```
    unsigned int offset = (c * kernel_h * kernel_w + h * kernel_w + w) * plane_size_col;
```
Without this change, the offset variable will overflow and produce negative index values and eventually fail on line 1859 in the code below:
```
     for (int w_col = w_col_start; w_col < w_col_end; ++w_col) {
                            val += data_col_[offset + h_col * coeff_h + w_col * coeff_w];
     }
```

If you don't want to use MATLAB to train the ONNX file yourself, I can provide it and a sample image. I can't upload it here because it exceeds the 25 MB limit. 



### Issue submission checklist

- [X] I report the issue, it's not a question
- [X] I checked the problem with documentation, FAQ, open issues, forum.opencv.org, Stack Overflow, etc and have not found any solution
- [X] I updated to the latest OpenCV version and the issue is still there
- [X] There is reproducer code and related data files (videos, images, onnx, etc)