Repository: opencv
Issue: restructuring CUDA backend to use its own computational graph and manage memory
Labels: RFC,category: dnn

##### System information (version)
- OpenCV => 4.5.4

### Internal Computational Graph

The importer's graph must be lowered into a new computational graph that is specific to the CUDA backend. This would enable better code design, less coupling, and better performance with the fine-grained control it would offer.

**Benefits:**
- CUDA backend code can be removed from `dnn.cpp` (fusions)
- tensors are uncoupled from the `BackendWrapper` system
  -  #15996
- graph scheduling
  - https://github.com/AlexeyAB/darknet/issues/7444
  - https://pytorch.org/blog/accelerating-pytorch-with-cuda-graphs/
  - https://github.com/opencv/opencv/pull/17236
  - YOLOv3 has three output blobs and the CUDA backend starts D2H transfer as soon as the blob is available (with the rest of the inference progressing in parallel). The topological ordering used in the DNN module computes the biggest blob last. Reordering it to compute the biggest blob first will reduce inference time.
- no irrelevant constraints on optimizations
  - #16306
- lower granularity of nodes; D2H, H2H, NCHW-NHWC conversion nodes, etc
  - these nodes are required to explicitly construct a CUDA graph
  - cuDNN converts between tensor formats because the CUDA backend doesn't support NHWC and tensor core convolutions require NHWC

### Kernels must be classes instead of functions

All the kernels currently have a CPU wrapper that invokes the kernel. The wrapper in many cases performs calculations (mostly negligible) for reducing tensor ranks, precomputing constants, etc. to improve kernel performance. These results can be cached in a class, unlike a function.

**Benefits:**
- amortize the CPU cost of executing kernels
- configuration to be saved into files (perhaps required for autotuning in the future?)
- can be incorporated into the computational graph
- easy to implement runtime compilation and autotuning
- cache performance information to do runtime autotuning during forward passes

Kernel arguments are stored in constant memory which is as fast as register access when all the threads in the warp access the same item (otherwise, the reads are serialized). A lot of precomputed information can be packed in a `struct` and passed onto the kernel. This frees up registers and also improves performance.

##### Issue submission checklist

 - [x] I report the issue, it's not a question
   <!--
   OpenCV team works with forum.opencv.org, Stack Overflow and other communities
   to discuss problems. Tickets with question without real issue statement will be
   closed.
   -->
 - [x] I checked the problem with documentation, FAQ, open issues,
       forum.opencv.org, Stack Overflow, etc and have not found solution
   <!--
   Places to check:
   * OpenCV documentation: https://docs.opencv.org
   * FAQ page: https://github.com/opencv/opencv/wiki/FAQ
   * OpenCV forum: https://forum.opencv.org
   * OpenCV issue tracker: https://github.com/opencv/opencv/issues?q=is%3Aissue
   * Stack Overflow branch: https://stackoverflow.com/questions/tagged/opencv
   -->
 - [x] I updated to latest OpenCV version and the issue is still there
   <!--
   master branch for OpenCV 4.x and 3.4 branch for OpenCV 3.x releases.
   OpenCV team supports only latest release for each branch.
   The ticket is closed, if the problem is not reproduced with modern version.
   -->
 - [ ] There is reproducer code and related data files: videos, images, onnx, etc
   <!--
   The best reproducer -- test case for OpenCV that we can add to the library.
   Recommendations for media files and binary files:
   * Try to reproduce the issue with images and videos in opencv_extra repository
     to reduce attachment size
   * Use PNG for images, if you report some CV related bug, but not image reader
     issue
   * Attach the image as archive to the ticket, if you report some reader issue.
     Image hosting services compress images and it breaks the repro code.
   * Provide ONNX file for some public model or ONNX file with with random weights,
     if you report ONNX parsing or handling issue. Architecture details diagram
     from netron tool can be very useful too. See https://lutzroeder.github.io/netron/
   -->