Repository: opencv
Issue: cv::VideoWriter does not support recording with variable frame rate
Labels: feature,category: videoio,RFC

### System Information

OpenCV 4.7.0
Arch Linux
gcc version 12.2.1 20230201

### Detailed description

FFMPEG backend of cv::VideoWriter class uses incorrect algorithm to compute pts of output frames.

In CvVideoWriter_FFMPEG::writeFrame:
https://github.com/opencv/opencv/blob/725e440d278aca07d35a5e8963ef990572b07316/modules/videoio/src/cap_ffmpeg_impl.hpp#L2462-L2474

Here I see `hw_frame->pts = frame_idx` (or `picture->pts = frame_idx`) and then `frame_idx++`. This only works for sources with constant fps so that output frames will always have constant delta no matter how fast input device produces them. My USB webcam produces frames with variable fps (it decreases fps in order to increase exposure time when insufficient lighting), but cv::VideoWriter doesn't know this because cv::VideoCapture erase information about original pts values. Because of this, the generated file plays too fast.

The more correct way to calculate output pts is "original pts" minus "stream start_time" like in ffmpeg.c program (in `adjust_frame_pts_to_encoder_tb` and `do_audio_out` functions):
https://github.com/FFmpeg/FFmpeg/blob/ea3d24bbe3c58b171e55fe2151fc7ffaca3ab3d2/fftools/ffmpeg.c#L709-L710

But as I said cv::VideoWriter doesn't know this information. Alternatively it can be set to "frame grab time" minus "file creation time" from system clock like in motion program:
https://github.com/Motion-Project/motion/blob/09afbfdd6ab5cde58286e93561b862cdb9509820/src/ffmpeg.c#L457

I've found similar questions on stackoverflow and forum.opencv.org but they don't have any definitive answer. (https://stackoverflow.com/questions/17272058/writing-variable-framerate-videos-in-opencv, https://forum.opencv.org/t/opencv-usb-camera-video-writer-fast-playback-maybe-stuck-in-yuv/8676)

Is there any plans of changing this pts generation method to something more versatile?

### Steps to reproduce

main.cpp:

<details>
  <summary>Spoiler warning</summary>
  
```c++
#include <chrono>
#include <iostream>
#include <sstream>

#include <opencv2/core/types.hpp>
#include <opencv2/highgui.hpp>
#include <opencv2/imgproc.hpp>
#include <opencv2/videoio.hpp>

int main() {
  std::cout << cv::getBuildInformation() << std::endl;

  cv::VideoCapture cap;
  cap.open("/dev/video0", cv::CAP_V4L2);
  if (!cap.isOpened()) {
    std::cerr << "ERROR! Unable to open camera\n";
    return -1;
  }
  if (!cap.set(cv::CAP_PROP_FOURCC,
               cv::VideoWriter::fourcc('M', 'J', 'P', 'G')))
    std::cerr << "Failed to set fourcc" << std::endl;
  if (!cap.set(cv::CAP_PROP_FRAME_WIDTH, 1280))
    std::cerr << "Failed to set width" << std::endl;
  if (!cap.set(cv::CAP_PROP_FRAME_HEIGHT, 720))
    std::cerr << "Failed to set height" << std::endl;
  if (!cap.set(cv::CAP_PROP_FPS, 30))
    std::cerr << "Failed to set FPS" << std::endl;

  cv::VideoWriter writer;
  writer.open("out.mp4", cv::VideoWriter::fourcc('h', '2', '6', '4'), 30,
              cv::Size(1280, 720));
  if (!writer.isOpened()) {
    std::cerr << "ERROR! Unable to open writer\n";
    return -1;
  }

  std::clog << "backend = " << writer.getBackendName() << std::endl;

  auto t1 = std::chrono::system_clock::now();
  const auto start = t1;
  while (true) {
    cv::Mat frame;
    const bool ok = cap.read(frame);
    if (!ok || frame.empty())
      break;
    const auto t2 = std::chrono::system_clock::now();
    const auto duration = std::chrono::duration<double>(t2 - t1).count();
    std::clog << "duration = " << duration << " s" << std::endl;
    t1 = t2;

    std::ostringstream os;
    os << std::chrono::duration<double>(t2 - start).count() << " " << duration;
    cv::putText(frame, os.str(), cv::Point(20, 50), cv::FONT_HERSHEY_SIMPLEX,
                1.0, cv::Scalar(0, 0, 255), 2);
    cv::imshow("Frame", frame);
    if (cv::waitKey(1) >= 0)
      break;

    writer.write(frame);
  }
}

  ```
  
</details>

CMakeLists.txt:

```cmake
cmake_minimum_required(VERSION 3.5)

project(a CXX)

set(CMAKE_CXX_STANDARD 17)
set(CMAKE_CXX_STANDARD_REQUIRED ON)
set(CMAKE_CXX_EXTENSIONS OFF)
set(CMAKE_EXPORT_COMPILE_COMMANDS ON)

find_package(OpenCV REQUIRED)
add_executable(${PROJECT_NAME} main.cpp)
target_include_directories(${PROJECT_NAME} PUBLIC ${OpenCV_INCLUDE_DIRS})
target_link_libraries(${PROJECT_NAME} PUBLIC ${OpenCV_LIBS})
```

```bash
$ mkdir build && cd build
$ cmake -GNinja .. -DCMAKE_BUILD_TYPE=Debug
$ cmake --build .
```

Output:

<details>
  <summary>Spoiler warning</summary>
    
```bash
General configuration for OpenCV 4.7.0 =====================================
  Version control:               unknown

  Extra modules:
    Location (extra):            /build/opencv/src/opencv_contrib-4.7.0/modules
    Version control (extra):     unknown

  Platform:
    Timestamp:                   2023-03-06T15:04:10Z
    Host:                        Linux 6.1.12-arch1-1 x86_64
    CMake:                       3.25.2
    CMake generator:             Unix Makefiles
    CMake build tool:            /usr/bin/make
    Configuration:               Release

  CPU/HW features:
    Baseline:                    SSE SSE2
      requested:                 SSE3
      required:                  SSE2
      disabled:                  SSE3
    Dispatched code generation:  SSE4_1 SSE4_2 FP16 AVX AVX2 AVX512_SKX
      requested:                 SSE4_1 SSE4_2 AVX FP16 AVX2 AVX512_SKX
      SSE4_1 (16 files):         + SSE3 SSSE3 SSE4_1
      SSE4_2 (1 files):          + SSE3 SSSE3 SSE4_1 POPCNT SSE4_2
      FP16 (0 files):            + SSE3 SSSE3 SSE4_1 POPCNT SSE4_2 FP16 AVX
      AVX (4 files):             + SSE3 SSSE3 SSE4_1 POPCNT SSE4_2 AVX
      AVX2 (32 files):           + SSE3 SSSE3 SSE4_1 POPCNT SSE4_2 FP16 FMA3 AVX AVX2
      AVX512_SKX (5 files):      + SSE3 SSSE3 SSE4_1 POPCNT SSE4_2 FP16 FMA3 AVX AVX2 AVX_512F AVX512_COMMON AVX512_SKX

  C/C++:
    Built as dynamic libs?:      YES
    C++ standard:                11
    C++ Compiler:                /usr/bin/c++  (ver 12.2.1)
    C++ flags (Release):         -march=x86-64 -mtune=generic -O2 -pipe -fno-plt -fexceptions         -Wp,-D_FORTIFY_SOURCE=2 -Wformat -Werror=format-security         -fstack-clash-protection -fcf-protection -Wp,-D_GLIBCXX_ASSERTIONS -g -ffile-prefix-map=/build/opencv/src=/usr/src/debug/opencv -flto=auto   -fsigned-char -W -Wall -Wreturn-type -Wnon-virtual-dtor -Waddress -Wsequence-point -Wformat -Wformat-security -Wmissing-declarations -Wundef -Winit-self -Wpointer-arith -Wshadow -Wsign-promo -Wuninitialized -Wsuggest-override -Wno-delete-non-virtual-dtor -Wno-comment -Wimplicit-fallthrough=3 -Wno-strict-overflow -fdiagnostics-show-option -Wno-long-long -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections  -msse -msse2 -fvisibility=hidden -fvisibility-inlines-hidden -O3 -DNDEBUG  -DNDEBUG
    C++ flags (Debug):           -march=x86-64 -mtune=generic -O2 -pipe -fno-plt -fexceptions         -Wp,-D_FORTIFY_SOURCE=2 -Wformat -Werror=format-security         -fstack-clash-protection -fcf-protection -Wp,-D_GLIBCXX_ASSERTIONS -g -ffile-prefix-map=/build/opencv/src=/usr/src/debug/opencv -flto=auto   -fsigned-char -W -Wall -Wreturn-type -Wnon-virtual-dtor -Waddress -Wsequence-point -Wformat -Wformat-security -Wmissing-declarations -Wundef -Winit-self -Wpointer-arith -Wshadow -Wsign-promo -Wuninitialized -Wsuggest-override -Wno-delete-non-virtual-dtor -Wno-comment -Wimplicit-fallthrough=3 -Wno-strict-overflow -fdiagnostics-show-option -Wno-long-long -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections  -msse -msse2 -fvisibility=hidden -fvisibility-inlines-hidden -g  -DDEBUG -D_DEBUG
    C Compiler:                  /usr/bin/cc
    C flags (Release):           -march=x86-64 -mtune=generic -O2 -pipe -fno-plt -fexceptions         -Wp,-D_FORTIFY_SOURCE=2 -Wformat -Werror=format-security         -fstack-clash-protection -fcf-protection -g -ffile-prefix-map=/build/opencv/src=/usr/src/debug/opencv -flto=auto   -fsigned-char -W -Wall -Wreturn-type -Waddress -Wsequence-point -Wformat -Wformat-security -Wmissing-declarations -Wmissing-prototypes -Wstrict-prototypes -Wundef -Winit-self -Wpointer-arith -Wshadow -Wuninitialized -Wno-comment -Wimplicit-fallthrough=3 -Wno-strict-overflow -fdiagnostics-show-option -Wno-long-long -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections  -msse -msse2 -fvisibility=hidden -O3 -DNDEBUG  -DNDEBUG
    C flags (Debug):             -march=x86-64 -mtune=generic -O2 -pipe -fno-plt -fexceptions         -Wp,-D_FORTIFY_SOURCE=2 -Wformat -Werror=format-security         -fstack-clash-protection -fcf-protection -g -ffile-prefix-map=/build/opencv/src=/usr/src/debug/opencv -flto=auto   -fsigned-char -W -Wall -Wreturn-type -Waddress -Wsequence-point -Wformat -Wformat-security -Wmissing-declarations -Wmissing-prototypes -Wstrict-prototypes -Wundef -Winit-self -Wpointer-arith -Wshadow -Wuninitialized -Wno-comment -Wimplicit-fallthrough=3 -Wno-strict-overflow -fdiagnostics-show-option -Wno-long-long -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections  -msse -msse2 -fvisibility=hidden -g  -DDEBUG -D_DEBUG
    Linker flags (Release):      -Wl,--exclude-libs,libippicv.a -Wl,--exclude-libs,libippiw.a -Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now -flto=auto  -Wl,--gc-sections -Wl,--as-needed -Wl,--no-undefined  
    Linker flags (Debug):        -Wl,--exclude-libs,libippicv.a -Wl,--exclude-libs,libippiw.a -Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now -flto=auto  -Wl,--gc-sections -Wl,--as-needed -Wl,--no-undefined  
    ccache:                      NO
    Precompiled headers:         NO
    Extra dependencies:          dl m pthread rt
    3rdparty dependencies:

  OpenCV modules:
    To be built:                 alphamat aruco barcode bgsegm bioinspired calib3d ccalib core cvv datasets dnn dnn_objdetect dnn_superres dpm face features2d flann freetype fuzzy gapi hdf hfs highgui img_hash imgcodecs imgproc intensity_transform java line_descriptor mcc ml objdetect optflow phase_unwrapping photo plot python3 quality rapid reg rgbd saliency shape stereo stitching structured_light superres surface_matching text tracking video videoio videostab viz wechat_qrcode xfeatures2d ximgproc xobjdetect xphoto
    Disabled:                    world
    Disabled by dependency:      -
    Unavailable:                 cudaarithm cudabgsegm cudacodec cudafeatures2d cudafilters cudaimgproc cudalegacy cudaobjdetect cudaoptflow cudastereo cudawarping cudev julia matlab ovis python2 sfm ts
    Applications:                examples apps
    Documentation:               NO
    Non-free algorithms:         YES

  GUI:                           QT5
    QT:                          YES (ver 5.15.8 )
      QT OpenGL support:         YES (Qt5::OpenGL 5.15.8)
    GTK+:                        NO
    OpenGL support:              YES (/lib64/libGL.so /lib64/libGLU.so)
    VTK support:                 YES (ver 9.2.6)

  Media I/O: 
    ZLib:                        /lib64/libz.so (ver 1.2.13)
    JPEG:                        /lib64/libjpeg.so (ver 80)
    WEBP:                        /lib64/libwebp.so (ver encoder: 0x020f)
    PNG:                         /lib64/libpng.so (ver 1.6.39)
    TIFF:                        /lib64/libtiff.so (ver 42 / 4.5.0)
    JPEG 2000:                   OpenJPEG (ver 2.5.0)
    OpenEXR:                     OpenEXR::OpenEXR (ver 3.1.5)
    HDR:                         YES
    SUNRASTER:                   YES
    PXM:                         YES
    PFM:                         YES

  Video I/O:
    DC1394:                      YES (2.2.6)
    FFMPEG:                      YES
      avcodec:                   YES (60.3.100)
      avformat:                  YES (60.3.100)
      avutil:                    YES (58.2.100)
      swscale:                   YES (7.1.100)
      avresample:                NO
    GStreamer:                   YES (1.22.1)
    v4l/v4l2:                    YES (linux/videodev2.h)

  Parallel framework:            TBB (ver 2021.5 interface 12050)

  Trace:                         YES (with Intel ITT)

  Other third-party libraries:
    Intel IPP:                   2020.0.0 Gold [2020.0.0]
           at:                   /build/opencv/src/build/3rdparty/ippicv/ippicv_lnx/icv
    Intel IPP IW:                sources (2020.0.0)
              at:                /build/opencv/src/build/3rdparty/ippicv/ippicv_lnx/iw
    VA:                          YES
    Lapack:                      YES (/usr/lib/liblapack.so /usr/lib/libblas.so /usr/lib/libcblas.so)
    Eigen:                       YES (ver 3.4.0)
    Custom HAL:                  NO
    Protobuf:                    /lib64/libprotobuf.so (3.21.12)

  Vulkan:                        YES
    Include path:                /build/opencv/src/opencv-4.7.0/3rdparty/include
    Link libraries:              Dynamic load

  OpenCL:                        YES (INTELVA)
    Include path:                /build/opencv/src/opencv-4.7.0/3rdparty/include/opencl/1.2
    Link libraries:              Dynamic load

  Python 3:
    Interpreter:                 /usr/bin/python3 (ver 3.10.10)
    Libraries:                   /lib64/libpython3.10.so (ver 3.10.10)
    numpy:                       /usr/lib/python3.10/site-packages/numpy/core/include (ver 1.24.2)
    install path:                lib/python3.10/site-packages

  Python (for build):            /usr/bin/python3

  Java:                          
    ant:                         /bin/ant (ver 1.10.12)
    JNI:                         /usr/lib/jvm/default/include /usr/lib/jvm/default/include/linux /usr/lib/jvm/default/include
    Java wrappers:               YES
    Java tests:                  NO

  Install to:                    /usr
-----------------------------------------------------------------


OpenCV: FFMPEG: tag 0x34363268/'h264' is not supported with codec id 27 and format 'mp4 / MP4 (MPEG-4 Part 14)'
OpenCV: FFMPEG: fallback to use tag 0x31637661/'avc1'
backend = FFMPEG
duration = 0.349812 s
duration = 0.474604 s
duration = 0.0482199 s
duration = 0.0159359 s
duration = 0.0169632 s
duration = 0.016698 s
duration = 0.0218174 s
duration = 0.0592223 s
duration = 0.0645943 s
duration = 0.0643299 s
duration = 0.058207 s
duration = 0.0598891 s
duration = 0.0598736 s
duration = 0.0600158 s
duration = 0.0601155 s
duration = 0.0592842 s
duration = 0.0574868 s
duration = 0.0601092 s
duration = 0.0601084 s
duration = 0.0599635 s
duration = 0.0599895 s
duration = 0.0602683 s
duration = 0.0597205 s
duration = 0.0598167 s
duration = 0.0639994 s
duration = 0.0562123 s
duration = 0.059916 s
duration = 0.0602109 s
duration = 0.0599823 s
duration = 0.0641135 s
duration = 0.0600654 s
duration = 0.0594222 s
duration = 0.0602603 s
duration = 0.0600267 s
duration = 0.0599065 s
duration = 0.0594204 s
duration = 0.0583724 s
duration = 0.059966 s
duration = 0.0595144 s
duration = 0.0602499 s
duration = 0.0624571 s
duration = 0.0601156 s
duration = 0.0599125 s
duration = 0.0599919 s
duration = 0.0999594 s
duration = 0.101943 s
duration = 0.100334 s
duration = 0.0997533 s
duration = 0.100789 s
duration = 0.098958 s
duration = 0.100219 s
duration = 0.102174 s
duration = 0.0984294 s
duration = 0.103088 s
duration = 0.10048 s
duration = 0.0993025 s
duration = 0.0985663 s
duration = 0.101554 s
duration = 0.0989961 s
duration = 0.101855 s
duration = 0.0986272 s
duration = 0.100393 s
duration = 0.0974455 s
duration = 0.102552 s
duration = 0.09719 s
duration = 0.101947 s
duration = 0.100875 s
duration = 0.0974297 s
duration = 0.102765 s
duration = 0.0998138 s
duration = 0.0998938 s
duration = 0.100771 s
duration = 0.0962844 s
```
  
</details>

Below there is an example of generated file. The first number is the interval from file creation time point using std::chrono::system_clock. Second - is the interval between frames (in seconds). As you can see, 21 seconds have passed since the start of recording, but the file is only 8 seconds long.

https://user-images.githubusercontent.com/36479428/227533794-63093114-74e3-400d-b677-2b39a1a21b0b.mp4



### Issue submission checklist

- [x] I report the issue, it's not a question
- [X] I checked the problem with documentation, FAQ, open issues, forum.opencv.org, Stack Overflow, etc and have not found any solution
- [X] I updated to the latest OpenCV version and the issue is still there
- [X] There is reproducer code and related data files (videos, images, onnx, etc)