Repository: opencv
Issue: VideoCapture FFmpeg backend color shift when reading H.264 video with BT.709 color standard
Labels: feature,category: videoio,Hackathon

##### System information (version)
- OpenCV => 4.53
- Operating System / Platform => Windows 64 Bit
- Compiler => Visual Studio 2017

##### Detailed description

There is an issue with OpenCV VideoCapture when reading video frames using FFmpeg backend.  
The issue results a "color shift" when reading H.264 video stream that marked as [BT.709](https://en.wikipedia.org/wiki/Rec._709) color standard.  

The problem is reproduced (at least) under Windows 10 in C++ and Python interfaces and under Ubuntu 18.04 with Python interface.  

The issue is described in my answer in following post: [OpenCV read / write video color difference](https://stackoverflow.com/questions/68629271/opencv-read-write-video-color-difference).  

It looks like OpenCV video frame capturing ignores the color characteristics of the H.264 video stream, and applies YUV to RGB color conversion using BT.601 color conversion formula.  
Color description in a stream is described in H.264 section E.2.1 and tables E-3, E-4 and E-5.

##### Steps to reproduce

**Building synthetic video pattern** for reproducing the problem:
Use FFmpeg command line tool for creating a synthetic video to be used as input.
The following command generates an MP4 video file with H.264 codec, and BT.709 color standard:

`ffmpeg -y -f lavfi -src_range 1 -color_primaries bt709 -color_trc bt709 -colorspace bt709 -i testsrc=size=192x108:rate=1:duration=5 -vcodec libx264 -crf 17 -pix_fmt yuv444p -dst_range 1 -color_primaries bt709 -color_trc bt709 -colorspace bt709 -bsf:v h264_metadata=video_full_range_flag=1:colour_primaries=1:transfer_characteristics=1:matrix_coefficients=1 bt709_full_range.mp4`

- The above command uses yuv444p pixel format (instead of yuv420p) for getting more pure colors.
- The arguments -bsf:v h264_metadata=video_full_range_flag=1:colour_primaries=1:transfer_characteristics=1:matrix_coefficients=1 use Bitstream Filter for marking the H.264 stream as "full range" BT.709.

Using [MediaInfo](https://mediaarea.net/en/MediaInfo) tool, we can view the following color characteristics:

`colour_range:             Full 
colour_primaries:         BT.709 
transfer_characteristics: BT.709 
matrix_coefficients:      BT.709 
`

**Capturing the video using OpenCV:**  

The following C++ code grabs the first frame, and save it to `1.png` image file:

#include "opencv2/opencv.hpp"

```c++
void main()
{   
    cv::VideoCapture cap("bt709_full_range.mp4");

    cv::Mat frame;    
    cap >> frame;

    cv::imwrite("1.png", frame);

    cap.release();    
}
```

Converting bt709_full_range.mp4 into images sequence using FFmpeg:

`ffmpeg -i bt709_full_range.mp4 -pix_fmt rgb24 %03d.png`

The file name of the first "extracted" frame is 001.png.

 - The value of the red color pixels of OpenCV are RGB = [232, 0, 3].
 - The value of the red color pixels of FFmpeg are RGB = [254, 0, 0].
The original RGB value is probably [255, 0, 0] (value is 254 due to colors conversion).

Comparing the results:

 - The left side is 1.png (result of OpenCV)
 - The right side is 001.png (result of FFmpeg command line tool)

![1](https://user-images.githubusercontent.com/20123928/128600632-c8f20cb7-ecea-4ffb-bb56-470444825a3e.png)  ![2](https://user-images.githubusercontent.com/20123928/128600644-52eaf82a-94e7-4155-87d9-47a2b1c31875.png)

Using GStreamer backend (instead of FFmpeg) backend seems to solve the "color shifting" problem.