docs/EVAL.md

Evaluate the generated videos quality

You can easily calculate the following video quality metrics, which supports the batch-wise process.

• CLIP-SCORE: It uses the pretrained CLIP model to measure the cosine similarity between two modalities.
• FVD: Frechét Video Distance
• SSIM: structural similarity index measure
• LPIPS: learned perceptual image patch similarity
• PSNR: peak-signal-to-noise ratio

Requirement

Environment

• install Pytorch (torch>=1.7.1)
• install CLIP

      pip install git+https://github.com/openai/CLIP.git
  

• install clip-cose from PyPi

      pip install clip-score
  

• Other package

      pip install lpips
      pip install scipy (scipy==1.7.3/1.9.3, if you use 1.11.3, **you will calculate a WRONG FVD VALUE!!!**)
      pip install numpy
      pip install pillow
      pip install torchvision>=0.8.2
      pip install ftfy
      pip install regex
      pip install tqdm
  

Pretrain model

• FVD Before you cacluate FVD, you should first download the FVD pre-trained model. You can manually download any of the following and put it into FVD folder.
  • i3d_torchscript.pt from here
  • i3d_pretrained_400.pt from here

Other Notices

1. Make sure the pixel value of videos should be in [0, 1].
2. We average SSIM when images have 3 channels, ssim is the only metric extremely sensitive to gray being compared to b/w.
3. Because the i3d model downsamples in the time dimension, frames_num should > 10 when calculating FVD, so FVD calculation begins from 10-th frame, like upper example.
4. For grayscale videos, we multiply to 3 channels
5. data input specifications for clip_score

   • Image Files:All images should be stored in a single directory. The image files can be in either .png or .jpg format.

   • Text Files: All text data should be contained in plain text files in a separate directory. These text files should have the extension .txt.

   Note: The number of files in the image directory should be exactly equal to the number of files in the text directory. Additionally, the files in the image directory and text directory should be paired by file name. For instance, if there is a cat.png in the image directory, there should be a corresponding cat.txt in the text directory.

   Directory Structure Example:

     ├── path/to/image
     │   ├── cat.png
     │   ├── dog.png
     │   └── bird.jpg
     └── path/to/text
         ├── cat.txt
   

  ├── dog.txt
  └── bird.txt

6. data input specifications for fvd, psnr, ssim, lpips

Directory Structure Example:

  ├── path/to/generated_image
  │   ├── cat.mp4
  │   ├── dog.mp4
  │   └── bird.mp4
  └── path/to/real_image
      ├── cat.mp4
      ├── dog.mp4
      └── bird.mp4

Usage

# you change the file path and need to set the frame_num, resolution etc...

# clip_score cross modality
cd opensora/eval
bash script/cal_clip_score.sh



# fvd 
cd opensora/eval
bash script/cal_fvd.sh

# psnr
cd opensora/eval
bash eval/script/cal_psnr.sh


# ssim
cd opensora/eval
bash eval/script/cal_ssim.sh


# lpips
cd opensora/eval
bash eval/script/cal_lpips.sh

Acknowledgement

The evaluation codebase refers to clip-score and common_metrics.