vinthony commited on
Commit
6d4338e
1 Parent(s): f7ea283

Update constants.py

Browse files
Files changed (1) hide show
  1. constants.py +11 -14
constants.py CHANGED
@@ -4,36 +4,33 @@ TASK_INFO = [ 'Resolution', 'FPS', 'Open Source', 'Length', 'Speed', 'Motion', '
4
  TASK_INFO_v2 = ['Final Sum Score', 'Motion Quality', 'Text-Video Alignment', 'Visual Quality', 'Temporal Consistency', 'Resolution', 'FPS', 'Open Source', 'Length', 'Speed', 'Motion', 'Camera']
5
 
6
  AVG_INFO = ['Final Sum Score', 'Motion Quality', 'Text-Video Alignment', 'Visual Quality', 'Temporal Consistency']
7
- DATA_TITILE_TYPE = ["markdown", "number", "number", "number", "number", "number", "number", "number", "number", "number", "number", "number", "number", "number", "number"]
8
  CSV_DIR = "./file/result.csv"
9
 
10
- # COLUMN_NAMES = MODEL_INFO + TASK_INFO
11
  COLUMN_NAMES = MODEL_INFO + TASK_INFO_v2
12
 
13
- DATA_NUM = [3158, 1831, 4649, 978, 2447, 657, 97, 331, 85, 1740, 2077, 1192]
14
-
15
-
16
  LEADERBORAD_INTRODUCTION = """# EvalCrafter Leaderboard 🏆
17
 
18
  Welcome to the cutting-edge leaderboard for text-to-video generation, where we meticulously evaluate state-of-the-art generative models using our comprehensive framework, ensuring high-quality results that align with user opinions. Join us in this exciting journey towards excellence! 🛫
19
 
20
- More methods will be evalcrafted soon, stay tunned ❤️ Join our evaluation by sending an email 📧 (vinthony@gmail.com)! You may also read the [EvalCrafter paper](https://arxiv.org/abs/2310.11440) for more detailed information 🤗
21
  """
22
 
 
23
  TABLE_INTRODUCTION = """In the table below, we summarize each dimension performance of all the models. """
24
 
25
  LEADERBORAD_INFO = """
26
- The vision and language generative models have been overgrown in recent years. For video generation,
27
- various open-sourced models and public-available services are released for generating high-visual quality videos.
28
- However, these methods often use a few academic metrics, \eg, FVD or IS, to evaluate the performance. We argue that
29
- it is hard to judge the large conditional generative models from the simple metrics since these models are often trained
30
- on very large datasets with multi-aspect abilities. Thus, we propose a new framework and pipeline to exhaustively evaluate
31
  the performance of the generated videos. To achieve this, we first conduct a new prompt list for text-to-video generation
32
  by analyzing the real-world prompt list with the help of the large language model. Then, we evaluate the state-of-the-art video
33
  generative models on our carefully designed benchmarks, in terms of visual qualities, content qualities, motion qualities, and
34
- text-caption alignment with around 18 objective metrics. To obtain the final leaderboard of the models, we also fit a series of
35
- coefficients to align the objective metrics to the users' opinions. Based on the proposed opinion alignment method, our final score
36
- shows a higher correlation than simply averaging the metrics, showing the effectiveness of the proposed evaluation method.
37
  """
38
 
39
 
 
4
  TASK_INFO_v2 = ['Final Sum Score', 'Motion Quality', 'Text-Video Alignment', 'Visual Quality', 'Temporal Consistency', 'Resolution', 'FPS', 'Open Source', 'Length', 'Speed', 'Motion', 'Camera']
5
 
6
  AVG_INFO = ['Final Sum Score', 'Motion Quality', 'Text-Video Alignment', 'Visual Quality', 'Temporal Consistency']
7
+ DATA_TITILE_TYPE = ["markdown", "markdown", "number", "number", "number", "number", "number", "number", "number", "number", "number", "number", "number", "number", "number"]
8
  CSV_DIR = "./file/result.csv"
9
 
 
10
  COLUMN_NAMES = MODEL_INFO + TASK_INFO_v2
11
 
 
 
 
12
  LEADERBORAD_INTRODUCTION = """# EvalCrafter Leaderboard 🏆
13
 
14
  Welcome to the cutting-edge leaderboard for text-to-video generation, where we meticulously evaluate state-of-the-art generative models using our comprehensive framework, ensuring high-quality results that align with user opinions. Join us in this exciting journey towards excellence! 🛫
15
 
16
+ More methods will be evalcrafted soon, stay tunned ❤️ Join our evaluation by sending an email 📧 (vinthony@gmail.com)! You may also read the [Code](https://github.com/EvalCrafter/EvalCrafter), [Paper](https://arxiv.org/abs/2310.11440), and [Project page](https://evalcrafter.github.io/) for more detailed information 🤗
17
  """
18
 
19
+
20
  TABLE_INTRODUCTION = """In the table below, we summarize each dimension performance of all the models. """
21
 
22
  LEADERBORAD_INFO = """
23
+ The vision and language generative models have been overgrown in recent years. For video generation, various
24
+ open-sourced models and public-available services are released for generating high-visual quality videos. However,
25
+ these methods often use a few academic metrics, \eg, FVD or IS, to evaluate the performance. We argue that it is
26
+ hard to judge the large conditional generative models from the simple metrics since these models are often trained on
27
+ very large datasets with multi-aspect abilities. Thus, we propose a new framework and pipeline to exhaustively evaluate
28
  the performance of the generated videos. To achieve this, we first conduct a new prompt list for text-to-video generation
29
  by analyzing the real-world prompt list with the help of the large language model. Then, we evaluate the state-of-the-art video
30
  generative models on our carefully designed benchmarks, in terms of visual qualities, content qualities, motion qualities, and
31
+ text-caption alignment with around 17 objective metrics. To obtain the final leaderboard of the models, we also fit a series of
32
+ coefficients to align the objective metrics to the users' opinions. Based on the proposed opinion alignment method, our final
33
+ score shows a higher correlation than simply averaging the metrics, showing the effectiveness of the proposed evaluation method.
34
  """
35
 
36