Repository: open-compass/VLMEvalKit
Issue: OmniDocBench 默认的是哪一个版本，实测数据与官方数据对不上
Root cause: Documentation Deficiency

Hi，请问OmniDocBench默认测试的是一个版本，我使用Qwen2.5-VL-3B-Instruct，测得最终得分0.36，如下所示：
`
Edit_dist:
------------  --------
ALL_page_avg  0.360609
------------  --------
`
而Qwen2.5-VL技术报告中，OmniDocBench的en/zh版本测试结果为0.409/0.543，不管哪个版本和ALL_page_avg =0.36都不大能对上。

<img width="1109" height="229" alt="Image" src="https://github.com/user-attachments/assets/76d32fe8-2aa2-4add-856e-109bed01873b" />

Looking forward to your reply, thank you！

您好，想问一下您是否找到原因了呢？我使用DeepSeek-OCR模型在OmniDocBench上评测，分值相较于官方低很多。您是否知道OmniDocBench的分值含义呢
self.img_root:/home/zq/A911/ascend-tools/vlm_data/data/images/OmniDocBench
Loading data: 981it [00:00, 1141.39it/s]
load_success:981,load_fail:0
Loading data: 981it [00:00, 2514.72it/s]
load_table_success:428,load_table_fail:0
Cannot find pred for 0 samples.
Time out for plain text match of docstructbench_llm-raw-scihub-o.O-s00395-011-0222-8.pdf_10.jpg, match_gt2pred_simple will be used.
WARNING:pylatexenc.latex2text:WARNING: Error in configuration: macro '\frac' failed its substitution!
Time out for plain text match of jiaocaineedrop_jiaocai_needrop_en_715.jpg, match_gt2pred_simple will be used.
Time out for plain text match of yanbaor2_yanbaoPPT_5946.jpg, match_gt2pred_simple will be used.
/home/ma-user/anaconda3/envs/py310_testpip/lib/python3.10/site-packages/vlmeval/dataset/OmniDocBench/metrics.py:300: FutureWarning: DataFrameGroupBy.apply operated on the grouping columns. This behavior is deprecated, and in a future version of pandas the grouping columns will be excluded from the operation. Either pass `include_groups=False` to exclude the groupings or explicitly select the grouping columns after groupby to silence this warning.
  up_total_avg = df.groupby("image_name").apply(lambda x: x['Edit_num'].sum() / x['upper_len'].sum()) # page level, sum of edits divided by sum of max(gt,pred) lengths for each sample
text_block
Edit_dist:
------------  --------
ALL_page_avg  0.327432
------------  --------
====================================================================================================
----Anno Attribute---------------
Edit_dist:
--------------------------------------  --------
text_background: multi_colored          0.419213
text_background: single_colored         0.463535
text_background: white                  0.539955
text_language: text_en_ch_mixed         0.400074
text_language: text_english             0.430851
text_language: text_simplified_chinese  0.564498
text_rotate: horizontal                 0.891245
text_rotate: normal                     0.527424
text_rotate: rotate270                  0.996624
--------------------------------------  --------
====================================================================================================
sample_count:
--------------------------------------  -----
text_background: multi_colored            994
text_background: single_colored           600
text_background: white                  10040
text_language: text_en_ch_mixed           259
text_language: text_english              2944
text_language: text_simplified_chinese   8396
text_rotate: horizontal                    63
text_rotate: normal                     11400
text_rotate: rotate270                     30
--------------------------------------  -----
====================================================================================================
/home/ma-user/anaconda3/envs/py310_testpip/lib/python3.10/site-packages/vlmeval/dataset/OmniDocBench/metrics.py:111: FutureWarning: DataFrameGroupBy.apply operated on the grouping columns. This behavior is deprecated, and in a future version of pandas the grouping columns will be excluded from the operation. Either pass `include_groups=False` to exclude the groupings or explicitly select the grouping columns after groupby to silence this warning.
  up_total_avg = df.groupby(["image_name", "attribute"]).apply(lambda x: (x["score"]*x['upper_len']).sum() / x['upper_len'].sum()).groupby('attribute').mean()  # At page level, accumulate edits, denominator is sum of max(gt, pred) from each sample
Edit_dist:
--------------------------------  --------
ALL                               0.327432
None                              0.322465
colorful_backgroud                0.273653
data_source: PPT2PDF              0.262033
data_source: academic_literature  0.322449
data_source: book                 0.286129
data_source: colorful_textbook    0.276364
data_source: exam_paper           0.370504
data_source: magazine             0.288656
data_source: newspaper            0.532822
data_source: note                 0.363901
data_source: research_report      0.162019
fuzzy_scan                        0.307146
language: en_ch_mixed             0.332221
language: english                 0.292347
language: simplified_chinese      0.343675
layout: 1andmore_column           0.28034
layout: double_column             0.380913
layout: other_layout              0.452067
layout: single_column             0.262505
layout: three_column              0.348257
watermark                         0.319464
--------------------------------  --------
====================================================================================================
/home/ma-user/anaconda3/envs/py310_testpip/lib/python3.10/site-packages/vlmeval/dataset/OmniDocBench/metrics.py:300: FutureWarning: DataFrameGroupBy.apply operated on the grouping columns. This behavior is deprecated, and in a future version of pandas the grouping columns will be excluded from the operation. Either pass `include_groups=False` to exclude the groupings or explicitly select the grouping columns after groupby to silence this warning.
  up_total_avg = df.groupby("image_name").apply(lambda x: x['Edit_num'].sum() / x['upper_len'].sum()) # page level, sum of edits divided by sum of max(gt,pred) lengths for each sample
display_formula
Edit_dist:
------------  --------
ALL_page_avg  0.641538
------------  --------
====================================================================================================
----Anno Attribute---------------
Edit_dist:
-------------------------  --------
formula_type: handwriting  0.988333
formula_type: print        0.587018
-------------------------  --------
====================================================================================================
sample_count:
-------------------------  ---
formula_type: handwriting    4
formula_type: print        359
-------------------------  ---
====================================================================================================
/home/ma-user/anaconda3/envs/py310_testpip/lib/python3.10/site-packages/vlmeval/dataset/OmniDocBench/metrics.py:111: FutureWarning: DataFrameGroupBy.apply operated on the grouping columns. This behavior is deprecated, and in a future version of pandas the grouping columns will be excluded from the operation. Either pass `include_groups=False` to exclude the groupings or explicitly select the grouping columns after groupby to silence this warning.
  up_total_avg = df.groupby(["image_name", "attribute"]).apply(lambda x: (x["score"]*x['upper_len']).sum() / x['upper_len'].sum()).groupby('attribute').mean()  # At page level, accumulate edits, denominator is sum of max(gt, pred) from each sample
Edit_dist:
--------------------------------  --------
ALL                               0.641538
None                              0.636615
colorful_backgroud                0.668036
data_source: PPT2PDF              0.859259
data_source: academic_literature  0.587644
data_source: book                 0.548733
data_source: colorful_textbook    0.639781
data_source: exam_paper           0.656845
data_source: note                 1
fuzzy_scan                        0.70241
language: english                 0.504478
language: simplified_chinese      0.724605
layout: 1andmore_column           0.73331
layout: double_column             0.631132
layout: other_layout              0.710689
layout: single_column             0.591374
layout: three_column              0.520625
watermark                         0.635837
--------------------------------  --------
====================================================================================================
/home/ma-user/anaconda3/envs/py310_testpip/lib/python3.10/site-packages/vlmeval/dataset/OmniDocBench/metrics.py:300: FutureWarning: DataFrameGroupBy.apply operated on the grouping columns. This behavior is deprecated, and in a future version of pandas the grouping columns will be excluded from the operation. Either pass `include_groups=False` to exclude the groupings or explicitly select the grouping columns after groupby to silence this warning.
  up_total_avg = df.groupby("image_name").apply(lambda x: x['Edit_num'].sum() / x['upper_len'].sum()) # page level, sum of edits divided by sum of max(gt,pred) lengths for each sample
table
TEDS:
---  --------
all  0.479982
---  --------
====================================================================================================
TEDS_structure_only:
---  --------
all  0.520253
---  --------
====================================================================================================
Edit_dist:
------------  --------
ALL_page_avg  0.695405
------------  --------
====================================================================================================
----Anno Attribute---------------
Edit_dist:
----------------------------------  --------
include_background: False           0.704722
include_background: True            0.699145
include_equation: False             0.696304
include_equation: True              0.731005
include_photo: False                0.702872
language: table_en                  0.72194
language: table_en_ch_mixed         0.713077
language: table_simplified_chinese  0.69377
line: fewer_line                    0.684708
line: full_line                     0.720814
line: less_line                     0.693746
line: wireless_line                 0.671276
table_layout: horizontal            0.697931
table_layout: vertical              1
with_span: False                    0.702529
with_span: True                     0.703506
with_structured_text: False         0.706014
with_structured_text: True          0.616351
----------------------------------  --------
====================================================================================================
TEDS:
----------------------------------  --------
include_background: False           0.500174
include_background: True            0.439314
include_equation: False             0.492544
include_equation: True              0.426168
include_photo: False                0.479982
language: table_en                  0.475307
language: table_en_ch_mixed         0.52724
language: table_simplified_chinese  0.479595
line: fewer_line                    0.548602
line: full_line                     0.420799
line: less_line                     0.549747
line: wireless_line                 0.31713
table_layout: horizontal            0.487963
table_layout: vertical              0
with_span: False                    0.512647
with_span: True                     0.419443
with_structured_text: False         0.475811
with_structured_text: True          0.594822
----------------------------------  --------
====================================================================================================
TEDS_structure_only:
----------------------------------  --------
include_background: False           0.53943
include_background: True            0.481628
include_equation: False             0.526944
include_equation: True              0.491587
include_photo: False                0.520253
language: table_en                  0.516791
language: table_en_ch_mixed         0.552654
language: table_simplified_chinese  0.520103
line: fewer_line                    0.581726
line: full_line                     0.461688
line: less_line                     0.611525
line: wireless_line                 0.328135
table_layout: horizontal            0.528903
table_layout: vertical              0
with_span: False                    0.545351
with_span: True                     0.473737
with_structured_text: False         0.516497
with_structured_text: True          0.623675
----------------------------------  --------
====================================================================================================
sample_count:
----------------------------------  ---
include_background: False           286
include_background: True            142
include_equation: False             347
include_equation: True               81
include_photo: False                428
language: table_en                  128
language: table_en_ch_mixed          15
language: table_simplified_chinese  285
line: fewer_line                    147
line: full_line                     205
line: less_line                      62
line: wireless_line                  14
table_layout: horizontal            421
table_layout: vertical                7
with_span: False                    278
with_span: True                     150
with_structured_text: False         413
with_structured_text: True           15
----------------------------------  ---
====================================================================================================
/home/ma-user/anaconda3/envs/py310_testpip/lib/python3.10/site-packages/vlmeval/dataset/OmniDocBench/metrics.py:111: FutureWarning: DataFrameGroupBy.apply operated on the grouping columns. This behavior is deprecated, and in a future version of pandas the grouping columns will be excluded from the operation. Either pass `include_groups=False` to exclude the groupings or explicitly select the grouping columns after groupby to silence this warning.
  up_total_avg = df.groupby(["image_name", "attribute"]).apply(lambda x: (x["score"]*x['upper_len']).sum() / x['upper_len'].sum()).groupby('attribute').mean()  # At page level, accumulate edits, denominator is sum of max(gt, pred) from each sample
/home/ma-user/anaconda3/envs/py310_testpip/lib/python3.10/site-packages/vlmeval/dataset/OmniDocBench/metrics.py:117: FutureWarning: DataFrameGroupBy.apply operated on the grouping columns. This behavior is deprecated, and in a future version of pandas the grouping columns will be excluded from the operation. Either pass `include_groups=False` to exclude the groupings or explicitly select the grouping columns after groupby to silence this warning.
  page_avg = df.groupby(["image_name", "attribute"]).apply(lambda x: x["score"].mean()).groupby('attribute').mean()
/home/ma-user/anaconda3/envs/py310_testpip/lib/python3.10/site-packages/vlmeval/dataset/OmniDocBench/metrics.py:117: FutureWarning: DataFrameGroupBy.apply operated on the grouping columns. This behavior is deprecated, and in a future version of pandas the grouping columns will be excluded from the operation. Either pass `include_groups=False` to exclude the groupings or explicitly select the grouping columns after groupby to silence this warning.
  page_avg = df.groupby(["image_name", "attribute"]).apply(lambda x: x["score"].mean()).groupby('attribute').mean()
Edit_dist:
--------------------------------  --------
ALL                               0.695405
colorful_backgroud                0.755806
data_source: PPT2PDF              0.558462
data_source: academic_literature  0.703317
data_source: book                 0.695917
data_source: colorful_textbook    0.689004
data_source: exam_paper           0.763753
data_source: magazine             0.80849
data_source: newspaper            0.733429
data_source: note                 0.757532
data_source: research_report      0.635508
fuzzy_scan                        0.624045
language: en_ch_mixed             0.791152
language: english                 0.720799
language: simplified_chinese      0.677849
layout: 1andmore_column           0.682159
layout: double_column             0.743113
layout: other_layout              0.716459
layout: single_column             0.688306
layout: three_column              0.600284
watermark                         0.695048
--------------------------------  --------
====================================================================================================
TEDS:
--------------------------------  --------
ALL                               0.490723
colorful_backgroud                0.309028
data_source: PPT2PDF              0.561097
data_source: academic_literature  0.507312
data_source: book                 0.525603
data_source: colorful_textbook    0.517007
data_source: exam_paper           0.369347
data_source: magazine             0.14523
data_source: newspaper            0.365689
data_source: note                 0.348597
data_source: research_report      0.60854
fuzzy_scan                        0.681825
language: en_ch_mixed             0.330408
language: english                 0.481871
language: simplified_chinese      0.505825
layout: 1andmore_column           0.554786
layout: double_column             0.386501
layout: other_layout              0.417671
layout: single_column             0.502845
layout: three_column              0.493858
watermark                         0.494159
--------------------------------  --------
====================================================================================================
TEDS_structure_only:
--------------------------------  --------
ALL                               0.532104
colorful_backgroud                0.386745
data_source: PPT2PDF              0.592891
data_source: academic_literature  0.55012
data_source: book                 0.566893
data_source: colorful_textbook    0.583622
data_source: exam_paper           0.409375
data_source: magazine             0.426256
data_source: newspaper            0.467738
data_source: note                 0.385588
data_source: research_report      0.625599
fuzzy_scan                        0.699471
language: en_ch_mixed             0.346975
language: english                 0.523202
language: simplified_chinese      0.548981
layout: 1andmore_column           0.595087
layout: double_column             0.429087
layout: other_layout              0.519321
layout: single_column             0.532459
layout: three_column              0.721429
watermark                         0.551043
--------------------------------  --------
====================================================================================================
/home/ma-user/anaconda3/envs/py310_testpip/lib/python3.10/site-packages/vlmeval/dataset/OmniDocBench/metrics.py:300: FutureWarning: DataFrameGroupBy.apply operated on the grouping columns. This behavior is deprecated, and in a future version of pandas the grouping columns will be excluded from the operation. Either pass `include_groups=False` to exclude the groupings or explicitly select the grouping columns after groupby to silence this warning.
  up_total_avg = df.groupby("image_name").apply(lambda x: x['Edit_num'].sum() / x['upper_len'].sum()) # page level, sum of edits divided by sum of max(gt,pred) lengths for each sample
reading_order
Edit_dist:
------------  ------
ALL_page_avg  0.2996
------------  ------
====================================================================================================
----Anno Attribute---------------
sample_count:

====================================================================================================
/home/ma-user/anaconda3/envs/py310_testpip/lib/python3.10/site-packages/vlmeval/dataset/OmniDocBench/metrics.py:111: FutureWarning: DataFrameGroupBy.apply operated on the grouping columns. This behavior is deprecated, and in a future version of pandas the grouping columns will be excluded from the operation. Either pass `include_groups=False` to exclude the groupings or explicitly select the grouping columns after groupby to silence this warning.
  up_total_avg = df.groupby(["image_name", "attribute"]).apply(lambda x: (x["score"]*x['upper_len']).sum() / x['upper_len'].sum()).groupby('attribute').mean()  # At page level, accumulate edits, denominator is sum of max(gt, pred) from each sample
Edit_dist:
--------------------------------  --------
ALL                               0.2996
None                              0.311509
colorful_backgroud                0.246225
data_source: PPT2PDF              0.181233
data_source: academic_literature  0.310842
data_source: book                 0.225318
data_source: colorful_textbook    0.294244
data_source: exam_paper           0.342158
data_source: magazine             0.318104
data_source: newspaper            0.454075
data_source: note                 0.352873
data_source: research_report      0.15968
fuzzy_scan                        0.246205
language: en_ch_mixed             0.340637
language: english                 0.277293
language: simplified_chinese      0.304765
layout: 1andmore_column           0.26852
layout: double_column             0.344696
layout: other_layout              0.421162
layout: single_column             0.238163
layout: three_column              0.281645
watermark                         0.349008
--------------------------------  --------
====================================================================================================
The save path of End2End_Evaluation is: /home/zq/A911/ascend-tools/output_vlmevalkit/20260105_092249/deepseek_ocr/deepseek_ocr_OmniDocBench_End2End_Evaluation.json
The save path of overall metrics is: /home/zq/A911/ascend-tools/output_vlmevalkit/20260105_092249/deepseek_ocr/deepseek_ocr_OmniDocBench_overall.xlsx
/home/ma-user/anaconda3/envs/py310_testpip/lib/python3.10/site-packages/vlmeval/dataset/OmniDocBench/metrics.py:300: FutureWarning: DataFrameGroupBy.apply operated on the grouping columns. This behavior is deprecated, and in a future version of pandas the grouping columns will be excluded from the operation. Either pass `include_groups=False` to exclude the groupings or explicitly select the grouping columns after groupby to silence this warning.
  up_total_avg = df.groupby("image_name").apply(lambda x: x['Edit_num'].sum() / x['upper_len'].sum()) # page level, sum of edits divided by sum of max(gt,pred) lengths for each sample
TEDS:
---  ----------
all  0.00293189
---  ----------
====================================================================================================
TEDS_structure_only:
---  ----------
all  0.00340846
---  ----------
====================================================================================================
Edit_dist:
------------  --------
ALL_page_avg  0.874582
------------  --------
====================================================================================================
----Anno Attribute---------------
Edit_dist:
----------------------------------  --------
include_background: False           0.908535
include_background: True            0.887306
include_equation: False             0.897509
include_equation: True              0.924348
include_photo: False                0.901922
language: table_en                  0.920348
language: table_en_ch_mixed         0.848848
language: table_simplified_chinese  0.895218
line: fewer_line                    0.894961
line: full_line                     0.907197
line: less_line                     0.910202
line: wireless_line                 0.885772
table_layout: horizontal            0.901307
table_layout: vertical              0.967078
with_span: False                    0.909165
with_span: True                     0.888408
with_structured_text: False         0.905317
with_structured_text: True          0.755933
----------------------------------  --------
====================================================================================================
TEDS:
----------------------------------  -----------
include_background: False           0.00164908
include_background: True            0.00576729
include_equation: False             0.00350881
include_equation: True              3.56883e-07
include_photo: False                0.00293189
language: table_en                  0.000220895
language: table_en_ch_mixed         0.00621118
language: table_simplified_chinese  0.00413404
line: fewer_line                    0.000850525
line: full_line                     0.00537075
line: less_line                     0.000262261
line: wireless_line                 0.00529101
table_layout: horizontal            0.00295959
table_layout: vertical              0
with_span: False                    0.000704019
with_span: True                     0.00708887
with_structured_text: False         0.00300007
with_structured_text: True          0
----------------------------------  -----------
====================================================================================================
TEDS_structure_only:
----------------------------------  -----------
include_background: False           0.00168201
include_background: True            0.00722444
include_equation: False             0.00406258
include_equation: True              8.46883e-05
include_photo: False                0.00340846
language: table_en                  0.000290978
language: table_en_ch_mixed         0.00621118
language: table_simplified_chinese  0.00483697
line: fewer_line                    0.000907438
line: full_line                     0.0063996
line: less_line                     0.000262261
line: wireless_line                 0.00529101
table_layout: horizontal            0.00344066
table_layout: vertical              0
with_span: False                    0.000704019
with_span: True                     0.00845469
with_structured_text: False         0.00348773
with_structured_text: True          0
----------------------------------  -----------
====================================================================================================
sample_count:
----------------------------------  ---
include_background: False           515
include_background: True            233
include_equation: False             625
include_equation: True              123
include_photo: False                748
language: table_en                  242
language: table_en_ch_mixed          23
language: table_simplified_chinese  483
line: fewer_line                    298
line: full_line                     330
line: less_line                      93
line: wireless_line                  27
table_layout: horizontal            741
table_layout: vertical                7
with_span: False                    487
with_span: True                     261
with_structured_text: False         731
with_structured_text: True           17
----------------------------------  ---
====================================================================================================
/home/ma-user/anaconda3/envs/py310_testpip/lib/python3.10/site-packages/vlmeval/dataset/OmniDocBench/metrics.py:111: FutureWarning: DataFrameGroupBy.apply operated on the grouping columns. This behavior is deprecated, and in a future version of pandas the grouping columns will be excluded from the operation. Either pass `include_groups=False` to exclude the groupings or explicitly select the grouping columns after groupby to silence this warning.
  up_total_avg = df.groupby(["image_name", "attribute"]).apply(lambda x: (x["score"]*x['upper_len']).sum() / x['upper_len'].sum()).groupby('attribute').mean()  # At page level, accumulate edits, denominator is sum of max(gt, pred) from each sample
/home/ma-user/anaconda3/envs/py310_testpip/lib/python3.10/site-packages/vlmeval/dataset/OmniDocBench/metrics.py:117: FutureWarning: DataFrameGroupBy.apply operated on the grouping columns. This behavior is deprecated, and in a future version of pandas the grouping columns will be excluded from the operation. Either pass `include_groups=False` to exclude the groupings or explicitly select the grouping columns after groupby to silence this warning.
  page_avg = df.groupby(["image_name", "attribute"]).apply(lambda x: x["score"].mean()).groupby('attribute').mean()
/home/ma-user/anaconda3/envs/py310_testpip/lib/python3.10/site-packages/vlmeval/dataset/OmniDocBench/metrics.py:117: FutureWarning: DataFrameGroupBy.apply operated on the grouping columns. This behavior is deprecated, and in a future version of pandas the grouping columns will be excluded from the operation. Either pass `include_groups=False` to exclude the groupings or explicitly select the grouping columns after groupby to silence this warning.
  page_avg = df.groupby(["image_name", "attribute"]).apply(lambda x: x["score"].mean()).groupby('attribute').mean()
Edit_dist:
--------------------------------  --------
ALL                               0.874582
colorful_backgroud                0.884077
data_source: PPT2PDF              0.669443
data_source: academic_literature  0.896414
data_source: book                 0.853304
data_source: colorful_textbook    0.86477
data_source: exam_paper           0.956598
data_source: magazine             0.968481
data_source: newspaper            0.911238
data_source: note                 0.908109
data_source: research_report      0.833855
fuzzy_scan                        0.891993
language: en_ch_mixed             0.901451
language: english                 0.898897
language: simplified_chinese      0.862359
layout: 1andmore_column           0.897399
layout: double_column             0.959971
layout: other_layout              0.924973
layout: single_column             0.844235
layout: three_column              0.9329
watermark                         0.897469
--------------------------------  --------
====================================================================================================
TEDS:
--------------------------------  -----------
ALL                               0.00134935
colorful_backgroud                0.00251559
data_source: PPT2PDF              0
data_source: academic_literature  0.000921665
data_source: book                 0.00223214
data_source: colorful_textbook    0.00402494
data_source: exam_paper           0
data_source: magazine             0
data_source: newspaper            0
data_source: note                 0
data_source: research_report      0.00236259
fuzzy_scan                        0.00804989
language: en_ch_mixed             0
language: english                 0.000593962
language: simplified_chinese      0.0017655
layout: 1andmore_column           0.00126054
layout: double_column             0
layout: other_layout              0
layout: single_column             0.00184749
layout: three_column              0
watermark                         0
--------------------------------  -----------
====================================================================================================
TEDS_structure_only:
--------------------------------  -----------
ALL                               0.00167078
colorful_backgroud                0.00446429
data_source: PPT2PDF              0
data_source: academic_literature  0.00121408
data_source: book                 0.00223214
data_source: colorful_textbook    0.00714286
data_source: exam_paper           0
data_source: magazine             0
data_source: newspaper            0
data_source: note                 0
data_source: research_report      0.00264838
fuzzy_scan                        0.0142857
language: en_ch_mixed             0
language: english                 0.000782407
language: simplified_chinese      0.00216614
layout: 1andmore_column           0.0015689
layout: double_column             0
layout: other_layout              0
layout: single_column             0.0022853
layout: three_column              0
watermark                         0
--------------------------------  -----------
====================================================================================================
The save path of table_attribute is :/home/zq/A911/ascend-tools/output_vlmevalkit/20260105_092249/deepseek_ocr/deepseek_ocr_OmniDocBench_table_attribute.xlsx
[2026-01-05 14:20:46] [INFO] - The evaluation of model deepseek_ocr x dataset OmniDocBench has finished!
[2026-01-05 14:20:46] [INFO] - Evaluation Results:
[2026-01-05 14:20:46] [INFO] -
----------------------------  ------
text_block_Edit_dist_EN       0.292
text_block_Edit_dist_CH       0.344
display_formula_Edit_dist_EN  0.504
display_formula_Edit_dist_CH  0.725
display_formula_CDM_EN        -
display_formula_CDM_CH        -
table_TEDS_EN                 48.187
table_TEDS_CH                 50.582
table_Edit_dist_EN            0.721
table_Edit_dist_CH            0.678
reading_order_Edit_dist_EN    0.277
reading_order_Edit_dist_CH    0.305
overall_EN                    0.449
overall_CH                    0.513
----------------------------  ------
[2026-01-05 14:20:46] [INFO] - ======================================================================
[2026-01-05 14:20:46] [INFO] - =               Start Getting All Evaluation Reports               =
[2026-01-05 14:20:46] [INFO] - ======================================================================
[2026-01-05 14:20:46] [INFO] - *****************************************************************
[2026-01-05 14:20:46] [INFO] - *                Eval Benchmark Command Details                 *
[2026-01-05 14:20:46] [INFO] - * Test Datasets: OmniDocBench                                   *
[2026-01-05 14:20:46] [INFO] - * Eval Backend: VLMEvalKit                                      *
[2026-01-05 14:20:46] [INFO] - * Eval Mode: eval                                               *
[2026-01-05 14:20:46] [INFO] - * Judge Strategy auto                                           *
[2026-01-05 14:20:46] [INFO] - * OPENAI Backend: openai-chat                                   *
[2026-01-05 14:20:46] [INFO] - * Dataset Directory: ./vlm_data                                 *
[2026-01-05 14:20:46] [INFO] - * Num Workers: 2                                                *
[2026-01-05 14:20:46] [INFO] - * Concurrency: 10                                               *
[2026-01-05 14:20:46] [INFO] - * [1/1] Testing Service: deepseek_ocr                           *
[2026-01-05 14:20:46] [INFO] - *       Service URL: http://127.0.0.1:8001/v1/chat/completions  *
[2026-01-05 14:20:46] [INFO] - *       Max Output Length: None                                 *
[2026-01-05 14:20:46] [INFO] - *       Tokenizer Path: /home/zq/A911/DeepSeek-OCR              *
[2026-01-05 14:20:46] [INFO] - * Judge Service: Qwen2.5-7B-Instruct                            *
[2026-01-05 14:20:46] [INFO] - *       Service URL: http://127.0.0.1:8085/v1/chat/completions  *
[2026-01-05 14:20:46] [INFO] - *       Max Output Length: None                                 *
[2026-01-05 14:20:46] [INFO] - *       Tokenizer Path: /home/zq/A911/DeepSeek-OCR              *
[2026-01-05 14:20:46] [INFO] - * Work Directory: output_vlmevalkit/20260105_092249             *
[2026-01-05 14:20:46] [INFO] - *****************************************************************
[2026-01-05 14:20:46] [INFO] - The evaluation of model deepseek_ocr x dataset OmniDocBench!
[2026-01-05 14:20:46] [INFO] - Evaluation Results:
[2026-01-05 14:20:46] [INFO] -
----------------------------  ------
text_block_Edit_dist_EN       0.292
text_block_Edit_dist_CH       0.344
display_formula_Edit_dist_EN  0.504
display_formula_Edit_dist_CH  0.725
display_formula_CDM_EN        -
display_formula_CDM_CH        -
table_TEDS_EN                 48.187
table_TEDS_CH                 50.582
table_Edit_dist_EN            0.721
table_Edit_dist_CH            0.678
reading_order_Edit_dist_EN    0.277
reading_order_Edit_dist_CH    0.305
overall_EN                    0.449
overall_CH                    0.513
----------------------------  ------