Repository: open-compass/VLMEvalKit
Issue: How can I use VLMEvalKit to test the DeepSeek-OCR model and obtain its accuracy score on OmniDocBench?
Root cause: Documentation Deficiency

I noticed that the OmniDocBench dataset is already supported. Can we simply add the new model to the config.json file and use the OpenAI-style inference for it?

Hi! As far as I know, you can’t evaluate DeepSeek-OCR by only editing `config.json`, and the OpenAI-style interface won’t work directly. OmniDocBench is already supported, but DeepSeek-OCR is not OpenAI-API compatible. You need to implement a local model wrapper to load the model, run OCR on images, and return plain text outputs.

This is because `config.json` is only for registration, not implementation. It simply tells VLMEvalKit that the model exists, which interface to use, and what parameters to pass. The actual inference logic must be implemented in code. After that, you can run OmniDocBench using the local interface.
Hope this helps! I'm looking forward to your reply! 

> 您好！据我所知，您无法仅通过编辑文档来评估 DeepSeek-OCR `config.json`，OpenAI 风格的接口也无法直接使用。OmniDocBench 已经支持，但 DeepSeek-OCR 与 OpenAI API 不兼容。您需要实现一个本地模型封装器来加载模型、对图像运行 OCR 并返回纯文本输出。
> 
> 这是因为`config.json`该接口仅用于注册，不涉及实际实现。它只是告诉 VLMEvalKit 模型存在、使用哪个接口以及要传递哪些参数。实际的推理逻辑必须在代码中实现。之后，您可以使用本地接口运行 OmniDocBench。 希望这能帮到您！期待您的回复！

您好，我使用DeepSeek-OCR模型在OmniDocBench上评测，分值相较于官方低很多。您是否知道OmniDocBench的分值含义呢？以及tsv数据集内数据981条，但官方数据有1335


self.img_root:/home/zq/A911/ascend-tools/vlm_data/data/images/OmniDocBench
Loading data: 981it [00:00, 1141.39it/s]
load_success:981,load_fail:0
Loading data: 981it [00:00, 2514.72it/s]
load_table_success:428,load_table_fail:0
Cannot find pred for 0 samples.
Time out for plain text match of docstructbench_llm-raw-scihub-o.O-s00395-011-0222-8.pdf_10.jpg, match_gt2pred_simple will be used.
WARNING:pylatexenc.latex2text:WARNING: Error in configuration: macro '\frac' failed its substitution!
Time out for plain text match of jiaocaineedrop_jiaocai_needrop_en_715.jpg, match_gt2pred_simple will be used.
Time out for plain text match of yanbaor2_yanbaoPPT_5946.jpg, match_gt2pred_simple will be used.
/home/ma-user/anaconda3/envs/py310_testpip/lib/python3.10/site-packages/vlmeval/dataset/OmniDocBench/metrics.py:300: FutureWarning: DataFrameGroupBy.apply operated on the grouping columns. This behavior is deprecated, and in a future version of pandas the grouping columns will be excluded from the operation. Either pass include_groups=False to exclude the groupings or explicitly select the grouping columns after groupby to silence this warning.
up_total_avg = df.groupby("image_name").apply(lambda x: x['Edit_num'].sum() / x['upper_len'].sum()) # page level, sum of edits divided by sum of max(gt,pred) lengths for each sample
text_block
Edit_dist:

ALL_page_avg 0.327432

====================================================================================================
----Anno Attribute---------------
Edit_dist:

text_background: multi_colored 0.419213
text_background: single_colored 0.463535
text_background: white 0.539955
text_language: text_en_ch_mixed 0.400074
text_language: text_english 0.430851
text_language: text_simplified_chinese 0.564498
text_rotate: horizontal 0.891245
text_rotate: normal 0.527424
text_rotate: rotate270 0.996624

====================================================================================================
sample_count:

text_background: multi_colored 994
text_background: single_colored 600
text_background: white 10040
text_language: text_en_ch_mixed 259
text_language: text_english 2944
text_language: text_simplified_chinese 8396
text_rotate: horizontal 63
text_rotate: normal 11400
text_rotate: rotate270 30

====================================================================================================
/home/ma-user/anaconda3/envs/py310_testpip/lib/python3.10/site-packages/vlmeval/dataset/OmniDocBench/metrics.py:111: FutureWarning: DataFrameGroupBy.apply operated on the grouping columns. This behavior is deprecated, and in a future version of pandas the grouping columns will be excluded from the operation. Either pass include_groups=False to exclude the groupings or explicitly select the grouping columns after groupby to silence this warning.
up_total_avg = df.groupby(["image_name", "attribute"]).apply(lambda x: (x["score"]*x['upper_len']).sum() / x['upper_len'].sum()).groupby('attribute').mean() # At page level, accumulate edits, denominator is sum of max(gt, pred) from each sample
Edit_dist:

ALL 0.327432
None 0.322465
colorful_backgroud 0.273653
data_source: PPT2PDF 0.262033
data_source: academic_literature 0.322449
data_source: book 0.286129
data_source: colorful_textbook 0.276364
data_source: exam_paper 0.370504
data_source: magazine 0.288656
data_source: newspaper 0.532822
data_source: note 0.363901
data_source: research_report 0.162019
fuzzy_scan 0.307146
language: en_ch_mixed 0.332221
language: english 0.292347
language: simplified_chinese 0.343675
layout: 1andmore_column 0.28034
layout: double_column 0.380913
layout: other_layout 0.452067
layout: single_column 0.262505
layout: three_column 0.348257
watermark 0.319464

====================================================================================================
/home/ma-user/anaconda3/envs/py310_testpip/lib/python3.10/site-packages/vlmeval/dataset/OmniDocBench/metrics.py:300: FutureWarning: DataFrameGroupBy.apply operated on the grouping columns. This behavior is deprecated, and in a future version of pandas the grouping columns will be excluded from the operation. Either pass include_groups=False to exclude the groupings or explicitly select the grouping columns after groupby to silence this warning.
up_total_avg = df.groupby("image_name").apply(lambda x: x['Edit_num'].sum() / x['upper_len'].sum()) # page level, sum of edits divided by sum of max(gt,pred) lengths for each sample
display_formula
Edit_dist:

ALL_page_avg 0.641538

====================================================================================================
----Anno Attribute---------------
Edit_dist:

formula_type: handwriting 0.988333
formula_type: print 0.587018

====================================================================================================
sample_count:

formula_type: handwriting 4
formula_type: print 359

====================================================================================================
/home/ma-user/anaconda3/envs/py310_testpip/lib/python3.10/site-packages/vlmeval/dataset/OmniDocBench/metrics.py:111: FutureWarning: DataFrameGroupBy.apply operated on the grouping columns. This behavior is deprecated, and in a future version of pandas the grouping columns will be excluded from the operation. Either pass include_groups=False to exclude the groupings or explicitly select the grouping columns after groupby to silence this warning.
up_total_avg = df.groupby(["image_name", "attribute"]).apply(lambda x: (x["score"]*x['upper_len']).sum() / x['upper_len'].sum()).groupby('attribute').mean() # At page level, accumulate edits, denominator is sum of max(gt, pred) from each sample
Edit_dist:

ALL 0.641538
None 0.636615
colorful_backgroud 0.668036
data_source: PPT2PDF 0.859259
data_source: academic_literature 0.587644
data_source: book 0.548733
data_source: colorful_textbook 0.639781
data_source: exam_paper 0.656845
data_source: note 1
fuzzy_scan 0.70241
language: english 0.504478
language: simplified_chinese 0.724605
layout: 1andmore_column 0.73331
layout: double_column 0.631132
layout: other_layout 0.710689
layout: single_column 0.591374
layout: three_column 0.520625
watermark 0.635837

====================================================================================================
/home/ma-user/anaconda3/envs/py310_testpip/lib/python3.10/site-packages/vlmeval/dataset/OmniDocBench/metrics.py:300: FutureWarning: DataFrameGroupBy.apply operated on the grouping columns. This behavior is deprecated, and in a future version of pandas the grouping columns will be excluded from the operation. Either pass include_groups=False to exclude the groupings or explicitly select the grouping columns after groupby to silence this warning.
up_total_avg = df.groupby("image_name").apply(lambda x: x['Edit_num'].sum() / x['upper_len'].sum()) # page level, sum of edits divided by sum of max(gt,pred) lengths for each sample
table
TEDS:

all 0.479982

====================================================================================================
TEDS_structure_only:

all 0.520253

====================================================================================================
Edit_dist:

ALL_page_avg 0.695405

====================================================================================================
----Anno Attribute---------------
Edit_dist:

include_background: False 0.704722
include_background: True 0.699145
include_equation: False 0.696304
include_equation: True 0.731005
include_photo: False 0.702872
language: table_en 0.72194
language: table_en_ch_mixed 0.713077
language: table_simplified_chinese 0.69377
line: fewer_line 0.684708
line: full_line 0.720814
line: less_line 0.693746
line: wireless_line 0.671276
table_layout: horizontal 0.697931
table_layout: vertical 1
with_span: False 0.702529
with_span: True 0.703506
with_structured_text: False 0.706014
with_structured_text: True 0.616351

====================================================================================================
TEDS:

include_background: False 0.500174
include_background: True 0.439314
include_equation: False 0.492544
include_equation: True 0.426168
include_photo: False 0.479982
language: table_en 0.475307
language: table_en_ch_mixed 0.52724
language: table_simplified_chinese 0.479595
line: fewer_line 0.548602
line: full_line 0.420799
line: less_line 0.549747
line: wireless_line 0.31713
table_layout: horizontal 0.487963
table_layout: vertical 0
with_span: False 0.512647
with_span: True 0.419443
with_structured_text: False 0.475811
with_structured_text: True 0.594822

====================================================================================================
TEDS_structure_only:

include_background: False 0.53943
include_background: True 0.481628
include_equation: False 0.526944
include_equation: True 0.491587
include_photo: False 0.520253
language: table_en 0.516791
language: table_en_ch_mixed 0.552654
language: table_simplified_chinese 0.520103
line: fewer_line 0.581726
line: full_line 0.461688
line: less_line 0.611525
line: wireless_line 0.328135
table_layout: horizontal 0.528903
table_layout: vertical 0
with_span: False 0.545351
with_span: True 0.473737
with_structured_text: False 0.516497
with_structured_text: True 0.623675

====================================================================================================
sample_count:

include_background: False 286
include_background: True 142
include_equation: False 347
include_equation: True 81
include_photo: False 428
language: table_en 128
language: table_en_ch_mixed 15
language: table_simplified_chinese 285
line: fewer_line 147
line: full_line 205
line: less_line 62
line: wireless_line 14
table_layout: horizontal 421
table_layout: vertical 7
with_span: False 278
with_span: True 150
with_structured_text: False 413
with_structured_text: True 15

====================================================================================================
/home/ma-user/anaconda3/envs/py310_testpip/lib/python3.10/site-packages/vlmeval/dataset/OmniDocBench/metrics.py:111: FutureWarning: DataFrameGroupBy.apply operated on the grouping columns. This behavior is deprecated, and in a future version of pandas the grouping columns will be excluded from the operation. Either pass include_groups=False to exclude the groupings or explicitly select the grouping columns after groupby to silence this warning.
up_total_avg = df.groupby(["image_name", "attribute"]).apply(lambda x: (x["score"]*x['upper_len']).sum() / x['upper_len'].sum()).groupby('attribute').mean() # At page level, accumulate edits, denominator is sum of max(gt, pred) from each sample
/home/ma-user/anaconda3/envs/py310_testpip/lib/python3.10/site-packages/vlmeval/dataset/OmniDocBench/metrics.py:117: FutureWarning: DataFrameGroupBy.apply operated on the grouping columns. This behavior is deprecated, and in a future version of pandas the grouping columns will be excluded from the operation. Either pass include_groups=False to exclude the groupings or explicitly select the grouping columns after groupby to silence this warning.
page_avg = df.groupby(["image_name", "attribute"]).apply(lambda x: x["score"].mean()).groupby('attribute').mean()
/home/ma-user/anaconda3/envs/py310_testpip/lib/python3.10/site-packages/vlmeval/dataset/OmniDocBench/metrics.py:117: FutureWarning: DataFrameGroupBy.apply operated on the grouping columns. This behavior is deprecated, and in a future version of pandas the grouping columns will be excluded from the operation. Either pass include_groups=False to exclude the groupings or explicitly select the grouping columns after groupby to silence this warning.
page_avg = df.groupby(["image_name", "attribute"]).apply(lambda x: x["score"].mean()).groupby('attribute').mean()
Edit_dist:

ALL 0.695405
colorful_backgroud 0.755806
data_source: PPT2PDF 0.558462
data_source: academic_literature 0.703317
data_source: book 0.695917
data_source: colorful_textbook 0.689004
data_source: exam_paper 0.763753
data_source: magazine 0.80849
data_source: newspaper 0.733429
data_source: note 0.757532
data_source: research_report 0.635508
fuzzy_scan 0.624045
language: en_ch_mixed 0.791152
language: english 0.720799
language: simplified_chinese 0.677849
layout: 1andmore_column 0.682159
layout: double_column 0.743113
layout: other_layout 0.716459
layout: single_column 0.688306
layout: three_column 0.600284
watermark 0.695048

====================================================================================================
TEDS:

ALL 0.490723
colorful_backgroud 0.309028
data_source: PPT2PDF 0.561097
data_source: academic_literature 0.507312
data_source: book 0.525603
data_source: colorful_textbook 0.517007
data_source: exam_paper 0.369347
data_source: magazine 0.14523
data_source: newspaper 0.365689
data_source: note 0.348597
data_source: research_report 0.60854
fuzzy_scan 0.681825
language: en_ch_mixed 0.330408
language: english 0.481871
language: simplified_chinese 0.505825
layout: 1andmore_column 0.554786
layout: double_column 0.386501
layout: other_layout 0.417671
layout: single_column 0.502845
layout: three_column 0.493858
watermark 0.494159

====================================================================================================
TEDS_structure_only:

ALL 0.532104
colorful_backgroud 0.386745
data_source: PPT2PDF 0.592891
data_source: academic_literature 0.55012
data_source: book 0.566893
data_source: colorful_textbook 0.583622
data_source: exam_paper 0.409375
data_source: magazine 0.426256
data_source: newspaper 0.467738
data_source: note 0.385588
data_source: research_report 0.625599
fuzzy_scan 0.699471
language: en_ch_mixed 0.346975
language: english 0.523202
language: simplified_chinese 0.548981
layout: 1andmore_column 0.595087
layout: double_column 0.429087
layout: other_layout 0.519321
layout: single_column 0.532459
layout: three_column 0.721429
watermark 0.551043

====================================================================================================
/home/ma-user/anaconda3/envs/py310_testpip/lib/python3.10/site-packages/vlmeval/dataset/OmniDocBench/metrics.py:300: FutureWarning: DataFrameGroupBy.apply operated on the grouping columns. This behavior is deprecated, and in a future version of pandas the grouping columns will be excluded from the operation. Either pass include_groups=False to exclude the groupings or explicitly select the grouping columns after groupby to silence this warning.
up_total_avg = df.groupby("image_name").apply(lambda x: x['Edit_num'].sum() / x['upper_len'].sum()) # page level, sum of edits divided by sum of max(gt,pred) lengths for each sample
reading_order
Edit_dist:

ALL_page_avg 0.2996

====================================================================================================
----Anno Attribute---------------
sample_count:

====================================================================================================
/home/ma-user/anaconda3/envs/py310_testpip/lib/python3.10/site-packages/vlmeval/dataset/OmniDocBench/metrics.py:111: FutureWarning: DataFrameGroupBy.apply operated on the grouping columns. This behavior is deprecated, and in a future version of pandas the grouping columns will be excluded from the operation. Either pass include_groups=False to exclude the groupings or explicitly select the grouping columns after groupby to silence this warning.
up_total_avg = df.groupby(["image_name", "attribute"]).apply(lambda x: (x["score"]*x['upper_len']).sum() / x['upper_len'].sum()).groupby('attribute').mean() # At page level, accumulate edits, denominator is sum of max(gt, pred) from each sample
Edit_dist:

ALL 0.2996
None 0.311509
colorful_backgroud 0.246225
data_source: PPT2PDF 0.181233
data_source: academic_literature 0.310842
data_source: book 0.225318
data_source: colorful_textbook 0.294244
data_source: exam_paper 0.342158
data_source: magazine 0.318104
data_source: newspaper 0.454075
data_source: note 0.352873
data_source: research_report 0.15968
fuzzy_scan 0.246205
language: en_ch_mixed 0.340637
language: english 0.277293
language: simplified_chinese 0.304765
layout: 1andmore_column 0.26852
layout: double_column 0.344696
layout: other_layout 0.421162
layout: single_column 0.238163
layout: three_column 0.281645
watermark 0.349008

====================================================================================================
The save path of End2End_Evaluation is: /home/zq/A911/ascend-tools/output_vlmevalkit/20260105_092249/deepseek_ocr/deepseek_ocr_OmniDocBench_End2End_Evaluation.json
The save path of overall metrics is: /home/zq/A911/ascend-tools/output_vlmevalkit/20260105_092249/deepseek_ocr/deepseek_ocr_OmniDocBench_overall.xlsx
/home/ma-user/anaconda3/envs/py310_testpip/lib/python3.10/site-packages/vlmeval/dataset/OmniDocBench/metrics.py:300: FutureWarning: DataFrameGroupBy.apply operated on the grouping columns. This behavior is deprecated, and in a future version of pandas the grouping columns will be excluded from the operation. Either pass include_groups=False to exclude the groupings or explicitly select the grouping columns after groupby to silence this warning.
up_total_avg = df.groupby("image_name").apply(lambda x: x['Edit_num'].sum() / x['upper_len'].sum()) # page level, sum of edits divided by sum of max(gt,pred) lengths for each sample
TEDS:

all 0.00293189

====================================================================================================
TEDS_structure_only:

all 0.00340846

====================================================================================================
Edit_dist:

ALL_page_avg 0.874582

====================================================================================================
----Anno Attribute---------------
Edit_dist:

include_background: False 0.908535
include_background: True 0.887306
include_equation: False 0.897509
include_equation: True 0.924348
include_photo: False 0.901922
language: table_en 0.920348
language: table_en_ch_mixed 0.848848
language: table_simplified_chinese 0.895218
line: fewer_line 0.894961
line: full_line 0.907197
line: less_line 0.910202
line: wireless_line 0.885772
table_layout: horizontal 0.901307
table_layout: vertical 0.967078
with_span: False 0.909165
with_span: True 0.888408
with_structured_text: False 0.905317
with_structured_text: True 0.755933

====================================================================================================
TEDS:

include_background: False 0.00164908
include_background: True 0.00576729
include_equation: False 0.00350881
include_equation: True 3.56883e-07
include_photo: False 0.00293189
language: table_en 0.000220895
language: table_en_ch_mixed 0.00621118
language: table_simplified_chinese 0.00413404
line: fewer_line 0.000850525
line: full_line 0.00537075
line: less_line 0.000262261
line: wireless_line 0.00529101
table_layout: horizontal 0.00295959
table_layout: vertical 0
with_span: False 0.000704019
with_span: True 0.00708887
with_structured_text: False 0.00300007
with_structured_text: True 0

====================================================================================================
TEDS_structure_only:

include_background: False 0.00168201
include_background: True 0.00722444
include_equation: False 0.00406258
include_equation: True 8.46883e-05
include_photo: False 0.00340846
language: table_en 0.000290978
language: table_en_ch_mixed 0.00621118
language: table_simplified_chinese 0.00483697
line: fewer_line 0.000907438
line: full_line 0.0063996
line: less_line 0.000262261
line: wireless_line 0.00529101
table_layout: horizontal 0.00344066
table_layout: vertical 0
with_span: False 0.000704019
with_span: True 0.00845469
with_structured_text: False 0.00348773
with_structured_text: True 0

====================================================================================================
sample_count:

include_background: False 515
include_background: True 233
include_equation: False 625
include_equation: True 123
include_photo: False 748
language: table_en 242
language: table_en_ch_mixed 23
language: table_simplified_chinese 483
line: fewer_line 298
line: full_line 330
line: less_line 93
line: wireless_line 27
table_layout: horizontal 741
table_layout: vertical 7
with_span: False 487
with_span: True 261
with_structured_text: False 731
with_structured_text: True 17

====================================================================================================
/home/ma-user/anaconda3/envs/py310_testpip/lib/python3.10/site-packages/vlmeval/dataset/OmniDocBench/metrics.py:111: FutureWarning: DataFrameGroupBy.apply operated on the grouping columns. This behavior is deprecated, and in a future version of pandas the grouping columns will be excluded from the operation. Either pass include_groups=False to exclude the groupings or explicitly select the grouping columns after groupby to silence this warning.
up_total_avg = df.groupby(["image_name", "attribute"]).apply(lambda x: (x["score"]*x['upper_len']).sum() / x['upper_len'].sum()).groupby('attribute').mean() # At page level, accumulate edits, denominator is sum of max(gt, pred) from each sample
/home/ma-user/anaconda3/envs/py310_testpip/lib/python3.10/site-packages/vlmeval/dataset/OmniDocBench/metrics.py:117: FutureWarning: DataFrameGroupBy.apply operated on the grouping columns. This behavior is deprecated, and in a future version of pandas the grouping columns will be excluded from the operation. Either pass include_groups=False to exclude the groupings or explicitly select the grouping columns after groupby to silence this warning.
page_avg = df.groupby(["image_name", "attribute"]).apply(lambda x: x["score"].mean()).groupby('attribute').mean()
/home/ma-user/anaconda3/envs/py310_testpip/lib/python3.10/site-packages/vlmeval/dataset/OmniDocBench/metrics.py:117: FutureWarning: DataFrameGroupBy.apply operated on the grouping columns. This behavior is deprecated, and in a future version of pandas the grouping columns will be excluded from the operation. Either pass include_groups=False to exclude the groupings or explicitly select the grouping columns after groupby to silence this warning.
page_avg = df.groupby(["image_name", "attribute"]).apply(lambda x: x["score"].mean()).groupby('attribute').mean()
Edit_dist:

ALL 0.874582
colorful_backgroud 0.884077
data_source: PPT2PDF 0.669443
data_source: academic_literature 0.896414
data_source: book 0.853304
data_source: colorful_textbook 0.86477
data_source: exam_paper 0.956598
data_source: magazine 0.968481
data_source: newspaper 0.911238
data_source: note 0.908109
data_source: research_report 0.833855
fuzzy_scan 0.891993
language: en_ch_mixed 0.901451
language: english 0.898897
language: simplified_chinese 0.862359
layout: 1andmore_column 0.897399
layout: double_column 0.959971
layout: other_layout 0.924973
layout: single_column 0.844235
layout: three_column 0.9329
watermark 0.897469

====================================================================================================
TEDS:

ALL 0.00134935
colorful_backgroud 0.00251559
data_source: PPT2PDF 0
data_source: academic_literature 0.000921665
data_source: book 0.00223214
data_source: colorful_textbook 0.00402494
data_source: exam_paper 0
data_source: magazine 0
data_source: newspaper 0
data_source: note 0
data_source: research_report 0.00236259
fuzzy_scan 0.00804989
language: en_ch_mixed 0
language: english 0.000593962
language: simplified_chinese 0.0017655
layout: 1andmore_column 0.00126054
layout: double_column 0
layout: other_layout 0
layout: single_column 0.00184749
layout: three_column 0
watermark 0

====================================================================================================
TEDS_structure_only:

ALL 0.00167078
colorful_backgroud 0.00446429
data_source: PPT2PDF 0
data_source: academic_literature 0.00121408
data_source: book 0.00223214
data_source: colorful_textbook 0.00714286
data_source: exam_paper 0
data_source: magazine 0
data_source: newspaper 0
data_source: note 0
data_source: research_report 0.00264838
fuzzy_scan 0.0142857
language: en_ch_mixed 0
language: english 0.000782407
language: simplified_chinese 0.00216614
layout: 1andmore_column 0.0015689
layout: double_column 0
layout: other_layout 0
layout: single_column 0.0022853
layout: three_column 0
watermark 0

====================================================================================================
The save path of table_attribute is :/home/zq/A911/ascend-tools/output_vlmevalkit/20260105_092249/deepseek_ocr/deepseek_ocr_OmniDocBench_table_attribute.xlsx
[2026-01-05 14:20:46] [INFO] - The evaluation of model deepseek_ocr x dataset OmniDocBench has finished!
[2026-01-05 14:20:46] [INFO] - Evaluation Results:
[2026-01-05 14:20:46] [INFO] -

text_block_Edit_dist_EN 0.292
text_block_Edit_dist_CH 0.344
display_formula_Edit_dist_EN 0.504
display_formula_Edit_dist_CH 0.725
display_formula_CDM_EN -
display_formula_CDM_CH -
table_TEDS_EN 48.187
table_TEDS_CH 50.582
table_Edit_dist_EN 0.721
table_Edit_dist_CH 0.678
reading_order_Edit_dist_EN 0.277
reading_order_Edit_dist_CH 0.305
overall_EN 0.449
overall_CH 0.513

[2026-01-05 14:20:46] [INFO] - ======================================================================
[2026-01-05 14:20:46] [INFO] - = Start Getting All Evaluation Reports =
[2026-01-05 14:20:46] [INFO] - ======================================================================
[2026-01-05 14:20:46] [INFO] - *****************************************************************
[2026-01-05 14:20:46] [INFO] - * Eval Benchmark Command Details *
[2026-01-05 14:20:46] [INFO] - * Test Datasets: OmniDocBench *
[2026-01-05 14:20:46] [INFO] - * Eval Backend: VLMEvalKit *
[2026-01-05 14:20:46] [INFO] - * Eval Mode: eval *
[2026-01-05 14:20:46] [INFO] - * Judge Strategy auto *
[2026-01-05 14:20:46] [INFO] - * OPENAI Backend: openai-chat *
[2026-01-05 14:20:46] [INFO] - * Dataset Directory: ./vlm_data *
[2026-01-05 14:20:46] [INFO] - * Num Workers: 2 *
[2026-01-05 14:20:46] [INFO] - * Concurrency: 10 *
[2026-01-05 14:20:46] [INFO] - * [1/1] Testing Service: deepseek_ocr *
[2026-01-05 14:20:46] [INFO] - * Service URL: http://127.0.0.1:8001/v1/chat/completions *
[2026-01-05 14:20:46] [INFO] - * Max Output Length: None *
[2026-01-05 14:20:46] [INFO] - * Tokenizer Path: /home/zq/A911/DeepSeek-OCR *
[2026-01-05 14:20:46] [INFO] - * Judge Service: Qwen2.5-7B-Instruct *
[2026-01-05 14:20:46] [INFO] - * Service URL: http://127.0.0.1:8085/v1/chat/completions *
[2026-01-05 14:20:46] [INFO] - * Max Output Length: None *
[2026-01-05 14:20:46] [INFO] - * Tokenizer Path: /home/zq/A911/DeepSeek-OCR *
[2026-01-05 14:20:46] [INFO] - * Work Directory: output_vlmevalkit/20260105_092249 *
[2026-01-05 14:20:46] [INFO] - *****************************************************************
[2026-01-05 14:20:46] [INFO] - The evaluation of model deepseek_ocr x dataset OmniDocBench!
[2026-01-05 14:20:46] [INFO] - Evaluation Results:
[2026-01-05 14:20:46] [INFO] -

text_block_Edit_dist_EN 0.292
text_block_Edit_dist_CH 0.344
display_formula_Edit_dist_EN 0.504
display_formula_Edit_dist_CH 0.725
display_formula_CDM_EN -
display_formula_CDM_CH -
table_TEDS_EN 48.187
table_TEDS_CH 50.582
table_Edit_dist_EN 0.721
table_Edit_dist_CH 0.678
reading_order_Edit_dist_EN 0.277
reading_order_Edit_dist_CH 0.305
overall_EN 0.449
overall_CH 0.513