vidore_eval.md

How to run the Vidore Evaluation

If you want to run the vidore evaluation on the jina-embeddings-v4 model (and on the Document Retrieval Benchmark curated by Jina AI), you need to install requirements in this fork/branch (these changes should be merged with the source code of Vidore soon).

pip install vidore-benchmark[jina-v4]

You can run the evaluation with the following command:

vidore-benchmark evaluate-retriever \
    --model-class jev4 \
    --model-name jinaai/jina-embeddings-v4 \
    --collection-name jinaai/document-screenshot-retrieval-benchmark-small-684831c022c53b21c313b449 \
    --dataset-format qa \
    --split test