Repository: EvolvingLMMs-Lab/lmms-eval
Issue: the value of egoschema is N/A
Root cause: Validation Gap

After I finish the  eval,the output is :
```
limit: None,
num_fewshot: None,
batch_size: 1
|  Tasks  |Version|Filter|n-shot|  Metric  |Value|   |Stderr|
|---------|-------|------|-----:|----------|-----|---|------|
|egoschema|Yaml   |none  |     0|submission|N/A  |±  |N/A   |
```

in the log json, I found the  "answer "is null:
```
{
            "doc_id": 1,
            "target": null,
            "doc": {
                "question_idx": "00001",
                "question": "Based on the events of the video, how would you describe c's behavior in the grocery store and its purpose, without listing every action?",
                "video_idx": "001d2d1b-d2f9-4c39-810e-6e2087ff9d5a",
                "option": [
                    "A. C shopped carefully, selecting fridge items, interacting with staff, and frequently checking surroundings.",
                    "B. C acted meticulously in the store, with a focus on beverages and reading papers, repeatedly looking around to ensure no one interfered with the shopping experience, and interacting with employees as needed.",
                    "C. C's behavior appeared cautious and observant, aiming to shop for various items while paying attention to the surroundings.",
                    "D. C's behavior could be described as vigilant and thorough, as evidenced by selecting various items from the fridge, reading materials on the countertop, and observing their environment at multiple points throughout the video.",
                    "E. C behaved with precision and caution in the grocery store, opting for a step-by-step process of fridge interaction, reading materials, and engaging in a back-and-forth communication with the store staff using calculators."
                ],
                "answer": null
            },
            "arguments": [
                [
                    "Based on the events of the video, how would you describe c's behavior in the grocery store and its purpose, without listing every action?\nA. C shopped carefully, selecting fridge items, interacting with staff, and frequently checking surroundings.\nB. C acted meticulously in the store, with a focus on beverages and reading papers, repeatedly looking around to ensure no one interfered with the shopping experience, and interacting with employees as needed.\nC. C's behavior appeared cautious and observant, aiming to shop for various items while paying attention to the surroundings.\nD. C's behavior could be described as vigilant and thorough, as evidenced by selecting various items from the fridge, reading materials on the countertop, and observing their environment at multiple points throughout the video.\nE. C behaved with precision and caution in the grocery store, opting for a step-by-step process of fridge interaction, reading materials, and engaging in a back-and-forth communication with the store staff using calculators.\nAnswer with the option's letter from the given choices directly.",
                    1,
                    "egoschema",
                    "test"
                ]
            ],
            "resps": [
                [
                    "D."
                ]
            ],
            "filtered_resps": [
                "D."
            ],
            "submission": {
                "001d2d1b-d2f9-4c39-810e-6e2087ff9d5a": 3
            },
            "score": {
                "pred": 3,
                "ground_truth": null
            }
        },
```

hi @guoyanan1g, i'm running into this same issue, would you mind sharing your fix?

ah okay it's because the actual labels aren't released: 

https://huggingface.co/datasets/lmms-lab/egoschema

> ah okay it's because the actual labels aren't released:
> 
> https://huggingface.co/datasets/lmms-lab/egoschema

But how does they report their performance in paper?

How to know the ground_truth then?