Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Bug: Unable to calculate metrics from saved predictions using --predict_only and --from_log
Root cause: Configuration Error

**Description:**

I'm experiencing an issue with calculating metrics after saving predictions using `--predict_only` and then attempting to compute metrics with `--from_log`. It appears that the `from_log` model is not functioning correctly, possibly due to recent changes in the log format.

**Steps to Reproduce:**

1. **Run prediction and save outputs:**

   I used the following script to generate predictions and save them:

   ```bash
   python3 -m accelerate.commands.launch \
       --num_processes=2 \
       -m lmms_eval \
       --model $model \
       --model_args logs=./logs_vlm/$model/checkpoints/,model_name=$model \
       --tasks $task \
       --batch_size 1 \
       --log_samples \
       --limit 10 \
       --predict_only \
       --output_path "./logs_vlm/$model/$task" \
       --verbosity=DEBUG
   ```

2. **Attempt to calculate metrics from saved outputs:**

   Then, I tried to calculate metrics using the saved logs:

   ```bash
   python3 -m accelerate.commands.launch \
       --num_processes=2 \
       -m lmms_eval \
       --model from_log \
       --model_args logs=./logs_vlm/$model/$task/,model_name=$model \
       --tasks $task \
       --batch_size 1 \
       --log_samples \
       --limit 10 \
       --output_path "./logs_vlm/$model/$task" \
       --verbosity=DEBUG
   ```

**Expected Behavior:**

- After running the first script with `--predict_only`, predictions should be saved to the specified `--output_path`.
- Running the second script with `--model from_log` should load these saved predictions and compute the evaluation metrics.

**Actual Behavior:**

- The second script does not compute the metrics as expected.
- It seems that the `from_log` model is not correctly processing the saved logs.

**Environment:**

- **lmms-eval:** 0.2.4

**Request:**

- Please investigate and fix the `from_log` model so that it correctly processes logs and calculates metrics.
- If there have been changes to the log format, update `from_log` to be compatible with the new format.
- Alternatively, provide guidance on how to calculate metrics from saved outputs with the current version.

I believe addressing this issue is important for workflows that separate prediction and evaluation phases. Moreover, enhancing this functionality will improve support for offline mode, as discussed in issue #335 

**Thank you for your assistance!**

---



Thank you for raising up this issue. I believe an offline workaround without gpt eval can be done as the workflow mentioned in #335. For fixing this bugs, @pufanyi may I ask do you have time to look into this recently? If not, I will look into it and try to fix it in the future. Thank you!

> Thank you for raising up this issue. I believe an offline workaround without gpt eval can be done as the workflow mentioned in [#335](https://github.com/EvolvingLMMs-Lab/lmms-eval/issues/335). For fixing this bugs, [@pufanyi](https://github.com/pufanyi) may I ask do you have time to look into this recently? If not, I will look into it and try to fix it in the future. Thank you!感谢您提出这个问题。我认为可以按照 [#335](https://github.com/EvolvingLMMs-Lab/lmms-eval/issues/335) 中提到的工作流程，在不使用 gpt eval 的情况下，采用离线方式解决。关于修复这个 bug，请问您最近有时间查看一下吗？如果没有，我会在以后查看并尝试修复。谢谢！

Hello, I'd like to ask if the bug in `from_log` has been fixed. I'm also trying to calculate metrics using `llm judge` from saved prediction results, but I'm getting an error.


I think right now we have adopted the cache api for a more flexible and easy way to load and cache requests. Feel free to check [here](https://github.com/EvolvingLMMs-Lab/lmms-eval/blob/main/docs/caching.md) for further reference. We have implemented the caching logic in models such as async_openai and vllm. Example code are like this

https://github.com/EvolvingLMMs-Lab/lmms-eval/blob/c75f1a4e6848676b9f47c3b5df07b93f60034388/lmms_eval/models/chat/async_openai.py#L221-L229

For the rest of the model if you needed, you can add this feature using the caching api and feel free to PR. Thanks!