Repository: mlcommons/inference
Issue: Simplify retrieval of MoE artifacts
Root cause: Unimplemented Feature Gap

To run [the MoE benchmark](https://github.com/mlcommons/inference/tree/master/language/mixtral-8x7b), we need to download additionally 3 files [from HuggingFace](https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1/tree/main):
```
tokenizer.json
tokenizer.model
tokenizer_config.json
```

I wonder if the 3 files could be added to the MLCommons' rclone model checkpoint folder. This would simplify workflow automation of the MoE benchmark, and aid reproducibility. Otherwise, one needs to Sign Up and/or Log In:

![image](https://github.com/user-attachments/assets/b418938f-19fc-41e8-96b7-db9103c5ffdd)


In addition, we need to use:
- `rclone copy` to get [the model checkpoint](https://github.com/mlcommons/inference/tree/master/language/mixtral-8x7b#checkpoint);
- `rclone copyurl` to get [the preprocessed](https://github.com/mlcommons/inference/tree/master/language/mixtral-8x7b#preprocessed) and [the calibration](https://github.com/mlcommons/inference/tree/master/language/mixtral-8x7b#calibration-dataset) datasets.

Is there a reason for using different commands? Can we use `rclone copy` for both?

Note that the files in question keep getting updated on HuggingFace - also not good for reproducibility:

![image](https://github.com/user-attachments/assets/fe235853-7916-4a1a-8ffb-e45f96397a4a)

@nathanw-mlc Can this be done please?

If the files keep being updated on HG, is there a specific version of the files that we want to preserve and use?

@psyhtest The tokenizer seems to have changed 13 days back. Is there a recommended version?

@arjunsuresh I'm not sure. On the one hand, some submitters might have started collecting results before that. On the other hand, the latest may be more stable and aligned with mainstream. I think we need MoE TF members to comment.

I took a look at their latest change to the tokenizer: https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1/commit/bbae113847402a22031211225b5ee45c005de7dd
It does seem some of the logics are changing. It would be good if we can upload the version from ~1 months ago when we froze the code.
I haven't downloaded and tested the tokenizer yet due to bandwidth. I will follow up once I get to it.

I recommend checkout to commit in Feb to be safe btw
@szutenberg to comment as well.

@nvzhihanj @arjunsuresh @psyhtest @nathanw-mlc I agree with Zhihan that we should use the artifacts from 1 month ago, since they were used to compute the reference accuracy and freeze the benchmark. I would propose the following steps:
- Contact folks through the MoE chat and Inference mailing list that might have started collecting results and inform about the situation.
- Upload the artifacts to the bucket where the checkpoint is hosted
- Update the ReadMe, update the HF instructions for external download


Thank you @pgmpablo157321 @nvzhihanj

@nathanw-mlc can you please use the artifacts as of June 25?

@pgmpablo157321 can you please send this email to the inference WG? 

I've uploaded the three artifacts, as well as `special_tokens_map.json`, at @pgmpablo157321's request, to the model checkpoint directory. Now, running the command to download the model checkpoint directory will include those files. If you've already run this command, no need to delete the entire directory and download again. You can just point Rclone at your copy and run the command again; Rclone will check the extant files and download just the ones you don't yet have.

`rclone copy mlc-inference:mlcommons-inference-wg-public/mixtral_8x7b/mixtral-8x7b-instruct-v0.1 ./mixtral-8x7b-instruct-v0.1 -P`

You can also run the following command to download all Mixtral files (entire model checkpoint directory, as well as the calibration and preprocessed datasets).

`rclone copy mlc-inference:mlcommons-inference-wg-public/mixtral_8x7b ./ -P`

@pgmpablo157321  Can you help update the link in the mixtral readme^? Thank you!

@nvzhihanj The readme looks good to me. I though we included the option for downloading the model from HF directly, so that part would need to be updated, but it was not included.

We can include the link from the previous version in HF. What do you think? Is that necessary/good to have?

@pgmpablo157321 You mean `git clone -> cd xx/ -> git checkout <commit>` right? Yes that would be good too!