Spaces:
Running
on
CPU Upgrade
Feature request: Add non AutoModelForCausalLM models
As I can see that in some cases models that require trust_remote_code=True are manually added to the leaderboard, wouldnt it be good to also add at least 5 models like T5-xl/T5-xxl/flan-t5-xxl/ul2/flan-ul2 manually, though it does not support doing sequence-to-sequence models?
They are quite meaningful models. It can't be about hardware, one can run the benchmarks manually on local hardware.
It looks incomplete without those. especially the larger versions of T5. I hear there are differences between decoder only models and encoder-decoder models performance wise.
The T5 models are still popular fine-tunes on this platform. Can't really ignore that.
Hi! The T5 models are not yet there because we don't support the architecture atm.
Running a trust_remote_code=True model is "relatively easy", we inspect the code manually (which can be time consuming though, hence why it's exceptional) then run our backend with the added trust... option.
Running a T5 mode would need adding a full on new architecture/model class to our backend, which doesn't support them for now, adding the relevant tests in the suite, etc. (And it's not been on top of the todo list).
However, I'm hoping we'll be able to add support before next year.
I'll leave this issue open as it's on our todo for our next iteration (but don't expect super fast progress on this as it's not the top of the list).
Thank you for the reminder :)