How to fine-tune this model?
#2
by
alvations
- opened
When trying to finetune the model with accelerate, it's throwing an error:
>>> trainer.train()
...
Traceback (most recent call last):
...
File "../models/modules/transformers_modules/ltg/deberta-xxlarge-fixed/.../modeling_deberta.py", line 1312 in forward
assert labels is None, "only inference is supported for now"
AssertError: only inference is supported for now
Any plans to let users finetune the model? Or is it just inference only for the near future ~1-2 months?
Hi, to be honest, I don't know how to finetune DeBERTa on a causal-LM task and I haven't even tried :) One possibility is to simply finetune it with a MLM objective and then use the generation method proposed in our paper; another possibility is to finetune it on a prefix-LM objective and then use standard autoregressive generation. And there are probably many other possibilities, it's an uncharted territory.