Update Files
As instructed by
@Rocketknight1
, here this PR updates the modeling and config files for Phi-1.5. Also, it updates the new weights and necessary files.
did these changes just break all peft finetuned models? I'm now getting
microsoft/phi-1_5 does not appear to have a file named configuration_mixformer_sequential.py.
Checkout 'https://huggingface.co/microsoft/phi-1_5/main' for available files.
when I run ArtifactAI/phi-arxiv-physics-instruct. Swapping in the new config results in gibberish
No, they are not merged yet. I think this issue is related to problem you are facing.
I just proposed these changes to be able to transfer the weights to their official repo.
BTW you can now load phi from the transformers library itself :)
got it thank you!
I've completed a review here - the port looks good, and outputs should be unchanged compared to the original version. The main difference is that past_key_values is returned by default now, but this isn't a significant issue I think.
I'd support merging this (and the Phi-1 PR) once we do the next release of transformers and add library support for the class! The next release should be due either this or next week.
Thanks for the work everyone! I agree with what you said, and let's merge as soon as transformers release their 4.36+ version.
Hi @gugarosa ,
Will this extra functionality be added to the code on the Hub or directly to the transformers library?
Also, let me know if I could be of any help.
I think transformers will be better, so we can gradually move away from having two distinct codebases.
It is pretty simple to be honest, it is only the support to use MQA/GQA because we might need that soon.
Hi @gugarosa , if it's ok with you, I can open a PR to the library to add support for MQA and GQA.
Of course! Can we work together on https://github.com/huggingface/transformers/pull/28163? I added this morning a starting point which should make our lives easier :D
Even with the updated PR & installing transformers from source, i get the same error:
OSError: microsoft/phi-1_5 does not appear to have a file named configuration_mixformer_sequential.py. Checkout 'https://huggingface.co/microsoft/phi-1_5/main' for available files.