Spaces:
Paused
vercel deployment's chat stream is choppy
I am deploying chat-ui on vercel, and the output stream generation shown on UI is really choppy. Does anybody have any idea on how it can be solved? Does anybody here have a good deployment on vercel without this issue?
It works perfectly on my local machine.
My guess is that there is a delay from huggingface inference to vercel, and that is why only sections of the generated text is seen during streaming. But how to solve that?
e.g. the stream would show something like this, which does not make sense:
----------------------------------------------
The is cat namedo." is, it is true that cat
----------------------------------------------
it does not make sense, because when the stream finishes, it shows this instead:
----------------------------------------------
The answer to the question "there is a cat, named nemo." is: Yes, it is true that there exists a cat named Nemo.
----------------------------------------------
here is another example, does not make sense, during the stream:
----------------------------------------------
Thats! indeed is cat named. else would like to?
----------------------------------------------
final reply after the stream is completed:
----------------------------------------------
That's correct! There indeed is a cat named Nemo. Is there anything else you would like to know?
----------------------------------------------
@willyninja30
had the same issue, however, his was having problems on netlify, and works fine on vercel.
https://huggingface.co/spaces/huggingchat/chat-ui/discussions/176#64996d1fa3f227b05cbe7fca
not sure how he got it to work on vercel.
@willyninja30
would you know?
I have been trying to find a solution for some time, your help is greatly appreciated!
I am deploying chat-ui on vercel, and the output stream generation shown on UI is really choppy. Does anybody have any idea on how it can be solved? Does anybody here have a good deployment on vercel without this issue?
It works perfectly on my local machine.
My guess is that there is a delay from huggingface inference to vercel, and that is why only sections of the generated text is seen during streaming. But how to solve that?
e.g. the stream would show something like this, which does not make sense:
----------------------------------------------
The is cat namedo." is, it is true that cat
----------------------------------------------it does not make sense, because when the stream finishes, it shows this instead:
----------------------------------------------
The answer to the question "there is a cat, named nemo." is: Yes, it is true that there exists a cat named Nemo.
----------------------------------------------here is another example, does not make sense, during the stream:
----------------------------------------------
Thats! indeed is cat named. else would like to?
----------------------------------------------final reply after the stream is completed:
----------------------------------------------
That's correct! There indeed is a cat named Nemo. Is there anything else you would like to know?
----------------------------------------------@willyninja30 had the same issue, however, his was having problems on netlify, and works fine on vercel.
https://huggingface.co/spaces/huggingchat/chat-ui/discussions/176#64996d1fa3f227b05cbe7fca
not sure how he got it to work on vercel. @willyninja30 would you know?I have been trying to find a solution for some time, your help is greatly appreciated!
Hello, we are deploying on vercel without any issue. I'm not sure to understand, can you say in one sentence what's the issue you have so I can help?
The issue was fixed with one of the PRs about a week or week after this was originally posted. It is no longer reproducible.
It was fixed around December.