Hugging Face's logo

tiiuae
/
falcon-7b

Text Generation
Transformers
PyTorch
English
falcon
custom_code
text-generation-inference
6 papers
Model card Files Community
102

Update RotaryEmbedding caching

#33
by beibin79 - opened
base: refs/heads/main
←
from: refs/pr/33
Discussion Files changed
+11
-15
beibin79
  1. Save computation for multiple alternating calls with the same sequence length.
  2. Debug: the cos_sin(...) function could return (None, None) during multi-threading on some hardware. The latest dict design can resolve this issue.
Update RotaryEmbedding cachingb2c55572
Cannot merge
This branch has merge conflicts in the following files:
  • modeling_falcon.py

Sign up or log in to comment