olivierdehaene
/

optimized-santacoder

Text Generation

text-generation-inference

Model card Files Files and versions Community

OlivierDehaene commited on Jan 23, 2023

Commit

fba6000

•

1 Parent(s): 6102f15

fix forward

Files changed (1) hide show

modeling_gpt2_mq.py +1 -1

modeling_gpt2_mq.py CHANGED Viewed

@@ -148,7 +148,7 @@ class GPT2MQAttention(nn.Module):
         # (b, sq * num_heads, head_dim) x (b, head_dim, sk) -> (b, sq * num_heads, sk)
         if self.scale_attn_weights:
-            query *= self.inv_norm_factor
         attn_weights = torch.bmm(query, key)

         # (b, sq * num_heads, head_dim) x (b, head_dim, sk) -> (b, sq * num_heads, sk)
         if self.scale_attn_weights:
+            query = query * self.inv_norm_factor
         attn_weights = torch.bmm(query, key)