RuntimeError The size of tensor a (291) must match the size of tensor b (19) at non-singleton dimension 1
I have been using the SentenceTransformers implementation of nomic-ai/nomic-embed-text-v1.5 but have been receiving errors through repeated usage of the encode function:
nomic_model = SentenceTransformer(
"nomic-ai/nomic-embed-text-v1.5",
trust_remote_code=True,
)
text_input = ...
embeddings = nomic_model.encode(
["search_document: " + text_input], convert_to_tensor=True
)
The following 2 error messages are what I received when running the above code with different values of text_input consecutively within 30 seconds of each other:
The size of tensor a (19) must match the size of tensor b (9) at non-singleton dimension 1
The size of tensor a (291) must match the size of tensor b (19) at non-singleton dimension 1
What's the cause of this issue and how can it be resolved? I was thinking there may have been some form of caching involved since the config.json for the model has use_cache = True, and the tensor b size in the second message matches the tensor a size in the first message, but am not sure.
The stack trace for the issue seems to go back to apply_rotary_emb function in modeling_hf_nomic_bert.py in the following line:
return torch.cat(
[x[..., :ro_dim] * cos + rotate_half(x[..., :ro_dim], interleaved) * sin, x[..., ro_dim:]],
dim=-1,
)
Can you please post the full error log?
File "cdss/vector_db.py", line 23, in embed_text
embeddings = nomic_model.encode(
File "sentence_transformers/SentenceTransformer.py", line 357, in encode
out_features = self.forward(features)
File "torch/nn/modules/container.py", line 217, in forward
input = module(input)
File "torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "torch/nn/modules/module.py", line 1520, in _call_impl
return forward_call(*args, **kwargs)
File "sentence_transformers/models/Transformer.py", line 98, in forward
output_states = self.auto_model(**trans_features, return_dict=False)
File "torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "torch/nn/modules/module.py", line 1520, in _call_impl
return forward_call(*args, **kwargs)
File "/tmp/hf_home/modules/transformers_modules/nomic-ai/nomic-embed-text-v1-unsupervised/3916676c856f1e25a4cc7a4e0ac740ea6ca9723a/modeling_hf_nomic_bert.py", line 1070, in forward
sequence_output = self.encoder(hidden_states, attention_mask=attention_mask, return_dict=return_dict)
File "torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "torch/nn/modules/module.py", line 1520, in _call_impl
return forward_call(*args, **kwargs)
File "/tmp/hf_home/modules/transformers_modules/nomic-ai/nomic-embed-text-v1-unsupervised/3916676c856f1e25a4cc7a4e0ac740ea6ca9723a/modeling_hf_nomic_bert.py", line 957, in forward
hidden_states, hidden_states2, residual = layer(
File "torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "torch/nn/modules/module.py", line 1520, in _call_impl
return forward_call(*args, **kwargs)
File "/tmp/hf_home/modules/transformers_modules/nomic-ai/nomic-embed-text-v1-unsupervised/3916676c856f1e25a4cc7a4e0ac740ea6ca9723a/modeling_hf_nomic_bert.py", line 890, in forward
attn_outputs = self.attn(
File "torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "torch/nn/modules/module.py", line 1520, in _call_impl
return forward_call(*args, **kwargs)
File "/tmp/hf_home/modules/transformers_modules/nomic-ai/nomic-embed-text-v1-unsupervised/3916676c856f1e25a4cc7a4e0ac740ea6ca9723a/modeling_hf_nomic_bert.py", line 782, in forward
qkv = self.rotary_emb(qkv, seqlen_offset=past_len)
File "torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "torch/nn/modules/module.py", line 1520, in _call_impl
return forward_call(*args, **kwargs)
File "/tmp/hf_home/modules/transformers_modules/nomic-ai/nomic-embed-text-v1-unsupervised/3916676c856f1e25a4cc7a4e0ac740ea6ca9723a/modeling_hf_nomic_bert.py", line 619, in forward
q_rot = apply_rotary_emb(qkv[:, :, 0], self._cos_cached, self._sin_cached, seqlen_offset, self.interleaved)
File "/tmp/hf_home/modules/transformers_modules/nomic-ai/nomic-embed-text-v1-unsupervised/3916676c856f1e25a4cc7a4e0ac740ea6ca9723a/modeling_hf_nomic_bert.py", line 506, in apply_rotary_emb
[x[..., :ro_dim] * cos + rotate_half(x[..., :ro_dim], interleaved) * sin, x[..., ro_dim:]],
Hm i'm unable to reproduce your error
>>> from sentence_transformers import SentenceTransformer
>>>
>>> nomic_model = SentenceTransformer(
... "nomic-ai/nomic-embed-text-v1.5",
... trust_remote_code=True,
... )
modules.json: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 255/255 [00:00<00:00, 1.57MB/s]
config_sentence_transformers.json: 100%|███████████████████████████████████████████████████████████████████████████████████████████████| 140/140 [00:00<00:00, 1.01MB/s]
You try to use a model that was created with version 2.4.0.dev0, however, your version is 2.3.1. This might cause unexpected behavior or errors. In that case, try to update to the latest version.
README.md: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 69.0k/69.0k [00:00<00:00, 35.7MB/s]
sentence_bert_config.json: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████| 120/120 [00:00<00:00, 773kB/s]
<All keys matched successfully>
tokenizer_config.json: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████| 1.19k/1.19k [00:00<00:00, 8.67MB/s]
vocab.txt: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 232k/232k [00:00<00:00, 25.9MB/s]
tokenizer.json: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 711k/711k [00:00<00:00, 18.1MB/s]
special_tokens_map.json: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████| 695/695 [00:00<00:00, 6.84MB/s]
1_Pooling/config.json: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████| 286/286 [00:00<00:00, 2.64MB/s]
>>> text_input = "hi im zach"
>>> embeddings = nomic_model.encode(
... ["search_document: " + text_input], convert_to_tensor=True
... )
I'm using versions
sentence-transformers 2.3.1
transformers 4.37.2
It might be worth trying to redownload the model and wiping the current model from your disk. it seems like there's some diff
I'm using the following versions of the packages you mentioned:
sentence-transformers 2.6.1
transformers 4.39.3
Also, I am applying multithreading to call the encode method from multiple threads. I wasn't sure if the multithreading could contribute to the problem, but its not clear that error message I saw would be related to that.
Do you still see the same error if you downgrade versions?
We did not experience any issues when testing on other versions of these 2 packages. However, it seemed to fail periodically (locally and in production) when used within a multithreaded web application context wherein multiple threads were attempting to call the encode function. For the time being, we decided to resort to using another model. I'll close this discussion for now as we couldn't find a particular cause for the issue, but let us know if you are able to find us a possible cause for this error.
I'm happy to hop on a call (please email zach@nomic.ai) and try to help, however I'm not able to reproduce your issue so I'm not able to provide any guidance