nomic-ai

/

nomic-bert-2048

like 16

apologies this isn’t working for you. can you tell me how you’re launching the code/what your config looks like? it looks like it may be missing the bert prefix

No need to apologize, I appreciate the hard work (and amazingly swift response).
I was lazy, so I just added the sequence classification class to the config and called it with AutoModelForSequenceClassification:

model_path = "nomic-ai/nomic-bert-2048"
config = AutoConfig.from_pretrained(model_path, trust_remote_code=True)
config.auto_map["AutoModelForSequenceClassification"] = model_path + '--modeling_hf_nomic_bert.NomicBertForSequenceClassification'
model = AutoModelForSequenceClassification.from_pretrained(model_path, config=config, trust_remote_code=True)

Oh, I just saw this line:

remove_bert_prefix = cls != NomicBertForPreTraining

Feeling dumb 🤦‍♂️

i mean this code can be written in a much better way :)
if you look at the next line, it adds the bert prefix if the class is the sequence classification class. i’m surprised it’s failing though, I can check in the am on why this isn’t working

oh actually it seems like the codes not fully up to date with https://github.com/nomic-ai/contrastors/blob/main/src/contrastors/models/encoder/modeling_nomic_bert.py

i’ll fix this in the am

in the meantime you can update the local file that huggingface downloads for you to match more closely what’s in the repo ^

Thanks, I'll see if I can sort this out. I have to admit, this is kinda funny:

    if remove_bert:

        def remove_bert_prefix(key):
            key = re.sub(r"^bert.", "", key)
            return key

        state_dict = OrderedDict((remove_bert_prefix(k), v) for k, v in state_dict.items())

    if add_bert:

        def add_bert_prefix(key):
            # prepend bert. to the key
            if key.startswith("bert."):
                return key
            return f"bert.{key}"

        state_dict = OrderedDict((add_bert_prefix(k), v) for k, v in state_dict.items())

yeah...the code isn't great lol

ok i appreciate your patience :) it should work now
i ran with the following code

from transformers import AutoConfig, AutoModelForSequenceClassification
model_path = "nomic-ai/nomic-bert-2048"
config = AutoConfig.from_pretrained(model_path, trust_remote_code=True)
model = AutoModelForSequenceClassification.from_pretrained(model_path, config=config, trust_remote_code=True, strict=False)

I've updated the readme as well to reflect this!

I also cleaned up the code a bit thanks for pointing it out! let me know if you run into other issues

Legend. I got it running before you made the fix, but I'll switch across.