File size: 1,519 Bytes
1f41240 95fbcf3 6471836 1f41240 95fbcf3 6471836 95fbcf3 6471836 95fbcf3 6471836 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 |
---
license: apache-2.0
language:
- en
pipeline_tag: sentence-similarity
inference: false
---
# Monarch Mixer-BERT
The 80M checkpoint for M2-BERT-base from the paper [Monarch Mixer: A Simple Sub-Quadratic GEMM-Based Architecture](https://arxiv.org/abs/2310.12109).
This model has been pretrained with sequence length 2048, and it has been fine-tuned for long-context retrieval.
This model was trained by Jon Saad-Falcon, Dan Fu, and Simran Arora.
Check out our [GitHub](https://github.com/HazyResearch/m2/tree/main) for instructions on how to download and fine-tune it!
## How to use
You can load this model using Hugging Face `AutoModel`:
```python
from transformers import AutoModelForMaskedLM
model = AutoModelForMaskedLM.from_pretrained("togethercomputer/m2-bert-80M-2k-retrieval", trust_remote_code=True)
```
This model generates embeddings for retrieval. The embeddings have a dimensionality of 768:
```
from transformers import AutoTokenizer, AutoModelForMaskedLM
max_seq_length = 2048
testing_string = "Every morning, I make a cup of coffee to start my day."
model = AutoModelForMaskedLM.from_pretrained("togethercomputer/m2-bert-80M-2k-retrieval", trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased", model_max_length=max_seq_length)
input_ids = tokenizer([testing_string], return_tensors="pt", padding="max_length", return_token_type_ids=False, truncation=True, max_length=max_seq_length)
outputs = model(**input_ids)
embeddings = outputs['sentence_embedding']
```
|