Edit model card

TiRoBERTa BiEncoder Model

This is a sentence-transformers model for the Tigrinya language based on TiRoBERTa-base. The maps sentences & paragraphs to a 768 dimensional dense vector space and can be used for tasks like clustering or semantic search.

Using Sentence-Transformers

Using this model becomes easy when you have sentence-transformersinstalled:

pip install -U sentence-transformers

Then use the model as follows:

from sentence_transformers import SentenceTransformer

sentences = ["ሓደ ሰብኣይ ፈረስ ይጋልብ ኣሎ።", "ሓንቲ ጓል ክራር ትጻወት ኣላ።"]
model = SentenceTransformer('fgaim/tiroberta-bi-encoder')
embeddings = model.encode(sentences)

Using 🤗 Transformers

Use the transformers library as follows: Pass the input through the transformer model, then you have to apply the right pooling-operation on-top of the contextualized word embeddings.

import torch
from transformers import AutoModel, AutoTokenizer

# Mean Pooling - Take attention mask into account for correct averaging
def mean_pooling(model_output, attention_mask):
    token_embeddings = model_output[0]  # First element of model_output contains all token embeddings
    input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
    return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(input_mask_expanded.sum(1), min=1e-9)

# Sentences we want sentence embeddings for
sentences = ["ሓደ ሰብኣይ ፈረስ ይጋልብ ኣሎ።", "ሓንቲ ጓል ክራር ትጻወት ኣላ።"]

# Load model from HuggingFace Hub
tokenizer = AutoTokenizer.from_pretrained("fgaim/tiroberta-bi-encoder")
model = AutoModel.from_pretrained("fgaim/tiroberta-bi-encoder")

# Tokenize sentences
encoded_input = tokenizer(sentences, padding=True, truncation=True, return_tensors="pt")

# Compute token embeddings
with torch.no_grad():
    model_output = model(**encoded_input)

# Perform pooling. In this case, mean pooling.
sentence_embeddings = mean_pooling(model_output, encoded_input["attention_mask"])

print("Sentence embeddings:", sentence_embeddings)


Base Model

The model properties:

Model Size Layers Attn. Heads Hidden Size FFN Parameters Max. Seq
BASE 12 12 768 3072 125M 512

BiEncoder Model

  • Max Seq Length: 512
  • Word embedding dimension: 768
            'max_seq_length': 512,
            'do_lower_case': False
    ) # with Transformer model: RobertaModel

            'word_embedding_dimension': 768,
            'pooling_mode_cls_token': False,
            'pooling_mode_mean_tokens': True,
            'pooling_mode_max_tokens': False,
            'pooling_mode_mean_sqrt_len_tokens': False,
            'pooling_mode_weightedmean_tokens': False,
            'pooling_mode_lasttoken': False,
            'include_prompt': True,


If you use this model in your product or research, you can cite it as follows:

  author={Fitsum Gaim and Wonsuk Yang and Jong C. Park},
  title={Monolingual Pre-trained Language Models for Tigrinya},
  publisher={WiNLP 2021 co-located EMNLP 2021}
Downloads last month
Model size
125M params
Tensor type
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social visibility and check back later, or deploy to Inference Endpoints (dedicated) instead.