Edit model card

TiELECTRA BiEncoder Model

This model is a sentence-transformers model for the Tigrinya language based on TiELECTRA-small. The maps sentences & paragraphs to a 256 dimensional dense vector space and can be used for tasks like clustering or semantic search.

Using Sentence-Transformers

Using this model becomes easy when you have sentence-transformersinstalled:

pip install -U sentence-transformers

Then use the model as follows:

from sentence_transformers import SentenceTransformer
sentences = ["ሓደ ሰብኣይ ፈረስ ይጋልብ ኣሎ።", "ሓንቲ ጓል ክራር ትጻወት ኣላ።"]

model = SentenceTransformer('fgaim/tielectra-bi-encoder')
embeddings = model.encode(sentences)

Using 🤗 Transformers

Use the transformers library as follows: Pass the input through the transformer model, then you have to apply the right pooling-operation on-top of the contextualized word embeddings.

import torch
from transformers import AutoModel, AutoTokenizer

# Mean Pooling - Take attention mask into account for correct averaging
def mean_pooling(model_output, attention_mask):
    token_embeddings = model_output[0]  # First element of model_output contains all token embeddings
    input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
    return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(input_mask_expanded.sum(1), min=1e-9)

# Sentences we want sentence embeddings for
sentences = ["ሓደ ሰብኣይ ፈረስ ይጋልብ ኣሎ።", "ሓንቲ ጓል ክራር ትጻወት ኣላ።"]

# Load model from HuggingFace Hub
tokenizer = AutoTokenizer.from_pretrained("fgaim/tielectra-bi-encoder")
model = AutoModel.from_pretrained("fgaim/tielectra-bi-encoder")

# Tokenize sentences
encoded_input = tokenizer(sentences, padding=True, truncation=True, return_tensors="pt")

# Compute token embeddings
with torch.no_grad():
    model_output = model(**encoded_input)

# Perform pooling. In this case, mean pooling.
sentence_embeddings = mean_pooling(model_output, encoded_input["attention_mask"])

print("Sentence embeddings:", sentence_embeddings)


Base Model

The model properties:

Model Size Layers Attn. Heads Hidden Size FFN Parameters Max. Seq
SMALL 12 4 256 1024 14M 512

BiEncoder Model

  • Max Seq Length: 512
  • Word embedding dimension: 256
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: ElectraModel

  (1): Pooling({'word_embedding_dimension': 256, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})


If you use this model in your product or research, you can cite it as follows:

  author={Fitsum Gaim and Wonsuk Yang and Jong C. Park},
  title={Monolingual Pre-trained Language Models for Tigrinya},
  publisher={WiNLP 2021 co-located EMNLP 2021}
Downloads last month
Model size
13.5M params
Tensor type
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social visibility and check back later, or deploy to Inference Endpoints (dedicated) instead.