KielEmbed-Code

A specialized dense embedding model fine-tuned from microsoft/codebert-base for code and text representation.

This is a sentence-transformers model fine-tuned from microsoft/codebert-base. It maps sentences and code blocks into a 768-dimensional dense vector space optimized for semantic textual similarity, semantic search, and clustering tasks.

Model Details

Model Description

• Model Type: Sentence Transformer / Dense Embedding Backbone
• Base Model: microsoft/codebert-base
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity
• Supported Modality: Text & Code

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face Hub: kiel/KielEmbed-Code

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'RobertaModel'})
  (1): Pooling({'embedding_dimension': 768, 'pooling_mode': 'mean', 'include_prompt': True})
)
UsageDirect Usage (Sentence Transformers)First, install the Sentence Transformers library:Bashpip install -U sentence-transformers
Then load your model and run inference:Pythonfrom sentence_transformers import SentenceTransformer

# Load your custom fine-tuned CodeBERT model from the Hugging Face Hub
model = SentenceTransformer("kiel/KielEmbed-Code")

# Run inference
sentences = [
    '" Any decision on Charleroi will have huge implications for regional airports in France , " he said .',
    '" A bad decision on Charleroi would have huge implications for state-owned regional airports in France .',
    "He said the ferry 's crew will be interviewed and tested for drugs and alcohol .",
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
Training DetailsTraining DatasetKiel Code & Semantic CorpusSize: 10,000 training samplesColumns: text1, text2, and labelApproximate Token Statistics (First 100 samples):Text 1: Min: 12 tokens | Mean: 27.7 tokens | Max: 41 tokensText 2: Min: 15 tokens | Mean: 27.64 tokens | Max: 46 tokensLabel Distribution: Class 0 (~34.62%), Class 1 (~65.38%)Loss Function: CosineSimilarityLoss with parameters:JSON{
    "loss_fct": "torch.nn.modules.loss.MSELoss",
    "cos_score_transformation": "torch.nn.modules.linear.Identity"
}
Training HyperparametersPer Device Train Batch Size: 8Gradient Accumulation Steps: 4 (Effective batch size = 32)Learning Rate: 2e-05Number of Epochs: 1Warmup Steps: 0.1Mixed Precision: FP16 EnabledOptimizer: adamw_torch_fusedTraining LogsEpochStepTraining Loss0.165028.07520.3210024.76680.4815021.00550.6420026.55380.825026.06000.9630026.3027Total Training Time: 5.1 minutesFramework VersionsPython: 3.13.15Sentence Transformers: 5.7.0Transformers: 5.16.1PyTorch: 2.11.0+cu128Accelerate: 1.14.0Datasets: 4.8.5Tokenizers: 0.23.1Additional ResourcesTraining and Finetuning Embedding Models with Sentence Transformers: End-to-end guide for fine-tuning Sentence Transformer models.CitationBibTeXCode snippet@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "[https://arxiv.org/abs/1908.10084](https://arxiv.org/abs/1908.10084)",
}