enpaiva/embed-andegpt-H768

andegpt-embed

This is a sentence-transformers model finetuned from microsoft/mpnet-base. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: microsoft/mpnet-base
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 768 tokens
• Similarity Function: Cosine Similarity
• Language: es
• License: apache-2.0

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: MPNetModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("enpaiva/embed-andegpt-280724")
# Run inference
sentences = [
    '¿Cuál es el número del artículo que trata sobre la mínima sección permisible para una lámpara o grupo de lámparas?',
    'Reglamento de Baja Tensión de la ANDE: El 14.7.3 trata sobre: La mínima sección permisible para una lámpara, o grupo de lámparas que forman un solo artefacto de iluminación, será de 1 mm².',
    'Reglamento de Baja Tensión de la ANDE: El 19.2.1 trata sobre: La caída de tensión máxima permisible, es la siguiente:  a) Para iluminación, en general (19.1.1), hasta 4%. -2% en el alimentador, y -2% en el circuito (19.1.2). b) Para fuerza motriz y/o calefacción, hasta 5%. -4% en el alimentador, y -1% en el ramal. c) En el caso de clientes que reciban la energía a tensión diferente de las normales de utilización (19.1.3), hasta 4%.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Triplet

• Dataset: andegpt-dev
• Evaluated with TripletEvaluator

Metric	Value
cosine_accuracy	0.9971
dot_accuracy	0.0032
manhattan_accuracy	0.9968
euclidean_accuracy	0.9971
max_accuracy	0.9971

Training Details

Training Hyperparameters

Non-Default Hyperparameters

• prediction_loss_only: False
• learning_rate: 2e-05
• lr_scheduler_type: cosine
• log_level_replica: passive
• log_on_each_node: False
• logging_nan_inf_filter: False
• bf16: True
• batch_sampler: no_duplicates

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• prediction_loss_only: False
• per_device_train_batch_size: 8
• per_device_eval_batch_size: 8
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• learning_rate: 2e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 3
• max_steps: -1
• lr_scheduler_type: cosine
• lr_scheduler_kwargs: {}
• warmup_ratio: 0
• warmup_steps: 0
• log_level: passive
• log_level_replica: passive
• log_on_each_node: False
• logging_nan_inf_filter: False
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: True
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: False
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• dispatch_batches: None
• split_batches: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional

Training Logs

Epoch	Step	Training Loss	loss	andegpt-dev_max_accuracy
0	0	-	-	0.6136
0.0270	250	0.8269	0.3100	0.9658
0.0540	500	0.3667	0.2169	0.9721
0.0809	750	0.2305	0.1594	0.9801
0.1079	1000	0.1866	0.1372	0.9830
0.1349	1250	0.1639	0.1114	0.9859
0.1619	1500	0.1375	0.0983	0.9871
0.1889	1750	0.1082	0.0815	0.9886
0.2158	2000	0.1023	0.0723	0.9900
0.2428	2250	0.0777	0.0703	0.9905
0.2698	2500	0.0809	0.0656	0.9896
0.2968	2750	0.0639	0.0662	0.9891
0.3238	3000	0.0633	0.0590	0.9922
0.3507	3250	0.0545	0.0533	0.9930
0.3777	3500	0.0541	0.0458	0.9932
0.4047	3750	0.0475	0.0365	0.9947
0.4317	4000	0.0394	0.0330	0.9939
0.4587	4250	0.0561	0.0345	0.9939
0.4856	4500	0.0432	0.0327	0.9942
0.5126	4750	0.0417	0.0328	0.9944
0.5396	5000	0.0388	0.0252	0.9949
0.5666	5250	0.033	0.0284	0.9959
0.5936	5500	0.0243	0.0229	0.9964
0.6205	5750	0.023	0.0223	0.9959
0.6475	6000	0.0313	0.0209	0.9966
0.6745	6250	0.0285	0.0208	0.9961
0.7015	6500	0.022	0.0192	0.9961
0.7285	6750	0.0219	0.0235	0.9956
0.7555	7000	0.0258	0.0186	0.9954
0.7824	7250	0.0226	0.0230	0.9959
0.8094	7500	0.0226	0.0240	0.9961
0.8364	7750	0.0208	0.0173	0.9968
0.8634	8000	0.0147	0.0200	0.9956
0.8904	8250	0.0193	0.0147	0.9971
0.9173	8500	0.0254	0.0136	0.9968
0.9443	8750	0.0148	0.0132	0.9971
0.9713	9000	0.0174	0.0157	0.9968
0.9983	9250	0.0221	0.0144	0.9971

Framework Versions

• Python: 3.11.0
• Sentence Transformers: 3.0.1
• Transformers: 4.39.3
• PyTorch: 2.2.0+cu121
• Accelerate: 0.28.0
• Datasets: 2.20.0
• Tokenizers: 0.15.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply}, 
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}